A fixed CKAD set (ids 38-57, 140 points, 120 minutes), run with
`./exam sim a`. It is kept out of the random pool, so reruns are
comparable. The scenarios are original - a fictional river-freight company,
one Namespace per river - and practise the techniques the killer.sh CKAD
environment is known for without reusing its questions: exact names and
file outputs, commands written to scripts, several steps per task.
It covers what the pool never asks: debugging a container with no shell
through an ephemeral container (the value lives only in the process's
environment), a Kustomize patch with a hashed configMapGenerator, a Helm
release stuck in `failed`, a CronJob run by hand while suspended, fsGroup
on a shared volume, RBAC proven by the app's own log, Pods rejected by a
LimitRange, and the two-source NetworkPolicy trap (enforced by kindnet).
Verified on a throwaway cluster: every task scores 0 untouched and 7/7 when
solved, and a full sitting (baseline, seed all 20, solve, grade at the end)
scored 140/140 twice on the final code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VchE1CFkJqwYhXSMTPZUtD
The CKAD track was a 12-task starter bank whose quota (3/3/2/2/2) drew the
whole bank, so every mock was the same 12 tasks reshuffled, and about half
of the curriculum had no task at all.
Tasks 13-37 (tracks/ckad/bank_extra.py) give every missing competency at
least one task: build and export an image with podman (OCI archive), Job
completions and parallelism, a DaemonSet on every node, a native sidecar
that lets a Job finish, a PVC and an in-memory emptyDir, a canary by replica
ratio, rolling-update limits, a Kustomize overlay, a Helm values file and an
upgrade that keeps values, manifests on removed APIs, kubectl top, logs of a
crashing container, two stacked faults, OOMKilled, a custom resource, RBAC
for a ServiceAccount, Pod Security admission, ResourceQuota with LimitRange
defaults, Guaranteed QoS, ConfigMaps, Secrets, ServiceAccount tokens, an
egress NetworkPolicy tested with real traffic, a Service with two faults and
an Ingress with TLS.
A mock now draws 16 tasks weighted like the exam (DESIGN 3, DEPLOY 3, OBS 3,
ENV 4, NET 3) from a pool of 8/7/7/10/5. New ensure_metrics_server helper so
`kubectl top` works on a freshly built cluster. Starter tasks now delete
their Pods with --wait=true, so a re-seed cannot apply onto a Pod that is
still terminating, and setups that create bare Pods fail loudly if one is
missing. README gains a CKAD section.
Verified on a throwaway cluster: all 12 starter and 25 new tasks score 0
untouched and full when solved; four full 16-task mocks that together
include all 37 tasks scored 122/122, 118/118, 124/124 and 119/119.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mhxm4rsUfuE5HoytYbu2YW
Every changed task was seeded, graded untouched, solved and re-graded on a
separate kind cluster. The four sim sets were rehearsed as full sittings (real
baseline, seed everything, solve in order, grade at the end): A 150/150,
B 176/176, C 163/163, D 122/122. Every random-pool task was either passed in a
real session or cycled the same way.
Graders that could never pass, or passed for free:
Q306 (local-path cannot resize; now graded as an in-place resize), Q314 (GNU
grep prefixed file names), Q305 (plain `get events` has no SOURCE column and
events expire; now managedFields), Q410 (kindnet DOES enforce NetworkPolicy;
the expected answer is now derived from the CNI), Q35, Q212 (missing `]`),
Q101 (`can-i create pods/exec` asks about a pod NAMED exec; multi-type get),
Q109 (an emptyDir over nginx's /var/log crash-loops it), Q40 (TTL was
measured at grading time), Q121, Q106, Q104, Q41, Q26, Q403, Q409, Q417,
Q113, Q22, Q24/Q108 (flaky right after a rollout), and free points on
untouched Q24/Q102/Q108/Q119/Q122.
Reference solutions: S[43], S[107], S[121], S[106] rewritten; S[116] (a
Completed Job pod still pins its PVC), S[119] (a pods quota equal to replicas
deadlocks the rollout); DNS checks use FQDNs; the explainer and exam guide
are corrected (etcd restore is etcdutl's job on 3.6+).
Cross-session leaks: the pre-session baseline now also repairs Q114 (the
kube-system NetworkPolicy and CoreDNS replicas), Q117/Q39 kubelet CA paths,
Q317, kubelet drop-in binary paths, a moved scheduler manifest, leftover
static pods and the default StorageClass. Q115 releases its fixed PVs from
claims left by earlier sessions.
Conflicts: `breaks` may be a list and `solo` keeps Q102's etcd restore out of
mixed sessions. Q118, Q5/Q103, Q26 vs the W1 taints, set A 203/204/208/212 and
set B 305/309 no longer sabotage each other. Q317 now disables only CSR
signing (the old break killed the whole controller-manager). Q110/Q311 no
longer compare against the whole cluster at submit time. Two delete/recreate
races in setups are fixed.
Harness: the weak list clears after two clean passes in a row, weak entries
show the latest attempt's failures, domain scores use the exam's own tasks,
history entries can be marked synthetic (entry 11 is), the web UI lists
every sim set, `./exam grade` refuses mid-exam (--force), preflight checks W1.
Set D (`./exam sim d`): 17 original killer.sh-style tasks, ids 501-517,
122 points - control-plane inventory, event forensics with crictl, a broken
kubelet unit, kubelet certificates, manual scheduling with the scheduler
stopped, Helm housekeeping with a stuck release, an enforced AND-vs-OR
NetworkPolicy, dnsPolicy and access-mode troubleshooting, and more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audit of the bank and harness after a 23% mock result. Two of these were
silently corrupting every practice session on this cluster.
restore_baseline:
- The w1-taint cleanup removed nothing at all. `kubectl taint node N a- b- c-`
aborts on the first key that is not present and applies none of the removals,
so with only `workload` set the command reported five "not found" lines,
exited 1, and left the taint in place - the precise failure the block's own
comment says it exists to prevent. W1 then rejected pods in every later
session. Now one call per key. Verified by re-seeding workload=gpu:NoSchedule
and confirming a single restore_baseline run clears it.
- metrics-server was never repaired. Q23 seeds its scenario by stripping
--kubelet-insecure-tls, but carried no "breaks" key, so nothing tracked it
and nothing undid it. On kind the kubelet serving certs have no IP SANs, so
without that flag metrics-server never goes Ready, v1beta1.metrics.k8s.io
has no endpoints, aggregated discovery goes stale, and EVERY namespace
deletion in the cluster hangs on NamespaceDeletionDiscoveryFailure - which
is where the ~40 stuck namespaces came from, plus a dead `kubectl top`.
Q23 now declares breaks: metrics-server and the baseline restores the flag.
Baseline runs before seeding, so a session containing Q23 re-breaks it a
moment later as intended. Verified by stripping the flag and confirming one
restore_baseline run puts it back.
Session composition:
- pick() dropped any domain whose tasks are all "hard" from a normal session.
CKS SUPPLY and RUNTIME hold one hard task each, so `./exam start` returned
6 of 9 tasks with 40% of the blueprint silently absent. It now tops up from
the hard pool when a domain's normal pool is short.
- compose_exam() iterated QUOTA in declaration order, so a one-task domain
lost every "breaks" tie to a bigger domain that claimed the shared slot
first: CKS RUNTIME (only Q9, apiserver-flag) was never drawn in ANY mode
because SETUP went first and took that slot with Q2. Now walks domains
scarcest-first, and warns on a shortfall instead of truncating in silence.
CKS now yields 8/9 covering all six domains, warning SETUP 1/2.
- session_questions() rebuilt the list by filtering QUESTIONS (sorted by id),
discarding the session's own order - so compose_exam's rng.shuffle never
reached the candidate and every exam arrived grouped by domain and
difficulty, base tasks then hard then sim sets. Fixed to preserve the stored
order, and cmd_drill now shuffles explicit-id drills after conflict
resolution, so which task gets dropped stays deterministic while the order
does not.
Bank correctness (verified by grading each reference solution):
- CKAD Q2 was unpassable, 5/8. Its init container waited on `until nslookup
backend`, but busybox nslookup exits 1 even on a successful lookup because
it walks the whole search list and the trailing NXDOMAINs set the status -
the loop never terminates. The premise was wrong too: a ClusterIP Service
resolves from creation, endpoints or not, so DNS could never have gated it.
Now waits on a real connection; 8/8, and the pod genuinely holds at Init:0/1
until the backend Deployment exists. Same bug class as e3f5ecd.
- CKS Q1 was unsatisfiable as written, 6/10. The text asked for three
NetworkPolicies but the grader required a fourth: one podSelector cannot
cover both the web egress half and the db ingress half. Task text now asks
for four and says why.
- CKS Q4 check 4 was a false positive. `... || r=0` read a failed exec as
"no token mounted", awarding 3 points when the candidate had done nothing
and the probe pod simply could not start - which happens often on this
cluster. Now requires the pod to reach Running and exec to work. Verified
it awards the points before the fix and fails after, with pod creation
blocked by a quota.
- CKAD Q8's solution asserted a Pod's image cannot be changed in place.
It can - spec.containers[].image is mutable; `kubectl set image pod/...`
works and grades 7/7. Rewritten, noting which fields really are immutable.
Also: the `sim` usage and help strings hardcoded "a|b" and a "17-task" count,
wrong for a third set and wrong for CKAD/CKS, which define no sets at all. Both
are now generated from SETS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
