fix: taint cleanup removed nothing, Q23 leaked a broken metrics-server, CKS lost 40% of its blueprint
Audit of the bank and harness after a 23% mock result. Two of these were
silently corrupting every practice session on this cluster.
restore_baseline:
- The w1-taint cleanup removed nothing at all. `kubectl taint node N a- b- c-`
aborts on the first key that is not present and applies none of the removals,
so with only `workload` set the command reported five "not found" lines,
exited 1, and left the taint in place - the precise failure the block's own
comment says it exists to prevent. W1 then rejected pods in every later
session. Now one call per key. Verified by re-seeding workload=gpu:NoSchedule
and confirming a single restore_baseline run clears it.
- metrics-server was never repaired. Q23 seeds its scenario by stripping
--kubelet-insecure-tls, but carried no "breaks" key, so nothing tracked it
and nothing undid it. On kind the kubelet serving certs have no IP SANs, so
without that flag metrics-server never goes Ready, v1beta1.metrics.k8s.io
has no endpoints, aggregated discovery goes stale, and EVERY namespace
deletion in the cluster hangs on NamespaceDeletionDiscoveryFailure - which
is where the ~40 stuck namespaces came from, plus a dead `kubectl top`.
Q23 now declares breaks: metrics-server and the baseline restores the flag.
Baseline runs before seeding, so a session containing Q23 re-breaks it a
moment later as intended. Verified by stripping the flag and confirming one
restore_baseline run puts it back.
Session composition:
- pick() dropped any domain whose tasks are all "hard" from a normal session.
CKS SUPPLY and RUNTIME hold one hard task each, so `./exam start` returned
6 of 9 tasks with 40% of the blueprint silently absent. It now tops up from
the hard pool when a domain's normal pool is short.
- compose_exam() iterated QUOTA in declaration order, so a one-task domain
lost every "breaks" tie to a bigger domain that claimed the shared slot
first: CKS RUNTIME (only Q9, apiserver-flag) was never drawn in ANY mode
because SETUP went first and took that slot with Q2. Now walks domains
scarcest-first, and warns on a shortfall instead of truncating in silence.
CKS now yields 8/9 covering all six domains, warning SETUP 1/2.
- session_questions() rebuilt the list by filtering QUESTIONS (sorted by id),
discarding the session's own order - so compose_exam's rng.shuffle never
reached the candidate and every exam arrived grouped by domain and
difficulty, base tasks then hard then sim sets. Fixed to preserve the stored
order, and cmd_drill now shuffles explicit-id drills after conflict
resolution, so which task gets dropped stays deterministic while the order
does not.
Bank correctness (verified by grading each reference solution):
- CKAD Q2 was unpassable, 5/8. Its init container waited on `until nslookup
backend`, but busybox nslookup exits 1 even on a successful lookup because
it walks the whole search list and the trailing NXDOMAINs set the status -
the loop never terminates. The premise was wrong too: a ClusterIP Service
resolves from creation, endpoints or not, so DNS could never have gated it.
Now waits on a real connection; 8/8, and the pod genuinely holds at Init:0/1
until the backend Deployment exists. Same bug class as e3f5ecd.
- CKS Q1 was unsatisfiable as written, 6/10. The text asked for three
NetworkPolicies but the grader required a fourth: one podSelector cannot
cover both the web egress half and the db ingress half. Task text now asks
for four and says why.
- CKS Q4 check 4 was a false positive. `... || r=0` read a failed exec as
"no token mounted", awarding 3 points when the candidate had done nothing
and the probe pod simply could not start - which happens often on this
cluster. Now requires the pod to reach Running and exec to work. Verified
it awards the points before the fix and fails after, with pod creation
blocked by a quota.
- CKAD Q8's solution asserted a Pod's image cannot be changed in place.
It can - spec.containers[].image is mutable; `kubectl set image pod/...`
works and grades 7/7. Rewritten, noting which fields really are immutable.
Also: the `sim` usage and help strings hardcoded "a|b" and a "17-task" count,
wrong for a third set and wrong for CKAD/CKS, which define no sets at all. Both
are now generated from SETS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
This commit is contained in:
parent
69c8ec9557
commit
49fcb76166
69
exam
69
exam
@ -201,7 +201,12 @@ def session_questions(st: dict[str, Any] | None) -> list[dict[str, Any]]:
|
||||
if not st:
|
||||
return list(QUESTIONS)
|
||||
ids = st.get("ids") or [q["id"] for q in QUESTIONS]
|
||||
return [q for q in QUESTIONS if q["id"] in ids]
|
||||
# Preserve the session's own order. Filtering QUESTIONS (which is sorted by id) threw it away,
|
||||
# so compose_exam's rng.shuffle never reached the candidate and every exam arrived grouped by
|
||||
# domain and difficulty - base tasks first, then hard, then the sim sets. Real exams are not
|
||||
# ordered by difficulty, and practising triage on a sorted list teaches the wrong instinct.
|
||||
by_id = {q["id"]: q for q in QUESTIONS}
|
||||
return [by_id[i] for i in ids if i in by_id]
|
||||
|
||||
def q_by_id(n: int) -> dict[str, Any]:
|
||||
for q in QUESTIONS:
|
||||
@ -380,16 +385,40 @@ def no_conflicts(cands, n, taken=()):
|
||||
def pick(domain, n, hard, rng, exclude=(), taken=(), unseen_only=False):
|
||||
"""n tasks from a domain: unseen first, then least recently seen; hard sessions prefer hard tasks."""
|
||||
seen = last_seen()
|
||||
|
||||
def build(allow_hard):
|
||||
pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude
|
||||
and (hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)]
|
||||
and (allow_hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)]
|
||||
rng.shuffle(pool)
|
||||
pool.sort(key=lambda q: (0 if (hard and q["difficulty"] == "hard") else 1, seen.get(q["id"], -1)))
|
||||
return no_conflicts(pool, n, taken)
|
||||
return pool
|
||||
|
||||
got = no_conflicts(build(hard), n, taken)
|
||||
if len(got) < n and not hard:
|
||||
# A domain whose only tasks are "hard" contributes NOTHING to a normal session and drops
|
||||
# out of the blueprint entirely - CKS SUPPLY and RUNTIME (20% each) were both unreachable
|
||||
# via ./exam start. Better to hand over a hard task than to silently skip the domain.
|
||||
got = no_conflicts(build(True), n, taken)
|
||||
return got
|
||||
|
||||
def compose_exam(hard, rng):
|
||||
qs = []
|
||||
for d, n in (QUOTA_HARD if hard else QUOTA).items():
|
||||
qs += pick(d, n, hard, rng, taken=qs)
|
||||
quota = QUOTA_HARD if hard else QUOTA
|
||||
# Scarcest domain first. Two tasks that "breaks" the same thing can't share a session, so a
|
||||
# one-task domain loses every such tie to a bigger domain that claimed the shared slot earlier
|
||||
# and then contributes nothing: CKS RUNTIME (only Q9, apiserver-flag) was never drawn in ANY
|
||||
# mode because SETUP was iterated first and took that slot with Q2.
|
||||
order = sorted(quota, key=lambda d: len([q for q in POOL if q["domain"] == d]))
|
||||
qs, short = [], {}
|
||||
for d in order:
|
||||
got = pick(d, quota[d], hard, rng, taken=qs)
|
||||
if len(got) < quota[d]:
|
||||
short[d] = (len(got), quota[d])
|
||||
qs += got
|
||||
if short:
|
||||
# Never truncate in silence - a short session still gets scored out of its own total, so
|
||||
# the missing domains look like they were simply not on the blueprint.
|
||||
print(yellow(" warning: bank too small to fill the blueprint - "
|
||||
+ ", ".join(f"{d} {g}/{w}" for d, (g, w) in short.items())))
|
||||
rng.shuffle(qs)
|
||||
return qs
|
||||
|
||||
@ -612,7 +641,24 @@ done
|
||||
# w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from
|
||||
# dedicated/team/workload, Q107 from pool/lane, and Q307 pins maintenance=inprogress - miss one
|
||||
# and that node silently stops accepting pods for every later session.
|
||||
$K taint node "$W1" dedicated- workload- team- pool- lane- maintenance- >/dev/null 2>&1 || true
|
||||
# One key per call: `kubectl taint node N a- b-` aborts on the first key that is not present and
|
||||
# removes NOTHING, so the single taint that is actually set survives the "cleanup" and W1 keeps
|
||||
# rejecting pods in every later session - the exact failure this block exists to prevent.
|
||||
for _t in dedicated workload team pool lane maintenance; do
|
||||
$K taint node "$W1" "$_t-" >/dev/null 2>&1 || true
|
||||
done
|
||||
|
||||
# metrics-server: Q23 seeds its scenario by stripping --kubelet-insecure-tls, and kind's kubelet
|
||||
# serving certs have no IP SANs, so without that flag metrics-server never becomes Ready. Left
|
||||
# broken it does far more than disable `kubectl top`: the v1beta1.metrics.k8s.io APIService has no
|
||||
# endpoints, aggregated discovery goes stale, and then EVERY namespace deletion in the cluster
|
||||
# hangs on NamespaceDeletionDiscoveryFailure. Baseline runs before seeding, so a session that
|
||||
# actually contains Q23 re-breaks it a moment later, exactly as intended.
|
||||
if $K -n kube-system get deploy metrics-server >/dev/null 2>&1 \
|
||||
&& ! $K -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.containers[0].args}' | grep -q 'kubelet-insecure-tls'; then
|
||||
$K -n kube-system patch deploy metrics-server --type=json \
|
||||
-p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
# w1-block/w2-drain/w3-drain: Q307 cordons W1 and Q117 cordons W2, and Q208/Q309 have the
|
||||
# candidate drain W2/W3. A cordon left behind fails any later task that counts pods per node
|
||||
@ -776,6 +822,10 @@ def cmd_drill(args):
|
||||
if len(kept) < len(qs):
|
||||
dropped = [q["id"] for q in qs if q not in kept]
|
||||
print(yellow(f"Dropped Q{', Q'.join(map(str, dropped))}: they break the same thing as another task in this set. Drill them separately."))
|
||||
# Shuffle AFTER resolving conflicts, so which task gets dropped stays deterministic while the
|
||||
# order you meet them in does not. A drill listed by id walks you from easy to hard in
|
||||
# lockstep, which is the one ordering the real exam will never give you.
|
||||
random.Random().shuffle(kept)
|
||||
begin_session(kept, "drill", (minutes * 60) if minutes else per_task * len(kept), force, hard=hard)
|
||||
|
||||
def cmd_study(args):
|
||||
@ -806,7 +856,8 @@ def cmd_sim(args):
|
||||
force = "--force" in args
|
||||
names = [a.lower() for a in args if not a.startswith("--")]
|
||||
if not names or names[0] not in SETS:
|
||||
die("usage: ./exam sim a|b [--force]\n " + "\n ".join(f"{k}: {v} ({len(set_questions(k))} tasks)" for k, v in SETS.items()))
|
||||
die(f"usage: ./exam sim {'|'.join(SETS) or '<set>'} [--force]\n "
|
||||
+ "\n ".join(f"{k}: {v} ({len(set_questions(k))} tasks)" for k, v in SETS.items()))
|
||||
name = names[0]
|
||||
qs = set_questions(name)
|
||||
print(bold(f"{SETS[name]} - {len(qs)} tasks, {TRACK['minutes']} minutes, exam rules (no feedback until you submit)"))
|
||||
@ -1213,7 +1264,7 @@ def cmd_help(args):
|
||||
drill weak|N..|DOMAIN.. chosen tasks (--time MIN, --force, --hard)
|
||||
progress attempted/passed per domain, normal and hard
|
||||
solution N reference solution (drills: after grading N; exams: after submit)
|
||||
sim a|b fixed 17-task simulator set, exam rules, {TRACK['minutes']} min
|
||||
sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
|
||||
start [--hard] full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
|
||||
Requires every domain drilled once first (--force skips). Hard mode adds traps,
|
||||
exact-output tasks, compound failures, collateral checks and exam-style contexts.
|
||||
|
||||
@ -204,7 +204,7 @@ grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """),
|
||||
},
|
||||
# ---------------------------------------------------------------- WORK ----
|
||||
{
|
||||
"id": 23, "domain": "WORK", "title": "metrics-server repair and HPA",
|
||||
"id": 23, "domain": "WORK", "title": "metrics-server repair and HPA", "breaks": "metrics-server",
|
||||
"params": lambda r: {"ns": ns_pick(r, "scale"), "dep": name_pick(r, "api"), "max": r.choice([5, 6, 8]), "cpu": r.choice([60, 70, 75])},
|
||||
"task": """
|
||||
`metrics-server` was installed in `kube-system` but `kubectl top nodes` does not work.
|
||||
|
||||
@ -73,11 +73,16 @@ k -n %{ns} logs %{pod} -c adapter
|
||||
"task": """
|
||||
Namespace `%{ns}` has a Service `backend` with no endpoints yet.
|
||||
|
||||
1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that loops until
|
||||
the DNS name `backend` resolves, and a main container `app` (nginx:1.27).
|
||||
1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that blocks until
|
||||
Service `backend` actually **answers on port 80**, and a main container `app` (nginx:1.27).
|
||||
It will sit in `Init:0/1`.
|
||||
2. Then create Deployment `backend` (nginx:1.27, 1 replica, Pod label `app=backend`) so the
|
||||
Service gets an endpoint and your Pod starts.
|
||||
|
||||
Two traps here: a ClusterIP Service gets a DNS record the moment it is created, endpoints or not,
|
||||
so waiting on name resolution alone never blocks. And busybox `nslookup` exits non-zero even when
|
||||
it succeeds, because the search-domain attempts after the hit return NXDOMAIN. Wait on a real
|
||||
connection instead.
|
||||
""",
|
||||
"setup": NS + r"""
|
||||
mkns %{ns}
|
||||
@ -106,13 +111,17 @@ spec:
|
||||
initContainers:
|
||||
- name: wait-for-backend
|
||||
image: busybox:1.36
|
||||
command: [sh, -c, "until nslookup backend; do echo waiting; sleep 2; done"]
|
||||
command: [sh, -c, "until wget -T 2 -q -O- http://backend:80 >/dev/null 2>&1; do echo waiting; sleep 2; done"]
|
||||
containers: [{name: app, image: nginx:1.27}]
|
||||
EOF
|
||||
k -n %{ns} get pod %{pod} # Init:0/1
|
||||
k -n %{ns} create deploy backend --image=nginx:1.27
|
||||
k -n %{ns} patch deploy backend -p '{"spec":{"template":{"metadata":{"labels":{"app":"backend"}}}}}'
|
||||
# A ClusterIP Service only resolves once it has endpoints - that is what unblocks the init container.
|
||||
# A ClusterIP Service resolves in DNS from the moment it exists - endpoints change nothing about
|
||||
# that - so `until nslookup backend` would not block even before you add the Deployment. Worse,
|
||||
# busybox nslookup returns 1 even on a successful lookup, because it walks the whole search list
|
||||
# and the trailing NXDOMAINs set the exit status, so that loop never terminates either way.
|
||||
# With no endpoints, kube-proxy REJECTs the connection, so wget fails fast and the loop is real.
|
||||
""",
|
||||
},
|
||||
{
|
||||
@ -365,10 +374,12 @@ k -n %{ns} get pod %{pod}
|
||||
k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}'
|
||||
# ImagePullBackOff -> {WS}/q8/reason.txt
|
||||
k -n %{ns} describe pod %{pod} | tail -15 # "Failed to pull image ... not found"
|
||||
k -n %{ns} get pod %{pod} -o yaml > fix.yaml # set image: busybox:1.36, strip status/uid/resourceVersion
|
||||
k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml
|
||||
# A Pod's image can't be changed in place, so delete + recreate is the fix (a Deployment could
|
||||
# have been patched with `set image`).
|
||||
k -n %{ns} set image pod/%{pod} worker=busybox:1.36
|
||||
# `image` is one of the very few mutable fields on a running Pod, so no delete/recreate is needed
|
||||
# here - the kubelet just pulls the new image and restarts the container. Most other spec fields
|
||||
# (command, volumes, securityContext) really are immutable, and for those the fix is:
|
||||
# k -n %{ns} get pod %{pod} -o yaml > fix.yaml # edit, strip status/uid/resourceVersion
|
||||
# k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml
|
||||
""",
|
||||
},
|
||||
# ------------------------------------------------------------------- ENV ----
|
||||
|
||||
@ -33,10 +33,15 @@ QUESTIONS = [
|
||||
"task": """
|
||||
Namespace `%{ns}` runs `app=web` (port 8080) and `app=db` (port 5432).
|
||||
|
||||
Create three NetworkPolicies:
|
||||
Create four NetworkPolicies:
|
||||
1. `default-deny` — denies all ingress **and** egress for every Pod in the Namespace.
|
||||
2. `allow-dns` — lets every Pod reach DNS (UDP and TCP 53) in `kube-system`.
|
||||
3. `web-to-db` — lets `app=web` reach `app=db` on 5432, and lets `app=db` accept that ingress.
|
||||
3. `web-to-db` — **egress**: lets `app=web` reach `app=db` on 5432.
|
||||
4. `db-from-web` — **ingress**: lets `app=db` accept that traffic from `app=web` on 5432.
|
||||
|
||||
A NetworkPolicy has exactly one `podSelector`, so the egress half (selecting `web`) and the
|
||||
ingress half (selecting `db`) cannot live in the same object. Once a default deny is in place
|
||||
both halves are required — forgetting one is the classic CKS mistake.
|
||||
|
||||
(kindnet does not enforce policies; you are graded on the objects.)
|
||||
""",
|
||||
@ -221,11 +226,16 @@ eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{de
|
||||
p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1
|
||||
$K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""),
|
||||
C("a pod on the default ServiceAccount has no token", 3, r"""
|
||||
cleanup() { $K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; }
|
||||
$K -n %{ns} delete pod plain --ignore-not-found --wait=true >/dev/null 2>&1
|
||||
$K -n %{ns} run plain --image=busybox:1.36 --restart=Never -- sleep 120 >/dev/null 2>&1
|
||||
for i in $(seq 1 15); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && break; sleep 3; done
|
||||
r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null || r=0
|
||||
$K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; exit $r"""),
|
||||
ok=0; for i in $(seq 1 20); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && { ok=1; break; }; sleep 3; done
|
||||
# A probe pod that never ran proves nothing. Treating a failed exec as "no token mounted" handed
|
||||
# out these points for doing absolutely nothing whenever pod creation was blocked or slow.
|
||||
[ "$ok" = 1 ] || { cleanup; exit 1; }
|
||||
$K -n %{ns} exec plain -- true >/dev/null 2>&1 || { cleanup; exit 1; }
|
||||
r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token >/dev/null 2>&1 || r=0
|
||||
cleanup; exit $r"""),
|
||||
],
|
||||
"solution": """
|
||||
k -n %{ns} patch sa default -p '{"automountServiceAccountToken": false}'
|
||||
|
||||
Loading…
Reference in New Issue
Block a user