vumast3r e07ee1beab refactor: CKA_SOLUTIONS_EXPLAINED.md + 12 files

2026-10-05 16:30:46 -06:00

118 KiB

Raw Blame History

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. Every entry gives you the underlying Kubernetes concept (why the mechanism behaves the way it does), the exact commands/manifests that solve the task, and the gotchas the grader is actually checking for — the small details that separate a passing answer from a plausible-looking one. k is used throughout as the kubectl alias (already configured on the exam VMs).

The CKA track runs on Kubernetes v1.35 (kind cluster cka-sim-135, etcd 3.6, kubectl 1.35), the version the real CKA environment runs as of October 2026 — so etcdutl, not etcdctl, restores and inspects snapshots, and output matches what the exam's kubectl prints.

Practicing the same ground on Killercoda

This bank mirrors the real CKA curriculum weights (25/15/20/10/30% across the five domains), and so does Killercoda's free, browser-based CKA scenario set. Killercoda groups its own scenarios into five domains too — the mapping below cross-references its actual scenario titles against the equivalent questions in this bank, so you know exactly which free scenario to run when a topic here feels shaky.

Killercoda scenario	Killercoda domain	Equivalent question(s) here
StorageClass and Dynamic Provisioning	Storage	Q16
PV Access Modes	Storage	Q15
Volume Can't Be Mounted	Storage	Q35, Q58 (troubleshooting a binding/mount failure)
PVC Data Migration	Storage	Q59
Deployment and StatefulSet Rollouts	Workloads & Scheduling	Q07, Q51
Init Container and Secret	Workloads & Scheduling	Q28 + Q10/Q29 (init container and a broken Secret reference, combined)
DaemonSet HostPath Configurator	Workloads & Scheduling	Q26
Node Selector and Scheduling	Workloads & Scheduling	Q09 (this bank uses required nodeAffinity instead of nodeSelector — worth doing both)
Node Taints and Tolerations	Workloads & Scheduling	Q25, Q50
Scheduling Pod Affinity	Workloads & Scheduling	no direct equivalent here — this bank only exercises anti-affinity (Q50); do this one to cover positive affinity too
Scheduling Pod Anti Affinity	Workloads & Scheduling	Q50
Scheduling Priority	Workloads & Scheduling	Q24, Q50
NetworkPolicy Isolation	Services & Networking	Q11, Q54
Troubleshoot NetworkPolicies	Services & Networking	Q57 (a blocking NetworkPolicy is one of its three faults)
App Goes Public	Services & Networking	Q12 (NodePort here; LoadBalancer+autoscale adds Q23's HPA angle)
Gateway API Setup	Services & Networking	Q22, Q56
CoreDNS Misconfigured	Services & Networking	Q37, Q57
TLS Secret and HTTPS	Services & Networking	Q55
Apiserver Crash	Troubleshooting	Q66
Apiserver Misconfigured	Troubleshooting	Q66
Deployment Scaling Issue	Troubleshooting	Q23
Application Multi Container Issue	Troubleshooting	Q08, Q52 (troubleshooting angle on a multi-container pod)
Applications Misconfigured	Troubleshooting	Q18, Q62
Components Misconfigured	Troubleshooting	Q20, Q38, Q61
RBAC for CronJob	Cluster Architecture	Q01/Q44's RBAC pattern, applied to a CronJob's ServiceAccount specifically
RBAC User Permissions	Cluster Architecture	Q06, Q33, Q47
Operator Setup and CRD	Cluster Architecture	Q04, Q34, Q49 (CRD half only — this bank never writes an actual operator/controller)
Operator Namespace Creator	Cluster Architecture	no equivalent here — this bank doesn't cover writing controller logic (arguably CKAD/dev territory more than CKA)
Static Pod Move	Cluster Architecture	Q05, Q46
Cluster Setup and Node Join	Cluster Architecture	Q40 (token) + Q67 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note
Cluster Upgrade	Cluster Architecture	Q109 (the drain and uncordon around it) — the upgrade itself has no equivalent here, see gap note below
Cluster Certificate Management	Cluster Architecture	Q06, Q65

Real gaps worth knowing about

Two topics from Killercoda's list still have no faithful equivalent in this simulator, for concrete infrastructure reasons rather than oversight:

Cluster Upgrade — kubeadm upgrade plan/apply, draining nodes, upgrading kubelet/kubectl package versions in order. Not reproducible on this rig: the kind node images bake kubelet/kubeadm in as fixed binaries with no pkgs.k8s.io apt repo configured, so there's no real multi-version binary swap to perform here — faking it would just be theater. Practice this one on Killercoda, which provisions real upgradeable VMs.
Operator Namespace Creator — writing actual controller/reconciliation logic and shipping it as a running operator. Doing this properly would mean building and pushing a real container image, and arguably strays into CKAD/development territory more than CKA administration — but it's on the exam-relevant list, so don't be caught flat-footed if it comes up.

Two other gaps this bank previously had are now closed:

Apiserver Crash / Apiserver Misconfigured → Q66. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q61), just applied to kube-apiserver itself — with the added twist that every kubectl command is unusable until you fix it, so you have to work from the node directly (crictl, journalctl) rather than leaning on kubectl describe.
Cluster Setup and Node Join → partially closed by Q67. A full kubeadm init/join end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q67 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever kubeadm token create --print-join-command or your local kubeconfig hands you.

The exam-day skill this table is really pointing at, regardless of which platform you practice on: the diagnostic method is identical everywhere — read the actual error (describe, logs, journalctl -u kubelet), don't guess from memory, and verify with the tool the grader would use (auth can-i, get endpointslices, rollout status) rather than assuming a manifest applied cleanly means it's actually working.

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

Concept: RBAC is purely additive — a subject's permissions are the union of every rule bound to it, and there is no "deny" verb. That means least-privilege isn't just "grant what's needed", it's also "make sure nothing else grants more" (checked here by verifying deploy-bot still can't touch pods or leave the namespace). A Role/RoleBinding pair is namespace-scoped; a ClusterRole/ClusterRoleBinding pair would leak the grant cluster-wide, which is exactly what this task is testing you don't do.

Solution:

k -n rbac-app create sa deploy-bot

k -n rbac-app create role deploy-manager \
  --verb=get,list,watch,create,update,patch,delete \
  --resource=deployments.apps

k -n rbac-app create rolebinding deploy-bot-binding \
  --role=deploy-manager \
  --serviceaccount=rbac-app:deploy-bot

# Verify
k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot   # yes
k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot          # no
k auth can-i create deployments -n default --as=system:serviceaccount:rbac-app:deploy-bot    # no

Gotchas: the ServiceAccount identity for --as is always system:serviceaccount:<ns>:<name>, not just the name. Scoping --resource=deployments.apps (not bare deployments) keeps you honest about the API group — with a bare Role this doesn't usually matter since create role infers the group, but get in the habit for when a resource exists in multiple groups.

Q02 · etcd Snapshot

Concept: etcd is the cluster's single source of truth — every object you've ever created lives there. etcdctl snapshot save takes a consistent, point-in-time backup of the whole keyspace over etcd's own client port (2379), authenticated with the same mTLS certs the API server uses to talk to it. This is why the command needs --cacert/--cert/--key: etcd refuses unauthenticated clients.

Solution:

# On the control-plane node
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cka/etcd-snapshot.db

# Verify it's a real, non-corrupt backup
etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table

Gotchas: ETCDCTL_API=3 matters on older etcd builds where v2 is still the default API. If etcd only runs as a pod (no host-level etcdctl), the etcd container image is distroless — no shell, no tar, so kubectl cp cannot extract a file from it. Instead, run the snapshot command via kubectl exec writing into /var/lib/etcd (which is a hostPath mount), then mv it into place from the node itself. A snapshot under ~100KB is almost certainly a failed/empty capture, not a real backup — check the file size before you move on.

Q03 · Kustomize Overlay

Concept: Kustomize overlays let you reuse a base manifest set across environments without templating — a kustomization.yaml declares transformations (namespace injection, common labels, replica counts) that are applied on top of resources: at build time. Crucially, the labels: transformer with includeSelectors: false adds a label to metadata.labels without touching spec.selector, which would otherwise break the Deployment (selectors are immutable after creation and must keep matching pod template labels).

Solution:

mkdir -p {WS}/q03/overlays/prod && cd {WS}/q03/overlays/prod

cat > namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata: {name: store-prod}
EOF

cat > kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: store-prod
resources: [../../base, namespace.yaml]
labels:
  - pairs: {env: prod}
    includeSelectors: false
replicas:
  - name: store-api
    count: 3
EOF

kubectl apply -k .
k -n store-prod get deploy,svc -l env=prod

Gotchas: the replicas: generator (rather than a strategic-merge patch) is the cleanest way to bump replica count in newer Kustomize — no separate patch file needed. Folding the Namespace object into resources: alongside namespace: store-prod guarantees the namespace exists before anything is scheduled into it, in one apply -k.

Q04 · CustomResourceDefinition (CRD) & Custom Resource

Concept: A CRD registers a brand-new API type with the API server (<plural>.<group>); once Established, the server validates, stores, and serves objects of that kind exactly like a built-in resource — CRUD, RBAC, kubectl get, all of it. The Backup object you create afterward is a Custom Resource (CR) — a plain piece of structured data with no controller behind it in this task, since nothing is watching it.

Solution:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: backups.data.example.com}
spec:
  group: data.example.com
  scope: Namespaced
  names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule: {type: string}
                target: {type: string}

apiVersion: data.example.com/v1
kind: Backup
metadata: {name: nightly, namespace: default}
spec: {schedule: "0 2 * * *", target: "s3://backups/prod"}

k apply -f crd.yaml
k apply -f backup.yaml
k get bk        # shortName resolves once the CRD is Established

Gotchas: the single most common mistake is a mismatched apiVersion on the CR — it must be exactly <spec.group>/<version> from the CRD (here data.example.com/v1), not some other placeholder group. Get that wrong and you get no matches for kind "Backup" in version "..." even though the CRD applied fine. metadata.name on the CRD itself must be <plural>.<group> verbatim.

Q05 · Static Pod Creation

Concept: A static pod is defined by dropping a manifest file directly into a directory the kubelet on that node watches (staticPodPath in /var/lib/kubelet/config.yaml, default /etc/kubernetes/manifests) — the kubelet creates and manages it independently of the API server, then creates a read-only mirror pod (<name>-<node>) in the API so you can observe it with kubectl. This is exactly how kube-apiserver, kube-scheduler, etcd, and kube-controller-manager themselves run.

Solution:

{SHELL_W3}
grep staticPodPath /var/lib/kubelet/config.yaml   # confirm the real path

cat > /etc/kubernetes/manifests/static-web.yaml <<EOF
apiVersion: v1
kind: Pod
metadata: {name: static-web, namespace: default}
spec:
  containers: [{name: web, image: nginx:1.27}]
EOF
# kubelet polls the directory every ~20s — no restart needed.
# Back on the main terminal: k get pods -o wide | grep static-web

Gotchas: don't assume /etc/kubernetes/manifests — always grep staticPodPath first, since the task deliberately warns "the kubelet may not be configured the way you expect" in the harder variants. The mirror pod name is always <manifest-metadata-name>-<node-name>; deleting the mirror pod via kubectl delete does nothing permanent — the kubelet just recreates it from the file, so to actually remove a static pod you must delete the manifest file itself.

Q06 · Certificate Expiry & User CSR

Concept: kubeadm-managed clusters use a self-signed CA at /etc/kubernetes/pki/ca.{crt,key}; anyone holding that CA's private key can mint new client certificates. Kubernetes formalizes this via the CertificateSigningRequest API — you generate a keypair and CSR locally (the private key never leaves your machine), submit only the CSR to the cluster, an approver (a human, or kubectl certificate approve) signs it against the cluster CA, and you get back a client cert usable in a kubeconfig. Authorization is then just RBAC against the CN in that certificate.

Solution:

# Part A — on the control plane
kubeadm certs check-expiration | grep '^apiserver ' > {WS}/q06/apiserver-expiry.txt

# Part B — issue Jane a client cert
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr

cat <<EOF | k apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: jane}
spec:
  request: $(base64 -w0 jane.csr)
  signerName: kubernetes.io/kube-apiserver-client
  usages: [client auth]
EOF

k certificate approve jane
k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
k -n default create rolebinding jane-view --clusterrole=view --user=jane

Gotchas: signerName: kubernetes.io/kube-apiserver-client is required for a CSR meant to authenticate against the API server (as opposed to the kubelet-serving signer) — pick the wrong signer and approval succeeds but the cert won't be usable the way you expect. --user=jane in a RoleBinding works purely because Kubernetes doesn't have a User object — any CN in a client cert is a username by convention; there's nothing to "create" for the user besides the cert and the binding.

Q21 · Helm Release Install and Upgrade

Concept: Helm packages a set of templated manifests (a chart) plus a values.yaml of defaults; helm install renders the templates with your overrides and applies them as one release, and every helm upgrade re-renders and re-applies, incrementing the release's revision counter (visible via helm history) so you can always helm rollback to a prior state.

Solution:

k create ns %{ns}
helm install %{rel} {WS}/q21/chart -n %{ns} --set replicaCount=%{n}

# Upgrade in place, keeping the replicaCount override, flipping the Service type
helm upgrade %{rel} {WS}/q21/chart -n %{ns} --reuse-values --set service.type=NodePort

helm -n %{ns} history %{rel}     # REVISION 2, deployed

Gotchas: without --reuse-values (or repeating every prior --set), helm upgrade re-renders using the chart's defaults for anything you don't pass again — a common way to accidentally undo your own replicaCount override on upgrade. The rendered resource names come from the chart's _helpers.tpl (here <release>-chart), not the release name alone — check helm template output if you're unsure what a chart will actually name things.

Q33 · Aggregated ClusterRole

Concept: An aggregationRule turns a ClusterRole into a live query: the API server's ClusterRole aggregation controller continuously unions rules from every ClusterRole matching the given label selector into the aggregating role's own rules field. This lets plugin/operator ClusterRoles "attach" extra permissions to a well-known role (like the built-in admin/edit/view) just by carrying the right label, without anyone editing the parent role directly.

Solution:

cat <<EOF | k apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: %{cr}}
aggregationRule:
  clusterRoleSelectors:
    - matchLabels: {rbac.example.com/aggregate-to-%{cr}: "true"}
rules: []
EOF

k create clusterrole %{cr}-pods --verb=get,list,watch --resource=pods
k create clusterrole %{cr}-services --verb=get,list,watch --resource=services
k label clusterrole %{cr}-pods %{cr}-services rbac.example.com/aggregate-to-%{cr}=true

k create clusterrolebinding %{cr}-binding --clusterrole=%{cr} --user=%{user}
k get clusterrole %{cr} -o yaml   # .rules is now populated

Gotchas: rules: [] on the parent is mandatory at creation — you never hand-write rules on an aggregating ClusterRole, the controller owns that field. Aggregation is asynchronous; give it a couple of seconds and re-get before assuming it failed. The label key must match the selector exactly, including the value "true" as a string.

Q34 · Discover CRD Field with `kubectl explain`

Concept: Because a CRD carries a full OpenAPI v3 schema, kubectl explain <kind>.<path> --recursive works on custom types exactly as it does on built-ins — it's reading the schema straight from the CRD, including each field's description. This is the fastest way to find an undocumented field's exact path when you can't just read the CRD YAML (or are being tested on whether you know the tool).

Solution:

k explain %{kind}.spec --recursive
k explain %{kind}.spec.tuning.%{field}      # confirms the description text
echo spec.tuning.%{field} > {WS}/q34/path.txt

cat <<EOF | k apply -f -
apiVersion: demo.io/v1
kind: %{kind}
metadata: {name: sample, namespace: default}
spec:
  tuning: {%{field}: %{n}}
EOF

Gotchas: --recursive is what makes nested object fields (like spec.tuning.*) show up in one pass instead of having to explain one level at a time. Write the path with dots and no leading ./kind prefix — just spec.tuning.<field>, matching what the checker expects verbatim.

Q40 · Kubeadm Join Token Creation

Concept: Nodes join a kubeadm cluster using a bootstrap token (a short-lived, narrowly-scoped credential stored as a Secret in kube-system) plus a hash of the cluster CA's public key, so a joining node can verify it's talking to the real control plane before trusting it. kubeadm token create --print-join-command bundles both into one ready-to-run command.

Solution:

{SHELL_CP}
kubeadm token create --ttl %{ttl} --print-join-command > {WS}/q40/join.sh
kubeadm token list   # confirm it's active with the right TTL

Gotchas: the printed command already includes --discovery-token-ca-cert-hash sha256:... — don't reconstruct it by hand. The token's expiry is derived from its bootstrap-token-<id> Secret's expiration data field, so if the grader checks TTL precisely, that's where it's actually computed from — not from wall-clock guessing.

Q41 · Component Versions Inspection

Concept: There's no single "cluster version" — the API server, kubelet, container runtime, and etcd are independently versioned components, and a real administrator needs to know how to read each one off the live cluster rather than assume they all match. kubectl version reports the server (API) version; node-level facts live under .status.nodeInfo on the Node object; etcd's version is baked into its static pod's container image tag since it has no separate kubectl subcommand to ask.

Solution:

k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])'   # apiserver
k get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}'               # kubelet
k get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'      # runtime
k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'   # parse tag after the colon

Gotchas: write exactly the value the cluster reports (gitVersion, not major/minor composed by hand) — a hand-typed guess like v1.35 instead of the exact v1.35.8 string will fail an exact-match check.

Q44 · [HARD] Exact RBAC with an Existing Over-Grant

Concept: This is Q01's lesson turned adversarial: someone already bound cluster-admin to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants and that nothing broader survives.

Solution:

k get clusterrolebinding -o wide | grep %{sa}
k delete clusterrolebinding %{sa}-legacy-admin

k -n %{ns} create role %{sa}-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps
k -n %{ns} patch role %{sa}-role --type=json \
  -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["pods/exec"],"verbs":["create"]}}]'
k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:%{sa}

k create clusterrole %{sa}-nodes --verb=list --resource=nodes
k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa}

k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}

Gotchas: pods/exec is a subresource, not a resource — it can only be added via a rule with resources: ["pods/exec"], which kubectl create role can't express directly, hence the JSON-patch to append it. Deleting the cluster-admin binding is worth as many points as everything else combined — a technically-correct scoped Role means nothing if the old admin binding is still sitting there granting everything anyway.

Q45 · [HARD] etcd Restore

Concept: Restoring etcd doesn't overwrite the live data directory in place — etcdutl snapshot restore materializes a brand-new data directory from the snapshot, and you then repoint the etcd static pod's hostPath volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise.

Solution:

{SHELL_CP}
etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore

# In /etc/kubernetes/manifests/etcd.yaml, change the etcd-data hostPath volume:
#   volumes:
#   - hostPath: {path: /var/lib/etcd-restore}
#     name: etcd-data
vi /etc/kubernetes/manifests/etcd.yaml

watch crictl ps                     # wait for the new etcd static pod
kubectl -n default get cm %{cm}     # data is back

Gotchas: never restore into the existing /var/lib/etcd directory — always a fresh path, then swap the mount, so a bad restore doesn't destroy your only copy of the working data. After the swap, verify the whole control plane, not just etcd — a bad data directory can leave etcd itself healthy but apiserver unable to talk to it.

Q46 · [HARD] Static Pod on a Relocated Manifest Path

Concept: Same mechanism as Q05, with the deliberate trap that staticPodPath has been changed on this node — dropping a manifest into the default /etc/kubernetes/manifests here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default.

Solution:

k get nodes -l tier=edge
{SHELL_W3}
grep staticPodPath /var/lib/kubelet/config.yaml    # e.g. /etc/kubernetes/static

cat > /etc/kubernetes/static/%{pod}.yaml <<EOF
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
spec:
  containers:
    - name: web
      image: nginx:1.27
      resources: {requests: {cpu: 50m, memory: 32Mi}}
EOF

Gotchas: check every node's config if you're not sure which one is "the edge node" — kubectl get nodes -l tier=edge finds it via the label, not guesswork. A restart kubelet isn't required for the file to be picked up, but doesn't hurt if you want it to reload immediately.

Q47 · [HARD] Self-Contained Kubeconfig Creation

Concept: A kubeconfig can reference certs either by file path (certificate-authority, client-certificate) or by embedding their base64 contents directly (certificate-authority-data, client-certificate-data). Only the embedded form is truly portable — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing.

Solution:

openssl genrsa -out %{user}.key 2048
openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr

k apply -f - <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: %{user}}
spec: {request: $(base64 -w0 %{user}.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
EOF
k certificate approve %{user}
k get csr %{user} -o jsonpath='{.status.certificate}' | base64 -d > %{user}.crt

k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}

KC={WS}/q47/%{user}.kubeconfig
k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')

kubectl --kubeconfig $KC config set-cluster lab --server=$SERVER --certificate-authority=ca.crt --embed-certs
kubectl --kubeconfig $KC config set-credentials %{user} --client-certificate=%{user}.crt --client-key=%{user}.key --embed-certs
kubectl --kubeconfig $KC config set-context %{user}@lab --cluster=lab --user=%{user} --namespace=%{ns}
kubectl --kubeconfig $KC config use-context %{user}@lab

kubectl --kubeconfig $KC get pods    # works with zero extra flags

Gotchas: --embed-certs is the flag that matters — without it, kubectl config set-* writes file paths instead of inlining data, silently defeating the whole point. --namespace on set-context is what makes the context "default" into %{ns} so the final test command needs no -n flag at all. O=developers in the CSR subject only matters if a ClusterRoleBinding elsewhere keys off the developers group — for this task it's the RoleBinding on the username that actually grants access.

Q48 · [HARD] Kustomize with Generators and Patches

Concept: This combines every non-trivial Kustomize feature at once: a strategic-merge patch (patches: [{path: replicas.yaml}]) for a simple field override, a JSON 6902 patch for a structural addition that strategic-merge can't express well (injecting a whole envFrom array element), and a configMapGenerator, which — critically — appends a content hash to the generated name and then automatically rewrites every reference to that ConfigMap elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place.

Solution:

mkdir -p {WS}/q48/overlays/prod && cd {WS}/q48/overlays/prod

cat > replicas.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec: {replicas: %{rep}}
EOF

cat > envfrom.yaml <<EOF
- op: add
  path: /spec/template/spec/containers/0/envFrom
  value: [{configMapRef: {name: web-config}}]
EOF

cat > kustomization.yaml <<EOF
namespace: %{ns}
resources: [../../base]
labels:
  - pairs: {tier: prod}
    includeSelectors: false
configMapGenerator:
  - name: web-config
    literals: [LOG_LEVEL=%{lvl}]
patches:
  - path: replicas.yaml
  - path: envfrom.yaml
    target: {kind: Deployment, name: web}
EOF

k create ns %{ns}
k apply -k .

Gotchas: write configMapRef: {name: web-config} (the unhashed logical name) in the JSON patch — Kustomize rewrites it to web-config-<hash> at build time automatically; hardcoding a guessed hash would break the moment the literal changes. includeSelectors: false on the labels transformer is what keeps tier=prod off spec.selector.matchLabels (which must never change on an existing Deployment).

Q49 · [HARD] CRD with Validation and Defaults

Concept: A CRD's OpenAPI v3 schema is enforced by the API server itself, before an object is ever persisted: minimum/maximum reject an out-of-range number at admission time, required forces a field to be present, and default retroactively fills in a value the caller left out. This is "structural schema" validation, and it's what separates a CRD from just being a loosely-typed blob in etcd.

Solution:

# CRD schema, under spec.versions[0].schema.openAPIV3Schema.properties.spec:
required: [replicas, image]
properties:
  replicas: {type: integer, minimum: 1, maximum: 10}
  environment: {type: string, enum: [prod, stage, dev], default: dev}
  image: {type: string}

k apply -f crd.yaml

# Step 2 — a valid object
k apply -f - <<EOF
apiVersion: deploy.example.com/v1
kind: %{kind}
metadata: {name: good}
spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
EOF

# Step 3 — capture the rejection of an invalid one
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: bad}\nspec: {replicas: 0, image: x}\n' \
  | k apply -f - 2> {WS}/q49/rejected.txt

# Step 4 — confirm defaulting
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \
  | k apply -f -
k get <kind> defaulted -o jsonpath='{.spec.environment}'   # -> dev

Gotchas: capture the rejection with 2> file (stderr), not > file — kubectl apply's error goes to stderr. Defaulting only fires when a field is absent from the request entirely, not when it's set to an empty value — and it only applies going forward, so editing an existing object to remove the field afterward does not retroactively re-apply the default.

Q65 · [HARD] Renew Control Plane Certificates

Concept: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with kubeadm certs renew all. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup.

Solution:

{SHELL_CP}
kubeadm certs renew all

mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/

cp /etc/kubernetes/admin.conf ~/.kube/config

kubeadm certs check-expiration | grep '^apiserver '

Gotchas: forgetting to refresh admin.conf is the most common failure here — kubeadm certs renew all issues a new client cert for admin.conf too, but your shell's own ~/.kube/config still has the old (now-mismatched, though not necessarily expired) credentials until you copy the new file over. Verify the actual serving certificate over the wire (openssl s_client ... | openssl x509 -noout -serial) matches the file on disk — that's proof the process actually reloaded, not just that the file changed.

Q67 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane

Concept: kubeadm join doesn't trust the control plane on faith — it verifies the server it's talking to by hashing the CA certificate's SubjectPublicKeyInfo with SHA-256 and comparing it against the --discovery-token-ca-cert-hash you supply, which is exactly the computation this task has you reproduce by hand instead of trusting whatever kubeadm token create --print-join-command prints. Separately, the address a joining node needs is the API server's real address on the cluster network — not the 127.0.0.1:<port> your own kubeconfig uses, which only works because kind port-forwards that specifically for your host machine.

Solution:

{SHELL_CP}
kubeadm token create --ttl 2h

# the exact hash kubeadm join itself verifies against
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
  | openssl rsa -pubin -outform der 2>/dev/null \
  | openssl dgst -sha256 -hex | sed 's/^.* //'

exit
kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}'

JOIN_TOKEN=<token>
CA_HASH=sha256:<hash>
API_ENDPOINT=<ip>:<port>

Gotchas: the kubernetes Service's own Endpoints object (kubectl get endpoints kubernetes) is the reliable source for the real advertised address — it's populated directly from kube-apiserver's --advertise-address, which is what any node on the cluster's own network actually reaches, as opposed to kubectl cluster-info/config view, which shows the address your specific machine uses (potentially through a completely different port-forward or load balancer). Recomputing the CA hash yourself rather than trusting a printed value is the actual security property kubeadm join is built around — a client blindly copying a hash it was handed defeats the whole point of a discovery hash.

Q107 · Kernel Prerequisites for a Kubernetes Node

Concept: before kubeadm runs, every machine needs the overlay and br_netfilter kernel modules and three sysctls (net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, net.ipv4.ip_forward) so that bridged Pod traffic passes through iptables and the node routes between Pods. modprobe and sysctl -w change the running kernel only; the files under /etc/modules-load.d and /etc/sysctl.d are what make the settings survive a reboot — which is what this task grades. Swap and the cgroup driver are the other two preflight classics: kubeadm refuses a node with active swap unless the kubelet says failSwapOn: false, and the kubelet's cgroupDriver must match the runtime's.

Solution:

{SHELL_W3}
printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
# on a real machine: modprobe overlay && modprobe br_netfilter && sysctl --system
sysctl -n net.ipv4.ip_forward
cat /proc/swaps                         # any line under the header = swap is on
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml

Gotchas: use the file name the task gives (it is drawn per session). The simulator's nodes are containers, so modprobe br_netfilter cannot work here and /proc/sys/net/bridge does not exist — write the files and move on; on the exam you would also apply them. /proc/swaps inside a node container shows the host's swap, which is why the answer here may be on.

Q108 · The Container Runtime Behind a Pod

Concept: the kubelet does not run containers itself — it drives a runtime (containerd here) through the CRI socket named by containerRuntimeEndpoint in /var/lib/kubelet/config.yaml. crictl speaks that same protocol, so it shows the node's view of a Pod: the sandbox (crictl pods), its containers (crictl ps), and everything about one container (crictl inspect: PID, log path, mounts). It keeps working when the API server is down, which is what makes it the tool for control-plane troubleshooting.

Solution:

{SHELL_W2}
crictl version                                    # RuntimeName / RuntimeVersion
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
crictl ps --no-trunc --name app --pod $(crictl pods --name <pod> -q)
crictl inspect <id> | grep -E '"pid"|logPath'

Gotchas: crictl ps truncates IDs to 13 characters — --no-trunc (or kubectl get pod -o jsonpath='{.status.containerStatuses[0].containerID}', minus the containerd:// prefix) gives the full one. The PID is the process on the node; inside the container that process is PID 1. The log path under /var/log/pods/<namespace>_<pod>_<uid>/ is the file kubectl logs reads.

Q109 · Drain the Control-Plane Node and Bring It Back

Concept: kubectl drain is cordon plus eviction — the first half of every node upgrade or maintenance window. It leaves DaemonSet Pods (they would be recreated at once) and static Pods (the kubelet owns them), and it refuses to go on when it meets emptyDir data or Pods without a controller until you say so explicitly. The node then stays SchedulingDisabled until kubectl uncordon. A Pod that can only run on the drained node (here: a nodeSelector for the control plane) sits in Pending for the duration.

Solution:

kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
kubectl get nodes > {WS}/q109/drained.txt
kubectl -n <ns> get pod -l app=<deployment>          # Pending
echo Pending > {WS}/q109/phase.txt
kubectl uncordon {CP}
kubectl -n <ns> rollout status deploy <deployment>

Gotchas: --force is only for bare Pods and deletes them for good — do not add it by reflex. CoreDNS usually sits on the control-plane node in a fresh kubeadm cluster, so the drain moves it to the workers; it does not move back after the uncordon, and that is fine. Save kubectl get nodes before you uncordon — the file is the proof the node was drained.

Q110 · Prepare the Join of a Second Control-Plane Node

Concept: a joining control-plane node needs the cluster's CA, front-proxy CA, etcd CA and service-account keys. kubeadm init phase upload-certs --upload-certs encrypts them with a one-off certificate key and stores them for two hours in the Secret kubeadm-certs; kubeadm join --control-plane --certificate-key <key> downloads and decrypts them. All of this presupposes a controlPlaneEndpoint — the one stable address (a load balancer or VIP in production) that every node and kubeconfig uses instead of one particular API server.

Solution:

kubectl -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint
{SHELL_CP}
kubeadm init phase upload-certs --upload-certs       # last line: the certificate key
kubeadm token create --print-join-command --certificate-key <key>

Gotchas: without --certificate-key (or --control-plane) --print-join-command prints a worker's join line. A key made with kubeadm certs certificate-key alone is just a random key — it only works if the certificates were uploaded with it (upload-certs --certificate-key <key>), so the simplest route is to let upload-certs generate it and copy what it prints.

Q111 · Who Leads, and Who Is in etcd

Concept: in a highly-available control plane every kube-apiserver serves at once, but only one kube-scheduler and one kube-controller-manager act — they elect a leader through a Lease in kube-system, and the holder identity is <node>_<uuid>. etcd elects its own leader and needs a majority of members to accept writes (3 members survive the loss of 1, 5 of 2). Stacked means an etcd member runs on each control-plane node, external that etcd has hosts of its own.

Solution:

kubectl get nodes -l node-role.kubernetes.io/control-plane
kubectl -n kube-system get pods -l component=etcd -o wide
kubectl -n kube-system get lease kube-scheduler kube-controller-manager
{SHELL_CP}
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
       ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table                     # IS LEADER

Gotchas: write the node name for the two leases, not the whole holder identity — the uuid changes with every restart of the component. ETCDCTL_API=3 is not needed any more; etcdctl 3.6 only warns about it. endpoint status --cluster asks every member through its advertised client URL, which fails when a node's address has changed; without --cluster you ask the local member, whose answer already names the leader.

Q112 · CNI, CSI and CRI: What This Cluster Plugs In

Concept: three interfaces keep networking, storage and container execution out of Kubernetes itself. CRI: the kubelet's socket to the runtime. CNI: for every Pod sandbox the runtime runs the plugin chain of the first configuration file in /etc/cni/net.d (binaries in /opt/cni/bin) — here ptp wires the veth pair, its host-local IPAM plugin hands out an address from the node's Pod CIDR, and portmap serves hostPorts. CSI: a storage driver registers a CSIDriver object and a StorageClass names it as provisioner.

Solution:

{SHELL_W1}
ls /etc/cni/net.d/ && cat /etc/cni/net.d/*.conflist
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
exit
kubectl get csidrivers
kubectl get sc

Gotchas: cni-plugins asks for the type of each plugin in the chain, in order — not the network's name (kindnet) and not the IPAM plugin, which has its own line. "No resources found" for CSIDrivers is the answer 0: the local-path provisioner of this cluster is a plain external provisioner, not a CSI driver.

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

Concept: RollingUpdate (the default Deployment strategy) replaces pods gradually, bounded by maxSurge (how many extra pods beyond replicas are allowed mid-rollout) and maxUnavailable (how many can be missing). Setting maxSurge: 1, maxUnavailable: 0 means Kubernetes must always create the new-image replacement pod and wait for it to be Ready before tearing down an old one — a zero-downtime rollout at the cost of briefly running one extra pod.

Solution:

k -n apps set image deploy web-frontend nginx=nginx:1.27
k -n apps scale deploy web-frontend --replicas=4
k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k -n apps rollout status deploy web-frontend

Gotchas: set image deploy/<name> <container-name>=<image> needs the container name, not the deployment name — get that wrong and it silently no-ops or errors depending on kubectl version. All 4 replicas must reach status.availableReplicas, not just readyReplicas — available additionally requires the pod to have stayed ready for minReadySeconds (0 by default, but still a distinct field the grader checks).

Q08 · Sidecar Container & Shared Volume

Concept: Two containers in the same pod share the same network namespace and can share storage via a volume mounted into both — emptyDir is the simplest option, an initially-empty directory that lives as long as the pod does (not persistent across pod restarts). This is the classic log-shipping sidecar pattern: one container produces data, a second reads and forwards it, with no direct coupling between them beyond the shared filesystem path.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: log-shipper, namespace: apps}
spec:
  volumes: [{name: logs, emptyDir: {}}]
  containers:
    - name: app
      image: busybox:1.36
      command: [sh, -c, "while true; do date >> /var/log/app.log; sleep 2; done"]
      volumeMounts: [{name: logs, mountPath: /var/log}]
    - name: shipper
      image: busybox:1.36
      command: [sh, -c, "tail -f /var/log/app.log"]
      volumeMounts: [{name: logs, mountPath: /var/log}]

k -n apps logs log-shipper -c shipper

Gotchas: -c <container> is required on both logs and exec the moment a pod has more than one container — omitting it either errors or (on older versions) defaults to the first container, silently reading the wrong logs.

Q09 · Node Affinity & Resource Requests/Limits

Concept: nodeSelector is a blunt equality match; nodeAffinity with requiredDuringSchedulingIgnoredDuringExecution expresses the same "must be on a node matching this label" constraint but supports richer operators (In, NotIn, Exists...) and is the mechanism the exam expects when a task explicitly says "do not use nodeSelector." requests tell the scheduler how much capacity to reserve when placing the pod; limits are enforced by the kubelet/runtime at runtime (CPU throttled, memory OOM-killed if exceeded).

Solution:

apiVersion: apps/v1
kind: Deployment
metadata: {name: cache, namespace: apps}
spec:
  replicas: 2
  selector: {matchLabels: {app: cache}}
  template:
    metadata: {labels: {app: cache}}
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions: [{key: disktype, operator: In, values: [ssd]}]
      containers:
        - name: redis
          image: redis:7-alpine
          resources:
            requests: {cpu: 100m, memory: 64Mi}
            limits: {cpu: 200m, memory: 128Mi}

Gotchas: the field really is requiredDuringSchedulingIgnoredDuringExecution — "ignored during execution" means if the node's label later changes, already-running pods are not evicted; only new scheduling decisions respect it. If nothing else in the cluster carries disktype=ssd, pods sit Pending forever with no error beyond an event — always confirm the label exists on the intended node first.

Q10 · ConfigMap and Secret Consumption

Concept: There are three distinct ways to get config into a container, and this task deliberately exercises all of them: a single key via env[].valueFrom.configMapKeyRef (one named env var), an entire object via envFrom.secretRef (every key becomes an env var, named after the key), and a full object mounted as files via a volume of type configMap (each key becomes a file, the value its contents).

Solution:

k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug
k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'

apiVersion: v1
kind: Pod
metadata: {name: config-pod, namespace: apps}
spec:
  volumes: [{name: cfg, configMap: {name: app-config}}]
  containers:
    - name: app
      image: busybox:1.36
      command: [sleep, "3600"]
      env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}}]
      envFrom: [{secretRef: {name: db-creds}}]
      volumeMounts: [{name: cfg, mountPath: /etc/config}]

Gotchas: env (singular value) and envFrom (whole object) are separate fields on the container spec — mixing up which mechanism the task wants for which source is the most common slip here. Quote values containing shell-special characters ('password=S3cr3t!') when using --from-literal so the ! doesn't trigger history expansion in an interactive shell.

Q23 · Metrics Server Repair & HPA

Concept: metrics-server scrapes each kubelet's /metrics/resource endpoint over TLS to compute CPU/memory usage; by default it validates the kubelet's serving certificate against a CA, which fails on clusters (like local kind/minikube setups) where kubelets use self-signed serving certs metrics-server doesn't trust — the fix is --kubelet-insecure-tls, which skips that validation (fine for a lab, not for production). The HorizontalPodAutoscaler then reads exactly this metrics pipeline to decide when to scale a Deployment.

Solution:

k -n kube-system edit deploy metrics-server
# add under spec.template.spec.containers[0].args: --kubelet-insecure-tls

k top nodes    # wait ~30s, confirm data flows

k -n %{ns} autoscale deploy %{dep} --min=2 --max=%{max} --cpu-percent=%{cpu}
k -n %{ns} get hpa %{dep}

Gotchas: kubectl top can take up to a metrics-server scrape interval (~15-60s) to report data after the fix lands — a top nodes failure ten seconds after the edit doesn't necessarily mean the fix is wrong. --cpu-percent on autoscale maps to averageUtilization in the resulting HPA spec, not a raw millicore value.

Q24 · PriorityClass

Concept: PriorityClass gives pods a numeric priority the scheduler uses to decide preemption order under resource pressure — higher-priority pending pods can bump lower-priority running ones off a node. It's a cluster-scoped object referenced by name from spec.template.spec.priorityClassName on a workload; the priority is stamped onto each resulting pod's .spec.priority at admission time (changing the PriorityClass value later does not retroactively update already-running pods).

Solution:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: %{pc}}
value: %{val}
globalDefault: false
description: "%{pc} workloads"

k -n %{ns} patch deploy payments -p '{"spec":{"template":{"spec":{"priorityClassName":"%{pc}"}}}}'
k -n %{ns} rollout status deploy payments

Gotchas: patching priorityClassName changes the pod template, which triggers a new rollout — the old pods (still at the old/no priority) get replaced, so you must wait for rollout status before checking that live pods show the new priority value; checking too early sees stale pods.

Q25 · Taints & Tolerations

Concept: A taint on a node (key=value:effect) repels pods unless they carry a matching toleration — NoSchedule blocks new scheduling outright. Tolerating a taint doesn't attract a pod to that node, it only removes the repulsion; to actually land pods there you still need a separate positive signal like nodeSelector or nodeAffinity. This pairing (taint+toleration for "may schedule here" plus a selector for "must schedule here") is the standard way to dedicate a node to specific workloads.

Solution:

k taint node {W1} %{key}=%{val}:NoSchedule

spec:
  template:
    spec:
      nodeSelector: {kubernetes.io/hostname: "{W1}"}
      tolerations:
        - key: %{key}
          operator: Equal
          value: "%{val}"
          effect: NoSchedule

Gotchas: operator: Equal requires value to match exactly; operator: Exists ignores value entirely and matches on key alone — using the wrong operator/value combination is the single most common reason a toleration silently fails to match. Without the nodeSelector, tolerating the taint only means the pod could land on {W1} — the scheduler might still place it elsewhere if other nodes are untainted and otherwise equally suitable.

Q26 · DaemonSet on All Nodes (including Control Plane)

Concept: A DaemonSet schedules exactly one pod per node automatically — no replicas field, it self-scales to desiredNumberScheduled = number of matching nodes. Control-plane nodes carry a node-role.kubernetes.io/control-plane:NoSchedule taint by default specifically to keep ordinary workloads off them; a DaemonSet must explicitly tolerate that taint to also run there, which is exactly what node-level agents (kube-proxy, CNI plugins, log/metrics collectors) do in real clusters.

Solution:

apiVersion: apps/v1
kind: DaemonSet
metadata: {name: %{ds}, namespace: %{ns}}
spec:
  selector: {matchLabels: {app: %{ds}}}
  template:
    metadata: {labels: {app: %{ds}}}
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: busybox:1.36
          command: [sleep, "3600"]
          resources: {requests: {cpu: 10m, memory: 16Mi}}

Gotchas: .status.numberReady and .status.desiredNumberScheduled should both equal the total node count once healthy — if they don't match, the toleration is the first thing to check (kubectl describe ds will show 0/1 nodes are available: 1 node(s) had untolerated taint).

Q27 · CronJob Configuration

Concept: A CronJob is a template that spawns a new Job on schedule; concurrencyPolicy: Forbid skips a scheduled run entirely if the previous run's Job is still active (as opposed to Allow, the default, which lets them overlap), and the history limits bound how many completed/failed Job objects are kept around for inspection. activeDeadlineSeconds on the Job template is a hard wall-clock kill switch independent of the pod's own exit behavior.

Solution:

apiVersion: batch/v1
kind: CronJob
metadata: {name: %{cj}, namespace: %{ns}}
spec:
  schedule: '*/%{every} * * * *'
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      activeDeadlineSeconds: 30
      template:
        spec:
          restartPolicy: Never
          containers: [{name: date, image: busybox:1.36, command: ["date"]}]

k -n %{ns} create job %{cj}-manual --from=cronjob/%{cj}

Gotchas: restartPolicy: Never (or OnFailure) is mandatory on a Job's pod template — Always (the Pod-level default) is rejected outright for Jobs, since a Job's whole point is running to completion, not restarting forever. create job --from=cronjob/<name> is the correct one-off trigger — it clones the CronJob's pod template into a standalone Job without waiting for the schedule.

Q28 · Init Containers

Concept: initContainers run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared emptyDir to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q52), a plain init container fully exits before the main container's process even begins.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
  volumes: [{name: content, emptyDir: {}}]
  initContainers:
    - name: setup
      image: busybox:1.36
      command: [sh, -c, "echo '%{text}' > /work/index.html"]
      volumeMounts: [{name: content, mountPath: /work}]
  containers:
    - name: web
      image: nginx:1.25
      volumeMounts: [{name: content, mountPath: /usr/share/nginx/html}]

Gotchas: the mount path differs deliberately between the two containers (/work vs /usr/share/nginx/html) — it's the same underlying emptyDir, just mounted at whatever path each container needs; they don't have to agree on the path, only on the volume name.

Q29 · Immutable ConfigMap Rotation

Concept: immutable: true on a ConfigMap is a performance/safety feature — it tells the API server (and lets kubelets skip) watching that object for changes, and any attempt to edit its data is rejected outright. The only way to "change" an immutable ConfigMap's effective content is the classic versioned-name rollout: create a new ConfigMap object with a new name, repoint the Deployment's volume reference at it, and let the resulting pod template change trigger a normal rolling update.

Solution:

k -n %{ns} create cm %{cm}-v2 --from-literal=mode=new
k -n %{ns} patch cm %{cm}-v2 -p '{"immutable":true}'

k -n %{ns} patch deploy %{dep} --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"%{cm}-v2"}]'

k -n %{ns} rollout status deploy %{dep}

Gotchas: trying to kubectl edit or patch the original immutable ConfigMap's data fails with a clear API error — that's the mechanism working as designed, not a bug to fight around. Repointing the volume's configMap.name field is itself a pod-template change, so it automatically drives a rolling update — you don't need to separately rollout restart.

Q31 · Parallel Job

Concept: completions sets the total number of successful pod completions a Job needs before it's considered done; parallelism bounds how many pods run at once toward that total; backoffLimit caps how many pod failures the Job tolerates (with exponential backoff) before giving up entirely. This "fixed completion count" pattern is Kubernetes' model for embarrassingly-parallel batch work.

Solution:

apiVersion: batch/v1
kind: Job
metadata: {name: %{job}, namespace: %{ns}}
spec:
  completions: %{c}
  parallelism: %{p}
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers: [{name: runner, image: busybox:1.36, command: ["sh", "-c", "echo done; sleep 2"]}]

k -n %{ns} wait --for=condition=complete job/%{job} --timeout=120s

Gotchas: .status.succeeded must reach exactly completions, not just "some pods ran" — with parallelism < completions the Job launches new pods in waves as earlier ones finish, so checking too early under-counts.

Q32 · ResourceQuota & LimitRange

Concept: ResourceQuota caps the sum of requests/limits (and object counts) across an entire namespace; LimitRange instead governs individual containers — setting defaults for pods that don't specify resources at all, and floor/ceiling bounds for ones that do. The two are complementary: LimitRange makes sure every pod has some request/limit (which quota accounting requires to even function sensibly), and quota then caps the namespace-wide total.

Solution:

k -n %{ns} create quota %{ns}-quota --hard=pods=%{pods},requests.cpu=1,limits.memory=2Gi

apiVersion: v1
kind: LimitRange
metadata: {name: %{ns}-defaults, namespace: %{ns}}
spec:
  limits:
    - type: Container
      default: {cpu: 500m, memory: 256Mi}
      defaultRequest: {cpu: %{dcpu}, memory: 128Mi}
      max: {cpu: 500m, memory: 512Mi}
      min: {cpu: 50m, memory: 64Mi}

Gotchas: once a ResourceQuota with requests.cpu/limits.memory exists in a namespace, the API server rejects any new pod whose containers don't specify those resource fields — this is precisely why the LimitRange's default/defaultRequest matter: they're what let the plain worker Deployment (created with no resources at all) get admitted, by injecting values before the quota check runs.

Q50 · [HARD] Spread, Tolerate, Prioritise — Under a Quota

Concept: This stacks four scheduling primitives that each solve a different problem: pod anti-affinity (requiredDuringSchedulingIgnoredDuringExecution on topologyKey: kubernetes.io/hostname) forces one replica per node by refusing to co-locate pods matching a label selector; a toleration lets pods land on an otherwise-repelling tainted node; PriorityClass affects preemption, not placement directly; and a ResourceQuota caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it.

Solution:

apiVersion: apps/v1
kind: Deployment
metadata: {name: %{dep}, namespace: %{ns}}
spec:
  replicas: 3
  selector: {matchLabels: {app: %{dep}}}
  template:
    metadata: {labels: {app: %{dep}}}
    spec:
      priorityClassName: %{dep}-prio
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector: {matchLabels: {app: %{dep}}}
              topologyKey: kubernetes.io/hostname
      tolerations:
        - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule}
      containers:
        - name: app
          image: nginx:1.27
          resources: {requests: {cpu: 100m, memory: 64Mi}}

k create priorityclass %{dep}-prio --value=50000
# the seeded quota (pods: 2, requests.cpu: 250m) cannot hold 3 x 100m - raise BOTH, keep the quota
k -n %{ns} patch quota %{ns}-quota -p '{"spec":{"hard":{"pods":"3","requests.cpu":"300m"}}}'

Gotchas: with exactly 3 worker nodes and replicas: 3 under required anti-affinity, there is zero slack — if the quota's pods cap is lower than 3, or CPU requests don't fit the quota's requests.cpu budget, pods stay Pending with no scheduling error beyond an event; check kubectl -n %{ns} get quota and size requests accordingly rather than raising replicas blindly.

Q51 · [HARD] Rollback to a Specific Revision, Then Pause

Concept: Every kubectl rollout undo --to-revision=N targets a specific historical ReplicaSet by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. rollout pause freezes a Deployment's controller from acting on template changes: you can edit spec.template afterward all you like (image, env, whatever) and the controller will not create a new ReplicaSet or touch running pods until you resume — a safe way to stage a change for later.

Solution:

k -n %{ns} rollout history deploy %{dep}
k -n %{ns} rollout undo deploy %{dep} --to-revision=%{rev}
k -n %{ns} scale deploy %{dep} --replicas=3
k -n %{ns} rollout status deploy %{dep}

k -n %{ns} rollout pause deploy %{dep}
k -n %{ns} set image deploy/%{dep} nginx=nginx:1.27
k -n %{ns} set env deploy/%{dep} RELEASE=next
# leave it paused — no rollout status wait here, by design

Gotchas: rollout history only shows revisions if .spec.revisionHistoryLimit hasn't pruned them and --record/change-cause tracking was in effect — if the image alone doesn't tell you which revision is nginx:1.25, add --revision=N to rollout history to inspect each one's pod template. The whole point of pausing before editing is that no new ReplicaSet should exist for the post-pause template — if you edited before pausing, a rollout would have already started.

Q52 · [HARD] Native Sidecar with Probes and an Exact Report

Concept: Kubernetes 1.28+ added native sidecars: an initContainers entry with restartPolicy: Always starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with terminationGracePeriodSeconds, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
  terminationGracePeriodSeconds: 5
  initContainers:
    - name: agent
      image: busybox:1.36
      restartPolicy: Always
      command: [sh, -c, "while true; do date >> /var/log/agent/agent.log; sleep 1; done"]
      volumeMounts: [{name: logs, mountPath: /var/log/agent}]
  containers:
    - name: app
      image: nginx:1.27
      volumeMounts: [{name: logs, mountPath: /var/log/agent}]   # NOT /var/log - see Gotchas
      readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5}
      livenessProbe: {tcpSocket: {port: 80}}
  volumes: [{name: logs, emptyDir: {}}]

k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \
  | tr ' ' '\n' | grep . | sort > {WS}/q52/containers.txt
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q52/ip.txt

Gotchas: never mount an emptyDir over /var/log in an nginx container — it hides nginx's own /var/log/nginx and nginx dies with could not open error log file, crash-looping forever. A volume mount replaces the directory's contents; mount a subdirectory instead. restartPolicy: Always goes inside the initContainer entry itself, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own sort byte-for-byte.

Q53 · [HARD] Cluster Audit to Exact Files

Concept: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always -o json piped through jq/python3, never hand-transcribing what kubectl get prints to the terminal.

Solution:

# <namespace>/<pod> for pods (excluding kube-system) with no readiness probe on their first container
k get pods -A -o json | python3 -c '
import sys,json
for p in json.load(sys.stdin)["items"]:
    if p["metadata"]["namespace"]=="kube-system": continue
    if not p["spec"]["containers"][0].get("readinessProbe"):
        print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q53/no-readiness.txt

# <node> <count> of Running pods per node, all namespaces
k get pods -A -o json | python3 -c '
import sys,json,collections
c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"]
    if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
for n in sorted(c): print(n, c[n])' > {WS}/q53/per-node.txt

# every distinct image running in kube-system
k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
  | tr ' ' '\n' | sort -u > {WS}/q53/images.txt

Gotchas: no header lines, no trailing commentary — the checker does a literal sort-then-compare against each file's contents. Excluding kube-system from the first file but restricting to kube-system in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners.

Q64 · [HARD] The Context's Namespace

Concept: A kubeconfig context bundles a cluster, a user, and a default namespace (contexts[].context.namespace) — any command run without an explicit -n/--namespace inherits that default, falling back to literal default only if the context doesn't set one. kubectl config view --minify collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's default.

Solution:

k config view --minify -o jsonpath='{..namespace}' > {WS}/q64/ns.txt

# No -n anywhere below — everything inherits the context's namespace
k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
k label deploy %{dep} owner=ac --overwrite
k expose deploy %{dep} --port=9000 --target-port=9000

Gotchas: it's tempting to just assume the context's namespace is default and pass -n default explicitly to "be safe" — that defeats the entire point of the task and the checker specifically verifies nothing landed in default by mistake. {..namespace} (double-dot JSONPath) walks the whole minified document since the exact key path depends on kubeconfig structure version; .contexts[0].context.namespace also works once you know there's only one context after --minify.

Q105 · Autoscale on CPU, Scale Down Calmly

Concept: a HorizontalPodAutoscaler compares a metric with a target and sets the replica count between minReplicas and maxReplicas. CPU utilisation is usage divided by the container's CPU request, so a container without one can never be scaled on it. behavior shapes the change: scaleDown.stabilizationWindowSeconds makes the HPA act on the highest recommendation of that window (no flapping), and policies cap how fast replicas go — here one Pod per minute.

Solution:

kubectl -n <ns> set resources deploy <dep> -c web --requests=cpu=100m
kubectl -n <ns> autoscale deploy <dep> --min=<min> --max=<max> --cpu=<cpu>%     # older kubectl: --cpu-percent=<cpu>
kubectl -n <ns> edit hpa <dep>
#   spec:
#     behavior:
#       scaleDown:
#         stabilizationWindowSeconds: <win>
#         policies: [{type: Pods, value: 1, periodSeconds: 60}]
kubectl -n <ns> get hpa <dep>

Gotchas: kubectl autoscale cannot write behavior — edit the object afterwards or write the autoscaling/v2 YAML from the start. TARGETS shows <unknown> until metrics-server delivers data (it may not be installed here), but minReplicas is enforced immediately either way — which is what the last check looks at. set resources without -c changes every container; with it, only web.

Q106 · Health Checks That Check the Wrong Thing

Concept: the kubelet acts on probes, not on the application: a failing liveness probe gets the container killed and restarted, a failing readiness probe keeps the Pod out of every Service's endpoints without restarting it. So an application that is perfectly healthy looks broken when its probes point at the wrong port or path. The event text tells the two apart: connection refused is a port nobody listens on, HTTP probe failed with statuscode: 404 is a path the server does not have.

Solution:

kubectl -n <ns> describe pod -l app=<dep> --show-events | grep -E 'Liveness|Readiness|Unhealthy|Killing'
echo Unhealthy > {WS}/q106/reason.txt
kubectl -n <ns> get cm <dep>-conf -o yaml          # listen <port>; the liveness and readiness locations
kubectl -n <ns> edit deploy <dep>                  # livenessProbe port, readinessProbe path
kubectl -n <ns> rollout status deploy <dep>

Gotchas: --show-events costs nothing on the CKA's kubectl 1.35, which shows Events anyway, and is required from kubectl 1.36 on, where describe with a selector that matches several Pods leaves them out. Keep the probes HTTP and keep their timings — the graders compare them. Fixing only the readiness path produces Pods that turn Ready and are killed ten seconds later, so wait and look at RESTARTS before you move on.

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

Concept: NetworkPolicies are allow-list only and scoped by podSelector — the moment any policy selects a pod for a given policyTypes direction, that direction becomes default-deny for that pod except for what the policy(ies) explicitly permit; pods not selected by any policy remain fully open. A policy needs a CNI that enforces NetworkPolicy (Calico, Cilium, etc.) to actually have effect — plain kubectl apply succeeding only proves the object is well-formed, not that traffic is blocked, which is why lab environments grade the spec, not live traffic.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: backend-policy, namespace: netpol}
spec:
  podSelector: {matchLabels: {app: backend}}
  policyTypes: [Ingress]
  ingress:
    - from: [{podSelector: {matchLabels: {app: frontend}}}]
      ports: [{protocol: TCP, port: 8080}]

Gotchas: omitting policyTypes: [Egress] is what leaves egress unrestricted — policyTypes doesn't default to "whatever directions have rules," you must list exactly the directions you intend to lock down. A single ingress rule with one from entry and one ports entry is strictly narrower than the same rule split across multiple list entries (which OR together) — keep it to one of each so nothing broader sneaks in.

Q12 · NodePort Service

Concept: type: NodePort allocates (or, if you specify one, reserves) a port in the 30000-32767 range on every node, forwarding traffic on <any-node-IP>:nodePort through to the Service's ClusterIP:port, which in turn load-balances to matching pods via targetPort. It's built on top of ClusterIP — a NodePort Service still gets a normal cluster-internal IP too.

Solution:

apiVersion: v1
kind: Service
metadata: {name: shop-svc, namespace: svc}
spec:
  type: NodePort
  selector: {app: shop}
  ports: [{port: 80, targetPort: 80, nodePort: 30080}]

Gotchas: "has ready endpoints" isn't automatic just because the Service object exists — the selector must actually match labels on running, ready pods, or the EndpointSlice stays empty and the NodePort accepts connections that go nowhere.

Q13 · Ingress Routing

Concept: An Ingress is a declarative routing table an Ingress controller (nginx, etc.) reads and turns into real reverse-proxy config — the object itself is inert without a controller watching ingressClassName, but its structure (host, path rules, backend service+port) is exactly what any controller expects. Multiple paths under one host are matched in order by pathType; Prefix matches on path segments, not raw string prefixes (/api matches /api/v1 but not /apix).

Solution:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: app-ingress, namespace: ingress-app}
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com
      http:
        paths:
          - {path: /api, pathType: Prefix, backend: {service: {name: api, port: {number: 8080}}}}
          - {path: /,    pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}

Gotchas: list the more specific path (/api) before the catch-all / — most controllers evaluate in list order for equal-specificity prefix matches, and putting / first can shadow /api entirely depending on the controller.

Q14 · Cluster DNS Querying

Concept: CoreDNS auto-generates two DNS record families cluster-wide: Service A-records at <svc>.<namespace>.svc.cluster.local, and (if pod DNS is enabled) pod A-records at <pod-ip-with-dashes>.<namespace>.pod.cluster.local. Every pod's /etc/resolv.conf gets a search list that includes the local namespace's svc.cluster.local suffix, which is why a short name like hello often resolves too — but the fully-qualified form is what's actually being tested here since it works regardless of search-domain configuration.

Solution:

k -n dns-check run dns-test --image=busybox:1.36 -- sleep 3600
echo 'hello.dns-check.svc.cluster.local' > {WS}/q14/svc-fqdn.txt
k -n dns-check exec dns-test -- nslookup hello.dns-check.svc.cluster.local > {WS}/q14/nslookup.txt

Gotchas: the FQDN check tolerates an optional trailing dot (...cluster.local., the technically-correct fully-qualified form with the root zone) — but don't add anything else. nslookup's output includes the resolver's own address as the first two lines; the checker just greps for the Service's actual ClusterIP appearing somewhere in the file, so raw redirected output is fine.

Q22 · Gateway API HTTPRoute

Concept: The Gateway API is the newer, role-oriented replacement for Ingress: a GatewayClass (cluster-scoped, names a controller implementation), a Gateway (a specific listener set, e.g. host+port), and an HTTPRoute (namespace-scoped routing rules that attach to a Gateway via parentRefs) are three separate objects, deliberately split so infra teams own the Gateway and app teams own their own HTTPRoutes independently.

Solution:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: {name: %{gc}}
spec: {controllerName: example.com/gateway}
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{gw}, namespace: %{ns}}
spec:
  gatewayClassName: %{gc}
  listeners: [{name: http, protocol: HTTP, port: 80, hostname: %{host}.example.com}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}-route, namespace: %{ns}}
spec:
  parentRefs: [{name: %{gw}}]
  rules:
    - matches: [{path: {type: PathPrefix, value: /api}}]
      backendRefs: [{name: api, port: 8080}]
    - backendRefs: [{name: web, port: 80}]

Gotchas: the CRDs (gateways.gateway.networking.k8s.io, httproutes...) must exist before any of this — install the standard-channel release first (kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml) or every subsequent apply fails with "no matches for kind." Order matters in rules: — put the specific /api match before the catch-all rule with no matches at all (which matches everything).

Q30 · Named Ports in Service

Concept: A container port can carry a name alongside its number; a Service's targetPort can then reference that name instead of a number, and so can probes (httpGet.port). This indirection means the container can change its actual listening port without anyone needing to update the Service or probe — only the pod spec's ports[].containerPort needs to change, since everything else points at the stable name.

Solution:

# container:
ports: [{name: %{pname}, containerPort: 80}]
readinessProbe: {httpGet: {path: /, port: %{pname}}}

apiVersion: v1
kind: Service
metadata: {name: %{dep}, namespace: %{ns}}
spec:
  selector: {app: %{app}}
  ports: [{name: http, port: 8080, targetPort: %{pname}}]

Gotchas: targetPort: %{pname} must be the literal string name, not the number — YAML will happily accept either syntactically, but a numeric targetPort silently ignores the named-port feature entirely and just targets that raw port number.

Q42 · Egress-Only NetworkPolicy

Concept: An egress policy needs its DNS rule spelled out explicitly — without an allowed egress path to CoreDNS on port 53, a default-deny-egress pod can't resolve any name at all, including ones needed for the very traffic you're trying to permit (like an external CIDR by hostname). This is the most common real-world NetworkPolicy gotcha: people lock down egress and then wonder why everything breaks, DNS included.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: %{app}-egress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: %{app}}}
  policyTypes: [Egress]
  egress:
    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}]
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
    - to: [{ipBlock: {cidr: %{cidr}}}]
      ports: [{protocol: TCP, port: %{port}}]

Gotchas: combining namespaceSelector and podSelector in the same to[] entry means "pods matching X in namespaces matching Y" (an AND) — putting them in two separate to[] list entries instead would mean "any pod in namespace Y" OR "any pod matching X anywhere," which is far broader than intended. kubernetes.io/metadata.name is the automatically-applied label every namespace carries with its own name, so it's the reliable way to target "the kube-system namespace" without assuming someone hand-labeled it.

Q43 · Headless & ExternalName Services

Concept: A headless Service (clusterIP: None) skips virtual-IP load-balancing entirely — DNS for the Service name returns the individual pod IPs directly, and for a StatefulSet specifically, CoreDNS additionally publishes a stable per-pod record <pod-name>.<service-name>.<namespace>.svc.cluster.local, which is what lets StatefulSet members address each other individually. An ExternalName Service is pure DNS aliasing — no selector, no endpoints, just a CNAME to an external hostname, letting in-cluster code refer to an external dependency via a normal-looking Service name.

Solution:

apiVersion: v1
kind: Service
metadata: {name: %{sts}, namespace: %{ns}}
spec:
  clusterIP: None
  selector: {app: %{sts}}
  ports: [{port: 5432, targetPort: 5432}]
---
apiVersion: v1
kind: Service
metadata: {name: legacy, namespace: %{ns}}
spec:
  type: ExternalName
  externalName: %{ext}

# full name: busybox's nslookup applies no search domains, so the short form returns NXDOMAIN
k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts}.%{ns}.svc.cluster.local > {WS}/q43/pod0.txt

Gotchas: the headless Service's name must match spec.serviceName on the StatefulSet — that field is what wires the StatefulSet's per-pod DNS into that specific headless Service; a mismatched name means the stable pod hostnames never resolve, even though both objects individually look fine. type: ExternalName Services never get a ClusterIP or endpoints at all — that's expected, not a misconfiguration.

Q54 · [HARD] Zero-Trust Namespace

Concept: True zero-trust starts from default-deny both directions, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: %{ns}}
spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-dns, namespace: %{ns}}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}}]
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: api-ingress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: api}}
  policyTypes: [Ingress]
  ingress: [{from: [{namespaceSelector: {matchLabels: {team: %{from}}}}], ports: [{port: 8080}]}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: api-to-db, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: api}}
  policyTypes: [Egress]
  egress: [{to: [{podSelector: {matchLabels: {app: db}}}], ports: [{port: 5432}]}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: db-ingress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: db}}
  policyTypes: [Ingress]
  ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{port: 5432}]}]

k label ns %{from} team=%{from}

Gotchas: podSelector: {} (empty selector) means "every pod in this namespace" — don't confuse it with an empty from/to array, which instead means "match nothing" (fully deny that direction). This task's setup deliberately leaves a temp-allow-all policy behind as a trap; deleting every contradicting leftover policy is graded explicitly, not just adding the five you're told about.

Q55 · [HARD] TLS Ingress with a Generated Certificate

Concept: TLS termination at the Ingress layer needs a kubernetes.io/tls Secret holding tls.crt/tls.key, referenced from spec.tls[].secretName; for a self-signed lab cert, openssl req -x509 generates both the key and a self-signed certificate in one step (no separate CA involved) with the CN acting as the hostname it's valid for.

Solution:

openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt
k -n %{ns} create secret tls %{host}-tls --cert=tls.crt --key=tls.key

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: %{ns}
  annotations: {nginx.ingress.kubernetes.io/ssl-redirect: "true"}
spec:
  ingressClassName: nginx
  tls: [{hosts: [%{host}], secretName: %{host}-tls}]
  rules:
    - host: %{host}
      http:
        paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}]

openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q55/fp.txt

Gotchas: -nodes on openssl req means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which kubectl create secret tls can't supply interactively. The fingerprint recorded in fp.txt must match the certificate actually stored in the Secret, not just the local tls.crt file — if you regenerate the cert after creating the Secret, they'll drift apart.

Q56 · [HARD] Gateway API Canary with Header Match

Concept: HTTPRoute rules are evaluated top-to-bottom for a match; a rule with a header matches clause only applies to requests carrying that header, so stacking a header-matched 100%-to-v2 rule before a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. parentRefs[].sectionName scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener.

Solution:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: {name: %{host}-gc}
spec: {controllerName: example.com/gw}
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{host}-gw, namespace: %{ns}}
spec:
  gatewayClassName: %{host}-gc
  listeners:
    - {name: web, protocol: HTTP, port: 80, hostname: %{host}.example.com}
    - {name: api, protocol: HTTP, port: 80, hostname: api.%{host}.example.com}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}-route, namespace: %{ns}}
spec:
  parentRefs: [{name: %{host}-gw, sectionName: web}]
  rules:
    - matches: [{headers: [{name: X-Canary, value: always}]}]
      backendRefs: [{name: v2, port: 80}]
    - backendRefs:
        - {name: v1, port: 80, weight: %{w2}}
        - {name: v2, port: 80, weight: %{w}}

Gotchas: sectionName: web is what confines this route to just the web listener — omit it and the route attaches to every listener on the Gateway, including api, which the check explicitly verifies against. Rule order is significant and not sorted by specificity automatically — the header-matched rule must physically come first in the rules list.

Q57 · [HARD] DNS Down: Three Faults

Concept: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream forward target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check each layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found.

Solution:

k -n kube-system get netpol
k -n kube-system delete netpol maintenance-lockdown

k -n kube-system scale deploy coredns --replicas=2

k -n kube-system edit cm coredns
# forward . 10.255.255.254  ->  forward . /etc/resolv.conf

k -n kube-system rollout restart deploy coredns
# FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works
k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local

Gotchas: fixing the Corefile alone looks successful (kubectl edit returns cleanly) but does nothing while replicas are still at 0 and/or a NetworkPolicy blocks ingress to the pods — always verify the actual symptom (nslookup from a real pod) rather than trusting that each individual fix "should" have worked. rollout restart is necessary after editing the ConfigMap because CoreDNS doesn't hot-reload Corefile changes on its own.

Q113 · Pod to Pod, Across Nodes

Concept: the Kubernetes network model in one task — every Pod has its own address, taken from its node's Pod CIDR (.spec.podCIDR), and any Pod reaches any other Pod on any node at that address without NAT; the CNI plugin is what makes it true. A Service is not involved at all. The exception is a hostNetwork: true Pod, which lives in the node's own network namespace: its address is the node's, outside every Pod CIDR, and it shares the node's ports.

Solution:

kubectl -n <ns> get pods -o wide
kubectl -n <ns> get pod <web> -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt
kubectl -n <ns> exec <client> -- wget -qO- -T3 http://<web ip> > {WS}/q113/direct.html
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/q113/pod-cidrs.txt
echo <edge pod> > {WS}/q113/outside.txt; echo hostNetwork > {WS}/q113/why.txt

Gotchas: pod-cidrs.txt wants every node, the control plane included. Compare the odd Pod's IP with kubectl get nodes -o wide and you see it is its node's INTERNAL-IP. Pod addresses change whenever a Pod is replaced, which is the whole reason Services exist — never hard-code one outside an exercise like this.

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

Concept: A PersistentVolume (PV) is a piece of storage capacity a cluster admin provisions (here, a hostPath directory — fine for single-node labs, unsafe for real multi-node clusters since it's tied to one specific host); a PersistentVolumeClaim (PVC) is a request for storage that the control plane binds to a matching PV based on storageClassName, accessModes, and capacity; a Pod then references the PVC (never the PV directly) via a persistentVolumeClaim volume. persistentVolumeReclaimPolicy: Retain means the underlying data survives even after the PVC is deleted (the PV just moves to Released, as explored in Q35).

Solution:

apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-data}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath: {path: /mnt/data}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pvc-data, namespace: storage}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  resources: {requests: {storage: 500Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: data-pod, namespace: storage}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: pvc-data}}]
  containers: [{name: web, image: nginx:1.25, volumeMounts: [{name: d, mountPath: /usr/share/data}]}]

Gotchas: binding requires storageClassName to match exactly on both sides (including both being explicitly manual, since an empty string and an unset field are different for binding purposes) — a PVC requesting less capacity than the PV offers is fine (the PVC gets the whole PV), but a class mismatch leaves the PVC Pending indefinitely with no PV candidate.

Q16 · Default StorageClass & Dynamic Provisioning

Concept: Exactly one StorageClass in a cluster should carry the storageclass.kubernetes.io/is-default-class: "true" annotation — if a PVC omits storageClassName entirely, the admission controller stamps in whichever class currently holds that annotation. volumeBindingMode: WaitForFirstConsumer delays actual volume provisioning until a pod referencing the PVC is scheduled, so the provisioner can pick storage local to (or otherwise appropriate for) the node the pod actually landed on — critical for node-local provisioners like rancher.io/local-path.

Solution:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-local
  annotations: {storageclass.kubernetes.io/is-default-class: "true"}
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

k annotate sc standard storageclass.kubernetes.io/is-default-class-    # remove the trailing dash = delete the annotation

Gotchas: two StorageClasses both marked default is an explicit error condition the API server tolerates but which breaks class-omitted PVCs unpredictably (the "oldest wins" tiebreak is not something to rely on) — always remove the old default's annotation, don't just add the new one. With WaitForFirstConsumer, a bare kubectl get pvc right after creation correctly shows Pending until a consuming pod is scheduled — that's expected, not a failure.

Q35 · Re-bind a Released PersistentVolume

Concept: When a PVC bound to a Retain-policy PV is deleted, the PV doesn't become Available again automatically — it moves to Released and keeps a stale spec.claimRef pointing at the now-gone PVC's UID, which blocks any new PVC from binding to it (Kubernetes won't silently hand pre-existing data to just any new claim). Manually clearing claimRef is the explicit, deliberate step that says "yes, this data is safe to reuse."

Solution:

k patch pv %{pv} --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
k get pv %{pv}   # now Available

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  volumeName: %{pv}
  resources: {requests: {storage: 500Mi}}

Gotchas: setting spec.volumeName on the PVC is what forces a specific PV rather than letting the binder pick any matching Available one by capacity/class alone — without it, if any other manual-class PV happens to exist, you could bind to the wrong one and never notice.

Q36 · Data Persistence Across Pod Restarts

Concept: The entire point of a PVC is that its backing storage outlives any individual pod — a pod is disposable, the volume is not. This task proves that end-to-end: one pod writes and exits, a completely separate pod (potentially on a different node, if the storage backend supports it) mounts the same claim and reads back exactly what was written.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: writer, namespace: %{ns}}
spec:
  restartPolicy: Never
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: w, image: busybox:1.36, command: [sh, -c, "echo %{text} > /data/marker.txt"], volumeMounts: [{name: d, mountPath: /data}]}]

k -n %{ns} wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer

apiVersion: v1
kind: Pod
metadata: {name: reader, namespace: %{ns}}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: r, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]

Gotchas: restartPolicy: Never on the writer is required so it actually reaches Succeeded and stays there instead of the pod-level default (Always) endlessly re-running the same echo — and only after Succeeded is it safe to assume the write landed on disk before you spin up the reader.

Q58 · [HARD] Bind the Right Local Volume

Concept: local PersistentVolumes are tied to a specific node via nodeAffinity — the scheduler must place any pod using that volume on exactly that node, which is why choosing the wrong one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's spec.selector (label match) is the tool for picking a specific PV among several of the same StorageClass without editing the PVs themselves — exactly the constraint this task imposes.

Solution:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-fast
  selector: {matchLabels: {media: nvme}}
  resources: {requests: {storage: 1Gi}}

apiVersion: v1
kind: Pod
metadata: {name: consumer, namespace: %{ns}}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: c, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]

Gotchas: check kubectl get pv -l media --show-labels and cross-reference each PV's nodeAffinity against kubectl get node -l disk=nvme before writing the selector — guessing which of two identically-classed PVs is the right one gets it wrong half the time. The consumer pod's node placement is a consequence of the local PV's affinity, not something you set directly — don't add your own nodeSelector, just trust the PVC/PV binding to constrain it.

Q59 · [HARD] Migrate Data Between Claims

Concept: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway Job that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's persistentVolumeReclaimPolicy to Retain before deleting its PVC is what keeps the original data recoverable (Released, not gone) in case the migration needs to be redone.

Solution:

k patch pv $(k -n %{ns} get pvc old-data -o jsonpath='{.spec.volumeName}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: new-data, namespace: %{ns}}
spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
---
apiVersion: batch/v1
kind: Job
metadata: {name: migrate, namespace: %{ns}}
spec:
  template:
    spec:
      restartPolicy: Never
      volumes:
        - {name: src, persistentVolumeClaim: {claimName: old-data}}
        - {name: dst, persistentVolumeClaim: {claimName: new-data}}
      containers:
        - name: cp
          image: busybox:1.36
          command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
          volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]

k -n %{ns} wait --for=condition=complete job/migrate --timeout=120s
# re-point legacy: swap claimName old-data -> new-data, then recreate the pod
k -n %{ns} delete pod legacy --now
k -n %{ns} apply -f legacy.yaml
# the Job's Completed pod still references old-data and pins it - delete that pod first
k -n %{ns} delete pod -l job-name=migrate
k -n %{ns} delete pvc old-data

Gotchas: a pod that references a claim holds it in pvc-protection even after it has Completed — delete pvc old-data hangs in Terminating until the Job's pod is gone (k describe pvc old-data names it under "Used By"). Deleting the pod leaves the Job object and its succeeded: 1 intact. cp -a /src/. /dst/ (trailing /.) copies the contents of /src into /dst, preserving permissions/timestamps — dropping the trailing dot instead creates a nested /dst/src/ directory, which silently fails any check expecting files directly under the new mount. Repointing legacy's claim requires deleting and recreating the pod (volumes[].persistentVolumeClaim.claimName isn't mutable in place) — but the task requires the Deployment/PVC naming to stay the same, not the pod's UID, so this is safe here.

Domain 5: Troubleshooting (30%)

Q17 · Node `NotReady` (Kubelet Inactive)

Concept: A node's Ready condition is entirely driven by whether its kubelet is actively heartbeating to the API server — no other signal makes a node Ready. If the kubelet process is stopped or crash-looping, the node is marked NotReady (and eventually pods get evicted) purely from the API server's perspective, even though the underlying machine is perfectly healthy.

Solution:

k get node {W2}                      # NotReady
{SHELL_W2}
systemctl status kubelet             # inactive/disabled
systemctl enable --now kubelet
journalctl -u kubelet -f --no-pager  # confirm "Successfully registered node"

Gotchas: systemctl start kubelet brings it up for this boot only — enable --now (or enable + start) is what's actually being graded, since the task requires the fix to survive a reboot.

Q18 · Broken Deployment and Service

Concept: Two independent failures are layered here — a bad image tag (ImagePullBackOff, preventing any pod from ever becoming Ready) and a Service whose selector/targetPort don't actually match the Deployment's pod labels/container port (so even a healthy pod would get zero traffic). Fixing only one leaves the app still down; kubectl describe on both objects is what surfaces each root cause independently.

Solution:

k -n troubleshoot describe deploy payments | grep -i image   # nginx:1.99-nope
k -n troubleshoot set image deploy payments payments=nginx:1.27

k -n troubleshoot get svc payments -o yaml
# selector: app: payment (typo) -- container labels are app: payments
# targetPort: 8080 -- container actually listens on 80

k -n troubleshoot patch svc payments -p \
  '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'

k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments

Gotchas: an off-by-one-character selector typo (payment vs payments) produces a Service with zero endpoints and no error at all — kubectl describe svc and get endpointslices are the tools that actually surface "selector matches nothing," since the Service object itself validates fine syntactically.

Q19 · CrashLoopBackOff Diagnosis

Concept: CrashLoopBackOff means the container keeps exiting non-zero and the kubelet is backing off restart attempts — the reason is almost always in the container's own logs, and once a container has already restarted, kubectl logs --previous is what retrieves the crashed instance's output rather than the (empty, just-started) current one.

Solution:

k -n troubleshoot logs crasher --previous | tee {WS}/q19/crasher.log
# "FATAL: required environment variable DB_HOST is not set"

k -n troubleshoot get pod crasher -o yaml > crasher.yaml
# add under spec.containers[0]:
#   env: [{name: DB_HOST, value: db.internal}]
# strip status/uid/resourceVersion/creationTimestamp before reapplying

k -n troubleshoot delete pod crasher --now
k apply -f crasher.yaml

Gotchas: a bare Pod's env field can't be patched with kubectl set env in a way that survives — since the task requires keeping the same name/namespace/image/command, editing the exported YAML and recreating it is more reliable than trying to kubectl edit a running pod (many fields are immutable post-creation, so a straight edit often just errors).

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

Concept: Pods stuck Pending forever with no scheduling errors at all (not even "insufficient resources") is the signature of the scheduler itself being down — nothing is deciding placement, so pods just sit unassigned indefinitely. Since kube-scheduler runs as a static pod, a broken manifest (here, a typo'd binary name) means the kubelet can't even start the container, and kubectl logs/describe on the resulting pod reveals the exec failure.

Solution:

k -n kube-system get pods                    # kube-scheduler in Error/CrashLoopBackOff
k -n kube-system logs kube-scheduler-{CP}     # exec: "kube-schedulerr": executable file not found

{SHELL_CP}
vi /etc/kubernetes/manifests/kube-scheduler.yaml
# command: [kube-schedulerr]  ->  command: [kube-scheduler]

k -n kube-system get pods -l component=kube-scheduler   # back to Running

Gotchas: editing a static pod manifest doesn't need a kubectl apply or restart command — the kubelet watches the manifest directory itself and recreates the pod automatically within seconds of the file changing.

Q37 · Cluster DNS Misconfiguration (Invalid Forwarding)

Concept: CoreDNS's Corefile (stored as the coredns ConfigMap) controls how it resolves names it doesn't own directly — a forward directive pointing at an unreachable or wrong upstream breaks every external/non-cluster lookup, and depending on plugin order can break in-cluster lookups too if it short-circuits before the kubernetes plugin gets a chance.

Solution:

k -n kube-system get pods -l k8s-app=kube-dns
k -n kube-system logs -l k8s-app=kube-dns --tail=20

k -n kube-system get cm coredns -o yaml
# forward . 10.255.255.1   (wrong)

k -n kube-system edit cm coredns
# forward . 10.255.255.1  ->  forward . /etc/resolv.conf

k -n kube-system rollout restart deploy coredns
# FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works
k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local

Gotchas: forward . /etc/resolv.conf tells CoreDNS to use the node's own resolver config for anything outside the cluster zone — that's the correct, portable default; pointing forward at a hardcoded IP address is exactly the kind of change that breaks the moment that IP stops being reachable, which is the fault being simulated here.

Q38 · Controller-Manager Crash (Invalid Flag)

Concept: When kube-controller-manager is down, existing workloads keep running untouched, but nothing new gets reconciled — a Deployment you create gets no ReplicaSet at all, since it's the controller-manager's Deployment controller that would normally create one. This distinguishes it from a scheduler failure (Q20), where pods do get created but stay unplaced.

Solution:

k -n kube-system get pods                          # controller-manager crashing
k -n kube-system logs kube-controller-manager-{CP}  # invalid flag: --leader-elect-lease-duration=1s

{SHELL_CP}
vi /etc/kubernetes/manifests/kube-controller-manager.yaml   # remove the bad flag

k create deploy probe --image=busybox:1.36 -- sleep 60
k get rs           # a ReplicaSet now appears -> controller-manager is healthy again
k delete deploy probe

Gotchas: "no ReplicaSet or pods, and nothing shows in describe deploy" is the tell — a scheduling or image problem still produces a ReplicaSet and pod objects (just unhealthy ones); a missing ReplicaSet altogether means the Deployment controller itself never ran.

Q39 · Kubelet Configuration Drift (Certificate Path)

Concept: /var/lib/kubelet/config.yaml's clientCAFile (or similar cert-path fields) points at a specific file the kubelet needs to start correctly — if that path was "tuned" to point at a file that doesn't exist, the kubelet fails at startup and the node reads as NotReady, indistinguishable at the kubectl get node level from the process simply being stopped (Q17). journalctl -u kubelet is what actually distinguishes a config error from a stopped service.

Solution:

{SHELL_W1}
systemctl status kubelet
journalctl -u kubelet -n 30 --no-pager
# "open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory"

grep -n clientCAFile /var/lib/kubelet/config.yaml
sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml

systemctl restart kubelet && systemctl enable kubelet
systemctl status kubelet

Gotchas: a "NotReady" node always deserves a journalctl -u kubelet read before assuming it's simply stopped — a config error like this one causes the service to be in a failed/restarting state, not cleanly inactive, and blindly enable --now-ing without fixing the underlying path just loops the same crash.

Q60 · [HARD] Worker Node: Three Things Wrong

Concept: This stacks three unrelated failure modes onto one node — a cordon (spec.unschedulable: true, which stops new scheduling but is unrelated to Ready status), a wrong clientCAFile path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work."

Solution:

{SHELL_W2}
systemctl status kubelet
sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml
systemctl enable --now kubelet
systemctl status kubelet
exit

k uncordon {W2}

k run test-pod --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 60
k get pod test-pod -o wide
k delete pod test-pod --now

Gotchas: a node can show Ready in kubectl get nodes while still cordoned (Ready,SchedulingDisabled) — fixing the kubelet alone gets you Ready but pods still won't schedule there until you separately uncordon; this task's grading explicitly checks both conditions independently plus an actual pod landing there.

Q61 · [HARD] Two Control-Plane Components Down

Concept: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay Pending (scheduler dead) and brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads.

Solution:

k -n kube-system get pods    # scheduler CrashLoop, controller-manager ImagePullBackOff
{SHELL_CP}

vi /etc/kubernetes/manifests/kube-scheduler.yaml
# kube-schedulerr -> kube-scheduler

vi /etc/kubernetes/manifests/kube-controller-manager.yaml
# strip the broken image-tag suffix back to a real released version

k -n kube-system get pods -w
k create deploy probe --image=busybox:1.36 -- sleep 60
k get deploy probe   # Available within a minute -> both fixed
k delete deploy probe

Gotchas: don't stop at the first component you fix reporting Running — this task's actual grading criterion is a brand-new Deployment reaching Available, which requires both the scheduler (to place the pod) and the controller-manager (to even create the ReplicaSet) working correctly at the same time.

Q62 · [HARD] Application Down for Four Reasons

Concept: Four independent faults are stacked on one app: a ResourceQuota capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's env references (CreateContainerConfigError), and a Service selector/targetPort mismatch (Q18's pattern again) — and critically, the fix must happen in place: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut.

Solution:

k -n %{ns} get all
k -n %{ns} describe deploy %{dep}
k -n %{ns} describe rs -l app=%{dep}          # "exceeded quota: pods, requested: pods=1"

# 4, not 3: the rolling update needs a surge pod, and with maxUnavailable 0 (25% of 3 rounds
# down) the controller will not remove an old pod first - pods=3 deadlocks the rollout
k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"4"}}}'
k -n %{ns} set image deploy %{dep} web=nginx:1.27

k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}'
k -n %{ns} rollout restart deploy %{dep}

k -n %{ns} patch svc %{dep} -p \
  '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"port":80,"targetPort":80}]}}'

k -n %{ns} get deploy %{dep}
k -n %{ns} get endpoints %{dep}

Gotchas: patch/edit in place (never delete+apply) is what preserves metadata.uid on the Deployment and spec.clusterIP on the Service — those two identifiers are explicitly captured before the task starts and diffed afterward, so even a perfectly-configured recreated object fails this check. Fix the quota first — with pods: "1" in force, no amount of image/env/Service fixing gets you past 1 available replica.

Q63 · [HARD] Cluster Facts, Exactly

Concept: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of kube-controller-manager/kube-apiserver (not in any single friendly kubectl get field), the cluster DNS address is just the kube-dns Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an empty mode value means the default, iptables — not "not configured"), and per-node pod CIDR lives on the Node object itself.

Solution:

k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr
k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:'   # empty -> write "iptables"
k get node %{node} -o jsonpath='{.spec.podCIDR}'

pod-cidr=<value>
service-cidr=<value>
dns-ip=<value>
proxy-mode=<value or iptables if empty>
<node>-pod-cidr=<value>

Gotchas: kube-proxy's ConfigMap often has mode: "" (an explicit empty string) rather than the key being absent entirely — both mean "use the default, iptables" and the file must say iptables either way, not literally blank. Write exactly the key=value format the task specifies, one per line, no extra whitespace — this is graded with exact string matches, not fuzzy comparison.

Q66 · [HARD] Nothing Works: kube-apiserver Itself Is Down

Concept: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves kubectl itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every kubectl command depends on, so once it's down, kubectl describe/logs/get all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage static pods — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q61 (repair a static pod manifest), just reached without kubectl in hand.

Solution:

kubectl get nodes    # connection refused / timeout -- confirms it's the apiserver, not your network

{SHELL_CP}
crictl ps -a | grep apiserver               # see it crash-looping
journalctl -u kubelet -n 50 --no-pager | grep -i apiserver

vi /etc/kubernetes/manifests/kube-apiserver.yaml
# fix whatever's broken in the command/args (e.g. a typo'd binary name)

exit
kubectl get nodes                           # back to normal
kubectl -n kube-system get pods -l component=kube-apiserver

Gotchas: don't reach for kubectl describe/logs first here — they need the very component that's down, so they'll just hang or refuse to connect. crictl talks to containerd directly on the node and works regardless of API server state, making it (along with journalctl -u kubelet) your only real diagnostic tool until the fix lands. Once fixed, double-check the other static pods (etcd, scheduler, controller-manager) are still healthy — it's easy to nick something else while editing manifests under pressure with no kubectl to sanity-check against.

Q103 · Find the Heaviest Pods with `kubectl top`

Concept: kubectl top reads the metrics API (metrics.k8s.io), which metrics-server serves from what each kubelet reports. It shows usage — not requests or limits, which are in kubectl describe node under "Allocated resources". --sort-by=cpu|memory puts the largest first; --containers splits a Pod into its containers; -A covers every Namespace.

Solution:

kubectl top pod -n <ns> --sort-by=cpu --no-headers | head -1 | awk '{print $1}' > {WS}/q103/cpu.txt
kubectl top pod -n <ns> --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/q103/memory.txt
echo 'kubectl top node' > {WS}/q103/nodes.sh
echo 'kubectl top pod -A --sort-by=memory' > {WS}/q103/pods.sh

Gotchas: give metrics-server about a minute after the Pods start — until then top says metrics not available yet for them. Metrics API not available means metrics-server itself is missing or not Ready (that is Q23's scenario, which is why the two never share a session). The scripts run in a plain shell: spell out kubectl, an alias k does not exist there.

Q104 · One Replica Is Logging Errors

Concept: kubectl logs reads one container's stdout/stderr. With -l it reads every Pod of a label at once — but then shows only the last 10 lines per Pod unless --tail says otherwise, and you cannot tell the Pods apart without --prefix. An init container's log is read like any other container's, with -c, long after it has finished.

Solution:

kubectl -n <ns> logs -l app=<app> --prefix --tail=-1 | grep ERROR | head -3      # [pod/<app>-N/router] ERROR ...
echo <app>-N > {WS}/q104/bad-pod.txt
kubectl -n <ns> logs <app>-N | grep ERROR > {WS}/q104/errors.log
kubectl -n <ns> logs <app>-0 -c migrate                                          # schema version ... is current
echo "kubectl -n <ns> logs -l app=<app> --prefix --tail=2" > {WS}/q104/tail.sh

Gotchas: errors.log must hold the ERROR lines and nothing else — redirecting the whole log fails the check. kubectl logs statefulset/<app> (or deploy/...) picks one Pod; only a selector covers them all. --tail=2 with --prefix gives two lines per Pod, each starting [pod/<name>/<container>].

118 KiB Raw Blame History

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

Practicing the same ground on Killercoda

Real gaps worth knowing about

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

Q02 · etcd Snapshot

Q03 · Kustomize Overlay

Q04 · CustomResourceDefinition (CRD) & Custom Resource

Q05 · Static Pod Creation

Q06 · Certificate Expiry & User CSR

Q21 · Helm Release Install and Upgrade

Q33 · Aggregated ClusterRole

Q34 · Discover CRD Field with kubectl explain

Q40 · Kubeadm Join Token Creation

Q41 · Component Versions Inspection

Q44 · [HARD] Exact RBAC with an Existing Over-Grant

Q45 · [HARD] etcd Restore

Q46 · [HARD] Static Pod on a Relocated Manifest Path

Q47 · [HARD] Self-Contained Kubeconfig Creation

Q48 · [HARD] Kustomize with Generators and Patches

Q49 · [HARD] CRD with Validation and Defaults

Q65 · [HARD] Renew Control Plane Certificates

Q67 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane

Q107 · Kernel Prerequisites for a Kubernetes Node

Q108 · The Container Runtime Behind a Pod

Q109 · Drain the Control-Plane Node and Bring It Back

Q110 · Prepare the Join of a Second Control-Plane Node

Q111 · Who Leads, and Who Is in etcd

Q112 · CNI, CSI and CRI: What This Cluster Plugs In

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

Q08 · Sidecar Container & Shared Volume

Q09 · Node Affinity & Resource Requests/Limits

Q10 · ConfigMap and Secret Consumption

Q23 · Metrics Server Repair & HPA

Q24 · PriorityClass

Q25 · Taints & Tolerations

Q26 · DaemonSet on All Nodes (including Control Plane)

Q27 · CronJob Configuration

Q28 · Init Containers

Q29 · Immutable ConfigMap Rotation

Q31 · Parallel Job

Q32 · ResourceQuota & LimitRange

Q50 · [HARD] Spread, Tolerate, Prioritise — Under a Quota

Q51 · [HARD] Rollback to a Specific Revision, Then Pause

Q52 · [HARD] Native Sidecar with Probes and an Exact Report

Q53 · [HARD] Cluster Audit to Exact Files

Q64 · [HARD] The Context's Namespace

Q105 · Autoscale on CPU, Scale Down Calmly

Q106 · Health Checks That Check the Wrong Thing

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

Q12 · NodePort Service

Q13 · Ingress Routing

Q14 · Cluster DNS Querying

Q22 · Gateway API HTTPRoute

Q30 · Named Ports in Service

Q42 · Egress-Only NetworkPolicy

Q43 · Headless & ExternalName Services

Q54 · [HARD] Zero-Trust Namespace

Q55 · [HARD] TLS Ingress with a Generated Certificate

Q56 · [HARD] Gateway API Canary with Header Match

Q57 · [HARD] DNS Down: Three Faults

Q113 · Pod to Pod, Across Nodes

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

Q16 · Default StorageClass & Dynamic Provisioning

Q35 · Re-bind a Released PersistentVolume

Q36 · Data Persistence Across Pod Restarts

Q58 · [HARD] Bind the Right Local Volume

Q59 · [HARD] Migrate Data Between Claims

Domain 5: Troubleshooting (30%)

Q17 · Node NotReady (Kubelet Inactive)

Q18 · Broken Deployment and Service

Q19 · CrashLoopBackOff Diagnosis

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

Q37 · Cluster DNS Misconfiguration (Invalid Forwarding)

Q38 · Controller-Manager Crash (Invalid Flag)

Q39 · Kubelet Configuration Drift (Certificate Path)

118 KiB

Raw Blame History

Q34 · Discover CRD Field with `kubectl explain`

Q17 · Node `NotReady` (Kubelet Inactive)

Q103 · Find the Heaviest Pods with `kubectl top`