vumast3r 3959c072e6 docs: rewrite solutions guide with deeper explanations for all 65 questions

Expand every entry with a concept section explaining the underlying
mechanism, exact commands/YAML pulled from tracks/cka/solutions.py,
and gotchas tied to what each question's grader checks actually
verify. Doc grows from ~750 to ~1540 lines.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-09-10 21:05:35 -06:00

93 KiB

Raw Blame History

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. Every entry gives you the underlying Kubernetes concept (why the mechanism behaves the way it does), the exact commands/manifests that solve the task, and the gotchas the grader is actually checking for — the small details that separate a passing answer from a plausible-looking one. k is used throughout as the kubectl alias (already configured on the exam VMs).

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

Concept: RBAC is purely additive — a subject's permissions are the union of every rule bound to it, and there is no "deny" verb. That means least-privilege isn't just "grant what's needed", it's also "make sure nothing else grants more" (checked here by verifying deploy-bot still can't touch pods or leave the namespace). A Role/RoleBinding pair is namespace-scoped; a ClusterRole/ClusterRoleBinding pair would leak the grant cluster-wide, which is exactly what this task is testing you don't do.

Solution:

k -n rbac-app create sa deploy-bot

k -n rbac-app create role deploy-manager \
  --verb=get,list,watch,create,update,patch,delete \
  --resource=deployments.apps

k -n rbac-app create rolebinding deploy-bot-binding \
  --role=deploy-manager \
  --serviceaccount=rbac-app:deploy-bot

# Verify
k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot   # yes
k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot          # no
k auth can-i create deployments -n default --as=system:serviceaccount:rbac-app:deploy-bot    # no

Gotchas: the ServiceAccount identity for --as is always system:serviceaccount:<ns>:<name>, not just the name. Scoping --resource=deployments.apps (not bare deployments) keeps you honest about the API group — with a bare Role this doesn't usually matter since create role infers the group, but get in the habit for when a resource exists in multiple groups.

Q02 · etcd Snapshot

Concept: etcd is the cluster's single source of truth — every object you've ever created lives there. etcdctl snapshot save takes a consistent, point-in-time backup of the whole keyspace over etcd's own client port (2379), authenticated with the same mTLS certs the API server uses to talk to it. This is why the command needs --cacert/--cert/--key: etcd refuses unauthenticated clients.

Solution:

# On the control-plane node
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cka/etcd-snapshot.db

# Verify it's a real, non-corrupt backup
etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table

Gotchas: ETCDCTL_API=3 matters on older etcd builds where v2 is still the default API. If etcd only runs as a pod (no host-level etcdctl), the etcd container image is distroless — no shell, no tar, so kubectl cp cannot extract a file from it. Instead, run the snapshot command via kubectl exec writing into /var/lib/etcd (which is a hostPath mount), then mv it into place from the node itself. A snapshot under ~100KB is almost certainly a failed/empty capture, not a real backup — check the file size before you move on.

Q03 · Kustomize Overlay

Concept: Kustomize overlays let you reuse a base manifest set across environments without templating — a kustomization.yaml declares transformations (namespace injection, common labels, replica counts) that are applied on top of resources: at build time. Crucially, the labels: transformer with includeSelectors: false adds a label to metadata.labels without touching spec.selector, which would otherwise break the Deployment (selectors are immutable after creation and must keep matching pod template labels).

Solution:

mkdir -p {WS}/q03/overlays/prod && cd {WS}/q03/overlays/prod

cat > namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata: {name: store-prod}
EOF

cat > kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: store-prod
resources: [../../base, namespace.yaml]
labels:
  - pairs: {env: prod}
    includeSelectors: false
replicas:
  - name: store-api
    count: 3
EOF

kubectl apply -k .
k -n store-prod get deploy,svc -l env=prod

Gotchas: the replicas: generator (rather than a strategic-merge patch) is the cleanest way to bump replica count in newer Kustomize — no separate patch file needed. Folding the Namespace object into resources: alongside namespace: store-prod guarantees the namespace exists before anything is scheduled into it, in one apply -k.

Q04 · CustomResourceDefinition (CRD) & Custom Resource

Concept: A CRD registers a brand-new API type with the API server (<plural>.<group>); once Established, the server validates, stores, and serves objects of that kind exactly like a built-in resource — CRUD, RBAC, kubectl get, all of it. The Backup object you create afterward is a Custom Resource (CR) — a plain piece of structured data with no controller behind it in this task, since nothing is watching it.

Solution:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: backups.data.example.com}
spec:
  group: data.example.com
  scope: Namespaced
  names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule: {type: string}
                target: {type: string}

apiVersion: data.example.com/v1
kind: Backup
metadata: {name: nightly, namespace: default}
spec: {schedule: "0 2 * * *", target: "s3://backups/prod"}

k apply -f crd.yaml
k apply -f backup.yaml
k get bk        # shortName resolves once the CRD is Established

Gotchas: the single most common mistake is a mismatched apiVersion on the CR — it must be exactly <spec.group>/<version> from the CRD (here data.example.com/v1), not some other placeholder group. Get that wrong and you get no matches for kind "Backup" in version "..." even though the CRD applied fine. metadata.name on the CRD itself must be <plural>.<group> verbatim.

Q05 · Static Pod Creation

Concept: A static pod is defined by dropping a manifest file directly into a directory the kubelet on that node watches (staticPodPath in /var/lib/kubelet/config.yaml, default /etc/kubernetes/manifests) — the kubelet creates and manages it independently of the API server, then creates a read-only mirror pod (<name>-<node>) in the API so you can observe it with kubectl. This is exactly how kube-apiserver, kube-scheduler, etcd, and kube-controller-manager themselves run.

Solution:

{SHELL_W3}
grep staticPodPath /var/lib/kubelet/config.yaml   # confirm the real path

cat > /etc/kubernetes/manifests/static-web.yaml <<EOF
apiVersion: v1
kind: Pod
metadata: {name: static-web, namespace: default}
spec:
  containers: [{name: web, image: nginx:1.27}]
EOF
# kubelet polls the directory every ~20s — no restart needed.
# Back on the main terminal: k get pods -o wide | grep static-web

Gotchas: don't assume /etc/kubernetes/manifests — always grep staticPodPath first, since the task deliberately warns "the kubelet may not be configured the way you expect" in the harder variants. The mirror pod name is always <manifest-metadata-name>-<node-name>; deleting the mirror pod via kubectl delete does nothing permanent — the kubelet just recreates it from the file, so to actually remove a static pod you must delete the manifest file itself.

Q06 · Certificate Expiry & User CSR

Concept: kubeadm-managed clusters use a self-signed CA at /etc/kubernetes/pki/ca.{crt,key}; anyone holding that CA's private key can mint new client certificates. Kubernetes formalizes this via the CertificateSigningRequest API — you generate a keypair and CSR locally (the private key never leaves your machine), submit only the CSR to the cluster, an approver (a human, or kubectl certificate approve) signs it against the cluster CA, and you get back a client cert usable in a kubeconfig. Authorization is then just RBAC against the CN in that certificate.

Solution:

# Part A — on the control plane
kubeadm certs check-expiration | grep '^apiserver ' > {WS}/q06/apiserver-expiry.txt

# Part B — issue Jane a client cert
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr

cat <<EOF | k apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: jane}
spec:
  request: $(base64 -w0 jane.csr)
  signerName: kubernetes.io/kube-apiserver-client
  usages: [client auth]
EOF

k certificate approve jane
k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
k -n default create rolebinding jane-view --clusterrole=view --user=jane

Gotchas: signerName: kubernetes.io/kube-apiserver-client is required for a CSR meant to authenticate against the API server (as opposed to the kubelet-serving signer) — pick the wrong signer and approval succeeds but the cert won't be usable the way you expect. --user=jane in a RoleBinding works purely because Kubernetes doesn't have a User object — any CN in a client cert is a username by convention; there's nothing to "create" for the user besides the cert and the binding.

Q21 · Helm Release Install and Upgrade

Concept: Helm packages a set of templated manifests (a chart) plus a values.yaml of defaults; helm install renders the templates with your overrides and applies them as one release, and every helm upgrade re-renders and re-applies, incrementing the release's revision counter (visible via helm history) so you can always helm rollback to a prior state.

Solution:

k create ns %{ns}
helm install %{rel} {WS}/q21/chart -n %{ns} --set replicaCount=%{n}

# Upgrade in place, keeping the replicaCount override, flipping the Service type
helm upgrade %{rel} {WS}/q21/chart -n %{ns} --reuse-values --set service.type=NodePort

helm -n %{ns} history %{rel}     # REVISION 2, deployed

Gotchas: without --reuse-values (or repeating every prior --set), helm upgrade re-renders using the chart's defaults for anything you don't pass again — a common way to accidentally undo your own replicaCount override on upgrade. The rendered resource names come from the chart's _helpers.tpl (here <release>-chart), not the release name alone — check helm template output if you're unsure what a chart will actually name things.

Q33 · Aggregated ClusterRole

Concept: An aggregationRule turns a ClusterRole into a live query: the API server's ClusterRole aggregation controller continuously unions rules from every ClusterRole matching the given label selector into the aggregating role's own rules field. This lets plugin/operator ClusterRoles "attach" extra permissions to a well-known role (like the built-in admin/edit/view) just by carrying the right label, without anyone editing the parent role directly.

Solution:

cat <<EOF | k apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: %{cr}}
aggregationRule:
  clusterRoleSelectors:
    - matchLabels: {rbac.example.com/aggregate-to-%{cr}: "true"}
rules: []
EOF

k create clusterrole %{cr}-pods --verb=get,list,watch --resource=pods
k create clusterrole %{cr}-services --verb=get,list,watch --resource=services
k label clusterrole %{cr}-pods %{cr}-services rbac.example.com/aggregate-to-%{cr}=true

k create clusterrolebinding %{cr}-binding --clusterrole=%{cr} --user=%{user}
k get clusterrole %{cr} -o yaml   # .rules is now populated

Gotchas: rules: [] on the parent is mandatory at creation — you never hand-write rules on an aggregating ClusterRole, the controller owns that field. Aggregation is asynchronous; give it a couple of seconds and re-get before assuming it failed. The label key must match the selector exactly, including the value "true" as a string.

Q34 · Discover CRD Field with `kubectl explain`

Concept: Because a CRD carries a full OpenAPI v3 schema, kubectl explain <kind>.<path> --recursive works on custom types exactly as it does on built-ins — it's reading the schema straight from the CRD, including each field's description. This is the fastest way to find an undocumented field's exact path when you can't just read the CRD YAML (or are being tested on whether you know the tool).

Solution:

k explain %{kind}.spec --recursive
k explain %{kind}.spec.tuning.%{field}      # confirms the description text
echo spec.tuning.%{field} > {WS}/q34/path.txt

cat <<EOF | k apply -f -
apiVersion: demo.io/v1
kind: %{kind}
metadata: {name: sample, namespace: default}
spec:
  tuning: {%{field}: %{n}}
EOF

Gotchas: --recursive is what makes nested object fields (like spec.tuning.*) show up in one pass instead of having to explain one level at a time. Write the path with dots and no leading ./kind prefix — just spec.tuning.<field>, matching what the checker expects verbatim.

Q40 · Kubeadm Join Token Creation

Concept: Nodes join a kubeadm cluster using a bootstrap token (a short-lived, narrowly-scoped credential stored as a Secret in kube-system) plus a hash of the cluster CA's public key, so a joining node can verify it's talking to the real control plane before trusting it. kubeadm token create --print-join-command bundles both into one ready-to-run command.

Solution:

{SHELL_CP}
kubeadm token create --ttl %{ttl} --print-join-command > {WS}/q40/join.sh
kubeadm token list   # confirm it's active with the right TTL

Gotchas: the printed command already includes --discovery-token-ca-cert-hash sha256:... — don't reconstruct it by hand. The token's expiry is derived from its bootstrap-token-<id> Secret's expiration data field, so if the grader checks TTL precisely, that's where it's actually computed from — not from wall-clock guessing.

Q41 · Component Versions Inspection

Concept: There's no single "cluster version" — the API server, kubelet, container runtime, and etcd are independently versioned components, and a real administrator needs to know how to read each one off the live cluster rather than assume they all match. kubectl version reports the server (API) version; node-level facts live under .status.nodeInfo on the Node object; etcd's version is baked into its static pod's container image tag since it has no separate kubectl subcommand to ask.

Solution:

k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])'   # apiserver
k get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}'               # kubelet
k get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'      # runtime
k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'   # parse tag after the colon

Gotchas: write exactly the value the cluster reports (gitVersion, not major/minor composed by hand) — a hand-typed guess like v1.30 instead of the exact v1.30.4 string will fail an exact-match check.

Q101 · [HARD] Exact RBAC with an Existing Over-Grant

Concept: This is Q01's lesson turned adversarial: someone already bound cluster-admin to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants and that nothing broader survives.

Solution:

k get clusterrolebinding -o wide | grep %{sa}
k delete clusterrolebinding %{sa}-legacy-admin

k -n %{ns} create role %{sa}-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps
k -n %{ns} patch role %{sa}-role --type=json \
  -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["pods/exec"],"verbs":["create"]}}]'
k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:%{sa}

k create clusterrole %{sa}-nodes --verb=list --resource=nodes
k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa}

k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}

Gotchas: pods/exec is a subresource, not a resource — it can only be added via a rule with resources: ["pods/exec"], which kubectl create role can't express directly, hence the JSON-patch to append it. Deleting the cluster-admin binding is worth as many points as everything else combined — a technically-correct scoped Role means nothing if the old admin binding is still sitting there granting everything anyway.

Q102 · [HARD] etcd Restore

Concept: Restoring etcd doesn't overwrite the live data directory in place — etcdutl snapshot restore materializes a brand-new data directory from the snapshot, and you then repoint the etcd static pod's hostPath volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise.

Solution:

{SHELL_CP}
etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore

# In /etc/kubernetes/manifests/etcd.yaml, change the etcd-data hostPath volume:
#   volumes:
#   - hostPath: {path: /var/lib/etcd-restore}
#     name: etcd-data
vi /etc/kubernetes/manifests/etcd.yaml

watch crictl ps                     # wait for the new etcd static pod
kubectl -n default get cm %{cm}     # data is back

Gotchas: never restore into the existing /var/lib/etcd directory — always a fresh path, then swap the mount, so a bad restore doesn't destroy your only copy of the working data. After the swap, verify the whole control plane, not just etcd — a bad data directory can leave etcd itself healthy but apiserver unable to talk to it.

Q103 · [HARD] Static Pod on a Relocated Manifest Path

Concept: Same mechanism as Q05, with the deliberate trap that staticPodPath has been changed on this node — dropping a manifest into the default /etc/kubernetes/manifests here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default.

Solution:

k get nodes -l tier=edge
{SHELL_W3}
grep staticPodPath /var/lib/kubelet/config.yaml    # e.g. /etc/kubernetes/static

cat > /etc/kubernetes/static/%{pod}.yaml <<EOF
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
spec:
  containers:
    - name: web
      image: nginx:1.27
      resources: {requests: {cpu: 50m, memory: 32Mi}}
EOF

Gotchas: check every node's config if you're not sure which one is "the edge node" — kubectl get nodes -l tier=edge finds it via the label, not guesswork. A restart kubelet isn't required for the file to be picked up, but doesn't hurt if you want it to reload immediately.

Q104 · [HARD] Self-Contained Kubeconfig Creation

Concept: A kubeconfig can reference certs either by file path (certificate-authority, client-certificate) or by embedding their base64 contents directly (certificate-authority-data, client-certificate-data). Only the embedded form is truly portable — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing.

Solution:

openssl genrsa -out %{user}.key 2048
openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr

k apply -f - <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: %{user}}
spec: {request: $(base64 -w0 %{user}.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
EOF
k certificate approve %{user}
k get csr %{user} -o jsonpath='{.status.certificate}' | base64 -d > %{user}.crt

k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}

KC={WS}/q104/%{user}.kubeconfig
k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')

kubectl --kubeconfig $KC config set-cluster lab --server=$SERVER --certificate-authority=ca.crt --embed-certs
kubectl --kubeconfig $KC config set-credentials %{user} --client-certificate=%{user}.crt --client-key=%{user}.key --embed-certs
kubectl --kubeconfig $KC config set-context %{user}@lab --cluster=lab --user=%{user} --namespace=%{ns}
kubectl --kubeconfig $KC config use-context %{user}@lab

kubectl --kubeconfig $KC get pods    # works with zero extra flags

Gotchas: --embed-certs is the flag that matters — without it, kubectl config set-* writes file paths instead of inlining data, silently defeating the whole point. --namespace on set-context is what makes the context "default" into %{ns} so the final test command needs no -n flag at all. O=developers in the CSR subject only matters if a ClusterRoleBinding elsewhere keys off the developers group — for this task it's the RoleBinding on the username that actually grants access.

Q105 · [HARD] Kustomize with Generators and Patches

Concept: This combines every non-trivial Kustomize feature at once: a strategic-merge patch (patches: [{path: replicas.yaml}]) for a simple field override, a JSON 6902 patch for a structural addition that strategic-merge can't express well (injecting a whole envFrom array element), and a configMapGenerator, which — critically — appends a content hash to the generated name and then automatically rewrites every reference to that ConfigMap elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place.

Solution:

mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod

cat > replicas.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec: {replicas: %{rep}}
EOF

cat > envfrom.yaml <<EOF
- op: add
  path: /spec/template/spec/containers/0/envFrom
  value: [{configMapRef: {name: web-config}}]
EOF

cat > kustomization.yaml <<EOF
namespace: %{ns}
resources: [../../base]
labels:
  - pairs: {tier: prod}
    includeSelectors: false
configMapGenerator:
  - name: web-config
    literals: [LOG_LEVEL=%{lvl}]
patches:
  - path: replicas.yaml
  - path: envfrom.yaml
    target: {kind: Deployment, name: web}
EOF

k create ns %{ns}
k apply -k .

Gotchas: write configMapRef: {name: web-config} (the unhashed logical name) in the JSON patch — Kustomize rewrites it to web-config-<hash> at build time automatically; hardcoding a guessed hash would break the moment the literal changes. includeSelectors: false on the labels transformer is what keeps tier=prod off spec.selector.matchLabels (which must never change on an existing Deployment).

Q106 · [HARD] CRD with Validation and Defaults

Concept: A CRD's OpenAPI v3 schema is enforced by the API server itself, before an object is ever persisted: minimum/maximum reject an out-of-range number at admission time, required forces a field to be present, and default retroactively fills in a value the caller left out. This is "structural schema" validation, and it's what separates a CRD from just being a loosely-typed blob in etcd.

Solution:

# CRD schema, under spec.versions[0].schema.openAPIV3Schema.properties.spec:
required: [replicas, image]
properties:
  replicas: {type: integer, minimum: 1, maximum: 10}
  environment: {type: string, enum: [prod, stage, dev], default: dev}
  image: {type: string}

k apply -f crd.yaml

# Step 2 — a valid object
k apply -f - <<EOF
apiVersion: deploy.example.com/v1
kind: %{kind}
metadata: {name: good}
spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
EOF

# Step 3 — capture the rejection of an invalid one
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: bad}\nspec: {replicas: 0, image: x}\n' \
  | k apply -f - 2> {WS}/q106/rejected.txt

# Step 4 — confirm defaulting
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \
  | k apply -f -
k get <kind> defaulted -o jsonpath='{.spec.environment}'   # -> dev

Gotchas: capture the rejection with 2> file (stderr), not > file — kubectl apply's error goes to stderr. Defaulting only fires when a field is absent from the request entirely, not when it's set to an empty value — and it only applies going forward, so editing an existing object to remove the field afterward does not retroactively re-apply the default.

Q122 · [HARD] Renew Control Plane Certificates

Concept: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with kubeadm certs renew all. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup.

Solution:

{SHELL_CP}
kubeadm certs renew all

mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/

cp /etc/kubernetes/admin.conf ~/.kube/config

kubeadm certs check-expiration | grep '^apiserver '

Gotchas: forgetting to refresh admin.conf is the most common failure here — kubeadm certs renew all issues a new client cert for admin.conf too, but your shell's own ~/.kube/config still has the old (now-mismatched, though not necessarily expired) credentials until you copy the new file over. Verify the actual serving certificate over the wire (openssl s_client ... | openssl x509 -noout -serial) matches the file on disk — that's proof the process actually reloaded, not just that the file changed.

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

Concept: RollingUpdate (the default Deployment strategy) replaces pods gradually, bounded by maxSurge (how many extra pods beyond replicas are allowed mid-rollout) and maxUnavailable (how many can be missing). Setting maxSurge: 1, maxUnavailable: 0 means Kubernetes must always create the new-image replacement pod and wait for it to be Ready before tearing down an old one — a zero-downtime rollout at the cost of briefly running one extra pod.

Solution:

k -n apps set image deploy web-frontend nginx=nginx:1.27
k -n apps scale deploy web-frontend --replicas=4
k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k -n apps rollout status deploy web-frontend

Gotchas: set image deploy/<name> <container-name>=<image> needs the container name, not the deployment name — get that wrong and it silently no-ops or errors depending on kubectl version. All 4 replicas must reach status.availableReplicas, not just readyReplicas — available additionally requires the pod to have stayed ready for minReadySeconds (0 by default, but still a distinct field the grader checks).

Q08 · Sidecar Container & Shared Volume

Concept: Two containers in the same pod share the same network namespace and can share storage via a volume mounted into both — emptyDir is the simplest option, an initially-empty directory that lives as long as the pod does (not persistent across pod restarts). This is the classic log-shipping sidecar pattern: one container produces data, a second reads and forwards it, with no direct coupling between them beyond the shared filesystem path.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: log-shipper, namespace: apps}
spec:
  volumes: [{name: logs, emptyDir: {}}]
  containers:
    - name: app
      image: busybox:1.36
      command: [sh, -c, "while true; do date >> /var/log/app.log; sleep 2; done"]
      volumeMounts: [{name: logs, mountPath: /var/log}]
    - name: shipper
      image: busybox:1.36
      command: [sh, -c, "tail -f /var/log/app.log"]
      volumeMounts: [{name: logs, mountPath: /var/log}]

k -n apps logs log-shipper -c shipper

Gotchas: -c <container> is required on both logs and exec the moment a pod has more than one container — omitting it either errors or (on older versions) defaults to the first container, silently reading the wrong logs.

Q09 · Node Affinity & Resource Requests/Limits

Concept: nodeSelector is a blunt equality match; nodeAffinity with requiredDuringSchedulingIgnoredDuringExecution expresses the same "must be on a node matching this label" constraint but supports richer operators (In, NotIn, Exists...) and is the mechanism the exam expects when a task explicitly says "do not use nodeSelector." requests tell the scheduler how much capacity to reserve when placing the pod; limits are enforced by the kubelet/runtime at runtime (CPU throttled, memory OOM-killed if exceeded).

Solution:

apiVersion: apps/v1
kind: Deployment
metadata: {name: cache, namespace: apps}
spec:
  replicas: 2
  selector: {matchLabels: {app: cache}}
  template:
    metadata: {labels: {app: cache}}
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions: [{key: disktype, operator: In, values: [ssd]}]
      containers:
        - name: redis
          image: redis:7-alpine
          resources:
            requests: {cpu: 100m, memory: 64Mi}
            limits: {cpu: 200m, memory: 128Mi}

Gotchas: the field really is requiredDuringSchedulingIgnoredDuringExecution — "ignored during execution" means if the node's label later changes, already-running pods are not evicted; only new scheduling decisions respect it. If nothing else in the cluster carries disktype=ssd, pods sit Pending forever with no error beyond an event — always confirm the label exists on the intended node first.

Q10 · ConfigMap and Secret Consumption

Concept: There are three distinct ways to get config into a container, and this task deliberately exercises all of them: a single key via env[].valueFrom.configMapKeyRef (one named env var), an entire object via envFrom.secretRef (every key becomes an env var, named after the key), and a full object mounted as files via a volume of type configMap (each key becomes a file, the value its contents).

Solution:

k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug
k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'

apiVersion: v1
kind: Pod
metadata: {name: config-pod, namespace: apps}
spec:
  volumes: [{name: cfg, configMap: {name: app-config}}]
  containers:
    - name: app
      image: busybox:1.36
      command: [sleep, "3600"]
      env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}}]
      envFrom: [{secretRef: {name: db-creds}}]
      volumeMounts: [{name: cfg, mountPath: /etc/config}]

Gotchas: env (singular value) and envFrom (whole object) are separate fields on the container spec — mixing up which mechanism the task wants for which source is the most common slip here. Quote values containing shell-special characters ('password=S3cr3t!') when using --from-literal so the ! doesn't trigger history expansion in an interactive shell.

Q23 · Metrics Server Repair & HPA

Concept: metrics-server scrapes each kubelet's /metrics/resource endpoint over TLS to compute CPU/memory usage; by default it validates the kubelet's serving certificate against a CA, which fails on clusters (like local kind/minikube setups) where kubelets use self-signed serving certs metrics-server doesn't trust — the fix is --kubelet-insecure-tls, which skips that validation (fine for a lab, not for production). The HorizontalPodAutoscaler then reads exactly this metrics pipeline to decide when to scale a Deployment.

Solution:

k -n kube-system edit deploy metrics-server
# add under spec.template.spec.containers[0].args: --kubelet-insecure-tls

k top nodes    # wait ~30s, confirm data flows

k -n %{ns} autoscale deploy %{dep} --min=2 --max=%{max} --cpu-percent=%{cpu}
k -n %{ns} get hpa %{dep}

Gotchas: kubectl top can take up to a metrics-server scrape interval (~15-60s) to report data after the fix lands — a top nodes failure ten seconds after the edit doesn't necessarily mean the fix is wrong. --cpu-percent on autoscale maps to averageUtilization in the resulting HPA spec, not a raw millicore value.

Q24 · PriorityClass

Concept: PriorityClass gives pods a numeric priority the scheduler uses to decide preemption order under resource pressure — higher-priority pending pods can bump lower-priority running ones off a node. It's a cluster-scoped object referenced by name from spec.template.spec.priorityClassName on a workload; the priority is stamped onto each resulting pod's .spec.priority at admission time (changing the PriorityClass value later does not retroactively update already-running pods).

Solution:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: %{pc}}
value: %{val}
globalDefault: false
description: "%{pc} workloads"

k -n %{ns} patch deploy payments -p '{"spec":{"template":{"spec":{"priorityClassName":"%{pc}"}}}}'
k -n %{ns} rollout status deploy payments

Gotchas: patching priorityClassName changes the pod template, which triggers a new rollout — the old pods (still at the old/no priority) get replaced, so you must wait for rollout status before checking that live pods show the new priority value; checking too early sees stale pods.

Q25 · Taints & Tolerations

Concept: A taint on a node (key=value:effect) repels pods unless they carry a matching toleration — NoSchedule blocks new scheduling outright. Tolerating a taint doesn't attract a pod to that node, it only removes the repulsion; to actually land pods there you still need a separate positive signal like nodeSelector or nodeAffinity. This pairing (taint+toleration for "may schedule here" plus a selector for "must schedule here") is the standard way to dedicate a node to specific workloads.

Solution:

k taint node {W1} %{key}=%{val}:NoSchedule

spec:
  template:
    spec:
      nodeSelector: {kubernetes.io/hostname: "{W1}"}
      tolerations:
        - key: %{key}
          operator: Equal
          value: "%{val}"
          effect: NoSchedule

Gotchas: operator: Equal requires value to match exactly; operator: Exists ignores value entirely and matches on key alone — using the wrong operator/value combination is the single most common reason a toleration silently fails to match. Without the nodeSelector, tolerating the taint only means the pod could land on {W1} — the scheduler might still place it elsewhere if other nodes are untainted and otherwise equally suitable.

Q26 · DaemonSet on All Nodes (including Control Plane)

Concept: A DaemonSet schedules exactly one pod per node automatically — no replicas field, it self-scales to desiredNumberScheduled = number of matching nodes. Control-plane nodes carry a node-role.kubernetes.io/control-plane:NoSchedule taint by default specifically to keep ordinary workloads off them; a DaemonSet must explicitly tolerate that taint to also run there, which is exactly what node-level agents (kube-proxy, CNI plugins, log/metrics collectors) do in real clusters.

Solution:

apiVersion: apps/v1
kind: DaemonSet
metadata: {name: %{ds}, namespace: %{ns}}
spec:
  selector: {matchLabels: {app: %{ds}}}
  template:
    metadata: {labels: {app: %{ds}}}
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: busybox:1.36
          command: [sleep, "3600"]
          resources: {requests: {cpu: 10m, memory: 16Mi}}

Gotchas: .status.numberReady and .status.desiredNumberScheduled should both equal the total node count once healthy — if they don't match, the toleration is the first thing to check (kubectl describe ds will show 0/1 nodes are available: 1 node(s) had untolerated taint).

Q27 · CronJob Configuration

Concept: A CronJob is a template that spawns a new Job on schedule; concurrencyPolicy: Forbid skips a scheduled run entirely if the previous run's Job is still active (as opposed to Allow, the default, which lets them overlap), and the history limits bound how many completed/failed Job objects are kept around for inspection. activeDeadlineSeconds on the Job template is a hard wall-clock kill switch independent of the pod's own exit behavior.

Solution:

apiVersion: batch/v1
kind: CronJob
metadata: {name: %{cj}, namespace: %{ns}}
spec:
  schedule: '*/%{every} * * * *'
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      activeDeadlineSeconds: 30
      template:
        spec:
          restartPolicy: Never
          containers: [{name: date, image: busybox:1.36, command: ["date"]}]

k -n %{ns} create job %{cj}-manual --from=cronjob/%{cj}

Gotchas: restartPolicy: Never (or OnFailure) is mandatory on a Job's pod template — Always (the Pod-level default) is rejected outright for Jobs, since a Job's whole point is running to completion, not restarting forever. create job --from=cronjob/<name> is the correct one-off trigger — it clones the CronJob's pod template into a standalone Job without waiting for the schedule.

Q28 · Init Containers

Concept: initContainers run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared emptyDir to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q109), a plain init container fully exits before the main container's process even begins.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
  volumes: [{name: content, emptyDir: {}}]
  initContainers:
    - name: setup
      image: busybox:1.36
      command: [sh, -c, "echo '%{text}' > /work/index.html"]
      volumeMounts: [{name: content, mountPath: /work}]
  containers:
    - name: web
      image: nginx:1.25
      volumeMounts: [{name: content, mountPath: /usr/share/nginx/html}]

Gotchas: the mount path differs deliberately between the two containers (/work vs /usr/share/nginx/html) — it's the same underlying emptyDir, just mounted at whatever path each container needs; they don't have to agree on the path, only on the volume name.

Q29 · Immutable ConfigMap Rotation

Concept: immutable: true on a ConfigMap is a performance/safety feature — it tells the API server (and lets kubelets skip) watching that object for changes, and any attempt to edit its data is rejected outright. The only way to "change" an immutable ConfigMap's effective content is the classic versioned-name rollout: create a new ConfigMap object with a new name, repoint the Deployment's volume reference at it, and let the resulting pod template change trigger a normal rolling update.

Solution:

k -n %{ns} create cm %{cm}-v2 --from-literal=mode=new
k -n %{ns} patch cm %{cm}-v2 -p '{"immutable":true}'

k -n %{ns} patch deploy %{dep} --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"%{cm}-v2"}]'

k -n %{ns} rollout status deploy %{dep}

Gotchas: trying to kubectl edit or patch the original immutable ConfigMap's data fails with a clear API error — that's the mechanism working as designed, not a bug to fight around. Repointing the volume's configMap.name field is itself a pod-template change, so it automatically drives a rolling update — you don't need to separately rollout restart.

Q31 · Parallel Job

Concept: completions sets the total number of successful pod completions a Job needs before it's considered done; parallelism bounds how many pods run at once toward that total; backoffLimit caps how many pod failures the Job tolerates (with exponential backoff) before giving up entirely. This "fixed completion count" pattern is Kubernetes' model for embarrassingly-parallel batch work.

Solution:

apiVersion: batch/v1
kind: Job
metadata: {name: %{job}, namespace: %{ns}}
spec:
  completions: %{c}
  parallelism: %{p}
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers: [{name: runner, image: busybox:1.36, command: ["sh", "-c", "echo done; sleep 2"]}]

k -n %{ns} wait --for=condition=complete job/%{job} --timeout=120s

Gotchas: .status.succeeded must reach exactly completions, not just "some pods ran" — with parallelism < completions the Job launches new pods in waves as earlier ones finish, so checking too early under-counts.

Q32 · ResourceQuota & LimitRange

Concept: ResourceQuota caps the sum of requests/limits (and object counts) across an entire namespace; LimitRange instead governs individual containers — setting defaults for pods that don't specify resources at all, and floor/ceiling bounds for ones that do. The two are complementary: LimitRange makes sure every pod has some request/limit (which quota accounting requires to even function sensibly), and quota then caps the namespace-wide total.

Solution:

k -n %{ns} create quota %{ns}-quota --hard=pods=%{pods},requests.cpu=1,limits.memory=2Gi

apiVersion: v1
kind: LimitRange
metadata: {name: %{ns}-defaults, namespace: %{ns}}
spec:
  limits:
    - type: Container
      default: {cpu: 500m, memory: 256Mi}
      defaultRequest: {cpu: %{dcpu}, memory: 128Mi}
      max: {cpu: 500m, memory: 512Mi}
      min: {cpu: 50m, memory: 64Mi}

Gotchas: once a ResourceQuota with requests.cpu/limits.memory exists in a namespace, the API server rejects any new pod whose containers don't specify those resource fields — this is precisely why the LimitRange's default/defaultRequest matter: they're what let the plain worker Deployment (created with no resources at all) get admitted, by injecting values before the quota check runs.

Q107 · [HARD] Spread, Tolerate, Prioritise — Under a Quota

Concept: This stacks four scheduling primitives that each solve a different problem: pod anti-affinity (requiredDuringSchedulingIgnoredDuringExecution on topologyKey: kubernetes.io/hostname) forces one replica per node by refusing to co-locate pods matching a label selector; a toleration lets pods land on an otherwise-repelling tainted node; PriorityClass affects preemption, not placement directly; and a ResourceQuota caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it.

Solution:

apiVersion: apps/v1
kind: Deployment
metadata: {name: %{dep}, namespace: %{ns}}
spec:
  replicas: 3
  selector: {matchLabels: {app: %{dep}}}
  template:
    metadata: {labels: {app: %{dep}}}
    spec:
      priorityClassName: %{dep}-prio
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector: {matchLabels: {app: %{dep}}}
              topologyKey: kubernetes.io/hostname
      tolerations:
        - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule}
      containers:
        - name: app
          image: nginx:1.27
          resources: {requests: {cpu: 100m, memory: 64Mi}}

k create priorityclass %{dep}-prio --value=50000

Gotchas: with exactly 3 worker nodes and replicas: 3 under required anti-affinity, there is zero slack — if the quota's pods cap is lower than 3, or CPU requests don't fit the quota's requests.cpu budget, pods stay Pending with no scheduling error beyond an event; check kubectl -n %{ns} get quota and size requests accordingly rather than raising replicas blindly.

Q108 · [HARD] Rollback to a Specific Revision, Then Pause

Concept: Every kubectl rollout undo --to-revision=N targets a specific historical ReplicaSet by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. rollout pause freezes a Deployment's controller from acting on template changes: you can edit spec.template afterward all you like (image, env, whatever) and the controller will not create a new ReplicaSet or touch running pods until you resume — a safe way to stage a change for later.

Solution:

k -n %{ns} rollout history deploy %{dep}
k -n %{ns} rollout undo deploy %{dep} --to-revision=%{rev}
k -n %{ns} scale deploy %{dep} --replicas=3
k -n %{ns} rollout status deploy %{dep}

k -n %{ns} rollout pause deploy %{dep}
k -n %{ns} set image deploy/%{dep} nginx=nginx:1.27
k -n %{ns} set env deploy/%{dep} RELEASE=next
# leave it paused — no rollout status wait here, by design

Gotchas: rollout history only shows revisions if .spec.revisionHistoryLimit hasn't pruned them and --record/change-cause tracking was in effect — if the image alone doesn't tell you which revision is nginx:1.25, add --revision=N to rollout history to inspect each one's pod template. The whole point of pausing before editing is that no new ReplicaSet should exist for the post-pause template — if you edited before pausing, a rollout would have already started.

Q109 · [HARD] Native Sidecar with Probes and an Exact Report

Concept: Kubernetes 1.28+ added native sidecars: an initContainers entry with restartPolicy: Always starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with terminationGracePeriodSeconds, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
  terminationGracePeriodSeconds: 5
  initContainers:
    - name: agent
      image: busybox:1.36
      restartPolicy: Always
      command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"]
      volumeMounts: [{name: logs, mountPath: /var/log}]
  containers:
    - name: app
      image: nginx:1.27
      volumeMounts: [{name: logs, mountPath: /var/log}]
      readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5}
      livenessProbe: {tcpSocket: {port: 80}}
  volumes: [{name: logs, emptyDir: {}}]

k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \
  | tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt

Gotchas: restartPolicy: Always goes inside the initContainer entry itself, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own sort byte-for-byte.

Q110 · [HARD] Cluster Audit to Exact Files

Concept: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always -o json piped through jq/python3, never hand-transcribing what kubectl get prints to the terminal.

Solution:

# <namespace>/<pod> for pods (excluding kube-system) with no readiness probe on their first container
k get pods -A -o json | python3 -c '
import sys,json
for p in json.load(sys.stdin)["items"]:
    if p["metadata"]["namespace"]=="kube-system": continue
    if not p["spec"]["containers"][0].get("readinessProbe"):
        print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q110/no-readiness.txt

# <node> <count> of Running pods per node, all namespaces
k get pods -A -o json | python3 -c '
import sys,json,collections
c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"]
    if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
for n in sorted(c): print(n, c[n])' > {WS}/q110/per-node.txt

# every distinct image running in kube-system
k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
  | tr ' ' '\n' | sort -u > {WS}/q110/images.txt

Gotchas: no header lines, no trailing commentary — the checker does a literal sort-then-compare against each file's contents. Excluding kube-system from the first file but restricting to kube-system in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners.

Q121 · [HARD] The Context's Namespace

Concept: A kubeconfig context bundles a cluster, a user, and a default namespace (contexts[].context.namespace) — any command run without an explicit -n/--namespace inherits that default, falling back to literal default only if the context doesn't set one. kubectl config view --minify collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's default.

Solution:

k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt

# No -n anywhere below — everything inherits the context's namespace
k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
k label deploy %{dep} owner=ac --overwrite
k expose deploy %{dep} --port=9000 --target-port=9000

Gotchas: it's tempting to just assume the context's namespace is default and pass -n default explicitly to "be safe" — that defeats the entire point of the task and the checker specifically verifies nothing landed in default by mistake. {..namespace} (double-dot JSONPath) walks the whole minified document since the exact key path depends on kubeconfig structure version; .contexts[0].context.namespace also works once you know there's only one context after --minify.

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

Concept: NetworkPolicies are allow-list only and scoped by podSelector — the moment any policy selects a pod for a given policyTypes direction, that direction becomes default-deny for that pod except for what the policy(ies) explicitly permit; pods not selected by any policy remain fully open. A policy needs a CNI that enforces NetworkPolicy (Calico, Cilium, etc.) to actually have effect — plain kubectl apply succeeding only proves the object is well-formed, not that traffic is blocked, which is why lab environments grade the spec, not live traffic.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: backend-policy, namespace: netpol}
spec:
  podSelector: {matchLabels: {app: backend}}
  policyTypes: [Ingress]
  ingress:
    - from: [{podSelector: {matchLabels: {app: frontend}}}]
      ports: [{protocol: TCP, port: 8080}]

Gotchas: omitting policyTypes: [Egress] is what leaves egress unrestricted — policyTypes doesn't default to "whatever directions have rules," you must list exactly the directions you intend to lock down. A single ingress rule with one from entry and one ports entry is strictly narrower than the same rule split across multiple list entries (which OR together) — keep it to one of each so nothing broader sneaks in.

Q12 · NodePort Service

Concept: type: NodePort allocates (or, if you specify one, reserves) a port in the 30000-32767 range on every node, forwarding traffic on <any-node-IP>:nodePort through to the Service's ClusterIP:port, which in turn load-balances to matching pods via targetPort. It's built on top of ClusterIP — a NodePort Service still gets a normal cluster-internal IP too.

Solution:

apiVersion: v1
kind: Service
metadata: {name: shop-svc, namespace: svc}
spec:
  type: NodePort
  selector: {app: shop}
  ports: [{port: 80, targetPort: 80, nodePort: 30080}]

Gotchas: "has ready endpoints" isn't automatic just because the Service object exists — the selector must actually match labels on running, ready pods, or the EndpointSlice stays empty and the NodePort accepts connections that go nowhere.

Q13 · Ingress Routing

Concept: An Ingress is a declarative routing table an Ingress controller (nginx, etc.) reads and turns into real reverse-proxy config — the object itself is inert without a controller watching ingressClassName, but its structure (host, path rules, backend service+port) is exactly what any controller expects. Multiple paths under one host are matched in order by pathType; Prefix matches on path segments, not raw string prefixes (/api matches /api/v1 but not /apix).

Solution:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: app-ingress, namespace: ingress-app}
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com
      http:
        paths:
          - {path: /api, pathType: Prefix, backend: {service: {name: api, port: {number: 8080}}}}
          - {path: /,    pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}

Gotchas: list the more specific path (/api) before the catch-all / — most controllers evaluate in list order for equal-specificity prefix matches, and putting / first can shadow /api entirely depending on the controller.

Q14 · Cluster DNS Querying

Concept: CoreDNS auto-generates two DNS record families cluster-wide: Service A-records at <svc>.<namespace>.svc.cluster.local, and (if pod DNS is enabled) pod A-records at <pod-ip-with-dashes>.<namespace>.pod.cluster.local. Every pod's /etc/resolv.conf gets a search list that includes the local namespace's svc.cluster.local suffix, which is why a short name like hello often resolves too — but the fully-qualified form is what's actually being tested here since it works regardless of search-domain configuration.

Solution:

k -n dns-check run dns-test --image=busybox:1.36 -- sleep 3600
echo 'hello.dns-check.svc.cluster.local' > {WS}/q14/svc-fqdn.txt
k -n dns-check exec dns-test -- nslookup hello.dns-check.svc.cluster.local > {WS}/q14/nslookup.txt

Gotchas: the FQDN check tolerates an optional trailing dot (...cluster.local., the technically-correct fully-qualified form with the root zone) — but don't add anything else. nslookup's output includes the resolver's own address as the first two lines; the checker just greps for the Service's actual ClusterIP appearing somewhere in the file, so raw redirected output is fine.

Q22 · Gateway API HTTPRoute

Concept: The Gateway API is the newer, role-oriented replacement for Ingress: a GatewayClass (cluster-scoped, names a controller implementation), a Gateway (a specific listener set, e.g. host+port), and an HTTPRoute (namespace-scoped routing rules that attach to a Gateway via parentRefs) are three separate objects, deliberately split so infra teams own the Gateway and app teams own their own HTTPRoutes independently.

Solution:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: {name: %{gc}}
spec: {controllerName: example.com/gateway}
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{gw}, namespace: %{ns}}
spec:
  gatewayClassName: %{gc}
  listeners: [{name: http, protocol: HTTP, port: 80, hostname: %{host}.example.com}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}-route, namespace: %{ns}}
spec:
  parentRefs: [{name: %{gw}}]
  rules:
    - matches: [{path: {type: PathPrefix, value: /api}}]
      backendRefs: [{name: api, port: 8080}]
    - backendRefs: [{name: web, port: 80}]

Gotchas: the CRDs (gateways.gateway.networking.k8s.io, httproutes...) must exist before any of this — install the standard-channel release first (kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml) or every subsequent apply fails with "no matches for kind." Order matters in rules: — put the specific /api match before the catch-all rule with no matches at all (which matches everything).

Q30 · Named Ports in Service

Concept: A container port can carry a name alongside its number; a Service's targetPort can then reference that name instead of a number, and so can probes (httpGet.port). This indirection means the container can change its actual listening port without anyone needing to update the Service or probe — only the pod spec's ports[].containerPort needs to change, since everything else points at the stable name.

Solution:

# container:
ports: [{name: %{pname}, containerPort: 80}]
readinessProbe: {httpGet: {path: /, port: %{pname}}}

apiVersion: v1
kind: Service
metadata: {name: %{dep}, namespace: %{ns}}
spec:
  selector: {app: %{app}}
  ports: [{name: http, port: 8080, targetPort: %{pname}}]

Gotchas: targetPort: %{pname} must be the literal string name, not the number — YAML will happily accept either syntactically, but a numeric targetPort silently ignores the named-port feature entirely and just targets that raw port number.

Q42 · Egress-Only NetworkPolicy

Concept: An egress policy needs its DNS rule spelled out explicitly — without an allowed egress path to CoreDNS on port 53, a default-deny-egress pod can't resolve any name at all, including ones needed for the very traffic you're trying to permit (like an external CIDR by hostname). This is the most common real-world NetworkPolicy gotcha: people lock down egress and then wonder why everything breaks, DNS included.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: %{app}-egress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: %{app}}}
  policyTypes: [Egress]
  egress:
    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}]
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
    - to: [{ipBlock: {cidr: %{cidr}}}]
      ports: [{protocol: TCP, port: %{port}}]

Gotchas: combining namespaceSelector and podSelector in the same to[] entry means "pods matching X in namespaces matching Y" (an AND) — putting them in two separate to[] list entries instead would mean "any pod in namespace Y" OR "any pod matching X anywhere," which is far broader than intended. kubernetes.io/metadata.name is the automatically-applied label every namespace carries with its own name, so it's the reliable way to target "the kube-system namespace" without assuming someone hand-labeled it.

Q43 · Headless & ExternalName Services

Concept: A headless Service (clusterIP: None) skips virtual-IP load-balancing entirely — DNS for the Service name returns the individual pod IPs directly, and for a StatefulSet specifically, CoreDNS additionally publishes a stable per-pod record <pod-name>.<service-name>.<namespace>.svc.cluster.local, which is what lets StatefulSet members address each other individually. An ExternalName Service is pure DNS aliasing — no selector, no endpoints, just a CNAME to an external hostname, letting in-cluster code refer to an external dependency via a normal-looking Service name.

Solution:

apiVersion: v1
kind: Service
metadata: {name: %{sts}, namespace: %{ns}}
spec:
  clusterIP: None
  selector: {app: %{sts}}
  ports: [{port: 5432, targetPort: 5432}]
---
apiVersion: v1
kind: Service
metadata: {name: legacy, namespace: %{ns}}
spec:
  type: ExternalName
  externalName: %{ext}

k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt

Gotchas: the headless Service's name must match spec.serviceName on the StatefulSet — that field is what wires the StatefulSet's per-pod DNS into that specific headless Service; a mismatched name means the stable pod hostnames never resolve, even though both objects individually look fine. type: ExternalName Services never get a ClusterIP or endpoints at all — that's expected, not a misconfiguration.

Q111 · [HARD] Zero-Trust Namespace

Concept: True zero-trust starts from default-deny both directions, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either.

Solution:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: %{ns}}
spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-dns, namespace: %{ns}}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}}]
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: api-ingress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: api}}
  policyTypes: [Ingress]
  ingress: [{from: [{namespaceSelector: {matchLabels: {team: %{from}}}}], ports: [{port: 8080}]}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: api-to-db, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: api}}
  policyTypes: [Egress]
  egress: [{to: [{podSelector: {matchLabels: {app: db}}}], ports: [{port: 5432}]}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: db-ingress, namespace: %{ns}}
spec:
  podSelector: {matchLabels: {app: db}}
  policyTypes: [Ingress]
  ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{port: 5432}]}]

k label ns %{from} team=%{from}

Gotchas: podSelector: {} (empty selector) means "every pod in this namespace" — don't confuse it with an empty from/to array, which instead means "match nothing" (fully deny that direction). This task's setup deliberately leaves a temp-allow-all policy behind as a trap; deleting every contradicting leftover policy is graded explicitly, not just adding the five you're told about.

Q112 · [HARD] TLS Ingress with a Generated Certificate

Concept: TLS termination at the Ingress layer needs a kubernetes.io/tls Secret holding tls.crt/tls.key, referenced from spec.tls[].secretName; for a self-signed lab cert, openssl req -x509 generates both the key and a self-signed certificate in one step (no separate CA involved) with the CN acting as the hostname it's valid for.

Solution:

openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt
k -n %{ns} create secret tls %{host}-tls --cert=tls.crt --key=tls.key

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: %{ns}
  annotations: {nginx.ingress.kubernetes.io/ssl-redirect: "true"}
spec:
  ingressClassName: nginx
  tls: [{hosts: [%{host}], secretName: %{host}-tls}]
  rules:
    - host: %{host}
      http:
        paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}]

openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt

Gotchas: -nodes on openssl req means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which kubectl create secret tls can't supply interactively. The fingerprint recorded in fp.txt must match the certificate actually stored in the Secret, not just the local tls.crt file — if you regenerate the cert after creating the Secret, they'll drift apart.

Q113 · [HARD] Gateway API Canary with Header Match

Concept: HTTPRoute rules are evaluated top-to-bottom for a match; a rule with a header matches clause only applies to requests carrying that header, so stacking a header-matched 100%-to-v2 rule before a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. parentRefs[].sectionName scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener.

Solution:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: {name: %{host}-gc}
spec: {controllerName: example.com/gw}
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{host}-gw, namespace: %{ns}}
spec:
  gatewayClassName: %{host}-gc
  listeners:
    - {name: web, protocol: HTTP, port: 80, hostname: %{host}.example.com}
    - {name: api, protocol: HTTP, port: 80, hostname: api.%{host}.example.com}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}-route, namespace: %{ns}}
spec:
  parentRefs: [{name: %{host}-gw, sectionName: web}]
  rules:
    - matches: [{headers: [{name: X-Canary, value: always}]}]
      backendRefs: [{name: v2, port: 80}]
    - backendRefs:
        - {name: v1, port: 80, weight: %{w2}}
        - {name: v2, port: 80, weight: %{w}}

Gotchas: sectionName: web is what confines this route to just the web listener — omit it and the route attaches to every listener on the Gateway, including api, which the check explicitly verifies against. Rule order is significant and not sorted by specificity automatically — the header-matched rule must physically come first in the rules list.

Q114 · [HARD] DNS Down: Three Faults

Concept: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream forward target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check each layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found.

Solution:

k -n kube-system get netpol
k -n kube-system delete netpol maintenance-lockdown

k -n kube-system scale deploy coredns --replicas=2

k -n kube-system edit cm coredns
# forward . 10.255.255.254  ->  forward . /etc/resolv.conf

k -n kube-system rollout restart deploy coredns
k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

Gotchas: fixing the Corefile alone looks successful (kubectl edit returns cleanly) but does nothing while replicas are still at 0 and/or a NetworkPolicy blocks ingress to the pods — always verify the actual symptom (nslookup from a real pod) rather than trusting that each individual fix "should" have worked. rollout restart is necessary after editing the ConfigMap because CoreDNS doesn't hot-reload Corefile changes on its own.

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

Concept: A PersistentVolume (PV) is a piece of storage capacity a cluster admin provisions (here, a hostPath directory — fine for single-node labs, unsafe for real multi-node clusters since it's tied to one specific host); a PersistentVolumeClaim (PVC) is a request for storage that the control plane binds to a matching PV based on storageClassName, accessModes, and capacity; a Pod then references the PVC (never the PV directly) via a persistentVolumeClaim volume. persistentVolumeReclaimPolicy: Retain means the underlying data survives even after the PVC is deleted (the PV just moves to Released, as explored in Q35).

Solution:

apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-data}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath: {path: /mnt/data}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pvc-data, namespace: storage}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  resources: {requests: {storage: 500Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: data-pod, namespace: storage}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: pvc-data}}]
  containers: [{name: web, image: nginx:1.25, volumeMounts: [{name: d, mountPath: /usr/share/data}]}]

Gotchas: binding requires storageClassName to match exactly on both sides (including both being explicitly manual, since an empty string and an unset field are different for binding purposes) — a PVC requesting less capacity than the PV offers is fine (the PVC gets the whole PV), but a class mismatch leaves the PVC Pending indefinitely with no PV candidate.

Q16 · Default StorageClass & Dynamic Provisioning

Concept: Exactly one StorageClass in a cluster should carry the storageclass.kubernetes.io/is-default-class: "true" annotation — if a PVC omits storageClassName entirely, the admission controller stamps in whichever class currently holds that annotation. volumeBindingMode: WaitForFirstConsumer delays actual volume provisioning until a pod referencing the PVC is scheduled, so the provisioner can pick storage local to (or otherwise appropriate for) the node the pod actually landed on — critical for node-local provisioners like rancher.io/local-path.

Solution:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-local
  annotations: {storageclass.kubernetes.io/is-default-class: "true"}
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

k annotate sc standard storageclass.kubernetes.io/is-default-class-    # remove the trailing dash = delete the annotation

Gotchas: two StorageClasses both marked default is an explicit error condition the API server tolerates but which breaks class-omitted PVCs unpredictably (the "oldest wins" tiebreak is not something to rely on) — always remove the old default's annotation, don't just add the new one. With WaitForFirstConsumer, a bare kubectl get pvc right after creation correctly shows Pending until a consuming pod is scheduled — that's expected, not a failure.

Q35 · Re-bind a Released PersistentVolume

Concept: When a PVC bound to a Retain-policy PV is deleted, the PV doesn't become Available again automatically — it moves to Released and keeps a stale spec.claimRef pointing at the now-gone PVC's UID, which blocks any new PVC from binding to it (Kubernetes won't silently hand pre-existing data to just any new claim). Manually clearing claimRef is the explicit, deliberate step that says "yes, this data is safe to reuse."

Solution:

k patch pv %{pv} --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
k get pv %{pv}   # now Available

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  volumeName: %{pv}
  resources: {requests: {storage: 500Mi}}

Gotchas: setting spec.volumeName on the PVC is what forces a specific PV rather than letting the binder pick any matching Available one by capacity/class alone — without it, if any other manual-class PV happens to exist, you could bind to the wrong one and never notice.

Q36 · Data Persistence Across Pod Restarts

Concept: The entire point of a PVC is that its backing storage outlives any individual pod — a pod is disposable, the volume is not. This task proves that end-to-end: one pod writes and exits, a completely separate pod (potentially on a different node, if the storage backend supports it) mounts the same claim and reads back exactly what was written.

Solution:

apiVersion: v1
kind: Pod
metadata: {name: writer, namespace: %{ns}}
spec:
  restartPolicy: Never
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: w, image: busybox:1.36, command: [sh, -c, "echo %{text} > /data/marker.txt"], volumeMounts: [{name: d, mountPath: /data}]}]

k -n %{ns} wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer

apiVersion: v1
kind: Pod
metadata: {name: reader, namespace: %{ns}}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: r, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]

Gotchas: restartPolicy: Never on the writer is required so it actually reaches Succeeded and stays there instead of the pod-level default (Always) endlessly re-running the same echo — and only after Succeeded is it safe to assume the write landed on disk before you spin up the reader.

Q115 · [HARD] Bind the Right Local Volume

Concept: local PersistentVolumes are tied to a specific node via nodeAffinity — the scheduler must place any pod using that volume on exactly that node, which is why choosing the wrong one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's spec.selector (label match) is the tool for picking a specific PV among several of the same StorageClass without editing the PVs themselves — exactly the constraint this task imposes.

Solution:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-fast
  selector: {matchLabels: {media: nvme}}
  resources: {requests: {storage: 1Gi}}

apiVersion: v1
kind: Pod
metadata: {name: consumer, namespace: %{ns}}
spec:
  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
  containers: [{name: c, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]

Gotchas: check kubectl get pv -l media --show-labels and cross-reference each PV's nodeAffinity against kubectl get node -l disk=nvme before writing the selector — guessing which of two identically-classed PVs is the right one gets it wrong half the time. The consumer pod's node placement is a consequence of the local PV's affinity, not something you set directly — don't add your own nodeSelector, just trust the PVC/PV binding to constrain it.

Q116 · [HARD] Migrate Data Between Claims

Concept: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway Job that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's persistentVolumeReclaimPolicy to Retain before deleting its PVC is what keeps the original data recoverable (Released, not gone) in case the migration needs to be redone.

Solution:

k patch pv $(k -n %{ns} get pvc old-data -o jsonpath='{.spec.volumeName}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: new-data, namespace: %{ns}}
spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
---
apiVersion: batch/v1
kind: Job
metadata: {name: migrate, namespace: %{ns}}
spec:
  template:
    spec:
      restartPolicy: Never
      volumes:
        - {name: src, persistentVolumeClaim: {claimName: old-data}}
        - {name: dst, persistentVolumeClaim: {claimName: new-data}}
      containers:
        - name: cp
          image: busybox:1.36
          command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
          volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]

k -n %{ns} wait --for=condition=complete job/migrate --timeout=120s
# re-point legacy: swap claimName old-data -> new-data, then recreate the pod
k -n %{ns} delete pod legacy --now
k -n %{ns} apply -f legacy.yaml
k -n %{ns} delete pvc old-data

Gotchas: cp -a /src/. /dst/ (trailing /.) copies the contents of /src into /dst, preserving permissions/timestamps — dropping the trailing dot instead creates a nested /dst/src/ directory, which silently fails any check expecting files directly under the new mount. Repointing legacy's claim requires deleting and recreating the pod (volumes[].persistentVolumeClaim.claimName isn't mutable in place) — but the task requires the Deployment/PVC naming to stay the same, not the pod's UID, so this is safe here.

Domain 5: Troubleshooting (30%)

Q17 · Node `NotReady` (Kubelet Inactive)

Concept: A node's Ready condition is entirely driven by whether its kubelet is actively heartbeating to the API server — no other signal makes a node Ready. If the kubelet process is stopped or crash-looping, the node is marked NotReady (and eventually pods get evicted) purely from the API server's perspective, even though the underlying machine is perfectly healthy.

Solution:

k get node {W2}                      # NotReady
{SHELL_W2}
systemctl status kubelet             # inactive/disabled
systemctl enable --now kubelet
journalctl -u kubelet -f --no-pager  # confirm "Successfully registered node"

Gotchas: systemctl start kubelet brings it up for this boot only — enable --now (or enable + start) is what's actually being graded, since the task requires the fix to survive a reboot.

Q18 · Broken Deployment and Service

Concept: Two independent failures are layered here — a bad image tag (ImagePullBackOff, preventing any pod from ever becoming Ready) and a Service whose selector/targetPort don't actually match the Deployment's pod labels/container port (so even a healthy pod would get zero traffic). Fixing only one leaves the app still down; kubectl describe on both objects is what surfaces each root cause independently.

Solution:

k -n troubleshoot describe deploy payments | grep -i image   # nginx:1.99-nope
k -n troubleshoot set image deploy payments payments=nginx:1.27

k -n troubleshoot get svc payments -o yaml
# selector: app: payment (typo) -- container labels are app: payments
# targetPort: 8080 -- container actually listens on 80

k -n troubleshoot patch svc payments -p \
  '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'

k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments

Gotchas: an off-by-one-character selector typo (payment vs payments) produces a Service with zero endpoints and no error at all — kubectl describe svc and get endpointslices are the tools that actually surface "selector matches nothing," since the Service object itself validates fine syntactically.

Q19 · CrashLoopBackOff Diagnosis

Concept: CrashLoopBackOff means the container keeps exiting non-zero and the kubelet is backing off restart attempts — the reason is almost always in the container's own logs, and once a container has already restarted, kubectl logs --previous is what retrieves the crashed instance's output rather than the (empty, just-started) current one.

Solution:

k -n troubleshoot logs crasher --previous | tee {WS}/q19/crasher.log
# "FATAL: required environment variable DB_HOST is not set"

k -n troubleshoot get pod crasher -o yaml > crasher.yaml
# add under spec.containers[0]:
#   env: [{name: DB_HOST, value: db.internal}]
# strip status/uid/resourceVersion/creationTimestamp before reapplying

k -n troubleshoot delete pod crasher --now
k apply -f crasher.yaml

Gotchas: a bare Pod's env field can't be patched with kubectl set env in a way that survives — since the task requires keeping the same name/namespace/image/command, editing the exported YAML and recreating it is more reliable than trying to kubectl edit a running pod (many fields are immutable post-creation, so a straight edit often just errors).

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

Concept: Pods stuck Pending forever with no scheduling errors at all (not even "insufficient resources") is the signature of the scheduler itself being down — nothing is deciding placement, so pods just sit unassigned indefinitely. Since kube-scheduler runs as a static pod, a broken manifest (here, a typo'd binary name) means the kubelet can't even start the container, and kubectl logs/describe on the resulting pod reveals the exec failure.

Solution:

k -n kube-system get pods                    # kube-scheduler in Error/CrashLoopBackOff
k -n kube-system logs kube-scheduler-{CP}     # exec: "kube-schedulerr": executable file not found

{SHELL_CP}
vi /etc/kubernetes/manifests/kube-scheduler.yaml
# command: [kube-schedulerr]  ->  command: [kube-scheduler]

k -n kube-system get pods -l component=kube-scheduler   # back to Running

Gotchas: editing a static pod manifest doesn't need a kubectl apply or restart command — the kubelet watches the manifest directory itself and recreates the pod automatically within seconds of the file changing.

Q37 · Cluster DNS Misconfiguration (Invalid Forwarding)

Concept: CoreDNS's Corefile (stored as the coredns ConfigMap) controls how it resolves names it doesn't own directly — a forward directive pointing at an unreachable or wrong upstream breaks every external/non-cluster lookup, and depending on plugin order can break in-cluster lookups too if it short-circuits before the kubernetes plugin gets a chance.

Solution:

k -n kube-system get pods -l k8s-app=kube-dns
k -n kube-system logs -l k8s-app=kube-dns --tail=20

k -n kube-system get cm coredns -o yaml
# forward . 10.255.255.1   (wrong)

k -n kube-system edit cm coredns
# forward . 10.255.255.1  ->  forward . /etc/resolv.conf

k -n kube-system rollout restart deploy coredns
k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

Gotchas: forward . /etc/resolv.conf tells CoreDNS to use the node's own resolver config for anything outside the cluster zone — that's the correct, portable default; pointing forward at a hardcoded IP address is exactly the kind of change that breaks the moment that IP stops being reachable, which is the fault being simulated here.

Q38 · Controller-Manager Crash (Invalid Flag)

Concept: When kube-controller-manager is down, existing workloads keep running untouched, but nothing new gets reconciled — a Deployment you create gets no ReplicaSet at all, since it's the controller-manager's Deployment controller that would normally create one. This distinguishes it from a scheduler failure (Q20), where pods do get created but stay unplaced.

Solution:

k -n kube-system get pods                          # controller-manager crashing
k -n kube-system logs kube-controller-manager-{CP}  # invalid flag: --leader-elect-lease-duration=1s

{SHELL_CP}
vi /etc/kubernetes/manifests/kube-controller-manager.yaml   # remove the bad flag

k create deploy probe --image=busybox:1.36 -- sleep 60
k get rs           # a ReplicaSet now appears -> controller-manager is healthy again
k delete deploy probe

Gotchas: "no ReplicaSet or pods, and nothing shows in describe deploy" is the tell — a scheduling or image problem still produces a ReplicaSet and pod objects (just unhealthy ones); a missing ReplicaSet altogether means the Deployment controller itself never ran.

Q39 · Kubelet Configuration Drift (Certificate Path)

Concept: /var/lib/kubelet/config.yaml's clientCAFile (or similar cert-path fields) points at a specific file the kubelet needs to start correctly — if that path was "tuned" to point at a file that doesn't exist, the kubelet fails at startup and the node reads as NotReady, indistinguishable at the kubectl get node level from the process simply being stopped (Q17). journalctl -u kubelet is what actually distinguishes a config error from a stopped service.

Solution:

{SHELL_W1}
systemctl status kubelet
journalctl -u kubelet -n 30 --no-pager
# "open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory"

grep -n clientCAFile /var/lib/kubelet/config.yaml
sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml

systemctl restart kubelet && systemctl enable kubelet
systemctl status kubelet

Gotchas: a "NotReady" node always deserves a journalctl -u kubelet read before assuming it's simply stopped — a config error like this one causes the service to be in a failed/restarting state, not cleanly inactive, and blindly enable --now-ing without fixing the underlying path just loops the same crash.

Q117 · [HARD] Worker Node: Three Things Wrong

Concept: This stacks three unrelated failure modes onto one node — a cordon (spec.unschedulable: true, which stops new scheduling but is unrelated to Ready status), a wrong clientCAFile path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work."

Solution:

{SHELL_W2}
systemctl status kubelet
sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml
systemctl enable --now kubelet
systemctl status kubelet
exit

k uncordon {W2}

k run test-pod --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 60
k get pod test-pod -o wide
k delete pod test-pod --now

Gotchas: a node can show Ready in kubectl get nodes while still cordoned (Ready,SchedulingDisabled) — fixing the kubelet alone gets you Ready but pods still won't schedule there until you separately uncordon; this task's grading explicitly checks both conditions independently plus an actual pod landing there.

Q118 · [HARD] Two Control-Plane Components Down

Concept: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay Pending (scheduler dead) and brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads.

Solution:

k -n kube-system get pods    # scheduler CrashLoop, controller-manager ImagePullBackOff
{SHELL_CP}

vi /etc/kubernetes/manifests/kube-scheduler.yaml
# kube-schedulerr -> kube-scheduler

vi /etc/kubernetes/manifests/kube-controller-manager.yaml
# strip the broken image-tag suffix back to a real released version

k -n kube-system get pods -w
k create deploy probe --image=busybox:1.36 -- sleep 60
k get deploy probe   # Available within a minute -> both fixed
k delete deploy probe

Gotchas: don't stop at the first component you fix reporting Running — this task's actual grading criterion is a brand-new Deployment reaching Available, which requires both the scheduler (to place the pod) and the controller-manager (to even create the ReplicaSet) working correctly at the same time.

Q119 · [HARD] Application Down for Four Reasons

Concept: Four independent faults are stacked on one app: a ResourceQuota capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's env references (CreateContainerConfigError), and a Service selector/targetPort mismatch (Q18's pattern again) — and critically, the fix must happen in place: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut.

Solution:

k -n %{ns} get all
k -n %{ns} describe deploy %{dep}
k -n %{ns} describe rs -l app=%{dep}          # "exceeded quota: pods, requested: pods=1"

k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}'
k -n %{ns} set image deploy %{dep} web=nginx:1.27

k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}'
k -n %{ns} rollout restart deploy %{dep}

k -n %{ns} patch svc %{dep} -p \
  '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"port":80,"targetPort":80}]}}'

k -n %{ns} get deploy %{dep}
k -n %{ns} get endpoints %{dep}

Gotchas: patch/edit in place (never delete+apply) is what preserves metadata.uid on the Deployment and spec.clusterIP on the Service — those two identifiers are explicitly captured before the task starts and diffed afterward, so even a perfectly-configured recreated object fails this check. Fix the quota first — with pods: "1" in force, no amount of image/env/Service fixing gets you past 1 available replica.

Q120 · [HARD] Cluster Facts, Exactly

Concept: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of kube-controller-manager/kube-apiserver (not in any single friendly kubectl get field), the cluster DNS address is just the kube-dns Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an empty mode value means the default, iptables — not "not configured"), and per-node pod CIDR lives on the Node object itself.

Solution:

k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr
k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:'   # empty -> write "iptables"
k get node %{node} -o jsonpath='{.spec.podCIDR}'

pod-cidr=<value>
service-cidr=<value>
dns-ip=<value>
proxy-mode=<value or iptables if empty>
<node>-pod-cidr=<value>

Gotchas: kube-proxy's ConfigMap often has mode: "" (an explicit empty string) rather than the key being absent entirely — both mean "use the default, iptables" and the file must say iptables either way, not literally blank. Write exactly the key=value format the task specifies, one per line, no extra whitespace — this is graded with exact string matches, not fuzzy comparison.

93 KiB Raw Blame History

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

Q02 · etcd Snapshot

Q03 · Kustomize Overlay

Q04 · CustomResourceDefinition (CRD) & Custom Resource

Q05 · Static Pod Creation

Q06 · Certificate Expiry & User CSR

Q21 · Helm Release Install and Upgrade

Q33 · Aggregated ClusterRole

Q34 · Discover CRD Field with kubectl explain

Q40 · Kubeadm Join Token Creation

Q41 · Component Versions Inspection

Q101 · [HARD] Exact RBAC with an Existing Over-Grant

Q102 · [HARD] etcd Restore

Q103 · [HARD] Static Pod on a Relocated Manifest Path

Q104 · [HARD] Self-Contained Kubeconfig Creation

Q105 · [HARD] Kustomize with Generators and Patches

Q106 · [HARD] CRD with Validation and Defaults

Q122 · [HARD] Renew Control Plane Certificates

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

Q08 · Sidecar Container & Shared Volume

Q09 · Node Affinity & Resource Requests/Limits

Q10 · ConfigMap and Secret Consumption

Q23 · Metrics Server Repair & HPA

Q24 · PriorityClass

Q25 · Taints & Tolerations

Q26 · DaemonSet on All Nodes (including Control Plane)

Q27 · CronJob Configuration

Q28 · Init Containers

Q29 · Immutable ConfigMap Rotation

Q31 · Parallel Job

Q32 · ResourceQuota & LimitRange

Q107 · [HARD] Spread, Tolerate, Prioritise — Under a Quota

Q108 · [HARD] Rollback to a Specific Revision, Then Pause

Q109 · [HARD] Native Sidecar with Probes and an Exact Report

Q110 · [HARD] Cluster Audit to Exact Files

Q121 · [HARD] The Context's Namespace

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

Q12 · NodePort Service

Q13 · Ingress Routing

Q14 · Cluster DNS Querying

Q22 · Gateway API HTTPRoute

Q30 · Named Ports in Service

Q42 · Egress-Only NetworkPolicy

Q43 · Headless & ExternalName Services

Q111 · [HARD] Zero-Trust Namespace

Q112 · [HARD] TLS Ingress with a Generated Certificate

Q113 · [HARD] Gateway API Canary with Header Match

Q114 · [HARD] DNS Down: Three Faults

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

Q16 · Default StorageClass & Dynamic Provisioning

Q35 · Re-bind a Released PersistentVolume

Q36 · Data Persistence Across Pod Restarts

Q115 · [HARD] Bind the Right Local Volume

Q116 · [HARD] Migrate Data Between Claims

Domain 5: Troubleshooting (30%)

Q17 · Node NotReady (Kubelet Inactive)

Q18 · Broken Deployment and Service

Q19 · CrashLoopBackOff Diagnosis

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

Q37 · Cluster DNS Misconfiguration (Invalid Forwarding)

Q38 · Controller-Manager Crash (Invalid Flag)

Q39 · Kubelet Configuration Drift (Certificate Path)

Q117 · [HARD] Worker Node: Three Things Wrong

Q118 · [HARD] Two Control-Plane Components Down

Q119 · [HARD] Application Down for Four Reasons

Q120 · [HARD] Cluster Facts, Exactly

93 KiB

Raw Blame History

Q34 · Discover CRD Field with `kubectl explain`

Q17 · Node `NotReady` (Kubelet Inactive)