# CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. Every entry gives you the **underlying Kubernetes concept** (why the mechanism behaves the way it does), the **exact commands/manifests** that solve the task, and the **gotchas the grader is actually checking for** — the small details that separate a passing answer from a plausible-looking one. `k` is used throughout as the `kubectl` alias (already configured on the exam VMs).

---

## Domain 1: Cluster Architecture, Installation & Configuration (25%)

### Q01 · RBAC for a Deployment Bot
- **Concept**: RBAC is purely additive — a subject's permissions are the union of every rule bound to it, and there is no "deny" verb. That means least-privilege isn't just "grant what's needed", it's also "make sure nothing else grants more" (checked here by verifying `deploy-bot` still can't touch pods or leave the namespace). A `Role`/`RoleBinding` pair is namespace-scoped; a `ClusterRole`/`ClusterRoleBinding` pair would leak the grant cluster-wide, which is exactly what this task is testing you *don't* do.
- **Solution**:
  ```bash
  k -n rbac-app create sa deploy-bot

  k -n rbac-app create role deploy-manager \
    --verb=get,list,watch,create,update,patch,delete \
    --resource=deployments.apps

  k -n rbac-app create rolebinding deploy-bot-binding \
    --role=deploy-manager \
    --serviceaccount=rbac-app:deploy-bot

  # Verify
  k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot   # yes
  k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot          # no
  k auth can-i create deployments -n default --as=system:serviceaccount:rbac-app:deploy-bot    # no
  ```
- **Gotchas**: the ServiceAccount identity for `--as` is always `system:serviceaccount:<ns>:<name>`, not just the name. Scoping `--resource=deployments.apps` (not bare `deployments`) keeps you honest about the API group — with a bare `Role` this doesn't usually matter since `create role` infers the group, but get in the habit for when a resource exists in multiple groups.

---

### Q02 · etcd Snapshot
- **Concept**: etcd is the cluster's single source of truth — every object you've ever created lives there. `etcdctl snapshot save` takes a **consistent, point-in-time** backup of the whole keyspace over etcd's own client port (2379), authenticated with the same mTLS certs the API server uses to talk to it. This is why the command needs `--cacert`/`--cert`/`--key`: etcd refuses unauthenticated clients.
- **Solution**:
  ```bash
  # On the control-plane node
  ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    snapshot save /opt/cka/etcd-snapshot.db

  # Verify it's a real, non-corrupt backup
  etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
  ```
- **Gotchas**: `ETCDCTL_API=3` matters on older etcd builds where v2 is still the default API. If etcd only runs as a pod (no host-level `etcdctl`), the etcd container image is **distroless** — no shell, no `tar`, so `kubectl cp` cannot extract a file from it. Instead, run the snapshot command via `kubectl exec` writing into `/var/lib/etcd` (which is a `hostPath` mount), then `mv` it into place from the node itself. A snapshot under ~100KB is almost certainly a failed/empty capture, not a real backup — check the file size before you move on.

---

### Q03 · Kustomize Overlay
- **Concept**: Kustomize overlays let you reuse a base manifest set across environments without templating — a `kustomization.yaml` declares transformations (namespace injection, common labels, replica counts) that are applied on top of `resources:` at build time. Crucially, the `labels:` transformer with `includeSelectors: false` adds a label to `metadata.labels` **without** touching `spec.selector`, which would otherwise break the Deployment (selectors are immutable after creation and must keep matching pod template labels).
- **Solution**:
  ```bash
  mkdir -p {WS}/q03/overlays/prod && cd {WS}/q03/overlays/prod

  cat > namespace.yaml <<EOF
  apiVersion: v1
  kind: Namespace
  metadata: {name: store-prod}
  EOF

  cat > kustomization.yaml <<EOF
  apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization
  namespace: store-prod
  resources: [../../base, namespace.yaml]
  labels:
    - pairs: {env: prod}
      includeSelectors: false
  replicas:
    - name: store-api
      count: 3
  EOF

  kubectl apply -k .
  k -n store-prod get deploy,svc -l env=prod
  ```
- **Gotchas**: the `replicas:` generator (rather than a strategic-merge patch) is the cleanest way to bump replica count in newer Kustomize — no separate patch file needed. Folding the `Namespace` object into `resources:` alongside `namespace: store-prod` guarantees the namespace exists before anything is scheduled into it, in one `apply -k`.

---

### Q04 · CustomResourceDefinition (CRD) & Custom Resource
- **Concept**: A CRD registers a brand-new API type with the API server (`<plural>.<group>`); once `Established`, the server validates, stores, and serves objects of that `kind` exactly like a built-in resource — CRUD, RBAC, `kubectl get`, all of it. The `Backup` object you create afterward is a **Custom Resource (CR)** — a plain piece of structured data with no controller behind it in this task, since nothing is watching it.
- **Solution**:
  ```yaml
  apiVersion: apiextensions.k8s.io/v1
  kind: CustomResourceDefinition
  metadata: {name: backups.data.example.com}
  spec:
    group: data.example.com
    scope: Namespaced
    names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
    versions:
      - name: v1
        served: true
        storage: true
        schema:
          openAPIV3Schema:
            type: object
            properties:
              spec:
                type: object
                properties:
                  schedule: {type: string}
                  target: {type: string}
  ```
  ```yaml
  apiVersion: data.example.com/v1
  kind: Backup
  metadata: {name: nightly, namespace: default}
  spec: {schedule: "0 2 * * *", target: "s3://backups/prod"}
  ```
  ```bash
  k apply -f crd.yaml
  k apply -f backup.yaml
  k get bk        # shortName resolves once the CRD is Established
  ```
- **Gotchas**: the single most common mistake is a mismatched `apiVersion` on the CR — it must be exactly `<spec.group>/<version>` from the CRD (here `data.example.com/v1`), **not** some other placeholder group. Get that wrong and you get `no matches for kind "Backup" in version "..."` even though the CRD applied fine. `metadata.name` on the CRD itself must be `<plural>.<group>` verbatim.

---

### Q05 · Static Pod Creation
- **Concept**: A static pod is defined by dropping a manifest file directly into a directory the **kubelet** on that node watches (`staticPodPath` in `/var/lib/kubelet/config.yaml`, default `/etc/kubernetes/manifests`) — the kubelet creates and manages it independently of the API server, then creates a read-only **mirror pod** (`<name>-<node>`) in the API so you can observe it with `kubectl`. This is exactly how kube-apiserver, kube-scheduler, etcd, and kube-controller-manager themselves run.
- **Solution**:
  ```bash
  {SHELL_W3}
  grep staticPodPath /var/lib/kubelet/config.yaml   # confirm the real path

  cat > /etc/kubernetes/manifests/static-web.yaml <<EOF
  apiVersion: v1
  kind: Pod
  metadata: {name: static-web, namespace: default}
  spec:
    containers: [{name: web, image: nginx:1.27}]
  EOF
  # kubelet polls the directory every ~20s — no restart needed.
  # Back on the main terminal: k get pods -o wide | grep static-web
  ```
- **Gotchas**: don't assume `/etc/kubernetes/manifests` — always `grep staticPodPath` first, since the task deliberately warns "the kubelet may not be configured the way you expect" in the harder variants. The mirror pod name is always `<manifest-metadata-name>-<node-name>`; deleting the mirror pod via `kubectl delete` does nothing permanent — the kubelet just recreates it from the file, so to actually remove a static pod you must delete the manifest file itself.

---

### Q06 · Certificate Expiry & User CSR
- **Concept**: kubeadm-managed clusters use a self-signed CA at `/etc/kubernetes/pki/ca.{crt,key}`; anyone holding that CA's private key can mint new client certificates. Kubernetes formalizes this via the `CertificateSigningRequest` API — you generate a keypair and CSR locally (the private key never leaves your machine), submit only the CSR to the cluster, an approver (a human, or `kubectl certificate approve`) signs it against the cluster CA, and you get back a client cert usable in a kubeconfig. Authorization is then just RBAC against the CN in that certificate.
- **Solution**:
  ```bash
  # Part A — on the control plane
  kubeadm certs check-expiration | grep '^apiserver ' > {WS}/q06/apiserver-expiry.txt

  # Part B — issue Jane a client cert
  openssl genrsa -out jane.key 2048
  openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr

  cat <<EOF | k apply -f -
  apiVersion: certificates.k8s.io/v1
  kind: CertificateSigningRequest
  metadata: {name: jane}
  spec:
    request: $(base64 -w0 jane.csr)
    signerName: kubernetes.io/kube-apiserver-client
    usages: [client auth]
  EOF

  k certificate approve jane
  k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
  k -n default create rolebinding jane-view --clusterrole=view --user=jane
  ```
- **Gotchas**: `signerName: kubernetes.io/kube-apiserver-client` is required for a CSR meant to authenticate against the API server (as opposed to the kubelet-serving signer) — pick the wrong signer and approval succeeds but the cert won't be usable the way you expect. `--user=jane` in a RoleBinding works purely because Kubernetes doesn't have a User object — any CN in a client cert *is* a username by convention; there's nothing to "create" for the user besides the cert and the binding.

---

### Q21 · Helm Release Install and Upgrade
- **Concept**: Helm packages a set of templated manifests (a *chart*) plus a `values.yaml` of defaults; `helm install` renders the templates with your overrides and applies them as one **release**, and every `helm upgrade` re-renders and re-applies, incrementing the release's revision counter (visible via `helm history`) so you can always `helm rollback` to a prior state.
- **Solution**:
  ```bash
  k create ns %{ns}
  helm install %{rel} {WS}/q21/chart -n %{ns} --set replicaCount=%{n}

  # Upgrade in place, keeping the replicaCount override, flipping the Service type
  helm upgrade %{rel} {WS}/q21/chart -n %{ns} --reuse-values --set service.type=NodePort

  helm -n %{ns} history %{rel}     # REVISION 2, deployed
  ```
- **Gotchas**: without `--reuse-values` (or repeating every prior `--set`), `helm upgrade` re-renders using the chart's **defaults** for anything you don't pass again — a common way to accidentally undo your own `replicaCount` override on upgrade. The rendered resource names come from the chart's `_helpers.tpl` (here `<release>-chart`), not the release name alone — check `helm template` output if you're unsure what a chart will actually name things.

---

### Q33 · Aggregated ClusterRole
- **Concept**: An `aggregationRule` turns a ClusterRole into a live query: the API server's ClusterRole aggregation controller continuously unions `rules` from every ClusterRole matching the given label selector into the aggregating role's own `rules` field. This lets plugin/operator ClusterRoles "attach" extra permissions to a well-known role (like the built-in `admin`/`edit`/`view`) just by carrying the right label, without anyone editing the parent role directly.
- **Solution**:
  ```bash
  cat <<EOF | k apply -f -
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata: {name: %{cr}}
  aggregationRule:
    clusterRoleSelectors:
      - matchLabels: {rbac.example.com/aggregate-to-%{cr}: "true"}
  rules: []
  EOF

  k create clusterrole %{cr}-pods --verb=get,list,watch --resource=pods
  k create clusterrole %{cr}-services --verb=get,list,watch --resource=services
  k label clusterrole %{cr}-pods %{cr}-services rbac.example.com/aggregate-to-%{cr}=true

  k create clusterrolebinding %{cr}-binding --clusterrole=%{cr} --user=%{user}
  k get clusterrole %{cr} -o yaml   # .rules is now populated
  ```
- **Gotchas**: `rules: []` on the parent is mandatory at creation — you never hand-write rules on an aggregating ClusterRole, the controller owns that field. Aggregation is asynchronous; give it a couple of seconds and re-`get` before assuming it failed. The label key must match the selector **exactly**, including the value `"true"` as a string.

---

### Q34 · Discover CRD Field with `kubectl explain`
- **Concept**: Because a CRD carries a full OpenAPI v3 schema, `kubectl explain <kind>.<path> --recursive` works on custom types exactly as it does on built-ins — it's reading the schema straight from the CRD, including each field's `description`. This is the fastest way to find an undocumented field's exact path when you can't just read the CRD YAML (or are being tested on whether you know the tool).
- **Solution**:
  ```bash
  k explain %{kind}.spec --recursive
  k explain %{kind}.spec.tuning.%{field}      # confirms the description text
  echo spec.tuning.%{field} > {WS}/q34/path.txt

  cat <<EOF | k apply -f -
  apiVersion: demo.io/v1
  kind: %{kind}
  metadata: {name: sample, namespace: default}
  spec:
    tuning: {%{field}: %{n}}
  EOF
  ```
- **Gotchas**: `--recursive` is what makes nested object fields (like `spec.tuning.*`) show up in one pass instead of having to `explain` one level at a time. Write the path with dots and no leading `.`/`kind` prefix — just `spec.tuning.<field>`, matching what the checker expects verbatim.

---

### Q40 · Kubeadm Join Token Creation
- **Concept**: Nodes join a kubeadm cluster using a **bootstrap token** (a short-lived, narrowly-scoped credential stored as a `Secret` in `kube-system`) plus a hash of the cluster CA's public key, so a joining node can verify it's talking to the real control plane before trusting it. `kubeadm token create --print-join-command` bundles both into one ready-to-run command.
- **Solution**:
  ```bash
  {SHELL_CP}
  kubeadm token create --ttl %{ttl} --print-join-command > {WS}/q40/join.sh
  kubeadm token list   # confirm it's active with the right TTL
  ```
- **Gotchas**: the printed command already includes `--discovery-token-ca-cert-hash sha256:...` — don't reconstruct it by hand. The token's expiry is derived from its `bootstrap-token-<id>` Secret's `expiration` data field, so if the grader checks TTL precisely, that's where it's actually computed from — not from wall-clock guessing.

---

### Q41 · Component Versions Inspection
- **Concept**: There's no single "cluster version" — the API server, kubelet, container runtime, and etcd are independently versioned components, and a real administrator needs to know how to read each one off the live cluster rather than assume they all match. `kubectl version` reports the server (API) version; node-level facts live under `.status.nodeInfo` on the Node object; etcd's version is baked into its static pod's container image tag since it has no separate `kubectl` subcommand to ask.
- **Solution**:
  ```bash
  k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])'   # apiserver
  k get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}'               # kubelet
  k get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'      # runtime
  k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'   # parse tag after the colon
  ```
- **Gotchas**: write exactly the value the cluster reports (`gitVersion`, not `major`/`minor` composed by hand) — a hand-typed guess like `v1.30` instead of the exact `v1.30.4` string will fail an exact-match check.

---

### Q101 · [HARD] Exact RBAC with an Existing Over-Grant
- **Concept**: This is Q01's lesson turned adversarial: someone already bound `cluster-admin` to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants **and** that nothing broader survives.
- **Solution**:
  ```bash
  k get clusterrolebinding -o wide | grep %{sa}
  k delete clusterrolebinding %{sa}-legacy-admin

  k -n %{ns} create role %{sa}-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps
  k -n %{ns} patch role %{sa}-role --type=json \
    -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["pods/exec"],"verbs":["create"]}}]'
  k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:%{sa}

  k create clusterrole %{sa}-nodes --verb=list --resource=nodes
  k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa}

  k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}
  ```
- **Gotchas**: `pods/exec` is a **subresource**, not a resource — it can only be added via a rule with `resources: ["pods/exec"]`, which `kubectl create role` can't express directly, hence the JSON-patch to append it. Deleting the `cluster-admin` binding is worth as many points as everything else combined — a technically-correct scoped Role means nothing if the old admin binding is still sitting there granting everything anyway.

---

### Q102 · [HARD] etcd Restore
- **Concept**: Restoring etcd doesn't overwrite the live data directory in place — `etcdutl snapshot restore` materializes a **brand-new** data directory from the snapshot, and you then repoint the etcd static pod's `hostPath` volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise.
- **Solution**:
  ```bash
  {SHELL_CP}
  etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore

  # In /etc/kubernetes/manifests/etcd.yaml, change the etcd-data hostPath volume:
  #   volumes:
  #   - hostPath: {path: /var/lib/etcd-restore}
  #     name: etcd-data
  vi /etc/kubernetes/manifests/etcd.yaml

  watch crictl ps                     # wait for the new etcd static pod
  kubectl -n default get cm %{cm}     # data is back
  ```
- **Gotchas**: never restore into the existing `/var/lib/etcd` directory — always a fresh path, then swap the mount, so a bad restore doesn't destroy your only copy of the working data. After the swap, verify the *whole* control plane, not just etcd — a bad data directory can leave etcd itself healthy but apiserver unable to talk to it.

---

### Q103 · [HARD] Static Pod on a Relocated Manifest Path
- **Concept**: Same mechanism as Q05, with the deliberate trap that `staticPodPath` has been changed on this node — dropping a manifest into the default `/etc/kubernetes/manifests` here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default.
- **Solution**:
  ```bash
  k get nodes -l tier=edge
  {SHELL_W3}
  grep staticPodPath /var/lib/kubelet/config.yaml    # e.g. /etc/kubernetes/static

  cat > /etc/kubernetes/static/%{pod}.yaml <<EOF
  apiVersion: v1
  kind: Pod
  metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
  spec:
    containers:
      - name: web
        image: nginx:1.27
        resources: {requests: {cpu: 50m, memory: 32Mi}}
  EOF
  ```
- **Gotchas**: check *every* node's config if you're not sure which one is "the edge node" — `kubectl get nodes -l tier=edge` finds it via the label, not guesswork. A `restart kubelet` isn't required for the file to be picked up, but doesn't hurt if you want it to reload immediately.

---

### Q104 · [HARD] Self-Contained Kubeconfig Creation
- **Concept**: A kubeconfig can reference certs either by file path (`certificate-authority`, `client-certificate`) or by embedding their base64 contents directly (`certificate-authority-data`, `client-certificate-data`). Only the embedded form is truly **portable** — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing.
- **Solution**:
  ```bash
  openssl genrsa -out %{user}.key 2048
  openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr

  k apply -f - <<EOF
  apiVersion: certificates.k8s.io/v1
  kind: CertificateSigningRequest
  metadata: {name: %{user}}
  spec: {request: $(base64 -w0 %{user}.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
  EOF
  k certificate approve %{user}
  k get csr %{user} -o jsonpath='{.status.certificate}' | base64 -d > %{user}.crt

  k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}

  KC={WS}/q104/%{user}.kubeconfig
  k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
  SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')

  kubectl --kubeconfig $KC config set-cluster lab --server=$SERVER --certificate-authority=ca.crt --embed-certs
  kubectl --kubeconfig $KC config set-credentials %{user} --client-certificate=%{user}.crt --client-key=%{user}.key --embed-certs
  kubectl --kubeconfig $KC config set-context %{user}@lab --cluster=lab --user=%{user} --namespace=%{ns}
  kubectl --kubeconfig $KC config use-context %{user}@lab

  kubectl --kubeconfig $KC get pods    # works with zero extra flags
  ```
- **Gotchas**: `--embed-certs` is the flag that matters — without it, `kubectl config set-*` writes file paths instead of inlining data, silently defeating the whole point. `--namespace` on `set-context` is what makes the context "default" into `%{ns}` so the final test command needs no `-n` flag at all. `O=developers` in the CSR subject only matters if a ClusterRoleBinding elsewhere keys off the `developers` group — for this task it's the RoleBinding on the username that actually grants access.

---

### Q105 · [HARD] Kustomize with Generators and Patches
- **Concept**: This combines every non-trivial Kustomize feature at once: a **strategic-merge patch** (`patches: [{path: replicas.yaml}]`) for a simple field override, a **JSON 6902 patch** for a structural addition that strategic-merge can't express well (injecting a whole `envFrom` array element), and a `configMapGenerator`, which — critically — appends a content hash to the generated name and then **automatically rewrites every reference to that ConfigMap** elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place.
- **Solution**:
  ```bash
  mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod

  cat > replicas.yaml <<EOF
  apiVersion: apps/v1
  kind: Deployment
  metadata: {name: web}
  spec: {replicas: %{rep}}
  EOF

  cat > envfrom.yaml <<EOF
  - op: add
    path: /spec/template/spec/containers/0/envFrom
    value: [{configMapRef: {name: web-config}}]
  EOF

  cat > kustomization.yaml <<EOF
  namespace: %{ns}
  resources: [../../base]
  labels:
    - pairs: {tier: prod}
      includeSelectors: false
  configMapGenerator:
    - name: web-config
      literals: [LOG_LEVEL=%{lvl}]
  patches:
    - path: replicas.yaml
    - path: envfrom.yaml
      target: {kind: Deployment, name: web}
  EOF

  k create ns %{ns}
  k apply -k .
  ```
- **Gotchas**: write `configMapRef: {name: web-config}` (the **unhashed** logical name) in the JSON patch — Kustomize rewrites it to `web-config-<hash>` at build time automatically; hardcoding a guessed hash would break the moment the literal changes. `includeSelectors: false` on the `labels` transformer is what keeps `tier=prod` off `spec.selector.matchLabels` (which must never change on an existing Deployment).

---

### Q106 · [HARD] CRD with Validation and Defaults
- **Concept**: A CRD's OpenAPI v3 schema is enforced by the API server itself, before an object is ever persisted: `minimum`/`maximum` reject an out-of-range number at admission time, `required` forces a field to be present, and `default` retroactively fills in a value the caller left out. This is "structural schema" validation, and it's what separates a CRD from just being a loosely-typed blob in etcd.
- **Solution**:
  ```yaml
  # CRD schema, under spec.versions[0].schema.openAPIV3Schema.properties.spec:
  required: [replicas, image]
  properties:
    replicas: {type: integer, minimum: 1, maximum: 10}
    environment: {type: string, enum: [prod, stage, dev], default: dev}
    image: {type: string}
  ```
  ```bash
  k apply -f crd.yaml

  # Step 2 — a valid object
  k apply -f - <<EOF
  apiVersion: deploy.example.com/v1
  kind: %{kind}
  metadata: {name: good}
  spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
  EOF

  # Step 3 — capture the rejection of an invalid one
  printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: bad}\nspec: {replicas: 0, image: x}\n' \
    | k apply -f - 2> {WS}/q106/rejected.txt

  # Step 4 — confirm defaulting
  printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \
    | k apply -f -
  k get <kind> defaulted -o jsonpath='{.spec.environment}'   # -> dev
  ```
- **Gotchas**: capture the rejection with `2> file` (stderr), not `> file` — `kubectl apply`'s error goes to stderr. Defaulting only fires when a field is **absent** from the request entirely, not when it's set to an empty value — and it only applies going forward, so editing an existing object to remove the field afterward does *not* retroactively re-apply the default.

---

### Q122 · [HARD] Renew Control Plane Certificates
- **Concept**: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with `kubeadm certs renew all`. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup.
- **Solution**:
  ```bash
  {SHELL_CP}
  kubeadm certs renew all

  mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/

  cp /etc/kubernetes/admin.conf ~/.kube/config

  kubeadm certs check-expiration | grep '^apiserver '
  ```
- **Gotchas**: forgetting to refresh `admin.conf` is the most common failure here — `kubeadm certs renew all` issues a new client cert for `admin.conf` too, but your **shell's own** `~/.kube/config` still has the old (now-mismatched, though not necessarily expired) credentials until you copy the new file over. Verify the actual serving certificate over the wire (`openssl s_client ... | openssl x509 -noout -serial`) matches the file on disk — that's proof the process actually reloaded, not just that the file changed.

---

## Domain 2: Workloads & Scheduling (15%)

### Q07 · Rolling Update Strategy & Scaling
- **Concept**: `RollingUpdate` (the default `Deployment` strategy) replaces pods gradually, bounded by `maxSurge` (how many *extra* pods beyond `replicas` are allowed mid-rollout) and `maxUnavailable` (how many can be missing). Setting `maxSurge: 1, maxUnavailable: 0` means Kubernetes must always create the new-image replacement pod and wait for it to be Ready *before* tearing down an old one — a zero-downtime rollout at the cost of briefly running one extra pod.
- **Solution**:
  ```bash
  k -n apps set image deploy web-frontend nginx=nginx:1.27
  k -n apps scale deploy web-frontend --replicas=4
  k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
  k -n apps rollout status deploy web-frontend
  ```
- **Gotchas**: `set image deploy/<name> <container-name>=<image>` needs the **container name**, not the deployment name — get that wrong and it silently no-ops or errors depending on kubectl version. All 4 replicas must reach `status.availableReplicas`, not just `readyReplicas` — available additionally requires the pod to have stayed ready for `minReadySeconds` (0 by default, but still a distinct field the grader checks).

---

### Q08 · Sidecar Container & Shared Volume
- **Concept**: Two containers in the same pod share the same network namespace and can share storage via a **volume** mounted into both — `emptyDir` is the simplest option, an initially-empty directory that lives as long as the pod does (not persistent across pod restarts). This is the classic log-shipping sidecar pattern: one container produces data, a second reads and forwards it, with no direct coupling between them beyond the shared filesystem path.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: log-shipper, namespace: apps}
  spec:
    volumes: [{name: logs, emptyDir: {}}]
    containers:
      - name: app
        image: busybox:1.36
        command: [sh, -c, "while true; do date >> /var/log/app.log; sleep 2; done"]
        volumeMounts: [{name: logs, mountPath: /var/log}]
      - name: shipper
        image: busybox:1.36
        command: [sh, -c, "tail -f /var/log/app.log"]
        volumeMounts: [{name: logs, mountPath: /var/log}]
  ```
  ```bash
  k -n apps logs log-shipper -c shipper
  ```
- **Gotchas**: `-c <container>` is required on both `logs` and `exec` the moment a pod has more than one container — omitting it either errors or (on older versions) defaults to the first container, silently reading the wrong logs.

---

### Q09 · Node Affinity & Resource Requests/Limits
- **Concept**: `nodeSelector` is a blunt equality match; `nodeAffinity` with `requiredDuringSchedulingIgnoredDuringExecution` expresses the same "must be on a node matching this label" constraint but supports richer operators (`In`, `NotIn`, `Exists`...) and is the mechanism the exam expects when a task explicitly says "do not use nodeSelector." `requests` tell the scheduler how much capacity to reserve when placing the pod; `limits` are enforced by the kubelet/runtime at runtime (CPU throttled, memory OOM-killed if exceeded).
- **Solution**:
  ```yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata: {name: cache, namespace: apps}
  spec:
    replicas: 2
    selector: {matchLabels: {app: cache}}
    template:
      metadata: {labels: {app: cache}}
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions: [{key: disktype, operator: In, values: [ssd]}]
        containers:
          - name: redis
            image: redis:7-alpine
            resources:
              requests: {cpu: 100m, memory: 64Mi}
              limits: {cpu: 200m, memory: 128Mi}
  ```
- **Gotchas**: the field really is `requiredDuringSchedulingIgnoredDuringExecution` — "ignored during execution" means if the node's label later changes, already-running pods are **not** evicted; only new scheduling decisions respect it. If nothing else in the cluster carries `disktype=ssd`, pods sit `Pending` forever with no error beyond an event — always confirm the label exists on the intended node first.

---

### Q10 · ConfigMap and Secret Consumption
- **Concept**: There are three distinct ways to get config into a container, and this task deliberately exercises all of them: a single key via `env[].valueFrom.configMapKeyRef` (one named env var), an entire object via `envFrom.secretRef` (every key becomes an env var, named after the key), and a full object mounted as files via a `volume` of type `configMap` (each key becomes a file, the value its contents).
- **Solution**:
  ```bash
  k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug
  k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'
  ```
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: config-pod, namespace: apps}
  spec:
    volumes: [{name: cfg, configMap: {name: app-config}}]
    containers:
      - name: app
        image: busybox:1.36
        command: [sleep, "3600"]
        env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}}]
        envFrom: [{secretRef: {name: db-creds}}]
        volumeMounts: [{name: cfg, mountPath: /etc/config}]
  ```
- **Gotchas**: `env` (singular value) and `envFrom` (whole object) are separate fields on the container spec — mixing up which mechanism the task wants for which source is the most common slip here. Quote values containing shell-special characters (`'password=S3cr3t!'`) when using `--from-literal` so the `!` doesn't trigger history expansion in an interactive shell.

---

### Q23 · Metrics Server Repair & HPA
- **Concept**: `metrics-server` scrapes each kubelet's `/metrics/resource` endpoint over TLS to compute CPU/memory usage; by default it validates the kubelet's serving certificate against a CA, which fails on clusters (like local `kind`/`minikube` setups) where kubelets use self-signed serving certs metrics-server doesn't trust — the fix is `--kubelet-insecure-tls`, which skips that validation (fine for a lab, not for production). The **HorizontalPodAutoscaler** then reads exactly this metrics pipeline to decide when to scale a Deployment.
- **Solution**:
  ```bash
  k -n kube-system edit deploy metrics-server
  # add under spec.template.spec.containers[0].args: --kubelet-insecure-tls

  k top nodes    # wait ~30s, confirm data flows

  k -n %{ns} autoscale deploy %{dep} --min=2 --max=%{max} --cpu-percent=%{cpu}
  k -n %{ns} get hpa %{dep}
  ```
- **Gotchas**: `kubectl top` can take up to a metrics-server scrape interval (~15-60s) to report data after the fix lands — a `top nodes` failure ten seconds after the edit doesn't necessarily mean the fix is wrong. `--cpu-percent` on `autoscale` maps to `averageUtilization` in the resulting HPA spec, not a raw millicore value.

---

### Q24 · PriorityClass
- **Concept**: `PriorityClass` gives pods a numeric priority the scheduler uses to decide preemption order under resource pressure — higher-priority pending pods can bump lower-priority running ones off a node. It's a cluster-scoped object referenced by name from `spec.template.spec.priorityClassName` on a workload; the priority is stamped onto each resulting pod's `.spec.priority` at admission time (changing the PriorityClass value later does **not** retroactively update already-running pods).
- **Solution**:
  ```yaml
  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata: {name: %{pc}}
  value: %{val}
  globalDefault: false
  description: "%{pc} workloads"
  ```
  ```bash
  k -n %{ns} patch deploy payments -p '{"spec":{"template":{"spec":{"priorityClassName":"%{pc}"}}}}'
  k -n %{ns} rollout status deploy payments
  ```
- **Gotchas**: patching `priorityClassName` changes the pod template, which triggers a **new rollout** — the old pods (still at the old/no priority) get replaced, so you must wait for `rollout status` before checking that live pods show the new priority value; checking too early sees stale pods.

---

### Q25 · Taints & Tolerations
- **Concept**: A taint on a node (`key=value:effect`) repels pods unless they carry a matching **toleration** — `NoSchedule` blocks new scheduling outright. Tolerating a taint doesn't *attract* a pod to that node, it only removes the repulsion; to actually land pods there you still need a separate positive signal like `nodeSelector` or `nodeAffinity`. This pairing (taint+toleration for "may schedule here" plus a selector for "must schedule here") is the standard way to dedicate a node to specific workloads.
- **Solution**:
  ```bash
  k taint node {W1} %{key}=%{val}:NoSchedule
  ```
  ```yaml
  spec:
    template:
      spec:
        nodeSelector: {kubernetes.io/hostname: "{W1}"}
        tolerations:
          - key: %{key}
            operator: Equal
            value: "%{val}"
            effect: NoSchedule
  ```
- **Gotchas**: `operator: Equal` requires `value` to match exactly; `operator: Exists` ignores `value` entirely and matches on key alone — using the wrong operator/value combination is the single most common reason a toleration silently fails to match. Without the `nodeSelector`, tolerating the taint only means the pod *could* land on `{W1}` — the scheduler might still place it elsewhere if other nodes are untainted and otherwise equally suitable.

---

### Q26 · DaemonSet on All Nodes (including Control Plane)
- **Concept**: A `DaemonSet` schedules exactly one pod per node automatically — no `replicas` field, it self-scales to `desiredNumberScheduled` = number of matching nodes. Control-plane nodes carry a `node-role.kubernetes.io/control-plane:NoSchedule` taint by default specifically to keep ordinary workloads off them; a DaemonSet must explicitly tolerate that taint to also run there, which is exactly what node-level agents (kube-proxy, CNI plugins, log/metrics collectors) do in real clusters.
- **Solution**:
  ```yaml
  apiVersion: apps/v1
  kind: DaemonSet
  metadata: {name: %{ds}, namespace: %{ns}}
  spec:
    selector: {matchLabels: {app: %{ds}}}
    template:
      metadata: {labels: {app: %{ds}}}
      spec:
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule
        containers:
          - name: agent
            image: busybox:1.36
            command: [sleep, "3600"]
            resources: {requests: {cpu: 10m, memory: 16Mi}}
  ```
- **Gotchas**: `.status.numberReady` and `.status.desiredNumberScheduled` should both equal the total node count once healthy — if they don't match, the toleration is the first thing to check (`kubectl describe ds` will show `0/1 nodes are available: 1 node(s) had untolerated taint`).

---

### Q27 · CronJob Configuration
- **Concept**: A `CronJob` is a template that spawns a new `Job` on schedule; `concurrencyPolicy: Forbid` skips a scheduled run entirely if the previous run's Job is still active (as opposed to `Allow`, the default, which lets them overlap), and the history limits bound how many completed/failed Job objects are kept around for inspection. `activeDeadlineSeconds` on the Job template is a hard wall-clock kill switch independent of the pod's own exit behavior.
- **Solution**:
  ```yaml
  apiVersion: batch/v1
  kind: CronJob
  metadata: {name: %{cj}, namespace: %{ns}}
  spec:
    schedule: '*/%{every} * * * *'
    concurrencyPolicy: Forbid
    successfulJobsHistoryLimit: 3
    failedJobsHistoryLimit: 1
    jobTemplate:
      spec:
        activeDeadlineSeconds: 30
        template:
          spec:
            restartPolicy: Never
            containers: [{name: date, image: busybox:1.36, command: ["date"]}]
  ```
  ```bash
  k -n %{ns} create job %{cj}-manual --from=cronjob/%{cj}
  ```
- **Gotchas**: `restartPolicy: Never` (or `OnFailure`) is mandatory on a Job's pod template — `Always` (the Pod-level default) is rejected outright for Jobs, since a Job's whole point is running to completion, not restarting forever. `create job --from=cronjob/<name>` is the correct one-off trigger — it clones the CronJob's pod template into a standalone Job without waiting for the schedule.

---

### Q28 · Init Containers
- **Concept**: `initContainers` run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared `emptyDir` to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q109), a plain init container fully exits before the main container's process even begins.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: %{pod}, namespace: %{ns}}
  spec:
    volumes: [{name: content, emptyDir: {}}]
    initContainers:
      - name: setup
        image: busybox:1.36
        command: [sh, -c, "echo '%{text}' > /work/index.html"]
        volumeMounts: [{name: content, mountPath: /work}]
    containers:
      - name: web
        image: nginx:1.25
        volumeMounts: [{name: content, mountPath: /usr/share/nginx/html}]
  ```
- **Gotchas**: the mount path differs deliberately between the two containers (`/work` vs `/usr/share/nginx/html`) — it's the same underlying `emptyDir`, just mounted at whatever path each container needs; they don't have to agree on the path, only on the volume `name`.

---

### Q29 · Immutable ConfigMap Rotation
- **Concept**: `immutable: true` on a ConfigMap is a performance/safety feature — it tells the API server (and lets kubelets skip) watching that object for changes, and any attempt to edit its `data` is rejected outright. The only way to "change" an immutable ConfigMap's effective content is the classic **versioned-name rollout**: create a new ConfigMap object with a new name, repoint the Deployment's volume reference at it, and let the resulting pod template change trigger a normal rolling update.
- **Solution**:
  ```bash
  k -n %{ns} create cm %{cm}-v2 --from-literal=mode=new
  k -n %{ns} patch cm %{cm}-v2 -p '{"immutable":true}'

  k -n %{ns} patch deploy %{dep} --type=json \
    -p='[{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"%{cm}-v2"}]'

  k -n %{ns} rollout status deploy %{dep}
  ```
- **Gotchas**: trying to `kubectl edit` or `patch` the *original* immutable ConfigMap's `data` fails with a clear API error — that's the mechanism working as designed, not a bug to fight around. Repointing the volume's `configMap.name` field is itself a pod-template change, so it automatically drives a rolling update — you don't need to separately `rollout restart`.

---

### Q31 · Parallel Job
- **Concept**: `completions` sets the total number of successful pod completions a Job needs before it's considered done; `parallelism` bounds how many pods run at once toward that total; `backoffLimit` caps how many pod failures the Job tolerates (with exponential backoff) before giving up entirely. This "fixed completion count" pattern is Kubernetes' model for embarrassingly-parallel batch work.
- **Solution**:
  ```yaml
  apiVersion: batch/v1
  kind: Job
  metadata: {name: %{job}, namespace: %{ns}}
  spec:
    completions: %{c}
    parallelism: %{p}
    backoffLimit: 2
    template:
      spec:
        restartPolicy: Never
        containers: [{name: runner, image: busybox:1.36, command: ["sh", "-c", "echo done; sleep 2"]}]
  ```
  ```bash
  k -n %{ns} wait --for=condition=complete job/%{job} --timeout=120s
  ```
- **Gotchas**: `.status.succeeded` must reach exactly `completions`, not just "some pods ran" — with `parallelism < completions` the Job launches new pods in waves as earlier ones finish, so checking too early under-counts.

---

### Q32 · ResourceQuota & LimitRange
- **Concept**: `ResourceQuota` caps the **sum** of requests/limits (and object counts) across an entire namespace; `LimitRange` instead governs individual containers — setting defaults for pods that don't specify `resources` at all, and floor/ceiling bounds for ones that do. The two are complementary: LimitRange makes sure every pod has *some* request/limit (which quota accounting requires to even function sensibly), and quota then caps the namespace-wide total.
- **Solution**:
  ```bash
  k -n %{ns} create quota %{ns}-quota --hard=pods=%{pods},requests.cpu=1,limits.memory=2Gi
  ```
  ```yaml
  apiVersion: v1
  kind: LimitRange
  metadata: {name: %{ns}-defaults, namespace: %{ns}}
  spec:
    limits:
      - type: Container
        default: {cpu: 500m, memory: 256Mi}
        defaultRequest: {cpu: %{dcpu}, memory: 128Mi}
        max: {cpu: 500m, memory: 512Mi}
        min: {cpu: 50m, memory: 64Mi}
  ```
- **Gotchas**: once a `ResourceQuota` with `requests.cpu`/`limits.memory` exists in a namespace, the API server **rejects** any new pod whose containers don't specify those resource fields — this is precisely why the LimitRange's `default`/`defaultRequest` matter: they're what let the plain `worker` Deployment (created with no resources at all) get admitted, by injecting values before the quota check runs.

---

### Q107 · [HARD] Spread, Tolerate, Prioritise — Under a Quota
- **Concept**: This stacks four scheduling primitives that each solve a different problem: **pod anti-affinity** (`requiredDuringSchedulingIgnoredDuringExecution` on `topologyKey: kubernetes.io/hostname`) forces one replica per node by refusing to co-locate pods matching a label selector; a **toleration** lets pods land on an otherwise-repelling tainted node; **PriorityClass** affects preemption, not placement directly; and a **ResourceQuota** caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it.
- **Solution**:
  ```yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata: {name: %{dep}, namespace: %{ns}}
  spec:
    replicas: 3
    selector: {matchLabels: {app: %{dep}}}
    template:
      metadata: {labels: {app: %{dep}}}
      spec:
        priorityClassName: %{dep}-prio
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector: {matchLabels: {app: %{dep}}}
                topologyKey: kubernetes.io/hostname
        tolerations:
          - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule}
        containers:
          - name: app
            image: nginx:1.27
            resources: {requests: {cpu: 100m, memory: 64Mi}}
  ```
  ```bash
  k create priorityclass %{dep}-prio --value=50000
  ```
- **Gotchas**: with exactly 3 worker nodes and `replicas: 3` under required anti-affinity, there is zero slack — if the quota's `pods` cap is lower than 3, or CPU requests don't fit the quota's `requests.cpu` budget, pods stay `Pending` with no scheduling error beyond an event; check `kubectl -n %{ns} get quota` and size requests accordingly rather than raising replicas blindly.

---

### Q108 · [HARD] Rollback to a Specific Revision, Then Pause
- **Concept**: Every `kubectl rollout undo --to-revision=N` targets a specific historical **ReplicaSet** by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. `rollout pause` freezes a Deployment's controller from acting on template changes: you can edit `spec.template` afterward all you like (image, env, whatever) and the controller will **not** create a new ReplicaSet or touch running pods until you `resume` — a safe way to stage a change for later.
- **Solution**:
  ```bash
  k -n %{ns} rollout history deploy %{dep}
  k -n %{ns} rollout undo deploy %{dep} --to-revision=%{rev}
  k -n %{ns} scale deploy %{dep} --replicas=3
  k -n %{ns} rollout status deploy %{dep}

  k -n %{ns} rollout pause deploy %{dep}
  k -n %{ns} set image deploy/%{dep} nginx=nginx:1.27
  k -n %{ns} set env deploy/%{dep} RELEASE=next
  # leave it paused — no rollout status wait here, by design
  ```
- **Gotchas**: `rollout history` only shows revisions if `.spec.revisionHistoryLimit` hasn't pruned them and `--record`/change-cause tracking was in effect — if the image alone doesn't tell you which revision is `nginx:1.25`, add `--revision=N` to `rollout history` to inspect each one's pod template. The whole point of pausing *before* editing is that no new ReplicaSet should exist for the post-pause template — if you edited before pausing, a rollout would have already started.

---

### Q109 · [HARD] Native Sidecar with Probes and an Exact Report
- **Concept**: Kubernetes 1.28+ added **native sidecars**: an `initContainers` entry with `restartPolicy: Always` starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with `terminationGracePeriodSeconds`, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: %{pod}, namespace: %{ns}}
  spec:
    terminationGracePeriodSeconds: 5
    initContainers:
      - name: agent
        image: busybox:1.36
        restartPolicy: Always
        command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"]
        volumeMounts: [{name: logs, mountPath: /var/log}]
    containers:
      - name: app
        image: nginx:1.27
        volumeMounts: [{name: logs, mountPath: /var/log}]
        readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5}
        livenessProbe: {tcpSocket: {port: 80}}
    volumes: [{name: logs, emptyDir: {}}]
  ```
  ```bash
  k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \
    | tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt
  k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt
  ```
- **Gotchas**: `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte.

---

### Q110 · [HARD] Cluster Audit to Exact Files
- **Concept**: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always `-o json` piped through `jq`/`python3`, never hand-transcribing what `kubectl get` prints to the terminal.
- **Solution**:
  ```bash
  # <namespace>/<pod> for pods (excluding kube-system) with no readiness probe on their first container
  k get pods -A -o json | python3 -c '
  import sys,json
  for p in json.load(sys.stdin)["items"]:
      if p["metadata"]["namespace"]=="kube-system": continue
      if not p["spec"]["containers"][0].get("readinessProbe"):
          print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q110/no-readiness.txt

  # <node> <count> of Running pods per node, all namespaces
  k get pods -A -o json | python3 -c '
  import sys,json,collections
  c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"]
      if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
  for n in sorted(c): print(n, c[n])' > {WS}/q110/per-node.txt

  # every distinct image running in kube-system
  k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
    | tr ' ' '\n' | sort -u > {WS}/q110/images.txt
  ```
- **Gotchas**: no header lines, no trailing commentary — the checker does a literal `sort`-then-compare against each file's contents. Excluding `kube-system` from the first file but restricting *to* `kube-system` in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners.

---

### Q121 · [HARD] The Context's Namespace
- **Concept**: A kubeconfig **context** bundles a cluster, a user, and a default namespace (`contexts[].context.namespace`) — any command run without an explicit `-n`/`--namespace` inherits that default, falling back to literal `default` only if the context doesn't set one. `kubectl config view --minify` collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's `default`.
- **Solution**:
  ```bash
  k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt

  # No -n anywhere below — everything inherits the context's namespace
  k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
  k label deploy %{dep} owner=ac --overwrite
  k expose deploy %{dep} --port=9000 --target-port=9000
  ```
- **Gotchas**: it's tempting to just assume the context's namespace is `default` and pass `-n default` explicitly to "be safe" — that defeats the entire point of the task and the checker specifically verifies nothing landed in `default` by mistake. `{..namespace}` (double-dot JSONPath) walks the whole minified document since the exact key path depends on kubeconfig structure version; `.contexts[0].context.namespace` also works once you know there's only one context after `--minify`.

---

## Domain 3: Services & Networking (20%)

### Q11 · NetworkPolicy (Ingress Restriction)
- **Concept**: NetworkPolicies are **allow-list only** and scoped by `podSelector` — the moment any policy selects a pod for a given `policyTypes` direction, that direction becomes default-deny for that pod except for what the policy(ies) explicitly permit; pods not selected by any policy remain fully open. A policy needs a CNI that enforces NetworkPolicy (Calico, Cilium, etc.) to actually have effect — plain `kubectl apply` succeeding only proves the object is well-formed, not that traffic is blocked, which is why lab environments grade the spec, not live traffic.
- **Solution**:
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: backend-policy, namespace: netpol}
  spec:
    podSelector: {matchLabels: {app: backend}}
    policyTypes: [Ingress]
    ingress:
      - from: [{podSelector: {matchLabels: {app: frontend}}}]
        ports: [{protocol: TCP, port: 8080}]
  ```
- **Gotchas**: omitting `policyTypes: [Egress]` is what leaves egress **unrestricted** — `policyTypes` doesn't default to "whatever directions have rules," you must list exactly the directions you intend to lock down. A single `ingress` rule with one `from` entry and one `ports` entry is strictly narrower than the same rule split across multiple list entries (which OR together) — keep it to one of each so nothing broader sneaks in.

---

### Q12 · NodePort Service
- **Concept**: `type: NodePort` allocates (or, if you specify one, reserves) a port in the `30000-32767` range on **every** node, forwarding traffic on `<any-node-IP>:nodePort` through to the Service's `ClusterIP:port`, which in turn load-balances to matching pods via `targetPort`. It's built on top of `ClusterIP` — a NodePort Service still gets a normal cluster-internal IP too.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Service
  metadata: {name: shop-svc, namespace: svc}
  spec:
    type: NodePort
    selector: {app: shop}
    ports: [{port: 80, targetPort: 80, nodePort: 30080}]
  ```
- **Gotchas**: "has ready endpoints" isn't automatic just because the Service object exists — the `selector` must actually match labels on **running, ready** pods, or the EndpointSlice stays empty and the NodePort accepts connections that go nowhere.

---

### Q13 · Ingress Routing
- **Concept**: An `Ingress` is a declarative routing table an **Ingress controller** (nginx, etc.) reads and turns into real reverse-proxy config — the object itself is inert without a controller watching `ingressClassName`, but its structure (host, path rules, backend service+port) is exactly what any controller expects. Multiple `paths` under one `host` are matched in order by `pathType`; `Prefix` matches on path segments, not raw string prefixes (`/api` matches `/api/v1` but not `/apix`).
- **Solution**:
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata: {name: app-ingress, namespace: ingress-app}
  spec:
    ingressClassName: nginx
    rules:
      - host: shop.example.com
        http:
          paths:
            - {path: /api, pathType: Prefix, backend: {service: {name: api, port: {number: 8080}}}}
            - {path: /,    pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}
  ```
- **Gotchas**: list the more specific path (`/api`) **before** the catch-all `/` — most controllers evaluate in list order for equal-specificity prefix matches, and putting `/` first can shadow `/api` entirely depending on the controller.

---

### Q14 · Cluster DNS Querying
- **Concept**: CoreDNS auto-generates two DNS record families cluster-wide: Service A-records at `<svc>.<namespace>.svc.cluster.local`, and (if pod DNS is enabled) pod A-records at `<pod-ip-with-dashes>.<namespace>.pod.cluster.local`. Every pod's `/etc/resolv.conf` gets a `search` list that includes the local namespace's `svc.cluster.local` suffix, which is why a **short** name like `hello` often resolves too — but the fully-qualified form is what's actually being tested here since it works regardless of search-domain configuration.
- **Solution**:
  ```bash
  k -n dns-check run dns-test --image=busybox:1.36 -- sleep 3600
  echo 'hello.dns-check.svc.cluster.local' > {WS}/q14/svc-fqdn.txt
  k -n dns-check exec dns-test -- nslookup hello.dns-check.svc.cluster.local > {WS}/q14/nslookup.txt
  ```
- **Gotchas**: the FQDN check tolerates an optional trailing dot (`...cluster.local.`, the technically-correct fully-qualified form with the root zone) — but don't add anything else. `nslookup`'s output includes the resolver's own address as the first two lines; the checker just greps for the Service's actual ClusterIP appearing somewhere in the file, so raw redirected output is fine.

---

### Q22 · Gateway API HTTPRoute
- **Concept**: The Gateway API is the newer, role-oriented replacement for Ingress: a `GatewayClass` (cluster-scoped, names a controller implementation), a `Gateway` (a specific listener set, e.g. host+port), and an `HTTPRoute` (namespace-scoped routing rules that attach to a Gateway via `parentRefs`) are three separate objects, deliberately split so infra teams own the Gateway and app teams own their own HTTPRoutes independently.
- **Solution**:
  ```yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: GatewayClass
  metadata: {name: %{gc}}
  spec: {controllerName: example.com/gateway}
  ---
  apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata: {name: %{gw}, namespace: %{ns}}
  spec:
    gatewayClassName: %{gc}
    listeners: [{name: http, protocol: HTTP, port: 80, hostname: %{host}.example.com}]
  ---
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata: {name: %{host}-route, namespace: %{ns}}
  spec:
    parentRefs: [{name: %{gw}}]
    rules:
      - matches: [{path: {type: PathPrefix, value: /api}}]
        backendRefs: [{name: api, port: 8080}]
      - backendRefs: [{name: web, port: 80}]
  ```
- **Gotchas**: the CRDs (`gateways.gateway.networking.k8s.io`, `httproutes...`) must exist before any of this — install the standard-channel release first (`kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml`) or every subsequent `apply` fails with "no matches for kind." Order matters in `rules:` — put the specific `/api` match before the catch-all rule with no `matches` at all (which matches everything).

---

### Q30 · Named Ports in Service
- **Concept**: A container port can carry a `name` alongside its number; a Service's `targetPort` can then reference that **name** instead of a number, and so can probes (`httpGet.port`). This indirection means the container can change its actual listening port without anyone needing to update the Service or probe — only the pod spec's `ports[].containerPort` needs to change, since everything else points at the stable name.
- **Solution**:
  ```yaml
  # container:
  ports: [{name: %{pname}, containerPort: 80}]
  readinessProbe: {httpGet: {path: /, port: %{pname}}}
  ```
  ```yaml
  apiVersion: v1
  kind: Service
  metadata: {name: %{dep}, namespace: %{ns}}
  spec:
    selector: {app: %{app}}
    ports: [{name: http, port: 8080, targetPort: %{pname}}]
  ```
- **Gotchas**: `targetPort: %{pname}` must be the literal **string** name, not the number — YAML will happily accept either syntactically, but a numeric `targetPort` silently ignores the named-port feature entirely and just targets that raw port number.

---

### Q42 · Egress-Only NetworkPolicy
- **Concept**: An egress policy needs its **DNS rule spelled out explicitly** — without an allowed egress path to CoreDNS on port 53, a default-deny-egress pod can't resolve any name at all, including ones needed for the very traffic you're trying to permit (like an external CIDR by hostname). This is the most common real-world NetworkPolicy gotcha: people lock down egress and then wonder why *everything* breaks, DNS included.
- **Solution**:
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: %{app}-egress, namespace: %{ns}}
  spec:
    podSelector: {matchLabels: {app: %{app}}}
    policyTypes: [Egress]
    egress:
      - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}]
        ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
      - to: [{ipBlock: {cidr: %{cidr}}}]
        ports: [{protocol: TCP, port: %{port}}]
  ```
- **Gotchas**: combining `namespaceSelector` **and** `podSelector` in the same `to[]` entry means "pods matching X *in* namespaces matching Y" (an AND) — putting them in two separate `to[]` list entries instead would mean "any pod in namespace Y" OR "any pod matching X anywhere," which is far broader than intended. `kubernetes.io/metadata.name` is the automatically-applied label every namespace carries with its own name, so it's the reliable way to target "the kube-system namespace" without assuming someone hand-labeled it.

---

### Q43 · Headless & ExternalName Services
- **Concept**: A **headless Service** (`clusterIP: None`) skips virtual-IP load-balancing entirely — DNS for the Service name returns the individual pod IPs directly, and for a `StatefulSet` specifically, CoreDNS additionally publishes a stable per-pod record `<pod-name>.<service-name>.<namespace>.svc.cluster.local`, which is what lets StatefulSet members address each other individually. An **ExternalName** Service is pure DNS aliasing — no selector, no endpoints, just a CNAME to an external hostname, letting in-cluster code refer to an external dependency via a normal-looking Service name.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Service
  metadata: {name: %{sts}, namespace: %{ns}}
  spec:
    clusterIP: None
    selector: {app: %{sts}}
    ports: [{port: 5432, targetPort: 5432}]
  ---
  apiVersion: v1
  kind: Service
  metadata: {name: legacy, namespace: %{ns}}
  spec:
    type: ExternalName
    externalName: %{ext}
  ```
  ```bash
  k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt
  ```
- **Gotchas**: the headless Service's name **must match** `spec.serviceName` on the StatefulSet — that field is what wires the StatefulSet's per-pod DNS into that specific headless Service; a mismatched name means the stable pod hostnames never resolve, even though both objects individually look fine. `type: ExternalName` Services never get a `ClusterIP` or endpoints at all — that's expected, not a misconfiguration.

---

### Q111 · [HARD] Zero-Trust Namespace
- **Concept**: True zero-trust starts from **default-deny both directions**, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either.
- **Solution**:
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: default-deny, namespace: %{ns}}
  spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
  ---
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: allow-dns, namespace: %{ns}}
  spec:
    podSelector: {}
    policyTypes: [Egress]
    egress:
      - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}}]
        ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
  ---
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: api-ingress, namespace: %{ns}}
  spec:
    podSelector: {matchLabels: {app: api}}
    policyTypes: [Ingress]
    ingress: [{from: [{namespaceSelector: {matchLabels: {team: %{from}}}}], ports: [{port: 8080}]}]
  ---
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: api-to-db, namespace: %{ns}}
  spec:
    podSelector: {matchLabels: {app: api}}
    policyTypes: [Egress]
    egress: [{to: [{podSelector: {matchLabels: {app: db}}}], ports: [{port: 5432}]}]
  ---
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: db-ingress, namespace: %{ns}}
  spec:
    podSelector: {matchLabels: {app: db}}
    policyTypes: [Ingress]
    ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{port: 5432}]}]
  ```
  ```bash
  k label ns %{from} team=%{from}
  ```
- **Gotchas**: `podSelector: {}` (empty selector) means "every pod in this namespace" — don't confuse it with an empty `from`/`to` array, which instead means "match nothing" (fully deny that direction). This task's setup deliberately leaves a `temp-allow-all` policy behind as a trap; deleting every contradicting leftover policy is graded explicitly, not just adding the five you're told about.

---

### Q112 · [HARD] TLS Ingress with a Generated Certificate
- **Concept**: TLS termination at the Ingress layer needs a `kubernetes.io/tls` Secret holding `tls.crt`/`tls.key`, referenced from `spec.tls[].secretName`; for a self-signed lab cert, `openssl req -x509` generates both the key and a self-signed certificate in one step (no separate CA involved) with the `CN` acting as the hostname it's valid for.
- **Solution**:
  ```bash
  openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt
  k -n %{ns} create secret tls %{host}-tls --cert=tls.crt --key=tls.key
  ```
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: web
    namespace: %{ns}
    annotations: {nginx.ingress.kubernetes.io/ssl-redirect: "true"}
  spec:
    ingressClassName: nginx
    tls: [{hosts: [%{host}], secretName: %{host}-tls}]
    rules:
      - host: %{host}
        http:
          paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}]
  ```
  ```bash
  openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt
  ```
- **Gotchas**: `-nodes` on `openssl req` means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which `kubectl create secret tls` can't supply interactively. The fingerprint recorded in `fp.txt` must match the certificate **actually stored in the Secret**, not just the local `tls.crt` file — if you regenerate the cert after creating the Secret, they'll drift apart.

---

### Q113 · [HARD] Gateway API Canary with Header Match
- **Concept**: `HTTPRoute` rules are evaluated top-to-bottom for a match; a rule with a header `matches` clause only applies to requests carrying that header, so stacking a header-matched 100%-to-`v2` rule *before* a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. `parentRefs[].sectionName` scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener.
- **Solution**:
  ```yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: GatewayClass
  metadata: {name: %{host}-gc}
  spec: {controllerName: example.com/gw}
  ---
  apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata: {name: %{host}-gw, namespace: %{ns}}
  spec:
    gatewayClassName: %{host}-gc
    listeners:
      - {name: web, protocol: HTTP, port: 80, hostname: %{host}.example.com}
      - {name: api, protocol: HTTP, port: 80, hostname: api.%{host}.example.com}
  ---
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata: {name: %{host}-route, namespace: %{ns}}
  spec:
    parentRefs: [{name: %{host}-gw, sectionName: web}]
    rules:
      - matches: [{headers: [{name: X-Canary, value: always}]}]
        backendRefs: [{name: v2, port: 80}]
      - backendRefs:
          - {name: v1, port: 80, weight: %{w2}}
          - {name: v2, port: 80, weight: %{w}}
  ```
- **Gotchas**: `sectionName: web` is what confines this route to just the `web` listener — omit it and the route attaches to *every* listener on the Gateway, including `api`, which the check explicitly verifies against. Rule order is significant and not sorted by specificity automatically — the header-matched rule must physically come first in the `rules` list.

---

### Q114 · [HARD] DNS Down: Three Faults
- **Concept**: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream `forward` target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check **each** layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found.
- **Solution**:
  ```bash
  k -n kube-system get netpol
  k -n kube-system delete netpol maintenance-lockdown

  k -n kube-system scale deploy coredns --replicas=2

  k -n kube-system edit cm coredns
  # forward . 10.255.255.254  ->  forward . /etc/resolv.conf

  k -n kube-system rollout restart deploy coredns
  k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
  ```
- **Gotchas**: fixing the Corefile alone looks successful (`kubectl edit` returns cleanly) but does nothing while replicas are still at 0 and/or a NetworkPolicy blocks ingress to the pods — always verify the actual symptom (`nslookup` from a real pod) rather than trusting that each individual fix "should" have worked. `rollout restart` is necessary after editing the ConfigMap because CoreDNS doesn't hot-reload `Corefile` changes on its own.

---

## Domain 4: Storage (10%)

### Q15 · PersistentVolume, PVC & Pod Mounting
- **Concept**: A `PersistentVolume` (PV) is a piece of storage capacity a cluster admin provisions (here, a `hostPath` directory — fine for single-node labs, unsafe for real multi-node clusters since it's tied to one specific host); a `PersistentVolumeClaim` (PVC) is a *request* for storage that the control plane binds to a matching PV based on `storageClassName`, `accessModes`, and capacity; a Pod then references the PVC (never the PV directly) via a `persistentVolumeClaim` volume. `persistentVolumeReclaimPolicy: Retain` means the underlying data survives even after the PVC is deleted (the PV just moves to `Released`, as explored in Q35).
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: PersistentVolume
  metadata: {name: pv-data}
  spec:
    capacity: {storage: 1Gi}
    accessModes: [ReadWriteOnce]
    persistentVolumeReclaimPolicy: Retain
    storageClassName: manual
    hostPath: {path: /mnt/data}
  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata: {name: pvc-data, namespace: storage}
  spec:
    accessModes: [ReadWriteOnce]
    storageClassName: manual
    resources: {requests: {storage: 500Mi}}
  ---
  apiVersion: v1
  kind: Pod
  metadata: {name: data-pod, namespace: storage}
  spec:
    volumes: [{name: d, persistentVolumeClaim: {claimName: pvc-data}}]
    containers: [{name: web, image: nginx:1.25, volumeMounts: [{name: d, mountPath: /usr/share/data}]}]
  ```
- **Gotchas**: binding requires `storageClassName` to match **exactly** on both sides (including both being explicitly `manual`, since an empty string and an unset field are different for binding purposes) — a PVC requesting less capacity than the PV offers is fine (the PVC gets the whole PV), but a class mismatch leaves the PVC `Pending` indefinitely with no PV candidate.

---

### Q16 · Default StorageClass & Dynamic Provisioning
- **Concept**: Exactly one StorageClass in a cluster should carry the `storageclass.kubernetes.io/is-default-class: "true"` annotation — if a PVC omits `storageClassName` entirely, the admission controller stamps in whichever class currently holds that annotation. `volumeBindingMode: WaitForFirstConsumer` delays actual volume provisioning until a pod referencing the PVC is scheduled, so the provisioner can pick storage local to (or otherwise appropriate for) the node the pod actually landed on — critical for node-local provisioners like `rancher.io/local-path`.
- **Solution**:
  ```yaml
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: fast-local
    annotations: {storageclass.kubernetes.io/is-default-class: "true"}
  provisioner: rancher.io/local-path
  volumeBindingMode: WaitForFirstConsumer
  reclaimPolicy: Delete
  ```
  ```bash
  k annotate sc standard storageclass.kubernetes.io/is-default-class-    # remove the trailing dash = delete the annotation
  ```
- **Gotchas**: two StorageClasses both marked default is an explicit error condition the API server tolerates but which breaks class-omitted PVCs unpredictably (the "oldest wins" tiebreak is not something to rely on) — always remove the old default's annotation, don't just add the new one. With `WaitForFirstConsumer`, a bare `kubectl get pvc` right after creation correctly shows `Pending` until a consuming pod is scheduled — that's expected, not a failure.

---

### Q35 · Re-bind a Released PersistentVolume
- **Concept**: When a PVC bound to a `Retain`-policy PV is deleted, the PV doesn't become `Available` again automatically — it moves to `Released` and keeps a stale `spec.claimRef` pointing at the now-gone PVC's UID, which blocks any new PVC from binding to it (Kubernetes won't silently hand pre-existing data to just any new claim). Manually clearing `claimRef` is the explicit, deliberate step that says "yes, this data is safe to reuse."
- **Solution**:
  ```bash
  k patch pv %{pv} --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
  k get pv %{pv}   # now Available
  ```
  ```yaml
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata: {name: %{pvc}, namespace: %{ns}}
  spec:
    accessModes: [ReadWriteOnce]
    storageClassName: manual
    volumeName: %{pv}
    resources: {requests: {storage: 500Mi}}
  ```
- **Gotchas**: setting `spec.volumeName` on the PVC is what forces a **specific** PV rather than letting the binder pick any matching `Available` one by capacity/class alone — without it, if any other `manual`-class PV happens to exist, you could bind to the wrong one and never notice.

---

### Q36 · Data Persistence Across Pod Restarts
- **Concept**: The entire point of a PVC is that its backing storage outlives any individual pod — a pod is disposable, the volume is not. This task proves that end-to-end: one pod writes and exits, a completely separate pod (potentially on a different node, if the storage backend supports it) mounts the same claim and reads back exactly what was written.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: writer, namespace: %{ns}}
  spec:
    restartPolicy: Never
    volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
    containers: [{name: w, image: busybox:1.36, command: [sh, -c, "echo %{text} > /data/marker.txt"], volumeMounts: [{name: d, mountPath: /data}]}]
  ```
  ```bash
  k -n %{ns} wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer
  ```
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: reader, namespace: %{ns}}
  spec:
    volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
    containers: [{name: r, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]
  ```
- **Gotchas**: `restartPolicy: Never` on the writer is required so it actually reaches `Succeeded` and stays there instead of the pod-level default (`Always`) endlessly re-running the same `echo` — and only after `Succeeded` is it safe to assume the write landed on disk before you spin up the reader.

---

### Q115 · [HARD] Bind the Right Local Volume
- **Concept**: `local` PersistentVolumes are tied to a specific node via `nodeAffinity` — the scheduler must place any pod using that volume on exactly that node, which is why choosing the *wrong* one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's `spec.selector` (label match) is the tool for picking a specific PV among several of the same StorageClass **without editing the PVs themselves** — exactly the constraint this task imposes.
- **Solution**:
  ```yaml
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata: {name: %{pvc}, namespace: %{ns}}
  spec:
    accessModes: [ReadWriteOnce]
    storageClassName: local-fast
    selector: {matchLabels: {media: nvme}}
    resources: {requests: {storage: 1Gi}}
  ```
  ```yaml
  apiVersion: v1
  kind: Pod
  metadata: {name: consumer, namespace: %{ns}}
  spec:
    volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
    containers: [{name: c, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]
  ```
- **Gotchas**: check `kubectl get pv -l media --show-labels` and cross-reference each PV's `nodeAffinity` against `kubectl get node -l disk=nvme` **before** writing the selector — guessing which of two identically-classed PVs is the right one gets it wrong half the time. The consumer pod's node placement is a *consequence* of the local PV's affinity, not something you set directly — don't add your own `nodeSelector`, just trust the PVC/PV binding to constrain it.

---

### Q116 · [HARD] Migrate Data Between Claims
- **Concept**: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway `Job` that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's `persistentVolumeReclaimPolicy` to `Retain` *before* deleting its PVC is what keeps the original data recoverable (`Released`, not gone) in case the migration needs to be redone.
- **Solution**:
  ```bash
  k patch pv $(k -n %{ns} get pvc old-data -o jsonpath='{.spec.volumeName}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
  ```
  ```yaml
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata: {name: new-data, namespace: %{ns}}
  spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
  ---
  apiVersion: batch/v1
  kind: Job
  metadata: {name: migrate, namespace: %{ns}}
  spec:
    template:
      spec:
        restartPolicy: Never
        volumes:
          - {name: src, persistentVolumeClaim: {claimName: old-data}}
          - {name: dst, persistentVolumeClaim: {claimName: new-data}}
        containers:
          - name: cp
            image: busybox:1.36
            command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
            volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
  ```
  ```bash
  k -n %{ns} wait --for=condition=complete job/migrate --timeout=120s
  # re-point legacy: swap claimName old-data -> new-data, then recreate the pod
  k -n %{ns} delete pod legacy --now
  k -n %{ns} apply -f legacy.yaml
  k -n %{ns} delete pvc old-data
  ```
- **Gotchas**: `cp -a /src/. /dst/` (trailing `/.`) copies the **contents** of `/src` into `/dst`, preserving permissions/timestamps — dropping the trailing dot instead creates a nested `/dst/src/` directory, which silently fails any check expecting files directly under the new mount. Repointing `legacy`'s claim requires deleting and recreating the pod (`volumes[].persistentVolumeClaim.claimName` isn't mutable in place) — but the task requires the *Deployment/PVC* naming to stay the same, not the pod's UID, so this is safe here.

---

## Domain 5: Troubleshooting (30%)

### Q17 · Node `NotReady` (Kubelet Inactive)
- **Concept**: A node's `Ready` condition is entirely driven by whether its kubelet is actively heartbeating to the API server — no other signal makes a node `Ready`. If the kubelet process is stopped or crash-looping, the node is marked `NotReady` (and eventually pods get evicted) purely from the API server's perspective, even though the underlying machine is perfectly healthy.
- **Solution**:
  ```bash
  k get node {W2}                      # NotReady
  {SHELL_W2}
  systemctl status kubelet             # inactive/disabled
  systemctl enable --now kubelet
  journalctl -u kubelet -f --no-pager  # confirm "Successfully registered node"
  ```
- **Gotchas**: `systemctl start kubelet` brings it up for this boot only — `enable --now` (or `enable` + `start`) is what's actually being graded, since the task requires the fix to survive a reboot.

---

### Q18 · Broken Deployment and Service
- **Concept**: Two independent failures are layered here — a bad image tag (`ImagePullBackOff`, preventing any pod from ever becoming Ready) and a Service whose `selector`/`targetPort` don't actually match the Deployment's pod labels/container port (so even a healthy pod would get zero traffic). Fixing only one leaves the app still down; `kubectl describe` on both objects is what surfaces each root cause independently.
- **Solution**:
  ```bash
  k -n troubleshoot describe deploy payments | grep -i image   # nginx:1.99-nope
  k -n troubleshoot set image deploy payments payments=nginx:1.27

  k -n troubleshoot get svc payments -o yaml
  # selector: app: payment (typo) -- container labels are app: payments
  # targetPort: 8080 -- container actually listens on 80

  k -n troubleshoot patch svc payments -p \
    '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'

  k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments
  ```
- **Gotchas**: an off-by-one-character selector typo (`payment` vs `payments`) produces a Service with **zero** endpoints and no error at all — `kubectl describe svc` and `get endpointslices` are the tools that actually surface "selector matches nothing," since the Service object itself validates fine syntactically.

---

### Q19 · CrashLoopBackOff Diagnosis
- **Concept**: `CrashLoopBackOff` means the container keeps exiting non-zero and the kubelet is backing off restart attempts — the *reason* is almost always in the container's own logs, and once a container has already restarted, `kubectl logs --previous` is what retrieves the crashed instance's output rather than the (empty, just-started) current one.
- **Solution**:
  ```bash
  k -n troubleshoot logs crasher --previous | tee {WS}/q19/crasher.log
  # "FATAL: required environment variable DB_HOST is not set"

  k -n troubleshoot get pod crasher -o yaml > crasher.yaml
  # add under spec.containers[0]:
  #   env: [{name: DB_HOST, value: db.internal}]
  # strip status/uid/resourceVersion/creationTimestamp before reapplying

  k -n troubleshoot delete pod crasher --now
  k apply -f crasher.yaml
  ```
- **Gotchas**: a bare Pod's `env` field can't be patched with `kubectl set env` in a way that survives — since the task requires keeping the same name/namespace/image/command, editing the exported YAML and recreating it is more reliable than trying to `kubectl edit` a running pod (many fields are immutable post-creation, so a straight edit often just errors).

---

### Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)
- **Concept**: Pods stuck `Pending` forever with no scheduling errors at all (not even "insufficient resources") is the signature of the **scheduler itself being down** — nothing is deciding placement, so pods just sit unassigned indefinitely. Since kube-scheduler runs as a static pod, a broken manifest (here, a typo'd binary name) means the kubelet can't even start the container, and `kubectl logs`/`describe` on the resulting pod reveals the exec failure.
- **Solution**:
  ```bash
  k -n kube-system get pods                    # kube-scheduler in Error/CrashLoopBackOff
  k -n kube-system logs kube-scheduler-{CP}     # exec: "kube-schedulerr": executable file not found

  {SHELL_CP}
  vi /etc/kubernetes/manifests/kube-scheduler.yaml
  # command: [kube-schedulerr]  ->  command: [kube-scheduler]

  k -n kube-system get pods -l component=kube-scheduler   # back to Running
  ```
- **Gotchas**: editing a static pod manifest doesn't need a `kubectl apply` or restart command — the kubelet watches the manifest directory itself and recreates the pod automatically within seconds of the file changing.

---

### Q37 · Cluster DNS Misconfiguration (Invalid Forwarding)
- **Concept**: CoreDNS's `Corefile` (stored as the `coredns` ConfigMap) controls how it resolves names it doesn't own directly — a `forward` directive pointing at an unreachable or wrong upstream breaks **every** external/non-cluster lookup, and depending on plugin order can break in-cluster lookups too if it short-circuits before the `kubernetes` plugin gets a chance.
- **Solution**:
  ```bash
  k -n kube-system get pods -l k8s-app=kube-dns
  k -n kube-system logs -l k8s-app=kube-dns --tail=20

  k -n kube-system get cm coredns -o yaml
  # forward . 10.255.255.1   (wrong)

  k -n kube-system edit cm coredns
  # forward . 10.255.255.1  ->  forward . /etc/resolv.conf

  k -n kube-system rollout restart deploy coredns
  k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
  ```
- **Gotchas**: `forward . /etc/resolv.conf` tells CoreDNS to use the **node's own** resolver config for anything outside the cluster zone — that's the correct, portable default; pointing `forward` at a hardcoded IP address is exactly the kind of change that breaks the moment that IP stops being reachable, which is the fault being simulated here.

---

### Q38 · Controller-Manager Crash (Invalid Flag)
- **Concept**: When `kube-controller-manager` is down, existing workloads keep running untouched, but nothing **new** gets reconciled — a Deployment you create gets no ReplicaSet at all, since it's the controller-manager's Deployment controller that would normally create one. This distinguishes it from a scheduler failure (Q20), where pods *do* get created but stay unplaced.
- **Solution**:
  ```bash
  k -n kube-system get pods                          # controller-manager crashing
  k -n kube-system logs kube-controller-manager-{CP}  # invalid flag: --leader-elect-lease-duration=1s

  {SHELL_CP}
  vi /etc/kubernetes/manifests/kube-controller-manager.yaml   # remove the bad flag

  k create deploy probe --image=busybox:1.36 -- sleep 60
  k get rs           # a ReplicaSet now appears -> controller-manager is healthy again
  k delete deploy probe
  ```
- **Gotchas**: "no ReplicaSet or pods, and nothing shows in `describe deploy`" is the tell — a scheduling or image problem still produces a ReplicaSet and pod objects (just unhealthy ones); a **missing** ReplicaSet altogether means the Deployment controller itself never ran.

---

### Q39 · Kubelet Configuration Drift (Certificate Path)
- **Concept**: `/var/lib/kubelet/config.yaml`'s `clientCAFile` (or similar cert-path fields) points at a specific file the kubelet needs to start correctly — if that path was "tuned" to point at a file that doesn't exist, the kubelet fails at startup and the node reads as `NotReady`, indistinguishable at the `kubectl get node` level from the process simply being stopped (Q17). `journalctl -u kubelet` is what actually distinguishes a config error from a stopped service.
- **Solution**:
  ```bash
  {SHELL_W1}
  systemctl status kubelet
  journalctl -u kubelet -n 30 --no-pager
  # "open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory"

  grep -n clientCAFile /var/lib/kubelet/config.yaml
  sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml

  systemctl restart kubelet && systemctl enable kubelet
  systemctl status kubelet
  ```
- **Gotchas**: a "NotReady" node always deserves a `journalctl -u kubelet` read before assuming it's simply stopped — a config error like this one causes the service to be in a **failed/restarting** state, not cleanly `inactive`, and blindly `enable --now`-ing without fixing the underlying path just loops the same crash.

---

### Q117 · [HARD] Worker Node: Three Things Wrong
- **Concept**: This stacks three unrelated failure modes onto one node — a cordon (`spec.unschedulable: true`, which stops new scheduling but is unrelated to `Ready` status), a wrong `clientCAFile` path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work."
- **Solution**:
  ```bash
  {SHELL_W2}
  systemctl status kubelet
  sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml
  systemctl enable --now kubelet
  systemctl status kubelet
  exit

  k uncordon {W2}

  k run test-pod --image=busybox:1.36 --restart=Never \
    --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 60
  k get pod test-pod -o wide
  k delete pod test-pod --now
  ```
- **Gotchas**: a node can show `Ready` in `kubectl get nodes` while **still cordoned** (`Ready,SchedulingDisabled`) — fixing the kubelet alone gets you `Ready` but pods still won't schedule there until you separately `uncordon`; this task's grading explicitly checks both conditions independently plus an actual pod landing there.

---

### Q118 · [HARD] Two Control-Plane Components Down
- **Concept**: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay `Pending` (scheduler dead) *and* brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads.
- **Solution**:
  ```bash
  k -n kube-system get pods    # scheduler CrashLoop, controller-manager ImagePullBackOff
  {SHELL_CP}

  vi /etc/kubernetes/manifests/kube-scheduler.yaml
  # kube-schedulerr -> kube-scheduler

  vi /etc/kubernetes/manifests/kube-controller-manager.yaml
  # strip the broken image-tag suffix back to a real released version

  k -n kube-system get pods -w
  k create deploy probe --image=busybox:1.36 -- sleep 60
  k get deploy probe   # Available within a minute -> both fixed
  k delete deploy probe
  ```
- **Gotchas**: don't stop at the first component you fix reporting `Running` — this task's actual grading criterion is a brand-new Deployment reaching `Available`, which requires *both* the scheduler (to place the pod) and the controller-manager (to even create the ReplicaSet) working correctly at the same time.

---

### Q119 · [HARD] Application Down for Four Reasons
- **Concept**: Four independent faults are stacked on one app: a `ResourceQuota` capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's `env` references (`CreateContainerConfigError`), and a Service `selector`/`targetPort` mismatch (Q18's pattern again) — and critically, the fix must happen **in place**: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut.
- **Solution**:
  ```bash
  k -n %{ns} get all
  k -n %{ns} describe deploy %{dep}
  k -n %{ns} describe rs -l app=%{dep}          # "exceeded quota: pods, requested: pods=1"

  k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}'
  k -n %{ns} set image deploy %{dep} web=nginx:1.27

  k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}'
  k -n %{ns} rollout restart deploy %{dep}

  k -n %{ns} patch svc %{dep} -p \
    '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"port":80,"targetPort":80}]}}'

  k -n %{ns} get deploy %{dep}
  k -n %{ns} get endpoints %{dep}
  ```
- **Gotchas**: `patch`/`edit` in place (never `delete`+`apply`) is what preserves `metadata.uid` on the Deployment and `spec.clusterIP` on the Service — those two identifiers are explicitly captured before the task starts and diffed afterward, so even a perfectly-configured *recreated* object fails this check. Fix the quota **first** — with `pods: "1"` in force, no amount of image/env/Service fixing gets you past 1 available replica.

---

### Q120 · [HARD] Cluster Facts, Exactly
- **Concept**: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of `kube-controller-manager`/`kube-apiserver` (not in any single friendly `kubectl get` field), the cluster DNS address is just the `kube-dns` Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an *empty* mode value means the default, `iptables` — not "not configured"), and per-node pod CIDR lives on the Node object itself.
- **Solution**:
  ```bash
  k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr
  k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
  k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
  k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:'   # empty -> write "iptables"
  k get node %{node} -o jsonpath='{.spec.podCIDR}'
  ```
  ```
  pod-cidr=<value>
  service-cidr=<value>
  dns-ip=<value>
  proxy-mode=<value or iptables if empty>
  <node>-pod-cidr=<value>
  ```
- **Gotchas**: kube-proxy's ConfigMap often has `mode: ""` (an explicit empty string) rather than the key being absent entirely — both mean "use the default, iptables" and the file must say `iptables` either way, not literally blank. Write exactly the `key=value` format the task specifies, one per line, no extra whitespace — this is graded with exact string matches, not fuzzy comparison.