cka-sim/CKA_SOLUTIONS_EXPLAINED.md

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. It breaks down the underlying Kubernetes concepts, the step-by-step commands, why specific flags are used, and how to verify your solution before submitting.

---

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

• Concept: Role-Based Access Control (RBAC) separates permissions using Roles (namespaced) and ClusterRoles (cluster-wide), bound to users/ServiceAccounts via RoleBindings or ClusterRoleBindings.
• Step-by-Step:
  1. Create the ServiceAccount in the target namespace:

     k -n rbac-app create sa deploy-bot
     

  2. Create the Role with explicit verbs on deployments.apps:

     k -n rbac-app create role deploy-manager \
       --verb=get,list,watch,create,update,patch,delete \
       --resource=deployments.apps
     

  3. Bind the Role to the ServiceAccount:

     k -n rbac-app create rolebinding deploy-bot-binding \
       --role=deploy-manager \
       --serviceaccount=rbac-app:deploy-bot
     

  4. Verify using kubectl auth can-i:

     k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot # yes
     k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot        # no
     

---

Q02 · etcd Snapshot

• Concept: etcd stores all cluster state. Snapshots require TLS certificates (CA cert, server cert, server key) and ETCDCTL_API=3.
• Step-by-Step:
  1. SSH / access the control plane node (podman exec -it <control-plane> bash or ssh <control-plane>).
  2. Run etcdctl snapshot save pointing to the etcd TLS certificates:

     ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
       --cacert=/etc/kubernetes/pki/etcd/ca.crt \
       --cert=/etc/kubernetes/pki/etcd/server.crt \
       --key=/etc/kubernetes/pki/etcd/server.key \
       snapshot save /opt/cka/etcd-snapshot.db
     

  3. Verify snapshot integrity:

     etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
     

• If the node has no etcdctl: run it inside the etcd pod instead. The etcd image is distroless (no sh, no cat, no tar), so kubectl cp cannot copy the file out — write the snapshot into /var/lib/etcd, which is a hostPath, and move it on the node:

  kubectl -n kube-system exec etcd-<cp> -- etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/snap.db
  # on the node: mv /var/lib/etcd/snap.db /opt/cka/etcd-snapshot.db
  

  ./etcd-tools.sh installs both binaries on the control-plane node for good (setup.sh runs it).

---

Q03 · Kustomize Overlay

• Concept: Kustomize allows environment-specific customization (like prod overlays) without templating YAMLs.
• Step-by-Step:
  1. Create the overlay folder: mkdir -p workspace/q03/overlays/prod && cd workspace/q03/overlays/prod
  2. Create namespace.yaml for store-prod.
  3. Write kustomization.yaml referencing ../../base, adding namespace, labels, and replica count.
  4. Apply via kubectl apply -k .
  5. Verify: k -n store-prod get deploy,pod -l env=prod

---

Q04 · CustomResourceDefinition (CRD) & Custom Resource

• Concept: CRDs extend Kubernetes APIs with custom objects. Schema validation uses OpenAPI v3.
• Step-by-Step:
  1. Create the CustomResourceDefinition YAML specifying group, names, scope: Namespaced, and openAPIV3Schema.
  2. Apply the CRD: k apply -f crd.yaml.
  3. Instantiate the Custom Resource (Backup named nightly).
  4. Verify: k get bk or k get backups.

---

Q05 · Static Pod Creation

• Concept: Static pods are managed directly by the Kubelet on a specific node without API server scheduling. Kubelet creates a mirror pod in the API server.
• Step-by-Step:
  1. SSH / access the target node.
  2. Check static pod path: grep staticPodPath /var/lib/kubelet/config.yaml (usually /etc/kubernetes/manifests).
  3. Create the pod manifest in that directory: /etc/kubernetes/manifests/static-web.yaml.
  4. Verify: Kubelet creates the mirror pod within seconds: k get pods -o wide | grep static-web.

---

Q06 · Certificate Expiry & User CSR

• Concept: Kubeadm manages control-plane certs. Kubernetes CSR API (certificates.k8s.io/v1) signs and approves client certificates.
• Step-by-Step:
  1. On the control plane: kubeadm certs check-expiration | grep '^apiserver ' > apiserver-expiry.txt.
  2. Generate user private key and CSR with OpenSSL:

     openssl genrsa -out jane.key 2048
     openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
     

  3. Submit CertificateSigningRequest with base64 -w0 jane.csr and signer kubernetes.io/kube-apiserver-client.
  4. Approve: k certificate approve jane.
  5. Extract certificate: k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt.
  6. Create RoleBinding binding user jane to ClusterRole view.

---

Q21 · Helm Release Install and Upgrade

• Concept: Helm packages manifests as charts; --set overrides values.yaml at install/upgrade time, and every upgrade bumps the release revision (visible via helm history/helm status).
• Step-by-Step:
  1. Install into a new namespace with an override: helm install <rel> {WS}/q21/chart -n <ns> --create-namespace --set replicaCount=<n>.
  2. Verify revision 1: helm status <rel> -n <ns>.
  3. Upgrade the same release, changing the Service type: helm upgrade <rel> {WS}/q21/chart -n <ns> --set replicaCount=<n> --set service.type=NodePort (repeat any values you want kept, or add --reuse-values).
  4. Verify: helm status <rel> -n <ns> shows REVISION: 2 and k -n <ns> get svc shows type: NodePort.

---

Q33 · Aggregated ClusterRole

• Concept: Aggregated ClusterRoles automatically aggregate permissions from other ClusterRoles that have matching labels defined in aggregationRule.clusterRoleSelectors.
• Step-by-Step:
  1. Create the parent ClusterRole with aggregationRule: {clusterRoleSelectors: [{matchLabels: {rbac.example.com/aggregate-to-parent: "true"}}]} and rules: [].
  2. Create sub-ClusterRoles (pods, services) with required verbs.
  3. Label the sub-ClusterRoles to trigger aggregation.
  4. Bind the parent ClusterRole to the user via ClusterRoleBinding.
  5. Verify: k get clusterrole <name> -o yaml (rules will be automatically populated).

---

Q34 · Discover CRD Field with kubectl explain

• Concept: kubectl explain <resource>.<field> --recursive inspects the live schema for any built-in or custom resource.
• Step-by-Step:
  1. Inspect CRD schema: k explain <kind>.spec --recursive.
  2. Check field description: k explain <kind>.spec.tuning.<field>.
  3. Write path to file: echo spec.tuning.<field> > path.txt.
  4. Create sample custom resource with that field configured.

---

Q40 · Kubeadm Join Token Creation

• Concept: kubeadm token create generates authentication bootstrap tokens used by new nodes to join the cluster.
• Step-by-Step:
  1. On control plane: kubeadm token create --ttl 1h --print-join-command.
  2. Save output to join.sh.
  3. Verify with kubeadm token list.

---

Q41 · Component Versions Inspection

• Concept: Control-plane components run as static pods; Kubelet runs as a systemd service.
• Step-by-Step:
  1. Server version: k version.
  2. Node versions: k get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion} {.status.nodeInfo.containerRuntimeVersion}'.
  3. etcd version: k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'.

---

Q101 · [HARD] RBAC Cleanup & Granular Privileges

• Concept: Principle of least privilege. Remove over-privileged admin bindings before granting surgical access.
• Step-by-Step:
  1. Find and delete over-granting binding: k delete clusterrolebinding <sa>-legacy-admin.
  2. Create Role for deployments & replicasets: k -n <ns> create role <sa>-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps.
  3. Add pods/exec create permissions to the Role.
  4. Create ClusterRole with list on nodes and bind it.
  5. Verify: k auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>.

---

Q102 · [HARD] etcd Restore

• Concept: Restoring etcd from a snapshot requires stopping etcd, restoring data to a new data directory, and updating the etcd static pod hostPath volume.
• Step-by-Step:
  1. On control plane, restore snapshot: etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore.
  2. Edit /etc/kubernetes/manifests/etcd.yaml:
     • Update the etcd-data hostPath volume path from /var/lib/etcd to /var/lib/etcd-restore.
  3. Wait for Kubelet to restart etcd and the API server to reconnect (watch crictl ps).
  4. Verify: k get cm <restored-cm>.

---

Q103 · [HARD] Static Pod on Relocated Path

• Concept: Some nodes may have custom staticPodPath configured in /var/lib/kubelet/config.yaml instead of /etc/kubernetes/manifests.
• Step-by-Step:
  1. Find target node: k get nodes -l tier=edge.
  2. On node, inspect grep staticPodPath /var/lib/kubelet/config.yaml.
  3. Write pod manifest directly into that specific directory.
  4. Verify: k get pods -n <ns> -o wide.

---

Q104 · [HARD] Self-Contained Kubeconfig Creation

• Concept: A portable kubeconfig embeds all certificates using --embed-certs rather than referencing file paths.
• Step-by-Step:
  1. Issue & approve CSR for the user; save user.crt and user.key.
  2. Extract cluster CA: k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt.
  3. Set cluster, credentials, context, and current-context with --kubeconfig=<file> --embed-certs.
  4. Verify: kubectl --kubeconfig=<file> get pods.

---

Q105 · [HARD] Kustomize with Generators and Patches

• Concept: Kustomize overlays combine namespace/labels transformers, a strategic-merge patches entry for simple field changes, a JSON 6902 patches entry for structural additions, and a configMapGenerator (which appends a content hash to the name and auto-rewrites every reference to it within the same kustomization).
• Step-by-Step:
  1. overlays/prod/kustomization.yaml:

     resources: [../../base]
     namespace: <ns>
     labels:
       - pairs: {tier: prod}
         includeSelectors: false
     patches:
       - path: replicas.yaml
       - path: envfrom.yaml
         target: {kind: Deployment, name: web}
     configMapGenerator:
       - name: web-config
         literals: [LOG_LEVEL=<lvl>]
     

  2. replicas.yaml (strategic merge patch): a partial Deployment with metadata.name: web and spec.replicas: <rep>.
  3. envfrom.yaml (JSON 6902 patch): - op: add, path: /spec/template/spec/containers/0/envFrom, value: [{configMapRef: {name: web-config}}].
  4. Apply: k apply -k {WS}/q105/overlays/prod.
  5. Verify: k -n <ns> get deploy web -o jsonpath='{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}' returns the hashed name (e.g. web-config-<hash>), and labels.tier=prod is set without touching spec.selector.

---

Q106 · [HARD] CRD with Validation and Defaults

• Concept: An OpenAPI v3 schema on a CRD is enforced by the API server itself — minimum/maximum reject out-of-range numbers before the object is ever persisted, required forces fields to be present, and default fills in a value the caller omitted.
• Step-by-Step:
  1. In spec.versions[0].schema.openAPIV3Schema.properties.spec.properties, define replicas: {type: integer, minimum: 1, maximum: 10}, environment: {type: string, enum: [prod, stage, dev], default: dev}, image: {type: string}, and set required: [replicas, image].
  2. Apply the CRD: k apply -f crd.yaml.
  3. Create the good CR with replicas: 3, image: nginx:1.27, environment: <env>.
  4. Attempt an invalid CR (replicas: 0) and capture the rejection: k apply -f bad.yaml 2> rejected.txt (the API server refuses it because 0 < minimum).
  5. Create the defaulted CR with only replicas and image set — the API server injects environment: dev automatically.
  6. Verify: k -n default get <kind> defaulted -o jsonpath='{.spec.environment}' prints dev.

---

Q122 · [HARD] Renew Control Plane Certificates

• Concept: kubeadm certs renew all generates new certificates for all control-plane components. Static pods must be bounced to reload the new certs.
• Step-by-Step:
  1. On control plane: kubeadm certs renew all.
  2. Bounce static pods: mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/.
  3. Update admin config: cp /etc/kubernetes/admin.conf ~/.kube/config.
  4. Verify: kubeadm certs check-expiration.

---

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

• Concept: maxSurge controls how many extra pods can be created during updates; maxUnavailable controls how many pods can be unavailable.
• Step-by-Step:
  1. Update image: k -n apps set image deploy web-frontend nginx=nginx:1.27.
  2. Scale replicas: k -n apps scale deploy web-frontend --replicas=4.
  3. Patch strategy: k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'.
  4. Verify: k -n apps rollout status deploy web-frontend.

---

Q08 · Sidecar Container & Shared Volume

• Concept: Multi-container pods share volumes (emptyDir) and localhost network namespace.
• Step-by-Step:
  1. Define Pod with an emptyDir volume mounted at /var/log in both the app container and sidecar container.
  2. App writes logs to /var/log/app.log; sidecar runs tail -f /var/log/app.log.
  3. Verify: k -n apps logs log-shipper -c shipper.

---

Q09 · Node Affinity & Resource Requests/Limits

• Concept: nodeAffinity matches node labels (requiredDuringSchedulingIgnoredDuringExecution). Resource requests are used for scheduling; limits enforce runtime boundaries.
• Step-by-Step:
  1. Create deployment template: k -n apps create deploy cache --image=redis:7-alpine --replicas=2 $do > cache.yaml.
  2. Add affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution with matchExpressions: [{key: disktype, operator: In, values: [ssd]}].
  3. Add resources: {requests: {cpu: 100m, memory: 64Mi}, limits: {cpu: 200m, memory: 128Mi}}.
  4. Apply and verify scheduling: k -n apps get pods -o wide.

---

Q10 · ConfigMap & Secret Consumption

• Concept: ConfigMaps and Secrets can be mounted as files via volumes, loaded as individual environment variables (valueFrom), or loaded entirely (envFrom).
• Step-by-Step:
  1. Create ConfigMap: k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug.
  2. Create Secret: k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'.
  3. In Pod spec: mount ConfigMap as volume at /etc/config, inject APP_MODE from CM, and use envFrom for db-creds.
  4. Verify: k -n apps exec config-pod -- ls /etc/config and k -n apps exec config-pod -- env.

---

Q23 · Metrics Server Repair & HPA

• Concept: Metrics Server provides CPU/memory metrics for kubectl top and Horizontal Pod Autoscalers. In non-production/kind environments, --kubelet-insecure-tls is required.
• Step-by-Step:
  1. Edit metrics-server deployment: k -n kube-system edit deploy metrics-server. Add --kubelet-insecure-tls under container args.
  2. Verify: k top nodes.
  3. Create HPA: k -n <ns> autoscale deploy <dep> --min=2 --max=<max> --cpu-percent=<cpu>.
  4. Verify: k -n <ns> get hpa.

---

Q24 · PriorityClass

• Concept: PriorityClasses determine pod preemption and scheduling order. Higher value = higher priority.
• Step-by-Step:
  1. Create PriorityClass with value: <integer> and globalDefault: false.
  2. Assign priorityClassName: <name> in the Deployment pod template spec.
  3. Verify: k -n <ns> rollout status deploy <dep>.

---

Q25 · Taints & Tolerations

• Concept: Taints repel pods (NoSchedule, PreferNoSchedule, NoExecute). Pods require matching tolerations to be scheduled on tainted nodes.
• Step-by-Step:
  1. Taint node: k taint node <node> key=val:NoSchedule.
  2. Add toleration to deployment pod spec:

     tolerations:
       - key: "key"
         operator: "Equal"
         value: "val"
         effect: "NoSchedule"
     

  3. Set nodeSelector: {kubernetes.io/hostname: <node>}.

---

Q26 · DaemonSet on All Nodes (including Control Plane)

• Concept: DaemonSets ensure all (or some) nodes run a copy of a pod. To run on control-plane nodes, they must tolerate control-plane taints (node-role.kubernetes.io/control-plane:NoSchedule).
• Step-by-Step:
  1. Write DaemonSet YAML with toleration:

     tolerations:
       - key: node-role.kubernetes.io/control-plane
         operator: Exists
         effect: NoSchedule
     

  2. Apply and verify: k -n <ns> get pods -o wide -l app=<ds>.

---

Q27 · CronJob Configuration

• Concept: CronJobs create Jobs on a time schedule (schedule: "*/5 * * * *"). concurrencyPolicy: Forbid ensures jobs do not overlap.
• Step-by-Step:
  1. Create CronJob: k -n <ns> create cj <name> --image=busybox:1.36 --schedule='*/5 * * * *' $do -- date > cj.yaml.
  2. Add concurrencyPolicy: Forbid, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1, activeDeadlineSeconds: 30.
  3. Trigger manual execution: k -n <ns> create job <name>-manual --from=cronjob/<name>.

---

Q28 · Init Containers

• Concept: Init containers run to completion sequentially before application containers start.
• Step-by-Step:
  1. Define initContainers block in Pod spec with shared volume mount (/work).
  2. Init container writes content (echo 'hello' > /work/index.html).
  3. Main container mounts shared volume at /usr/share/nginx/html.
  4. Verify: k -n <ns> exec <pod> -c web -- cat /usr/share/nginx/html/index.html.

---

Q29 · Immutable ConfigMap Rotation

• Concept: Setting immutable: true on ConfigMaps prevents accidental updates and reduces API server load. To update apps, create a new ConfigMap (v2) and update Deployment volume references.
• Step-by-Step:
  1. Create cm-v2 with immutable: true.
  2. Patch Deployment volume to point to cm-v2.
  3. Monitor rolling update: k -n <ns> rollout status deploy <dep>.

---

Q31 · Parallel Job

• Concept: Batch Jobs manage batch tasks. completions is total tasks to complete; parallelism is max concurrent pods.
• Step-by-Step:
  1. Define Job with completions: 4, parallelism: 2, backoffLimit: 2, restartPolicy: Never.
  2. Apply and wait: k -n <ns> wait --for=condition=complete job/<name> --timeout=120s.

---

Q32 · ResourceQuota & LimitRange

• Concept: ResourceQuota enforces aggregate consumption limits in a namespace; LimitRange enforces default/min/max resource requests and limits on individual containers.
• Step-by-Step:
  1. Create quota: k -n <ns> create quota <name> --hard=pods=4,requests.cpu=1,limits.memory=2Gi.
  2. Create LimitRange with default, defaultRequest, min, max.
  3. Verify: k -n <ns> describe quota and k -n <ns> describe limitrange.

---

Q107 · [HARD] Anti-Affinity under Strict Quota

• Concept: Spread replicas across distinct nodes using podAntiAffinity while ensuring total resource requests do not breach namespace quotas.
• Step-by-Step:
  1. Inspect quota: k -n <ns> get quota.
  2. Configure deployment with podAntiAffinity on topologyKey: kubernetes.io/hostname.
  3. Size container requests to fit within namespace quota.
  4. Verify: Each pod runs on a distinct node (k get pods -o wide).

---

Q108 · [HARD] Rollback Revision and Pause Rollout

• Concept: rollout undo --to-revision=N rolls back to a specific ReplicaSet revision. rollout pause halts further container rollout.
• Step-by-Step:
  1. Inspect history: k -n <ns> rollout history deploy <dep>.
  2. Undo: k -n <ns> rollout undo deploy <dep> --to-revision=<rev>.
  3. Pause: k -n <ns> rollout pause deploy <dep>.

---

Q109 · [HARD] Native Sidecar Container (K8s 1.28+)

• Concept: Native sidecar containers are defined in initContainers with restartPolicy: Always. They start before app containers and keep running throughout the pod's life.
• Step-by-Step:
  1. Define initContainer with restartPolicy: Always and readiness probe.
  2. Define main container with liveness probe.
  3. Verify: k -n <ns> get pod <pod> shows all containers running.

---

Q110 · [HARD] Cluster Audit to Exact Files

• Concept: "Extract exact facts" tasks reward scripting -o json output through jq/python3 instead of eyeballing kubectl get — the checker diffs your file byte-for-byte against the sorted, header-free expected output.
• Step-by-Step:
  1. Dump everything once: k get pods -A -o json > /tmp/pods.json.
  2. no-readiness.txt: filter pods (excluding kube-system) whose first container has no readinessProbe, print <namespace>/<pod>, sort into the file.
  3. per-node.txt: group Running pods by .spec.nodeName, print <node> <count>, sorted by node name.
  4. images.txt: from kube-system pods only, collect every .spec.containers[*].image, dedupe, sort.
  5. Verify: each file has no header row and is already sort-stable — re-running sort -c on it should exit 0.

---

Q121 · [HARD] The Context's Namespace

• Concept: A kubeconfig context carries its own default namespace (contexts[].context.namespace); commands run without -n land there. kubectl config view --minify shows only the active context, so it's the safe way to read that default without guessing.
• Step-by-Step:
  1. Discover the context's default namespace: k config view --minify -o jsonpath='{..namespace}'.
  2. Create the Deployment and Service without -n/--namespace so they inherit that default (do not assume it's default).
  3. Write the namespace name you discovered in step 1 to {WS}/q121/ns.txt.
  4. Verify: k get deploy,svc <dep> (still no -n) shows them Running/present, and k -n default get deploy <dep> returns not-found — confirming nothing leaked into default.

---

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

• Concept: By default, pods accept traffic from any source. A NetworkPolicy selects target pods with podSelector and restricts traffic using ingress/egress rules.
• Step-by-Step:
  1. Create NetworkPolicy in namespace:

     apiVersion: networking.k8s.io/v1
     kind: NetworkPolicy
     metadata: {name: allow-frontend, namespace: marketing}
     spec:
       podSelector: {matchLabels: {app: api}}
       policyTypes: [Ingress]
       ingress:
         - from:
             - podSelector: {matchLabels: {role: frontend}}
           ports:
             - protocol: TCP
               port: 80
     

  2. Verify: k -n marketing describe netpol allow-frontend.

---

Q12 · NodePort Service

• Concept: NodePort exposes the Service on each Node's IP at a static port (default range: 30000–32767).
• Step-by-Step:
  1. Expose deployment: k -n web expose deploy web-server --type=NodePort --port=80 --target-port=80 $do > svc.yaml.
  2. Edit spec.ports[0].nodePort: 30080.
  3. Apply and verify: k -n web get svc web-server and k -n web get endpoints web-server.

---

Q13 · Ingress Routing

• Concept: Ingress manages external HTTP/HTTPS routing to cluster Services based on hostname and URI path prefixes.
• Step-by-Step:
  1. Create Ingress imperatively:

     k -n web create ingress web-ingress \
       --class=nginx \
       --rule="order.example.com/api*=order-service:8080"
     

  2. Verify: k -n web describe ingress web-ingress.

---

Q14 · Cluster DNS Querying

• Concept: Services get DNS names in the format <service>.<ns>.svc.cluster.local. Pods with IP a.b.c.d get <a-b-c-d>.<ns>.pod.cluster.local.
• Step-by-Step:
  1. Run transient debug pod: k run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup web-service.apps.svc.cluster.local.
  2. Query pod IP record: nslookup 10-244-1-5.apps.pod.cluster.local.

---

Q22 · Gateway API Configuration

• Concept: Gateway API is the modern successor to Ingress. GatewayClass defines the controller, Gateway defines listeners/ports, and HTTPRoute defines routing rules.
• Step-by-Step:
  1. Create GatewayClass (controllerName: example.com/gw).
  2. Create Gateway with HTTP listener on port 80.
  3. Create HTTPRoute attaching to the Gateway with path prefix matching /api forwarding to target service.
  4. Verify: k -n <ns> get gateway,httproute.

---

Q30 · Named Ports in Service

• Concept: Services can map targetPort to the name of a container port rather than a numeric port.
• Step-by-Step:
  1. In Pod/Deployment container: ports: [{name: http-web, containerPort: 80}].
  2. In Service: ports: [{name: http, port: 8080, targetPort: http-web}].
  3. Verify: k -n <ns> get endpoints <svc>.

---

Q42 · Egress-Only NetworkPolicy

• Concept: Egress NetworkPolicies restrict outbound traffic from pods to specific IP CIDR blocks or namespaces.
• Step-by-Step:
  1. Define policy with policyTypes: [Egress].
  2. Add egress: [{to: [{ipBlock: {cidr: 192.168.1.0/24}}], ports: [{port: 443}]}].
  3. Verify: k -n <ns> describe netpol <name>.

---

Q43 · Headless & ExternalName Services

• Concept: clusterIP: None (Headless) returns direct Pod IPs on DNS queries. type: ExternalName returns a CNAME pointing to an external domain.
• Step-by-Step:
  1. Create Headless Service: spec.clusterIP: None, spec.selector: {app: db}.
  2. Create ExternalName Service: spec.type: ExternalName, spec.externalName: api.partner.com.
  3. Verify: k -n <ns> get svc.

---

Q111 · [HARD] Zero-Trust Namespace Network Policies

• Concept: Zero-trust architecture starts with default-deny on both ingress and egress, then explicitly allows DNS to kube-system (port 53 UDP/TCP) and tiered service-to-service communication.
• Step-by-Step:
  1. Create default-deny for both [Ingress, Egress].
  2. Create allow-dns egress policy to kube-system on port 53.
  3. Create api-ingress from external labeled namespace.
  4. Create api-to-db egress and db-ingress policies on port 5432.

---

Q112 · [HARD] TLS Ingress with Certificate & Fingerprint

• Concept: Secure Ingress with TLS Secrets containing tls.crt and tls.key.
• Step-by-Step:
  1. Generate TLS key and self-signed certificate:

     openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=<host>" -keyout tls.key -out tls.crt
     

  2. Create TLS secret: k -n <ns> create secret tls <name>-tls --cert=tls.crt --key=tls.key.
  3. Create Ingress with tls: [{hosts: [<host>], secretName: <name>-tls}] and annotation nginx.ingress.kubernetes.io/ssl-redirect: "true".
  4. Extract fingerprint: openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > fp.txt.

---

Q113 · [HARD] Gateway API Canary with Header Match

• Concept: Gateway API HTTPRoute rules are evaluated in order; a rule with a header matches entry only applies to requests carrying that header, letting you layer a canary override (100% to one backend) on top of a default weighted split. parentRefs[].sectionName pins the route to one specific Gateway listener instead of all of them.
• Step-by-Step:
  1. Create a GatewayClass (controllerName: example.com/gw) and a Gateway with two listeners — web for <host>.example.com and api for api.<host>.example.com, both port 80.
  2. Create HTTPRoute with spec.parentRefs: [{name: <gw>, sectionName: web}] so it only attaches to the web listener.
  3. Rule 1 (checked first): matches: [{headers: [{name: X-Canary, value: always}]}], backendRefs: [{name: v2, weight: 100}].
  4. Rule 2 (default, no header match): backendRefs: [{name: v1, weight: <w2>}, {name: v2, weight: <w>}].
  5. Verify: k -n <ns> get httproute <route> -o yaml shows both rules in order; a request with X-Canary: always always lands on v2, everything else splits by weight.

---

Q114 · [HARD] Multi-Fault CoreDNS Repair

• Concept: CoreDNS failures can be caused by blocking NetworkPolicies in kube-system, deployment scaled to 0, or bad upstream forwarding in Corefile.
• Step-by-Step:
  1. Check and delete rogue NetworkPolicy in kube-system: k -n kube-system delete netpol maintenance-lockdown.
  2. Scale CoreDNS back to 2 replicas: k -n kube-system scale deploy coredns --replicas=2.
  3. Fix Corefile in ConfigMap: k -n kube-system edit cm coredns (change forward . <bad-ip> to forward . /etc/resolv.conf).
  4. Restart: k -n kube-system rollout restart deploy coredns.

---

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

• Concept: PersistentVolume (PV) represents physical cluster storage; PersistentVolumeClaim (PVC) is a request for storage by a user.
• Step-by-Step:
  1. Create PV: capacity: 1Gi, accessModes: [ReadWriteOnce], persistentVolumeReclaimPolicy: Retain, storageClassName: manual, hostPath: {path: /mnt/data}.
  2. Create PVC: storageClassName: manual, requests.storage: 500Mi.
  3. Create Pod referencing PVC under spec.volumes and mounting it in the container.
  4. Verify: k get pv and k -n storage get pvc show Bound.

---

Q16 · Default StorageClass & Dynamic Provisioning

• Concept: When a PVC omits storageClassName, Kubernetes provisions storage using the StorageClass annotated with storageclass.kubernetes.io/is-default-class: "true".
• Step-by-Step:
  1. Create new StorageClass with annotation storageclass.kubernetes.io/is-default-class: "true".
  2. Remove default annotation from old StorageClass: k annotate sc <old-sc> storageclass.kubernetes.io/is-default-class-.
  3. Create PVC without storageClassName and attach to Pod.
  4. Verify: Storage is dynamically created and bound upon pod scheduling (WaitForFirstConsumer).

---

Q35 · Re-bind Released PersistentVolume

• Concept: When a PVC is deleted, a PV with Retain policy enters Released status because spec.claimRef still points to the old PVC UID. Removing spec.claimRef returns it to Available.
• Step-by-Step:
  1. Remove old claimRef: k patch pv <pv> --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'.
  2. Verify PV is now Available: k get pv <pv>.
  3. Create new PVC specifying volumeName: <pv>.
  4. Verify: Status transitions back to Bound.

---

Q36 · Data Persistence Across Pod Restarts

• Concept: Data written to a PersistentVolume survives pod termination and rescheduling.
• Step-by-Step:
  1. Create PVC.
  2. Run writer Pod (restartPolicy: Never) writing data to /data/marker.txt.
  3. Wait for writer Pod completion: k wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer.
  4. Run reader Pod mounting the same PVC.
  5. Verify data: k exec reader -- cat /data/marker.txt.

---

Q115 · [HARD] Local PV with Node Affinity Matching

• Concept: Local PersistentVolumes use nodeAffinity to bind to specific worker nodes. Pods mounting the claim are automatically scheduled onto the matching node.
• Step-by-Step:
  1. Inspect PV labels and node affinity: k get pv -l media --show-labels.
  2. Create PVC matching selector: {matchLabels: {media: nvme}}.
  3. Create Pod mounting the PVC.
  4. Verify: Pod schedules onto the node containing the NVMe disk without explicit nodeSelector.

---

Q116 · [HARD] Data Migration Between PVCs via Batch Job

• Concept: Migrating data between volumes requires running a migration Job mounting both the old claim (src) and new claim (dst).
• Step-by-Step:
  1. Create new-data PVC.
  2. Run batch Job mounting old-data to /src and new-data to /dst running cp -a /src/. /dst/.
  3. Wait for Job completion: k wait --for=condition=complete job/migrate.
  4. Update application Pod to reference new-data claim.
  5. Delete old-data claim (PV remains retained).

---

Domain 5: Troubleshooting (30%)

Q17 · Node NotReady (Kubelet Inactive)

• Concept: A node reports NotReady when the local kubelet service is stopped, disabled, or failing to communicate with the API server.
• Step-by-Step:
  1. Identify broken node: k get nodes.
  2. SSH / access the node: podman exec -it <node> bash or ssh <node>.
  3. Check status: systemctl status kubelet.
  4. Enable & start service: systemctl enable --now kubelet.
  5. Check logs: journalctl -u kubelet -f --no-pager.
  6. Verify: Node transitions to Ready.

---

Q18 · Deployment & Service Misconfiguration

• Concept: Pods in ImagePullBackOff have invalid image names/tags; Services without endpoints have mismatched spec.selector or incorrect targetPort.
• Step-by-Step:
  1. Fix Deployment image: k -n troubleshoot set image deploy payments payments=nginx:1.27.
  2. Inspect Service: k -n troubleshoot get svc payments -o yaml.
  3. Fix typo in selector (app: payments) and match targetPort: 80 to container port.
  4. Verify: k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments.

---

Q19 · CrashLoopBackOff Diagnosis

• Concept: CrashLoopBackOff means the application process started and crashed. Check previous logs with --previous.
• Step-by-Step:
  1. Read crash logs: k -n troubleshoot logs crasher --previous | tee crasher.log.
  2. Locate missing configuration: FATAL: required environment variable DB_HOST is not set.
  3. Export pod YAML, add env: [{name: DB_HOST, value: db.internal}], strip metadata, delete and recreate pod.
  4. Verify: Pod reaches Running status and stays healthy.

---

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

• Concept: Static pods for kube-apiserver, kube-scheduler, kube-controller-manager, and etcd live in /etc/kubernetes/manifests/. Kubelet watches and restarts them automatically.
• Step-by-Step:
  1. Check failing control-plane pod: k -n kube-system get pods.
  2. Check logs: k -n kube-system logs kube-scheduler-<cp>.
  3. SSH into control plane.
  4. Edit /etc/kubernetes/manifests/kube-scheduler.yaml: fix binary command typo (kube-schedulerr -> kube-scheduler).
  5. Verify: Kubelet reloads the manifest and pods schedule normally.

---

Q37 · CoreDNS Misconfiguration (Invalid Forwarding)

• Concept: CoreDNS forwards external queries based on the Corefile in ConfigMap coredns in kube-system.
• Step-by-Step:
  1. Inspect logs: k -n kube-system logs -l k8s-app=kube-dns.
  2. Edit ConfigMap: k -n kube-system edit cm coredns.
  3. Replace invalid forward IP (10.255.255.1) with /etc/resolv.conf.
  4. Restart deployment: k -n kube-system rollout restart deploy coredns.
  5. Verify: k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.

---

Q38 · Controller-Manager Crash (Invalid Flag)

• Concept: Invalid CLI flags in static pod manifests prevent components from starting.
• Step-by-Step:
  1. Check logs: k -n kube-system logs kube-controller-manager-<cp>.
  2. Error: invalid flag: --leader-elect-lease-duration=1s.
  3. On control plane, edit /etc/kubernetes/manifests/kube-controller-manager.yaml and remove the bad flag.
  4. Verify: Controller manager restarts and reconciles deployments and ReplicaSets.

---

Q39 · Kubelet Configuration Drift (Certificate Path)

• Concept: Kubelet configuration lives at /var/lib/kubelet/config.yaml. Broken certificate paths prevent Kubelet from starting.
• Step-by-Step:
  1. On broken worker, inspect journalctl -u kubelet -n 30 --no-pager.
  2. Error: open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory.
  3. In /var/lib/kubelet/config.yaml, fix clientCAFile: /etc/kubernetes/pki/ca.crt.
  4. Restart kubelet: systemctl restart kubelet && systemctl enable kubelet.
  5. Verify: systemctl status kubelet is active.

---

Q117 · [HARD] Multi-Fault Worker Node (Dead Kubelet + Wrong CA + Cordoned)

• Concept: Killer.sh-style compound failures combine multiple root causes on a single node.
• Step-by-Step:
  1. Check node in cluster: k get node <node> (shows NotReady,SchedulingDisabled).
  2. Access node, check systemctl status kubelet.
  3. Fix wrong CA file in /var/lib/kubelet/config.yaml (ca-wrong.crt -> ca.crt).
  4. Start kubelet: systemctl enable --now kubelet.
  5. Exit node back to main terminal.
  6. Uncordon the node: k uncordon <node>.
  7. Verify: Test scheduling on the node (k run test-pod --overrides='{"spec":{"nodeName":"<node>"}}').

---

Q118 · [HARD] Two Control Plane Components Down Simultaneously

• Concept: Both kube-scheduler and kube-controller-manager fail for different reasons (command typo + broken image tag).
• Step-by-Step:
  1. On control plane, fix kube-scheduler.yaml command typo (kube-schedulerr -> kube-scheduler).
  2. Fix kube-controller-manager.yaml image tag (remove broken suffix).
  3. Verify: k -n kube-system get pods -w shows all control-plane pods Running.

---

Q119 · [HARD] Application Failure (4 Compound Reasons)

• Concept: Quota exhaustion + image tag typo + missing Secret key + broken Service selector.
• Step-by-Step:
  1. Patch namespace quota hard limit from 1 to 3 pods.
  2. Fix deployment image tag to nginx:1.27.
  3. Add missing key to Secret DATABASE_URL and restart rollout.
  4. Fix Service selector and targetPort.
  5. Verify: All 3 deployment replicas are Available with active endpoints.

---

Q120 · [HARD] Extract Cluster Networking Facts

• Concept: Extract network configuration directly from control-plane manifests and ConfigMaps.
• Step-by-Step:
  1. Pod CIDR: k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr.
  2. Service CIDR: k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range.
  3. CoreDNS ClusterIP: k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'.
  4. Kube-proxy mode: k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:' (default: iptables).
  5. Node PodCIDR: k get node <node> -o jsonpath='{.spec.podCIDR}'.