vumast3r/cka-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
71f57de121
cka-sim/CKA_SOLUTIONS_EXPLAINED.md

32 KiB
Raw Blame History

CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. It breaks down the underlying Kubernetes concepts, the step-by-step commands, why specific flags are used, and how to verify your solution before submitting.

Domain 1: Cluster Architecture, Installation & Configuration (25%)

Q01 · RBAC for a Deployment Bot

  • Concept: Role-Based Access Control (RBAC) separates permissions using Roles (namespaced) and ClusterRoles (cluster-wide), bound to users/ServiceAccounts via RoleBindings or ClusterRoleBindings.
  • Step-by-Step:
    1. Create the ServiceAccount in the target namespace: 
      k -n rbac-app create sa deploy-bot
    2. Create the Role with explicit verbs on deployments.apps: 
      k -n rbac-app create role deploy-manager \
  --verb=get,list,watch,create,update,patch,delete \
  --resource=deployments.apps
    3. Bind the Role to the ServiceAccount: 
      k -n rbac-app create rolebinding deploy-bot-binding \
  --role=deploy-manager \
  --serviceaccount=rbac-app:deploy-bot
    4. Verify using kubectl auth can-i: 
      k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot # yes
k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot        # no

Q02 · etcd Snapshot

  • Concept: etcd stores all cluster state. Snapshots require TLS certificates (CA cert, server cert, server key) and ETCDCTL_API=3.
  • Step-by-Step:
    1. SSH / access the control plane node (podman exec -it <control-plane> bash or ssh <control-plane>).
    2. Run etcdctl snapshot save pointing to the etcd TLS certificates: 
      ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cka/etcd-snapshot.db
    3. Verify snapshot integrity: 
      etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table

Q03 · Kustomize Overlay

  • Concept: Kustomize allows environment-specific customization (like prod overlays) without templating YAMLs.
  • Step-by-Step:
    1. Create the overlay folder: mkdir -p workspace/q03/overlays/prod && cd workspace/q03/overlays/prod
    2. Create namespace.yaml for store-prod.
    3. Write kustomization.yaml referencing ../../base, adding namespace, labels, and replica count.
    4. Apply via kubectl apply -k .
    5. Verify: k -n store-prod get deploy,pod -l env=prod

Q04 · CustomResourceDefinition (CRD) & Custom Resource

  • Concept: CRDs extend Kubernetes APIs with custom objects. Schema validation uses OpenAPI v3.
  • Step-by-Step:
    1. Create the CustomResourceDefinition YAML specifying group, names, scope: Namespaced, and openAPIV3Schema.
    2. Apply the CRD: k apply -f crd.yaml.
    3. Instantiate the Custom Resource (Backup named nightly).
    4. Verify: k get bk or k get backups.

Q05 · Static Pod Creation

  • Concept: Static pods are managed directly by the Kubelet on a specific node without API server scheduling. Kubelet creates a mirror pod in the API server.
  • Step-by-Step:
    1. SSH / access the target node.
    2. Check static pod path: grep staticPodPath /var/lib/kubelet/config.yaml (usually /etc/kubernetes/manifests).
    3. Create the pod manifest in that directory: /etc/kubernetes/manifests/static-web.yaml.
    4. Verify: Kubelet creates the mirror pod within seconds: k get pods -o wide | grep static-web.

Q06 · Certificate Expiry & User CSR

  • Concept: Kubeadm manages control-plane certs. Kubernetes CSR API (certificates.k8s.io/v1) signs and approves client certificates.
  • Step-by-Step:
    1. On the control plane: kubeadm certs check-expiration | grep '^apiserver ' > apiserver-expiry.txt.
    2. Generate user private key and CSR with OpenSSL: 
      openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
    3. Submit CertificateSigningRequest with base64 -w0 jane.csr and signer kubernetes.io/kube-apiserver-client.
    4. Approve: k certificate approve jane.
    5. Extract certificate: k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt.
    6. Create RoleBinding binding user jane to ClusterRole view.

Q33 · Aggregated ClusterRole

  • Concept: Aggregated ClusterRoles automatically aggregate permissions from other ClusterRoles that have matching labels defined in aggregationRule.clusterRoleSelectors.
  • Step-by-Step:
    1. Create the parent ClusterRole with aggregationRule: {clusterRoleSelectors: [{matchLabels: {rbac.example.com/aggregate-to-parent: "true"}}]} and rules: [].
    2. Create sub-ClusterRoles (pods, services) with required verbs.
    3. Label the sub-ClusterRoles to trigger aggregation.
    4. Bind the parent ClusterRole to the user via ClusterRoleBinding.
    5. Verify: k get clusterrole <name> -o yaml (rules will be automatically populated).

Q34 · Discover CRD Field with kubectl explain

  • Concept: kubectl explain <resource>.<field> --recursive inspects the live schema for any built-in or custom resource.
  • Step-by-Step:
    1. Inspect CRD schema: k explain <kind>.spec --recursive.
    2. Check field description: k explain <kind>.spec.tuning.<field>.
    3. Write path to file: echo spec.tuning.<field> > path.txt.
    4. Create sample custom resource with that field configured.

Q40 · Kubeadm Join Token Creation

  • Concept: kubeadm token create generates authentication bootstrap tokens used by new nodes to join the cluster.
  • Step-by-Step:
    1. On control plane: kubeadm token create --ttl 1h --print-join-command.
    2. Save output to join.sh.
    3. Verify with kubeadm token list.

Q41 · Component Versions Inspection

  • Concept: Control-plane components run as static pods; Kubelet runs as a systemd service.
  • Step-by-Step:
    1. Server version: k version.
    2. Node versions: k get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion} {.status.nodeInfo.containerRuntimeVersion}'.
    3. etcd version: k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'.

Q101 · [HARD] RBAC Cleanup & Granular Privileges

  • Concept: Principle of least privilege. Remove over-privileged admin bindings before granting surgical access.
  • Step-by-Step:
    1. Find and delete over-granting binding: k delete clusterrolebinding <sa>-legacy-admin.
    2. Create Role for deployments & replicasets: k -n <ns> create role <sa>-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps.
    3. Add pods/exec create permissions to the Role.
    4. Create ClusterRole with list on nodes and bind it.
    5. Verify: k auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>.

Q102 · [HARD] etcd Restore

  • Concept: Restoring etcd from a snapshot requires stopping etcd, restoring data to a new data directory, and updating the etcd static pod hostPath volume.
  • Step-by-Step:
    1. On control plane, restore snapshot: etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore.
    2. Edit /etc/kubernetes/manifests/etcd.yaml:
      • Update the etcd-data hostPath volume path from /var/lib/etcd to /var/lib/etcd-restore.
    3. Wait for Kubelet to restart etcd and the API server to reconnect (watch crictl ps).
    4. Verify: k get cm <restored-cm>.

Q103 · [HARD] Static Pod on Relocated Path

  • Concept: Some nodes may have custom staticPodPath configured in /var/lib/kubelet/config.yaml instead of /etc/kubernetes/manifests.
  • Step-by-Step:
    1. Find target node: k get nodes -l tier=edge.
    2. On node, inspect grep staticPodPath /var/lib/kubelet/config.yaml.
    3. Write pod manifest directly into that specific directory.
    4. Verify: k get pods -n <ns> -o wide.

Q104 · [HARD] Self-Contained Kubeconfig Creation

  • Concept: A portable kubeconfig embeds all certificates using --embed-certs rather than referencing file paths.
  • Step-by-Step:
    1. Issue & approve CSR for the user; save user.crt and user.key.
    2. Extract cluster CA: k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt.
    3. Set cluster, credentials, context, and current-context with --kubeconfig=<file> --embed-certs.
    4. Verify: kubectl --kubeconfig=<file> get pods.

Q122 · [HARD] Renew Control Plane Certificates

  • Concept: kubeadm certs renew all generates new certificates for all control-plane components. Static pods must be bounced to reload the new certs.
  • Step-by-Step:
    1. On control plane: kubeadm certs renew all.
    2. Bounce static pods: mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/.
    3. Update admin config: cp /etc/kubernetes/admin.conf ~/.kube/config.
    4. Verify: kubeadm certs check-expiration.

Domain 2: Workloads & Scheduling (15%)

Q07 · Rolling Update Strategy & Scaling

  • Concept: maxSurge controls how many extra pods can be created during updates; maxUnavailable controls how many pods can be unavailable.
  • Step-by-Step:
    1. Update image: k -n apps set image deploy web-frontend nginx=nginx:1.27.
    2. Scale replicas: k -n apps scale deploy web-frontend --replicas=4.
    3. Patch strategy: k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'.
    4. Verify: k -n apps rollout status deploy web-frontend.

Q08 · Sidecar Container & Shared Volume

  • Concept: Multi-container pods share volumes (emptyDir) and localhost network namespace.
  • Step-by-Step:
    1. Define Pod with an emptyDir volume mounted at /var/log in both the app container and sidecar container.
    2. App writes logs to /var/log/app.log; sidecar runs tail -f /var/log/app.log.
    3. Verify: k -n apps logs log-shipper -c shipper.

Q09 · Node Affinity & Resource Requests/Limits

  • Concept: nodeAffinity matches node labels (requiredDuringSchedulingIgnoredDuringExecution). Resource requests are used for scheduling; limits enforce runtime boundaries.
  • Step-by-Step:
    1. Create deployment template: k -n apps create deploy cache --image=redis:7-alpine --replicas=2 $do > cache.yaml.
    2. Add affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution with matchExpressions: [{key: disktype, operator: In, values: [ssd]}].
    3. Add resources: {requests: {cpu: 100m, memory: 64Mi}, limits: {cpu: 200m, memory: 128Mi}}.
    4. Apply and verify scheduling: k -n apps get pods -o wide.

Q10 · ConfigMap & Secret Consumption

  • Concept: ConfigMaps and Secrets can be mounted as files via volumes, loaded as individual environment variables (valueFrom), or loaded entirely (envFrom).
  • Step-by-Step:
    1. Create ConfigMap: k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug.
    2. Create Secret: k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'.
    3. In Pod spec: mount ConfigMap as volume at /etc/config, inject APP_MODE from CM, and use envFrom for db-creds.
    4. Verify: k -n apps exec config-pod -- ls /etc/config and k -n apps exec config-pod -- env.

Q23 · Metrics Server Repair & HPA

  • Concept: Metrics Server provides CPU/memory metrics for kubectl top and Horizontal Pod Autoscalers. In non-production/kind environments, --kubelet-insecure-tls is required.
  • Step-by-Step:
    1. Edit metrics-server deployment: k -n kube-system edit deploy metrics-server. Add --kubelet-insecure-tls under container args.
    2. Verify: k top nodes.
    3. Create HPA: k -n <ns> autoscale deploy <dep> --min=2 --max=<max> --cpu-percent=<cpu>.
    4. Verify: k -n <ns> get hpa.

Q24 · PriorityClass

  • Concept: PriorityClasses determine pod preemption and scheduling order. Higher value = higher priority.
  • Step-by-Step:
    1. Create PriorityClass with value: <integer> and globalDefault: false.
    2. Assign priorityClassName: <name> in the Deployment pod template spec.
    3. Verify: k -n <ns> rollout status deploy <dep>.

Q25 · Taints & Tolerations

  • Concept: Taints repel pods (NoSchedule, PreferNoSchedule, NoExecute). Pods require matching tolerations to be scheduled on tainted nodes.
  • Step-by-Step:
    1. Taint node: k taint node <node> key=val:NoSchedule.
    2. Add toleration to deployment pod spec: 
      tolerations:
  - key: "key"
    operator: "Equal"
    value: "val"
    effect: "NoSchedule"
    3. Set nodeSelector: {kubernetes.io/hostname: <node>}.

Q26 · DaemonSet on All Nodes (including Control Plane)

  • Concept: DaemonSets ensure all (or some) nodes run a copy of a pod. To run on control-plane nodes, they must tolerate control-plane taints (node-role.kubernetes.io/control-plane:NoSchedule).
  • Step-by-Step:
    1. Write DaemonSet YAML with toleration: 
      tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
    2. Apply and verify: k -n <ns> get pods -o wide -l app=<ds>.

Q27 · CronJob Configuration

  • Concept: CronJobs create Jobs on a time schedule (schedule: "*/5 * * * *"). concurrencyPolicy: Forbid ensures jobs do not overlap.
  • Step-by-Step:
    1. Create CronJob: k -n <ns> create cj <name> --image=busybox:1.36 --schedule='*/5 * * * *' $do -- date > cj.yaml.
    2. Add concurrencyPolicy: Forbid, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1, activeDeadlineSeconds: 30.
    3. Trigger manual execution: k -n <ns> create job <name>-manual --from=cronjob/<name>.

Q28 · Init Containers

  • Concept: Init containers run to completion sequentially before application containers start.
  • Step-by-Step:
    1. Define initContainers block in Pod spec with shared volume mount (/work).
    2. Init container writes content (echo 'hello' > /work/index.html).
    3. Main container mounts shared volume at /usr/share/nginx/html.
    4. Verify: k -n <ns> exec <pod> -c web -- cat /usr/share/nginx/html/index.html.

Q29 · Immutable ConfigMap Rotation

  • Concept: Setting immutable: true on ConfigMaps prevents accidental updates and reduces API server load. To update apps, create a new ConfigMap (v2) and update Deployment volume references.
  • Step-by-Step:
    1. Create cm-v2 with immutable: true.
    2. Patch Deployment volume to point to cm-v2.
    3. Monitor rolling update: k -n <ns> rollout status deploy <dep>.

Q31 · Parallel Job

  • Concept: Batch Jobs manage batch tasks. completions is total tasks to complete; parallelism is max concurrent pods.
  • Step-by-Step:
    1. Define Job with completions: 4, parallelism: 2, backoffLimit: 2, restartPolicy: Never.
    2. Apply and wait: k -n <ns> wait --for=condition=complete job/<name> --timeout=120s.

Q32 · ResourceQuota & LimitRange

  • Concept: ResourceQuota enforces aggregate consumption limits in a namespace; LimitRange enforces default/min/max resource requests and limits on individual containers.
  • Step-by-Step:
    1. Create quota: k -n <ns> create quota <name> --hard=pods=4,requests.cpu=1,limits.memory=2Gi.
    2. Create LimitRange with default, defaultRequest, min, max.
    3. Verify: k -n <ns> describe quota and k -n <ns> describe limitrange.

Q107 · [HARD] Anti-Affinity under Strict Quota

  • Concept: Spread replicas across distinct nodes using podAntiAffinity while ensuring total resource requests do not breach namespace quotas.
  • Step-by-Step:
    1. Inspect quota: k -n <ns> get quota.
    2. Configure deployment with podAntiAffinity on topologyKey: kubernetes.io/hostname.
    3. Size container requests to fit within namespace quota.
    4. Verify: Each pod runs on a distinct node (k get pods -o wide).

Q108 · [HARD] Rollback Revision and Pause Rollout

  • Concept: rollout undo --to-revision=N rolls back to a specific ReplicaSet revision. rollout pause halts further container rollout.
  • Step-by-Step:
    1. Inspect history: k -n <ns> rollout history deploy <dep>.
    2. Undo: k -n <ns> rollout undo deploy <dep> --to-revision=<rev>.
    3. Pause: k -n <ns> rollout pause deploy <dep>.

Q109 · [HARD] Native Sidecar Container (K8s 1.28+)

  • Concept: Native sidecar containers are defined in initContainers with restartPolicy: Always. They start before app containers and keep running throughout the pod's life.
  • Step-by-Step:
    1. Define initContainer with restartPolicy: Always and readiness probe.
    2. Define main container with liveness probe.
    3. Verify: k -n <ns> get pod <pod> shows all containers running.

Domain 3: Services & Networking (20%)

Q11 · NetworkPolicy (Ingress Restriction)

  • Concept: By default, pods accept traffic from any source. A NetworkPolicy selects target pods with podSelector and restricts traffic using ingress/egress rules.
  • Step-by-Step:
    1. Create NetworkPolicy in namespace: 
      apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-frontend, namespace: marketing}
spec:
  podSelector: {matchLabels: {app: api}}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: {matchLabels: {role: frontend}}
      ports:
        - protocol: TCP
          port: 80
    2. Verify: k -n marketing describe netpol allow-frontend.

Q12 · NodePort Service

  • Concept: NodePort exposes the Service on each Node's IP at a static port (default range: 30000–32767).
  • Step-by-Step:
    1. Expose deployment: k -n web expose deploy web-server --type=NodePort --port=80 --target-port=80 $do > svc.yaml.
    2. Edit spec.ports[0].nodePort: 30080.
    3. Apply and verify: k -n web get svc web-server and k -n web get endpoints web-server.

Q13 · Ingress Routing

  • Concept: Ingress manages external HTTP/HTTPS routing to cluster Services based on hostname and URI path prefixes.
  • Step-by-Step:
    1. Create Ingress imperatively: 
      k -n web create ingress web-ingress \
  --class=nginx \
  --rule="order.example.com/api*=order-service:8080"
    2. Verify: k -n web describe ingress web-ingress.

Q14 · Cluster DNS Querying

  • Concept: Services get DNS names in the format <service>.<ns>.svc.cluster.local. Pods with IP a.b.c.d get <a-b-c-d>.<ns>.pod.cluster.local.
  • Step-by-Step:
    1. Run transient debug pod: k run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup web-service.apps.svc.cluster.local.
    2. Query pod IP record: nslookup 10-244-1-5.apps.pod.cluster.local.

Q22 · Gateway API Configuration

  • Concept: Gateway API is the modern successor to Ingress. GatewayClass defines the controller, Gateway defines listeners/ports, and HTTPRoute defines routing rules.
  • Step-by-Step:
    1. Create GatewayClass (controllerName: example.com/gw).
    2. Create Gateway with HTTP listener on port 80.
    3. Create HTTPRoute attaching to the Gateway with path prefix matching /api forwarding to target service.
    4. Verify: k -n <ns> get gateway,httproute.

Q30 · Named Ports in Service

  • Concept: Services can map targetPort to the name of a container port rather than a numeric port.
  • Step-by-Step:
    1. In Pod/Deployment container: ports: [{name: http-web, containerPort: 80}].
    2. In Service: ports: [{name: http, port: 8080, targetPort: http-web}].
    3. Verify: k -n <ns> get endpoints <svc>.

Q42 · Egress-Only NetworkPolicy

  • Concept: Egress NetworkPolicies restrict outbound traffic from pods to specific IP CIDR blocks or namespaces.
  • Step-by-Step:
    1. Define policy with policyTypes: [Egress].
    2. Add egress: [{to: [{ipBlock: {cidr: 192.168.1.0/24}}], ports: [{port: 443}]}].
    3. Verify: k -n <ns> describe netpol <name>.

Q43 · Headless & ExternalName Services

  • Concept: clusterIP: None (Headless) returns direct Pod IPs on DNS queries. type: ExternalName returns a CNAME pointing to an external domain.
  • Step-by-Step:
    1. Create Headless Service: spec.clusterIP: None, spec.selector: {app: db}.
    2. Create ExternalName Service: spec.type: ExternalName, spec.externalName: api.partner.com.
    3. Verify: k -n <ns> get svc.

Q111 · [HARD] Zero-Trust Namespace Network Policies

  • Concept: Zero-trust architecture starts with default-deny on both ingress and egress, then explicitly allows DNS to kube-system (port 53 UDP/TCP) and tiered service-to-service communication.
  • Step-by-Step:
    1. Create default-deny for both [Ingress, Egress].
    2. Create allow-dns egress policy to kube-system on port 53.
    3. Create api-ingress from external labeled namespace.
    4. Create api-to-db egress and db-ingress policies on port 5432.

Q112 · [HARD] TLS Ingress with Certificate & Fingerprint

  • Concept: Secure Ingress with TLS Secrets containing tls.crt and tls.key.
  • Step-by-Step:
    1. Generate TLS key and self-signed certificate: 
      openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=<host>" -keyout tls.key -out tls.crt
    2. Create TLS secret: k -n <ns> create secret tls <name>-tls --cert=tls.crt --key=tls.key.
    3. Create Ingress with tls: [{hosts: [<host>], secretName: <name>-tls}] and annotation nginx.ingress.kubernetes.io/ssl-redirect: "true".
    4. Extract fingerprint: openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > fp.txt.

Q114 · [HARD] Multi-Fault CoreDNS Repair

  • Concept: CoreDNS failures can be caused by blocking NetworkPolicies in kube-system, deployment scaled to 0, or bad upstream forwarding in Corefile.
  • Step-by-Step:
    1. Check and delete rogue NetworkPolicy in kube-system: k -n kube-system delete netpol maintenance-lockdown.
    2. Scale CoreDNS back to 2 replicas: k -n kube-system scale deploy coredns --replicas=2.
    3. Fix Corefile in ConfigMap: k -n kube-system edit cm coredns (change forward . <bad-ip> to forward . /etc/resolv.conf).
    4. Restart: k -n kube-system rollout restart deploy coredns.

Domain 4: Storage (10%)

Q15 · PersistentVolume, PVC & Pod Mounting

  • Concept: PersistentVolume (PV) represents physical cluster storage; PersistentVolumeClaim (PVC) is a request for storage by a user.
  • Step-by-Step:
    1. Create PV: capacity: 1Gi, accessModes: [ReadWriteOnce], persistentVolumeReclaimPolicy: Retain, storageClassName: manual, hostPath: {path: /mnt/data}.
    2. Create PVC: storageClassName: manual, requests.storage: 500Mi.
    3. Create Pod referencing PVC under spec.volumes and mounting it in the container.
    4. Verify: k get pv and k -n storage get pvc show Bound.

Q16 · Default StorageClass & Dynamic Provisioning

  • Concept: When a PVC omits storageClassName, Kubernetes provisions storage using the StorageClass annotated with storageclass.kubernetes.io/is-default-class: "true".
  • Step-by-Step:
    1. Create new StorageClass with annotation storageclass.kubernetes.io/is-default-class: "true".
    2. Remove default annotation from old StorageClass: k annotate sc <old-sc> storageclass.kubernetes.io/is-default-class-.
    3. Create PVC without storageClassName and attach to Pod.
    4. Verify: Storage is dynamically created and bound upon pod scheduling (WaitForFirstConsumer).

Q35 · Re-bind Released PersistentVolume

  • Concept: When a PVC is deleted, a PV with Retain policy enters Released status because spec.claimRef still points to the old PVC UID. Removing spec.claimRef returns it to Available.
  • Step-by-Step:
    1. Remove old claimRef: k patch pv <pv> --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'.
    2. Verify PV is now Available: k get pv <pv>.
    3. Create new PVC specifying volumeName: <pv>.
    4. Verify: Status transitions back to Bound.

Q36 · Data Persistence Across Pod Restarts

  • Concept: Data written to a PersistentVolume survives pod termination and rescheduling.
  • Step-by-Step:
    1. Create PVC.
    2. Run writer Pod (restartPolicy: Never) writing data to /data/marker.txt.
    3. Wait for writer Pod completion: k wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer.
    4. Run reader Pod mounting the same PVC.
    5. Verify data: k exec reader -- cat /data/marker.txt.

Q115 · [HARD] Local PV with Node Affinity Matching

  • Concept: Local PersistentVolumes use nodeAffinity to bind to specific worker nodes. Pods mounting the claim are automatically scheduled onto the matching node.
  • Step-by-Step:
    1. Inspect PV labels and node affinity: k get pv -l media --show-labels.
    2. Create PVC matching selector: {matchLabels: {media: nvme}}.
    3. Create Pod mounting the PVC.
    4. Verify: Pod schedules onto the node containing the NVMe disk without explicit nodeSelector.

Q116 · [HARD] Data Migration Between PVCs via Batch Job

  • Concept: Migrating data between volumes requires running a migration Job mounting both the old claim (src) and new claim (dst).
  • Step-by-Step:
    1. Create new-data PVC.
    2. Run batch Job mounting old-data to /src and new-data to /dst running cp -a /src/. /dst/.
    3. Wait for Job completion: k wait --for=condition=complete job/migrate.
    4. Update application Pod to reference new-data claim.
    5. Delete old-data claim (PV remains retained).

Domain 5: Troubleshooting (30%)

Q17 · Node NotReady (Kubelet Inactive)

  • Concept: A node reports NotReady when the local kubelet service is stopped, disabled, or failing to communicate with the API server.
  • Step-by-Step:
    1. Identify broken node: k get nodes.
    2. SSH / access the node: podman exec -it <node> bash or ssh <node>.
    3. Check status: systemctl status kubelet.
    4. Enable & start service: systemctl enable --now kubelet.
    5. Check logs: journalctl -u kubelet -f --no-pager.
    6. Verify: Node transitions to Ready.

Q18 · Deployment & Service Misconfiguration

  • Concept: Pods in ImagePullBackOff have invalid image names/tags; Services without endpoints have mismatched spec.selector or incorrect targetPort.
  • Step-by-Step:
    1. Fix Deployment image: k -n troubleshoot set image deploy payments payments=nginx:1.27.
    2. Inspect Service: k -n troubleshoot get svc payments -o yaml.
    3. Fix typo in selector (app: payments) and match targetPort: 80 to container port.
    4. Verify: k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments.

Q19 · CrashLoopBackOff Diagnosis

  • Concept: CrashLoopBackOff means the application process started and crashed. Check previous logs with --previous.
  • Step-by-Step:
    1. Read crash logs: k -n troubleshoot logs crasher --previous | tee crasher.log.
    2. Locate missing configuration: FATAL: required environment variable DB_HOST is not set.
    3. Export pod YAML, add env: [{name: DB_HOST, value: db.internal}], strip metadata, delete and recreate pod.
    4. Verify: Pod reaches Running status and stays healthy.

Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)

  • Concept: Static pods for kube-apiserver, kube-scheduler, kube-controller-manager, and etcd live in /etc/kubernetes/manifests/. Kubelet watches and restarts them automatically.
  • Step-by-Step:
    1. Check failing control-plane pod: k -n kube-system get pods.
    2. Check logs: k -n kube-system logs kube-scheduler-<cp>.
    3. SSH into control plane.
    4. Edit /etc/kubernetes/manifests/kube-scheduler.yaml: fix binary command typo (kube-schedulerr -> kube-scheduler).
    5. Verify: Kubelet reloads the manifest and pods schedule normally.

Q37 · CoreDNS Misconfiguration (Invalid Forwarding)

  • Concept: CoreDNS forwards external queries based on the Corefile in ConfigMap coredns in kube-system.
  • Step-by-Step:
    1. Inspect logs: k -n kube-system logs -l k8s-app=kube-dns.
    2. Edit ConfigMap: k -n kube-system edit cm coredns.
    3. Replace invalid forward IP (10.255.255.1) with /etc/resolv.conf.
    4. Restart deployment: k -n kube-system rollout restart deploy coredns.
    5. Verify: k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.

Q38 · Controller-Manager Crash (Invalid Flag)

  • Concept: Invalid CLI flags in static pod manifests prevent components from starting.
  • Step-by-Step:
    1. Check logs: k -n kube-system logs kube-controller-manager-<cp>.
    2. Error: invalid flag: --leader-elect-lease-duration=1s.
    3. On control plane, edit /etc/kubernetes/manifests/kube-controller-manager.yaml and remove the bad flag.
    4. Verify: Controller manager restarts and reconciles deployments and ReplicaSets.

Q39 · Kubelet Configuration Drift (Certificate Path)

  • Concept: Kubelet configuration lives at /var/lib/kubelet/config.yaml. Broken certificate paths prevent Kubelet from starting.
  • Step-by-Step:
    1. On broken worker, inspect journalctl -u kubelet -n 30 --no-pager.
    2. Error: open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory.
    3. In /var/lib/kubelet/config.yaml, fix clientCAFile: /etc/kubernetes/pki/ca.crt.
    4. Restart kubelet: systemctl restart kubelet && systemctl enable kubelet.
    5. Verify: systemctl status kubelet is active.

Q117 · [HARD] Multi-Fault Worker Node (Dead Kubelet + Wrong CA + Cordoned)

  • Concept: Killer.sh-style compound failures combine multiple root causes on a single node.
  • Step-by-Step:
    1. Check node in cluster: k get node <node> (shows NotReady,SchedulingDisabled).
    2. Access node, check systemctl status kubelet.
    3. Fix wrong CA file in /var/lib/kubelet/config.yaml (ca-wrong.crt -> ca.crt).
    4. Start kubelet: systemctl enable --now kubelet.
    5. Exit node back to main terminal.
    6. Uncordon the node: k uncordon <node>.
    7. Verify: Test scheduling on the node (k run test-pod --overrides='{"spec":{"nodeName":"<node>"}}').

Q118 · [HARD] Two Control Plane Components Down Simultaneously

  • Concept: Both kube-scheduler and kube-controller-manager fail for different reasons (command typo + broken image tag).
  • Step-by-Step:
    1. On control plane, fix kube-scheduler.yaml command typo (kube-schedulerr -> kube-scheduler).
    2. Fix kube-controller-manager.yaml image tag (remove broken suffix).
    3. Verify: k -n kube-system get pods -w shows all control-plane pods Running.

Q119 · [HARD] Application Failure (4 Compound Reasons)

  • Concept: Quota exhaustion + image tag typo + missing Secret key + broken Service selector.
  • Step-by-Step:
    1. Patch namespace quota hard limit from 1 to 3 pods.
    2. Fix deployment image tag to nginx:1.27.
    3. Add missing key to Secret DATABASE_URL and restart rollout.
    4. Fix Service selector and targetPort.
    5. Verify: All 3 deployment replicas are Available with active endpoints.

Q120 · [HARD] Extract Cluster Networking Facts

  • Concept: Extract network configuration directly from control-plane manifests and ConfigMaps.
  • Step-by-Step:
    1. Pod CIDR: k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr.
    2. Service CIDR: k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range.
    3. CoreDNS ClusterIP: k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'.
    4. Kube-proxy mode: k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:' (default: iptables).
    5. Node PodCIDR: k get node <node> -o jsonpath='{.spec.podCIDR}'.