# CKA Simulator — Step-by-Step Solutions & Conceptual Guide

This guide explains how to approach and solve every question in the CKA exam bank. It breaks down the **underlying Kubernetes concepts**, **the step-by-step commands**, **why specific flags are used**, and **how to verify your solution** before submitting.

---

## Domain 1: Cluster Architecture, Installation & Configuration (25%)

### Q01 · RBAC for a Deployment Bot
- **Concept**: Role-Based Access Control (RBAC) separates permissions using `Roles` (namespaced) and `ClusterRoles` (cluster-wide), bound to users/ServiceAccounts via `RoleBindings` or `ClusterRoleBindings`.
- **Step-by-Step**:
  1. Create the ServiceAccount in the target namespace:
     ```bash
     k -n rbac-app create sa deploy-bot
     ```
  2. Create the Role with explicit verbs on `deployments.apps`:
     ```bash
     k -n rbac-app create role deploy-manager \
       --verb=get,list,watch,create,update,patch,delete \
       --resource=deployments.apps
     ```
  3. Bind the Role to the ServiceAccount:
     ```bash
     k -n rbac-app create rolebinding deploy-bot-binding \
       --role=deploy-manager \
       --serviceaccount=rbac-app:deploy-bot
     ```
  4. **Verify** using `kubectl auth can-i`:
     ```bash
     k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot # yes
     k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot        # no
     ```

---

### Q02 · etcd Snapshot
- **Concept**: etcd stores all cluster state. Snapshots require TLS certificates (CA cert, server cert, server key) and `ETCDCTL_API=3`.
- **Step-by-Step**:
  1. SSH / access the control plane node (`podman exec -it <control-plane> bash` or `ssh <control-plane>`).
  2. Run `etcdctl snapshot save` pointing to the etcd TLS certificates:
     ```bash
     ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
       --cacert=/etc/kubernetes/pki/etcd/ca.crt \
       --cert=/etc/kubernetes/pki/etcd/server.crt \
       --key=/etc/kubernetes/pki/etcd/server.key \
       snapshot save /opt/cka/etcd-snapshot.db
     ```
  3. **Verify** snapshot integrity:
     ```bash
     etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
     ```
- **If the node has no `etcdctl`**: run it inside the etcd pod instead. The etcd image is
  distroless (no `sh`, no `cat`, no `tar`), so `kubectl cp` cannot copy the file out — write the
  snapshot into `/var/lib/etcd`, which is a hostPath, and move it on the node:
  ```bash
  kubectl -n kube-system exec etcd-<cp> -- etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/snap.db
  # on the node: mv /var/lib/etcd/snap.db /opt/cka/etcd-snapshot.db
  ```
  `./etcd-tools.sh` installs both binaries on the control-plane node for good (`setup.sh` runs it).

---

### Q03 · Kustomize Overlay
- **Concept**: Kustomize allows environment-specific customization (like `prod` overlays) without templating YAMLs.
- **Step-by-Step**:
  1. Create the overlay folder: `mkdir -p workspace/q03/overlays/prod && cd workspace/q03/overlays/prod`
  2. Create `namespace.yaml` for `store-prod`.
  3. Write `kustomization.yaml` referencing `../../base`, adding namespace, labels, and replica count.
  4. Apply via `kubectl apply -k .`
  5. **Verify**: `k -n store-prod get deploy,pod -l env=prod`

---

### Q04 · CustomResourceDefinition (CRD) & Custom Resource
- **Concept**: CRDs extend Kubernetes APIs with custom objects. Schema validation uses OpenAPI v3.
- **Step-by-Step**:
  1. Create the `CustomResourceDefinition` YAML specifying `group`, `names`, `scope: Namespaced`, and `openAPIV3Schema`.
  2. Apply the CRD: `k apply -f crd.yaml`.
  3. Instantiate the Custom Resource (`Backup` named `nightly`).
  4. **Verify**: `k get bk` or `k get backups`.

---

### Q05 · Static Pod Creation
- **Concept**: Static pods are managed directly by the Kubelet on a specific node without API server scheduling. Kubelet creates a mirror pod in the API server.
- **Step-by-Step**:
  1. SSH / access the target node.
  2. Check static pod path: `grep staticPodPath /var/lib/kubelet/config.yaml` (usually `/etc/kubernetes/manifests`).
  3. Create the pod manifest in that directory: `/etc/kubernetes/manifests/static-web.yaml`.
  4. **Verify**: Kubelet creates the mirror pod within seconds: `k get pods -o wide | grep static-web`.

---

### Q06 · Certificate Expiry & User CSR
- **Concept**: Kubeadm manages control-plane certs. Kubernetes CSR API (`certificates.k8s.io/v1`) signs and approves client certificates.
- **Step-by-Step**:
  1. On the control plane: `kubeadm certs check-expiration | grep '^apiserver ' > apiserver-expiry.txt`.
  2. Generate user private key and CSR with OpenSSL:
     ```bash
     openssl genrsa -out jane.key 2048
     openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
     ```
  3. Submit `CertificateSigningRequest` with `base64 -w0 jane.csr` and signer `kubernetes.io/kube-apiserver-client`.
  4. Approve: `k certificate approve jane`.
  5. Extract certificate: `k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt`.
  6. Create RoleBinding binding user `jane` to ClusterRole `view`.

---

### Q33 · Aggregated ClusterRole
- **Concept**: Aggregated ClusterRoles automatically aggregate permissions from other ClusterRoles that have matching labels defined in `aggregationRule.clusterRoleSelectors`.
- **Step-by-Step**:
  1. Create the parent ClusterRole with `aggregationRule: {clusterRoleSelectors: [{matchLabels: {rbac.example.com/aggregate-to-parent: "true"}}]}` and `rules: []`.
  2. Create sub-ClusterRoles (`pods`, `services`) with required verbs.
  3. Label the sub-ClusterRoles to trigger aggregation.
  4. Bind the parent ClusterRole to the user via `ClusterRoleBinding`.
  5. **Verify**: `k get clusterrole <name> -o yaml` (rules will be automatically populated).

---

### Q34 · Discover CRD Field with `kubectl explain`
- **Concept**: `kubectl explain <resource>.<field> --recursive` inspects the live schema for any built-in or custom resource.
- **Step-by-Step**:
  1. Inspect CRD schema: `k explain <kind>.spec --recursive`.
  2. Check field description: `k explain <kind>.spec.tuning.<field>`.
  3. Write path to file: `echo spec.tuning.<field> > path.txt`.
  4. Create sample custom resource with that field configured.

---

### Q40 · Kubeadm Join Token Creation
- **Concept**: `kubeadm token create` generates authentication bootstrap tokens used by new nodes to join the cluster.
- **Step-by-Step**:
  1. On control plane: `kubeadm token create --ttl 1h --print-join-command`.
  2. Save output to `join.sh`.
  3. Verify with `kubeadm token list`.

---

### Q41 · Component Versions Inspection
- **Concept**: Control-plane components run as static pods; Kubelet runs as a systemd service.
- **Step-by-Step**:
  1. Server version: `k version`.
  2. Node versions: `k get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion} {.status.nodeInfo.containerRuntimeVersion}'`.
  3. etcd version: `k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'`.

---

### Q101 · [HARD] RBAC Cleanup & Granular Privileges
- **Concept**: Principle of least privilege. Remove over-privileged admin bindings before granting surgical access.
- **Step-by-Step**:
  1. Find and delete over-granting binding: `k delete clusterrolebinding <sa>-legacy-admin`.
  2. Create Role for deployments & replicasets: `k -n <ns> create role <sa>-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps`.
  3. Add `pods/exec` create permissions to the Role.
  4. Create ClusterRole with `list` on `nodes` and bind it.
  5. **Verify**: `k auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>`.

---

### Q102 · [HARD] etcd Restore
- **Concept**: Restoring etcd from a snapshot requires stopping etcd, restoring data to a new data directory, and updating the etcd static pod hostPath volume.
- **Step-by-Step**:
  1. On control plane, restore snapshot: `etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore`.
  2. Edit `/etc/kubernetes/manifests/etcd.yaml`:
     - Update the `etcd-data` hostPath volume path from `/var/lib/etcd` to `/var/lib/etcd-restore`.
  3. Wait for Kubelet to restart etcd and the API server to reconnect (`watch crictl ps`).
  4. **Verify**: `k get cm <restored-cm>`.

---

### Q103 · [HARD] Static Pod on Relocated Path
- **Concept**: Some nodes may have custom `staticPodPath` configured in `/var/lib/kubelet/config.yaml` instead of `/etc/kubernetes/manifests`.
- **Step-by-Step**:
  1. Find target node: `k get nodes -l tier=edge`.
  2. On node, inspect `grep staticPodPath /var/lib/kubelet/config.yaml`.
  3. Write pod manifest directly into that specific directory.
  4. **Verify**: `k get pods -n <ns> -o wide`.

---

### Q104 · [HARD] Self-Contained Kubeconfig Creation
- **Concept**: A portable kubeconfig embeds all certificates using `--embed-certs` rather than referencing file paths.
- **Step-by-Step**:
  1. Issue & approve CSR for the user; save `user.crt` and `user.key`.
  2. Extract cluster CA: `k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt`.
  3. Set cluster, credentials, context, and current-context with `--kubeconfig=<file> --embed-certs`.
  4. **Verify**: `kubectl --kubeconfig=<file> get pods`.

---

### Q122 · [HARD] Renew Control Plane Certificates
- **Concept**: `kubeadm certs renew all` generates new certificates for all control-plane components. Static pods must be bounced to reload the new certs.
- **Step-by-Step**:
  1. On control plane: `kubeadm certs renew all`.
  2. Bounce static pods: `mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/`.
  3. Update admin config: `cp /etc/kubernetes/admin.conf ~/.kube/config`.
  4. **Verify**: `kubeadm certs check-expiration`.

---

## Domain 2: Workloads & Scheduling (15%)

### Q07 · Rolling Update Strategy & Scaling
- **Concept**: `maxSurge` controls how many extra pods can be created during updates; `maxUnavailable` controls how many pods can be unavailable.
- **Step-by-Step**:
  1. Update image: `k -n apps set image deploy web-frontend nginx=nginx:1.27`.
  2. Scale replicas: `k -n apps scale deploy web-frontend --replicas=4`.
  3. Patch strategy: `k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'`.
  4. **Verify**: `k -n apps rollout status deploy web-frontend`.

---

### Q08 · Sidecar Container & Shared Volume
- **Concept**: Multi-container pods share volumes (`emptyDir`) and localhost network namespace.
- **Step-by-Step**:
  1. Define Pod with an `emptyDir` volume mounted at `/var/log` in both the app container and sidecar container.
  2. App writes logs to `/var/log/app.log`; sidecar runs `tail -f /var/log/app.log`.
  3. **Verify**: `k -n apps logs log-shipper -c shipper`.

---

### Q09 · Node Affinity & Resource Requests/Limits
- **Concept**: `nodeAffinity` matches node labels (`requiredDuringSchedulingIgnoredDuringExecution`). Resource `requests` are used for scheduling; `limits` enforce runtime boundaries.
- **Step-by-Step**:
  1. Create deployment template: `k -n apps create deploy cache --image=redis:7-alpine --replicas=2 $do > cache.yaml`.
  2. Add `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` with `matchExpressions: [{key: disktype, operator: In, values: [ssd]}]`.
  3. Add `resources: {requests: {cpu: 100m, memory: 64Mi}, limits: {cpu: 200m, memory: 128Mi}}`.
  4. Apply and verify scheduling: `k -n apps get pods -o wide`.

---

### Q10 · ConfigMap & Secret Consumption
- **Concept**: ConfigMaps and Secrets can be mounted as files via volumes, loaded as individual environment variables (`valueFrom`), or loaded entirely (`envFrom`).
- **Step-by-Step**:
  1. Create ConfigMap: `k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug`.
  2. Create Secret: `k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'`.
  3. In Pod spec: mount ConfigMap as volume at `/etc/config`, inject `APP_MODE` from CM, and use `envFrom` for `db-creds`.
  4. **Verify**: `k -n apps exec config-pod -- ls /etc/config` and `k -n apps exec config-pod -- env`.

---

### Q23 · Metrics Server Repair & HPA
- **Concept**: Metrics Server provides CPU/memory metrics for `kubectl top` and Horizontal Pod Autoscalers. In non-production/kind environments, `--kubelet-insecure-tls` is required.
- **Step-by-Step**:
  1. Edit metrics-server deployment: `k -n kube-system edit deploy metrics-server`. Add `--kubelet-insecure-tls` under container args.
  2. Verify: `k top nodes`.
  3. Create HPA: `k -n <ns> autoscale deploy <dep> --min=2 --max=<max> --cpu-percent=<cpu>`.
  4. **Verify**: `k -n <ns> get hpa`.

---

### Q24 · PriorityClass
- **Concept**: PriorityClasses determine pod preemption and scheduling order. Higher value = higher priority.
- **Step-by-Step**:
  1. Create `PriorityClass` with `value: <integer>` and `globalDefault: false`.
  2. Assign `priorityClassName: <name>` in the Deployment pod template spec.
  3. **Verify**: `k -n <ns> rollout status deploy <dep>`.

---

### Q25 · Taints & Tolerations
- **Concept**: Taints repel pods (`NoSchedule`, `PreferNoSchedule`, `NoExecute`). Pods require matching tolerations to be scheduled on tainted nodes.
- **Step-by-Step**:
  1. Taint node: `k taint node <node> key=val:NoSchedule`.
  2. Add toleration to deployment pod spec:
     ```yaml
     tolerations:
       - key: "key"
         operator: "Equal"
         value: "val"
         effect: "NoSchedule"
     ```
  3. Set `nodeSelector: {kubernetes.io/hostname: <node>}`.

---

### Q26 · DaemonSet on All Nodes (including Control Plane)
- **Concept**: DaemonSets ensure all (or some) nodes run a copy of a pod. To run on control-plane nodes, they must tolerate control-plane taints (`node-role.kubernetes.io/control-plane:NoSchedule`).
- **Step-by-Step**:
  1. Write `DaemonSet` YAML with toleration:
     ```yaml
     tolerations:
       - key: node-role.kubernetes.io/control-plane
         operator: Exists
         effect: NoSchedule
     ```
  2. Apply and verify: `k -n <ns> get pods -o wide -l app=<ds>`.

---

### Q27 · CronJob Configuration
- **Concept**: CronJobs create Jobs on a time schedule (`schedule: "*/5 * * * *"`). `concurrencyPolicy: Forbid` ensures jobs do not overlap.
- **Step-by-Step**:
  1. Create CronJob: `k -n <ns> create cj <name> --image=busybox:1.36 --schedule='*/5 * * * *' $do -- date > cj.yaml`.
  2. Add `concurrencyPolicy: Forbid`, `successfulJobsHistoryLimit: 3`, `failedJobsHistoryLimit: 1`, `activeDeadlineSeconds: 30`.
  3. Trigger manual execution: `k -n <ns> create job <name>-manual --from=cronjob/<name>`.

---

### Q28 · Init Containers
- **Concept**: Init containers run to completion sequentially before application containers start.
- **Step-by-Step**:
  1. Define `initContainers` block in Pod spec with shared volume mount (`/work`).
  2. Init container writes content (`echo 'hello' > /work/index.html`).
  3. Main container mounts shared volume at `/usr/share/nginx/html`.
  4. **Verify**: `k -n <ns> exec <pod> -c web -- cat /usr/share/nginx/html/index.html`.

---

### Q29 · Immutable ConfigMap Rotation
- **Concept**: Setting `immutable: true` on ConfigMaps prevents accidental updates and reduces API server load. To update apps, create a new ConfigMap (`v2`) and update Deployment volume references.
- **Step-by-Step**:
  1. Create `cm-v2` with `immutable: true`.
  2. Patch Deployment volume to point to `cm-v2`.
  3. Monitor rolling update: `k -n <ns> rollout status deploy <dep>`.

---

### Q31 · Parallel Job
- **Concept**: Batch Jobs manage batch tasks. `completions` is total tasks to complete; `parallelism` is max concurrent pods.
- **Step-by-Step**:
  1. Define Job with `completions: 4`, `parallelism: 2`, `backoffLimit: 2`, `restartPolicy: Never`.
  2. Apply and wait: `k -n <ns> wait --for=condition=complete job/<name> --timeout=120s`.

---

### Q32 · ResourceQuota & LimitRange
- **Concept**: `ResourceQuota` enforces aggregate consumption limits in a namespace; `LimitRange` enforces default/min/max resource requests and limits on individual containers.
- **Step-by-Step**:
  1. Create quota: `k -n <ns> create quota <name> --hard=pods=4,requests.cpu=1,limits.memory=2Gi`.
  2. Create LimitRange with `default`, `defaultRequest`, `min`, `max`.
  3. **Verify**: `k -n <ns> describe quota` and `k -n <ns> describe limitrange`.

---

### Q107 · [HARD] Anti-Affinity under Strict Quota
- **Concept**: Spread replicas across distinct nodes using `podAntiAffinity` while ensuring total resource requests do not breach namespace quotas.
- **Step-by-Step**:
  1. Inspect quota: `k -n <ns> get quota`.
  2. Configure deployment with `podAntiAffinity` on `topologyKey: kubernetes.io/hostname`.
  3. Size container requests to fit within namespace quota.
  4. **Verify**: Each pod runs on a distinct node (`k get pods -o wide`).

---

### Q108 · [HARD] Rollback Revision and Pause Rollout
- **Concept**: `rollout undo --to-revision=N` rolls back to a specific ReplicaSet revision. `rollout pause` halts further container rollout.
- **Step-by-Step**:
  1. Inspect history: `k -n <ns> rollout history deploy <dep>`.
  2. Undo: `k -n <ns> rollout undo deploy <dep> --to-revision=<rev>`.
  3. Pause: `k -n <ns> rollout pause deploy <dep>`.

---

### Q109 · [HARD] Native Sidecar Container (K8s 1.28+)
- **Concept**: Native sidecar containers are defined in `initContainers` with `restartPolicy: Always`. They start before app containers and keep running throughout the pod's life.
- **Step-by-Step**:
  1. Define initContainer with `restartPolicy: Always` and readiness probe.
  2. Define main container with liveness probe.
  3. **Verify**: `k -n <ns> get pod <pod>` shows all containers running.

---

## Domain 3: Services & Networking (20%)

### Q11 · NetworkPolicy (Ingress Restriction)
- **Concept**: By default, pods accept traffic from any source. A `NetworkPolicy` selects target pods with `podSelector` and restricts traffic using ingress/egress rules.
- **Step-by-Step**:
  1. Create NetworkPolicy in namespace:
     ```yaml
     apiVersion: networking.k8s.io/v1
     kind: NetworkPolicy
     metadata: {name: allow-frontend, namespace: marketing}
     spec:
       podSelector: {matchLabels: {app: api}}
       policyTypes: [Ingress]
       ingress:
         - from:
             - podSelector: {matchLabels: {role: frontend}}
           ports:
             - protocol: TCP
               port: 80
     ```
  2. **Verify**: `k -n marketing describe netpol allow-frontend`.

---

### Q12 · NodePort Service
- **Concept**: `NodePort` exposes the Service on each Node's IP at a static port (default range: 30000–32767).
- **Step-by-Step**:
  1. Expose deployment: `k -n web expose deploy web-server --type=NodePort --port=80 --target-port=80 $do > svc.yaml`.
  2. Edit `spec.ports[0].nodePort: 30080`.
  3. Apply and verify: `k -n web get svc web-server` and `k -n web get endpoints web-server`.

---

### Q13 · Ingress Routing
- **Concept**: Ingress manages external HTTP/HTTPS routing to cluster Services based on hostname and URI path prefixes.
- **Step-by-Step**:
  1. Create Ingress imperatively:
     ```bash
     k -n web create ingress web-ingress \
       --class=nginx \
       --rule="order.example.com/api*=order-service:8080"
     ```
  2. **Verify**: `k -n web describe ingress web-ingress`.

---

### Q14 · Cluster DNS Querying
- **Concept**: Services get DNS names in the format `<service>.<ns>.svc.cluster.local`. Pods with IP `a.b.c.d` get `<a-b-c-d>.<ns>.pod.cluster.local`.
- **Step-by-Step**:
  1. Run transient debug pod: `k run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup web-service.apps.svc.cluster.local`.
  2. Query pod IP record: `nslookup 10-244-1-5.apps.pod.cluster.local`.

---

### Q22 · Gateway API Configuration
- **Concept**: Gateway API is the modern successor to Ingress. `GatewayClass` defines the controller, `Gateway` defines listeners/ports, and `HTTPRoute` defines routing rules.
- **Step-by-Step**:
  1. Create `GatewayClass` (`controllerName: example.com/gw`).
  2. Create `Gateway` with HTTP listener on port 80.
  3. Create `HTTPRoute` attaching to the Gateway with path prefix matching `/api` forwarding to target service.
  4. **Verify**: `k -n <ns> get gateway,httproute`.

---

### Q30 · Named Ports in Service
- **Concept**: Services can map `targetPort` to the `name` of a container port rather than a numeric port.
- **Step-by-Step**:
  1. In Pod/Deployment container: `ports: [{name: http-web, containerPort: 80}]`.
  2. In Service: `ports: [{name: http, port: 8080, targetPort: http-web}]`.
  3. **Verify**: `k -n <ns> get endpoints <svc>`.

---

### Q42 · Egress-Only NetworkPolicy
- **Concept**: Egress NetworkPolicies restrict outbound traffic from pods to specific IP CIDR blocks or namespaces.
- **Step-by-Step**:
  1. Define policy with `policyTypes: [Egress]`.
  2. Add `egress: [{to: [{ipBlock: {cidr: 192.168.1.0/24}}], ports: [{port: 443}]}]`.
  3. **Verify**: `k -n <ns> describe netpol <name>`.

---

### Q43 · Headless & ExternalName Services
- **Concept**: `clusterIP: None` (Headless) returns direct Pod IPs on DNS queries. `type: ExternalName` returns a CNAME pointing to an external domain.
- **Step-by-Step**:
  1. Create Headless Service: `spec.clusterIP: None`, `spec.selector: {app: db}`.
  2. Create ExternalName Service: `spec.type: ExternalName`, `spec.externalName: api.partner.com`.
  3. **Verify**: `k -n <ns> get svc`.

---

### Q111 · [HARD] Zero-Trust Namespace Network Policies
- **Concept**: Zero-trust architecture starts with default-deny on both ingress and egress, then explicitly allows DNS to kube-system (port 53 UDP/TCP) and tiered service-to-service communication.
- **Step-by-Step**:
  1. Create `default-deny` for both `[Ingress, Egress]`.
  2. Create `allow-dns` egress policy to `kube-system` on port 53.
  3. Create `api-ingress` from external labeled namespace.
  4. Create `api-to-db` egress and `db-ingress` policies on port 5432.

---

### Q112 · [HARD] TLS Ingress with Certificate & Fingerprint
- **Concept**: Secure Ingress with TLS Secrets containing `tls.crt` and `tls.key`.
- **Step-by-Step**:
  1. Generate TLS key and self-signed certificate:
     ```bash
     openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=<host>" -keyout tls.key -out tls.crt
     ```
  2. Create TLS secret: `k -n <ns> create secret tls <name>-tls --cert=tls.crt --key=tls.key`.
  3. Create Ingress with `tls: [{hosts: [<host>], secretName: <name>-tls}]` and annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`.
  4. Extract fingerprint: `openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > fp.txt`.

---

### Q114 · [HARD] Multi-Fault CoreDNS Repair
- **Concept**: CoreDNS failures can be caused by blocking NetworkPolicies in `kube-system`, deployment scaled to 0, or bad upstream forwarding in `Corefile`.
- **Step-by-Step**:
  1. Check and delete rogue NetworkPolicy in `kube-system`: `k -n kube-system delete netpol maintenance-lockdown`.
  2. Scale CoreDNS back to 2 replicas: `k -n kube-system scale deploy coredns --replicas=2`.
  3. Fix `Corefile` in ConfigMap: `k -n kube-system edit cm coredns` (change `forward . <bad-ip>` to `forward . /etc/resolv.conf`).
  4. Restart: `k -n kube-system rollout restart deploy coredns`.

---

## Domain 4: Storage (10%)

### Q15 · PersistentVolume, PVC & Pod Mounting
- **Concept**: `PersistentVolume` (PV) represents physical cluster storage; `PersistentVolumeClaim` (PVC) is a request for storage by a user.
- **Step-by-Step**:
  1. Create PV: `capacity: 1Gi`, `accessModes: [ReadWriteOnce]`, `persistentVolumeReclaimPolicy: Retain`, `storageClassName: manual`, `hostPath: {path: /mnt/data}`.
  2. Create PVC: `storageClassName: manual`, `requests.storage: 500Mi`.
  3. Create Pod referencing PVC under `spec.volumes` and mounting it in the container.
  4. **Verify**: `k get pv` and `k -n storage get pvc` show `Bound`.

---

### Q16 · Default StorageClass & Dynamic Provisioning
- **Concept**: When a PVC omits `storageClassName`, Kubernetes provisions storage using the StorageClass annotated with `storageclass.kubernetes.io/is-default-class: "true"`.
- **Step-by-Step**:
  1. Create new StorageClass with annotation `storageclass.kubernetes.io/is-default-class: "true"`.
  2. Remove default annotation from old StorageClass: `k annotate sc <old-sc> storageclass.kubernetes.io/is-default-class-`.
  3. Create PVC without `storageClassName` and attach to Pod.
  4. **Verify**: Storage is dynamically created and bound upon pod scheduling (`WaitForFirstConsumer`).

---

### Q35 · Re-bind Released PersistentVolume
- **Concept**: When a PVC is deleted, a PV with `Retain` policy enters `Released` status because `spec.claimRef` still points to the old PVC UID. Removing `spec.claimRef` returns it to `Available`.
- **Step-by-Step**:
  1. Remove old claimRef: `k patch pv <pv> --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'`.
  2. Verify PV is now `Available`: `k get pv <pv>`.
  3. Create new PVC specifying `volumeName: <pv>`.
  4. **Verify**: Status transitions back to `Bound`.

---

### Q36 · Data Persistence Across Pod Restarts
- **Concept**: Data written to a PersistentVolume survives pod termination and rescheduling.
- **Step-by-Step**:
  1. Create PVC.
  2. Run writer Pod (`restartPolicy: Never`) writing data to `/data/marker.txt`.
  3. Wait for writer Pod completion: `k wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer`.
  4. Run reader Pod mounting the same PVC.
  5. Verify data: `k exec reader -- cat /data/marker.txt`.

---

### Q115 · [HARD] Local PV with Node Affinity Matching
- **Concept**: Local PersistentVolumes use `nodeAffinity` to bind to specific worker nodes. Pods mounting the claim are automatically scheduled onto the matching node.
- **Step-by-Step**:
  1. Inspect PV labels and node affinity: `k get pv -l media --show-labels`.
  2. Create PVC matching `selector: {matchLabels: {media: nvme}}`.
  3. Create Pod mounting the PVC.
  4. **Verify**: Pod schedules onto the node containing the NVMe disk without explicit `nodeSelector`.

---

### Q116 · [HARD] Data Migration Between PVCs via Batch Job
- **Concept**: Migrating data between volumes requires running a migration Job mounting both the old claim (`src`) and new claim (`dst`).
- **Step-by-Step**:
  1. Create `new-data` PVC.
  2. Run batch Job mounting `old-data` to `/src` and `new-data` to `/dst` running `cp -a /src/. /dst/`.
  3. Wait for Job completion: `k wait --for=condition=complete job/migrate`.
  4. Update application Pod to reference `new-data` claim.
  5. Delete `old-data` claim (PV remains retained).

---

## Domain 5: Troubleshooting (30%)

### Q17 · Node `NotReady` (Kubelet Inactive)
- **Concept**: A node reports `NotReady` when the local `kubelet` service is stopped, disabled, or failing to communicate with the API server.
- **Step-by-Step**:
  1. Identify broken node: `k get nodes`.
  2. SSH / access the node: `podman exec -it <node> bash` or `ssh <node>`.
  3. Check status: `systemctl status kubelet`.
  4. Enable & start service: `systemctl enable --now kubelet`.
  5. Check logs: `journalctl -u kubelet -f --no-pager`.
  6. **Verify**: Node transitions to `Ready`.

---

### Q18 · Deployment & Service Misconfiguration
- **Concept**: Pods in `ImagePullBackOff` have invalid image names/tags; Services without endpoints have mismatched `spec.selector` or incorrect `targetPort`.
- **Step-by-Step**:
  1. Fix Deployment image: `k -n troubleshoot set image deploy payments payments=nginx:1.27`.
  2. Inspect Service: `k -n troubleshoot get svc payments -o yaml`.
  3. Fix typo in selector (`app: payments`) and match `targetPort: 80` to container port.
  4. **Verify**: `k -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments`.

---

### Q19 · `CrashLoopBackOff` Diagnosis
- **Concept**: `CrashLoopBackOff` means the application process started and crashed. Check previous logs with `--previous`.
- **Step-by-Step**:
  1. Read crash logs: `k -n troubleshoot logs crasher --previous | tee crasher.log`.
  2. Locate missing configuration: `FATAL: required environment variable DB_HOST is not set`.
  3. Export pod YAML, add `env: [{name: DB_HOST, value: db.internal}]`, strip metadata, delete and recreate pod.
  4. **Verify**: Pod reaches `Running` status and stays healthy.

---

### Q20 · Control Plane Component Failure (Kube-Scheduler Static Pod)
- **Concept**: Static pods for `kube-apiserver`, `kube-scheduler`, `kube-controller-manager`, and `etcd` live in `/etc/kubernetes/manifests/`. Kubelet watches and restarts them automatically.
- **Step-by-Step**:
  1. Check failing control-plane pod: `k -n kube-system get pods`.
  2. Check logs: `k -n kube-system logs kube-scheduler-<cp>`.
  3. SSH into control plane.
  4. Edit `/etc/kubernetes/manifests/kube-scheduler.yaml`: fix binary command typo (`kube-schedulerr` -> `kube-scheduler`).
  5. **Verify**: Kubelet reloads the manifest and pods schedule normally.

---

### Q37 · CoreDNS Misconfiguration (Invalid Forwarding)
- **Concept**: CoreDNS forwards external queries based on the `Corefile` in ConfigMap `coredns` in `kube-system`.
- **Step-by-Step**:
  1. Inspect logs: `k -n kube-system logs -l k8s-app=kube-dns`.
  2. Edit ConfigMap: `k -n kube-system edit cm coredns`.
  3. Replace invalid forward IP (`10.255.255.1`) with `/etc/resolv.conf`.
  4. Restart deployment: `k -n kube-system rollout restart deploy coredns`.
  5. **Verify**: `k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default`.

---

### Q38 · Controller-Manager Crash (Invalid Flag)
- **Concept**: Invalid CLI flags in static pod manifests prevent components from starting.
- **Step-by-Step**:
  1. Check logs: `k -n kube-system logs kube-controller-manager-<cp>`.
  2. Error: `invalid flag: --leader-elect-lease-duration=1s`.
  3. On control plane, edit `/etc/kubernetes/manifests/kube-controller-manager.yaml` and remove the bad flag.
  4. **Verify**: Controller manager restarts and reconciles deployments and ReplicaSets.

---

### Q39 · Kubelet Configuration Drift (Certificate Path)
- **Concept**: Kubelet configuration lives at `/var/lib/kubelet/config.yaml`. Broken certificate paths prevent Kubelet from starting.
- **Step-by-Step**:
  1. On broken worker, inspect `journalctl -u kubelet -n 30 --no-pager`.
  2. Error: `open /etc/kubernetes/pki/ca-bundle.crt: no such file or directory`.
  3. In `/var/lib/kubelet/config.yaml`, fix `clientCAFile: /etc/kubernetes/pki/ca.crt`.
  4. Restart kubelet: `systemctl restart kubelet && systemctl enable kubelet`.
  5. **Verify**: `systemctl status kubelet` is active.

---

### Q117 · [HARD] Multi-Fault Worker Node (Dead Kubelet + Wrong CA + Cordoned)
- **Concept**: Killer.sh-style compound failures combine multiple root causes on a single node.
- **Step-by-Step**:
  1. Check node in cluster: `k get node <node>` (shows `NotReady,SchedulingDisabled`).
  2. Access node, check `systemctl status kubelet`.
  3. Fix wrong CA file in `/var/lib/kubelet/config.yaml` (`ca-wrong.crt` -> `ca.crt`).
  4. Start kubelet: `systemctl enable --now kubelet`.
  5. Exit node back to main terminal.
  6. **Uncordon** the node: `k uncordon <node>`.
  7. **Verify**: Test scheduling on the node (`k run test-pod --overrides='{"spec":{"nodeName":"<node>"}}'`).

---

### Q118 · [HARD] Two Control Plane Components Down Simultaneously
- **Concept**: Both `kube-scheduler` and `kube-controller-manager` fail for different reasons (command typo + broken image tag).
- **Step-by-Step**:
  1. On control plane, fix `kube-scheduler.yaml` command typo (`kube-schedulerr` -> `kube-scheduler`).
  2. Fix `kube-controller-manager.yaml` image tag (remove broken suffix).
  3. **Verify**: `k -n kube-system get pods -w` shows all control-plane pods Running.

---

### Q119 · [HARD] Application Failure (4 Compound Reasons)
- **Concept**: Quota exhaustion + image tag typo + missing Secret key + broken Service selector.
- **Step-by-Step**:
  1. Patch namespace quota hard limit from 1 to 3 pods.
  2. Fix deployment image tag to `nginx:1.27`.
  3. Add missing key to Secret `DATABASE_URL` and restart rollout.
  4. Fix Service selector and targetPort.
  5. **Verify**: All 3 deployment replicas are Available with active endpoints.

---

### Q120 · [HARD] Extract Cluster Networking Facts
- **Concept**: Extract network configuration directly from control-plane manifests and ConfigMaps.
- **Step-by-Step**:
  1. Pod CIDR: `k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr`.
  2. Service CIDR: `k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range`.
  3. CoreDNS ClusterIP: `k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'`.
  4. Kube-proxy mode: `k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:'` (default: `iptables`).
  5. Node PodCIDR: `k get node <node> -o jsonpath='{.spec.podCIDR}'`.