# CKA study guide The 27 objectives of the Certified Kubernetes Administrator curriculum, in its own words and order: what to know, the fastest commands, the traps, the documentation pages the exam lets you open, and the tasks in this simulator that practise each one. _Generated by `./exam --track cka notes all --md` from `tracks/cka/curriculum.py` - edit that file, not this one._ ``` ./exam objectives # the path and your progress ./exam study 3.2 # one objective: these notes, then its tasks, no clock, solutions open ./exam drill 3.2 # the same tasks against the clock ``` ## Storage (10%) ### 1.1 Implement storage classes and dynamic volume provisioning `./exam study 1.1` (or `storageclasses`) **Know** - A StorageClass names a provisioner and its settings. A claim that asks for the class gets a PersistentVolume made for it - that is dynamic provisioning. - Fields: provisioner, reclaimPolicy (Delete by default, or Retain), volumeBindingMode (Immediate or WaitForFirstConsumer), allowVolumeExpansion, parameters. - The default class carries the annotation storageclass.kubernetes.io/is-default-class: "true"; a claim with no storageClassName gets it. `storageClassName: ""` asks for no class at all. - A StatefulSet's volumeClaimTemplates make one claim per replica (data-web-0, data-web-1 ...). **Fast** ``` k get sc # (default) marks the default class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: {name: fast-local} provisioner: rancher.io/local-path reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer --- k patch sc standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' k patch sc fast-local -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' k get pvc,pv; k describe pvc data # the Events say why a claim is Pending ``` **Traps** - With WaitForFirstConsumer a claim stays Pending until a Pod uses it. That is not an error. - Leave exactly one class marked default: take the annotation off the old one as well as putting it on the new one. - provisioner, reclaimPolicy and volumeBindingMode cannot be edited - delete the class and create it again. **Docs** - - - **Tasks** - Q16 Default StorageClass and dynamic provisioning - Q36 Data survives the pod - Q59 Migrate data between claims - Q99 Per-replica storage for a StatefulSet _(Simulator Set D (killer.sh-style))_ ### 1.2 Configure volume types, access modes and reclaim policies `./exam study 1.2` (or `volumes`) **Know** - Volume types you meet: hostPath (a directory of whatever node the Pod lands on), local (a disk of one node, needs nodeAffinity), emptyDir, nfs, csi, and the configMap / secret / projected volumes. - Access modes: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany, ReadWriteOncePod (one Pod). A claim binds only to a volume that offers its mode. - The reclaim policy is on the PersistentVolume: Retain keeps the volume and its data when the claim goes (it becomes Released), Delete removes both. **Fast** ``` apiVersion: v1 kind: PersistentVolume metadata: {name: pv-data} spec: capacity: {storage: 1Gi} accessModes: [ReadWriteOnce] persistentVolumeReclaimPolicy: Retain storageClassName: manual hostPath: {path: /mnt/data} --- k patch pv pv-data -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' k patch pv pv-data --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]' # Released -> Available k get pv -o custom-columns=NAME:.metadata.name,MODES:.spec.accessModes,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase,CLAIM:.spec.claimRef.name ``` **Traps** - A Released volume never binds again by itself: remove its claimRef first. - A `local` volume needs spec.nodeAffinity; a hostPath volume does not pin the Pod to the node that has the data. - The local-path provisioner of this cluster only does ReadWriteOnce and ReadWriteOncePod - a ReadWriteMany claim stays Pending. **Docs** - - - **Tasks** - Q15 PersistentVolume, claim and pod - Q35 Re-bind a Released PersistentVolume - Q58 Bind the right local volume - Q68 Provision geo-fence persistent storage _(Simulator Set C)_ - Q72 Migrate ride history to a Retain volume _(Simulator Set C)_ - Q102 A claim that can never bind _(Simulator Set D (killer.sh-style))_ ### 1.3 Manage persistent volumes and persistent volume claims `./exam study 1.3` (or `claims`) **Know** - A claim binds to one volume, and only if class, access mode and size fit (the volume may be larger); `volumeName` or a selector narrows it to one volume. - Phases: a volume goes Available, Bound, Released; a claim Pending, Bound. - A Pod mounts a claim of its own Namespace: volumes[].persistentVolumeClaim.claimName, then volumeMounts. - A claim in use is protected: deleting it waits (Terminating) until no Pod references it, finished Pods included. **Fast** ``` apiVersion: v1 kind: PersistentVolumeClaim metadata: {name: pvc-data, namespace: storage} spec: accessModes: [ReadWriteOnce] storageClassName: manual # omit for the default class resources: {requests: {storage: 500Mi}} --- # in the Pod: volumes: [{name: data, persistentVolumeClaim: {claimName: pvc-data}}] containers: [{..., volumeMounts: [{name: data, mountPath: /data}]}] k get pvc,pv -A; k describe pvc pvc-data # Used By: and Events: k describe pod POD | tail # unbound claim -> the Pod stays Pending ``` **Traps** - There is no `kubectl create pvc` or `create pv` - copy both from the docs. - For a hand-made volume the claim's storageClassName has to equal the volume's exactly; leaving it out means the default class, not none. - Apart from growing the request (and only if the class allows expansion) a claim cannot be edited. - To move data between claims, run a Pod or Job that mounts both. **Docs** - - **Tasks** - Q15 PersistentVolume, claim and pod - Q36 Data survives the pod - Q35 Re-bind a Released PersistentVolume - Q58 Bind the right local volume - Q59 Migrate data between claims - Q68 Provision geo-fence persistent storage _(Simulator Set C)_ - Q72 Migrate ride history to a Retain volume _(Simulator Set C)_ - Q81 Fix the fare calculation pods _(Simulator Set C)_ - Q99 Per-replica storage for a StatefulSet _(Simulator Set D (killer.sh-style))_ - Q102 A claim that can never bind _(Simulator Set D (killer.sh-style))_ ## Troubleshooting (30%) ### 2.1 Troubleshoot clusters and nodes `./exam study 2.1` (or `nodes`) **Know** - Start from the cluster: `k get nodes`, then `k describe node` - Conditions, Taints, Unschedulable, Events. - A NotReady node is nearly always its kubelet: on the node, `systemctl status kubelet` and `journalctl -u kubelet` name the cause. - What the kubelet needs: its unit and drop-in (which binary, which flags), /var/lib/kubelet/config.yaml, /etc/kubernetes/kubelet.conf (its kubeconfig), its certificates in /var/lib/kubelet/pki, and a running container runtime. - Ready but SchedulingDisabled is a cordon, not a fault: `k uncordon`. **Fast** ``` k get nodes -o wide; k describe node NODE | sed -n '/Conditions/,/Addresses/p' ssh NODE systemctl status kubelet; journalctl -u kubelet --no-pager | tail -30 systemctl cat kubelet # the unit and its drop-ins: ExecStart, EnvironmentFile cat /var/lib/kubelet/config.yaml; cat /var/lib/kubelet/kubeadm-flags.env systemctl daemon-reload; systemctl enable --now kubelet; systemctl restart kubelet systemctl status containerd; crictl ps -a k uncordon NODE; k describe node NODE | grep -i taint ``` **Traps** - A kubelet that was stopped is often disabled too: `enable --now`, or it is gone again after a reboot. - After changing a unit file nothing happens until `systemctl daemon-reload`. - There may be more than one fault. After every fix read the journal again instead of assuming you are done. **Docs** - - - **Tasks** - Q17 Node NotReady - Q39 kubelet misconfiguration - Q60 Worker node: three things wrong - Q92 Kubelet unit points at nothing _(Simulator Set D (killer.sh-style))_ - Q90 Event forensics around kube-proxy _(Simulator Set D (killer.sh-style))_ - Q95 The kubelet's two certificates _(Simulator Set D (killer.sh-style))_ ### 2.2 Troubleshoot cluster components `./exam study 2.2` (or `components`) **Know** - kube-apiserver, kube-scheduler, kube-controller-manager and etcd run as static Pods: the kubelet starts whatever is in /etc/kubernetes/manifests and restarts it when the file changes. - Read the symptom: Pods stay Pending with no events - scheduler. Deployments make no ReplicaSet or Pods - controller-manager. kubectl is refused - API server (or etcd behind it). - With the API server up: `k -n kube-system get pods`, `logs`, `describe`. With it down: `crictl ps -a`, `crictl logs`, /var/log/pods and the kubelet's journal on the node. **Fast** ``` k -n kube-system get pods; k -n kube-system logs kube-scheduler-NODE; k -n kube-system describe pod kube-scheduler-NODE ssh CONTROL-PLANE ls /etc/kubernetes/manifests; vi /etc/kubernetes/manifests/kube-scheduler.yaml crictl ps -a | grep -E 'kube-|etcd'; crictl logs CONTAINER-ID ls /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/ # its logs when kubectl cannot ask for them journalctl -u kubelet --no-pager | grep -i -E 'manifest|apiserver' | tail ``` **Traps** - The faults are small: a misspelt image, binary or flag, a wrong path in a volume or certificate flag, a wrong port in a probe. - Do not leave a backup copy inside the manifests directory - the kubelet runs every file there. - After saving a manifest give the kubelet up to a minute before deciding the fix did not work. **Docs** - - - **Tasks** - Q20 Control plane component failure - Q38 Deployments never create pods - Q61 Two control-plane components down - Q66 Nothing works: kube-apiserver itself is down - Q97 Stop the scheduler, schedule by hand _(Simulator Set D (killer.sh-style))_ - Q86 How the control plane is run _(Simulator Set D (killer.sh-style))_ ### 2.3 Monitor cluster and application resource usage `./exam study 2.3` (or `usage`) **Know** - `kubectl top node|pod` shows live CPU and memory from metrics-server through the metrics API (metrics.k8s.io); `--containers` splits a Pod up, `--sort-by=cpu|memory` orders it. - `k describe node` shows what is allocated - the sum of requests and limits - which is not what is used. - `kubectl get` sorts and shapes everything else: --sort-by, -o custom-columns, -o jsonpath, -A. **Fast** ``` k top node; k top pod -A --sort-by=memory | head; k top pod -n NS --containers k describe node NODE | grep -A8 'Allocated resources' k get pods -A --sort-by='.status.containerStatuses[0].restartCount' k -n kube-system get deploy metrics-server; k get apiservice v1beta1.metrics.k8s.io k -n kube-system logs deploy/metrics-server ``` **Traps** - `Metrics API not available` means metrics-server is missing or not Ready (on this cluster it needs --kubelet-insecure-tls). - `kubectl top --sort-by` puts the largest first; `kubectl get --sort-by` puts it last. - A new Pod has no metrics for about a minute. **Docs** - - - **Tasks** - Q103 Find the heaviest Pods with kubectl top - Q23 metrics-server repair and HPA - Q53 Namespace audit to exact files - Q85 Investigate vehicle tracker health _(Simulator Set C)_ - Q89 Sorting done by kubectl _(Simulator Set D (killer.sh-style))_ ### 2.4 Manage and evaluate container output streams `./exam study 2.4` (or `logs`) **Know** - `kubectl logs` shows what a container wrote to stdout and stderr. The runtime keeps it on the node under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/, and it goes when the Pod is deleted. - Pick the container with -c (init containers too) or take --all-containers; --previous is the run before the last restart. - Narrow with --tail=N, --since=10m, --timestamps; follow with -f; take every Pod of a label with -l and tell them apart with --prefix. - On a node, without the API server: `crictl logs CONTAINER-ID`. **Fast** ``` k logs POD -c CONTAINER # --all-containers --prefix k logs POD --previous # the crashed run k logs POD --tail=50 --since=10m --timestamps k logs -l app=web --prefix --tail=5 # every Pod with the label k logs deploy/web; k logs job/render # one Pod of it k logs POD | grep ERROR > errors.log crictl ps -a; crictl logs --tail=20 CONTAINER-ID; ls /var/log/pods/ ``` **Traps** - With -l, only the last 10 lines of each Pod are shown unless you set --tail (--tail=-1 for everything). - `kubectl logs deploy/x` reads one Pod, not all replicas. - --previous fails when the container has never restarted. **Docs** - - - **Tasks** - Q19 CrashLoopBackOff diagnosis - Q104 One replica is logging errors - Q8 Multi-container pod with shared volume - Q85 Investigate vehicle tracker health _(Simulator Set C)_ ### 2.5 Troubleshoot services and networking `./exam study 2.5` (or `netdebug`) **Know** - Work outwards from the Pod: is it Ready, does the Service have endpoints (selector against Pod labels), is targetPort the port the container listens on, does the name resolve, does a NetworkPolicy stand in the way. - DNS is a Service like any other: the CoreDNS Pods, the kube-dns Service and its endpoints, the Corefile in ConfigMap coredns. - Underneath: kube-proxy (a DaemonSet) turns Services into rules on every node, and the CNI plugin connects the Pods. **Fast** ``` k get svc,endpointslices -o wide; k describe svc SVC # Endpoints: = nothing selected k get pods -l KEY=VALUE --show-labels -o wide k run tmp --rm -it --restart=Never --image=busybox:1.36 -- sh # wget -qO- -T3 http://SVC:PORT ; nslookup SVC.NS.svc.cluster.local k -n kube-system get pods -l k8s-app=kube-dns -o wide; k -n kube-system logs -l k8s-app=kube-dns k -n kube-system get cm coredns -o yaml; k -n kube-system get svc,endpointslices -l k8s-app=kube-dns k -n kube-system get ds kube-proxy; k -n kube-system logs ds/kube-proxy k get netpol -A ``` **Traps** - A Service with no endpoints still resolves - it is the connection that fails. - Test names with the fully qualified form (SVC.NS.svc.cluster.local): busybox's nslookup misreports short ones. - A NetworkPolicy in kube-system can cut DNS off for the whole cluster. - Fix what is broken and leave the rest alone - the graders check what you did not have to touch. **Docs** - - **Tasks** - Q18 Broken deployment and service - Q62 Application down for four reasons - Q37 Cluster DNS is broken - Q57 DNS down: three faults - Q73 Fix ETA service connectivity _(Simulator Set C)_ - Q101 Resolves everything except the cluster _(Simulator Set D (killer.sh-style))_ - Q70 Fix the driver onboarding app _(Simulator Set C)_ - Q90 Event forensics around kube-proxy _(Simulator Set D (killer.sh-style))_ ## Workloads & Scheduling (15%) ### 3.1 Understand application deployments and how to perform rolling update and rollbacks `./exam study 3.1` (or `rollouts`) **Know** - Any change to `spec.template` starts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default). - Every rollout is a revision. `kubectl rollout`: status, history, undo, pause, resume, restart. - minReadySeconds, revisionHistoryLimit and progressDeadlineSeconds tune how careful and how patient a rollout is. **Fast** ``` k set image deploy/web nginx=nginx:1.27 # CONTAINER=IMAGE k scale deploy/web --replicas=4 k rollout status deploy/web k rollout history deploy/web; k rollout history deploy/web --revision=2 # that revision's template k rollout undo deploy/web --to-revision=2 k rollout pause deploy/web; k rollout resume deploy/web k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' ``` **Traps** - To go back to a particular image, look at each revision's template first - "the previous one" is not always the one asked for. - An undo makes a new revision out of the old one, so the numbers keep climbing. - A paused Deployment records changes but rolls nothing out until it is resumed. - maxSurge and maxUnavailable cannot both be 0. **Docs** - **Tasks** - Q7 Rolling update & scaling - Q51 Rollback to a specific revision, then pause ### 3.2 Use ConfigMaps and Secrets to configure applications `./exam study 3.2` (or `config`) **Know** - Both hold keys. A key becomes an environment variable (configMapKeyRef / secretKeyRef, or every key at once with envFrom) or a file (a configMap / secret volume; `items` picks and renames keys). - Secret data is base64 - an encoding, not encryption. Types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson. - Mounted files follow a change after a short delay. Environment variables and subPath mounts never do: restart the Pods. - `immutable: true` freezes one for good. To change the content, make a new object and point the workload at it. **Fast** ``` k create cm app-config --from-literal=APP_MODE=production --from-file=app.properties k create secret generic db-creds --from-literal=username=admin --from-literal=password='S3cr3t!' k create secret tls web-tls --cert=tls.crt --key=tls.key k get secret db-creds -o jsonpath='{.data.password}' | base64 -d env: - {name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}} - {name: DB_PASS, valueFrom: {secretKeyRef: {name: db-creds, key: password}}} envFrom: [{configMapRef: {name: app-config}}, {secretRef: {name: db-creds}}] volumes: [{name: creds, secret: {secretName: db-creds, defaultMode: 0400}}] k set env deploy/web --from=configmap/app-config; k rollout restart deploy/web ``` **Traps** - A secret volume names it with `secretName`; a configMap volume uses `name`. - A Pod that references a missing ConfigMap, Secret or key waits in CreateContainerConfigError. - `--from-file` makes one key named after the file; `--from-env-file` makes one key per line. **Docs** - - - **Tasks** - Q10 ConfigMap and Secret consumption - Q29 Immutable ConfigMap rotation - Q79 Surge pricing configuration _(Simulator Set C)_ - Q93 Secrets as a read-only volume and renamed env vars _(Simulator Set D (killer.sh-style))_ - Q84 Enforce TLS 1.3 on the booking portal _(Simulator Set C)_ ### 3.3 Configure workload autoscaling `./exam study 3.3` (or `autoscaling`) **Know** - A HorizontalPodAutoscaler (autoscaling/v2) keeps a workload between minReplicas and maxReplicas by a metric - usually average CPU or memory utilisation, which is usage divided by the container's request. - It reads the metrics API, so it needs metrics-server, and the containers need a request for the resource it scales on. - `behavior` shapes the change: scaleUp and scaleDown each take a stabilizationWindowSeconds and policies (so many Pods or Percent per periodSeconds). **Fast** ``` k set resources deploy web --requests=cpu=100m k autoscale deploy web --min=2 --max=6 --cpu=60% # kubectl before 1.34: --cpu-percent=60 k get hpa; k describe hpa web apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: {name: web} spec: scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: web} minReplicas: 2 maxReplicas: 6 metrics: - type: Resource resource: {name: cpu, target: {type: Utilization, averageUtilization: 60}} behavior: scaleDown: stabilizationWindowSeconds: 300 policies: [{type: Pods, value: 1, periodSeconds: 60}] ``` **Traps** - TARGETS ``: no metrics-server, or no request on the containers. - `kubectl autoscale` cannot write `behavior` - edit the object or write the YAML. - minReplicas is enforced at once, even without metrics. **Docs** - - - **Tasks** - Q105 Autoscale on CPU, scale down calmly - Q23 metrics-server repair and HPA - Q82 Autoscale the route planner on memory _(Simulator Set C)_ ### 3.4 Understand the primitives used to create robust, self-healing, application deployments `./exam study 3.4` (or `selfhealing`) **Know** - Controllers put back what is missing: a Deployment its replicas, a DaemonSet one Pod per node, a StatefulSet named replicas with their own storage, a Job its completions (backoffLimit retries), a CronJob its Jobs. - Probes tell the kubelet about health: liveness restarts the container, readiness takes the Pod out of Service endpoints, startup holds both off until the app is up. - Init containers run to completion, in order, before the app starts; a native sidecar (an init container with restartPolicy: Always) keeps running beside it. - A PodDisruptionBudget (minAvailable or maxUnavailable) limits how many Pods a drain may take away at once. **Fast** ``` containers: - name: app startupProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12} readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 5} livenessProbe: {tcpSocket: {port: 80}, initialDelaySeconds: 15, periodSeconds: 10} # exec: {exec: {command: [cat, /tmp/ready]}} k create job render --image=busybox:1.36 -- sh -c 'echo done' k create cronjob digest --image=busybox:1.36 --schedule='*/5 * * * *' -- date; k create job run-1 --from=cronjob/digest k create pdb web --selector=app=web --min-available=2 k create deploy x --image=busybox:1.36 $do > ds.yaml # DaemonSet: change kind, drop replicas / strategy / status k describe pod POD | grep -E 'Liveness|Readiness|Startup'; k get events --field-selector reason=Unhealthy ``` **Traps** - Probes belong to a container. No kubectl flag adds one - edit the YAML. - Restarts with Unhealthy and Killing events but a clean log: the liveness probe is wrong, not the app. - A DaemonSet reaches control-plane nodes only with a toleration for node-role.kubernetes.io/control-plane. - A Job's Pod template cannot be edited: delete the Job and create it again. **Docs** - - - - **Tasks** - Q106 Health checks that check the wrong thing - Q8 Multi-container pod with shared volume - Q28 Init container prepares content - Q26 DaemonSet on every node - Q27 CronJob with a manual run - Q31 Parallel Job - Q52 Native sidecar with probes and an exact report - Q53 Namespace audit to exact files - Q76 Add an init container to the trip logger _(Simulator Set C)_ - Q88 Ready only when its dependency answers _(Simulator Set D (killer.sh-style))_ - Q70 Fix the driver onboarding app _(Simulator Set C)_ ### 3.5 Configure Pod admission and scheduling (limits, node affinity, etc.) `./exam study 3.5` (or `scheduling`) **Know** - Requests are what the scheduler reserves, limits what the node enforces. A LimitRange gives containers defaults and bounds; a ResourceQuota caps a Namespace's totals - both act at admission. - Where a Pod may go: nodeSelector, node affinity (required or preferred), Pod affinity and anti-affinity with a topologyKey, topologySpreadConstraints, or nodeName, which skips the scheduler. - Taints keep Pods off a node (NoSchedule, PreferNoSchedule, NoExecute); a toleration lets a Pod through. - A PriorityClass orders Pods when room is short; the Pod names it in priorityClassName. **Fast** ``` k label node NODE disktype=ssd k taint node NODE dedicated=gpu:NoSchedule; k taint node NODE dedicated- # the trailing - removes it tolerations: [{key: dedicated, operator: Equal, value: gpu, effect: NoSchedule}] affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: [{matchExpressions: [{key: disktype, operator: In, values: [ssd]}]}] podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - {labelSelector: {matchLabels: {app: web}}, topologyKey: kubernetes.io/hostname} k create priorityclass high --value=100000 --description='critical workloads' k create quota team -n NS --hard=pods=5,requests.cpu=1,limits.memory=2Gi k describe pod POD | tail # FailedScheduling: 0/4 nodes are available: ... ``` **Traps** - A toleration lets a Pod onto a tainted node but does not send it there - pair it with a nodeSelector or affinity. - Once a quota covers CPU or memory, every new Pod must state them or get them from a LimitRange. - In a LimitRange, `default` is the limit and `defaultRequest` the request. - Scheduling fields of a running Pod cannot be changed: change the Deployment, or delete and recreate the Pod. **Docs** - - - - **Tasks** - Q9 Scheduling with node affinity and resources - Q25 Taints and tolerations - Q24 PriorityClass - Q32 ResourceQuota and LimitRange - Q50 Spread, tolerate, prioritise — under a quota - Q78 Priority for the rider support portal _(Simulator Set C)_ - Q96 A Pod for the control plane only _(Simulator Set D (killer.sh-style))_ - Q81 Fix the fare calculation pods _(Simulator Set C)_ - Q97 Stop the scheduler, schedule by hand _(Simulator Set D (killer.sh-style))_ ## Cluster Architecture, Installation & Configuration (25%) ### 4.1 Manage role based access control (RBAC) `./exam study 4.1` (or `rbac`) **Know** - A Role (one Namespace) or ClusterRole (everywhere, and cluster-scoped resources) lists verbs on resources; a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts. - A RoleBinding may point at a ClusterRole: the rules then apply in that one Namespace only. - A user is a certificate: CN is the name, O the groups. Create a key and CSR, submit a CertificateSigningRequest, approve it, put the certificate in a kubeconfig. - An aggregated ClusterRole has no rules of its own: the controller fills them in from every ClusterRole carrying the label it selects. **Fast** ``` k create role deploy-manager -n NS --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps k create rolebinding deploy-bot -n NS --role=deploy-manager --serviceaccount=NS:deploy-bot # --user=jane --group=devs k create clusterrole node-reader --verb=get,list,watch --resource=nodes k create clusterrolebinding node-reader --clusterrole=node-reader --serviceaccount=NS:SA k auth can-i create deployments -n NS --as=system:serviceaccount:NS:deploy-bot; k auth can-i --list --as=jane -n NS openssl genrsa -out jane.key 2048; openssl req -new -key jane.key -subj '/CN=jane/O=developers' -out jane.csr # CertificateSigningRequest: spec.request = base64 -w0 of the .csr, signerName kubernetes.io/kube-apiserver-client, usages [client auth] k certificate approve jane; k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt k config set-credentials jane --client-certificate=jane.crt --client-key=jane.key --embed-certs ``` **Traps** - "Exactly these permissions" also means finding what is already granted: `k get rolebindings,clusterrolebindings -A -o wide | grep NAME`. - Deployments are in the group apps: --resource=deployments.apps; a subresource is deployments.apps/scale, pods/log. - The same ServiceAccount is `--serviceaccount=NS:NAME` in a binding and `system:serviceaccount:NS:NAME` after `--as`. - Nodes, PersistentVolumes and Namespaces are cluster-scoped: they need a ClusterRole and a ClusterRoleBinding. **Docs** - - **Tasks** - Q1 RBAC for a deployment bot - Q33 Aggregated ClusterRole - Q6 Certificates: expiry check and user CSR - Q44 Exact RBAC with an existing over-grant - Q47 User onboarding to a working kubeconfig - Q75 Cluster-wide read access for the auth team _(Simulator Set C)_ - Q91 API resources and a Role census _(Simulator Set D (killer.sh-style))_ ### 4.2 Prepare underlying infrastructure for installing a Kubernetes cluster `./exam study 4.2` (or `infra`) **Know** - Every machine needs, before kubeadm: a container runtime (containerd) using the same cgroup driver as the kubelet (systemd), the kernel modules overlay and br_netfilter, the sysctls net.bridge.bridge-nf-call-iptables and net.ipv4.ip_forward, and swap off or accounted for. - kubeadm, kubelet and kubectl come from the package repository of one minor version (pkgs.k8s.io) and are held, so that a system update does not move them. - Also: a unique hostname, MAC address and product_uuid per node, and the ports open - 6443 (API), 10250 (kubelet), 2379-2380 (etcd). **Fast** ``` printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf; modprobe overlay; modprobe br_netfilter cat > /etc/sysctl.d/k8s.conf < - **Tasks** - Q107 Kernel prerequisites for a Kubernetes node - Q108 The container runtime behind a Pod - Q41 Component versions ### 4.3 Create and manage Kubernetes clusters using kubeadm `./exam study 4.3` (or `kubeadm`) **Know** - `kubeadm init` makes the control plane: certificates in /etc/kubernetes/pki, kubeconfigs in /etc/kubernetes, static Pod manifests in /etc/kubernetes/manifests, then CoreDNS and kube-proxy. A CNI plugin is yours to add. - `kubeadm join` adds a node with a bootstrap token and the hash of the cluster CA; `kubeadm token create --print-join-command` writes the whole line. - What the cluster was built with is in ConfigMap kubeadm-config (kube-system). `kubeadm certs check-expiration` lists every certificate it manages. - A static Pod is a manifest in the kubelet's staticPodPath; it shows up in the API as NAME-NODE and can only be changed or removed through the file. **Fast** ``` kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=HOST:6443 kubeadm token create --ttl 2h --print-join-command; kubeadm token list openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex # the CA hash kubeadm certs check-expiration; openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate k -n kube-system get cm kubeadm-config -o yaml grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $do > /etc/kubernetes/manifests/web.yaml ``` **Traps** - Tokens expire (24 hours unless you say otherwise) - make a new one rather than hunting for the old. - Check staticPodPath before you write a static Pod: it is not always /etc/kubernetes/manifests. - Deleting a static Pod with kubectl only deletes its mirror; the kubelet puts it straight back. **Docs** - - - - **Tasks** - Q40 kubeadm join token - Q5 Static pod - Q6 Certificates: expiry check and user CSR - Q67 Prepare a node join without shell access to the control plane - Q46 Static pod on a relocated manifest path - Q94 Static Pod on the control plane behind a Service _(Simulator Set D (killer.sh-style))_ - Q95 The kubelet's two certificates _(Simulator Set D (killer.sh-style))_ - Q80 Plan the control-plane upgrade _(Simulator Set C)_ ### 4.4 Manage the lifecycle of Kubernetes clusters `./exam study 4.4` (or `lifecycle`) **Know** - Upgrade one minor version at a time, control plane first: the kubeadm package, `kubeadm upgrade plan` and `apply`, drain, the kubelet and kubectl packages, restart the kubelet, uncordon. Workers: `kubeadm upgrade node` in place of `apply`. - Maintenance on a node: `kubectl drain` cordons it and evicts its Pods; `kubectl uncordon` gives it back. - etcd: `etcdctl snapshot save` backs it up; `etcdutl snapshot restore --data-dir` unpacks a snapshot into a new directory that etcd's manifest must then point at. - `kubeadm certs renew all` renews the control-plane certificates; the static Pods have to be restarted to load them. **Fast** ``` k drain NODE --ignore-daemonsets --delete-emptydir-data; k uncordon NODE apt-get install -y kubeadm='1.36.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.36.x # workers: kubeadm upgrade node apt-get install -y kubelet='1.36.*' kubectl='1.36.*'; systemctl daemon-reload; systemctl restart kubelet etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db etcdutl snapshot restore /opt/backup.db --data-dir /var/lib/etcd-restore # then etcd.yaml: the hostPath of volume etcd-data kubeadm certs renew all; kubeadm certs check-expiration ``` **Traps** - drain stops at DaemonSet Pods (--ignore-daemonsets), at emptyDir data (--delete-emptydir-data) and at bare Pods (--force, which deletes them for good). - A drained node stays cordoned until you uncordon it. - With etcd 3.6, as here, restore exists only in etcdutl - `etcdctl snapshot restore` is gone. - Restoring etcd takes the whole cluster back to the snapshot: everything created since is lost. **Docs** - - - - **Tasks** - Q109 Drain the control-plane node and bring it back - Q2 etcd snapshot - Q65 Renew control-plane certificates - Q45 etcd restore - Q80 Plan the control-plane upgrade _(Simulator Set C)_ ### 4.5 Implement and configure a highly-available control plane `./exam study 4.5` (or `ha`) **Know** - Highly available means several control-plane nodes (three as a rule) behind one stable address - the controlPlaneEndpoint, a load balancer or virtual IP, fixed at `kubeadm init`. - etcd is either stacked (a member on every control-plane node) or external (hosts of its own). It needs a majority to accept writes: 3 members survive the loss of 1, 5 of 2. - More control-plane nodes join with `kubeadm join --control-plane --certificate-key KEY`; the key comes from `kubeadm init phase upload-certs --upload-certs` and is good for two hours. - Every kube-apiserver serves at once. Of the schedulers and controller-managers only the leader acts - they elect it through a Lease in kube-system. **Fast** ``` k get nodes -l node-role.kubernetes.io/control-plane k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint kubeadm init --control-plane-endpoint=LB:6443 --upload-certs kubeadm init phase upload-certs --upload-certs # prints the certificate key kubeadm token create --print-join-command --certificate-key KEY # ... --control-plane --certificate-key KEY k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER = NODE_uuid etcdctl member list -w table; etcdctl endpoint status -w table; etcdctl endpoint health # with the four etcd flags k -n kube-system get pods -l component=etcd -o wide ``` **Traps** - A cluster initialised without a controlPlaneEndpoint cannot simply be given a second control-plane node. - An even number of etcd members tolerates no more failures than the odd number below it. - The certificate key expires after two hours - upload the certificates again rather than reusing an old key. **Docs** - - **Tasks** - Q110 Prepare the join of a second control-plane node - Q111 Who leads, and who is in etcd - Q86 How the control plane is run _(Simulator Set D (killer.sh-style))_ - Q87 Cluster facts to an exact file _(Simulator Set D (killer.sh-style))_ ### 4.6 Use Helm and Kustomize to install cluster components `./exam study 4.6` (or `helmkustomize`) **Know** - Helm: a chart is the package, a release one installation of it in a Namespace, and each install, upgrade or rollback a revision. Values come from the chart, then -f files, then --set. - `helm template` renders a chart without installing it; `helm show values` prints its defaults; `helm get values` what a release was given. - Kustomize: a kustomization.yaml lists resources and changes them - namespace, namePrefix, labels, images, replicas, patches - and generates ConfigMaps and Secrets. An overlay lists a base and changes only what differs. **Fast** ``` helm repo add NAME URL && helm repo update; helm search repo NAME --versions helm install REL CHART -n NS --create-namespace --set replicaCount=2 -f values.yaml # --version X helm upgrade REL CHART -n NS --reuse-values --set image.tag=1.27 helm template REL CHART -n NS --set replicaCount=3 > manifest.yaml helm list -A; helm history REL -n NS; helm rollback REL 1 -n NS; helm uninstall REL -n NS k kustomize DIR | less; k apply -k DIR # kustomization.yaml: resources: [../../base] namespace: prod namePrefix: prod- images: [{name: nginx, newTag: "1.27"}] # replicas: [{name: web, count: 3}] patches: [{path: patch.yaml}] configMapGenerator: [{name: cfg, literals: [k=v]}] ``` **Traps** - Releases live in a Namespace: without -n Helm looks in `default`. - An upgrade without --reuse-values (or the same -f and --set again) puts every value back to the chart's default. - In a kustomization, `images`, `replicas` and patches use the names of the base, before any prefix. - `namespace:` in a kustomization does not create the Namespace. **Docs** - - - **Tasks** - Q21 Helm release install and upgrade - Q3 Kustomize overlay - Q48 Kustomize with generators and patches - Q69 Render a chart, then install it _(Simulator Set C)_ - Q98 Helm release housekeeping _(Simulator Set D (killer.sh-style))_ ### 4.7 Understand extension interfaces (CNI, CSI, CRI, etc.) `./exam study 4.7` (or `interfaces`) **Know** - CRI: the kubelet drives the container runtime (containerd, CRI-O) through a socket; `crictl` speaks the same protocol, so it works on a node when kubectl does not. - CNI: for every Pod the runtime calls the plugins named in /etc/cni/net.d (binaries in /opt/cni/bin) to give it an address and wire it up. The plugin - kindnet, Calico, Cilium, Flannel - usually runs as a DaemonSet. - CSI: a storage driver runs as Pods, registers a CSIDriver object, and a StorageClass names it as provisioner. **Fast** ``` k get nodes -o wide # CONTAINER-RUNTIME column crictl version; crictl pods; crictl ps -a; crictl images; crictl inspect CONTAINER-ID grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml; cat /etc/crictl.yaml ls /etc/cni/net.d; cat /etc/cni/net.d/*; ls /opt/cni/bin k -n kube-system get ds # the CNI's DaemonSet beside kube-proxy k get csidrivers; k get csinodes; k get sc # PROVISIONER ``` **Traps** - Not every CNI plugin enforces NetworkPolicy (Flannel does not; kindnet in this cluster does). - Without a CNI plugin the nodes stay NotReady: `cni plugin not initialized`. - The runtime takes the first configuration file in /etc/cni/net.d, in alphabetical order. **Docs** - - - **Tasks** - Q112 CNI, CSI and CRI: what this cluster plugs in - Q108 The container runtime behind a Pod - Q41 Component versions - Q63 Cluster facts, exactly - Q77 Audit the CNI, then lock a namespace down _(Simulator Set C)_ - Q87 Cluster facts to an exact file _(Simulator Set D (killer.sh-style))_ ### 4.8 Understand CRDs, install and configure operators `./exam study 4.8` (or `crds`) **Know** - A CustomResourceDefinition adds a kind to the API: group, names (kind, plural, singular, shortNames), scope (Namespaced or Cluster) and versions, each with an OpenAPI v3 schema. - The schema validates custom resources - types, required fields, enum, minimum, maximum, defaults - and feeds `kubectl explain`. - An operator is a CRD plus a controller (a Deployment with RBAC) that acts on those objects. Installing one means applying its manifests or its Helm chart, then creating a custom resource. **Fast** ``` apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: {name: backups.data.example.com} # PLURAL.GROUP spec: group: data.example.com scope: Namespaced names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]} versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object required: [schedule] properties: schedule: {type: string} retention: {type: integer, minimum: 1, default: 7} k get crd; k api-resources --api-group=data.example.com; k explain backup.spec --recursive ``` **Traps** - The CRD's name must be PLURAL.GROUP, and exactly one version has `storage: true`. - A custom resource's apiVersion is GROUP/VERSION of the CRD, not apiextensions.k8s.io/v1. - `default` and validation only work inside a structural schema: every level needs its `type`. - Deleting a CRD deletes every object of that kind. **Docs** - - - **Tasks** - Q4 CustomResourceDefinition - Q34 Discover a CRD field with kubectl explain - Q49 CRD with validation and defaults - Q91 API resources and a Role census _(Simulator Set D (killer.sh-style))_ ## Services & Networking (20%) ### 5.1 Understand connectivity between Pods `./exam study 5.1` (or `connectivity`) **Know** - Every Pod has its own IP address, and any Pod reaches any other Pod on any node at that address without NAT. The CNI plugin makes that true. - Each node hands out Pod addresses from its own range (spec.podCIDR), a slice of the cluster's Pod CIDR; Services get theirs from the separate service CIDR. - A hostNetwork Pod lives in the node's network: it has the node's address and shares its ports. - Containers of one Pod share a network namespace and talk over localhost. **Fast** ``` k get pods -o wide -A; k get nodes -o custom-columns=NODE:.metadata.name,PODCIDR:.spec.podCIDR,IP:.status.addresses[0].address k exec POD -- wget -qO- -T3 http://POD-IP:PORT k -n kube-system get cm kubeadm-config -o yaml | grep -E 'podSubnet|serviceSubnet' k -n kube-system get pods -o wide | grep -E 'kube-proxy|kindnet'; k -n kube-system get cm kube-proxy -o yaml | grep mode ``` **Traps** - A Pod's address changes whenever the Pod is replaced - that is what Services are for. - Reaching a Pod by IP proves the network; reaching it by Service name also needs endpoints and DNS. - If only the Pods of one node are unreachable, look at that node's CNI and kube-proxy Pods. **Docs** - - **Tasks** - Q113 Pod to Pod, across nodes - Q63 Cluster facts, exactly - Q71 Expose and verify the dispatch API _(Simulator Set C)_ - Q77 Audit the CNI, then lock a namespace down _(Simulator Set C)_ - Q87 Cluster facts to an exact file _(Simulator Set D (killer.sh-style))_ ### 5.2 Define and enforce Network Policies `./exam study 5.2` (or `netpol`) **Know** - Pods accept and send everything until a NetworkPolicy selects them. From then on, for the policyTypes it lists, only what some policy allows gets through. - Policies only allow, and they add up: there is no deny rule and no ordering. - Peers: podSelector (same Namespace), namespaceSelector, ipBlock. Two selectors in one list item mean AND; two list items mean OR. - Default deny: `podSelector: {}` with the policyTypes and no rules. An egress lock-down also has to allow DNS (UDP and TCP 53). **Fast** ``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: {name: backend-policy, namespace: NS} spec: podSelector: {matchLabels: {app: backend}} policyTypes: [Ingress, Egress] ingress: - from: [{podSelector: {matchLabels: {app: frontend}}}] ports: [{protocol: TCP, port: 8080}] egress: - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}] ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}] k describe netpol backend-policy -n NS; k exec CLIENT -- wget -qO- -T2 http://backend:8080 ``` **Traps** - There is no `kubectl create networkpolicy` - copy the skeleton from the docs. - `namespaceSelector` and `podSelector` in the same item: Pods with that label in those Namespaces. In two items: either. - A missing `ingress` denies everything; `ingress: [{}]` allows everything. - A Namespace is matched by its labels; kubernetes.io/metadata.name=NAME is always there. **Docs** - **Tasks** - Q11 NetworkPolicy - Q42 Egress-only NetworkPolicy - Q54 Zero-trust namespace - Q83 NetworkPolicy for the safety monitor _(Simulator Set C)_ - Q100 Namespace AND pod selector, enforced _(Simulator Set D (killer.sh-style))_ - Q77 Audit the CNI, then lock a namespace down _(Simulator Set C)_ ### 5.3 Use ClusterIP, NodePort, LoadBalancer service types and endpoints `./exam study 5.3` (or `services`) **Know** - A Service selects Pods by label and forwards `port` to `targetPort` - a number, or the name of a container port. - ClusterIP is reachable inside the cluster; NodePort adds a port (30000-32767) on every node; LoadBalancer adds an external address from the cloud; ExternalName is a DNS alias; headless (clusterIP: None) gives every Pod its own DNS record. - The Pods behind a Service are its endpoints, kept in EndpointSlices; only Ready Pods are in them. **Fast** ``` k expose deploy shop --name=shop-svc --port=80 --target-port=80 --type=NodePort k create svc nodeport shop-svc --tcp=80:80 --node-port=30080 # then check the selector k create svc externalname legacy --external-name=db.example.com k get svc,endpointslices -o wide; k describe svc shop-svc # Endpoints: k get pods -l app=shop --show-labels k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://shop-svc ``` **Traps** - `kubectl expose` copies the workload's selector; `kubectl create svc` selects app=, which may match nothing. - `expose` cannot choose the nodePort - use `create svc nodeport --node-port` or edit the YAML. - A LoadBalancer stays where nothing provides one; it still works as a NodePort. - A headless Service for a StatefulSet has to carry the name in the StatefulSet's serviceName. **Docs** - - **Tasks** - Q12 NodePort Service - Q30 Named ports - Q43 Headless and ExternalName services - Q64 The context's namespace - Q71 Expose and verify the dispatch API _(Simulator Set C)_ - Q73 Fix ETA service connectivity _(Simulator Set C)_ - Q94 Static Pod on the control plane behind a Service _(Simulator Set D (killer.sh-style))_ ### 5.4 Use the Gateway API to manage Ingress traffic `./exam study 5.4` (or `gateway`) **Know** - Three kinds, three owners: a GatewayClass names the controller, a Gateway opens listeners (port, protocol, hostname, which routes may attach), an HTTPRoute attaches to a Gateway (parentRefs) and sends requests to Services (backendRefs). - An HTTPRoute rule matches on path, header, method or query, and may split traffic between backends by weight - that is a canary. - The kinds are CRDs (gateway.networking.k8s.io): they have to be installed, and a controller has to implement them. **Fast** ``` apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: {name: web-gw, namespace: NS} spec: gatewayClassName: example listeners: [{name: http, protocol: HTTP, port: 80, hostname: shop.example.com}] --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: {name: web, namespace: NS} spec: parentRefs: [{name: web-gw}] hostnames: [shop.example.com] rules: - matches: [{path: {type: PathPrefix, value: /}}] backendRefs: [{name: v1, port: 80, weight: 90}, {name: v2, port: 80, weight: 10}] k get gatewayclass,gateway,httproute -A; k describe httproute web -n NS ``` **Traps** - A route in another Namespace attaches only if the listener's allowedRoutes lets it. - A header match belongs in `matches`, beside or instead of the path - and several rules are tried most specific first. - No Gateway controller runs in this simulator, so these tasks are graded on the objects. **Docs** - - **Tasks** - Q22 Gateway API HTTPRoute - Q56 Gateway API canary with header match - Q74 Fleet dashboard canary with Gateway API _(Simulator Set C)_ ### 5.5 Know how to use Ingress controllers and Ingress resources `./exam study 5.5` (or `ingress`) **Know** - An Ingress routes HTTP(S) by host and path to Services of its own Namespace. It does nothing without an ingress controller, chosen with ingressClassName (an IngressClass names the controller). - Each path has a pathType (Prefix, Exact or ImplementationSpecific) and a backend: service.name plus service.port.number or .name. - TLS: spec.tls lists hosts and the kubernetes.io/tls Secret that holds their certificate. **Fast** ``` k create ingress app-ingress -n NS --class=nginx \ --rule='shop.example.com/api*=api:8080,tls=shop-tls' --rule='shop.example.com/*=web:80,tls=shop-tls' openssl req -x509 -nodes -newkey rsa:2048 -days 365 -subj '/CN=shop.example.com' -keyout tls.key -out tls.crt k create secret tls shop-tls -n NS --cert=tls.crt --key=tls.key k get ingress,ingressclass -A; k describe ingress app-ingress -n NS ``` **Traps** - In --rule a trailing * means pathType Prefix; without it the path is Exact. - The backend port is the Service's port, not the container's. - No ingress controller runs in this simulator, so Ingress tasks are graded on the object - on the exam, curl it. **Docs** - - **Tasks** - Q13 Ingress routing - Q55 TLS Ingress with a generated certificate ### 5.6 Understand and use CoreDNS `./exam study 5.6` (or `coredns`) **Know** - CoreDNS is a Deployment in kube-system behind the Service kube-dns; its configuration, the Corefile, is in ConfigMap coredns: the `kubernetes` plugin answers for cluster.local, `forward` sends the rest upstream. - Names: SVC.NS.svc.cluster.local for a Service, POD.SVC.NS.svc.cluster.local for the Pods of a headless Service. - A Pod's /etc/resolv.conf points at the kube-dns address and lists the search domains that make short names work. dnsPolicy decides: ClusterFirst (default), Default (the node's resolver), None with dnsConfig. **Fast** ``` k -n kube-system get deploy coredns; k -n kube-system get pods -l k8s-app=kube-dns -o wide k -n kube-system get svc kube-dns; k -n kube-system get endpointslices -l k8s-app=kube-dns k -n kube-system get cm coredns -o yaml; k -n kube-system edit cm coredns k -n kube-system logs -l k8s-app=kube-dns; k -n kube-system rollout restart deploy coredns k run tmp --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc.cluster.local k exec POD -- cat /etc/resolv.conf ``` **Traps** - A typo in the Corefile puts CoreDNS in CrashLoopBackOff - its log names the plugin and the line. - DNS can be broken in several places at once: the Corefile, the Deployment (replicas, image), the kube-dns Service (selector, ports), a NetworkPolicy. - A Pod with dnsPolicy: Default resolves the internet but not cluster names. **Docs** - - - **Tasks** - Q14 Cluster DNS - Q37 Cluster DNS is broken - Q57 DNS down: three faults - Q101 Resolves everything except the cluster _(Simulator Set D (killer.sh-style))_