Free to read · CKA · Cluster architecture and kubectl fluency
Control plane components up close
How the control plane and the nodes of a kubeadm cluster work together, how the API is organised, and the kubectl and exam habits that make every later module faster.
Lesson · 12 min. This module is free on this site. The rest of the course, the quizzes and the labs open with a free account.
Why this matters
On a kubeadm cluster the control plane is four static pods. When one of them breaks, kubectl may stop working, so you have to fix it from the node's shell. Knowing where each component's configuration lives, which ports it uses and how to read its logs without the API is a core troubleshooting skill.
Where the control plane lives
kubeadm writes one manifest per component into the kubelet's static pod directory:
ls /etc/kubernetes/manifests
# etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
The kubelet watches that directory. Change a file and the kubelet stops the old pod and starts a new one. Delete the file and the pod goes away. The API shows a read-only mirror pod named <component>-<node>, for example kube-apiserver-controlplane in kube-system. Editing or deleting the mirror pod with kubectl does not change the real pod. Always edit the file.
Certificates and kubeconfig files sit next to it:
| Path | What it holds |
|---|---|
/etc/kubernetes/pki/ | Cluster CA, API server, front-proxy and service-account keys |
/etc/kubernetes/pki/etcd/ | etcd CA, server, peer and healthcheck certificates |
/etc/kubernetes/admin.conf | Admin kubeconfig |
/etc/kubernetes/scheduler.conf | Scheduler's kubeconfig |
/etc/kubernetes/controller-manager.conf | Controller manager's kubeconfig |
/etc/kubernetes/kubelet.conf | This node's kubelet kubeconfig |
kube-apiserver
The API server is the only component that reads and writes etcd. Everything else, including the scheduler and the controller manager, is just an API client. It listens on port 6443.
Flags worth recognising in kube-apiserver.yaml:
- --advertise-address=10.0.0.10
- --authorization-mode=Node,RBAC
- --enable-admission-plugins=NodeRestriction
- --etcd-servers=https://127.0.0.1:2379
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
Two details trip people up. etcd's client port is 2379; 2380 is the peer port that etcd members use among themselves. And the API server must trust the etcd CA (pki/etcd/ca.crt), not the cluster CA (pki/ca.crt). They are different CAs on a kubeadm cluster.
Health endpoints help when you are unsure whether it is up:
kubectl get --raw='/readyz?verbose' | tail -3
curl -k https://localhost:6443/livez
etcd
etcd stores every object as a key under /registry/.... kubeadm runs it as a static pod with its data in /var/lib/etcd on the host. It serves clients on 2379 and talks to other members on 2380, all over TLS with its own CA.
etcd needs a majority of its members to accept writes. One member is a cluster of one; three members survive one failure; five survive two. Even numbers add risk without adding tolerance.
To talk to it you pass the etcd CA and a client certificate. On a kubeadm control plane node the server certificate works for this:
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health
kube-scheduler
The scheduler watches for pods with no spec.nodeName. For each one it filters nodes (enough requests left, taints tolerated, affinity met, ports free) and then scores the survivors. It writes the winner into a binding. It listens on 10259 for health and metrics, and it reaches the API with /etc/kubernetes/scheduler.conf.
If the scheduler is down, new pods stay Pending and their events are empty. Compare that to a pod the scheduler could not place: that one has a FailedScheduling event explaining why.
kube-controller-manager
One binary, dozens of control loops: Deployment, ReplicaSet, Job, CronJob, node lifecycle, endpoints, service accounts, namespace cleanup and more. It listens on 10257. Flags worth knowing:
- --allocate-node-cidrs=true
- --cluster-cidr=192.168.0.0/16
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --use-service-account-credentials=true
If it is down, objects are stored but nothing downstream appears: a Deployment with no ReplicaSet, a Job with no pod, a deleted namespace stuck in Terminating.
The scheduler and the controller manager both run --leader-elect=true. In an HA cluster only the leader acts; the others stand by.
Debugging when kubectl is dead
If the API server is down you cannot use kubectl logs. Go one layer lower, to the container runtime and the kubelet.
# Is the container running, restarting or gone?
crictl ps -a --name kube-apiserver
# Its logs, even after it exited
crictl logs <container-id> 2>&1 | tail -20
# The same logs as files, kept by the kubelet
ls /var/log/pods/ | grep apiserver
tail -20 /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log
# Did the kubelet even accept the manifest?
journalctl -u kubelet --since "10 min ago" | grep -i apiserver | tail
A typical fix loop: read the error, fix the manifest, then wait. The kubelet notices the change within seconds, but the API server takes up to a minute to report healthy. Use watch crictl ps rather than hammering kubectl.
When a pod does not start at all and no container exists, the manifest is probably invalid YAML or has a field the kubelet rejects. The kubelet log says so; the container runtime has nothing to show.
Common mistakes
- Keeping a backup copy such as
kube-apiserver.yaml.bakinside/etc/kubernetes/manifests. The kubelet reads every non-hidden file there and may run the backup. Keep backups in/rootor another directory. - Using
kubectl editonkube-apiserver-controlplane. It edits the mirror, not the source. - Pointing
--etcd-serversat port 2380 or using the cluster CA for etcd. - Misspelt flags. The binary refuses to start with
unknown flag; the log names it exactly. - Restarting the kubelet in a panic. It is rarely needed for a manifest change and it hides the timeline in the logs.
Exam angle
Broken control plane tasks are short if you go straight to the evidence: crictl ps -a, then crictl logs, then the manifest. Do not guess at flags. The log line usually names the bad flag or file path, and the kubernetes.io reference page for the component lists every valid flag.
Practise it
Each of these labs breaks one control plane component in a different way:
Practise this on a real cluster in your account: T3, T4, T5, T6, T33.