> ## Documentation Index > Fetch the complete documentation index at: https://docs.karchunt.com/llms.txt > Use this file to discover all available pages before exploring further. # Control Plane Components ## kube-apiserver It is designed to scale horizontally, meaning you can deploy and run multiple instances of kube-apiserver to balance the traffic between them, ensuring high availability and reliability of the Kubernetes API. It is the **front-end** of the Kubernetes control plane that **exposes** the Kubernetes HTTP API. It's the **entry point** for all the REST commands used to **manage (orchestrate) the cluster operations**. Remember, **kube-apiserver** is the **only component** that **interacts** with the **etcd database**, and it serves as the **primary gateway** for all other components like (kube-scheduler, controller-manager, kubelet, kube-proxy) to **interact** with the **cluster data**. You can download the kube-apiserver binary from the [kube-apiserver releases](https://kubernetes.io/releases/download/) and **run it manually on your control-plane node**. However, this method is **not recommended for production** environments as it requires manual configuration and management of the kube-apiserver process. If you're using **kubeadm** to setup your Kubernetes cluster, the **kube-apiserver** will be **automatically deployed as a static pod** on the control-plane node. You can use the `kubectl get pods -n kube-system` command to find the **kube-apiserver pod**. The kube-apiserver will be running as a static pod, which means it will be **managed by the kubelet** and will **automatically restart** if it crashes. ```bash icon="terminal" title="Guide to check kube-apiserver status" theme={"theme":"github-dark-dimmed"} # Check the kube-apiserver pod kubectl get pods -n kube-system # Check the kube-apiserver pod config options cat /etc/kubernetes/manifests/kube-apiserver.yaml # Check the kube-apiserver running process ps -aux | grep kube-apiserver # Check the kube-apiserver service cat /etc/systemd/system/kube-apiserver.service ``` ### Process flow of getting data from the cluster ```mermaid theme={"theme":"github-dark-dimmed"} flowchart LR user["User (kubectl get)"] --"HTTPS REST Request"--> api["kube-apiserver"] api --"Query State"--> etcd[("ETCD Cluster")] etcd --"Return Data"--> api api --"Formatted Output"--> user ``` You can interact with kube-apiserver by calling the Kubernetes API directly as well. From the above diagram, we can see that **all the requests** `kubectl` from the user will **first go to** the **kube-apiserver**. 1. The user sends a request to the **kube-apiserver** using `kubectl` or any other client. 2. The request is received by the `kube-apiserver`, which will **authenticate and validate** the request. 3. The `kube-apiserver` will then **query** the `etcd` cluster to **retrieve the current state** of the cluster or to **update the state** based on the request. 4. The `etcd` cluster will **return the requested data** or **acknowledge** the update to the `kube-apiserver`. 5. Finally, the `kube-apiserver` will **format** the **response** and **send it back** to the user. ### Process flow of creating a new pod ```mermaid theme={"theme":"github-dark-dimmed"} graph TD 1[1 User/kubectl Apply Manifest/API] --> 2(2 kube-apiserver) 2 -->|3 Store State| 3[(3 etcd)] 2 -->|4 Watch Event| 4(4 Scheduler) 4 -->|5 Filter & Score Nodes| 4 4 -->|6 Bind Pod to Node| 2 2 -->|7 Watch Event| 5(7 Kubelet on Node) 5 -->|8 Call| 6(8 Container Runtime) 6 -->|9 Pull Image| 7(9 Image Registry) 6 -->|10 Create & Start| 8(10 Container) 5 -->|11 Update Status| 2 2 -->|12 Update State| 3 ``` All these steps are very similar when a change happens in the cluster. The kube-apiserver will always be the central point of communication between all the components in the cluster. From the above diagram, let's assume that the user wants to create a new pod using **API**. ```bash title="Create Pod via API" icon="terminal" theme={"theme":"github-dark-dimmed"} # Reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/ curl -X POST /api/v1/namespace/default/pods ``` 1. The user sends a request to the **kube-apiserver** to create a new pod. 2. The request is received by the `kube-apiserver`, which will **authenticate and validate** the request. 3. The `kube-apiserver` wilL then **store the state** of the new pod (Create a Pod Object without the node assign) in the `etcd` cluster. 4. The `kube-apiserver` will **watch for events** related to the new pod and **notify the scheduler**. 5. The `scheduler` will **filter and score the nodes** to determine the best node for the new pod. 6. The `scheduler` will **bind the pod to the selected node** and **update the state** in the `etcd` cluster. 7. The `kubelet` on the selected node will **watch for events** related to the new pod and **call the container runtime** to create the container. 8. The `container runtime` will **pull the necessary image** from the image registry. 9. The `container runtime` will **create and start the container** on the node. 10. The `kubelet` will **update the status** of the pod back to the `kube-apiserver`. 11. The `kube-apiserver` will **update the state** of the pod in the `etcd` cluster. ## etcd You can refer to the [etcd documentation](https://etcd.io/) for more details about etcd, including its architecture, features, and how to use it effectively in a Kubernetes environment. It's a **distributed key-value store** that is used to **store all the cluster data**, including the state of the cluster, configuration, date, and metadata. The `kube-apiserver` interacts with `etcd` to **read and write data** about the **cluster's state**, making it an essential part of the Kubernetes architecture. All information you see when you run the `kubectl get` command is from the **ETCD server**. Remember all changes made to the cluster like adding additional nodes, deploying pods, etc, will be **updated** in the **ETCD server**. You can refer to the [etcd releases](https://github.com/etcd-io/etcd/releases) to download the etcd binary and follow the [etcd installation instructions](https://etcd.io/docs/) to set up an etcd server on your control-plane node. However, this method is suitable for learning and testing purposes but is **not recommended for production environments** due to the complexity of managing and maintaining the `etcd` cluster manually. There is one important configuration option to note when setting up `etcd` manually: `--advertise-client-urls 'http://{IPADDRESS}:2379'`. This option specifies the address that `etcd` listens on for client requests. The default port for `etcd` is `2379`. When configuring the `kube-apiserver`, you need to ensure that it is set to connect to this URL, as the `kube-apiserver` will use this URL to communicate with the `etcd` server. If you're using **kubeadm** to set up your Kubernetes cluster, the `etcd` server will be **automatically deployed as a static pod** on the control-plane node. You can use the `kubectl get pods -n kube-system` command to find the `etcd` pod. The `etcd` server will be running as a static pod, which means it will be **managed by the kubelet** and will **automatically restart** if it crashes. ```bash icon="terminal" title="Guide to check etcd status" theme={"theme":"github-dark-dimmed"} # Check the etcd pod kubectl get pods -n kube-system # Check the etcd pod config options cat /etc/kubernetes/manifests/etcd.yaml # Check the etcd running process ps -aux | grep etcd # Check the etcd service cat /etc/systemd/system/etcd.service ``` You can also use the `etcdctl` command-line tool to interact with the `etcd` server directly. For example, you can run the following command to get all keys stored by Kubernetes in `etcd`: ```bash icon="terminal" title="Get all keys from etcd" theme={"theme":"github-dark-dimmed"} kubectl exec etcd-controlplane -n kube-system -- etcdctl get / --prefix --keys-only ``` You will notice that the root directory is the **registry**, and below that are various Kubernetes objects like nodes, pods, deployments, etc, as it stores data in a specific directory structure. ## kube-controller manager Ideally, each controller should run in its own process, but to reduce complexity, they're all compiled into a single binary and run in a single process. This design choice simplifies the deployment and management of the controllers while still allowing for scalability and reliability. It is responsible for **managing various controllers** that are responsible for **ensuring the desired state of the cluster**. Each controller has different functions to take care of its side, such as: * **Node Controller** - It is responsible for **monitoring the nodes** and **taking action** when a node goes down or becomes unresponsive. * **Replication Controller** - It is responsible for **ensuring that the desired number of pod replicas** are running at any given time. * **Endpoints Controller** - It is responsible for **managing the endpoints** that are used to connect services to pods. * **Service Account & Token Controllers** - They are responsible for **managing service accounts and tokens** that are used for authentication and authorization in the cluster. ```mermaid theme={"theme":"github-dark-dimmed"} graph TD subgraph A[kube-controller manager] B[Node controller] C[Replication controller] D[Endpoints controller] E[More controllers...] end ``` The controller is a **process** that is responsible for **monitoring the state** of the various components and **resolving situations** when the **actual state does not match the desired state**. You can download the kube-controller-manager binary from the [kube-controller-manager releases](https://kubernetes.io/releases/download/) and **run it manually on your control-plane node**. However, this method is **not recommended for production** environments as it requires manual configuration and management of the kube-controller-manager process. If you're using **kubeadm** to setup your Kubernetes cluster, the **kube-controller-manager** will be **automatically deployed as a static pod** on the control-plane node. You can use the `kubectl get pods -n kube-system` command to find the **kube-controller-manager pod**. The kube-controller-manager will be running as a static pod, which means it will be **managed by the kubelet** and will **automatically restart** if it crashes. ```bash icon="terminal" title="Guide to check kube-controller-manager status" theme={"theme":"github-dark-dimmed"} # Check the kube-controller-manager pod kubectl get pods -n kube-system # Check the kube-controller-manager pod config options cat /etc/kubernetes/manifests/kube-controller-manager.yaml # Check the kube-controller-manager running process ps -aux | grep kube-controller-manager # Check the kube-controller-manager service cat /etc/systemd/system/kube-controller-manager.service ``` ### Node Controller ```mermaid theme={"theme":"github-dark-dimmed"} flowchart RL subgraph "Master Node (Control Plane)" node-controller --> controller-manager controller-manager --> kube-apiserver end node1[Worker Node 1] --> kube-apiserver node2[Worker Node 2] --> kube-apiserver node3[Worker Node 3] --> kube-apiserver ``` The **node controller** is responsible for **monitoring the nodes** in the cluster and **taking action** when a node goes down or becomes unresponsive. It does this by **watching for events** related to the nodes and **updating the status** of the nodes in the `etcd` cluster. ```bash theme={"theme":"github-dark-dimmed"} kcserver@kcserver:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kcserver Ready control-plane 40m v1.34.6+k3s1 ``` Nodes are **tested every 5 seconds** to ensure the node is **healthy**. If a node **fails to respond (not receiving heartbeats)** within **40 seconds**, the node controller will mark the node as **"NotReady"**. The node controller gives the node **5 minutes to recover** before it is marked as **"Unreachable"**. If the node remains in the "Unreachable" state for **more than 5 minutes**, the node controller will **remove all the pods from that node** and will be **provisioned those pods** to **other healthy nodes** in the cluster as long as the pod is **part of a deployment** or a **replica set**. ### Replication Controller ```mermaid theme={"theme":"github-dark-dimmed"} flowchart RL subgraph "Master Node (Control Plane)" replication-controller --> controller-manager controller-manager --> kube-apiserver end node1[Worker Node 1] --> kube-apiserver node2[Worker Node 2] --> kube-apiserver node3[Worker Node 3] --> kube-apiserver ``` The **replication controller** is responsible for **ensuring that the desired number of pod replicas** are running at any given time. It does this by **watching for events** related to the pods and **updating the status** of the pods in the `etcd` cluster. If the replication controller detects that a pod is not running or has been deleted, it will **create a new pod** to replace it, ensuring that the desired number of replicas is maintained. ## kube-scheduler Remember, **kubelet** is the one who will **place and create the pod** on the **node**. It is responsible for **identifying and scheduling** the pods on nodes. It **only decides** which pod goes to which node, but it does not actually deploy the pod on the node, that is the responsibility of the kubelet. There are some factors that it will take into account when making scheduling decisions, such as: * **Resource Requirements** - CPU and memory requirements of the pod * **Constraints** - Hardware, software, and policy constraints * **Affinity and Anti-affinity** - Rules about which pods should be co-located or separated based on node or other pod labels * **Data Locality** - Scheduling pods close to the data they need to access * **Inter-workload Interference** - Avoiding scheduling pods that may interfere with each other on the same node * **Deadlines** - Scheduling pods based on their deadlines and priorities The reason why we need a scheduler is because there could be multiple nodes in the cluster, and we need a way to determine which node is the best fit based on the pod requirements. For example, if a pod requires a certain amount of CPU and memory, the scheduler will need to analyze the available resources on each node to determine which node can accommodate the pod. You can download the kube-scheduler binary from the [kube-scheduler releases](https://kubernetes.io/releases/download/) and **run it manually on your control-plane node**. However, this method is **not recommended for production** environments as it requires manual configuration and management of the kube-scheduler process. If you're using **kubeadm** to setup your Kubernetes cluster, the **kube-scheduler** will be **automatically deployed as a static pod** on the control-plane node. You can use the `kubectl get pods -n kube-system` command to find the **kube-scheduler pod**. The kube-scheduler will be running as a static pod, which means it will be **managed by the kubelet** and will **automatically restart** if it crashes. ```bash icon="terminal" title="Guide to check kube-scheduler status" theme={"theme":"github-dark-dimmed"} # Check the kube-scheduler pod kubectl get pods -n kube-system # Check the kube-scheduler pod config options cat /etc/kubernetes/manifests/kube-scheduler.yaml # Check the kube-scheduler running process ps -aux | grep kube-scheduler # Check the kube-scheduler service cat /etc/systemd/system/kube-scheduler.service ``` ### Nodes Ranking ```mermaid theme={"theme":"github-dark-dimmed"} --- title: kube-scheduler ranking --- flowchart LR pod["Pod (CPU: 8)"] subgraph "A List of nodes" node1["Node 1 (CPU: 4)"] node2["Node 2 (CPU: 10)"] node3["Node 3 (CPU: 20)"] end pod ---> node3 ``` To determine which node is the best fit for a pod, it will rank the nodes. Here is an example, currently we have one pod with **CPU requirements of 10**. The kube scheduler will be going through **2 phases** to identify and schedule the pod on the **best node**. 1. The kube scheduler will **filter out** those **nodes** that **do not fit the requirements**. So in this case, node 1 will be filtered out as node 1 only has 4 CPUs. 2. (**Rank nodes**) By using a **priority function** or class, the kube scheduler **assigns a score** and **calculates** how much **free space** is available on the nodes after the pod is placed. The **highest score** after calculation will place the pod on that node. * Assuming the priority score is **5** * Score on node 2 = `10 - 5 = 5` * Score on node 3 = `20 - 5 = 15` (Win)