Openshift UPI: Introduction Kubelet, the only daemon I love

I believe the Kubelet static pod deserves its headlines, considering it plays a significant role in our proof-of-concept. Kubelet is the component managing our KeepAlived, HAProxy, and CoreDNS, making sure these applications are healthy and up and running all the time. Therefore, before we start using Kubelet in our POC, let's start this blog by defining Kubelet, how it works, and Kubelet Static Pods?

What is Kubelet?

The Kubelet is a critical component of the Kubernetes system, responsible for managing and monitoring each node in the cluster. As each node in a Kubernetes cluster can be thought of as a small Kubernetes deployment in and of itself, the Kubelet plays a vital role in ensuring that each node is healthy and running its workloads as expected.

The Kubelet receives information about the state of the cluster from the Kubernetes API server and responds to any changes in that state by managing the node accordingly. For example, API server detects that a pod is deleted from the cluster, it will notify all of the nodes in the cluster. The Kubelet on each node will then remove any running containers associated with that Pod.

Kubelet is also responsible for maintaining a local cache of all of the pods in the cluster. This cache makes Kubelet quickly respond to any changes without querying the API server for each shift.

How does the Kubelet Static pod work?

The static Pod is a manifest that describes how to run the application using Kubelet daemon sets without the Kubernetes API server observing them. The manifest includes settings for liveness and readiness probes, resource limits, logging, and other options. The Kubelet uses this manifest to create containers on the node that it is running on.

The Kubelet is the primary component of any Kubernetes node and is responsible for maintaining a list of pods running on the node. Pods can either be static or dynamic. Static pods are defined in a JSON or YAML file, and Kubelet watches for changes to these files. Kubelet creates dynamic pods based on instructions from the Kubernetes API server. Kubelet manages static pods by creating and deleting them as needed, based on the contents of the pod manifest files. It also monitors the health of static pods, and if a pod dies, Kubelet will restart it. Kubelet ensures that the pods running on a node are always in the desired state.

Use cases of Static Pods

Static pods are commonly utilized by software that bootstraps Kubernetes. For example, static pods are used by Kubeadm to bring up Kubernetes control plane components such as API-Server, Controller-Manager, and Kube-Scheduler. One of the reasons it chose static Pods is that it can not generate these as standard pods since the Kube API Server is not yet ready.

When Kubeadm starts the Kubernetes control plane, it creates Pod manifests for the API-Server, Controller-Manager and Kube-Scheduler in a directory that Kubelet watches. The control plane components are then brought up as static pods by Kubelet. Kubelet may regularly sync Pod manifests; you can variously configure this setting. You can use,

• A web URL configured using --manifest-url argument to the Kubelet process. -or watch a directory on the host file system specified with the --pod-manifest-path parameter to Kubelet Process
• You can use the Kubelet config file to configure this setting using the staticPodPath yaml key.

Take a look at the following output of the Kubelet running process,

root    2508 9.1 0.5 4897580 221580 ?   Ssl 14:44  3:28 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=192.168.7.143 --address=192.168.7.143 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --hostname-override= --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4b6376a0a30fd543e43ed46f53722300e58271fe9d615329334c703cd46dbf94 --system-reserved=cpu=500m,memory=1Gi --v=2

The output showing kubelet is using config --config=/etc/kubernetes/kubelet.conf, and if you cat this file, you will see the following,

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
 x509:
  clientCAFile: /etc/kubernetes/kubelet-ca.crt
 anonymous:
  enabled: false
cgroupDriver: systemd
cgroupRoot: /
clusterDNS:
 - 172.30.0.10
clusterDomain: cluster.local
containerLogMaxSize: 50Mi
maxPods: 250
kubeAPIQPS: 50
kubeAPIBurst: 100
rotateCertificates: true
serializeImagePulls: false
staticPodPath: /etc/kubernetes/manifests
systemCgroups: /system.slice
systemReserved:
 ephemeral-storage: 1Gi
featureGates:
 APIPriorityAndFairness: true
 LegacyNodeRoleBehavior: false
 NodeDisruptionExclusion: true
 RotateKubeletServerCertificate: true
 ServiceNodeExclusion: true
 SupportPodPidsLimit: true
 DownwardAPIHugePages: true
 PodSecurity: true
 CSIMigrationAWS: false
 CSIMigrationAzureDisk: false
 CSIMigrationAzureFile: false
 CSIMigrationGCE: false
 CSIMigrationOpenStack: false
 CSIMigrationvSphere: false
serverTLSBootstrap: true
tlsMinVersion: VersionTLS12
tlsCipherSuites:
 - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
 - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

where staticPodPath is pointed to /etc/kubernetes/manifests directory. So this means any static pods manifest in this directory kubelet will keep watching for the changes and adjust accordingly.

We'll handle our Hyperconverged OCP cluster components using the same method.

Using Kubelet static Pod in Hyperconverged OCP Cluster

Now we understand Kubelet and how it works, and we have gathered all static pods manifest files for KeepAlived, HAProxy, and CoreDNS components in our previous blog post. We can start creating custom ignition config files. So the idea is to generate the default ignition config using the openshift-install tool and merge this default config using the Filetranspiler tool to create the custom-ignition config. Finally, we can upload this ignition config to some webserver and fetch that config during the installation process.

Let's see how we can achieve that in the upcoming blog.