journalctl to debug Kubernetes

The command journalctl -x -f is used to view the logs of all units in the system, with the -x option providing additional output that includes the unit's process ID and exit code. The -f option is used to "follow" the logs, so that new log entries will be displayed as they are generated.

This command is useful when debugging system-wide issues or when trying to track down the source of an error. For example, if you are experiencing a problem with a service that is failing to start, you can use this command to view the logs for that service and see if there are any error messages or other clues as to what might be causing the problem.

It's important to note that, the amount of logs that this command will show will be huge, you may want to filter the logs by using grep command or by using other options like -u to filter by unit,-p to filter by priority and so on.

Here are a few examples of using journalctl to debug Kubernetes issues:

Please keep in mind that these examples are written for a system that runs systemd as the service manager. If you are using other service manager, the command may different.

• To view the logs for all pods in a specific namespace:

journalctl -u kubelet

• To view the logs for a specific pod in a namespace:

journalctl -u kubelet -n kube-system -f | grep <pod-name>

• To view the logs for a specific container in a pod:

journalctl -u kubelet -n kube-system -f | grep <container-name>

• To view the logs for a specific node:

journalctl -u kubelet -n kube-system -f | grep <node-name>

• To see the logs of kube-apiserver:

journalctl -u kube-apiserver

• To see the logs of kube-controller-manager:

journalctl -u kube-controller-manager

• To see the logs of kube-scheduler:

journalctl -u kube-scheduler

Failed to add match 'kube-system': Invalid argument

The error message "Failed to add match 'kube-system': Invalid argument" suggests that the -n option is not supported by your version of journalctl or the service manager you are using.

journalctl uses systemd as the service manager, if your system is not running on systemd, the -n option will not work.

Instead of filtering by namespace, you can filter by the kubelet service, which should return all the logs related to Kubernetes.

Here's an example command that will show all logs related to kubelet :

journalctl -u kubelet -f
journalctl -xeu kubelet -f

You can also filter the logs by pod,node,container name etc. using grep command.

example :

journalctl -u kubelet -f | grep <pod-name>
journalctl -xeu kubelet -f | grep kube-apiserver

Some other debug command I used frequently (mostly OpenShift)

journalctl -b -f -u kubelet.service -u crio.service

The -b option specifies that the command should show logs from the current and previous boot, which can be useful when trying to troubleshoot issues that occurred during a recent reboot of the system. The -f option is used to "follow" the logs, so that new log entries will be displayed as they are generated. The -u option is used to filter the logs by unit, which in this case is kubelet.service and crio.service.

To collect the bootkube.service service logs from the host during bootstrapping, execute the following journalctl command:

journalctl -b -f -u bootkube.service