Introduction
The ability to view the history of deleted pods in a Kubernetes (K8s) cluster is vital for debugging and auditing. When a pod is deleted, whether intentionally or due to a system event, its historical data can provide valuable insights into the reasons for its termination and the state of the cluster at that time. In this tutorial, we will explore several methods to view the history of deleted pods in Kubernetes, ranging from basic commands to more advanced techniques.
Understanding Pod Deletion
In Kubernetes, when a pod is deleted, it is removed from the API server and is no longer visible using the standard kubectl get pods command. However, the events and logs related to the pod may still be available for some time. Understanding where and how this information is stored is key to retrieving a deleted pod’s history.
Kubernetes Events
An important source of information regarding pod deletion can be found in Kubernetes events. To view these events, use the following command:
kubectl get events --sort-by='.lastTimestamp'
This will show you a list of all cluster events, sorted by their last timestamp. Look for events related to the deletion of pods.
Output example:
LAST SEEN TYPE REASON OBJECT MESSAGE
1m Normal SuccessfulDelete pod/my-pod Deleted pod: my-pod
Auditing
Kubernetes auditing is another powerful mechanism that logs all API calls, including those to delete pods. Ensure that the audit log feature is enabled and configured in your cluster to leverage this functionality.
Kubectl Get Pods
To get started, first see if any information about the deleted pod is still available via the kubectl get pods command by including the --include-uninitialized flag:
kubectl get pods --include-uninitialized -n
Using Kubectl to Retrieve Deleted Pod Information
One way to potentially find information about a deleted pod is by using kubectl to describe nodes and look for information about evicted or otherwise terminated pods:
kubectl describe node
You might find references to the pod in the node events or in the description of what the node is currently running or has recently terminated.
Exploring Pod Logs
If the pod was writing logs to stdout/stderr, and the container runtime is configured to retain these logs for a period after the container is deleted, you can attempt to retrieve the logs with:
kubectl logs --previous -n
Note that this only works if the pod crashed and restarted at least once and if the logs haven’t yet been garbage-collected.
Advanced Techniques
For more sophisticated analysis, you can use logging and monitoring systems like Elastic Stack (ELK), Prometheus, and Grafana to observe the state and performance of a pod before its deletion. These tools collect and store data over time, which could include information about pod deletions.
Another advanced technique involves querying the etcd database. etcd is a distributed key-value store used by Kubernetes to store all cluster data. If you have direct access to etcd, you can find data about deleted pods:
ETCDCTL_API=3 etcdctl get / --prefix --keys-only
However, this method requires deep knowledge of the etcd structure and careful attention not to modify the etcd contents.
Using a K8s Audit Log
If your cluster is configured with audit logging, you can search through the audit log files to find information about deleted pods. Audit logs are stored on the Kubernetes API server or sent to a remote logging service depending on the configuration. Use UNIX-like commands or your logging service’s search capabilities to search for deletion events:
grep 'delete' /var/log/kubernetes/audit.log
Modify the path and command to match your environment and logging configuration.
Kubernetes Dashboard
If you’re using the Kubernetes Dashboard, you can browse through the event logs for any traces of the deleted pod. Look for the namespace of the deleted pod and search for relevant events by timestamp and pod name.
See also: Working with Kubernetes Dashboard: A Practical Guide (with examples).
Customizing Event Retention
By default, Kubernetes events are only stored for 1 hour. For long-term access to this data, consider altering the event TTL (Time To Live) with a Kubernetes controller or utilizing a third-party tool for event collection.
Conclusion
As we’ve seen, while deleted pods disappear from a Kubernetes cluster’s immediate view, there are multiple ways to uncover their history. From using kubectl commands to deploying advanced logging and monitoring solutions, each approach has its use cases depending on the data’s detail and retention requirements.
