Detect Kubernetes pods with status "terminating"

Question

kubectl get pod output shows a status field, which for the most part corresponds to the phase of the pods. It also shows pods being deleted as "terminating" which, as noted in the documentation, is not an actual phase.

Note: When a Pod is being deleted, it is shown as Terminating by some kubectl commands. This Terminating status is not one of the Pod phases. A Pod is granted a term to terminate gracefully, which defaults to 30 seconds.

We sometimes have pods stuck in the status "terminating," so I want to detect this in our monitoring. How can I determine this status from the API (I'm usingcurl for these queries), when the status is not an actual phase and thus is not shown as such in the query response?

Answer 1

When a pod is being deleted Kubernetes adds a label "deletionTimestamp" to the pod metadata. This seems to be what prompts kubectl to display a pod as "terminating."

With jq for instance you could report the status like this:

curl ... | jq -r '[.items[] | {
  "namespace": .metadata.namespace,
  "name": .metadata.name,
  "status": (if .metadata.deletionTimestamp then "Terminating" else .status.phase end)
}]'

which would produce output like this:

[
  {
    "namespace": "my-namespace",
    "name": "pod-A",
    "status": "Running"
  },
  {
    "namespace": "my-namespace",
    "name": "pod-B",
    "status": "Terminating"
  },
  ...
]