The Kubernetes CLI offers a powerful command to monitor and react to changes in your cluster: the kubectl wait command.
This command enables you to block execution (i.e. wait) until a specific condition is met, such as:
- a specified resource is deleted ;
- a specified resource transitions to a specific state
Waiting for resource deletion: kubectl wait --for=delete
In this case, you will use the --for=delete option as follows, for example on a pod :
$ kubectl wait pod/rabbitmq-7575b7f589-dsdhl --for=delete --timeout=-1s
pod/rabbitmq-7575b7f589-dsdhl condition met
This option works with all Kubernetes resource types.
Please note: the default timeout is 30 seconds. Here we use the value “-1s” which will be interpreted as “the maximum timeout”, which happens to be 1 week. (It is not possible to fully deactivate the timeout.)
Waiting for a condition: kubectl wait --for=condition
Some Kubernetes resources can have “conditions” that describe their status. The kubectl wait command allows you to wait until a specific condition is met.
As of Kubernetes 1.28, the following resources can have conditions:
- API Services
- Custom Resource Definitions (CRDs)
- DaemonSets
- Deployments
- Jobs
- Nodes
- Persistent Volume Claims
- Pods
- ReplicaSets
- Replication Controllers
- StatefulSets
- Namespaces
- Persistent Volumes
- Services
- HorizontalPodAutoscalers
Please note that while the conditions field is defined in the status of these resources, its implementation and usage can vary. For example, not all resources actively populate or update their conditions field.
The exact conditions depend on the resource type. For example, pods have the conditions Initialized, Ready, ContainersReady, PodScheduled and Unschedulable while deployments have the conditions Available and Progressing.
The following example lists the current condition status for a pod:
$ kubectl get pod/rabbitmq-7575b7f589-dsdhl -o "go-template={{range .status.conditions}}{{printf \"%s = %s\n\" .type .status}}{{end}}"
Initialized = True
Ready = True
ContainersReady = True
PodScheduled = True
Waiting for a pod to be ready: kubectl wait --for=condition=Ready
To wait for your pod to be ready, you will use the following kubectl wait command:
$ kubectl -n nxs-r1-prod wait pod/rabbitmq-7575b7f589-dsdhl --for=condition=Ready --timeout=-1s
pod/rabbitmq-7575b7f589-dsdhl condition met
Handling Services Without Conditions
As you can see in the list above, services unfortunately do not have a condition, but it could be very useful to check that at least one endpoint exists for the service in question (and thus that at least one pod is ready to serve user requests thanks to health checks).
It is possible to monitor the associated pods (or better, deployments) but a race condition exists between the moment the pod is declared ready for use and the moment the Endpoints Controller of the Controller Manager actually creates the associated endpoint.
One solution to this problem is to actively poll the Kubernetes API to monitor the creation of an endpoint using the following bash one-liner:
until [[ $(kubectl get endpoints/rabbitmq -o=jsonpath='{.subsets[*].addresses[*].ip}') ]]; do sleep 5; done
An alternative solution would be to poll the target service directly and wait for it to be available, the command to use in this case will depend on the service used.
Using kubectl wait for Scheduling with InitContainers
Kubernetes Pods can execute a number of initContainers. These are executed “synchronously” and in the order of their definition in the pod specification, before the execution of the “classic” containers. This allows for example to initialize a database, to prepare a volume, or to render configuration templates (see Konfplate, one of our projects on Github).
Associated with initContainers, the wait command allows you to block the execution of certain pods while waiting for some conditions to be met (for example, the completion of a Job) and thus to basically schedule the execution of resources on its Kubernetes cluster.
Conclusion
kubectl wait is a valuable command for managing Kubernetes resource states efficiently. Whether waiting for pod readiness, resource deletion, or ensuring that services have available endpoints.
Before you leave, we invite you to read this article to discover more special and advanced kubectl commands.
Do not miss our latest DevOps and Cloud Native blogposts! Follow Enix on Linkedin!