The Kubernetes CLI offers a very useful command allowing you to wait for events on your cluster: the kubectl wait command.
This command allows you to block (i.e, wait) until a particular event happens, for instance:
- a specified resource is deleted
- a specified resource transitions to a specific state
Waiting for resource deletion: kubectl wait --for=delete
In this case, you will use the --for=delete option as follows, for example on a pod :
$ kubectl wait pod/rabbitmq-7575b7f589-dsdhl --for=delete --timeout=-1s
pod/rabbitmq-7575b7f589-dsdhl condition met
This option can be used on all kinds of Kubernetes resources.
Please note: the default timeout is 30s (30 seconds). Here we use the value “-1s” which will be interpreted as “the maximum timeout”, which happens to be 1 week. (It is not possible to fully deactivate the timeout.)
Waiting for a condition: kubectl wait --for=condition
Some Kubernetes resources can have “conditions”. In Kubernetes 1.14, the following resources could have conditions:
- api services
- custom ressource definitions
- daemon sets
- deployments
- jobs
- nodes
- persistent volume claims
- pods
- replica sets
- replication controllers
- stateful sets
The exact conditions depend on the resource type. For example, pods have the conditions Initialized, Ready, ContainersReady, PodScheduled and Unschedulable while deployments have the conditions Available and Progressing.
The following example lists the current condition status for a pod:
$ kubectl get pod/rabbitmq-7575b7f589-dsdhl -o "go-template={{range .status.conditions}}{{printf \"%s = %s\n\" .type .status}}{{end}}"
Initialized = True
Ready = True
ContainersReady = True
PodScheduled = True
Waiting for a pod to be ready: kubectl wait --for=condition=Ready
To wait for your pod to be ready, you will use the following kubectl wait command:
$ kubectl -n nxs-r1-prod wait pod/rabbitmq-7575b7f589-dsdhl --for=condition=Ready --timeout=-1s
pod/rabbitmq-7575b7f589-dsdhl condition met
The case of services
As you can see in the list above, services unfortunately do not have a condition, but it could be very useful to check that at least one endpoint exists for the service in question (and thus that at least one pod is ready to serve user requests thanks to health checks).
It is possible to monitor the associated pods (or better, deployments) but a race condition exists between the moment the pod is declared ready for use and the moment the Endpoints Controller of the Controller Manager actually creates the associated endpoint.
One solution to this problem is to actively poll the Kubernetes API to monitor the creation of an endpoint using the following bash one-liner:
until [[ $(kubectl get endpoints/rabbitmq -o=jsonpath='{.subsets[*].addresses[*].ip}') ]]; do sleep 5; done
An alternative solution would be to poll the target service directly and wait for it to be available, the command to use in this case will depend on the service used.
Scheduling with initContainers
Kubernetes Pods can execute a number of initContainers. These are executed “synchronously” and in the order of their definition in the pod specification, before the execution of the “classic” containers. This allows for example to initialize a database, to prepare a volume, or to render configuration templates (see Konfplate, one of our projects on Github).
Associated with initContainers, the wait command allows you to block the execution of certain pods while waiting for some conditions to be met (for example, the completion of a Job) and thus to basically schedule the execution of resources on its Kubernetes cluster.
You can also read this article if you want to know more about other advanced kubectl commands.
Do not miss our latest DevOps and Cloud Native blogposts! Follow Enix on Twitter!