We have all seen these ugly security alerts caused by expired TLS certificates.
In this blog post, we will see a “belt and suspenders” approach to avoid this situation, by renewing them automatically and detecting certificates that might expire (before they do!). In pure Cloud Native fashion, our solution will even run entirely on Kubernetes!
X.509 certificates, what are they?
X.509 certificates are used to ensure the security of various actions in our Cloud Native world, and are widely used to guarantee the security of exchanges via HTTPS, so the APIs we all use also depend on them.
If you want to dig deeper into the subject, to know in what context to use a certificate of this type, I invite you to look at what a PKI can offer. Today, I will focus on its use in the context of a Kubernetes cluster.
X.509 certificates mainly contain credentials and use a signature based on RSA or ECDSA encryption to guarantee their authenticity and integrity.
To perfect the concept, certificates should not remain indefinitely valid. Each of them has a validity period, which once exceeded makes the certificate completely obsolete. Here you can see the problem coming: the expiration of a certificate can put any application, any service in jeopardy. They must therefore be carefully monitored and renewed as needed, or else you’ll have a very bad time…
The security of a certificate depends on the security of the organization that signs it. Historically, certificates were issued by well-known and trusted organizations (Verisign, DigiCert, …) and companies charged for the signature according to the certificate’s validity period and the expected level of verification.
Not so long ago, obtaining a certificate could involve manual cross-checks like faxing some company papers or verifying the company representative ID documents. In our Cloud Native era, the demand for certificates has skyrocketed and these manual mechanisms have indeed become obsolete. To address this problem, Let’s Encrypt was born: a platform allowing you to obtain certificates automatically and free of charge, and without compromising on security; nothing less!.
The result is short-lived certificates… to be renewed… very often!
Renewing your certificates very often
With Let’s Encrypt, signed certificates are only valid for 3 months. Certainly, there has to be a way to automate this for Kubernetes lovers like me and you, right?
Look no further, Cert Manager is here for you. It completely automates the creation of certificates and renews them as needed.
All you have to do is add some information in your Ingress object and roll on…
In addition to being used for applications running on the cluster, X.509 certificates are also used internally by Kubernetes, in order to authenticate communications with the API server or between its various components. These certificates are found all over the cluster: some on the main nodes (called master, or control plane), some on all nodes and others directly in the Kubernetes database (in the form of Secret).
In addition to not being centralized, these certificates are not all renewed automatically, depending on the application that manages them. And in the case of Kubernetes, it depends on how it was installed:
- A cluster created with
kubeadmdoes not renew any certificates by default. Therefore, after the initial validity period (varying from 6 months to one year), the cluster will no longer be able to function without human intervention. So you have to update them regularly! - If you use a Kubernetes distribution like Rancher or OpenShift, everything is done for you automatically.
In short, it all depends on the phases of the moon! A non-automated task or a badly configured automation and it’s guaranteed unavailability!
So let me sum up: in order to guarantee the availability of an infrastructure, it is essential to make sure that none of its certificates are close to expiring!
Monitoring certificate renewal in a Kubernetes infrastructure
Any Cloud Native infrastructure worthy of the name will collect some kind of metrics, one way or another. A very popular option to collect and expose metrics is Prometheus. Together with its additional component alert-manager, it can detect when some metrics cross specific thresholds and take automated action when that happens. If only we could expose to Prometheus the information of our cluster certificates, we would then be able to trigger an alert when one of them is close to expire.
This is where I say tadaaaa!!! : the X.509 certificate exporter is your open-source solution in Go to deal with this. Well ok I’m a big contributor so not quite impartial 😉 So I’ll let you judge for yourself!
It allows information about certificates to be exposed to Prometheus :
- Certificates present in files on a node: the user can fill in the paths to the files that contain one or more certificates in PEM format.So with the proper configuration, it is possible to monitor any system based on certificates in files.
- Certificates embedded in YAML configuration files: the tool has been designed for monitoring Kubernetes clusters, which use base64 embedded certificates in YAML configuration files. It also happens that these YAML files contain a path to a file in PEM format: the exporter takes care of everything.
- Certificates stored as a Kubernetes Secret: this includes, among other things, all application certificates (present and future) managed by Cert Manager, and also all TLS type S_ecrets._
In addition to their expiration, the exporter collects the information contained in the distinguished name of the certificate (e.g. the domain name in the case of an HTTPS certificate), allowing advanced filtering on the metrics, which can be displayed on a Grafana dashboard, in order to visualize the status of all certificates at a glance.
However, the number one feature remains to set up alerts in order to be notified in advance of an unexpected certificate expiration!
How to install x.509 Certificate Exporter?
The installation can be done using two commands via Helm.
# add our charts repository
helm repo add enix https://charts.enix.io
# install for TLS Secrets monitoring with prometheus-operator support
helm install x509-certificate-exporter enix/x509-certificate-exporter
By default only Secrets are supervised, but the Helm chart is configurable, and if compiling Go doesn’t scare you, the source code is available here.
This project is already used in production on several clusters of all types, notably on our Enix infrastructures and those of our customers and partners.
Your feedback will always be appreciated and listened to. Do come and tell us your ideas for improvement in the project outcomes!
If you like this software, your star on GitHub and Artifact Hub is welcome too 😉
And by the way… We’re also working on other Cloud Native projects here at Enix.
Do not miss our latest DevOps and Cloud Native blogposts! Follow Enix on Twitter!