Kubernetes Ingress Controller : migration handbook

In this article, we’ll present various techniques that can be used to migrate from one existing Kubernetes Ingress Controller to another one.

First things first: what is an ingress controller ?

Ingress is a popular Kubernetes API resource to route incoming HTTP (and HTTPS) requests to their corresponding pods. Developers and administrators can create ingress resources, which define routing rules. For instance, “requests to www.example.com/api are load balanced across pods associated with service XYZ.” These resources rely on one or more ingress controllers to implement the actual traffic handling, routing, and load balancing.

An ingress controller is essentially an HTTP load balancer, configured via the Kubernetes API. Whenever you send a request to a domain handled by an ingress controller, that ingress controller will receive the request, parse it, figure out its destination (i.e. which pod or pods should handle it), route it to the appropriate backend, and send the response back to you.

Note that Kubernetes users and docs will sometimes talk about “an ingress”, and depending on the context, this could designate an ingress resource (a set of rules indicating how to route HTTP traffic) or an ingress controller (the load balancer responsible for handling that HTTP traffic according to those rules).

Why migrate from one ingress controller to another?

Unfortunately, the original ingress API is relatively limited. It specifies how to route requests based on hostname and URL path, but in the modern web world, this may not be enough.

For example, we often need to specify an HTTP redirection to ensure that users access our sites and apps over a TLS secure connection (starting with the https:// URL scheme). We may also want to enable HSTS to improve security, or implement canary deployment/blue-green deployment, or facilitate authentication via Oauth2.

Since these features are fairly common requirements, various ingress controllers have implemented them; but since these features were not part of the original ingress specification, the level of implementation varies from one ingress controller to the next. This means that while pretty much every ingress controller implements the ingress specification, they all provide various extensions, and when these extensions are critical to our applications’ availability or performance, the choice of the ingress controller becomes a cornerstone of our Kubernetes cluster solution.

Some design decisions can have long-term consequences, and changing them can require complex and costly migrations. However, as we will see in the rest of this article, the choice of an ingress controller doesn’t have to be definitive, and there are solutions to migrate from one controller to another if the first choice turns out to be suboptimal or if our requirements have evolved. If you’re in the situation of making or challenging the choice of an ingress controller this spreadsheet about Kubernetes Ingress Controllers Comparison could be useful.

A typology of ingress controllers

Before talking about migration scenarios, we can arbitrarily classify ingress controllers into the following categories.

Ingress Controllers Migration Scenarios

Now that we’ve described the main ways that ingress controllers can be deployed on Kubernetes, let’s talk about migration scenarios - because these scenarios will mostly depend on how our ingress controllers are deployed.

Those scenarios are generic guides that need to be treated with care. Always try your migration steps first in a staging environment before proceeding with a production environment.

A) DNS switchover

In this scenario, both ingress controllers have their own external IP address, which means that the migration is done mainly by updating DNS records.

Scenario applicability

Note: there is no “Cloud Ingress to Cloud Ingress” scenario in the table above, as this is considered to be an extremely rare case (where a cloud provider would have at least two distinct ingress controllers). If you are in that scenario, the cloud provider will have a dedicated migration guide which you should then follow.

Expected Downtime

No downtime

Description

To avoid downtime the best option is to have both controllers running in parallel, check that everything works fine with the new ingress, and finally switch the DNS one ingress at a time to minimize risk.

Ingress controllers normally rely on the “ingress class” of an ingress resource to determine whether they should handle that resource or not. When migrating from one ingress controller to another, you can either create duplicate ingress resources (for both ingress controllers), or configure at least one of the ingress controllers to handle all ingress resources, ignoring their ingress class.

Note that the second option can lead to conflicts between the two ingress controllers, as they might “fight” to update the “status.loadbalancer” field of the ingress resources to reflect the external IP address of their load balancer. While this should be harmless in itself, it might trigger issues if you use a tool that relies on that field (for instance, ExternalDNS).

Since managing a given ingress resource with two ingress controllers simultaneously isn’t explicitly supported in Kubernetes, if you go that route, you should test it extensively in your staging environment and make sure that it doesn’t trigger interesting bugs in one of the ingress controllers.

And if these tests surface some issues (for instance, if the two controllers keep fighting over the same ingress object), go back to the initial plan and keep an ingress class for each controller, and switch when you’re ready.

When updating DNS records, it is advised to reduce their TTL to a lower value (e.g. between 30 and 300 seconds) some time before the migration to mitigate the impact of stale DNS records still pointing to an old (and potentially invalid!) IP address; and restore the TTL to its initial value when the migration is complete.

Examples

If you deploy NGINX Ingress with its Helm chart, you can add the following value:

--set controller.watchIngressWithoutClass=true

If you deploy HAProxy Kubernetes Ingress Controller with its Helm chart, the corresponding value is:

--set "controller.extraArgs[0]=--empty-ingress-class”

B) LoadBalancer IP address reuse

Scenario applicability

Expected Downtime

As low as a few seconds (depends on the cloud controller manager)

Description

In this scenario, conceptually, we reassign the public IP address of one LoadBalancer service to another. This is done by setting the field “spec.loadBalancerIP” in the new service.

There are a few important details to be aware of, if you want to go that route.

First, this feature is only supported on some cloud controller managers; so you should check the relevant documentation to know if it’s available in your specific cloud environment.

Next, it might require “locking” the corresponding IP address so that it doesn’t get deallocated and returned to the cloud provider’s pool.

Finally, the “spec.loadBalancerIP” field has been deprecated in Kubernetes 1.24 in favor of a vendor-specific annotation. Some providers still implement the field, and others require switching to the annotation instead. Again, you’ll need to check the relevant documentation (for instance for Google Cloud, or for Azure).

Here is what the different steps look like for this migration scenario:

1. deploy the new ingress controller with a LoadBalancer Service
2. in that new LoadBalancer Service, set “spec.loadBalancerIP” (or corresponding annotation in 1.24 or later) to the IP address of the LoadBalancer Service of the old ingress controller (which is still running); that LoadBalancer external IP may show up as <pending> for the time being
3. optionally, make a backup of the old LoadBalancer Service (with kubectl get service -o yaml …)
4. check that the new ingress controller works properly (e.g. with kubectl port-forward or any other technique)
5. when ready, delete the old LoadBalancer Service
6. the cloud controller manager will automatically reallocate the IP address of the old LoadBalancer to the new one.

Example

To set loadbalancerIP for Traefik:

service:
	spec:
		loadbalancerIP: <old_loadbalancer_IP>

C) Node rollout

Scenario applicability

Expected Downtime

• No downtime when rolling out to different nodes (requiring extra steps e.g. with DNS)
• Low downtime when rolling out to the same set of nodes (as low as the time needed to start an ingress controller pod; typically between a few seconds and a minute)

Description

In this scenario, the ingress controller runs on a set of nodes. This is typically achieved by running the ingress controller with a Daemon Set, either on the whole cluster (for smaller clusters) or on a designated subset of nodes (using a nodeSelector). HTTP and HTTPS traffic is sent to these nodes, on ports 80 and 443.

If the current ingress controller is deployed on a subset of nodes, it is possible to achieve zero downtime by deploying the new ingress controller on another set of nodes. The new ingress controller behavior can be tested extensively (without disturbing the current ingress controller), and once everything is ready, traffic can be switched over to the new ingress controller by updating DNS records: add records to the new nodes, remove the records of the old nodes, and once traffic has effectively moved to the new nodes, stop the old ingress controller.

If the current ingress controller is deployed on all the nodes of the cluster, the procedure would be different. To test the new ingress controller, it would be deployed on different port numbers (for instance, 8080 and 8443). Once the new ingress controller has been confirmed to work correctly, the actual migration can take place - with slightly different steps, depending on whether we’re using HostNetwork or HostPort.

When using HostPort, Kubernetes takes care of forwarding connections from the host ports (e.g. 80 and 443) to the pods. Since the ports are explicitly listed in the pod manifest, Kubernetes “knows” that these ports are allocated on the host, and will prevent two pods from using the same port: the first pod will be scheduled and start normally, and the second pod using the same port will remain in “Pending” state, as the scheduler is able to recognize that a required resource - the host port - is not available. We can rely on this behavior to execute a graceful handover from the old ingress controller to the new one, as follows:

1. create the Daemon Set for the new ingress controller (its pods will be created but remain “Pending”)
2. delete the Daemon Set of the old ingress controller with the “–cascade=orphan” flag (so that its pod keep running)
3. delete the old pods one by one, verifying that the new pods start correctly

If something goes wrong, this careful technique lets you quickly roll back to the old ingress controller by re-creating its Daemon Set.

When using HostNetwork, the ingress controller pods use the network namespace of the nodes on which they run. In other words, when the ingress controller listens on port 80, this is port 80 on the host; there is no extra hop or networking feature involved. From a network standpoint, it’s as if the network controller were running directly on the host, without any kind of virtualization or containerization. The downside of this mode is that Kubernetes doesn’t keep track of the ports allocated by the ingress controller, and won’t prevent two pods from trying to bind the same ports. If two pods (for instance, two different ingress controllers running on the same host) try to bind to port 80, the first one will succeed, while the second one will get an error message similar to “address already in use”.

The rollout will have to be carefully coordinated, by stopping each pod of the old ingress controller and creating its replacement in sequence.

One final note: migrating between HostNetwork and HostPort strategies is possible, but also requires careful coordination, since once again, Kubernetes won’t be aware of the ports in use by the pods using HostNetwork.

Example

To set kong host ports:

--set proxy.http.hostPort=8080 --set proxy.tls.hostPort=8443

Conclusions

In this article, we listed the most common scenarios that you may encounter when migrating from one ingress controller to another.

In practice, if you run Kubernetes in the cloud, you’re more likely to be in one of the two first scenarios. The third scenario is more common on small on-premises clusters.

In any case, ingress controllers are stateless components, and can therefore be changed “in flight” with minimal service disruption (if at all). The bottom line is that if your current ingress controller doesn’t meet all your requirements, it doesn’t have to keep accruing technical debt: you can migrate to a new one very safely!

Do not miss our latest DevOps and Cloud Native blogposts! Follow Enix on Twitter!