We began our collaboration with Institut Pasteur four years ago on one-off networking and Kubernetes rollout missions. We now handle the operational management of their platform used for scientific research.
Presentation of the Golden Banana award for the biggest Kubernetes cluster with GPUs!
A relaxed conversation hosted by Sébastien (Enix Partner) with their teams to explore their platform, their migration to K8s, and the behind-the-scenes of our collaboration!
Stéphane, can you tell us more about your research activities and your IT needs at Institut Pasteur?
Stéphane (CIO at Institut Pasteur): In life sciences, as in other research fields, research can be split into two broad categories: wet science and dry science. Wet science is the work of the researcher in a white coat with test tubes, as it’s often pictured, in a somewhat clichéd way. Dry science, or in silico science, relies on computing: developing algorithms, using highly specific software (e.g. AlphaFold), computational models, or building databases (genome banks, microscopy banks, etc.). In every case, researchers’ goal is to publish in the most prestigious journals. These scientists, sometimes supported by IT specialists or bioinformaticians, need robust and agile platforms. Beyond an HPC (High Performance Computing) cluster for heavy computation, adopting Kubernetes and automating application design and deployment have been central to achieving that agility.
Your platforms are on-premise, do you need any particular hardware?
Stéphane: All this research work requires a lot of IT resources. We try to optimize them as much as possible, especially since Institut Pasteur is a foundation that relies on public generosity for a third of its budget. For the control plane of our Kubernetes clusters, for example, we reused servers from our HPC cluster when we upgraded to higher-performance models. We also use GPUs, for instance for applications relying on inference, after training a model on the HPC cluster. To simplify, you could say we have two platforms: the Kubernetes platform for most applications and services, including those exposed on the internet, and the HPC platform for parallelized, high-performance computing.
Thomas, you’re the Kubernetes specialist at Pasteur, how did you get there?
Thomas (Deputy Head of IT Operations at Institut Pasteur): We were looking for more scalability and to better automate the researchers’ work. After evaluating OpenShift, we opted for a simpler, less costly approach, with a more open, tailor-made solution built around CI/CD (continuous integration and deployment) and Kubernetes along with its technology ecosystem.
How were we able to help you with your move to Kubernetes?
Thomas: We had already built a V1 with a single Kubernetes cluster. We wanted to industrialize it and roll it out more broadly for the researchers’ various use cases. It was hard to find expertise in these technologies. Stéphane put us in touch with you, and we found a team that could support us on Kubernetes and, more broadly, on the infrastructure technologies we were already using.
At first, we reviewed the components of our v1 cluster together. Some were kept, others replaced for our target architecture. After deploying our 4 new clusters, you helped us migrate our applications. It was particularly complex, as we had 60 to 80 projects to migrate to the new clusters, including our Data Lake, for example.
Since then, you’ve handled the operational management of the clusters and supported the scientists whenever they run into issues with their applications. That can mean networking questions within the cluster, matters related to putting their applications into production, or new, specific needs around Elasticsearch, for example.
You don’t just use Kubernetes on its own, but also an advanced CI/CD pipeline…
Thomas: Yes, we have an internal software forge and a continuous integration pipeline built on GitLab CI, which is the entry point for deploying our applications to the K8s clusters. A scientist, or any platform user, has specific rights to view and deploy to the infrastructure. We’ve also embraced the GitOps movement to encourage users to work more cleanly and version their changes.
Bryan, as more of a user on the researchers’ side, what has this new platform brought you?
Bryan (Research Engineer at Institut Pasteur): What people really like is the reproducibility aspect. Previously, when developers deployed their applications on VMs, everyone did it a bit their own way, with manual operations. Automating the application design pipeline and its deployment on K8s lets scientists deploy in a “cleaner” way that’s easier to maintain over time. In science, reproducibility is fundamental — we need to be able to rerun experiments months or years later, often even with new researchers.
Can you tell us about a specific use case?
Bryan: I started using K8s on a research project on phages. These are viruses used to kill targeted bacteria. Each researcher sequences their various phages and needs a platform to share, compare, and aggregate their results. I offered them a web platform deployed on the K8s cluster where they could submit their data. The K8s side and the CI/CD setup speed up the rollout of new features, with on-demand staging environments. Researchers really appreciate that kind of responsiveness.
Beyond K8s, how have your needs evolved in recent years?
Thomas: We took a building-block approach. The first big topic was setting up the dev and prod Kubernetes clusters. We then added a K8s cluster with GPUs and specific capabilities. I wanted us to set up Kubeflow to give Machine Learning researchers access to GPUs through a graphical interface.
We also had storage needs on top of our Isilon NFS array. We use a Ceph cluster that you also run for us. We also needed to collect logs and metrics with Elasticsearch, Kibana, and Vector. And we keep evolving the Kubernetes clusters — recently we added resource validation policies with Kyverno.
Lastly, a few words on how our teams work together day to day?
Thomas: From the start, your approach hasn’t been to replace the expertise already present within our teams. You position yourselves as an extension of our existing team, bringing your expertise on Kubernetes and, more broadly, on DevOps and infrastructure topics. We bridged our Rocket.Chat and your Slack together, and whenever we have questions or needs on the platforms, interactions between our teams are smooth and fast. That gives us peace of mind and frees up time so we can work on pure infrastructure topics and support researchers at the same time.
