Case Study: Jenkins Migration, Upgrades and Performance Improvements

Project goal

A customer with a SaaS application had an ageing Jenkins CI/CD setup, which deployed their monolithic application to GKE on GCP. It was slow and had recently stopped working. Their requirements were:

• Fix broken CI/CD pipeline
• Speed up deployments
• Upgrade Jenkins to a secure version
• Save costs on GCP
• Reduce carbon footprint of hosting solution

Skills and technologies used

• GCP
• Google Cloud Run
• GKE
• Kubernetes
• Helm
• Jenkins
• CI/CD
• PHP
• Docker
• Ansible

Solution

We rebuilt Jenkins using Ansible, making management easier on an ongoing basis; as part of this, Jenkins was upgraded to the latest version, fixing issues with outdated plugins, etc.

Infrequently used Kubernetes services were moved to Google Cloud Run, which retained their scalability when required but reduced the cost of running these services by 99%.

We moved the Helm Kubernetes configuration into a separate repository, allowing the changes to the Kubernetes configuration without creating a new build.

Builds and their resulting Docker images were tagged with a Git hash, allowing their deployment to development, staging and production environments without having to rerun static code analysis, unit tests and rebuild Docker images unnecessarily.

This vastly improved deployment times and allowed quick rollbacks if any new code caused issues.

We updated the Jenkins pipeline to use parallel jobs where possible, speeding up deployments by running pipeline steps concurrently.

We scheduled the build of base and helper Docker images daily rather than for every deployment, and we cached NPM and Composer packages to allow their quick reuse for each build.

Any changes to Google Cloud Run services were deployed simultaneously as the core monolith application, again parallel.

These changes reduced deployment time from thirty to five minutes and allowed bug fixes to be deployed to a staging environment, tested and promoted to production in half an hour instead of three.

Client-customised templates were shared into the application using an expensive GlusterFS setup, and a recent Kubernetes version removed support for mounting GlusterFS filesystems; and this prevented updates to that version. We built client-specific services to replace GlusterFS, which saved money and increased uptime.

We reconfigured the GKE cluster to use preemptible VM instances and reduced the instance size, resulting in no performance or reliability impact and reducing costs significantly.

The combination of moving the chunky Jenkins GKE cluster to an Ansible-managed instance, moving client-customised templates from GlusterFS to customised services, moving infrequently used services to Cloud Run and utilising preemptible VM instances saved the customer 75% in their monthly GCP bill. It also, of course, decreased the carbon footprint of the hosting significantly.