Daily Life as a Jenkins Administrator
Agent ecosystem overview
The second blog post focused on one specific part of your Kubernetes environment. I felt the urge to write more about it, so I thought a post about agents and managing them would be a good follow-up.
Our agents include:
-
200 user-managed agents running on Windows and Linux
-
100,000 Kubernetes pods created each month
| Every Jenkins agent runs on Java 21. |
The agents
User-managed agents
Users are allowed to add agents themselves. We provide guides for the most common operating systems (Windows, Linux, and macOS) to help them set up and configure their agents correctly, which also reduces our support effort. Users are responsible for maintaining and managing their agents. We offer support and guidance for their questions and issues. They can run builds directly on their agents, use any Docker image they choose, or use our agent image.
Kubernetes-managed agents
The agents are created as pods within our Kubernetes cluster. These agents are ephemeral and are automatically provisioned and deprovisioned based on the runtime of the build. We maintain the cluster environment. The pods run in a CinC setup (e.g. DinD). Kubernetes agents use only our agent image.
Which agent should a user choose?
When deciding which agent to use, consider the following points. Use our Kubernetes agent with our image if:
-
You are getting started with Jenkins
-
You have no special requirements
-
You don’t have the time to maintain your own agent
Use a user-managed agent if:
-
You have to use Windows OS
-
You need a connection to a hardware device (e.g. a dongle)
-
You have jobs which run longer than a few hours
-
You need full control over the environment
| Users can make a request to us if they need a new tool or package which can be added to the agent image. |
Our agent image
The image is based on a Linux distribution and includes a lot of different tools, configurations, environment variables, languages, and packages. This gives users a ready-to-use environment for their builds and makes things more convenient for us, too. Users can request changes or additions to the image. We also handle version bumps for existing tools and packages. For bigger changes, we announce them to users in advance. The image is regularly built to keep databases with security-relevant information up to date. As you can imagine, the image is rather large, which is why it is pre-pulled onto each node in the cluster and available from an internally hosted repository. And since all use the same image, we actually have less to put on the nodes.
Maintenance changes
Every change to the image triggers a series of Jenkins jobs to ensure that it remains stable and compatible with our build environment. This includes unit tests for individual tools and configurations, integration tests for common build scenarios, and performance tests to measure build times and resource usage.
Troubleshooting
Network
Most network issues are related to users not using our image (e.g. network proxy vars are missing or incorrectly configured). Another common cause is not using our internal package repositories. The remaining issues are usually temporary network problems.
Build issues
The most common issues:
-
The latest changes to the Jenkinsfile were not that good as expected (e.g. syntax or logic error, or relied on something from the internet that does not apply to our environment)
-
Missing the latest maintenance or shared library announcements
-
Building a job created in 2019 without updating it
-
Try to use an agent with Java 8
More about issues and some guidance on how to resolve them will be discussed in a future blog post.
| The last two points would of course never happen. ;) |
Third blog post & my ideas for the future
I have a few more topics in mind:
-
Issues, tips and tricks to make your life easier / user support
-
Plugin management
-
Backup and restore / disaster recovery
-
Team collaboration / shared libraries
-
Your choice
If you have feedback or questions, or would like one of these topics covered in the next blog post, please let me know in the comments below.