GSoC 2026 Final Blog: How It Was to Be a Contributor at Jenkins
Hello everyone!
I am Mallikarjun G D, and I have been a GSoC 2026 contributor for the project Continue: AI-Powered Chatbot for Quick Access to Jenkins Resources. Over the past few months, I have learnt a lot, not just about AI and Jenkins, but also about contributing to a large open-source project. So for my final GSoC blog, I wanted to share how the whole journey actually went.
Before GSoC
My journey with Jenkins started pretty randomly.
I was learning Jenkins, got curious about its open-source codebase, and while exploring the GitHub organization, I stumbled across the resources-ai-chatbot-plugin.
I found out it had started as a GSoC 2025 project, cloned it, tried running it, and eventually filtered the Issues page by good-first-issue.
And of course, I found a README issue the classic open-source starter pack.
I worked on a small setup guide, fixed a few missing links, and opened my first pull request.
And there it was my first open-source PR.
After that, I slowly moved from documentation issues toward code-related contributions. By the time GSoC applications opened, I had around five or six PRs and a much better understanding of the project.
I started my proposal early, focused on areas that matched both the project priorities and what I wanted to learn, went through the interview, and then came the selection.
That was when the real journey started.
During GSoC at Jenkins
The first few weeks were a mix of excitement and nervousness. Community bonding helped me understand the workflow, get comfortable with my mentors, and settle into regular coding, PR reviews, bi-weekly meetings, and weekly GSoC stand-ups.
One early lesson came while working on LLM-as-a-Judge. I was stuck improving the faithfulness score and kept tuning the same things. A mentor suggested changing the evaluation template itself, and that worked almost immediately. It taught me to step back and rethink the approach instead of endlessly tuning the same solution.
The hardest part was burnout when my college exams overlapped with the coding period. Until midterm, I had also put a lot of pressure on myself simply because it was GSoC. After midterm, that changed, and the work started feeling much more like normal open-source contribution and learning.
The project was also extended by two weeks, giving me some additional time to complete the remaining work.
GraphRAG gave me another memorable challenge. My first documentation-based graph was too small, so I researched further and found Jenkins Update Center data. Rebuilding around it gave me roughly 5,000 relationships, which was one of the most satisfying moments of the project.
Over time, I also started thinking more about testing, edge cases, maintainability, smaller PRs, and reusing existing architecture. Most importantly, I became much more comfortable asking questions and discussing ideas with maintainers.
By the end, GSoC felt less like something I had to prove myself through and more like something I genuinely enjoyed being part of.
The Features I worked on:
LLM-as-a-Judge Evaluation
The first major feature I worked on was an LLM-as-a-Judge evaluation pipeline.
The goal was to move away from manually judging whether chatbot changes were actually improving responses. I built an evaluation flow using a fixed golden dataset and DeepEval, measuring Faithfulness, Answer Relevancy, and Contextual Recall directly in CI.
This feature taught me a lot about evaluation beyond just looking at model outputs. I also had to deal with model runtime, quantization, CI limitations, caching, and making the evaluation stable enough to be useful during development.
GraphRAG for Relationship Queries
The next feature was GraphRAG, mainly for Jenkins plugin dependency queries.
Semantic retrieval works well for documentation questions, but it struggles when relationship direction matters. For example, "What does git depend on?" and "What depends on git?" look very similar to a vector retriever but need completely different answers.
I built a directional plugin graph using Jenkins documentation and Update Center data. The graph now contains roughly 5,000 relationships across around 1,400 plugins, while normal documentation queries still continue through semantic retrieval.
This was probably the feature that taught me the most about how retrieval systems work underneath, and when structured retrieval is more useful than semantic similarity.
Build Failure Diagnosis Agent
The third feature was the Build Failure Diagnosis Agent.
Jenkins console logs can become extremely long, even when only a few lines actually explain the failure. The agent extracts the relevant error section and sends that focused context to the chatbot.
While building this, security became an important part of the design. Build logs can contain API keys, tokens, and other sensitive values, so the extracted snippet is sanitized before it reaches an external LLM.
This feature made me think much more carefully about what data should reach an LLM in the first place, rather than simply sending everything and expecting the model to handle it.
Third-Party LLM Provider Support
One additional feature I worked on was third-party LLM provider integration.
Previously, the chatbot mainly depended on running a model locally. That made development difficult for contributors without a capable GPU, which was something I faced myself.
I added provider routing through LiteLLM, along with a frontend provider and model selector. The chatbot can now work with providers such as Groq, OpenRouter, Gemini, Claude, and OpenAI, while still supporting local inference.
Providers are configured through providers.json, and credentials use a per-user Bring Your Own Key setup.
Interestingly, this ended up being the feature I enjoyed working on the most. By this point I was much more comfortable with the codebase, and it felt less like figuring out how the project worked and more like actually building within it.
Jenkins Plugin Installation Experience
Another feature I worked on was improving the Jenkins plugin installation experience. Previously, installing the plugin did not automatically give users a complete chatbot interface because the frontend still had to be run separately.
The React frontend is now bundled directly inside the Jenkins plugin HPI, so installing the plugin through Jenkins makes the chatbot UI available without requiring a separate frontend server. The plugin connects to the FastAPI backend through a Jenkins-configured backend URL, with Jenkins acting as the same-origin API gateway between the frontend and backend.
The backend remains separately managed and can be started in a local setup. This keeps Python dependencies and backend services outside the Jenkins process while making the plugin itself much easier to install and use.
What Comes Next
Even though GSoC is ending, there are still a few things I would like to continue improving in the project.
Some of the next areas are:
-
Expand GraphRAG to cover more conflict and incompatibility relationships.
-
Integrate Jenkins authentication and authorization so user context stays private, anonymous users only access public information, and data is not exposed across users.
More importantly, I would like to continue contributing to Jenkins instead of treating the final evaluation as the end of the project.
Looking Back
The biggest change for me was probably how comfortable open source feels now.
Before GSoC, even opening a pull request felt like a big deal. Now, reading a large codebase, discussing ideas with maintainers, asking questions, getting reviews, and iterating on changes feels much more natural.
I also stopped looking at GSoC as something I had to constantly prove myself through. Somewhere along the way, it simply became a project I genuinely enjoyed working on and learning from.
That transition is probably what I will remember the most.
Acknowledgements
A huge thank you to my mentors, V. Sreenivas, Bervianto Leo Pratama, and Giovanni Vaccarino and special mention to Kris Stern for their guidance, reviews, and patience throughout the project.
A lot of my learning came from discussions where they did not just tell me what to change, but helped me understand why a different approach would be better.
I would also like to thank the Jenkins community, the GSoC organization team, and the other contributors I interacted with throughout the program. Being part of the community made the whole experience much more enjoyable.
Final Note
If there is one thing I would tell someone who is thinking about contributing to open source, it is to just start.
My journey began with a small README issue because I had no idea where else to begin. That one contribution slowly turned into more issues, more code, a GSoC proposal, and eventually several months of working with the Jenkins community.
You do not need to know everything before contributing. Ask questions, start small, and keep going.
Follow Along
-
Repository: jenkinsci/resources-ai-chatbot-plugin
-
GitHub: arjun-713
-
LinkedIn: Mallikarjun G D
About the author