GSoC 2026 Community Bonding Wrap-Up: Continuation of AI-Powered Chatbot for Jenkins Resources

I set up the local development environment on Fedora Linux — downloading the LLM model, running the data pipeline to populate the vector database, and getting the frontend and backend running together end to end.

I read through the full codebase: the RAG pipeline, plugin configuration, query flow between frontend and backend, and how responses are returned to the user.

I studied the DeepEval documentation and the NetworkX source and docs in preparation for the GraphRAG graph layer.

I designed the evaluation pipeline architecture — dataset format, judge model selection, which metrics to track, and how to store results so that runs remain comparable over time.

I ran a prototype pipeline on a small synthetic dataset in a personal repository before touching the main codebase. This surfaced a real problem early: the initial judge model choice was too slow at scale, which I was able to address before it blocked any coding-period work.

I fixed a frontend rendering bug where the plugin frontend wasn’t injected correctly into the Jenkins page: #370.

I set up the dataset branch with a file structure and versioning plan for the golden Q&A dataset, establishing the conventions that will carry through the coding period.

I attended sync meetings with mentors V. Sreenivas, Bervianto Leo Pratama, and Giovanni Vaccarino to align on approach and surface blockers early. The key decisions from these meetings were: biweekly progress update meetings will be the default cadence; day-to-day questions and blockers go through Slack to keep async communication fast; I will maintain a daily work log throughout the coding period to keep the team informed; and the golden Q&A dataset will be built manually rather than using DeepEval’s synthetic data generator — this keeps the ground-truth answers grounded in real Jenkins documentation and reduces the risk of hallucinated reference answers inflating eval scores.

Build 100 curated Q&A pairs covering plugin documentation, Jenkinsfile syntax, and common build failure cases. The dataset will be assembled manually (not synthesized) to ensure ground-truth answers are accurate and hallucination-free. This dataset is the foundation every subsequent eval run depends on.

Integrate DeepEval metrics (Answer Relevancy, Faithfulness, Contextual Recall) into the GitHub Actions pipeline and record a baseline score against the current chatbot.

Build the GraphRAG layer on top of the existing FAISS retriever using NetworkX and benchmark it against flat-RAG on plugin dependency queries — validating whether graph traversal genuinely improves recall on multi-hop plugin questions or only adds latency.

Build the agent in two stages: log ingestion and structured parsing first, then Presidio-based PII sanitization before context is passed to the LLM. Both stages will be covered by eval metrics before the feature is considered complete.