Git Plugin Performance Improvements

Goal: Enhance user experience by enabling faster checkouts

Status: Completed

Team

Details

The aim of the project is to find and improve potential areas in the git plugin for performance enhancement. The process of doing that was to find differences in performance between the two co-existing implementations for git, CLI git and JGit. It is logical to leverage those differences by creating a feature which would automatically switch git implementations optimally adjusting to a certain scenario or a use-case. It is also logical to fix existing performance issues within the git plugin to enhance its overall performance.

The project identified git SCM checkout as the area which showed potential scope for enhancements. Improving any portion of the process meant a direct reduction in the overall execution time of a Jenkins job.

Background of the project

The git plugin is one of the most widely used Jenkins component which provides git source code management functionalities to the Jenkins ecosystem. These functionalities or the git operations (like fetch or checkout) are provided using two mutually exclusive implementations, CliGitAPIImpl and JGitAPIImpl.

By default, only command line git is enabled for a user. To enable JGit and fully appreciate the benefits of the new features, go to this doc: Enabling JGit in the git plugin.

The task was to compare both of these available implementations for certain repository parameters (like size of git objects, number of branches) in multiple platforms to derive a so called universal rule which would help the plugin decide which implementation is optimal in a certain situation.

Project Deliverables

  • Remove a redundant fetch (JENKINS-49757) during initial checkout (PR-845)

    • A fix which ensures no repository/user data loss if second fetch is removed with automated tests to confirm the intended behavior

  • Performance comparison using JMH benchmarking framework (PR-553)

  • Performance enhancements in the git plugin (PR-931)

Performance comparison and its results

Here is the PR-521 which provides the code and results from the initial investigation of performance comparison between CliGitAPIImpl and JGitAPIImpl.

The graphical results for various benchmarks performed throughout the projects are present in:

Performance enhancement in the checkout step

The project has introduced a new feature called the GitToolChooser which recommends the optimal git implementation for a particular scenario.
The user doesn’t have to worry about choosing the correct implementation for the given remote repository.
Here is a graphical representation of the feature:

git perf improv

Evaluation Phase 1

  1. Integrate JMH benchmarks module into the git client plugin

    • Choose a git operation and write a benchmark

    • Validate the benchmark results

    • Repository

  2. Fix git redundant fetch issue JENKINS-49757

    • Create a fix and write automated test cases

    • Interactive testing before pushing it to production

  3. Design a strategy to implement performance improvements in the plugin

Evaluation Phase 2

  1. Implementing performance improvement from benchmarks:

    • GitToolChooser - A class which tries to recommend the optimal git implementation for a particular remote repository

    • Repository

  2. Widening the scope of benchmarking in the git client plugin

    • Benchmarks on multiple repository parameters

    • Benchmarks on multiple platforms

Resources

Evaluation Phase 3

This phase marked the release of the git plugin with all of the performance improvements performed during GSoC period. That includes addition of GitToolChooser, removing redundant second fetch from checkout and adding a JMH benchmark module in git client plugin.

  1. Git Plugin 4.4.0 released

  2. Git Client Plugin 3.4.0 released

Office hours

Office hours are scheduled each Wednesday at 14:30 UTC, with regular meeting notes available for anyone to read.

Links