External Fingerprint Storage Phase-2 Updates
As another great phase for the External Fingerprint Storage Project comes to an end, we summarise the work done during this phase in this blog post. It was an exciting and fruitful journey, just like the previous phase, and offered some great learning experience.
To understand what the project is about and the past progress, please refer to the phase 1 blog post.
We targeted four stories in this phase, namely fingerprint cleanup, fingerprint migration, refactoring the current implementation to use descriptors, and improved testing of the Redis Fingerprint Storage Plugin. We explain these stories in detail below.
This story involved extending the
FingerprintStorage API to allow external storage plugins to perform and configure
their own fingerprint cleanup strategies.
We added the following functionalities to Jenkins core API:
This allows external fingerprint storage implementations to implement their own custom fingerprint cleanup. The method is called periodically by Jenkins core.
FingerprintStorage#cleanFingerprint(Fingerprint fingerprint, TaskListener taskListener)
This is a reference implementation which can be called by external storage plugins to clean up a fingerprint. It is upto the plugin implementation to decide whether to use this method. They may choose to write a custom implementation.
We consume these new API functionalities in the Redis Fingerprint Storage plugin. The plugin uses cursors to traverse the fingerprints, updating the build information, and deleting the build-less fingerprints.
Earlier, fingerprint cleanup was always run periodically and there was no way to turn it off. We also added an option to allow the user to turn off fingerprint cleanup.
This was done because it may be the case that keeping redundant fingerprints in memory might be cheaper than the cleanup operation (especially in the case of external storages, which are cheaper these days).
Earlier, there was no support for fingerprints stored in the local storage. In this phase, we introduce migration support for users. The old fingerprints are now migrated to the new configured external storage whenever they are used (lazy migration). This allows gradual migration of old fingerprints from local disk storage to the new external storage.
Earlier, whenever an external fingerprint storage plugin was installed, it was enabled by default.
We refactored the implementation to make use of
Descriptor pattern so the fingerprint engine can now be selected
as a dropdown from the Jenkins configuration page.
The dropdown is shown only when multiple fingerprint storage engines are configured on the system.
Redis Fingerprint Storage Plugin was refactored
to use this new implementation.
We introduced new connection tests in the Redis Fingerprint Storage Plugin. These tests allow testing of cases like slow connection, breakage of connection to Redis, etc. These were implemented using the Toxiproxy module inside Testcontainers.
We introduced test for Configuration-as-code (JCasC) compatibility with the plugin. The documentation for configuring the plugin using JCasC was also added.
We introduced a suite of authentication tests, to verify the proper working of the Redis authentication system. Authentication uses the credentials plugin.
We strengthened our web UI testing to ensure that the configuration page for the plugin works properly as planned.
Please refer to the Jira Epic for this phase.
Changes in the Jenkins core (except migration) were released in Jenkins 2.248.
We drafted 1.0-rc-1 release for the Redis Fingerprint Storage Plugin to deliver the changes. This was an increment from the alpha release we had drafted at the end of the previous phase. The plugin is now available at https://plugins.jenkins.io/redis-fingerprint-storage/!
The latest release for the plugin can be downloaded from the update center, instructions for which can be found in the README of the plugin. We appreciate you trying out the plugin, and welcome any suggestions, feature requests, bug reports, etc.
Some of the topics we aim to tackle in the next phase include a new reference implementation (possibly backed by PostgreSQL), tracing, etc.