github.com/fluxcd/source-controller

Release date: 2022-04-28

This prerelease prevents Reconciling and ArtifactOutdated conditions from being set on a HelmRepository when the checksum of a cached repository index changes.

Fixes:

• helmrepo: same revision different checksum scenario #691

Release date: 2022-04-26

This prerelease improves the memory consumption while reconciling a HelmRepository, by only validating the YAML of a fetched index when the checksum of the retrieved file differs from the current Artifact.

Fixes:

• helm: optimise repository index loading #685
• tests: Fix flakiness of git related tests #686

Release date: 2022-04-22

This prerelease fixes a regression bug where the controller would panic in further to be identified edge-case scenarios in which a HelmRepository Artifact would not have a size.

Fixes:

• Fix panic when HelmRepository's artifact size is nil #683

Release date: 2022-04-19

This prerelease enables the Helm chart dependency manager to make use of the opt-in memory cache introduced in v0.23.0, revises the file permissions set by the controller, and updates various dependencies.

:warning: Breaking change: From this release on, the RUNTIME_NAMESPACE environment variable is no longer taken into account to configure the advertised HTTP/S address of the storage. Instead, variable substitution must be used, as described in the changelog entry for v0.5.2.

Improvements:

• Change all file permissions to octal format #653
• Enable dependency manager to use in-memory cache #667
• Update libgit2 image to v1.3.1 #671
• Remove hostname hyphen split block #672
• Update dependencies #675 #676 #677

Release date: 2022-04-12

This prerelease introduces new retention options for Garbage Collection, a new opt-in in-memory cache for HelmRepository index files, improves notifications following reconciling failures, brings ways to configure Key Exchange Algorithms, plus some extra housekeeping awesomeness.

Garbage Collection is enabled by default, and now its retention options are configurable with the flags: --artifact-retention-ttl (default: 60s) and --artifact-retention-records (default: 2). They define the minimum time to live and the maximum amount of artifacts to survive a collection.

A new notification is now emitted to identify recovery from failures. It is triggered when a failed reconciliation is followed by a successful one, and the notification message is the same that's sent in usual successful source reconciliation message about the stored artifact.

The opt-in in-memory cache for HelmRepository addresses issues where the index file is loaded and unmarshalled in concurrent reconciliation resulting in a heavy memory footprint. It can be configured using the flags: --helm-cache-max-size, --helm-cache-ttl, --helm-cache-purge-interval.

The Key Exchange Algorithms used when establishing SSH connections are based on the defaults configured upstream in go-git and golang.org/x/crypto. Now this can be overriden with the flag --ssh-kex-algos. Note this applies to the go-git gitImplementation or the libgit2 gitImplementation but only when Managed Transport is being used.

Managed Transport for libgit2 now introduces self-healing capabilities, to recover from failure when long-running connections become stale.

The exponental back-off retry can be configured with the new flags: --min-retry-delay (default: 750ms) and --max-retry-delay (default: 15min). Previously the defaults were set to 5ms and 1000s, which in some cases impaired the controller's ability to self-heal (e.g. retrying failing SSH connections).

Introduction of a secure directory loader which improves the handling of Helm charts paths.

Improvements:

• update toolkit.fluxcd.io docs links #651
• Add optional in-memory cache of HelmRepository index files #626
• Add flag to allow configuration of SSH kex algos #655
• Garbage collect with provided retention options #638
• Avoid event logging GC failure #659
• Add notify() in all the reconcilers #624
• Remove leftover timeout in reconcilers #660
• libgit2: managed transport improvements #658
• helm: introduce customized chart loaders #663
• Add flags to configure exponential back-off retry #664

Release date: 2022-03-30

This prerelease improves the Status API of the Source objects to reflect more accurate Status Condition information.

In addition, it also fixes a bug in go-git implementation due to which cloning public Git repository failed without any credentials since version 0.17.0, and some general stability improvements in the libgit2 experimental managed transport.

Improvements:

• Align fuzzing deps #644
• Separate positive polarity conditions for ArtifactInStorage #646
• Removes empty credentials from Basic Auth #648
• libgit2: fix access to nil t.stdin and improve observability #649

Release date: 2022-03-28

This prerelease improves on the experimental managed transport overall stability. Changes of note:

• SSH connections now being reused across git operations.
• Leaked HTTP connections are now fixed.
• The long-standing SSH intermittent errors are addressed by the cached connections.

Fixes:

• Various fixes for managed transport #637

Release date: 2022-03-25

This prerelease fixes a regression bug introduced in v0.22.0, which would cause a GitRepository to end up in a Stalled state if an include did not have an Artifact available.

Fixes:

• gitrepo: Do not stall when no included artifact #639
• Fix dpanic issue when logging odd number of args #641

Release date: 2022-03-23

This prerelease ensures (Kubernetes) Event annotations are prefixed with the FQDN of the Source API Group. For example, revision is now source.toolkit.fluxcd.io/revision.

This to facilitate improvements to the notification-controller, where annotations prefixed with the FQDN of the Group of the Involved Object will be transformed into "fields".

Fixes:

• Prefix event annotations with API Group FQDN #632

Release date: 2022-03-23

This prerelease fixes a regression in which .sourceignore rules for a GitRepository would not be matched correctly.

Fixes:

• fix: configure domain for .sourceignore rules #629

Release date: 2022-03-17

This prerelease comes with new reconcilers which make use of fluxcd/pkg utilities for common runtime operations, and graduates the API to v1beta2.

:warning: It is required to update the source-controller Custom Resource Definitions on your cluster and/or in Git.

Breaking changes

• Bucket resources do now take the provided etag for object storage items into account during the calculation of the revision. As a result, items will no longer be downloaded on every reconcile if none of them have changed.
• HelmChart resources do now advertise the observed chart name (.status.observedChartName) and Source (reference) Artifact revision (.status.observedSourceArtifactRevision) in the Status. The information is used to more efficiently react to source revision and/or chart changes.

Features and Improvements

API specifications in a user-friendly format

The new specifications for the v1beta2 API have been written in a new format with the aim to be more valuable to a user. Featuring separate sections with examples, and information on how to write and work with them.

Artifact now advertises size

The size (in bytes) of a tarball Artifact is now advertised in the Size (.size) field of the Artifact. This can be utilized by users to e.g. quickly see if .sourceignore rules have an effect, or be displayed in a UI.

Azure Blob Storage support for Bucket resources

The .spec.provider of a Bucket resource can now be set to azure to instruct the controller to use the Azure Blob Storage SDK while fetching objects. This allows for authenticating using Service Principals, Managed Identities and Shared Keys.

For more information, see the Bucket spec about the Azure provider.

Enhanced Kubernetes Conditions

Source API resources will now advertise more explicit Condition types (more about the types in "API changes"), provide Reconciling and Stalled Conditions where applicable for better integration with kstatus, and record the Observed Generation on the Condition.

For a detailed overview per Source type, refer to the spec:

• GitRepository
• HelmRepository
• HelmChart
• Bucket

Enhanced Kubernetes Events (and notifications)

The Kubernetes Events the controller emits have been reworked to provide a proper reason, and more informative messages. Users making use of the notification-controller will notice this as well, as this same information is used to compose notifications.

Experimental managed transport for libgit2 Git implementation

The libgit2 Git implementation supports a new experimental transport to improve reliability, adding timeout enforcement for Git network operations. Opt-in by setting the environment variable EXPERIMENTAL_GIT_TRANSPORT to true in the controller's Deployment. This will result in the low-level transport being handled by the controller, instead of libgit2. It may result in an increased number of timeout messages in the logs, however it will remove the ability of Git operations to make the controllers hang indefinitely.

Reuse of HTTP/S transport for Helm repository index and chart downloads

The Helm dependency has been updated to v3.8.1, with a patch applied from https://github.com/helm/helm/pull/10568. Using this patch, the HTTP transports are now managed by the controller, to prevent the clogging of thousands of open TCP connections on some instances.

Update of libgit2 Git implementation to v1.3.x

The libgit2 Git implementation has been updated to v1.3.x, allowing us to provide better error signals for authentication, certificate and transport failures. Effectively, this means that instead of a unable to clone: User error string, a descriptive one is now given.

In addition, NO_PROXY settings are now properly taken into account.

Preparation of support for rsa-ssh2-256/512

The dependency on golang.org/x/crypto has been updated to v0.0.0-20220315160706-3147a52a75dd, as preparation of support for rsa-ssh2-256/512. This should theoretically work out of the box for known_hosts entries and go-git Git provider credentials, but has not been widely tested.

API changes

The source.toolkit.fluxcd.io/v1beta2 API is backwards compatible with v1beta1.

• Introduction of Reconciling and Stalled Condition types for better integration with kstatus.
• Introduction of new Condition types to provide better signals and failure indications:
  • ArtifactOutdated: indicates the current Artifact of the Source is outdated.
  • SourceVerified: indicates the integrity of the Source has been verified.
  • FetchFailed: indicates a transient or persistent fetch failure of the upstream Source.
  • BuildFailed: indicates a transient or persistent build failure of a Source's Artifact.
  • StorageOperationFailed: indicates a transient or persistent failure related to storage.
  • IncludeUnavailable: indicates an include is not available. For example, because it does not exist, or does not have an Artifact.
• Introduction of a Size (in bytes) field (.status.artifact.size) in the Artifact object.
• Introduction of ObservedChartName (.status.observedChartName) and ObservedSourceArtifactRevision (.status.observedSourceArtifactRevision) fields in the HelmChart Status.
• Introduction of azure provider implementation for Bucket.

Updating the manifests in Git to v1beta2 can be done at any time after the source-controller upgrade.

Full list of changes

• Upgrade to golang-with-libgit2:1.1.1.6 and use static libraries for in development #562
• Initial fuzzing tests #572
• Validate libgit2 args are set correctly #574
• Download libgit2 libraries for fuzzing #572
• Upgrade libgit2 to 1.3.0 and git2go to v33 #573
• pkg/git: Include commit message and URL in checkout error #579
• Add support for multiple fuzz sanitizers #580
• Upgrade controller-runtime to v0.11.1 and docker/distribution to v2.8.0 #583
• Move to v1beta2 API and rewrite reconcilers #586
• git/libgit2: set CheckoutForce on branch strategy #589
• Reuse transport for Helm downloads #590
• Update object API version in the sample configs #591
• api: Move Status in CRD printcolumn to the end #592
• Update github.com/sosedoff/gitkit to v0.3.0 (CVE fix) #594
• Remove redundant reconciling condition in reconcileArtifact #595
• Implement Size field on archived artifacts #597
• Add native Azure Blob support #598
• Experimental managed transport for libgit2 operations #606
• Update Helm to patched v3.8.1 #609
• Add new condition StorageOperationFailedCondition #612
• Prioritize StorageOperationFailedCondition #613
• Update dependencies #600 #616
• api/v1beta2: add note on Condition polarity #622

Release date: 2022-02-07

This prerelease changes the default timeout of GitRepositories and Buckets from 20s to 60s. When using the libgit2 Git implementation, increasing the timeout helps avoid Error waiting on socket intermittent SSH cloning failures.

Improvements:

• Increase default timeout to 60s #570

Release date: 2022-01-27

This prerelease comes with a bug fix to ensure the libgit2 Git implementation respects hostnames with and without port while matching against known_hosts.

Fixes:

• Fix host mismatch in libgit2 #561