github.com/mvisonneau/gitlab-ci-pipelines-exporter

Changed

• Updated default metrics garbage collection intervals from 5 to 10m
• monitor: correctly sanitize the config output
• fixed an issue introduced in v0.5.1 causing the exporter to fail when the monitoring listener address was not defined
• fixed a bug which could cause nil pointer deferences on readiness checks
• Updated golang to 1.17

Added

• New monitoring CLI to retrieve information about the exporter
• New internal metrics about exporter's health:
  • gcpe_currently_queued_tasks_count - Number of tasks in the queue
  • gcpe_environments_count - Number of GitLab environments being exported
  • gcpe_executed_tasks_count - Number of tasks executed
  • gcpe_gitlab_api_requests_count - GitLab API requests count
  • gcpe_metrics_count - Number of GitLab pipelines metrics being exported
  • gcpe_projects_count - Number of GitLab projects being exported
  • gcpe_refs_count - Number of GitLab refs being exported

Changed

• fixed a goroutines/memory leak introduced in 0.5.0 which can occur when attempting to process more tasks than the ratelimit permits
• fixed a bug causing the refreshing of tag based jobs to not match any API results, generating lots of unnecessary requests
• webhooks: added more details when processing wildcards
• examples/quickstart: updated prometheus and grafana versions
• updated most libraries to their most recent versions

BREAKING CHANGES

• The configuration syntax has evolved, you can refer to the updated version of the doc

  • pull.maximum_gitlab_api_requests_per_second has moved to gitlab.maximum_requests_per_second
  • project_defaults.pull.refs.* has been redone in a hopefully more comprehensible fashion
    • project_defaults.pull.refs.(branches|tags|merge_requests).* parameters can now be used to have a finer granularity on the management of per-ref-kind settings
    • project_defaults.pull.refs.from.(pipelines|merge_requests) is not natively managed as part of the per-ref-kind specific parameters
  • project_defaults.pull.environments.name_regexp has moved to project_defaults.pull.environments.regexp
  • project_defaults.pull.environments.tags_regexp was removed to avoid confusion

• Logging configuration now has to be done as part of the config file instead of CLI flags:

log:
  level: info
  format: text

• By default, when exporting metrics for environments, stopped ones will not be considered any more.

Added

• New metric gitlab_ci_pipeline_queued_duration_seconds
• New metric gitlab_ci_pipeline_job_queued_duration_seconds
• (en|dis)able pulling branches / tags / merge_requests on a global or per-project basis
• Choose to export metrics for only the most 'n' recently updated refs pipelines
• Choose to export metrics for refs with pipelines 'updated in the last x seconds'
• Choose to export metrics for deleted branches or tags
• Choose to export metrics for available environments only

Changed

• Bumped all dependencies
• Enhanced the function signatures for the ref objects management
• Fixed a bug causing the jobs metrics pull to fail on ref.Kind=RefKindMergeRequest
• Fixed a bug causing the environments to be garbage collected on every run
• Fixed the error handling when comparing 2 refs which resulted into nil pointer dereferences
• Fixed the pulling of merge-request based pipelines
• Fixed unit tests on windows
• Replaced custom config mangement implementation with creasty/defaults and go-playground/validator
• Rewrote the non-OOP approach used so far for the controller

Changed

• Fixed a bug preventing to set output_sparse_status_metrics globally or at the wildcard level
• Updated all dependencies to their latest versions
• Reduced the default GitLab API request rate limit from 10 to 1rps

Added

• Snapcraft releases
• darwin/arm64 releases
• username label from the gitlab_ci_environment_information flag

Removed

• author_email label from the gitlab_ci_environment_information flag (replaced by username)

Changed

• Scoped down the projects fetched from wildcard onto the one starting with the owner's name to make it clearer for endusers
• Upgraded to go 1.16
• Bumped all dependencies to their latest versions

Added

• GPG sign released artifacts checksums
• Support for performing requests through a forward proxy using standard env variables
• New parameters to enable/disable/aggregate the export of the runner description which executed the job

Changed

• Fixed a bug on child/downstream pipelines pull when the trigger has not been fired yet
• Made the default config local file location not hidden as it makes very little sense
• Updated examples/webhook with ngrok+docker-compose instead of hashicorp waypoint
• Bumped goreleaser to 0.155.0 and leverage docker buildx
• Enhanced logging of the settings at startup to get more visibility on the interpreted parameters
• Fixed a bug preventing the webhooks from working properly when pull is completely disabled in conjunction of wildcards
• Updated all dependencies

Added

• When configured to export job metrics, it will now pull child/downstream pipelines jobs related ones as well
• New runner_description label for all the job related metrics.
• Release GitHub container registry based images: ghcr.io/mvisonneau/gitlab-ci-pipelines-exporter
• Release arm64v8 based container images as part of docker manifests in both docker.io and ghcr.io

Changed

• Ensure consistency of the exported metrics by distinguishing immutable from mutable labels used as metric key
• Updated all dependencies
• Migrated CI from Drone to GitHub actions

Added

• Implemented a max_age_seconds parameter to determine whether to pull a "stale ref" or not

Changed

• When fetching refs from pipelines, capped the maximum length to 100 in order to prevent the API call from failing
• Garbage collect merge requests refs in order to keep the depth value as a maximum
• Prevent potential nil pointers when exporting environments information
• Updated all dependencies

Added

• Automatically refresh pkg.go.dev on new releases

Changed

• Do not delete environments metrics when ref is a tag which may not be configured to be monitored for pipelines/jobs (#182)
• Fixed a bug making latest_commit_short_id labels reflect incorrect information on environment metrics when tags are used as refs
• Enhanced logging for jobs pulling function
• Bumped goreleaser to 0.147.1
• Bumped all dependencies

Added

• Export environments/deployments related metrics
• New environments/deployments and jobs grafana dashboards
• Documented the list of exported metrics
• Released .deb and .rpm packages
• More complete garbage collector capabilities
• Newly supported statuses for pipelines and jobs: created, waiting_for_resource, preparing, scheduled
• GitLab links for pipelines, jobs, environments and deployments in the dashboards

Changed

• Prefix new releases with ^v to make pkg.go.dev happy
• Bumped all dependencies
• Fixed race conditions during tests
• Always return coverage metric
• Enhanced the scheduling of the pull functions on init
• Improved webhook parsing functions performance
• Fixed a bug preventing the gitlab_ci_pipeline_run_count from being initialized correctly at startup
• Fixed the gitlab_ci_pipeline_job_run_count and gitlab_ci_pipeline_run_count metrics incrementing algorithm
• Improved the pipelines grafana dashboard
• Fixed a bug which could lead to an overwrite of the refs and environments at scale, inducing unecessary GitLab API calls and discrepancy for some metrics
• Optimized the storage layer implementation
• Ensure group wildcards only returns projects belonging directly to the group