github.com/cloudflare/pint

Fixed

• Fixed a regression causing poor query cache hit rate.

Added

• Added uptime field in prometheus configuration block. This field can be used to set a custom metric used for Prometheus uptime checks and by default uses up metric. If you have a Prometheus with a large number of scrape targets there might be a huge number of up time series making those uptime checks slow to run. If your Prometheus is configured to scrape itself, then you most likely want to use one of metrics exported by Prometheus, like prometheus_build_info:

  prometheus "prod" {
    uri    = "https://prometheus.example.com"
    uptime = "prometheus_build_info"
  }
  

Changed

• Refactored some queries used by promql/series check to avoid sending queries that might be very slow and/or return a huge amount of data.
• Prometheus query cache now takes into account the size of cached response. This makes memory usage needed for query cache more predictable. As a result the cache option for prometheus config block now means the number of time series cached instead of the number of responses cached and the default for this option is now 50000.

Fixed

• promql/vector_matching was sending expensive queries resulting in high memory usage, this is now fixed.

Added

• Added pint_prometheus_cache_evictions_total metric tracking the number of times cache results were evicted from query cache.
• Allow disabling individual checks for the entire file using # pint file/disable ... comments.

Changed

• Refactored query cache to only store queries that are requested more than once. This will avoid storing big responses that are never requested from the cache.

Fixed

• Config validation will now check for duplicated prometheus block names.

Fixed

• Fixed performance regression slowing down pint watch over time.

Added

• prometheus configuration block now accepts optional headers field, for setting request headers that will be attached to any request made to given Prometheus server. Example:

  prometheus "protected" {
  uri      = "https://prod.example.com"
  headers  = {
    "X-Auth": "secret",
    "X-User": "bob"
  }
  

Changed

• Prometheus range query handling was rewritten to improve memory usage caused by queries returning huge number of results. As a result pint should use up to 5x less memory.

Fixed

• Fixed false positive reports in promql/vector_matching for rules using on(...). Example:

  sum(foo) without(instance) * on(app_name) group_left() bar
  

• Don't log passwords when Prometheus URI is using basic authentication.

• Fixed false positive reports in alerts/template suggesting to use humanize on queries that already use round().

• Fixed false positive reports in alerts/comparison when bool modifier is used on a condition that is guarded by another condition. Example:

  alert: Foo
  expr: (foo > 1) > bool 1
  

• Fixed false positive reports in alerts/template warning about labels removed in a query despite being re-added by a join.

Fixed

• Fixed incorrect line number reporting on BitBucket annotations.

Fixed

• Fixed handling of symlinks when running pint lint and pint watch commands.

Added

• BitBucket only allows for annotations on modified lines, so when a high severity problem is reported on unmodified line pint will move that annotation to the first modified line, so it's still visible in BitBucket. Now pint will also add a note to that annotation to make it clear that the problem is really on a different line.

• alerts/template will now run extra checks to validate syntax of queries executed from within alerting rule templates.

  Example template using sum(xxx query that's missing closing ):

  {% raw %}

  - alert: ...
    expr: ...
    annotations:
      summary: |
        {{ with query "sum(xxx" }}
        {{ . | first | value | humanize }}
        {{ end }}
  

  {% endraw %}

• If a file is ignored pint will now note that using Information level annotation. This will make it more obvious that a CI check passed because pint didn't run any checks due to file being excluded.

Changed

• Prometheus rule files can be symlinked between directories. If the symlink source and target files are in a different directory they can end up querying different Prometheus server when running ping checks. This means that when modifying symlink target file checks must be executed against both symlink source and target. Until now pint was ignoring symlinks but starting with this release it will try to follow them. This means that if you modify a file that has symlinks pointing to them pint will try run checks against those symlinks too.

  NOTE: pint can only detect and check symlinks if they are located in the current working directory (as seen by running pint process) or its sub-directories.

Fixed

• Fixed a regression in promql/vector_matching that would cause a panic when parsing function calls with optional arguments.

Fixed

• promql/vector_matching was incorrectly handling queries containing function calls with multiple arguments.

Fixed

• Revert 'Use smaller buffers when decoding Prometheus API responses' change.

Fixed

• Use smaller buffers when decoding Prometheus API responses.

Fixed

• Fixed wrong request formatting for Prometheus metric metadata queries.

Changed

• Switched from using prometheus/client_golang API client to streaming JSON library prymitive/current

Fixed

• Avoid reporting same issue multiple times in promql/rate and promql/regexp checks.

Changed

• Updated Prometheus modules to v2.38.0. This adds support for toTime template function.

Fixed

• Fixed symlink handling when running pint lint.

Fixed

• Remove noisy debug logs.

Added

• Added pint_prometheus_cache_miss_total metric.

Changed

• Reduce log level for File parsed messages.

Fixed

• Purge expired cache entries faster to reduce memory usage.

Fixed

• Fix absent() handling in alerts/comparison #330.

Added

• Added --min-severity flag to the pint lint command. Default value is set to warning.

Fixed

• Fix a regression in promql/vector_matching introduced in previous release.
• Fix promql/series disable comments not working when there are multiple comments on a rule.
• promql/series no longer emits an information message metric is generated by alerts ....

Fixed

• Don't use topk in promql/vector_matching check to avoid false positives.

Added

• promql/rate check will now also validate deriv function usage.
• alerts/annotation check will now recommend using one of humanize functions if alert query is returning results based on rate() and the value is used in annotations.

Changed

• promql/series check now supports more flexible # pint disable promql/series(...) comments. Adding a comment # pint disable promql/series({cluster="dev"}) will disable this check for any metric selector with cluster="dev" matcher.

• query/cost check will now calculate how much Prometheus memory will be needed for storing results of given query. bytesPerSample option that was previously used to calculate this was removed.

• prometheus {} config block now allows to pass a list of paths to explicitly ignore by setting exclude option. Existing paths option was renamed to include for consistency. Example migration:

  prometheus "foo" {
    [...]
    paths = [ "rules/.*" ]
  }
  

  becomes

  prometheus "foo" {
    [...]
    include = [ "rules/.*" ]
  }
  

Fixed

• pint_last_run_checks and pint_last_run_checks_done were not updated properly.

Added

• Deduplicate reports where possible to avoid showing same issue twice.
• rule/link check for validating URIs found in alerting rule annotations.

Changed

• Add more details to BitBucket CI reports.
• More compact console output when running pint lint.

Added

• promql/range_query check.

Fixed

• Strict parsing mode shouldn't fail on template errors, those will be later reported by alerts/template check.

Changed

• All timeout options are now optional. This includes following config blocks:
  • prometheus { timeout = ... }
  • repository { bitbucket { timeout = ... } }
  • repository { github { timeout = ... } }
• pint will now try to discover all repository settings from environment variables when run as part of GitHub Actions workflow and so it doesn't need any repository { github { ... } } configuration block for that anymore. Setting GITHUB_AUTH_TOKEN is the only requirement for GitHub Actions now.

Fixed

• Fixed line reporting on some strict parser errors.

Added

• Added --base-branch flag to pint ci command.

Added

• Added rate limit for Prometheus API requests with a default value of 100 requests per second. To customise it set rateLimit field inside selected prometheus server definition.
• Added pint_last_run_checks and pint_last_run_checks_done metrics to track progress when running pint watch.

Fixed

• Improved range query cache efficiency.

Added

• Added extra global configuration for promql/series check. See check documentation for details.
• prometheus server definition in pint config file can now accept optional cache field (defaults to 10000) to allow fine tuning of built-in Prometheus API query caching.
• Added pint_prometheus_cache_size metric that exposes the number of entries currently in the query cache.

Fixed

• Improved error reporting when strict mode is enabled.

Fixed

• Fixed high memory usage when running range queries against Prometheus servers.

Changed

• The way pint sends API requests to Prometheus was changed to improve performance.

  First change is that each prometheus server definition in pint config file can now accept optional concurrency field (defaults to 16) that sets a limit on how many concurrent requests can that server receive. There is a new metric that tracks how many queries are currently being run for each Prometheus server - pint_prometheus_queries_running.

  Second change is that range queries will now be split into smaller queries, so if pint needs to run a range query on one week of metrics, then it will break this down into multiple queries each for a two hour slot, and then merge all the results. Previously it would try to run a single query for a whole week and if that failed it would reduce time range until a query would succeed.

Fixed

• Strict parsing mode didn't fully validate rule group files, this is now fixed and pint runs the same set of checks as Prometheus.
• Fixed promql/series handling of rules with {__name__=~"foo|bar"} queries.
• If Prometheus was stopped or restarted promql/series would occasionally report metrics as "sometimes present". This check will now try to find time ranges with no metrics in Prometheus and ignore these when checking if metrics are present.

Fixed

• pint_prometheus_queries_total and pint_prometheus_cache_hits_total metric wasn't always correctly updated.
• Ignore unknown metric types in promql/rate.

Added

• promql/rate check will now report if rate() or irate() function is being passed a non-counter metric.

Fixed

• pint will now correctly handle YAML anchors.

Added

• Parsing files in relaxed mode will now try to find rules inside multi-line strings #252. This allows direct linting of k8s manifests like the one below:

  ---
  kind: ConfigMap
  apiVersion: v1
  metadata:
    name: example-app-alerts
    labels:
    app: example-app
  data:
    alerts: |
      groups:
        - name: example-app-alerts
          rules:
            - alert: Example_Is_Down
              expr: kube_deployment_status_replicas_available{namespace="example-app"} < 1
              for: 5m
              labels:
                priority: "2"
                environment: production
              annotations:
                summary: "No replicas for Example have been running for 5 minutes"
  

Fixed

• Fixed incorrect line reported when pint fails to unmarshal YAML file.

Added

• Allow fine tuning promql/series check with extra control comments # pint rule/set promql/series min-age ... and # pint rule/set promql/series ignore/label-value ... See promql/series for details.
• promql/regexp will report redundant use of regex anchors.

Changed

• promql/series will now report missing metrics only if they were last seen over 2 hours ago by default. This can be customised per rule with comments.

Fixed

• Fix problem line reporting for rule/owner check.
• Add missing rule/owner documentation page.

Fixed

• Fixed false positive reports from promql/series check when running pint watch.

Added

• Added pint_last_run_duration_seconds metric.
• Added --require-owner flag support to pint ci command.

Fixed

• Better handling of YAML unmarshal errors.

Fixed

• Fixed false positive reports from alerts/template check when absent() is used inside a binary expression.

Fixed

• File parse errors didn't report correct line numbers when running pint ci.

Fixed

• File parse errors were not reported correctly when running pint ci.

Fixed

• Handle 504 Gateway Timeout HTTP responses from Prometheus same as query timeouts and retry with a shorter range query.

Added

• When running pint ci all checks will be skipped if any commit contains [skip ci] or [no ci] string in the commit message.

Changed

• By default pint will now parse all files in strict mode, where all rule files must have the exact syntax Prometheus expects:

  groups:
    - name: example
      rules:
        - record: ...
          expr: ...
  

  Previous releases were only looking for individual rules so groups object wasn't required. Now pint will fail to read any file that doesn't follow Prometheus syntax exactly. To enable old behaviour add parser { relaxed = ["(.+)", ...]} option in the config file. See Configuration for details. To enable old (relaxed) behaviour for all files add:

  parser {
  relaxed = ["(.*)"]
  }
  

Fixed

• Improved promql/vector_matching checks to detect more issues.
• Fixed reporting of problems detected on unmodified lines when running pint ci.