github.com/open-policy-agent/opa

This release focuses on bugfixes and documentation improvements, as well as a few small performance improvements.

Runtime, Tooling, SDK

• runtime: Update rule index's trie node scalar handling so that numerics compare correctly (#5585) authored by @ashutosh-narkar reported by @alvarogomez93
• ast: Improve error information when metadata yaml fails to compile (#4475) authored and reported by @johanfylling
• bundle: Retain metadata annotations for Wasm entrypoints during inspection (#5588) authored and reported by @johanfylling
• compile: Allow object generating rules to be annotated as entrypoints (#5577) authored and reported by @johanfylling
• plugins/discovery: Support for persisting and loading discovery bundle from disk (#2886) authored by @ashutosh-narkar reported by @anderseknert
• perf: Use json.Encode to avoid extra allocation (authored by @anderseknert)
• opa inspect: Fix prefix error when inspecting bundle from root (#5503) authored by @harikannan512 reported by @HarshPathakhp
• topdown: http.send to cache responses based on status code (#5617) authored by @ashutosh-narkar
• types: Add GoDoc about named types (authored by @wata727)
• deps: Remove github.com/pkg/errors dependency (authored by @Iceber)

Docs

• Update entrypoint documentation (#5565) authored by @johanfylling reported by @robertgartman
• Add missing folder argument in bundle build example (authored by @charlieegan3)
• Clarify crypto.x509.parse_certificates docs (authored by @charlieegan3)
• Added AWS S3 Web Identity Credentials info to tutorial (authored by @vishrana)
• docs/graphql: non-nullable id argument and typo fix (authored by @philipaconrad)

Website + Ecosystem

• Ecosystem:

  • ccbr (authored by @niuzhi)

• Website:

  • Show prominent warning when viewing old docs (authored by @charlieegan3)
  • Prevent navbar clipping on narrow screens + sticky nav (authored by @charlieegan3)

Miscellaneous

Dependency bumps:

• build: bump golang 1.19.4 -> 1.19.5 (authored by @yanggangtony)
• ci: aquasecurity/trivy-action from 0.8.0 to 0.9.0
• github.com/containerd/containerd from 1.6.15 to 1.6.16
• google.golang.org/grpc from 1.51.0 to 1.52.3

This release rolls in security fixes from recent patch releases, along with a number of bugfixes, and a new builtin function.

Improved error reporting available in opa eval

A common frustration when writing policies in OPA is when an error happens, causing a rule to unexpectedly return undefined. Using --strict-builtin-errors would allow finding the first error encountered during evaluation, but terminates execution immediately.

To improve the debugging experience, it is now possible to display all of the errors encountered during normal evaluation of a policy, via the new --show-builtin-errors option.

Consider the following error-filled policy, multi-error.rego:

package play

this_errors(number) := result {
        result := number / 0
}

this_errors_too(number) := result {
        result := number / 0
}

res1 := this_errors(1)

res2 := this_errors_too(1)

Using --strict-builtin-errors, we would only see the first divide by zero error:

opa eval --strict-builtin-errors -d multi-error.rego data.play

1 error occurred: multi-error.rego:4: eval_builtin_error: div: divide by zero

Using --show-builtin-errors shows both divide by zero issues though:

opa eval --show-builtin-errors -d multi-error.rego data.play -f pretty

2 errors occurred:
multi-error.rego:4: eval_builtin_error: div: divide by zero
multi-error.rego:8: eval_builtin_error: div: divide by zero

By showing more errors up front, we hope this will improve the overall policy writing experience.

New Built-in Function: time.format

It is now possible to format a time value from nanoseconds to a formatted timestamp string via a built-in function. The builtin accepts 3 argument formats, each allowing for different options:

1. A number representing the nanoseconds since the epoch (UTC).
2. A two-element array of the nanoseconds, and a timezone string.
3. A three-element array of nanoseconds, timezone string, and a layout string (same format as for time.parse_ns).

See the documentation for all details.

Implemented by @burnerlee.

Optimization in rule indexing

Previously, every time the evaluator looked up a rule in the index, OPA performed checks for grounded refs over the entire index before looking up the rule.

Now, OPA performs all groundedness checks once at index construction time, which keeps index lookup times much more consistent as the number of indexed rules scales up.

Policies with large numbers of index-ready rules can expect a small performance lift, proportional to the number of indexed rules.

Bundle fetching with AWS Signing Version 4A

AWS has recently developed an extension to SigV4 called Signature Version 4A (SigV4A) which enables signatures that are valid in more than one AWS Region. This new signature method is required for signing multi-region API requests, such as Amazon S3 Multi-Region Access Points (MRAP).

OPA now supports this new request signing method for bundle fetching, which means that you can use an S3 MRAP as a bundle source. This is configured via the new services[<your_service_name>].credentials.s3_signing.signature_version field.

See the the documentation for more details.

Implemented by @jwineinger

Runtime

• rego: Check store modules before skipping parsing (authored by @charlieegan3)
• topdown/rego: Add BuiltinErrorList support to rego package, add to eval command (authored by @charlieegan3)
• topdown: Fix evaluator's re-wrapping of NDBCache errors (authored by @srenatus)
• Fix potential memory leak from http.send in interquery cache (authored by @asleire)
• ast/parser: Detect function rule head + contains keyword (#5525) authored and reported by @philipaconrad
• ast/visit: Add SomeDecl to visitor walks (#5480) authored by @srenatus
• ast/visit: Include LazyObject in visitor walks (#5479) authored by @srenatus reported by @benweint

Tooling, SDK

• topdown: cache undefined rule evaluations (#593) authored by @edpaget reported by @tsdandall
• topdown: Specify host verification policy for http redirects (#5388) authored and reported by @ashutosh-narkar
• providers/aws: Refactor + Fix 2x Authorization header append issue (#5472) authored by @philipaconrad reported by @Hiieu
• Add support to enable ND builtin cache via discovery (#5457) authored by @ashutosh-narkar reported by @asadali
• format: Only use ref heads for all rule heads if necessary (#5449) authored and reported by @srenatus
• opa inspect: Fix path of data namespaces on windows (authored by @shm12)
• ast+cmd: Only enforcing schemas annotations if --schema flag is used (authored by @johanfylling)
• sdk: Allow use of a query tracer (authored by @charlieegan3)
• sdk: Allow use of metrics, profilers, and instrumentation (authored by @charlieegan3)
• sdk: Return provenance information in Result types (authored by @charlieegan3)
• sdk: Allow use of StrictBuiltinErrors (authored by @charlieegan3)
• Allow print calls in IR (authored by @anderseknert)
• tester/runner: Fix panic'ing case in utility function (#5496) authored and reported by @philipaconrad

Docs

• Community page updates (authored by @anderseknert)

• Update Hugo version, update deprecated Page fields (authored by @charlieegan3)

• docs: Update TLS-based Authentication Example (#5521) authored by @charlieegan3 reported by @jjthom87

• docs: Update opa eval flags to link to bundle docs (authored by @charlieegan3)

• docs: Make SDK first option for Go integraton (authored by @anderseknert)

• docs: Fix typo on Policy Language page. (authored by @mcdonagj)

• docs/integrations: Update kubescape repo links (authored by @dwertent)

• docs/oci: Corrected config section (authored by @ogazitt)

• website/frontpage: Update Learn More links (authored by @pauly4it)

• integrations.yaml: Ensure inventors listed in organizations (authored by @anderseknert)

• integrations: Fix malformed inventors item (authored by @anderseknert)

• Add Digraph to ADOPTERS.md (authored by @jamesphlewis)

Miscellaneous

• Remove changelog maintainer mention filter (authored by @anderseknert)
• Chore: Fix len check in the ast/visit_test error message (authored by @boranx)
• opa inspect: Fix wrong windows bundle tar files path separator (authored by @shm12)
• Add CHANGELOG.md to website build triggers (authored by @srenatus)

Dependency bumps:

• Golang 1.19.3 -> 1.19.4
• github.com/containerd/containerd from 1.6.10 -> 1.6.15
• github.com/dgraph-io/badger/v3
• golang.org/x/net to 0.5.0
• json5 and postcss-modules
• oras.land/oras-go from 1.2.1 -> 1.2.2

CI/Distribution fixes:

• Update base images for non debug builds (authored by @charlieegan3)
• Remove deprecated linters in golangci config (authored by @yanggangtony)

This is a bug fix release addressing a panic in opa test.

• tester/runner: Fix panic'ing case in utility function. (#5496) authored by @philipaconrad

This is a bug fix release addressing an issue that prevented OPA from fetching bundles stored in S3 buckets.

• providers/aws: Refactor + fix 2x Authorization header append issue. (#5472) authored by @philipaconrad, reported by @Hiieu

This is a second security fix to address CVE-2022-41717/GO-2022-1144.

We previously believed that upgrading the Golang version and its stdlib would be sufficient to address the problem. It turns out we also need to bump the x/net dependency to v0.4.0., a version that hadn't existed when v0.46.2 was released.

This release bumps the golang.org/x/net dependency to v0.4.0, and contains no other changes over v0.46.2.

Note that the affected code is OPA's HTTP server. So if you're using OPA as a Golang library, or if your confident that your OPA's HTTP interface is protected by other means (as it should be -- not exposed to the public internet), you're OK.

This is a bug fix release addressing two issues: one security issue, and one bug related to formatting backwards-compatibility.

Golang security fix CVE-2022-41717

An attacker can cause excessive memory growth in a Go server accepting HTTP/2 requests.

Since we advise against running an OPA service exposed to the general public of the internet, potential attackers would be limited to people that are already capable of sending direct requests to the OPA service.

opa fmt and backwards compatibility (#5449)

In v0.46.1, it was possible that opa fmt would format a rule in such a way that:

1. Before formatting, it was working fine with older OPA versions, and
2. after formatting, it would only work with OPA version >= 0.46.1.

This backwards incompatibility wasn't intended, and has now been fixed.

This release contains a mix of bugfixes, optimizations, and new features.

New Built-in Function: object.keys

It is now possible to conveniently retrieve an object's keys via a built-in function.

Before, you had to resort to constructs like

import future.keywords.in

keys[k] {
    _ = input[k]
}

allow if "my_key" in keys

Now, you can simply do

import future.keywords.in

allow if "my_key" in object.keys(input)

See the documentation for all details.

Implemented by @kevinswiber.

New Built-in Function: AWS Signature v4 Request Signing

It is now possible to use a built-in function to prepare a request with a signature, so that it can be used with AWS endpoints that use request signing for authentication.

See this example:

req := {"method": "get", "url": "https://examplebucket.s3.amazonaws.com/data"}
aws_config := {
    "aws_access_key": "MYAWSACCESSKEYGOESHERE",
    "aws_secret_access_key": "MYAWSSECRETACCESSKEYGOESHERE",
    "aws_service": "s3",
    "aws_region": "us-east-1",
}
example_verify_resource {
    resp := http.send(providers.aws.sign_req(req, aws_config, time.now_ns()))
    # process response from AWS ...
}

See the documentation on the new built-in for all details.

Reported by @jicowan and implemented by @philipaconrad.

Performance improvements for object.get and in operator

Before, using object.get and in had come with a performance penalty that wasn't to be expected just from the look of the calls: Since they have been implemented using built-in functions (obvious for object.get, not obvious for "admin" in input.user.roles), all of their operands had to be read from the store (if applicable) and converted into AST types.

Now, we use shallow references ("lazy objects") for store reads in the evaluator. In these two cases, this can bring huge performance improvements, when the object argument of these two calls is a ref into the base document (like data.users):

object.get(data.roles, input.role, [])
{ "id": 12 } in data.users

Tooling, SDK, and Runtime

• opa eval: Added --strict to enable strict code checking in evaluation (#5182) authored by @Parsifal-M
• opa fmt: Remove { true } block following else head
• opa fmt: Generate new wildcards for else and chained function heads in the parser (#5347). This fixes superfluous introductions of _1 instead of _ in when formatting functions that use wildcard arguments, like f(_) := true.
• opa fmt: Fix assignment rewrite in else formatting (#5348)
• OCI Download: Set auth credentials only if needed (#5212) authored by @carabasdaniel
• Server: Differentiate between "missing" and "undefined doc" in default decision (#5344)

Topdown and Rego

• http.send: Fix interquery cache size calculation with concurrent requests (#5359) reported and authored by @asleire
• http.send: Remove socket query param for unix sockets (#5313) reported and authored by @michivi
• Annotations: Add type coercion guards to avoid panics (#5368)
• Compiler: Provide more accurate error locations for some with unused vars (#4238)
• Optimization: Read lazy objects from the store (#5325). This improves the performance of x in data.foo and object.get(data.bar, ...) calls significantly.
• Partial Evaluation: Skip comprehensions when checking eqs in copy propagation (#5367). This fixes a bug when optimization on bundles would change the outcome of the subsequent evaluation.
• Parser: Fix else error handling with ref heads -- errors had occurred at a later stage then desired, because an edge case slipped through the earlier check.
• Planner/IR: Fix ref heads processing -- the CallDynamic optimization wasn't planned properly; a bug introduced with ref heads.

Documentation

• Builtins: Mention base64 URL encoding specifically (#5406) reported by @phi1010
• Builtins: Include behavior with sets in json.patch (#5328)
• Comparison: small fix to table to match sample code and other tables (authored by @anlandu)
• Builtins: Document reference timestamp behavior for time.parse_ns
• Typo fixes, authored by @deining
• Golang integration: update example code, move SDK above low-level packages

Website + Ecosystem

• Ecosystem:
  • Add Easegress (authored by @localvar)
  • Add Terraform Cloud
• Website: Updated Footer Color (#5254), reported and authored by @UtkarshMishra12
• Website: Add "canonical" link to latest to help with SEO and ancient pages being returned by search engines.
• Website: Add experimental "OPA version" badge. (Still needs to be tested more thorougly before advertisting it.)

Miscellaneous

• Dependency bumps: Notably, we're now using wasmtime-go v3
• CI fixes:
  • Move performance tests to nightly tests
  • CLI: add simple bundle build tests
  • Nightly: Revamp how we're doing fuzz testing