github.com/hashicorp/consul

BREAKING CHANGES:

• extensions: The Lua extension now targets local proxy listeners for the configured service's upstreams, rather than remote downstream listeners for the configured service, when ListenerType is set to outbound in extension configuration. See CVE-2023-2816 changelog entry for more details. [GH-17415]

SECURITY:

• Update to UBI base image to 9.2. [GH-17513]
• Upgrade golang.org/x/net to address CVE-2022-41723 [GH-16754]
• Upgrade to use Go 1.20.4. This resolves vulnerabilities CVE-2023-24537(go/scanner), CVE-2023-24538(html/template), CVE-2023-24534(net/textproto) and CVE-2023-24536(mime/multipart). Also, golang.org/x/net has been updated to v0.7.0 to resolve CVEs CVE-2022-41721 , CVE-2022-27664 and [CVE-2022-41723 ](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h .) [GH-17240]
• extensions: Disable remote downstream proxy patching by Envoy Extensions other than AWS Lambda. Previously, an operator with service:write ACL permissions for an upstream service could modify Envoy proxy config for downstream services without equivalent permissions for those services. This issue only impacts the Lua extension. [CVE-2023-2816] [GH-17415]

FEATURES:

• hcp: Add new metrics sink to collect, aggregate and export server metrics to HCP in OTEL format. [GH-17460]

IMPROVEMENTS:

• Fixes a performance issue in Raft where commit latency can increase by 100x or more when under heavy load. For more details see https://github.com/hashicorp/raft/pull/541. [GH-17081]
• agent: add a configurable maximimum age (default: 7 days) to prevent servers re-joining a cluster with stale data [GH-17171]
• agent: add new metrics to track cpu disk and memory usage for server hosts (defaults to: enabled) [GH-17038]
• connect: update supported envoy versions to 1.22.11, 1.23.8, 1.24.6, 1.25.4 [GH-16889]
• envoy: add MaxEjectionPercent and BaseEjectionTime to passive health check configs. [GH-15979]
• hcp: Add support for linking existing Consul clusters to HCP management plane. [GH-16916]
• logging: change snapshot log header from agent.server.snapshot to agent.server.raft.snapshot [GH-17236]
• peering: allow re-establishing terminated peering from new token without deleting existing peering first. [GH-16776]
• peering: gRPC queries for TrustBundleList, TrustBundleRead, PeeringList, and PeeringRead now support blocking semantics, reducing network and CPU demand. The HTTP APIs for Peering List and Read have been updated to support blocking. [GH-17426]
• raft: Remove expensive reflection from raft/mesh hot path [GH-16552]
• xds: rename envoy_hcp_metrics_bind_socket_dir to envoy_telemetry_collector_bind_socket_dir to remove HCP naming references. [GH-17327]

BUG FIXES:

• Fix an bug where decoding some Config structs with unset pointer fields could fail with reflect: call of reflect.Value.Type on zero Value. [GH-17048]
• acl: (Enterprise only) Check permissions in correct partition/namespace when resolving service in non-default partition/namespace
• acl: Fix an issue where the anonymous token was synthesized in non-primary datacenters which could cause permission errors when federating clusters with ACL replication enabled. [GH-17231]
• acls: Fix ACL bug that can result in sidecar proxies having incorrect endpoints.
• connect: Fix multiple inefficient behaviors when querying service health. [GH-17241]
• gateways: Fix an bug where targeting a virtual service defined by a service-resolver was broken for HTTPRoutes. [GH-17055]
• grpc: ensure grpc resolver correctly uses lan/wan addresses on servers [GH-17270]
• namespaces: adjusts the return type from HTTP list API to return the api module representation of a namespace. This fixes an error with the consul namespace list command when a namespace has a deferred deletion timestamp.
• peering: Fix issue where modifying the list of exported services did not correctly replicate changes for services that exist in a non-default namespace. [GH-17456]
• peering: Fix issue where peer streams could incorrectly deregister services in various scenarios. [GH-17235]
• peering: ensure that merged central configs of peered upstreams for partitioned downstreams work [GH-17179]
• xds: Fix possible panic that can when generating clusters before the root certificates have been fetched. [GH-17185]

FEATURES:

• xds: Allow for configuring connect proxies to send service mesh telemetry to an HCP metrics collection service. [GH-16585]

BUG FIXES:

• audit-logging: (Enterprise only) Fix a bug where /agent/monitor and /agent/metrics endpoints return a Streaming not supported error when audit logs are enabled. This also fixes the delay receiving logs when running consul monitor against an agent with audit logs enabled. [GH-16700]
• ca: Fixes a bug where updating Vault CA Provider config would cause TLS issues in the service mesh [GH-16592]
• cache: revert cache refactor which could cause blocking queries to never return [GH-16818]
• gateway: (Enterprise only) Fix bug where namespace/partition would fail to unmarshal for TCPServices. [GH-16781]
• gateway: (Enterprise only) Fix bug where namespace/partition would fail to unmarshal. [GH-16651]
• gateway: (Enterprise only) Fix bug where parent refs and service refs for a route in the same namespace as the route would fallback to the default namespace if the namespace was not specified in the configuration rather than falling back to the routes namespace. [GH-16789]
• gateway: (Enterprise only) Fix bug where routes defined in a different namespace than a gateway would fail to register. [GH-16677].
• gateways: Adds validation to ensure the API Gateway has a listener defined when created [GH-16649]
• gateways: Fixes a bug API gateways using HTTP listeners were taking upwards of 15 seconds to get configured over xDS. [GH-16661]
• peering: (Consul Enterprise only) Fix issue where connect-enabled services with peer upstreams incorrectly required service:write access in the default namespace to query data, which was too restrictive. Now having service:write to any namespace is sufficient to query the peering data.
• peering: (Consul Enterprise only) Fix issue where resolvers, routers, and splitters referencing peer targets may not work correctly for non-default partitions and namespaces. Enterprise customers leveraging peering are encouraged to upgrade both servers and agents to avoid this problem.
• peering: Fix issue resulting in prepared query failover to cluster peers never un-failing over. [GH-16729]
• peering: Fixes a bug that can lead to peering service deletes impacting the state of local services [GH-16570]
• peering: Fixes a bug where the importing partition was not added to peered failover targets, which causes issues when the importing partition is a non-default partition. [GH-16675]
• raft_logstore: Fixes a bug where restoring a snapshot when using the experimental WAL storage backend causes a panic. [GH-16647]
• ui: fix PUT token request with adding missed AccessorID property to requestBody [GH-16660]
• ui: fix rendering issues on Overview and empty-states by addressing isHTMLSafe errors [GH-16574]

IMPROVEMENTS:

• cli: added -append-policy-id, -append-policy-name, -append-role-name, and -append-role-id flags to the consul token update command. These flags allow updates to a token's policies/roles without having to override them completely. [GH-16288]
• cli: added -append-service-identity and -append-node-identity flags to the consul token update command. These flags allow updates to a token's node identities/service identities without having to override them. [GH-16506]
• connect: Bump Envoy 1.22.5 to 1.22.7, 1.23.2 to 1.23.4, 1.24.0 to 1.24.2, add 1.25.1, remove 1.21.5 [GH-16274]
• mesh: Add ServiceResolver RequestTimeout for route timeouts to make request timeouts configurable [GH-16495]
• ui: support filtering API gateways in the ui and displaying their documentation links [GH-16508]

DEPRECATIONS:

• cli: Deprecate the -merge-node-identites and -merge-service-identities flags from the consul token update command in favor of: -append-node-identity and -append-service-identity. [GH-16506]
• cli: Deprecate the -merge-policies and -merge-roles flags from the consul token update command in favor of: -append-policy-id, -append-policy-name, -append-role-name, and -append-role-id. [GH-16288]

BUG FIXES:

• cli: Fixes an issue with consul connect envoy where a log to STDOUT could malform JSON when used with -bootstrap. [GH-16530]
• cli: Fixes an issue with consul connect envoy where grpc-disabled agents were not error-handled correctly. [GH-16530]
• cli: ensure acl token read -self works [GH-16445]
• cli: fix panic read non-existent acl policy [GH-16485]
• gateways: fix HTTPRoute bug where service weights could be less than or equal to 0 and result in a downstream envoy protocol error [GH-16512]
• gateways: fix HTTPRoute bug where services with a weight not divisible by 10000 are never registered properly [GH-16531]
• mesh: Fix resolution of service resolvers with subsets for external upstreams [GH-16499]
• proxycfg: ensure that an irrecoverable error in proxycfg closes the xds session and triggers a replacement proxycfg watcher [GH-16497]
• proxycfg: fix a bug where terminating gateways were not cleaning up deleted service resolvers for their referenced services [GH-16498]
• ui: Fix issue with lists and filters not rendering properly [GH-16444]

KNOWN ISSUES:

• connect: A race condition can cause some service instances to lose their ability to communicate in the mesh after 72 hours (LeafCertTTL) due to a problem with leaf certificate rotation. This bug is fixed in Consul v1.15.2 by GH-16818.

BREAKING CHANGES:

• acl errors: Delete and get requests now return descriptive errors when the specified resource cannot be found. Other ACL request errors provide more information about when a resource is missing. Add error for when the ACL system has not been bootstrapped.
  • Delete Token/Policy/AuthMethod/Role/BindingRule endpoints now return 404 when the resource cannot be found.
    • New error formats: "Requested * does not exist: ACL not found", "* not found in namespace $NAMESPACE: ACL not found"
  • Read Token/Policy/Role endpoints now return 404 when the resource cannot be found.
    • New error format: "Cannot find * to delete"
  • Logout now returns a 401 error when the supplied token cannot be found
    • New error format: "Supplied token does not exist"
  • Token Self endpoint now returns 404 when the token cannot be found.
    • New error format: "Supplied token does not exist" [GH-16105]
• acl: remove all acl migration functionality and references to the legacy acl system. [GH-15947]
• acl: remove all functionality and references for legacy acl policies. [GH-15922]
• config: Deprecate -join, -join-wan, start_join, and start_join_wan. These options are now aliases of -retry-join, -retry-join-wan, retry_join, and retry_join_wan, respectively. [GH-15598]
• connect: Add peer field to service-defaults upstream overrides. The addition of this field makes it possible to apply upstream overrides only to peer services. Prior to this change, overrides would be applied based on matching the namespace and name fields only, which means users could not have different configuration for local versus peer services. With this change, peer upstreams are only affected if the peer field matches the destination peer name. [GH-15956]
• connect: Consul will now error and exit when using the consul connect envoy command if the Envoy version is incompatible. To ignore this check use flag --ignore-envoy-compatibility [GH-15818]
• extensions: Refactor Lambda integration to get configured with the Envoy extensions field on service-defaults configuration entries. [GH-15817]
• ingress-gateway: upstream cluster will have empty outlier_detection if passive health check is unspecified [GH-15614]
• xds: Remove the connect.enable_serverless_plugin agent configuration option. Now Lambda integration is enabled by default. [GH-15710]

SECURITY:

• Upgrade to use Go 1.20.1. This resolves vulnerabilities CVE-2022-41724 in crypto/tls and CVE-2022-41723 in net/http. [GH-16263]

FEATURES:

• API Gateway (Beta) This version adds support for API gateway on VMs. API gateway provides a highly-configurable ingress for requests coming into a Consul network. For more information, refer to the API gateway documentation. [GH-16369]
• acl: Add new acl.tokens.config_file_registration config field which specifies the token used to register services and checks that are defined in config files. [GH-15828]
• acl: anonymous token is logged as 'anonymous token' instead of its accessor ID [GH-15884]
• cli: adds new CLI commands consul troubleshoot upstreams and consul troubleshoot proxy to troubleshoot Consul's service mesh configuration and network issues. [GH-16284]
• command: Adds the operator usage instances subcommand for displaying total services, connect service instances and billable service instances in the local datacenter or globally. [GH-16205]
• config-entry(ingress-gateway): support outlier detection (passive health check) for upstream cluster [GH-15614]
• connect: adds support for Envoy access logging. Access logging can be enabled using the proxy-defaults config entry. [GH-15864]
• xds: Add a built-in Envoy extension that inserts Lua HTTP filters. [GH-15906]
• xds: Insert originator service identity into Envoy's dynamic metadata under the consul namespace. [GH-15906]

IMPROVEMENTS:

• connect: for early awareness of Envoy incompatibilities, when using the consul connect envoy command the Envoy version will now be checked for compatibility. If incompatible Consul will error and exit. [GH-15818]
• grpc: client agents will switch server on error, and automatically retry on RESOURCE_EXHAUSTED responses [GH-15892]
• raft: add an operator api endpoint and a command to initiate raft leadership transfer. [GH-14132]
• acl: Added option to allow for an operator-generated bootstrap token to be passed to the acl bootstrap command. [GH-14437]
• agent: Give better error when client specifies wrong datacenter when auto-encrypt is enabled. [GH-14832]
• api: updated the go module directive to 1.18. [GH-15297]
• ca: support Vault agent auto-auth config for Vault CA provider using AWS/GCP authentication. [GH-15970]
• cli: always use name "global" for proxy-defaults config entries [GH-14833]
• cli: connect envoy command errors if grpc ports are not open [GH-15794]
• client: add support for RemoveEmptyTags in Prepared Queries templates. [GH-14244]
• connect: Warn if ACLs are enabled but a token is not provided to envoy [GH-15967]
• container: Upgrade container image to use to Alpine 3.17. [GH-16358]
• dns: support RFC 2782 SRV lookups for prepared queries using format _<query id or name>._tcp.query[.<datacenter>].<domain>. [GH-14465]
• ingress-gateways: Don't log error when gateway is registered without a config entry [GH-15001]
• licensing: (Enterprise Only) Consul Enterprise non-terminating production licenses do not degrade or terminate Consul upon expiration. They will only fail when trying to upgrade to a newer version of Consul. Evaluation licenses still terminate.
• raft: Added experimental wal backend for log storage. [GH-16176]
• sdk: updated the go module directive to 1.18. [GH-15297]
• telemetry: Added a consul.xds.server.streamsUnauthenticated metric to track the number of active xDS streams handled by the server that are unauthenticated because ACLs are not enabled or ACL tokens were missing. [GH-15967]
• ui: Update sidebar width to 280px [GH-16204]
• ui: update Ember version to 3.27; [GH-16227]

DEPRECATIONS:

• acl: Deprecate the token query parameter and warn when it is used for authentication. [GH-16009]
• cli: The -id flag on acl token operations has been changed to -accessor-id for clarity in documentation. The -id flag will continue to work, but operators should use -accessor-id in the future. [GH-16044]

BUG FIXES:

• agent configuration: Fix issue of using unix socket when https is used. [GH-16301]
• cache: refactor agent cache fetching to prevent unnecessary fetches on error [GH-14956]
• cli: fatal error if config file does not have HCL or JSON extension, instead of warn and skip [GH-15107]
• cli: fix ACL token processing unexpected precedence [GH-15274]
• peering: Fix bug where services were incorrectly imported as connect-enabled. [GH-16339]
• peering: Fix issue where mesh gateways would use the wrong address when contacting a remote peer with the same datacenter name. [GH-16257]
• peering: Fix issue where secondary wan-federated datacenters could not be used as peering acceptors. [GH-16230]

SECURITY:

• Update golang.org/x/net to v0.17.0 to address CVE-2023-39325 / CVE-2023-44487(x/net/http2). [GH-19225]
• Upgrade Go to 1.20.10. This resolves vulnerability CVE-2023-39325 / CVE-2023-44487(net/http). [GH-19225]
• Upgrade google.golang.org/grpc to 1.56.3. This resolves vulnerability CVE-2023-44487. [GH-19414]
• connect: update supported envoy versions to 1.24.12 to address CVE-2023-44487 [GH-19271]

BUG FIXES:

• Mesh Gateways: Fix a bug where replicated and peered mesh gateways with hostname-based WAN addresses fail to initialize. [GH-19268]
• api: add custom marshal/unmarshal for ServiceResolverConfigEntry.RequestTimeout so config entries that set this field can be read using the API. [GH-19031]
• ca: ensure Vault CA provider respects Vault Enterprise namespace configuration. [GH-19095]
• catalog api: fixes a bug with catalog api where filter query parameter was not working correctly for the /v1/catalog/services endpoint [GH-18322]
• connect: Fix bug where uncleanly closed xDS connections would influence connection balancing for too long and prevent envoy instances from starting. Two new configuration fields performance.grpc_keepalive_timeout and performance.grpc_keepalive_interval now exist to allow for configuration on how often these dead connections will be cleaned up. [GH-19339]

SECURITY:

• Upgrade to use Go 1.20.8. This resolves CVEs CVE-2023-39320 (cmd/go), CVE-2023-39318 (html/template), CVE-2023-39319 (html/template), CVE-2023-39321 (crypto/tls), and CVE-2023-39322 (crypto/tls) [GH-18742]

IMPROVEMENTS:

• Adds flag -append-filename (which works on values version, dc, node and status) to consul snapshot save command. Adding the flag -append-filename version,dc,node,status will add consul version, consul datacenter, node name and leader/follower (status) in the file name given in the snapshot save command before the file extension. [GH-18625]
• api: Add support for listing ACL tokens by service name. [GH-18667]
• command: Adds -since flag in consul debug command which internally calls hcdiag for debug information in the past. [GH-18797]
• log: Currently consul logs files like this consul-{timestamp}.log. This change makes sure that there is always consul.log file with the latest logs in it. [GH-18617]

BUG FIXES:

• api: Fix /v1/agent/self not returning latest configuration [GH-18681]
• ca: Vault provider now cleans up the previous Vault issuer and key when generating a new leaf signing certificate [GH-18779] [GH-18773]
• gateways: Fix a bug where gateway to service mappings weren't being cleaned up properly when externally registered proxies were being deregistered. [GH-18831]
• telemetry: emit consul version metric on a regular interval. [GH-18724]

SECURITY:

• Update golang.org/x/net to v0.13.0 to address CVE-2023-3978. [GH-18358]
• Upgrade golang.org/x/net to address CVE-2023-29406 [GH-18186]
• Upgrade to use Go 1.20.6. This resolves CVE-2023-29406(net/http) for uses of the standard library. A separate change updates dependencies on golang.org/x/net to use 0.12.0. [GH-18190]
• Upgrade to use Go 1.20.7. This resolves vulnerability CVE-2023-29409(crypto/tls). [GH-18358]

FEATURES:

• cli: consul members command uses -filter expression to filter members based on bexpr. [GH-18223]
• cli: consul watch command uses -filter expression to filter response from checks, services, nodes, and service. [GH-17780]
• reloadable config: Made enable_debug config reloadable and enable pprof command to work when config toggles to true [GH-17565]

IMPROVEMENTS:

• Fix some typos in metrics docs [GH-18080]
• acl: added builtin ACL policy that provides global read-only access (builtin/global-read-only) [GH-18319]
• acl: allow for a single slash character in policy names [GH-18319]
• connect: update supported envoy versions to 1.21.6, 1.22.11, 1.23.12, 1.24.10 [GH-18305]
• hcp: Removes requirement for HCP to provide a management token [GH-18140]
• xds: Explicitly enable WebSocket connection upgrades in HTTP connection manager [GH-18150]

BUG FIXES:

• Fix a bug that wrongly trims domains when there is an overlap with DC name. [GH-17160]
• connect/ca: Fixes a bug preventing CA configuration updates in secondary datacenters [GH-17846]
• connect: Fix incorrect protocol config merging for transparent proxy implicit upstreams. [GH-17894]
• connect: fix a bug with Envoy potentially starting with incomplete configuration by not waiting enough for initial xDS configuration. [GH-18024]
• snapshot: fix access denied and handle is invalid when we call snapshot save on windows - skip sync() for folders in windows in https://github.com/rboyer/safeio/pull/3 [GH-18302]

SECURITY:

• Update to UBI base image to 9.2. [GH-17513]

FEATURES:

• cli: consul operator raft list-peers command shows the number of commits each follower is trailing the leader by to aid in troubleshooting. [GH-17582]
• server: (Enterprise Only) allow automatic license utilization reporting. [GH-5102]

IMPROVEMENTS:

• connect: update supported envoy versions to 1.21.6, 1.22.11, 1.23.9, 1.24.7 [GH-17547]
• debug: change default setting of consul debug command. now default duration is 5ms and default log level is 'TRACE' [GH-17596]
• fix metric names in /docs/agent/telemetry [GH-17577]
• peering: gRPC queries for TrustBundleList, TrustBundleRead, PeeringList, and PeeringRead now support blocking semantics, reducing network and CPU demand. The HTTP APIs for Peering List and Read have been updated to support blocking. [GH-17426]
• raft: Remove expensive reflection from raft/mesh hot path [GH-16552]
• systemd: set service type to notify. [GH-16845]

BUG FIXES:

• cache: fix a few minor goroutine leaks in leaf certs and the agent cache [GH-17636]
• connect: reverts #17317 fix that caused a downstream error for Ingress/Mesh/Terminating GWs when their respective config entry does not already exist. [GH-17541]
• namespaces: (Enterprise only) fixes a bug where agent health checks stop syncing for all services on a node if the namespace of any service has been removed from the server.
• namespaces: (Enterprise only) fixes a bug where namespaces are stuck in a deferred deletion state indefinitely under some conditions. Also fixes the Consul query metadata present in the HTTP headers of the namespace read and list endpoints.
• namespaces: adjusts the return type from HTTP list API to return the api module representation of a namespace. This fixes an error with the consul namespace list command when a namespace has a deferred deletion timestamp.
• peering: Fix a bug that caused server agents to continue cleaning up peering resources even after loss of leadership. [GH-17483]
• peering: Fix issue where modifying the list of exported services did not correctly replicate changes for services that exist in a non-default namespace. [GH-17456]

SECURITY:

• Upgrade to use Go 1.20.4. This resolves vulnerabilities CVE-2023-24537(go/scanner), CVE-2023-24538(html/template), CVE-2023-24534(net/textproto) and CVE-2023-24536(mime/multipart). Also, golang.org/x/net has been updated to v0.7.0 to resolve CVEs CVE-2022-41721 , CVE-2022-27664 and [CVE-2022-41723 ](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h .) [GH-17240]

IMPROVEMENTS:

• connect: update supported envoy versions to 1.21.6, 1.22.11, 1.23.8, 1.24.6 [GH-16888]
• envoy: add MaxEjectionPercent and BaseEjectionTime to passive health check configs. [GH-15979]
• hcp: Add support for linking existing Consul clusters to HCP management plane. [GH-16916]
• logging: change snapshot log header from agent.server.snapshot to agent.server.raft.snapshot [GH-17236]
• peering: allow re-establishing terminated peering from new token without deleting existing peering first. [GH-16776]

BUG FIXES:

• Fix an bug where decoding some Config structs with unset pointer fields could fail with reflect: call of reflect.Value.Type on zero Value. [GH-17048]
• acl: (Enterprise only) Check permissions in correct partition/namespace when resolving service in non-default partition/namespace
• acls: Fix ACL bug that can result in sidecar proxies having incorrect endpoints.
• connect: Fix multiple inefficient behaviors when querying service health. [GH-17241]
• connect: fix a bug with Envoy potentially starting with incomplete configuration by not waiting enough for initial xDS configuration. [GH-17317]
• grpc: ensure grpc resolver correctly uses lan/wan addresses on servers [GH-17270]
• peering: Fix issue where peer streams could incorrectly deregister services in various scenarios. [GH-17235]
• proxycfg: ensure that an irrecoverable error in proxycfg closes the xds session and triggers a replacement proxycfg watcher [GH-16497]
• xds: Fix possible panic that can when generating clusters before the root certificates have been fetched. [GH-17185]

BUG FIXES:

• audit-logging: (Enterprise only) Fix a bug where /agent/monitor and /agent/metrics endpoints return a Streaming not supported error when audit logs are enabled. This also fixes the delay receiving logs when running consul monitor against an agent with audit logs enabled. [GH-16700]
• ca: Fixes a bug where updating Vault CA Provider config would cause TLS issues in the service mesh [GH-16592]
• peering: (Consul Enterprise only) Fix issue where connect-enabled services with peer upstreams incorrectly required service:write access in the default namespace to query data, which was too restrictive. Now having service:write to any namespace is sufficient to query the peering data.
• peering: (Consul Enterprise only) Fix issue where resolvers, routers, and splitters referencing peer targets may not work correctly for non-default partitions and namespaces. Enterprise customers leveraging peering are encouraged to upgrade both servers and agents to avoid this problem.
• peering: Fix issue resulting in prepared query failover to cluster peers never un-failing over. [GH-16729]
• peering: Fixes a bug that can lead to peering service deletes impacting the state of local services [GH-16570]
• peering: Fixes a bug where the importing partition was not added to peered failover targets, which causes issues when the importing partition is a non-default partition. [GH-16693]
• ui: fix PUT token request with adding missed AccessorID property to requestBody [GH-16660]

SECURITY:

• Upgrade to use Go 1.20.1. This resolves vulnerabilities CVE-2022-41724 in crypto/tls and CVE-2022-41723 in net/http. [GH-16263]

IMPROVEMENTS:

• container: Upgrade container image to use to Alpine 3.17. [GH-16358]
• mesh: Add ServiceResolver RequestTimeout for route timeouts to make request timeouts configurable [GH-16495]

BUG FIXES:

• mesh: Fix resolution of service resolvers with subsets for external upstreams [GH-16499]
• peering: Fix bug where services were incorrectly imported as connect-enabled. [GH-16339]
• peering: Fix issue where mesh gateways would use the wrong address when contacting a remote peer with the same datacenter name. [GH-16257]
• peering: Fix issue where secondary wan-federated datacenters could not be used as peering acceptors. [GH-16230]
• proxycfg: fix a bug where terminating gateways were not cleaning up deleted service resolvers for their referenced services [GH-16498]

BREAKING CHANGES:

• connect: Fix configuration merging for transparent proxy upstreams. Proxy-defaults and service-defaults config entries were not correctly merged for implicit upstreams in transparent proxy mode and would result in some configuration not being applied. To avoid issues when upgrading, ensure that any proxy-defaults or service-defaults have correct configuration for upstreams, since all fields will now be properly used to configure proxies. [GH-16000]
• peering: Newly created peering connections must use only lowercase characters in the name field. Existing peerings with uppercase characters will not be modified, but they may encounter issues in various circumstances. To maintain forward compatibility and avoid issues, it is recommended to destroy and re-create any invalid peering connections so that they do not have a name containing uppercase characters. [GH-15697]

FEATURES:

• connect: add flags envoy-ready-bind-port and envoy-ready-bind-address to the consul connect envoy command that allows configuration of readiness probe on proxy for any service kind. [GH-16015]
• deps: update to latest go-discover to provide ECS auto-discover capabilities. [GH-13782]

IMPROVEMENTS:

• acl: relax permissions on the WatchServers, WatchRoots and GetSupportedDataplaneFeatures gRPC endpoints to accept any valid ACL token [GH-15346]
• connect: Add support for ConsulResolver to specifies a filter expression [GH-15659]
• grpc: Use new balancer implementation to reduce periodic WARN logs when shuffling servers. [GH-15701]
• partition: (Consul Enterprise only) when loading service from on-disk config file or sending API request to agent endpoint, if the partition is unspecified, consul will default the partition in the request to agent's partition [GH-16024]

BUG FIXES:

• agent: Fix assignment of error when auto-reloading cert and key file changes. [GH-15769]
• agent: Fix issue where the agent cache would incorrectly mark protobuf objects as updated. [GH-15866]
• cli: Fix issue where consul connect envoy was unable to configure TLS over unix-sockets to gRPC. [GH-15913]
• connect: (Consul Enterprise only) Fix issue where upstream configuration from proxy-defaults and service-defaults was not properly merged. This could occur when a mixture of empty-strings and "default" were used for the namespace or partition fields.
• connect: Fix issue where service-resolver protocol checks incorrectly errored for failover peer targets. [GH-15833]
• connect: Fix issue where watches on upstream failover peer targets did not always query the correct data. [GH-15865]
• xds: fix bug where sessions for locally-managed services could fail with "this server has too many xDS streams open" [GH-15789]

SECURITY:

• Upgrade to use Go 1.19.4. This resolves a vulnerability where restricted files can be read on Windows. CVE-2022-41720 [GH-15705]
• Upgrades golang.org/x/net to prevent a denial of service by excessive memory usage caused by HTTP2 requests. CVE-2022-41717 [GH-15737]

FEATURES:

• ui: Add field for fallback server addresses to peer token generation form [GH-15555]

IMPROVEMENTS:

• connect: ensure all vault connect CA tests use limited privilege tokens [GH-15669]

BUG FIXES:

• agent: (Enterprise Only) Ensure configIntentionsConvertToList does not compare empty strings with populated strings when filtering intentions created prior to AdminPartitions.
• connect: Fix issue where DialedDirectly configuration was not used by Consul Dataplane. [GH-15760]
• connect: Fix peering failovers ignoring local mesh gateway configuration. [GH-15690]
• connect: Fixed issue where using Vault 1.11+ as CA provider in a secondary datacenter would eventually break Intermediate CAs [GH-15661]

FEATURES:

• connect: Add local_idle_timeout_ms to allow configuring the Envoy route idle timeout on local_app connect: Add IdleTimeout to service-router to allow configuring the Envoy route idle timeout [GH-14340]
• snapshot: (Enterprise Only) Add support for the snapshot agent to use an IAM role for authentication/authorization when managing snapshots in S3.

IMPROVEMENTS:

• dns: Add support for cluster peering .service and .node DNS queries. [GH-15596]

BUG FIXES:

• acl: avoid debug log spam in secondary datacenter servers due to management token not being initialized. [GH-15610]
• agent: Fixed issue where blocking queries with short waits could timeout on the client [GH-15541]
• ca: Fixed issue where using Vault as Connect CA with Vault-managed policies would error on start-up if the intermediate PKI mount existed but was empty [GH-15525]
• cli: (Enterprise Only) Fix issue where consul partition update subcommand was not registered and therefore not available through the cli.
• connect: Fixed issue where using Vault 1.11+ as CA provider would eventually break Intermediate CAs in primary datacenters. A separate fix is needed to address the same issue in secondary datacenters. [GH-15217] [GH-15253]
• namespace: (Enterprise Only) Fix a bug that caused blocking queries during namespace replication to timeout
• peering: better represent non-passing states during peer check flattening [GH-15615]
• peering: fix the limit of replication gRPC message; set to 8MB [GH-15503]

BUG FIXES:

• cli: Fix issue where consul connect envoy incorrectly uses the HTTPS API configuration for xDS connections. [GH-15466]
• sdk: Fix SDK testutil backwards compatibility by only configuring grpc_tls port for new Consul versions. [GH-15423]

KNOWN ISSUES:

• cli: consul connect envoy incorrectly enables TLS for gRPC connections when the HTTP API is TLS-enabled.

BREAKING CHANGES:

• config: Add new ports.grpc_tls configuration option. Introduce a new port to better separate TLS config from the existing ports.grpc config. The new ports.grpc_tls only supports TLS encrypted communication. The existing ports.grpc now only supports plain-text communication. [GH-15339]
• config: update 1.14 config defaults: Enable peering and connect by default. [GH-15302]
• config: update 1.14 config defaults: Set gRPC TLS port default value to 8503 [GH-15302]
• connect: Removes support for Envoy 1.20 [GH-15093]
• peering: Rename PeerName to Peer on prepared queries and exported services. [GH-14854]
• xds: Convert service mesh failover to use Envoy's aggregate clusters. This changes the names of some Envoy dynamic HTTP metrics. [GH-14178]

SECURITY:

• Ensure that data imported from peers is filtered by ACLs at the UI Nodes/Services endpoints CVE-2022-3920 [GH-15356]

FEATURES:

• DNS-proxy support via gRPC request. [GH-14811]
• cli: Add -node-name flag to redirect-traffic command to support running in environments without client agents. [GH-14933]
• cli: Add -consul-dns-port flag to the consul connect redirect-traffic command to allow forwarding DNS traffic to a specific Consul DNS port. [GH-15050]
• connect: Add Envoy connection balancing configuration fields. [GH-14616]
• grpc: Added metrics for external gRPC server. Added server_type=internal|external label to gRPC metrics. [GH-14922]
• http: Add new get-or-empty operation to the txn api. Refer to the API docs for more information. [GH-14474]
• peering: Add mesh gateway local mode support for cluster peering. [GH-14817]
• peering: Add support for stale queries for trust bundle lookups [GH-14724]
• peering: Add support to failover to services running on cluster peers. [GH-14396]
• peering: Add support to redirect to services running on cluster peers with service resolvers. [GH-14445]
• peering: Ensure un-exported services get deleted even if the un-export happens while cluster peering replication is down. [GH-14797]
• peering: add support for routine peering control-plane traffic through mesh gateways [GH-14981]
• sdk: Configure iptables to forward DNS traffic to a specific DNS port. [GH-15050]
• telemetry: emit memberlist size metrics and broadcast queue depth metric. [GH-14873]
• ui: Added support for central config merging [GH-14604]
• ui: Create peerings detail page [GH-14947]
• ui: Detect a TokenSecretID cookie and passthrough to localStorage [GH-14495]
• ui: Display notice banner on nodes index page if synthetic nodes are being filtered. [GH-14971]
• ui: Filter agentless (synthetic) nodes from the nodes list page. [GH-14970]
• ui: Filter out node health checks on agentless service instances [GH-14986]
• ui: Remove node meta on service instances when using agentless and consolidate external-source labels on service instances page if they all match. [GH-14921]
• ui: Removed reference to node name on service instance page when using agentless [GH-14903]
• ui: Use withCredentials for all HTTP API requests [GH-14343]
• xds: servers will limit the number of concurrent xDS streams they can handle to balance the load across all servers [GH-14397]

IMPROVEMENTS:

• peering: Add peering datacenter and partition to initial handshake. [GH-14889]
• xds: Added a rate limiter to the delivery of proxy config updates, to prevent updates to "global" resources such as wildcard intentions from overwhelming servers (see: xds.update_max_per_second config field) [GH-14960]
• xds: Removed a bottleneck in Envoy config generation, enabling a higher number of dataplanes per server [GH-14934]
• agent/hcp: add initial HashiCorp Cloud Platform integration [GH-14723]
• agent: Added configuration option cloud.scada_address. [GH-14936]
• api: Add filtering support to Catalog's List Services (v1/catalog/services) [GH-11742]
• api: Increase max number of operations inside a transaction for requests to /v1/txn (128) [GH-14599]
• auto-config: Relax the validation on auto-config JWT authorization to allow non-whitespace, non-quote characters in node names. [GH-15370]
• config-entry: Validate that service-resolver Failovers and Redirects only specify Partition and Namespace on Consul Enterprise. This prevents scenarios where OSS Consul would save service-resolvers that require Consul Enterprise. [GH-14162]
• connect: Add Envoy 1.24.0 to support matrix [GH-15093]
• connect: Bump Envoy 1.20 to 1.20.7, 1.21 to 1.21.5 and 1.22 to 1.22.5 [GH-14831]
• connect: service-router destinations have gained a RetryOn field for specifying the conditions when Envoy should retry requests beyond specific status codes and generic connection failure which already exists. [GH-12890]
• dns/peering: (Enterprise Only) Support addresses in the formats <servicename>.virtual.<namespace>.ns.<partition>.ap.<peername>.peer.consul and <servicename>.virtual.<partition>.ap.<peername>.peer.consul. This longer form address that allows specifying .peer would need to be used for tproxy DNS requests made within non-default partitions for imported services.
• dns: (Enterprise Only) All enterprise locality labels are now optional in DNS lookups. For example, service lookups support the following format: [<tag>.]<service>.service[.<namespace>.ns][.<partition>.ap][.<datacenter>.dc]<domain>. [GH-14679]
• integ test: fix flakiness due to test condition from retry app endoint [GH-15233]
• metrics: Service RPC calls less than 1ms are now emitted as a decimal number. [GH-12905]
• peering: adds an internally managed server certificate for automatic TLS between servers in peer clusters. [GH-14556]
• peering: require TLS for peering connections using server cert signed by Connect CA [GH-14796]
• peering: return information about the health of the peering when the leader is queried to read a peering. [GH-14747]
• raft: Allow nonVoter to initiate an election to avoid having an election infinite loop when a Voter is converted to NonVoter [GH-14897]
• raft: Cap maximum grpc wait time when heartbeating to heartbeatTimeout/2 [GH-14897]
• raft: Fix a race condition where the snapshot file is closed without being opened [GH-14897]
• telemetry: Added a consul.xds.server.streamStart metric to measure time taken to first generate xDS resources for an xDS stream. [GH-14957]
• ui: Improve guidance around topology visualisation [GH-14527]
• xds: Set max_ejection_percent on Envoy's outlier detection to 100% for peered services. [GH-14373]
• xds: configure Envoy alpn_protocols for connect-proxy and ingress-gateway based on service protocol. [GH-14356]

BUG FIXES:

• checks: Do not set interval as timeout value [GH-14619]
• checks: If set, use proxy address for automatically added sidecar check instead of service address. [GH-14433]
• cli: Fix Consul kv CLI 'GET' flags 'keys' and 'recurse' to be set together [GH-13493]
• connect: Fix issue where mesh-gateway settings were not properly inherited from configuration entries. [GH-15186]
• connect: fixed bug where endpoint updates for new xDS clusters could block for 15s before being sent to Envoy. [GH-15083]
• connect: strip port from DNS SANs for ingress gateway leaf certificate to avoid an invalid hostname error when using the Vault provider. [GH-15320]
• debug: fixed bug that caused consul debug CLI to error on ACL-disabled clusters [GH-15155]
• deps: update go-memdb, fixing goroutine leak [GH-15010] [GH-15068]
• grpc: Merge proxy-defaults and service-defaults in GetEnvoyBootstrapParams response. [GH-14869]
• metrics: Add duplicate metrics that have only a single "consul_" prefix for all existing metrics with double ("consul_consul_") prefix, with the intent to standardize on single prefixes. [GH-14475]
• namespace: (Enterprise Only) Fixed a bug where a client may incorrectly log that namespaces were not enabled in the local datacenter
• peering: Fix a bug that resulted in /v1/agent/metrics returning an error. [GH-15178]
• peering: fix nil pointer in calling handleUpdateService [GH-15160]
• peering: fix the error of wan address isn't taken by the peering token. [GH-15065]
• peering: when wan address is set, peering stream should use the wan address. [GH-15108]
• proxycfg(mesh-gateway): Fix issue where deregistered services are not removed from mesh-gateway clusters. [GH-15272]
• server: fix goroutine/memory leaks in the xDS subsystem (these were present regardless of whether or not xDS was in-use) [GH-14916]
• server: fixes the error trying to source proxy configuration for http checks, in case of proxies using consul-dataplane. [GH-14924]
• xds: Central service configuration (proxy-defaults and service-defaults) is now correctly applied to Consul Dataplane proxies [GH-14962]

NOTES:

• deps: Upgrade to use Go 1.19.2 [GH-15090]

BREAKING CHANGES:

• connect: Disable peering by default in connect proxies for Consul 1.13. This change was made to prevent inefficient polling queries from having a negative impact on server performance. Peering in Consul 1.13 is an experimental feature and is not recommended for use in production environments. If you still wish to use the experimental peering feature, ensure peering.enabled = true is set on all clients and servers. [GH-17731]

SECURITY:

• Update to UBI base image to 9.2. [GH-17513]

FEATURES:

• server: (Enterprise Only) allow automatic license utilization reporting. [GH-5102]

IMPROVEMENTS:

• debug: change default setting of consul debug command. now default duration is 5ms and default log level is 'TRACE' [GH-17596]
• systemd: set service type to notify. [GH-16845]

BUG FIXES:

• cache: fix a few minor goroutine leaks in leaf certs and the agent cache [GH-17636]
• namespaces: (Enterprise only) fixes a bug where namespaces are stuck in a deferred deletion state indefinitely under some conditions. Also fixes the Consul query metadata present in the HTTP headers of the namespace read and list endpoints.
• namespaces: adjusts the return type from HTTP list API to return the api module representation of a namespace. This fixes an error with the consul namespace list command when a namespace has a deferred deletion timestamp.
• peering: Fix a bug that caused server agents to continue cleaning up peering resources even after loss of leadership. [GH-17483]

SECURITY:

• Upgrade to use Go 1.20.1. This resolves vulnerabilities CVE-2022-41724 in crypto/tls and CVE-2022-41723 in net/http. [GH-16263]
• Upgrade to use Go 1.20.4. This resolves vulnerabilities CVE-2023-24537(go/scanner), CVE-2023-24538(html/template), CVE-2023-24534(net/textproto) and CVE-2023-24536(mime/multipart). Also, golang.org/x/net has been updated to v0.7.0 to resolve CVEs CVE-2022-41721 , CVE-2022-27664 and [CVE-2022-41723 ](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h .) [GH-17240]

IMPROVEMENTS:

• api: updated the go module directive to 1.18. [GH-15297]
• connect: update supported envoy versions to 1.20.7, 1.21.6, 1.22.11, 1.23.8 [GH-16891]
• sdk: updated the go module directive to 1.18. [GH-15297]

BUG FIXES:

• Fix an bug where decoding some Config structs with unset pointer fields could fail with reflect: call of reflect.Value.Type on zero Value. [GH-17048]
• audit-logging: (Enterprise only) Fix a bug where /agent/monitor and /agent/metrics endpoints return a Streaming not supported error when audit logs are enabled. This also fixes the delay receiving logs when running consul monitor against an agent with audit logs enabled. [GH-16700]
• ca: Fixes a bug where updating Vault CA Provider config would cause TLS issues in the service mesh [GH-16592]
• connect: Fix multiple inefficient behaviors when querying service health. [GH-17241]
• grpc: ensure grpc resolver correctly uses lan/wan addresses on servers [GH-17270]
• peering: Fixes a bug that can lead to peering service deletes impacting the state of local services [GH-16570]
• xds: Fix possible panic that can when generating clusters before the root certificates have been fetched. [GH-17185]

SECURITY:

• Upgrade to use Go 1.19.6. This resolves vulnerabilities CVE-2022-41724 in crypto/tls and CVE-2022-41723 in net/http. [GH-16299]

IMPROVEMENTS:

• xds: Removed a bottleneck in Envoy config generation. [GH-16269]
• container: Upgrade container image to use to Alpine 3.17. [GH-16358]
• mesh: Add ServiceResolver RequestTimeout for route timeouts to make request timeouts configurable [GH-16495]

BUG FIXES:

• mesh: Fix resolution of service resolvers with subsets for external upstreams [GH-16499]
• proxycfg: fix a bug where terminating gateways were not cleaning up deleted service resolvers for their referenced services [GH-16498]

FEATURES:

• connect: add flags envoy-ready-bind-port and envoy-ready-bind-address to the consul connect envoy command that allows configuration of readiness probe on proxy for any service kind. [GH-16015]
• deps: update to latest go-discover to provide ECS auto-discover capabilities. [GH-13782]

IMPROVEMENTS:

• grpc: Use new balancer implementation to reduce periodic WARN logs when shuffling servers. [GH-15701]
• partition: (Consul Enterprise only) when loading service from on-disk config file or sending API request to agent endpoint, if the partition is unspecified, consul will default the partition in the request to agent's partition [GH-16024]

BUG FIXES:

• agent: Fix assignment of error when auto-reloading cert and key file changes. [GH-15769]

SECURITY:

• Upgrade to use Go 1.18.9. This resolves a vulnerability where restricted files can be read on Windows. CVE-2022-41720 [GH-15706]
• Upgrades golang.org/x/net to prevent a denial of service by excessive memory usage caused by HTTP2 requests. CVE-2022-41717 [GH-15743]

IMPROVEMENTS:

• connect: ensure all vault connect CA tests use limited privilege tokens [GH-15669]

BUG FIXES:

• agent: (Enterprise Only) Ensure configIntentionsConvertToList does not compare empty strings with populated strings when filtering intentions created prior to AdminPartitions.
• cli: (Enterprise Only) Fix issue where consul partition update subcommand was not registered and therefore not available through the cli.
• connect: Fixed issue where using Vault 1.11+ as CA provider in a secondary datacenter would eventually break Intermediate CAs [GH-15661]

IMPROVEMENTS:

• auto-config: Relax the validation on auto-config JWT authorization to allow non-whitespace, non-quote characters in node names. [GH-15370]
• raft: Allow nonVoter to initiate an election to avoid having an election infinite loop when a Voter is converted to NonVoter [GH-14897]
• raft: Cap maximum grpc wait time when heartbeating to heartbeatTimeout/2 [GH-14897]
• raft: Fix a race condition where the snapshot file is closed without being opened [GH-14897]

BUG FIXES:

• agent: Fixed issue where blocking queries with short waits could timeout on the client [GH-15541]
• ca: Fixed issue where using Vault as Connect CA with Vault-managed policies would error on start-up if the intermediate PKI mount existed but was empty [GH-15525]
• connect: Fixed issue where using Vault 1.11+ as CA provider would eventually break Intermediate CAs in primary datacenters. A separate fix is needed to address the same issue in secondary datacenters. [GH-15217] [GH-15253]
• connect: fixed bug where endpoint updates for new xDS clusters could block for 15s before being sent to Envoy. [GH-15083]
• connect: strip port from DNS SANs for ingress gateway leaf certificate to avoid an invalid hostname error when using the Vault provider. [GH-15320]
• debug: fixed bug that caused consul debug CLI to error on ACL-disabled clusters [GH-15155]
• deps: update go-memdb, fixing goroutine leak [GH-15010] [GH-15068]
• namespace: (Enterprise Only) Fix a bug that caused blocking queries during namespace replication to timeout
• namespace: (Enterprise Only) Fixed a bug where a client may incorrectly log that namespaces were not enabled in the local datacenter
• peering: better represent non-passing states during peer check flattening [GH-15615]
• peering: fix the error of wan address isn't taken by the peering token. [GH-15065]
• peering: when wan address is set, peering stream should use the wan address. [GH-15108]

FEATURES:

• agent: Added a new config option rpc_client_timeout to tune timeouts for client RPC requests [GH-14965]
• config-entry(ingress-gateway): Added support for max_connections for upstream clusters [GH-14749]

IMPROVEMENTS:

• connect/ca: Log a warning message instead of erroring when attempting to update the intermediate pki mount when using the Vault provider. [GH-15035]
• connect: Added gateway options to Envoy proxy config for enabling tcp keepalives on terminating gateway upstreams and mesh gateways in remote datacenters. [GH-14800]
• connect: Bump Envoy 1.20 to 1.20.7, 1.21 to 1.21.5 and 1.22 to 1.22.5 [GH-14828]
• licensing: (Enterprise Only) Consul Enterprise production licenses do not degrade or terminate Consul upon expiration. They will only fail when trying to upgrade to a newer version of Consul. Evaluation licenses still terminate. [GH-1990]

BUG FIXES:

• agent: avoid leaking the alias check runner goroutine when the check is de-registered [GH-14935]
• ca: fix a masked bug in leaf cert generation that would not be notified of root cert rotation after the first one [GH-15005]
• cache: prevent goroutine leak in agent cache [GH-14908]
• checks: Fixed a bug that prevented registration of UDP health checks from agent configuration files, such as service definition files with embedded health check definitions. [GH-14885]
• connect: Fixed a bug where transparent proxy does not correctly spawn listeners for upstreams to service-resolvers. [GH-14751]
• snapshot-agent: (Enterprise only) Fix a bug when a session is not found in Consul, which leads the agent to panic.

BREAKING CHANGES:

• ca: If using Vault as the service mesh CA provider, the Vault policy used by Consul now requires the update capability on the intermediate PKI's tune mount configuration endpoint, such as /sys/mounts/connect_inter/tune. The breaking nature of this change is resolved in 1.13.3. Refer to upgrade guidance for more information.

SECURITY:

• auto-config: Added input validation for auto-config JWT authorization checks. Prior to this change, it was possible for malicious actors to construct requests which incorrectly pass custom JWT claim validation for the AutoConfig.InitialConfiguration endpoint. Now, only a subset of characters are allowed for the input before evaluating the bexpr. [GH-14577]
• connect: Added URI length checks to ConnectCA CSR requests. Prior to this change, it was possible for a malicious actor to designate multiple SAN URI values in a call to the ConnectCA.Sign endpoint. The endpoint now only allows for exactly one SAN URI to be specified. [GH-14579]

FEATURES:

• cli: Adds new subcommands for peering workflows. Refer to the CLI docs for more information. [GH-14423]
• connect: Server address changes are streamed to peers [GH-14285]
• service-defaults: Added support for local_request_timeout_ms and local_connect_timeout_ms in servicedefaults config entry [GH-14395]

IMPROVEMENTS:

• connect: Bump latest Envoy to 1.23.1 in test matrix [GH-14573]
• connect: expose new tracing configuration on envoy [GH-13998]
• envoy: adds additional Envoy outlier ejection parameters to passive health check configurations. [GH-14238]
• metrics: add labels of segment, partition, network area, network (lan or wan) to serf and memberlist metrics [GH-14161]
• peering: Validate peering tokens for server name conflicts [GH-14563]
• snapshot agent: (Enterprise only) Add support for path-based addressing when using s3 backend.
• ui: Reuse connections for requests to /v1/internal/ui/metrics-proxy/ [GH-14521]

BUG FIXES:

• agent: Fixes an issue where an agent that fails to start due to bad addresses won't clean up any existing listeners [GH-14081]
• api: Fix a breaking change caused by renaming QueryDatacenterOptions to QueryFailoverOptions. This adds QueryDatacenterOptions back as an alias to QueryFailoverOptions and marks it as deprecated. [GH-14378]
• ca: Fixed a bug with the Vault CA provider where the intermediate PKI mount and leaf cert role were not being updated when the CA configuration was changed. [GH-14516]
• cli: When launching a sidecar proxy with consul connect envoy or consul connect proxy, the -sidecar-for service ID argument is now treated as case-insensitive. [GH-14034]
• connect: Fix issue where auto_config and auto_encrypt could unintentionally enable TLS for gRPC xDS connections. [GH-14269]
• connect: Fixed a bug where old root CAs would be removed from the primary datacenter after switching providers and restarting the cluster. [GH-14598]
• connect: Fixed an issue where intermediate certificates could build up in the root CA because they were never being pruned after expiring. [GH-14429]
• connect: Fixed some spurious issues during peering establishment when a follower is dialed [GH-14119]
• envoy: validate name before deleting proxy default configurations. [GH-14290]
• peering: Fix issue preventing deletion and recreation of peerings in TERMINATED state. [GH-14364]
• rpc: Adds max jitter to client deadlines to prevent i/o deadline errors on blocking queries [GH-14233]
• tls: undo breaking change that prevented setting TLS for gRPC when using config flags available in Consul v1.11. [GH-14668]
• ui: Removed Overview page from HCP instalations [GH-14606]

BUG FIXES:

• agent: Fixed a compatibility issue when restoring snapshots from pre-1.13.0 versions of Consul [GH-14107] [GH-14149]
• connect: Fixed some spurious issues during peering establishment when a follower is dialed [GH-14119]

BREAKING CHANGES:

• config-entry: Exporting a specific service name across all namespace is invalid.
• connect: contains an upgrade compatibility issue when restoring snapshots containing service mesh proxy registrations from pre-1.13 versions of Consul [GH-14107]. Fixed in 1.13.1 [GH-14149]. Refer to 1.13 upgrade guidance for more information.
• connect: if using auto-encrypt or auto-config, TLS is required for gRPC communication between Envoy and Consul as of 1.13.0; this TLS for gRPC requirement will be removed in a future 1.13 patch release. Refer to 1.13 upgrade guidance for more information.
• connect: if a pre-1.13 Consul agent's HTTPS port was not enabled, upgrading to 1.13 may turn on TLS for gRPC communication for Envoy and Consul depending on the agent's TLS configuration. Refer to 1.13 upgrade guidance for more information.
• connect: Removes support for Envoy 1.19 [GH-13807]
• telemetry: config flag telemetry { disable_compat_1.9 = (true|false) } has been removed. Before upgrading you should remove this flag from your config if the flag is being used. [GH-13532]

FEATURES:

• Cluster Peering (Beta) This version adds a new model to federate Consul clusters for both service mesh and traditional service discovery. Cluster peering allows for service interconnectivity with looser coupling than the existing WAN federation. For more information refer to the cluster peering documentation.
• Transparent proxying through terminating gateways This version adds egress traffic control to destinations outside of Consul's catalog, such as APIs on the public internet. Transparent proxies can dial destinations defined in service-defaults and have the traffic routed through terminating gateways. For more information refer to the terminating gateway documentation.
• acl: It is now possible to login and logout using the gRPC API [GH-12935]
• agent: Added information about build date alongside other version information for Consul. Extended /agent/self endpoint and consul version commands to report this. Agent also reports build date in log on startup. [GH-13357]
• ca: Leaf certificates can now be obtained via the gRPC API: Sign [GH-12787]
• checks: add UDP health checks.. [GH-12722]
• cli: A new flag for config delete to delete a config entry in a valid config file, e.g., config delete -filename intention-allow.hcl [GH-13677]
• connect: Adds a new destination field to the service-default config entry that allows routing egress traffic through a terminating gateway in transparent proxy mode without modifying the catalog. [GH-13613]
• grpc: New gRPC endpoint to return envoy bootstrap parameters. [GH-12825]
• grpc: New gRPC endpoint to return envoy bootstrap parameters. [GH-1717]
• grpc: New gRPC service and endpoint to return the list of supported consul dataplane features [GH-12695]
• server: broadcast the public grpc port using lan serf and update the consul service in the catalog with the same data [GH-13687]
• streaming: Added topic that can be used to consume updates about the list of services in a datacenter [GH-13722]
• streaming: Added topics for ingress-gateway, mesh, service-intentions and service-resolver config entry events. [GH-13658]

IMPROVEMENTS:

• api: merge-central-config query parameter support added to /catalog/node-services/:node-name API, to view a fully resolved service definition (especially when not written into the catalog that way). [GH-13450]
• api: merge-central-config query parameter support added to /catalog/node-services/:node-name API, to view a fully resolved service definition (especially when not written into the catalog that way). [GH-2046]
• api: merge-central-config query parameter support added to some catalog and health endpoints to view a fully resolved service definition (especially when not written into the catalog that way). [GH-13001]
• api: add the ability to specify a path prefix for when consul is behind a reverse proxy or API gateway [GH-12914]
• catalog: Add per-node indexes to reduce watchset firing for unrelated nodes and services. [GH-12399]
• connect: add validation to ensure connect native services have a port or socketpath specified on catalog registration. This was the only missing piece to ensure all mesh services are validated for a port (or socketpath) specification on catalog registration. [GH-12881]
• ui: Add new CopyableCode component and use it in certain pre-existing areas [GH-13686]
• acl: Clarify node/service identities must be lowercase [GH-12807]
• command: Add support for enabling TLS in the Envoy Prometheus endpoint via the consul connect envoy command. Adds the -prometheus-ca-file, -prometheus-ca-path, -prometheus-cert-file and -prometheus-key-file flags. [GH-13481]
• connect: Add Envoy 1.23.0 to support matrix [GH-13807]
• connect: Added a max_inbound_connections setting to service-defaults for limiting the number of concurrent inbound connections to each service instance. [GH-13143]
• grpc: Add a new ServerDiscovery.WatchServers gRPC endpoint for being notified when the set of ready servers has changed. [GH-12819]
• telemetry: Added consul.raft.thread.main.saturation and consul.raft.thread.fsm.saturation metrics to measure approximate saturation of the Raft goroutines [GH-12865]
• ui: removed external dependencies for serving UI assets in favor of Go's native embed capabilities [GH-10996]
• ui: upgrade ember-composable-helpers to v5.x [GH-13394]

BUG FIXES:

• acl: Fixed a bug where the ACL down policy wasn't being applied on remote errors from the primary datacenter. [GH-12885]
• cli: when acl token read is used with the -self and -expanded flags, return an error instead of panicking [GH-13787]
• connect: Fixed a goroutine/memory leak that would occur when using the ingress gateway. [GH-13847]
• connect: Ingress gateways with a wildcard service entry should no longer pick up non-connect services as upstreams. connect: Terminating gateways with a wildcard service entry should no longer pick up connect services as upstreams. [GH-13958]
• proxycfg: Fixed a minor bug that would cause configuring a terminating gateway to watch too many service resolvers and waste resources doing filtering. [GH-13012]
• raft: upgrade to v1.3.8 which fixes a bug where non cluster member can still be able to participate in an election. [GH-12844]
• rpc: Adds a deadline to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions. [GH-8504] [GH-11500]
• serf: upgrade serf to v0.9.8 which fixes a bug that crashes Consul when serf keyrings are listed [GH-13062]
• ui: Fixes an issue where client side validation errors were not showing in certain areas [GH-14021]

FEATURES:

• deps: update to latest go-discover to provide ECS auto-discover capabilities. [GH-13782]

IMPROVEMENTS:

• grpc: Use new balancer implementation to reduce periodic WARN logs when shuffling servers. [GH-15701]

BUG FIXES:

• agent: Fix assignment of error when auto-reloading cert and key file changes. [GH-15769]

SECURITY:

• Upgrade to use Go 1.18.9. This resolves a vulnerability where restricted files can be read on Windows. CVE-2022-41720 [GH-15727]
• Upgrades golang.org/x/net to prevent a denial of service by excessive memory usage caused by HTTP2 requests. CVE-2022-41717 [GH-15746]

IMPROVEMENTS:

• connect: ensure all vault connect CA tests use limited privilege tokens [GH-15669]

BUG FIXES:

• agent: (Enterprise Only) Ensure configIntentionsConvertToList does not compare empty strings with populated strings when filtering intentions created prior to AdminPartitions.
• cli: (Enterprise Only) Fix issue where consul partition update subcommand was not registered and therefore not available through the cli.
• connect: Fixed issue where using Vault 1.11+ as CA provider in a secondary datacenter would eventually break Intermediate CAs [GH-15661]

BUG FIXES:

• agent: Fixed issue where blocking queries with short waits could timeout on the client [GH-15541]
• ca: Fixed issue where using Vault as Connect CA with Vault-managed policies would error on start-up if the intermediate PKI mount existed but was empty [GH-15525]
• connect: Fixed issue where using Vault 1.11+ as CA provider would eventually break Intermediate CAs in primary datacenters. A separate fix is needed to address the same issue in secondary datacenters. [GH-15217] [GH-15253]
• connect: fixed bug where endpoint updates for new xDS clusters could block for 15s before being sent to Envoy. [GH-15083]
• connect: strip port from DNS SANs for ingress gateway leaf certificate to avoid an invalid hostname error when using the Vault provider. [GH-15320]
• debug: fixed bug that caused consul debug CLI to error on ACL-disabled clusters [GH-15155]
• deps: update go-memdb, fixing goroutine leak [GH-15010] [GH-15068]
• namespace: (Enterprise Only) Fix a bug that caused blocking queries during namespace replication to timeout
• namespace: (Enterprise Only) Fixed a bug where a client may incorrectly log that namespaces were not enabled in the local datacenter

FEATURES:

• agent: Added a new config option rpc_client_timeout to tune timeouts for client RPC requests [GH-14965]
• agent: Added information about build date alongside other version information for Consul. Extended /agent/self endpoint and consul version commands to report this. Agent also reports build date in log on startup. [GH-13357]
• config-entry(ingress-gateway): Added support for max_connections for upstream clusters [GH-14749]

IMPROVEMENTS:

• connect/ca: Log a warning message instead of erroring when attempting to update the intermediate pki mount when using the Vault provider. [GH-15035]
• connect: Added gateway options to Envoy proxy config for enabling tcp keepalives on terminating gateway upstreams and mesh gateways in remote datacenters. [GH-14800]
• connect: Bump Envoy 1.20 to 1.20.7, 1.21 to 1.21.5 and 1.22 to 1.22.5 [GH-14829]
• licensing: (Enterprise Only) Consul Enterprise production licenses do not degrade or terminate Consul upon expiration. They will only fail when trying to upgrade to a newer version of Consul. Evaluation licenses still terminate. [GH-1990]

BUG FIXES:

• agent: avoid leaking the alias check runner goroutine when the check is de-registered [GH-14935]
• ca: fix a masked bug in leaf cert generation that would not be notified of root cert rotation after the first one [GH-15005]
• cache: prevent goroutine leak in agent cache [GH-14908]
• connect: Fixed a bug where transparent proxy does not correctly spawn listeners for upstreams to service-resolvers. [GH-14751]
• snapshot-agent: (Enterprise only) Fix a bug when a session is not found in Consul, which leads the agent to panic.

BREAKING CHANGES:

• ca: If using Vault as the service mesh CA provider, the Vault policy used by Consul now requires the update capability on the intermediate PKI's tune mount configuration endpoint, such as /sys/mounts/connect_inter/tune. The breaking nature of this change is resolved in 1.12.6. Refer to upgrade guidance for more information.

SECURITY:

• auto-config: Added input validation for auto-config JWT authorization checks. Prior to this change, it was possible for malicious actors to construct requests which incorrectly pass custom JWT claim validation for the AutoConfig.InitialConfiguration endpoint. Now, only a subset of characters are allowed for the input before evaluating the bexpr. [GH-14577]
• connect: Added URI length checks to ConnectCA CSR requests. Prior to this change, it was possible for a malicious actor to designate multiple SAN URI values in a call to the ConnectCA.Sign endpoint. The endpoint now only allows for exactly one SAN URI to be specified. [GH-14579]

IMPROVEMENTS:

• envoy: adds additional Envoy outlier ejection parameters to passive health check configurations. [GH-14238]
• metrics: add labels of segment, partition, network area, network (lan or wan) to serf and memberlist metrics [GH-14161]
• snapshot agent: (Enterprise only) Add support for path-based addressing when using s3 backend.
• ui: Reuse connections for requests to /v1/internal/ui/metrics-proxy/ [GH-14521]

BUG FIXES:

• ca: Fixed a bug with the Vault CA provider where the intermediate PKI mount and leaf cert role were not being updated when the CA configuration was changed. [GH-14516]
• cli: When launching a sidecar proxy with consul connect envoy or consul connect proxy, the -sidecar-for service ID argument is now treated as case-insensitive. [GH-14034]
• connect: Fixed a bug where old root CAs would be removed from the primary datacenter after switching providers and restarting the cluster. [GH-14598]
• connect: Fixed an issue where intermediate certificates could build up in the root CA because they were never being pruned after expiring. [GH-14429]
• envoy: validate name before deleting proxy default configurations. [GH-14290]
• rpc: Adds max jitter to client deadlines to prevent i/o deadline errors on blocking queries [GH-14233]
• ui: Removed Overview page from HCP instalations [GH-14606]

BUG FIXES:

• cli: when acl token read is used with the -self and -expanded flags, return an error instead of panicking [GH-13787]
• connect: Fixed a goroutine/memory leak that would occur when using the ingress gateway. [GH-13847]
• connect: Ingress gateways with a wildcard service entry should no longer pick up non-connect services as upstreams. connect: Terminating gateways with a wildcard service entry should no longer pick up connect services as upstreams. [GH-13958]
• ui: Fixes an issue where client side validation errors were not showing in certain areas [GH-14021]

IMPROVEMENTS:

• Support Vault namespaces in Connect CA by adding RootPKINamespace and IntermediatePKINamespace fields to the config. [GH-12904]
• connect: Update Envoy support matrix to latest patch releases (1.22.2, 1.21.3, 1.20.4, 1.19.5) [GH-13431]
• dns: Added support for specifying admin partition in node lookups. [GH-13421]
• telemetry: Added a consul.server.isLeader metric to track if a server is a leader or not. [GH-13304]

BUG FIXES:

• agent: Fixed a bug in HTTP handlers where URLs were being decoded twice [GH-13256]
• deps: Update go-grpc/grpc, resolving connection memory leak [GH-13051]
• fix a bug that caused an error when creating grpc or http2 ingress gateway listeners with multiple services [GH-13127]
• ui: Fix incorrect text on certain page empty states [GH-13409]
• xds: Fix a bug that resulted in Lambda services not using the payload-passthrough option as expected. [GH-13607]
• xds: Fix a bug where terminating gateway upstream clusters weren't configured properly when the service protocol was http2. [GH-13699]

BUG FIXES:

• kvs: Fixed a bug where query options were not being applied to KVS.Get RPC operations. [GH-13344]

FEATURES:

• xds: Add the ability to invoke AWS Lambdas through sidecar proxies. [GH-12956]

IMPROVEMENTS:

• config: introduce telemetry.retry_failed_connection in agent configuration to retry on failed connection to any telemetry backend. This prevents the agent from exiting if the given DogStatsD DNS name is unresolvable, for example. [GH-13091]
• sentinel: (Enterprise Only) Sentinel now uses SHA256 to generate policy ids
• xds: Envoy now inserts x-forwarded-client-cert for incoming proxy connections [GH-12878]

BUG FIXES:

• Fix a bug when configuring an add_headers directive named Host the header is not set for v1/internal/ui/metrics-proxy/ endpoint. [GH-13071]
• api: Fix a bug that causes partition to be ignored when creating a namespace [GH-12845]
• api: agent/self now returns version with +ent suffix for Enterprise Consul [GH-12961]
• areas: (Enterprise Only) Fixes a bug when using Yamux pool ( for servers version 1.7.3 and later), the entire pool was locked while connecting to a remote location, which could potentially take a long time. [GH-1368]
• ca: fix a bug that caused a non blocking leaf cert query after a blocking leaf cert query to block [GH-12820]
• config: fix backwards compatibility bug where setting the (deprecated) top-level verify_incoming option would enable TLS client authentication on the gRPC port [GH-13118]
• health: ensure /v1/health/service/:service endpoint returns the most recent results when a filter is used with streaming #12640 [GH-12640]
• rpc: Adds a deadline to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions. [GH-8504] [GH-11500]
• snapshot-agent: (Enterprise only) Fix a bug where providing the ACL token to the snapshot agent via a CLI or ENV variable without a license configured results in an error during license auto-retrieval.
• ui: Re-instate '...' icon for row actions [GH-13183]

NOTES:

• ci: change action to pull v1 instead of main [GH-12846]

BREAKING CHANGES:

• connect: Removes support for Envoy 1.17.4 [GH-12777]
• connect: Removes support for Envoy 1.18.6 [GH-12805]
• sdk: several changes to the testutil configuration structs (removed ACLMasterToken, renamed Master to InitialManagement, and AgentMaster to AgentRecovery) [GH-11827]
• telemetry: the disable_compat_1.9 option now defaults to true. 1.9 style consul.http... metrics can still be enabled by setting disable_compat_1.9 = false. However, we will remove these metrics in 1.13. [GH-12675]

FEATURES:

• acl: Add token information to PermissionDeniedErrors [GH-12567]
• acl: Added an AWS IAM auth method that allows authenticating to Consul using AWS IAM identities [GH-12583]
• ca: Root certificates can now be consumed from a gRPC streaming endpoint: WatchRoots [GH-12678]
• cli: The token read command now supports the -expanded flag to display detailed role and policy information for the token. [GH-12670]
• config: automatically reload config when a file changes using the auto-reload-config CLI flag or auto_reload_config config option. [GH-12329]
• server: Ensure that service-defaults Meta is returned with the response to the ConfigEntry.ResolveServiceConfig RPC. [GH-12529]
• server: discovery chains now include a response field named "Default" to indicate if they were not constructed from any service-resolver, service-splitter, or service-router config entries [GH-12511]
• server: ensure that service-defaults meta is incorporated into the discovery chain response [GH-12511]
• tls: it is now possible to configure TLS differently for each of Consul's listeners (i.e. HTTPS, gRPC and the internal multiplexed RPC listener) using the tls stanza [GH-12504]
• ui: Added support for AWS IAM Auth Methods [GH-12786]
• ui: Support connect-native services in the Topology view. [GH-12098]
• xds: Add the ability to invoke AWS Lambdas through terminating gateways. [GH-12681]
• xds: adding control of the mesh-wide min/max TLS versions and cipher suites from the mesh config entry [GH-12601]

IMPROVEMENTS:

• Refactor ACL denied error code and start improving error details [GH-12308]
• acl: Provide fuller detail in the error messsage when an ACL denies access. [GH-12470]
• agent: Allow client agents to perform keyring operations [GH-12442]
• agent: add additional validation to TLS config [GH-12522]
• agent: add support for specifying TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 and TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 cipher suites [GH-12522]
• agent: bump default min version for connections to TLS 1.2 [GH-12522]
• api: add QueryBackend to QueryMeta so an api user can determine if a query was served using which backend (streaming or blocking query). [GH-12791]
• ci: include 'enhancement' entry type in IMPROVEMENTS section of changelog. [GH-12376]
• ui: Exclude Service Instance Health from Health Check reporting on the Node listing page. The health icons on each individual row now only reflect Node health. [GH-12248]
• ui: Improve usability of Topology warning/information panels [GH-12305]
• ui: Slightly improve usability of main navigation [GH-12334]
• ui: Use @hashicorp/flight icons for all our icons. [GH-12209]
• Removed impediments to using a namespace prefixed IntermediatePKIPath in a CA definition. [GH-12655]
• acl: Improve handling of region-specific endpoints in the AWS IAM auth method. As part of this, the STSRegion field was removed from the auth method config. [GH-12774]
• api: Improve error message if service or health check not found by stating that the entity must be referred to by ID, not name [GH-10894]
• autopilot: Autopilot state is now tracked on Raft followers in addition to the leader. Stale queries may be used to query for the non-leaders state. [GH-12617]
• autopilot: The autopilot.healthy and autopilot.failure_tolerance metrics are now regularly emitted by all servers. [GH-12617]
• ci: Enable security scanning for CRT [GH-11956]
• connect: Add Envoy 1.21.1 to support matrix, remove 1.17.4 [GH-12777]
• connect: Add Envoy 1.22.0 to support matrix, remove 1.18.6 [GH-12805]
• connect: reduce raft apply on CA configuration when no change is performed [GH-12298]
• deps: update to latest go-discover to fix vulnerable transitive jwt-go dependency [GH-12739]
• grpc, xds: improved reliability of grpc and xds servers by adding recovery-middleware to return and log error in case of panic. [GH-10895]
• http: if a GET request has a non-empty body, log a warning that suggests a possible problem (parameters were meant for the query string, but accidentally placed in the body) [GH-11821]
• metrics: The consul.raft.boltdb.writeCapacity metric was added and indicates a theoretical number of writes/second that can be performed to Consul. [GH-12646]
• sdk: Add support for Partition and RetryJoin to the TestServerConfig struct. [GH-12126]
• telemetry: Add new leader label to consul.rpc.server.call and optional target_datacenter, locality, allow_stale, and blocking optional labels. [GH-12727]
• ui: In the datacenter selector order Datacenters by Primary, Local then alpanumerically [GH-12478]
• ui: Include details on ACL policy dispositions required for unauthorized views [GH-12354]
• ui: Move icons away from depending on a CSS preprocessor [GH-12461]
• version: Improved performance of the version.GetHumanVersion function by 50% on memory allocation. [GH-11507]

DEPRECATIONS:

• acl: The consul.acl.ResolveTokenToIdentity metric is no longer reported. The values that were previous reported as part of this metric will now be part of the consul.acl.ResolveToken metric. [GH-12166]
• agent: deprecate older syntax for specifying TLS min version values [GH-12522]
• agent: remove support for specifying insecure TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suites [GH-12522]
• config: setting cert_file, key_file, ca_file, ca_path, tls_min_version, tls_cipher_suites, verify_incoming, verify_incoming_rpc, verify_incoming_https, verify_outgoing and verify_server_hostname at the top-level is now deprecated, use the tls stanza instead [GH-12504]

BUG FIXES:

• acl: Fix parsing of IAM user and role tags in IAM auth method [GH-12797]
• dns: allow max of 63 character DNS labels instead of 64 per RFC 1123 [GH-12535]
• logging: fix a bug with incorrect severity syslog messages (all messages were sent with NOTICE severity). [GH-12079]
• ui: Added Tags tab to gateways(just like exists for non-gateway services) [GH-12400]
• ui: Ensure proxy instance health is taken into account in Service Instance Listings [GH-12279]
• ui: Fixes an issue with the version footer wandering when scrolling [GH-11850]

NOTES:

• Forked net/rpc to add middleware support: https://github.com/hashicorp/consul-net-rpc/ . [GH-12311]
• dependency: Upgrade to use Go 1.18.1 [GH-12808]

FEATURES:

• agent: Added a new config option rpc_client_timeout to tune timeouts for client RPC requests [GH-14965]
• config-entry(ingress-gateway): Added support for max_connections for upstream clusters [GH-14749]

IMPROVEMENTS:

• connect/ca: Log a warning message instead of erroring when attempting to update the intermediate pki mount when using the Vault provider. [GH-15035]
• connect: Added gateway options to Envoy proxy config for enabling tcp keepalives on terminating gateway upstreams and mesh gateways in remote datacenters. [GH-14800]
• connect: Bump Envoy 1.20 to 1.20.7 [GH-14830]

BUG FIXES:

• agent: avoid leaking the alias check runner goroutine when the check is de-registered [GH-14935]
• ca: fix a masked bug in leaf cert generation that would not be notified of root cert rotation after the first one [GH-15005]
• cache: prevent goroutine leak in agent cache [GH-14908]
• snapshot-agent: (Enterprise only) Fix a bug when a session is not found in Consul, which leads the agent to panic.

BUG FIXES:

• kvs: Fixed a bug where query options were not being applied to KVS.Get RPC operations. [GH-13344]

BREAKING CHANGES:

• ca: If using Vault as the service mesh CA provider, the Vault policy used by Consul now requires the update capability on the intermediate PKI's tune mount configuration endpoint, such as /sys/mounts/connect_inter/tune. The breaking nature of this change is resolved in 1.11.11. Refer to upgrade guidance for more information.

SECURITY:

• auto-config: Added input validation for auto-config JWT authorization checks. Prior to this change, it was possible for malicious actors to construct requests which incorrectly pass custom JWT claim validation for the AutoConfig.InitialConfiguration endpoint. Now, only a subset of characters are allowed for the input before evaluating the bexpr. [GH-14577]
• connect: Added URI length checks to ConnectCA CSR requests. Prior to this change, it was possible for a malicious actor to designate multiple SAN URI values in a call to the ConnectCA.Sign endpoint. The endpoint now only allows for exactly one SAN URI to be specified. [[GH-14579](https://github.com/hashicorp/consul/issues/14579)]

IMPROVEMENTS:

• metrics: add labels of segment, partition, network area, network (lan or wan) to serf and memberlist metrics [GH-14161]
• snapshot agent: (Enterprise only) Add support for path-based addressing when using s3 backend.

BUG FIXES:

• ca: Fixed a bug with the Vault CA provider where the intermediate PKI mount and leaf cert role were not being updated when the CA configuration was changed. [GH-14516]
• cli: When launching a sidecar proxy with consul connect envoy or consul connect proxy, the -sidecar-for service ID argument is now treated as case-insensitive. [GH-14034]
• connect: Fixed a bug where old root CAs would be removed from the primary datacenter after switching providers and restarting the cluster. [GH-14598]
• connect: Fixed an issue where intermediate certificates could build up in the root CA because they were never being pruned after expiring. [GH-14429]
• rpc: Adds a deadline to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions. [GH-8504] [GH-11500]
• rpc: Adds max jitter to client deadlines to prevent i/o deadline errors on blocking queries [GH-14233]

BUG FIXES:

• connect: Fixed a goroutine/memory leak that would occur when using the ingress gateway. [GH-13847]
• connect: Ingress gateways with a wildcard service entry should no longer pick up non-connect services as upstreams. connect: Terminating gateways with a wildcard service entry should no longer pick up connect services as upstreams. [GH-13958]

IMPROVEMENTS:

• connect: Update supported Envoy versions to 1.20.4, 1.19.5, 1.18.6, 1.17.4 [GH-13434]

BUG FIXES:

• agent: Fixed a bug in HTTP handlers where URLs were being decoded twice [GH-13265]
• fix a bug that caused an error when creating grpc or http2 ingress gateway listeners with multiple services [GH-13127]
• xds: Fix a bug where terminating gateway upstream clusters weren't configured properly when the service protocol was http2. [GH-13699]

IMPROVEMENTS:

• sentinel: (Enterprise Only) Sentinel now uses SHA256 to generate policy ids

BUG FIXES:

• Fix a bug when configuring an add_headers directive named Host the header is not set for v1/internal/ui/metrics-proxy/ endpoint. [GH-13071]
• areas: (Enterprise Only) Fixes a bug when using Yamux pool ( for servers version 1.7.3 and later), the entire pool was locked while connecting to a remote location, which could potentially take a long time. [GH-1368]
• ca: fix a bug that caused a non blocking leaf cert query after a blocking leaf cert query to block [GH-12820]
• health: ensure /v1/health/service/:service endpoint returns the most recent results when a filter is used with streaming #12640 [GH-12640]
• snapshot-agent: (Enterprise only) Fix a bug where providing the ACL token to the snapshot agent via a CLI or ENV variable without a license configured results in an error during license auto-retrieval.

NOTES:

• ci: change action to pull v1 instead of main [GH-12846]

SECURITY:

• agent: Added a new check field, disable_redirects, that allows for disabling the following of redirects for HTTP checks. The intention is to default this to true in a future release so that redirects must explicitly be enabled. [GH-12685]
• connect: Properly set SNI when configured for services behind a terminating gateway. [GH-12672]

IMPROVEMENTS:

• agent: improve log messages when a service with a critical health check is deregistered due to exceeding the deregister_critical_service_after timeout [GH-12725]
• xds: ensure that all connect timeout configs can apply equally to tproxy direct dial connections [GH-12711]

BUG FIXES:

• acl: (Enterprise Only) fixes a bug preventing ACL policies configured with datacenter restrictions from being created if the cluster had been upgraded to Consul 1.11+ from an earlier version.
• connect/ca: cancel old Vault renewal on CA configuration. Provide a 1 - 6 second backoff on repeated token renewal requests to prevent overwhelming Vault. [GH-12607]
• namespace: (Enterprise Only) Unreserve consul namespace to allow K8s namespace mirroring when deploying in consul K8s namespace .
• raft: upgrade to v1.3.6 which fixes a bug where a read replica node could attempt bootstrapping raft and prevent other nodes from bootstrapping at all [GH-12496]
• replication: Fixed a bug which could prevent ACL replication from continuing successfully after a leader election. [GH-12565]
• server: fix spurious blocking query suppression for discovery chains [GH-12512]
• ui: Fixes a visual bug where our loading icon can look cut off [GH-12479]
• usagemetrics: (Enterprise only) Fix a bug where Consul usage metrics stopped being reported when upgrading servers from 1.10 to 1.11 or later.

FEATURES:

• ca: support using an external root CA with the vault CA provider [GH-11910]

IMPROVEMENTS:

• connect: Update supported Envoy versions to include 1.19.3 and 1.18.6 [GH-12449]
• connect: update Envoy supported version of 1.20 to 1.20.2 [GH-12433]
• connect: update Envoy supported version of 1.20 to 1.20.2 [GH-12443]
• debug: reduce the capture time for trace to only a single interval instead of the full duration to make trace.out easier to open without running into OOM errors. [GH-12359]
• raft: add additional logging of snapshot restore progress [GH-12325]
• rpc: improve blocking queries for items that do not exist, by continuing to block until they exist (or the timeout). [GH-12110]
• sentinel: (Enterprise Only) Sentinel now uses SHA256 to generate policy ids
• server: conditionally avoid writing a config entry to raft if it was already the same [GH-12321]
• server: suppress spurious blocking query returns where multiple config entries are involved [GH-12362]

BUG FIXES:

• agent: Parse datacenter from Create/Delete requests for AuthMethods and BindingRules. [GH-12370]
• areas: (Enterprise Only) Fixes a bug when using Yamux pool ( for servers version 1.7.3 and later), the entire pool was locked while connecting to a remote location, which could potentially take a long time. [GH-1368]
• catalog: compare node names case insensitively in more places [GH-12444]
• checks: populate interval and timeout when registering services [GH-11138]
• local: fixes a data race in anti-entropy sync that could cause the wrong tags to be applied to a service when EnableTagOverride is used [GH-12324]
• raft: fixed a race condition in leadership transfer that could result in reelection of the current leader [GH-12325]
• server: (Enterprise only) Namespace deletion will now attempt to delete as many namespaced config entries as possible instead of halting on the first deletion that failed.
• server: partly fix config entry replication issue that prevents replication in some circumstances [GH-12307]
• state: fix bug blocking snapshot restore when a node check and node differed in casing of the node string [GH-12444]
• ui: Ensure we always display the Policy default preview in the Namespace editing form [GH-12316]
• ui: Fix missing helper javascript error [GH-12358]
• xds: Fixed Envoy http features such as outlier detection and retry policy not working correctly with transparent proxy. [GH-12385]

IMPROVEMENTS:

• connect: update Envoy supported version of 1.20 to 1.20.1 [GH-11895]
• sentinel: (Enterprise Only) Sentinel now uses SHA256 to generate policy ids
• streaming: Improved performance when the server is handling many concurrent subscriptions and has a high number of CPU cores [GH-12080]
• systemd: Support starting/stopping the systemd service for linux packages when the optional EnvironmentFile does not exist. [GH-12176]

BUG FIXES:

• Fix a data race when a service is added while the agent is shutting down.. [GH-12302]
• areas: (Enterprise Only) Fixes a bug when using Yamux pool ( for servers version 1.7.3 and later), the entire pool was locked while connecting to a remote location, which could potentially take a long time. [GH-1368]
• ca: adjust validation of PrivateKeyType/Bits with the Vault provider, to remove the error when the cert is created manually in Vault. [GH-12267]
• config-entry: fix a panic when creating an ingress gateway config-entry and a proxy service instance, where both provided the same upstream and downstream mapping. [GH-12277]
• connect: fixes bug where passthrough addressses for transparent proxies dialed directly weren't being cleaned up. [GH-12223]
• partitions: (Enterprise only) Do not leave a serf partition when the partition is deleted
• serf: update serf v0.9.7, complete the leave process if broadcasting leave timeout. [GH-12057]
• ui: Ensure proxy instance health is taken into account in Service Instance Listings [GH-12279]
• ui: Fix up a problem where occasionally an intention can visually disappear from the listing after saving [GH-12315]
• ui: Fixed a bug with creating multiple nested KVs in one interaction [GH-12081]
• ui: Include partition data when saving an intention from the topology visualization [GH-12317]
• xds: allow only one outstanding delta request at a time [GH-12236]
• xds: fix for delta xDS reconnect bug in LDS/CDS [GH-12174]
• xds: prevents tight loop where the Consul client agent would repeatedly re-send config that Envoy has rejected. [GH-12195]

FEATURES:

• ingress: allow setting TLS min version and cipher suites in ingress gateway config entries [GH-11576]

IMPROVEMENTS:

• api: URL-encode/decode resource names for v1/agent endpoints in API. [GH-11335]
• api: Return 404 when de-registering a non-existent check [GH-11950]
• connect: Add support for connecting to services behind a terminating gateway when using a transparent proxy. [GH-12049]
• http: when a user attempts to access the UI but can't because it's disabled, explain this and how to fix it [GH-11820]
• raft: Consul leaders will attempt to transfer leadership to another server as part of gracefully leaving the cluster. [GH-11376]
• ui: Added a notice for non-primary intention creation [GH-11985]

BUG FIXES:

• Mutate NodeService struct properly to avoid a data race. [GH-11940]
• Upgrade to raft 1.3.3 which fixes a bug where a read replica node can trigger a raft election and become a leader. [GH-11958]
• cli: Display assigned node identities in output of consul acl token list. [GH-11926]
• cli: when creating a private key, save the file with mode 0600 so that only the user has read permission. [GH-11781]
• config: include all config errors in the error message, previously some could be hidden. [GH-11918]
• memberlist: fixes a bug which prevented members from joining a cluster with large amounts of churn [GH-253] [GH-12042]
• snapshot: the snapshot save command now saves the snapshot with read permission for only the current user. [GH-11918]
• ui: Differentiate between Service Meta and Node Meta when choosing search fields in Service Instance listings [GH-11774]
• ui: Ensure a login buttons appear for some error states, plus text amends [GH-11892]
• ui: Ensure partition query parameter is passed through to all OIDC related API requests [GH-11979]
• ui: Fix an issue where attempting to delete a policy from the policy detail page when attached to a token would result in the delete button disappearing and no deletion being attempted [GH-11868]
• ui: Fixes a bug where proxy service health checks would sometimes not appear until refresh [GH-11903]
• ui: Fixes a bug with URL decoding within KV area [GH-11931]
• ui: Fixes a visual issue with some border colors [GH-11959]
• ui: Fixes an issue saving intentions when editing per service intentions [GH-11937]
• ui: Fixes an issue where once a 403 page is displayed in some circumstances its diffcult to click back to where you where before receiving a 403 [GH-11891]
• ui: Prevent disconnection notice appearing with auth change on certain pages [GH-11905]
• ui: Temporarily remove KV pre-flight check for KV list permissions [GH-11968]
• windows: Fixes a bug with empty log files when Consul is run as a Windows Service [GH-11960]
• xds: fix a deadlock when the snapshot channel already have a snapshot to be consumed. [GH-11924]

SECURITY:

• ci: Upgrade golang.org/x/net to address CVE-2021-44716 [GH-11854]

FEATURES:

• Admin Partitions (Consul Enterprise only) This version adds admin partitions, a new entity defining administrative and networking boundaries within a Consul deployment. For more information refer to the Admin Partition documentation. [GH-11855]
• networking: (Enterprise Only) Make segment_limit configurable, cap at 256.

BUG FIXES:

• ui: Fixes an issue with the version footer wandering when scrolling [GH-11850]

BREAKING CHANGES:

• acl: The legacy ACL system that was deprecated in Consul 1.4.0 has been removed. Before upgrading you should verify that nothing is still using the legacy ACL system. See the Migrate Legacy ACL Tokens Learn Guide for more information. [GH-11232]
• cli: consul acl set-agent-token master has been replaced with consul acl set-agent-token recovery [GH-11669]

SECURITY:

• namespaces: (Enterprise only) Creating or editing namespaces that include default ACL policies or ACL roles now requires acl:write permission in the default namespace. This change fixes CVE-2021-41805.
• rpc: authorize raft requests CVE-2021-37219 [GH-10925]

FEATURES:

• Admin Partitions (Consul Enterprise only) This version adds admin partitions, a new entity defining administrative and networking boundaries within a Consul deployment. For more information refer to the Admin Partition documentation.
• ca: Add a configurable TTL for Connect CA root certificates. The configuration is supported by the Vault and Consul providers. [GH-11428]
• ca: Add a configurable TTL to the AWS ACM Private CA provider root certificate. [GH-11449]
• health-checks: add support for h2c in http2 ping health checks [GH-10690]
• ui: Add UI support to use Vault as an external source for a service [GH-10769]
• ui: Adding support of Consul API Gateway as an external source. [GH-11371]
• ui: Adds a copy button to each composite row in tokens list page, if Secret ID returns an actual ID [GH-10735]
• ui: Adds visible Consul version information [GH-11803]
• ui: Topology - New views for scenarios where no dependencies exist or ACLs are disabled [GH-11280]

IMPROVEMENTS:

• acls: Show AuthMethodNamespace when reading/listing ACL tokens. [GH-10598]
• acl: replication routine to report the last error message. [GH-10612]
• agent: add variation of force-leave that exclusively works on the WAN [GH-11722]
• api: Enable setting query options on agent health and maintenance endpoints. [GH-10691]
• api: responses that contain only a partial subset of results, due to filtering by ACL policies, may now include an X-Consul-Results-Filtered-By-ACLs header [GH-11569]
• checks: add failures_before_warning setting for interval checks. [GH-10969]
• ci: Upgrade to use Go 1.17.5 [GH-11799]
• ci: Allow configuring graceful stop in testutil. [GH-10566]
• cli: Add -cas and -modify-index flags to the consul config delete command to support Check-And-Set (CAS) deletion of config entries [GH-11419]
• config: (Enterprise Only) Allow specifying permission mode for audit logs. [GH-10732]
• config: Support Check-And-Set (CAS) deletion of config entries [GH-11419]
• config: add dns_config.recursor_strategy flag to control the order which DNS recursors are queried [GH-10611]
• config: warn the user if client_addr is empty because client services won't be listening [GH-11461]
• connect/ca: cease including the common name field in generated x509 non-CA certificates [GH-10424]
• connect: Add low-level feature to allow an Ingress to retrieve TLS certificates from SDS. [GH-10903]
• connect: Consul will now generate a unique virtual IP for each connect-enabled service (this will also differ across namespace/partition in Enterprise). [GH-11724]
• connect: Support Vault auth methods for the Connect CA Vault provider. Currently, we support any non-deprecated auth methods the latest version of Vault supports (v1.8.5), which include AppRole, AliCloud, AWS, Azure, Cloud Foundry, GitHub, Google Cloud, JWT/OIDC, Kerberos, Kubernetes, LDAP, Oracle Cloud Infrastructure, Okta, Radius, TLS Certificates, and Username & Password. [GH-11573]
• connect: Support manipulating HTTP headers in the mesh. [GH-10613]
• connect: add Namespace configuration setting for Vault CA provider [GH-11477]
• connect: ingress gateways may now enable built-in TLS for a subset of listeners. [GH-11163]
• connect: service-resolver subset filters are validated for valid go-bexpr syntax on write [GH-11293]
• connect: update supported envoy versions to 1.19.1, 1.18.4, 1.17.4, 1.16.5 [GH-11115]
• connect: update supported envoy versions to 1.20.0, 1.19.1, 1.18.4, 1.17.4 [GH-11277]
• debug: Add a new /v1/agent/metrics/stream API endpoint for streaming of metrics [GH-10399]
• debug: rename cluster capture target to members, to be more consistent with the terms used by the API. [GH-10804]
• dns: Added a virtual endpoint for querying the assigned virtual IP for a service. [GH-11725]
• http: when a URL path is not found, include a message with the 404 status code to help the user understand why (e.g., HTTP API endpoint path not prefixed with /v1/) [GH-11818]
• raft: Added a configuration to disable boltdb freelist syncing [GH-11720]
• raft: Emit boltdb related performance metrics [GH-11720]
• raft: Use bbolt instead of the legacy boltdb implementation [GH-11720]
• sdk: Add support for iptable rules that allow DNS lookup redirection to Consul DNS. [GH-11480]
• segments: (Enterprise only) ensure that the serf_lan_allowed_cidrs applies to network segments [GH-11495]
• telemetry: add a new agent.tls.cert.expiry metric for tracking when the Agent TLS certificate expires. [GH-10768]
• telemetry: add a new mesh.active-root-ca.expiry metric for tracking when the root certificate expires. [GH-9924]
• telemetry: added metrics to track certificates expiry. [GH-10504]
• types: add TLSVersion and TLSCipherSuite [GH-11645]
• ui: Change partition URL segment prefix from - to _ [GH-11801]
• ui: Add upstream icons for upstreams and upstream instances [GH-11556]
• ui: Add uri guard to prevent future URL encoding issues [GH-11117]
• ui: Move the majority of our SASS variables to use native CSS custom properties [GH-11200]
• ui: Removed informational panel from the namespace selector menu when editing namespaces [GH-11130]
• ui: Update UI browser support to 'roughly ~2 years back' [GH-11505]
• ui: Update global notification styling [GH-11577]
• ui: added copy to clipboard button in code editor toolbars [GH-11474]

DEPRECATIONS:

• api: /v1/agent/token/agent_master is deprecated and will be removed in a future major release - use /v1/agent/token/agent_recovery instead [GH-11669]
• config: acl.tokens.master has been renamed to acl.tokens.initial_management, and acl.tokens.agent_master has been renamed to acl.tokens.agent_recovery - the old field names are now deprecated and will be removed in a future major release [GH-11665]
• tls: With the upgrade to Go 1.17, the ordering of tls_cipher_suites will no longer be honored, and tls_prefer_server_cipher_suites is now ignored. [GH-11364]

BUG FIXES:

• acl: (Enterprise only) fix namespace and namespace_prefix policy evaluation when both govern an authz request
• api: Fix default values used for optional fields in autopilot configuration update (POST to /v1/operator/autopilot/configuration) [GH-10558] [GH-10559]
• api: ensure new partition fields are omit empty for compatibility with older versions of consul [GH-11585]
• areas: (Enterprise Only) Fixes a bug when using Yamux pool ( for servers version 1.7.3 and later), the entire pool was locked while connecting to a remote location, which could potentially take a long time.
• areas: (Enterprise only) make the gRPC server tracker network area aware [GH-11748]
• ca: fixes a bug that caused non blocking leaf cert queries to return the same cached response regardless of ca rotation or leaf cert expiry [GH-11693]
• ca: fixes a bug that caused the SigningKeyID to be wrong in the primary DC, when the Vault provider is used, after a CA config creates a new root. [GH-11672]
• ca: fixes a bug that caused the intermediate cert used to sign leaf certs to be missing from the /connect/ca/roots API response when the Vault provider was used. [GH-11671]
• check root and intermediate CA expiry before using it to sign a leaf certificate. [GH-10500]
• connect/ca: ensure edits to the key type/bits for the connect builtin CA will regenerate the roots [GH-10330]
• connect/ca: require new vault mount points when updating the key type/bits for the vault connect CA provider [GH-10331]
• connect: fix race causing xDS generation to lock up when discovery chains are tracked for services that are no longer upstreams. [GH-11826]
• dns: Fixed an issue where on DNS requests made with .alt_domain response was returned as .domain [GH-11348]
• dns: return an empty answer when asked for an addr dns with type other then A and AAAA. [GH-10401]
• macos: fixes building with a non-Apple LLVM (such as installed via Homebrew) [GH-11586]
• namespaces: (Enterprise only) ensure the namespace replicator doesn't replicate deleted namespaces
• proxycfg: ensure all of the watches are canceled if they are cancelable [GH-11824]
• snapshot: (Enterprise only) fixed a bug where the snapshot agent would ignore the license_path setting in config files
• ui: Change partitions to expect [] from the listing API [GH-11791]
• ui: Don't offer to save an intention with a source/destination wildcard partition [GH-11804]
• ui: Ensure all types of data get reconciled with the backend data [GH-11237]
• ui: Ensure dc selector correctly shows the currently selected dc [GH-11380]
• ui: Ensure we check intention permissions for specific services when deciding whether to show action buttons for per service intention actions [GH-11409]
• ui: Ensure we filter tokens by policy when showing which tokens use a certain policy whilst editing a policy [GH-11311]
• ui: Ensure we show a readonly designed page for readonly intentions [GH-11767]
• ui: Filter the global intentions list by the currently selected parition rather than a wildcard [GH-11475]
• ui: Fix inline-code brand styling [GH-11578]
• ui: Fix visual issue with slight table header overflow [GH-11670]
• ui: Fixes an issue where under some circumstances after logging we present the data loaded previous to you logging in. [GH-11681]
• ui: Gracefully recover from non-existant DC errors [GH-11077]
• ui: Include Service.Namespace into available variables for dashboard_url_templates [GH-11640]
• ui: Revert to depending on the backend, 'post-user-action', to report permissions errors rather than using UI capabilities 'pre-user-action' [GH-11520]
• ui: Topology - Fix up Default Allow and Permissive Intentions notices [GH-11216]
• ui: code editor styling (layout consistency + wide screen support) [GH-11474]
• use the MaxQueryTime instead of RPCHoldTimeout for blocking RPC queries [GH-8978]. [GH-10299]
• windows: fixes arm and arm64 builds [GH-11586]

NOTES:

• Renamed the agent_master field to agent_recovery in the acl-tokens.json file in which tokens are persisted on-disk (when acl.enable_token_persistence is enabled) [GH-11744]