github.com/grafana/gcx

Breaking changes: Cloud stack creation

• gcx cloud stacks create now requires --org <org-slug>, including
  dry runs. The Cloud API validates access to the selected organisation.
• Successful creation defaults to YAML and returns only name, orgSlug,
  slug, status, and url in every output format. Use
  gcx cloud stacks get <slug> -o json for full details. Dry runs still
  return a request preview.
• Creation trims whitespace around --org and --name; stack slugs retain
  strict lowercase-alphanumeric validation. Blank values are rejected.

Update scripts before upgrading:

# Add the explicit destination organisation to existing create commands.
gcx cloud stacks create --org example-org --name demo --slug demo --region us --dry-run

# After creation, retrieve fields omitted from the concise result.
gcx cloud stacks get demo -o json

Cloud organisation discovery and creation

• Add gcx cloud orgs list to show organisation slugs and membership roles
  for the signed-in user. It uses the selected context's Cloud endpoint,
  including dev and ops, and supports structured output.
• Fresh Cloud OAuth logins request profile. Rerun gcx cloud login if
  your saved login lacks that scope.
• Allow two minutes for stack creation. If it times out, check
  gcx cloud stacks get <slug> with the same config/context before retrying:
  the stack may already exist or still be provisioning.
• Reject blank organisation values in cloud stacks list; that command
  already required --org before this release.

Other improvements

• Preserve Synthetic Monitoring spec.folderUid in check manifests,
  responses, and resource workflows. Omit it to retain the assignment;
  supply an empty string to clear it.
• Explain how --since or --from/--to can speed up trace-ID lookup.
  Use the correct time range; one that excludes the trace returns not found.
• Show an Interactive Learning link after text-mode login when the
  Pathfinder plugin is installed and enabled. JSON/YAML output is unchanged.
• Add a contributor RFC for unified alerting resource workflows; it does
  not introduce a new runtime alerting API.

Metrics and traces

• Add gcx metrics search metric-names/label-names/label-values and
  gcx datasources prometheus search-metric-names/search-label-names/search-label-values,
  backed by the experimental Prometheus/Mimir search API (#1374)
• Add experimental TraceByID options to gcx traces get and
  gcx datasources tempo get: --filter keeps the spans that match a TraceQL
  spanset expression, shaped by --keep-hierarchy, --match-depth and
  --ancestor-depth; --prune (with --prune-group-by, --prune-min-spans,
  --prune-max-parent-depth) collapses repeated sibling spans. Both are off
  unless set, and aim at shrinking large traces before --llm analysis (#1250)

Authentication and configuration

• Add Basic authentication to gcx login (#1389)
• List the available contexts when the requested context is not found (#1210)

Agent Observability

• Add gcx agento11y model-rates to set your own negotiated model prices (#1377)

Synthetic Monitoring and IRM

• Add gcx synthetic-monitoring queries list/get for named queries (#1373)
• Manage IRM OnCall integration alert templates and maintenance mode (#1188)

Application Observability

• Add gcx appo11y operations list/get (#1334)
• Show service metadata, Explore links and Knowledge Graph data for services
  (#1330, #1333)
• Resolve scoped Knowledge Graph services by their exact name (#1384)

Security and debugging

• --insecure-log-http-payload dumps now run at the innermost transport layer
  and show every header that reaches the wire, including the OAuth bearer
  token. They also show the OAuth token refresh exchange, which carries the
  refresh token and the rotated token pair. Treat captured debug logs as
  secrets (#1190)
• Label the dumps http request dump and http response dump, include
  Content-Length and Accept-Encoding in request dumps, report why a dump
  failed, and document that dumps log at Debug level (-vvv) (#1190)

Docs and internals

• Add guides for managing resources and diagnosing missing telemetry
  (#1413, #1368)
• Clarify the MkDocs setup and fix broken links (#1378, #1417)
• Move Knowledge Graph transport into a shared internal client (#1331, #1332)

Synthetic Monitoring

• Add ad hoc check execution with result polling (#1311)
• Add named backend datasource queries (#1307)
• Add optional script decoding for browser and scripted checks (#1309)

Datasources and queries

• Add StarTree Pinot datasource queries (#1255)
• Add Frontend Observability session retrieval (#1257)
• Preserve datasource plugin types in list and resource responses (#1356)

Dashboards, Assistant, and Agent Observability

• Add a render timeout option to dashboard snapshots (#1355)
• Add optional artifact downloads to experiment pulls (#1321)
• Read legacy, AI SDK, and shared Assistant transcripts (#1369)

Application Observability

• Require App Observability activation for service commands (#1329)

SLO, Fleet, and skills

• Move SLO definitions and reports to declarative adapters (#1342)
• Use gcx resources push and pull for new SLO workflows. Generic push
  matches natural keys. Deprecated SLO wrappers match UUIDs only (#1342)
• Show Fleet collector health attributes and timestamps (#1352)
• Reconcile installed skills with a release lifecycle catalog (#1339)

Security and telemetry

• Stop OAuth before network use when the credential store is not writable (#1313)
• Add the human or agent execution mode to the user agent (#1336)

Other

• Migrate Knowledge Graph, Alerting, and Agent Observability tables to shared
  codecs (#1345, #1346, #1347)
• Update Go and Kubernetes dependencies (#915, #798)
• Improve release, review, and ownership workflows (#1341, #1306, #1350,
  #1317, #1314)

Breaking changes

• Credential storage now fails closed when the OS keychain is unavailable, instead of writing to the config file in plaintext with a warning. You will now see a Keychain unavailable error in this scenario. A locked, as opposed to absent, keychain already failed closed and is unchanged. Set GCX_KEYCHAIN=off to opt in to a plaintext write. (#1271)
• Fleet Management permissions have changed: A grafana.com Cloud Access Policy token is no longer needed for interacting with gcx fleet. gcx will need write permissions when using the OAuth flow for all gcx fleet commands. Users will need grafana-collector-app:read for named routes, and wildcard routes need the Admin role or grafana-collector-app:admin, including some read-only commands such as tenant get-limits. (#1261)
• gcx synthetic-monitoring probes deploy emits different manifests, and the probe token is now provided through a Kubernetes Secret, defined by the SM_AGENT_API_TOKEN environment variable. To use this, regenerate the manifests and update anything that read the token out of the Deployment arguments before applying. Using --token still works but it is deprecated and hidden from help (#1262)
• gcx synthetic-monitoring probes reset-token now returns the new token value instead of just confirming the reset. The value is in gcx.synth.probe_token_reset. Anything parsing the previous result shape needs updating (#1262)
• gcx irm oncall schedules list-final-shifts takes a --timezone flag and sends the timezone of the schedule, rather than the host. Without the --timezone flag it now uses the schedule's timezone, falling back to UTC with a warning if the schedule does not have a timezone set. (#1187)

Datasources and queries

• Add BigQuery, MSSQL, and OpenSearch datasource support (#1106, #1059, #1252)
• Add gcx profiles series for series discovery (#1086)
• Add -o dot call-graph output to gcx profiles query (#1122)
• Add the experimental gcx traces baseline command (#1168)
• Require an Explore link on every query command (#1134)

Alerting and IRM

• Add gcx alert state-history and notification history (#1296, #1258)
• Add gcx irm incidents update, and honour incident severity (#1191)
• Let a discovery command list its own catalog (#1193)
• alert-groups list --limit 0 now returns every alert group (#1213)

Configuration and credentials

• Add the trusted credentials.keychain policy (#1266)
• Add GCX_KEYCHAIN to disable the OS credential store (#1242)
• Make configuration load and write parameters explicit (#1272)

Agent Observability and Instrumentation

• Add gcx agento11y experiments pull for experiment bundles (#1267)
• Add gcx agento11y rules list-scores for online scores (#1226)
• Add --fix-plan to gcx instrumentation check (#1115)

Telemetry

• Break down api command usage by route, method, and type (#1094)
• Report HTTP status, Kubernetes reason, and authentication method (#1118)
• Report a canceled invocation with the outcome canceled (#1109)

Fixes

• create -f and update -f accept the K8s envelope that gcx itself prints, instead of discarding every field of the file (#1187)
• --jq now works together with -o agents (#1287)
• Normalize unstructured jq numbers (#1235)
• Faro sends extraLogLabels under the label field (#1238)
• Browser checks keep their notification channels (#1299)

Other

• Render provider tables with one generic codec (#1274, #1285, #1286, #1289)
• Add declarative provider registration machinery (#931)
• Set the manager identity from the gcx version (#1166)
• Enforce the command naming conventions in CI (#1243)
• Upgrade the Bubble Tea stack to v2 (#1253)
• Refine the Grafana investigation skill (#1279, #1292)
• Define how to mark an experimental command (#1220)
• Update CONTRIBUTING, and add skill references (#1304, #1280)

Breaking changes

• A locked OS keychain is now its own fatal failure class (Keychain locked). gcx stops credential-consuming commands instead of falling back to a plaintext write, and the error explains how to unlock the keychain for the session (#1142)
• Remove the broken gcx assistant dashboard subcommand (#1230)

New features

• Add the Database Observability provider: gcx dbo11y instances list and gcx dbo11y instances get <name> (#1199)

Fixes

• Remove the duplicated --agent-id default from assistant help text (#1232)

Other

• Add keychain setup and troubleshooting reference docs (#1142)
• Teach the add-datasource skill the shared raw-SQL contract (#1227)
• Bump create-github-app-token to 0.3.1 and tune the CI review process (#1236, #1217)

Datasources

• Add support for the MySQL and Postgres datasources (#970, #969)
• Add support for the Elasticsearch datasource (#971)
• Add support for Azure Monitor and Google Cloud Monitoring (#965, #974)
• Restructure the generic query router to keep it simple (#1141)
• Add the experimental gcx traces diff command (#1165)

Providers

• Add notification commands to the knowledge graph provider (#1077)
• Add gcx irm incidents get-pir for post-incident reviews (#1163)
• Report otlpIngestEndpointURL from frontend applications (#1223)

Login and configuration

• Add --oauth-manual for hosts without a reachable callback (#1136)
• Let environment credentials override a keychain failure (#1066)
• Scope gcx config check to one explicit context (#1155)

Fixes

• Include otlp_url and otlp_username in the cluster configuration (#1198)
• Correct the order of dashboards in the output (#1201)

Performance

• Build the discovery registry one time (#925)
• Limit the parallel fan-out in resource commands (#925)

Other

• Detect the pi coding agent harness for agent mode (#1208)
• Report the batch resource volume as size buckets (#1108)
• Add a Grafana version support policy to the documentation (#1197)
• Regenerate the command line interface reference (#1224)

Breaking changes

• gcx agento11y collections add-conversations and gcx agento11y collections remove-conversation emit type: gcx.agento11y.collection_membership instead of gcx.aio11y.collection_membership. schema_version stays 1. No fields changed. Scripts that read type must accept the new value.

New features

• Add gcx profiles data-range for Pyroscope ingestion health.
• Add the same data-range command under gcx datasources pyroscope.
• Add gcx instrumentation explain and list-explanations for the finding explanations that ship with otel-checker (now v0.3.1).
• Add an EXPLAIN_ID column to gcx instrumentation check table output, and an explain_id field to its JSON output. Pass the value to gcx instrumentation explain.
• Add gcx assistant investigations list-evidence.
• Show the chat ID next to the investigation ID in investigations get.
• Keep full v2 investigation summaries and totals in list output.
• Filter IRM alert groups by escalation chain and by date window.
• Add agent support to gcx metrics list-names.
• Add agent support to metrics label matching.

Improvements

• Check contexts concurrently in gcx config check.
• Record target_kind in usage telemetry events.

Fixes

• Make gcx dev serve exit on Ctrl-C.
• Set AllowsEdits so gcx-managed dashboards stay editable in the UI.
• Keep configType in Fleet so OTel pipelines round-trip.
• Support UTF-8 label names in profiles.
• Match experiment report types and trial counts to the API.
• Send guard redaction config as redact with {id, regex}.
• Drop rule_id from the rules update PATCH body.
• Handle an unavailable keychain safely during verification.

Internal

• Rename the aio11y Go package to agento11y.
• Add the integrate-with-gcx and review-pr contributor skills.
• Add a command naming conventions guide.
• Restructure the Grafana Cloud documentation.
• Pin GCX_AGENT_MODE=false for the test tasks.
• Add data sources code owners for internal/datasources.

Features

• alert: add ruler subtree for datasource-managed rule writes
• kg: add entities correlate to resolve entities from alert labels
• kg: add entity quality reports (kg quality), folded into kg diagnose

Fixes

• alert: make GMA alert rules modifiable via the resources tier
• cloud: drop status/createdAt/updatedAt from stack regions output
• aio11y: point plugin id at renamed grafana-agento11y-app

Docs

• GA: remove public preview warnings, add GA notes across docs and README
• skills: harden agento11y-instrument and agento11y-prod-setup guidance
• stop prescribing Editor/Admin for service-account tokens; installation fixes

Breaking changes

• Naming convergence: verb-first subcommand renames across all providers
  (e.g. versions→list-versions, create→upsert, summary→stats)
• Config split into separate stacks, cloud entries, and contexts (auto-migrates)
• Agent output contract: one JSON document per finite command
• Profiles: profile-types renamed to list-profile-types at both mounts
• Alert: dropped create/update/apply aliases from templates upsert

Features

• Telemetry: first-run notice and default-on anonymous usage stats
• Output: list truncation contract with honest caps and agent-legible metadata
• aio11y: experiments v2 shapes and commands, plus conversation annotations
• kg: --dry-run for model-rules and prom-rules upsert
• Profiles: time-range flags for labels and list-profile-types

Fixes

• Cloud: stop misreporting invalid stack slugs as "slug already taken"
• appo11y: honor --config across the direct command tree
• aio11y: honor --config in direct agento11y CRUD commands
• irm: decode expanded webhook integration filters

Docs

• Added anonymous usage statistics page; fixed usage-stats command examples

• Register the Assistant as a first-class provider, moving mcp-servers into the resources pipeline
• Breaking: rename the aio11y command tree and skills to agento11y
• Breaking: rename the agento11y-eval-starter skill to agento11y-test-starter
• Add agento11y-instrument skill (setup → instrument → verify loop)
• Add agento11y-prod-setup skill for production evals and guards
• Bundle the agento11y-test-starter starter skill
• Breaking: remove the explore-datasources skill
• Retune skill descriptions and fix API drift in aio11y, SLO, and Synthetics guidance
• Export anonymous usage events as flat JSON over HTTP when telemetry is enabled
• Add profiles span and trace selectors; include trace IDs in exemplars
• Fix profiles --top totals to exclude the pre-window boundary point
• Add KG prom-rules schema command and --dry-run to suppressions create
• Route KG write API through the Asserts plugin proxy
• Fix KG entity/relationship deletes to use the collection-path API
• Support Dashboard V2 in the resource linter
• Descend into single-key list envelopes for --json discovery and selection
• Include an output-shape hint in --jq runtime errors
• Request full cloud scopes in the login cloud followup flow
• Derive SDK imports for generated dev import code from actual usage
• Disclose per-product Grafana Cloud costs in docs and help text
• Clarify gcx works with OSS and Enterprise, not just Cloud
• Mark the traces --llm flag experimental and document its Accept header

• Add full CRUD lifecycle for datasources (create, update, delete, health, schemas)
• Add --name filter to datasources list
• Parallelize per-plugin-group datasource list fetch for speed
• Add CRUD verbs and discovery for IRM OnCall resources
• Fix IRM build break from a semantic merge conflict
• Add OAuth login flow for gcx cloud login (grafana.com), marked experimental
• Expand OAuth scopes; drop k6 from write scopes; fix login help text/doc links
• Add anonymous usage stats core library (telemetry, not yet wired in)
• Add fail-safe client-side dry-run for APIs that ignore dryRun
• Add metrics cardinality analysis commands
• Add filter, group-by, and label discovery to appo11y services commands
• Add assistant mcp-servers command group
• Add knowledge graph entity & relationship write commands (gated, experimental)
• Add x-client-id header to Synthetic Monitoring requests
• Fix logs: distinguish structured metadata from indexed stream labels
• Skip /bootdata namespace validation when stack-id is set (faster config load)
• Dedupe HTTP client construction/error parsing across providers
• Dedupe Prometheus/Loki/CloudWatch onto shared query transport
• Dedupe aio11y marshal/status-check/decode pattern
• Dedupe datasources config-loading boilerplate
• Add drift check ensuring bundled skills match real gcx commands
• Switch lint to Go module mode, dropping the vendor/ requirement
• Update Go dependencies
• Fix docs links in README, CLI help text, and OAuth login docs
• Use main branch for deploy previews; update SM CODEOWNERS

• Query: fall back to /api/ds/query on any non-200 from the k8s query API
• Pyroscope: default the metrics query step from the datasource config
• Knowledge Graph: apply the selected time range to entity scopes lookup
• Agent mode: mark cloud-only commands via new agent.availability annotation
• Windows: open OAuth browser via explorer.exe so the login URL survives
• CI: auto-publish the Homebrew formula and drop dead docs jobs

• Add Athena datasource provider (query, list catalogs/databases/tables, describe-table)
• Introduce shared SQL query package, refactoring ClickHouse onto it
• Synthetic Monitoring now syncs via a Grafana datasource proxy client
• Set X-Grafana-Caller-Id header on all datasource queries
• Refine output formatting (format and jq handling)
• Remove obsolete publish-technical-documentation workflows

• Breaking: --log-http-payload has been renamed to --insecure-log-http-payload. The old flag name now exits with an error naming the replacement. Payload-dump behavior is unchanged — credentials, cookies, OAuth refresh tokens, and request bodies still appear in the dump when the flag is set. A startup warning is now printed to stderr when the flag is active.
• Breaking: gcx dev serve --address now defaults to 127.0.0.1 (loopback) instead of 0.0.0.0. Users who need LAN or remote access should pass --address 0.0.0.0 explicitly. The WebSocket livereload endpoint now rejects cross-origin requests from hosts other than localhost, 127.0.0.1, [::1], or the configured listen address.
• Soft-breaking: Custom Rego rules loaded via gcx dev lint --rules can no longer call http.send, any net.* builtin, or opa.runtime. No flag is provided to re-enable them. Bundled rules are unaffected.
• Added --jq for inline JSON transformation and improved agent-mode hints.
• Added OAuth login support for agent and non-interactive workflows.
• Improved config performance with lazy keychain checks and StackID caching.
• Moved stack management under the gcx cloud stacks command group.
• Added App O11y service list, get, map, and operation breakdown commands.
• Added Instrumentation Hub gcx instrumentation check diagnostics.
• Expanded Assistant investigations with Lodestone v2 capability detection.
• Expanded Knowledge Graph diagnose, meta metrics, and model-rules support.
• Improved IRM incident filtering, paging, and OnCall escalation output.
• Added metrics instant-query timestamps and richer Tempo tag-value output.
• Hardened local debug surfaces and JSON error output for agent workflows.

• Added gcx assistant conversation list and gcx assistant conversation get to inspect Assistant conversations and transcripts.
• Added support for continuing accessible non-CLI Assistant conversations by context ID, with a warning when the source is not CLI.
• Added automatic migration of token-shaped config secrets into the OS keychain, with plaintext fallback when the keychain is unavailable.
• Breaking: renamed gcx kg rules to gcx kg prom-rules to align with model-rules and relabel-rules.
• Added read-only gcx kg relabel-rules get for prologue, epilogue, and generated rule groups.
• Fixed kg diagnose verdicts so failing checks no longer report a healthy service, and reclassify scoped no-data checks as label-scope warnings when unscoped data exists.
• Shared Grafana datasource query transport and dataframe wire types across ClickHouse, InfluxDB, and Infinity clients.
• Hardened Claude GitHub Actions secret handling by scoping Vault outputs and restricting @claude triggers to trusted repository associations.
• Improved agent guidance for dashboard workflows and Tempo trace analysis, including recommending gcx traces get --llm -o json for full-trace analysis.
• Removed several small direct dependencies in favor of internal implementations.