github.com/neilotoole/sq

Fixed

• [#866], [#868]: Ingesting document sources (CSV, JSON, Excel, etc.) is now much faster, especially on Windows and other slow filesystems.
• [#865]: Fixed shell-completion regressions.
• [#863]: FORCE_COLOR handling now follows the force-color.org conventions.
• [#919]: sq inspect on an Oracle source now handles materialized views correctly.
• Various other bug fixes: [#859], [#902], [#911], [#915], [#916], [#918], [#920], [#923], [#926].

Added

• 🐥 #444: New driver for rqlite, the lightweight distributed database built on SQLite.
• #441: Revamped secrets handling. Source credentials no longer need to live as plaintext in sq.yml: source locations accept ${scheme:path} placeholders that are resolved at connect time, and secrets are consistently redacted in output unless you opt in to disclosure.
  • #714: Supported placeholder resolvers:
    • OS keychain: ${keyring:3d28xd3jcr}
    • Environment variables: ${env:DB_PASSWORD}
    • File contents: ${file:/run/secrets/db_pw}
    • 1Password: ${op://Private/sakila/dsn}
  • #717, #729: New global flags:
    • --reveal: show secret values in output; supersedes the now-deprecated --no-redact.
    • --expand: print resolved placeholder values instead of the verbatim ${scheme:path} text.
  • #716: sq config export dumps the active config to YAML, primarily for backups. With --expand, placeholders are resolved in-line, making the export a self-contained (but plaintext) snapshot.
• #660: sq inspect gained svg-erd and png-erd output formats that render the schema ERD directly to .svg or .png image files.
  • Uses an embedded Graphviz engine (via goccy/go-graphviz): the generated images are not the prettiest, but it's a start.
• #846: New format.decimal option and --format.decimal flag (string | number) controlling whether decimal values render as quoted strings or bare numbers in JSON and YAML output.

Changed

• #851: In colored output, structural punctuation is now rendered muted rather than bold, so values stand out.
• #846: YAML now renders decimal values as quoted strings by default, matching JSON. Set format.decimal=number to restore bare numbers.
• ☢️ #594: avg() now returns a consistent float on every SQL driver. Breaking for Postgres and MySQL, which previously returned a lossless decimal.
• ☢️ #839, #853: sum() over an integer or decimal column now returns a consistent decimal on every SQL driver, rendered in JSON as a quoted string (e.g. "20100").
• #610: The DuckDB driver now opens sources read-only for non-writing commands, and sq sql gained a --readonly (alias --ro) flag.
• ☢️ #728, #782: The redact config option is renamed to secrets.reveal with inverted polarity; existing configs are migrated automatically (after a backup).
• #692: sq inspect -f mermaid-erd now syntax-colors its erDiagram source when writing to a terminal.
• #758: For SQLite (and rqlite) sources, sq tbl copy now copies the source table's indexes and triggers to the destination.

Fixed

• #851: Two fixes to yaml output: a numeric value from an untyped column is now colorized, and a BLOB value is now encoded as base64.
• #844: On Oracle, a query whose result is a computed NUMBER with a fractional value no longer fails with a scan error; such numbers are now typed as decimal.
• #594: On SQL Server, avg() over an integer column no longer performs integer division and truncates the result.
• #741, #743: sq add shell completion now supports ClickHouse and Oracle.
• #699: sq inspect on SQLite sources no longer executes pragmas that write to the database or scan the whole file, which could make each inspect painfully slow.
• #720: Fixed SQLite driver path handling for source locations with connection params (e.g. sqlite3:///path/to/db?mode=ro).
• #750, #752, #757, #759: A batch of fixes to the CREATE TABLE DDL rewriting that backs table copy and column-kind alteration on the sqlite3 driver.
• #783: sq db dump cluster no longer writes the database password to the sq log file.
• #783: sq cache clear @src now clears every cache dir belonging to the source, including stale dirs left over from a changed location, schema, or ingest options.
• #783: sq add and sq mv no longer permit a source handle nested below an existing source's handle.
• #821: On the SQLite and rqlite drivers, renaming a table, adding a column, or truncating it no longer fails when a table or column name contains a double quote.
• #834: On the ClickHouse driver, a result-column name containing a dot is no longer truncated to the segment after the last dot.

Full Changelog: v0.53.0...v0.54.0

Added

• 🐥 [#444]: New driver for rqlite, the lightweight distributed database built on SQLite.
• [#441]: Revamped secrets handling. Source credentials no longer need to live as plaintext in sq.yml: source locations accept ${scheme:path} placeholders that are resolved at connect time, and secrets are consistently redacted in output unless you opt in to disclosure.
  • [#714]: Supported placeholder resolvers:
    • OS keychain: ${keyring:3d28xd3jcr}
    • Environment variables: ${env:DB_PASSWORD}
    • File contents: ${file:/run/secrets/db_pw}
    • 1Password: ${op://Private/sakila/dsn}
  • [#717], [#729]: New global flags:
    • --reveal: show secret values in output; supersedes the now-deprecated --no-redact.
    • --expand: print resolved placeholder values instead of the verbatim ${scheme:path} text.
  • [#716]: sq config export dumps the active config to YAML, primarily for backups. With --expand, placeholders are resolved in-line, making the export a self-contained (but plaintext) snapshot.
• [#660]: sq inspect gained svg-erd and png-erd output formats that render the schema ERD directly to .svg or .png image files.
  • Uses an embedded Graphviz engine (via goccy/go-graphviz): the generated images are not the prettiest, but it's a start.
• [#846]: New format.decimal option and --format.decimal flag (string | number) controlling whether decimal values render as quoted strings or bare numbers in JSON and YAML output.

Changed

• [#900]: HTTPS requests (used when downloading remote sources) now require a minimum of TLS 1.2, matching the Go standard library default. TLS 1.0 and 1.1, deprecated by RFC 8996, are no longer accepted.
• [#851]: In colored output, structural punctuation is now rendered muted rather than bold, so values stand out.
• [#846]: YAML now renders decimal values as quoted strings by default, matching JSON. Set format.decimal=number to restore bare numbers.
• ☢️ [#594]: avg() now returns a consistent float on every SQL driver. Breaking for Postgres and MySQL, which previously returned a lossless decimal.
• ☢️ [#839], [#853]: sum() over an integer or decimal column now returns a consistent decimal on every SQL driver, rendered in JSON as a quoted string (e.g. "20100").
• [#610]: The DuckDB driver now opens sources read-only for non-writing commands, and sq sql gained a --readonly (alias --ro) flag.
• ☢️ [#728], [#782]: The redact config option is renamed to secrets.reveal with inverted polarity; existing configs are migrated automatically (after a backup).
• [#692]: sq inspect -f mermaid-erd now syntax-colors its erDiagram source when writing to a terminal.
• [#758]: For SQLite (and rqlite) sources, sq tbl copy now copies the source table's indexes and triggers to the destination.

Fixed

• [#851]: Two fixes to yaml output: a numeric value from an untyped column is now colorized, and a BLOB value is now encoded as base64.
• [#844]: On Oracle, a query whose result is a computed NUMBER with a fractional value no longer fails with a scan error; such numbers are now typed as decimal.
• [#594]: On SQL Server, avg() over an integer column no longer performs integer division and truncates the result.
• [#741], [#743]: sq add shell completion now supports ClickHouse and Oracle.
• [#699]: sq inspect on SQLite sources no longer executes pragmas that write to the database or scan the whole file, which could make each inspect painfully slow.
• [#720]: Fixed SQLite driver path handling for source locations with connection params (e.g. sqlite3:///path/to/db?mode=ro).
• [#750], [#752], [#757], [#759]: A batch of fixes to the CREATE TABLE DDL rewriting that backs table copy and column-kind alteration on the sqlite3 driver.
• [#783]: sq db dump cluster no longer writes the database password to the sq log file.
• [#783]: sq cache clear @src now clears every cache dir belonging to the source, including stale dirs left over from a changed location, schema, or ingest options.
• [#783]: sq add and sq mv no longer permit a source handle nested below an existing source's handle.
• [#821]: On the SQLite and rqlite drivers, renaming a table, adding a column, or truncating it no longer fails when a table or column name contains a double quote.
• [#834]: On the ClickHouse driver, a result-column name containing a dot is no longer truncated to the segment after the last dot.

Added

• #601, #615, #628, #629, #640: New SLQ string-matching functions. See Query Guide for per-driver behavior and SQLite ASCII-CI quirks.
  • Case-sensitive literal substring matchers contains, startswith, and endswith.
  • Case-insensitive variants icontains, istartswith, and iendswith.
  • User-controlled wildcard matchers like and ilike: % and _ are wildcards.
• #648: sq inspect now has --markdown and --html output formats that generate schema documents with embedded entity relationship diagrams.
  • #658: sq inspect also gained a mermaid-erd output format (--format=mermaid-erd) that emits the bare Mermaid entity-relationship diagram source for source and table schema inspection, with no surrounding document.

Changed

• #637: Richer SLQ syntax-error reporting in both text and JSON error formats. You can set the new config option error.format.text.verbose to false if you prefer the previous (less-verbose) text error format.
  
• #617, #618: sq inspect now omits an index whose key positions are all expressions (previously MySQL and SQLite reported such an index with an empty columns list).

Fixed

• #617, #618: sq inspect now preserves the true key arity and position of indexes that contain functional/expression keys. A key like lower(b) in CREATE INDEX ix ON t (a, lower(b), c) is reported as an empty-string entry in the index's columns, so it is no longer indistinguishable from a real two-column (a, c). Applies to SQLite, DuckDB, MySQL, and Postgres.
• #612: DuckDB INTERVAL values now render in DuckDB's native, round-trippable text form (e.g. 1 year 2 months 3 days 04:05:06.789) instead of the previous ad-hoc Go-style μs string.
• #613: sq inspect of a SQL Server source no longer returns tables from schemas other than the source's current schema.
• #471: The SQLite driver no longer fails with unsupported Scan, storing driver.Value type string into type *time.Time when a DATETIME/DATE column stores text that mattn/go-sqlite3 doesn't natively convert — most commonly Rails' microsecond-precision datetime(6) columns. sq now re-runs the same timestamp parse and falls back to the raw text only when the value is genuinely unparseable.
• #658: sq <command> --format=FORMAT (e.g. sq inspect --format=xlsx) no longer fails with a spurious "diff does not support output format" error. The format check for sq diff was being applied to every command's --format flag; it's now scoped to sq diff itself.
• #652: The ClickHouse driver now creates a copied table in its target schema (ClickHouse database) when the copy specifies one, rather than always in the connection's current database.
• #484: --insert into a MySQL or Postgres table no longer fails when a same-named table exists in another schema.
• #633: A query using the single-segment @handle.table:alias form on the left of a pipeline (e.g. @sakila.actor:a | .a.first_name) no longer silently collapses to SELECT * FROM <table>.
• #646: A query whose alias is a reserved word (e.g. .actor | .first_name:count, @sakila.actor:count, or join(.film_actor:count, ...)) now applies that alias instead of silently dropping it. An alias that is an argument reference (e.g. :$x) is now rejected with a clear error rather than silently discarded.
• #445: Cross-source join no longer fails when the participating sources contain tables with the same name (e.g. @src1.actor | join(@src2.actor, .actor_id)).

Added

• [#601], [#615], [#628], [#629], [#640]: New SLQ string-matching functions. See Query Guide for per-driver behavior and SQLite ASCII-CI quirks.
  • Case-sensitive literal substring matchers contains, startswith, and endswith.
  • Case-insensitive variants icontains, istartswith, and iendswith.
  • User-controlled wildcard matchers like and ilike: % and _ are wildcards.
• [#648]: sq inspect now has --markdown and --html output formats that generate schema documents with embedded entity relationship diagrams.
  • [#658]: sq inspect also gained a mermaid-erd output format (--format=mermaid-erd) that emits the bare Mermaid entity-relationship diagram source for source and table schema inspection, with no surrounding document.

Changed

• [#637]: Richer SLQ syntax-error reporting in both text and JSON error formats. You can set the new config option error.format.text.verbose to false if you prefer the previous (less-verbose) text error format.
• [#617], [#618]: sq inspect now omits an index whose key positions are all expressions (previously MySQL and SQLite reported such an index with an empty columns list).

Fixed

• [#617], [#618]: sq inspect now preserves the true key arity and position of indexes that contain functional/expression keys. A key like lower(b) in CREATE INDEX ix ON t (a, lower(b), c) is reported as an empty-string entry in the index's columns, so it is no longer indistinguishable from a real two-column (a, c). Applies to SQLite, DuckDB, MySQL, and Postgres.
• [#612]: DuckDB INTERVAL values now render in DuckDB's native, round-trippable text form (e.g. 1 year 2 months 3 days 04:05:06.789) instead of the previous ad-hoc Go-style μs string.
• [#613]: sq inspect of a SQL Server source no longer returns tables from schemas other than the source's current schema.
• [#471]: The SQLite driver no longer fails with unsupported Scan, storing driver.Value type string into type *time.Time when a DATETIME/DATE column stores text that mattn/go-sqlite3 doesn't natively convert — most commonly Rails' microsecond-precision datetime(6) columns. sq now re-runs the same timestamp parse and falls back to the raw text only when the value is genuinely unparseable.
• [#658]: sq <command> --format=FORMAT (e.g. sq inspect --format=xlsx) no longer fails with a spurious "diff does not support output format" error. The format check for sq diff was being applied to every command's --format flag; it's now scoped to sq diff itself.
• [#652]: The ClickHouse driver now creates a copied table in its target schema (ClickHouse database) when the copy specifies one, rather than always in the connection's current database.
• [#484]: --insert into a MySQL or Postgres table no longer fails when a same-named table exists in another schema.
• [#633]: A query using the single-segment @handle.table:alias form on the left of a pipeline (e.g. @sakila.actor:a | .a.first_name) no longer silently collapses to SELECT * FROM <table>.
• [#646]: A query whose alias is a reserved word (e.g. .actor | .first_name:count, @sakila.actor:count, or join(.film_actor:count, ...)) now applies that alias instead of silently dropping it. An alias that is an argument reference (e.g. :$x) is now rejected with a clear error rather than silently discarded.
• [#445]: Cross-source join no longer fails when the participating sources contain tables with the same name (e.g. @src1.actor | join(@src2.actor, .actor_id)).

Added

• 🐥🦆 #437: Beta DuckDB support. The duck
  driver supports most of the DuckDB
  type system; and includes a standard set of in-tree extensions (json,
  parquet, icu, fts, httpfs, excel, inet, autocomplete,
  tpch, tpcds). See the
  driver docs.
• #498: sq inspect now reports
  foreign keys (fk.outgoing / fk.incoming per table), unique
  constraints, and indexes for every SQL driver (ClickHouse skipped — no
  foreign-key concept). Composite and cross-schema references are
  supported; see the driver-specific caveats in the
  inspect docs.
• #602: sq now features a
  --render-sql flag, which
  prints the SQL (derived from SLQ input) that would be executed against
  the target database, instead of running it. Honors --format with:
  • text or raw: the rendered SQL is printed.
  • json or yaml: a structured payload is printed containing the
    original SLQ, the rendered SQL, any
    --arg, the
    dialect and information about the sources that the query touches.

    # $ sq --render-sql --yaml '@sakila/pg.actor | join(@sakila/my.film_actor, .actor_id) | .first_name, .last_name, .film_id | .[0:5]'
    slq: "@sakila/pg.actor | join(@sakila/my.film_actor, .actor_id) | .first_name, .last_name, .film_id | .[0:5]"
    sql: SELECT "first_name", "last_name", "film_id" FROM "actor" INNER JOIN "film_actor" ON "actor"."actor_id" = "film_actor"."actor_id" LIMIT 5 OFFSET 0
    dialect: sqlite3
    sources:
      target: "@join_xukcx3ye"
      inputs:
      - "@sakila/pg"
      - "@sakila/my"

Changed

• ☢️ #499: libsq.SLQ2SQL now returns a structured
  *libsq.RenderResult type instead of string. Any libsq library
  consumers must update call sites; the sq CLI is unaffected.

Added

• 🐥🦆 [#437]: Beta DuckDB support. The duck driver supports most of the DuckDB type system; and includes a standard set of in-tree extensions (json, parquet, icu, fts, httpfs, excel, inet, autocomplete, tpch, tpcds). See the driver docs.
• [#498]: sq inspect now reports foreign keys (fk.outgoing / fk.incoming per table), unique constraints, and indexes for every SQL driver (ClickHouse skipped — no foreign-key concept). Composite and cross-schema references are supported; see the driver-specific caveats in the inspect docs.
• [#602]: sq now features a --render-sql flag, which prints the SQL (derived from SLQ input) that would be executed against the target database, instead of running it. Honors --format with:

  • text or raw: the rendered SQL is printed.

  • json or yaml: a structured payload is printed containing the original SLQ, the rendered SQL, any --arg, the dialect and information about the sources that the query touches.

    # $ sq --render-sql --yaml '@sakila/pg.actor | join(@sakila/my.film_actor, .actor_id) | .first_name, .last_name, .film_id | .[0:5]'
    slq: "@sakila/pg.actor | join(@sakila/my.film_actor, .actor_id) | .first_name, .last_name, .film_id | .[0:5]"
    sql: SELECT "first_name", "last_name", "film_id" FROM "actor" INNER JOIN "film_actor" ON "actor"."actor_id" = "film_actor"."actor_id" LIMIT 5 OFFSET 0
    dialect: sqlite3
    sources:
      target: "@join_xukcx3ye"
      inputs:
        - "@sakila/pg"
        - "@sakila/my"
    

Changed

• ☢️ [#499]: libsq.SLQ2SQL now returns a structured *libsq.RenderResult type instead of string. Any libsq library consumers must update call sites; the sq CLI is unaffected.

Added

• 🐥 #501: Beta Oracle support, built on the pure-Go go-ora driver. No Instant Client required.
  Kudos to @drluckyspin for indulging enterprise Larry habits.

Added

• 🐥 [#501]: Beta Oracle support, built on the pure-Go go-ora driver. No Instant Client required. Kudos to @drluckyspin for indulging enterprise Larry habits.

Added

• #552: Agent-focused skill content under skills/sq/ (prompts and
  references) to help AI assistants use sq, plus related nfpm/AUR packaging
  metadata. Kudos to @drluckyspin.

Changed

• The sq.io website is now hosted in the ./site/ dir instead
  of in a separate repo.

Fixed

• [#555], #556: ClickHouse driver now populates md.Catalog correctly.

Added

• [#552]: Agent-focused skill content under skills/sq/ (prompts and references) to help AI assistants use sq, plus related nfpm/AUR packaging metadata. Kudos to @drluckyspin.

Changed

• The sq.io website is now hosted in the ./site/ dir instead of in a separate repo.

Fixed

• [#555], [#556]: ClickHouse driver now populates md.Catalog correctly.

Added

• 🐥 #503: Initial ClickHouse support. This is a beta release of the ClickHouse functionality; please open an issue if you find problems.
  • See docs for usage.
  • Note that the docs are still WIP, see the ClickHouse README.md for additional info.
  • Kudos to @drluckyspin for the ClickHouse work.

Added

• 🐥 #503: Initial ClickHouse support. This is a beta release of the ClickHouse functionality; please open an issue if you find problems.
  • See docs for usage.
  • Note that the docs are still WIP, see the ClickHouse README.md for additional info.
  • Kudos to @drluckyspin for the ClickHouse work.

• #470: Previously, the --src.schema=[CATALOG.]SCHEMA flag did not allow
  the schema or catalog name to begin with a digit. This is now permitted.
  Kudos to @majiayu000.

  # Inspect schema "123schema" via the @pg1 source
  $ sq inspect @pg1 --src.schema=123schema
  
  # Inspect schema "789schema" in catalog "456catalog" via the @pg1 source
  $ sq inspect @pg1 --src.schema=456catalog.789schema

• #542: Upgrade codebase to Go 1.26.

Changed

• [#470]: Previously, the --src.schema=[CATALOG.]SCHEMA flag did not allow the schema or catalog name to begin with a digit. This is now permitted. Kudos to @majiayu000.

  # Inspect schema "123schema" via the @pg1 source
  $ sq inspect @pg1 --src.schema=123schema
  
  # Inspect schema "789schema" in catalog "456catalog" via the @pg1 source
  $ sq inspect @pg1 --src.schema=456catalog.789schema
  

• [#542]: Upgrade codebase to Go 1.26.

🫡: This patch release addresses issues reported by @Dialga.

Fixed

• #532: sq sql now uses the usql lib for SQL input mode determination
  (does the SQL input execute a query or a statement?) replacing the custom
  implementation introduced in v0.48.11. The usql impl is battle-tested:
  there's no point reinventing that wheel.

Changed

• #531: Several changes to sq version handling.
  • sq version now returns faster by reducing the update-check timeout from
    2s to 500ms. Additionally, the version check now uses the official
    homebrew-core formula instead of the legacy tap. There's still future work to be
    done here to make it possible to configure or disable this update-check behavior.
  • Relatedly, sq now warns instead of erroring when the config file's
    config.version is newer than the sq build version. This allows users to
    downgrade to older sq versions for testing or debugging, at the small risk
    of config schema incompatibilities (which will likely error out). There's
    future work to be done to improve how sq stamps the config schema version
    in sq.yml (currently it uses the build version rather than tracking actual
    config schema changes).

🫡: This patch release addresses issues reported by @Dialga.

Fixed

• [#532]: sq sql now uses the usql lib for SQL input mode determination (does the SQL input execute a query or a statement?) replacing the custom implementation introduced in v0.48.11. The usql impl is battle-tested: there's no point reinventing that wheel.

Changed

• [#531]: Several changes to sq version handling.
  • sq version now returns faster by reducing the update-check timeout from 2s to 500ms. Additionally, the version check now uses the official homebrew-core formula instead of the legacy tap. There's still future work to be done here to make it possible to configure or disable this update-check behavior.
  • Relatedly, sq now warns instead of erroring when the config file's config.version is newer than the sq build version. This allows users to downgrade to older sq versions for testing or debugging, at the small risk of config schema incompatibilities (which will likely error out). There's future work to be done to improve how sq stamps the config schema version in sq.yml (currently it uses the build version rather than tracking actual config schema changes).

Fixed

• #502: sq sql now properly executes single SQL statements (INSERT, UPDATE,
  DROP, etc.) instead of incorrectly running them as queries.

  • This fixes broken behavior with strict database drivers and ensures affected
    row counts are correctly reported. Note that sq sql is designed to accept
    only a single SQL statement/query in the SQL input string; behavior is
    undefined for multiple statements in the input.
  • Thanks to @drluckyspin for the fix.

• #520: sq add and sq ls erroneously printed source password for SQL Server URLs
  in some circumstances.

• #469: Column widths were too wide when using --no-header flag. Header text
  is now excluded from column width calculation when headers are disabled.
  Thanks to @majiayu000 for the fix.

Fixed

• [#502]: sq sql now properly executes single SQL statements (INSERT, UPDATE, DROP, etc.) instead of incorrectly running them as queries.

  • This fixes broken behavior with strict database drivers and ensures affected row counts are correctly reported. Note that sq sql is designed to accept only a single SQL statement/query in the SQL input string; behavior is undefined for multiple statements in the input.
  • Thanks to @drluckyspin for the fix.

• [#520]: sq add and sq ls erroneously printed source password for SQL Server URLs in some circumstances.

• [#469]: Column widths were too wide when using --no-header flag. Header text is now excluded from column width calculation when headers are disabled. Thanks to @majiayu000 for the fix.

Fixed

• [#506]: Fixed two XLSX-related issues (sadly, both are regression fixes):
  • Stdin detection: Fixed type detection failing for XLSX files created by various tools (e.g., the excelize library). These files have varying internal ZIP structures that the previous detection couldn't handle. Detection now scans ZIP local file headers instead of relying on fragile magic number heuristics.
  • Output colorization: Fixed XLSX binary output (--xlsx) being corrupted when written to stdout. The colorization decorator was modifying the binary data. XLSX format now bypasses colorization, like --raw output already does.

Changed

• [#504]: Updated golangci-lint to v2.7.2, along with Go dependencies and GitHub Actions workflow versions. Other tool versions have been updated too.

  Note that Go tool dependencies are now located in the tools/ directory, each with its own go.mod. Tools are invoked via go tool -modfile, e.g. go tool -modfile=tools/golangci-lint/go.mod golangci-lint. See the Makefile and tools/README.md for more detail.