Added

• #552: Agent-focused skill content under skills/sq/ (prompts and
  references) to help AI assistants use sq, plus related nfpm/AUR packaging
  metadata. Kudos to @drluckyspin.

Changed

• The sq.io website is now hosted in the ./site/ dir instead
  of in a separate repo.

Fixed

• [#555], #556: ClickHouse driver now populates md.Catalog correctly.

Added

• 🐥 #503: Initial ClickHouse support. This is a beta release of the ClickHouse functionality; please open an issue if you find problems.
  • See docs for usage.
  • Note that the docs are still WIP, see the ClickHouse README.md for additional info.
  • Kudos to @drluckyspin for the ClickHouse work.

• #470: Previously, the --src.schema=[CATALOG.]SCHEMA flag did not allow
  the schema or catalog name to begin with a digit. This is now permitted.
  Kudos to @majiayu000.

  # Inspect schema "123schema" via the @pg1 source
  $ sq inspect @pg1 --src.schema=123schema
  
  # Inspect schema "789schema" in catalog "456catalog" via the @pg1 source
  $ sq inspect @pg1 --src.schema=456catalog.789schema

• #542: Upgrade codebase to Go 1.26.

🫡: This patch release addresses issues reported by @Dialga.

Fixed

• #532: sq sql now uses the usql lib for SQL input mode determination
  (does the SQL input execute a query or a statement?) replacing the custom
  implementation introduced in v0.48.11. The usql impl is battle-tested:
  there's no point reinventing that wheel.

Changed

• #531: Several changes to sq version handling.
  • sq version now returns faster by reducing the update-check timeout from
    2s to 500ms. Additionally, the version check now uses the official
    homebrew-core formula instead of the legacy tap. There's still future work to be
    done here to make it possible to configure or disable this update-check behavior.
  • Relatedly, sq now warns instead of erroring when the config file's
    config.version is newer than the sq build version. This allows users to
    downgrade to older sq versions for testing or debugging, at the small risk
    of config schema incompatibilities (which will likely error out). There's
    future work to be done to improve how sq stamps the config schema version
    in sq.yml (currently it uses the build version rather than tracking actual
    config schema changes).

Fixed

• #502: sq sql now properly executes single SQL statements (INSERT, UPDATE,
  DROP, etc.) instead of incorrectly running them as queries.

  • This fixes broken behavior with strict database drivers and ensures affected
    row counts are correctly reported. Note that sq sql is designed to accept
    only a single SQL statement/query in the SQL input string; behavior is
    undefined for multiple statements in the input.
  • Thanks to @drluckyspin for the fix.

• #520: sq add and sq ls erroneously printed source password for SQL Server URLs
  in some circumstances.

• #469: Column widths were too wide when using --no-header flag. Header text
  is now excluded from column width calculation when headers are disabled.
  Thanks to @majiayu000 for the fix.

Fixed

• [#506]: Fixed two XLSX-related issues (sadly, both are regression fixes):
  • Stdin detection: Fixed type detection failing for XLSX files created by various tools (e.g., the excelize library). These files have varying internal ZIP structures that the previous detection couldn't handle. Detection now scans ZIP local file headers instead of relying on fragile magic number heuristics.
  • Output colorization: Fixed XLSX binary output (--xlsx) being corrupted when written to stdout. The colorization decorator was modifying the binary data. XLSX format now bypasses colorization, like --raw output already does.

Changed

• [#504]: Updated golangci-lint to v2.7.2, along with Go dependencies and GitHub Actions workflow versions. Other tool versions have been updated too.

  Note that Go tool dependencies are now located in the tools/ directory, each with its own go.mod. Tools are invoked via go tool -modfile, e.g. go tool -modfile=tools/golangci-lint/go.mod golangci-lint. See the Makefile and tools/README.md for more detail.

Fixed

• [#446]: A bufio.ErrTooLong was being returned by bufio.Scanner, when splitting lines from input that was too long (larger than bufio.MaxScanTokenSize, i.e. 64KB). This meant that sq wasn't able to parse large JSON files, amongst other problems. The maximum buffer size is now configurable via the new tuning.scan-buffer-limit option. Note that the buffer will start small and grow as needed, up to the limit.

  $ sq config set tuning.scan-buffer-limit 64MB   # or 1024B, 64KB, 1GB, etc.
  

  A more useful error message is also now returned when the buffer limit is exceeded (the error suggests adjusting tuning.scan-buffer-limit).

Changed

• Renamed config option tuning.buffer-mem-limit to tuning.buffer-spill-limit. The new name better reflects the purpose of the option.

Changed

• Updated Go dependencies (was failing some security vulnerability scans).

Small bugfix release.

Fixed

• [#415]: The JSON ingester could fail due to a bug when a JSON blob landed on the edge of a buffer.
• The JSON ingester wasn't able to handle the case where a post-sampling JSON field had a different kind from the kind determined by the sampling process. For example, let's say the sample size was 1000, and the field zip was determined to be of kind int, because values 0-1000 were all parseable as integers. But then the 1001st value was BX123, which obviously is not an integer. sq will now see the non-integer value, and alter the ingest DB schema to a compatible kind, e.g. text. This flexibility is powerful, but it does come at the cost of slower ingest speed. But that's a topic for another release.

This release features significant improvements to sq diff.

Added

• Previously sq diff --data diffed every row, which could get crazy with a large table. Now the command stops after N differences, where N is controlled by the --stop (-n) flag, or the new config option diff.stop. The default stop-after value is 3; set to 0 to show all differences.

  # Stop on first difference
  $ sq diff @prod.actor @staging.actor --data --stop 1
  
  # Stop after 5 differences, using the -n shorthand flag
  $ sq diff @prod.actor @staging.actor --data -n5
  

• [#353]: The performance of sq diff has been significantly improved. There's still more to do.

• Previously, sq diff --data compared the rendered (text) representation of each value. This could lead to inaccurate results, for example with two timestamp values in different time zones, but the text rendering omitted the time zone. Now, sq diff --data compares the raw values, not the rendered text. Note in particular with time values that both time and location components are compared.

• sq can now handle a SQLite DB on stdin. This is useful for testing, or for working with SQLite DBs in a pipeline.

  $ cat sakila.db | sq '.actor | .first_name, .last_name'
  

  It's also surprisingly handy in daily life, because there are sneaky SQLite DBs all around us. Let's see how many text messages I've sent and received over the years:

  $ cat ~/Library/Messages/chat.db | sq '.message | count'
  count
  215439
  

  I'm sure that number makes me an amateur with these millenials 👴🏻.

  Note that you'll need to enable macOS Full Disk Access to read the chat.db file.

• sq now allows you to use true and false literals in queries. Which, in hindsight, does seem like a bit of an oversight 😳. (Although previously you could usually get away with using 1 and 0).

  $ sq '.people | where(.is_alive == false)'
  name        is_alive
  Kubla Khan  false
  
  $ sq '.people | where(.is_alive == true)'
  name         is_alive
  Kaiser Soze  true
  

Changed

• ☢️ Previously, sq diff only exited non-zero on an error. Now, sq diff exits 0 when no differences, exits 1 if differences are found, and exits 2 on any error. This aligns with the behavior of GNU diff:

  Exit status is 0 if inputs are the same, 1 if different, 2 if trouble.
  

• Minor fiddling with the color scheme for some command output.

Patch release with changes to flags. See the earlier v0.47.0 release for recent headline features.

Added

• By default, sq prints source locations with the password redacted. This is a sensible default, but there are legitimate reasons to access the unredacted connection string. Thus a new global flag --no-redact (and a corresponding redact config option).

  # Default behavior: password is redacted
  $ sq src -v
  @sakila/pg12  postgres  postgres://sakila:xxxxx@192.168.50.132/sakila
  
  # Unredacted
  $ sq src -v --no-redact
  @sakila/pg12  postgres  postgres://sakila:p_ssW0rd@192.168.50.132/sakila
  

• Previously, if an error occurred when verbose was true, and error.format was text, sq would print a stack trace to stderr. This was poor default behavior, flooding the user terminal, so the default is now no stack trace. To restore the previous behavior, use the new -E (--error.stack) flag, or set the error.stack config option.

Changed

• The --src.schema flag (as used in sq inspect, sq sql, and the root sq cmd) now accepts --src.schema=CATALOG.. Note the . suffix on CATALOG.. This is in addition to the existing allowed forms SCHEMA and CATALOG.SCHEMA. This new CATALOG. form is effectively equivalent to CATALOG.CURRENT_SCHEMA.

  # Inspect using the default schema in the "sales" catalog
  $ sq inspect --src.schema=sales.
  

• The --src.schema flag is now validated. Previously, if you provided a non-existing catalog or schema value, sq would silently ignore it and use the defaults. This could mislead the user into thinking that they were getting valid results from the non-existent catalog or schema. Now an error is returned.

Minor bug fix release. See the earlier v0.47.0 release for recent headline features.

Fixed

• Shell completion for bash only worked for top-level commands, not for subcommands, flags, args, etc. This bug was due to an unnoticed behavior change in an imported library 🤦‍♂️. It's now fixed, and tests have been added.

Changed

• Shell completion now initially suggests only sources within the active group. Previously, all sources were suggested, potentially flooding the user with irrelevant suggestions. However, if the user continues to input a source handle that is outside the active group, completion will suggest all matching sources. This behavior is controlled via the new config option shell-completion.group-filter.

Yet another morning-after-the-big-release issue, a nasty little one this time. See the earlier v0.47.0 release for recent headline features.

Fixed

• sq was failing to write config when there was no pre-existing config file. This was due to a bug in the newly-introduced (as of v0.47.0) config locking mechanism. Fixed.

This is a tiny bugfix release for a runtime issue on some Linux distros. See the previous v0.47.0 release for recent headline features.

Fixed

• sq panicked on some Linux distros that don't include timezone data (tzdata). It's now explicitly imported.

This is a significant release, focused on improving i/o, responsiveness, and performance. The headline features are caching of ingested data for document sources such as CSV or Excel, and download caching for remote document sources. There are a lot of under-the-hood changes, so please open an issue if you encounter any weirdness.

Added

• Long-running operations (such as data ingestion, or file download) now result in a progress bar being displayed. Display of the progress bar is controlled by the new config options progress and progress.delay. You can also use the --no-progress flag to disable the progress bar.
  • 👉 The progress bar is rendered on stderr and is always zapped from the terminal when command output begins. It won't corrupt the output.
• [#307]: Ingested document sources (such as CSV or Excel) now make use of an ingest cache DB. Previously, ingestion of document source data occurred on each sq command. It is now a one-time cost; subsequent use of the document source utilizes the cache DB. Until, that is, the source document changes: then the ingest cache DB is invalidated and ingested again. This is a significantly improved experience for large document sources.
• There are several new commands to interact with the cache (although you shouldn't need to):
  • sq cache enable and sq cache disable control cache usage. You can also instead use the new ingest.cache config option.
  • sq cache clear clears the cache.
  • sq cache location prints the cache location on disk.
  • sq cache stat shows stats about the cache.
  • sq cache tree shows a tree view of the cache.
• [#24]: The download mechanism for remote document sources (e.g. a CSV file at https://sq.io/testdata/actor.csv) has been completely overhauled. Previously, sq would re-download the remote file on every command. Now, the remote file is downloaded and cached locally. Subsequent sq invocations check for staleness of the cached download, and re-download if necessary.
• As part of the download revamp, new config options have been introduced:
  • http.request.timeout is the timeout for the initial response from the server, and http.response.timeout is the timeout for reading the entire response body. We separate these two timeouts because it's possible that the server responds quickly, but then for a large file, the download takes too long.
  • https.insecure-skip-verify controls whether HTTPS connections verify the server's certificate. This is useful for remote files served with a self-signed certificate.
  • download.cache controls whether remote files are cached locally.
  • download.refresh.ok-on-err controls whether sq should continue with a stale cached download if an error occurred while trying to refresh the download. This is a sort of "Airplane Mode" for remote document sources: sq continues with the cached download when the network is unavailable.
• There are two more new config options introduced as part of the above work.
  • cache.lock.timeout controls the time that sq will wait for a lock on the cache DB. The cache lock is introduced for when you have multiple sq commands running concurrently, and you want to avoid them stepping on each other.
  • Similarly, config.lock.timeout controls the timeout for acquiring the (newly-introduced) lock on sq's config file. This helps prevent issues with multiple sq processes mutating the config concurrently.
• sq's own logs previously outputted in JSON format. Now there's a new log.format config option that permits setting the log format to json or text. The text format is more human-friendly, and is now the default.

Changed

• Ingestion performance for json and jsonl sources has been significantly improved.

Fixed

• Opening a DB connection now correctly honors conn.open-timeout.

Fixed

• sq sometimes failed to read from stdin if piped input was slow to arrive. This is now fixed.