Sling v1.5.26 (2026-08-17T10:10:23Z)

• Critical fix for DuckDB OOM issue.

v1.5.26 Latest

Latest

Compare

Choose a tag to compare

Sling v1.5.25 (2026-08-16T20:07:28Z)

New Features

• SQL Server named instances: host\instance and host/instance are parsed into host + instance. Warns when port 1433 is set alongside an instance name (the driver ignores the instance), and surfaces a clearer error when SQL Server Browser (UDP 1434) cannot resolve the instance.

• DuckDB max_line_size override: Connection property max_line_size now overrides DuckDB read_csv line limits across temp-file, named-pipe, and HTTP import paths.

Bug Fixes

• DuckDB large text/XML columns: read_csv max_line_size is raised to 256 MB for text-class columns (not only binary), so large text/ntext/xml/varchar(max)/clob values no longer fail at the 2 MB default (fixes #787). CSV error hints now suggest copy_method: arrow_http only for DuckDB-class connections.

• ADBC 3-part table names: Ingestion now targets the catalog/database in database.schema.table instead of the connection default database (fixes #785).

• Parquet TIME and UUID columns: Parquet writes with time/timez/uuid columns no longer panic from an Arrow builder type mismatch.

• Multi-file reader hangs and duplicate listings: Remote file bodies are opened lazily and prefetched with a bounded worker pool (issue #789), instead of holding one idle connection per file. Duplicate listing paths (e.g. Google Drive) are skipped.

• Missing merge primary key: Merge config generation no longer nil-dereferences when a specified primary key column is absent; it returns a clear error instead.

• Timestamp timezone through Arrow: Source timezone is preserved through Arrow conversion (e.g. Snowflake TIMESTAMP_TZ). Zone-labeled datetime stays TIMESTAMP instead of being promoted to TIMESTAMPTZ. Iceberg appends stamp timestamps as UTC so they match Iceberg timestamptz.

• DuckDB producer/consumer hangs: Import cancels the producer when the consumer stops, aborts Arrow/CSV flushes on cancelled streams, and kills stalled DuckDB queries after 60 minutes (SLING_DUCKDB_STALL_TIMEOUT; 0 disables).

• CDC snapshot logs: Per-stream CDC snapshot output is attributed to the correct stream instead of mixing into the group log.

• MySQL/MariaDB CDC timestamps: TIMESTAMP keeps its offset (timestamptz) and DATETIME stays datetime, so CDC rows match the snapshot path instead of writing TIMESTAMP_NTZ to targets.

v1.5.25

Compare

Choose a tag to compare

Sling v1.5.24 (2026-08-09T21:32:00Z)

New Features

• ADBC auto-setup: Automatically downloads the ADBC driver manager (conda-forge) when missing, and auto-installs drivers via the dbc CLI when a driver path is not found locally (disable with SLING_DISABLE_DBC_AUTO_INSTALL). Windows discovery for adbc_driver_manager.dll is included.

Bug Fixes

• Redshift AWS credential chain: Falls back to the default AWS credential chain (env vars, shared profiles, IAM roles) when Redshift/S3 credentials are not explicitly configured, with clearer errors and redacted credential logging.

• Soft-merge delete guard NULL-safe: CDC soft-merge _sling_synced_op != 'D' guards now use COALESCE across dialects so pre-CDC target rows with NULL _sling_synced_op are no longer skipped (including Redshift).

• ADBC connection keys and auth: Corrected driver property keys (e.g. Snowflake/Trino uri), added Snowflake PAT and key-pair (JWT) auth, fixed BigQuery auth_type handling, and improved adbc_uri overrides per database type.

• ADBC library load diagnostics: Load failures (outdated libstdc++/glibc, wrong arch, ADBC_DRIVER_MANAGER_LIB) now return actionable remediation steps, including an LD_PRELOAD hint on Linux when needed.

• Column select vs SQL expressions: * inside SQL/JSONPath expressions (e.g. concat('a*', id), '$[*].amount') is no longer treated as a column glob; unmatched globs warn instead of silently no-oping.

• DuckDB Arrow / concurrency: Upgraded DuckDB, fixed copy_method → arrow_http when DUCKDB_USE_ARROW is set, improved Arrow stream error handling, and fall back to CSV when Arrow is unavailable due to a locked instance.

• Bool vs int cast to string: Boolean→string casting no longer uses integer comparisons/CAST that break on PostgreSQL/Redshift; integers targeting string columns use a plain varchar cast.

• Incremental state time format: time.Time state values are stored as RFC3339 so watermarks round-trip correctly with cast.ToTime (avoids full re-pulls).

• GCS auth collision: Resolved multiple-credential-option errors with newer Google API clients; added GC_KEY_BODY, GC_KEY_FILE, KEYFILE, and GC_CRED_API_KEY aliases.

v1.5.24

Compare

Choose a tag to compare

Sling v1.5.23 (2026-08-03T16:51:29Z)

New Features

• Three-part object names (database.schema.table): Table names can now be qualified with a database/catalog for dialects that support it (Snowflake, Databricks, Trino, BigQuery, DuckDB/DuckLake/MotherDuck, Fabric, SQL Server, Azure, Azure DWH). The database qualifier now flows into metadata queries (columns, primary keys, indexes, DDL), Snowflake internal-stage creation, and temp tables, which inherit the target's database. Dialects that don't support it now reject 3-part names with a clear error instead of silently misbehaving.

• Oracle read performance (prefetch_rows): New connection property to control the number of rows fetched per network round-trip, defaulting to 100 instead of go-ora's 25. Large table reads are significantly faster; lower it (e.g. 25) for tables with large BLOB/CLOB columns to reduce memory usage.

• ScyllaDB improvements: Column metadata is now read from system_schema.columns (instead of select * limit 1), with proper native CQL → Sling type mapping, keyspace/table/view introspection, truncate/add column/drop column support, CQL-safe DROP TABLE generation, NetworkTopologyStrategy for keyspace creation, and json mapped to text.

• Upsert-by-insert for key-value stores: incremental and backfill modes with a primary key now work with direct writes for ScyllaDB, MongoDB, and Azure Table, where INSERT is already an upsert-by-PK — no more falling back to a temp table or erroring out.

• Go 1.26: Toolchain and dependencies bumped to Go 1.26.0.

Bug Fixes

• ClickHouse pending mutations block table drops: DropTable now kills in-flight mutations referencing the table before dropping it, avoiding failures when a previous delete/update mutation is still running.

• ClickHouse long loads fail with "bad connection" at commit: Large ClickHouse loads that held a single transaction open for the whole run would fail when the connection was reaped underneath mid-commit. Loads now commit on a wall-clock cadence (batch_max_duration, default 5 minutes for ClickHouse), and ClickHouse connections use a bounded max lifetime/idle time so the pool retires stale connections cleanly.

• S3 transfers truncated on large/slow objects: Removed the client-wide 300s HTTP timeout, which capped the entire request including body reads. Connect/handshake/header phases remain bounded by transport timeouts; a whole-request deadline can still be opted into via HTTP_TIMEOUT.

• SQL Server merge double-casting: The MERGE template now uses casted field variants (set_fields_casted, src_insert_fields_casted), since src is a derived table where the cast has already been applied — previously the cast was applied twice.

• SQL Server integer-to-string cast: Integers targeting a string column are now cast with a plain cast(... as varchar); only actual boolean columns get the 1/0 → 'true'/'false' conversion. Previously integer columns were incorrectly rendered as true/false.

• Nested format variables missing during pre-render: Nested maps (source.type, target.type, stream.table, …) are now built before target-object pre-rendering and refreshed afterwards, so JMESPath lookups resolve in all cases.

• Wildcard streams not matching explicit stream config: Wildcard expansion now matches explicit stream entries on schema.table first, so a 2-part stream definition still matches when the table renders with a database qualifier.

• Command/SSH hook output bypassing log sinks: command hook output (both local and SSH) is now routed through env.Println instead of writing straight to os.Stdout/os.Stderr, so it is properly captured by log sinks. Partial trailing lines are flushed at the end.

v1.5.23

Compare

Choose a tag to compare

Sling v1.5.22 (2026-07-26T23:16:01Z)

New Features

• ScyllaDB connector: New scylladb database connection type for basic import/export, including connection template, default port (9042), and NoSQL type registration.

• MariaDB CDC reader: Dedicated binlog reader for MariaDB (separate from MySQL) handling the 4-column SHOW BINLOG STATUS DDL, the MariaDB GTID scheme, and transaction-boundary reads when resuming by file/offset.

• CDC change_feed option: Reference a pre-provisioned, DBA-managed server-side CDC object instead of relying on the engine default — a PostgreSQL publication, a SQL Server capture instance, or an Oracle GoldenGate stream. Takes precedence over engine-specific properties (e.g. gg_stream).

• Composite unique keys: Nested unique key groups (e.g. unique: [[a, b]]) are now preserved as composite unique indexes in generated DDL instead of being flattened into separate single-column constraints.

• Redshift IAM role & profile auth: Added AWS_ROLE_ARN and AWS_PROFILE support for S3 authentication during Redshift COPY/UNLOAD, alongside the existing access-key/session-token paths. AWS_ROLE_ARN=default uses the cluster's attached default IAM role.

• Pipeline execution state & CLI args: Pipeline runtime state now exposes execution-level fields (ID, start time, status, errors), and CLI arguments are surfaced via SLING_CLI_ARGS_MAP with dashed keys normalized to underscores and list values (primary-key, select, streams) preserved as structured data.

Bug Fixes

• on_failure: defer in hook group loops: Deferred step failures no longer abort the group mid-loop. They are collected across all loop iterations (concurrent and sequential) and surfaced only after every iteration completes; hard failures still abort immediately.

• CDC replay_from in shared-reader groups: replay_from was silently ignored when multiple streams shared a CDC reader — the group computed its start position from each stream's already-advanced position. Stream positions are now rewound before the shared start position is computed, matching single-stream semantics.

• MariaDB CDC table filtering (#775): Wildcard-expanded stream names retained their identifier quoting (e.g. `sf`.`fakturs`), so no binlog rows event ever matched the filter set and runs reported "read 0 CDC events". Table names are now normalized (quotes stripped, lower-cased) before filtering.

• Replication runtime state race: ReplicationConfig.RuntimeState now locks state access to prevent race conditions from concurrent callers.

v1.5.22

Compare

Choose a tag to compare

Sling v1.5.21 (2026-07-15T13:52:59Z)

New Features

• CSV skip_lines source option: New skip_lines source option to skip a fixed number of leading lines (e.g. preamble rows) before the header is parsed.
• Runtime variables in stream SQL: Stream sql: queries are now re-rendered after start-hooks run, so {store.*}, {state.*}, and {execution.*} variables resolve at query time (also exposed in the format map).
• SQL Server pre-fetched access-token auth: Added ActiveDirectoryServicePrincipalAccessToken support — connect and BCP-import using a pre-fetched Azure AD access token (delivered via the security-token protocol, not the DSN).
• Redshift Serverless auto-detection: Connections to redshift-serverless.amazonaws.com now automatically use the Redshift connector.
• Composite table-key form: Primary/unique/partition keys now accept the wrapped/composite list form (e.g. primary: [[user_id, brand_id]]), flattened to behave identically to the flat form.
• API composite primary-key dedup: A composite primary_key declared on an API spec now deduplicates records at the extraction layer (first-seen wins), preventing duplicate-key violations on merge.

Bug Fixes

• StarRocks multi-chunk stream load: Probe the FE with an empty body to discover the CN/BE URL up front and reuse the redirect host across chunks — avoids re-uploading each chunk's full body and the resulting timeouts on large loads.
• Redshift merge/CDC SQL: Removed unsupported table aliases from DELETE/UPDATE statements in merge, delete-insert, and change-capture templates.
• Unsized decimal cast overflow: An explicit decimal cast (type only, no precision) no longer locks the sample-inferred precision, which could silently saturate/overflow later rows on strict engines like StarRocks.
• Explicitly-typed columns preserved during casting: Columns pinned via columns: are no longer reverted to string/decimal mid-stream when a value fails to cast.
• Oracle sqlldr UTF-8: Added CHARACTERSET AL32UTF8 to the sqlldr control template to correctly load multi-byte UTF-8 data.
• DuckDB http_timeout: The raised http_timeout is now applied on every session (not only when the httpfs extension is registered), with a http_timeout connection property to override it.
• DuckDB describe errors: A failing Describe (e.g. a missing table) now surfaces the real underlying error instead of returning an empty result.
• Snowflake private-key validation: Key-pair auth now returns actionable errors for wrong-format keys (PKCS#1, OpenSSH) and mismatched passphrase cases, with conversion instructions.
• API authenticator errors: Authenticate no longer dereferences a nil authenticator, and unsupported authentication types now return a clear error.
• Spurious API state warning: The "use SLING_STATE for incremental mode" warning is no longer shown for non-incremental API syncs.
• Deprecated queues: warning: Downgraded the deprecated top-level queues: spec-field warning to debug (queues are auto-detected).
• API rule messages: Retry/rule messages now render request-scoped variables so their interpolated suffixes resolve correctly.

v1.5.21

Compare

Choose a tag to compare

Sling v1.5.20 (2026-06-07T10:37:59Z)

See https://github.com/slingdata-io/sling-cli/ for more details.

v1.5.20

Compare

Choose a tag to compare

Sling v1.5.19 (2026-05-25T11:50:17Z)

New Features

• Column reordering and renames in select: The select clause now supports reordering and renaming columns inline. Combined with stricter handling of select exclusions and excluded PKs, this gives finer control over output schemas without needing downstream transforms.

• Oracle LOB / binary fidelity to Snowflake: End-to-end row-level fidelity for Oracle LONG, LONG RAW, and large BLOB columns when replicating to Snowflake. Snowflake COPY VIA STAGE now casts hex-encoded binary to BINARY with explicit sizing up to 64MB, Snowflake parquet file formats set binary_as_text = false, DuckDB staging decodes hex-encoded binary to BLOB for parquet, DuckDB read_csv max_line_size is raised for large hex blobs, and BinaryToHex conversion is optimized from O(n²) to O(n). Type inference is refined to preserve native binary types and apply explicit lengths.

• null_if function: New function for normalizing values to null when they match a target — useful for cleaning sentinel values out of API responses and CSV inputs.

• JSON-RPC error detection in endpoints: API endpoints now recognize JSON-RPC error envelopes and surface them through the standard rule/retry pipeline instead of treating them as successful payloads.

• Idle DB-connection reaping in sling lsp: The language server now reaps idle database connections instead of holding them open for the lifetime of the editor session.

• FileType.Exts method: New helper returns all valid extensions for a FileType, enabling correct detection of multi-extension formats (e.g. .xlsm now recognized as Excel file).

Bug Fixes

• Inline iterate.over preserved on dynamic API endpoints: Inner iterate.over is now preserved across dynamic endpoint expansion, and the over field's JSON tag is corrected so YAML/JSON specs round-trip correctly.

• BigQuery errors masked by context.Canceled: BigQuery job-wait logic is consolidated into waitForJob, which re-fetches job status with a detached context if job.Wait returns context.Canceled. Real BigQuery API errors now surface from LoadFromReader, CopyFromGCS, and CopyToGCS instead of being hidden behind a generic cancellation error.

• GCS path format in importViaGoogleStorage: Corrected the GCS URI format used when importing via Google Storage, fixing failures on BigQuery loads that route through GCS.

• Auth-sequence responses skipped default rules: Default retry/fail rules are now applied to authentication-sequence responses in dbio/api. Previously these responses bypassed the rule pipeline, producing unclear errors on auth failures. Rule application is also guarded for idempotency to prevent stacking on re-compilation.

• Endpoint rule stacking on re-compilation: Added an idempotency guard to checkResponse so endpoint rules aren't duplicated when specs are re-compiled, and ensured mutations to setup/teardown call requests/responses persist by indexing directly into slices.

• CDC metadata column validation: Added support for retrieving and validating CDC metadata columns after the initial snapshot, with additional debug logging on CDC operations to make troubleshooting easier.

• Temp-table double-drop: Skip dropping a temp table that has already been marked as dropped, and make the temp-table cleanup key unique to avoid collisions between concurrent tasks.

v1.5.19

Compare

Choose a tag to compare

Sling v1.5.18 (2026-05-12T20:15:08Z)

New Features

• Queue-only API endpoints: New queue_only flag on API spec endpoints. Such endpoints drain synchronously to populate downstream queues and emit no records of their own, and are hidden from broad enumeration (sling conns discover, wildcard replication expansion) while remaining addressable by name. queue_only is also honored as a defaults value in the spec.

• Inline iterate lists/objects for dynamic API endpoints: Dynamic endpoints now accept native YAML arrays and objects in iterate (in addition to JSON literals and JMESPath strings). Nested YAML maps are normalized to map[string]any for consistent template/JMESPath use.

• sling conns exec output formats and --limit: sling conns exec now supports --output csv|json|arrow with streaming (Arrow IPC / CSV go straight to stdout, logs to stderr) and a --limit flag (default 100, 0 = unlimited). The limit is pushed down via the dialect's limit_sql template so the database truncates server-side. sling conns discover also gains CSV output.

Bug Fixes

• CDC nil target options on shared-reader path: CDC's shared-reader write path bypassed TaskExecution.Execute() and left target options (ColumnCasing, AddNewColumns, AdjustColumnType) nil. Now calls Config.SetDefault() on the cloned config so system-wide defaults apply.

• Dataflow.Pause timeout on slow stream starts: Pause now waits up to a configurable timeout (SLING_PAUSE_TIMEOUT, default 30s) and returns false if any stream can't pause in time or the dataflow context is canceled, instead of using a short random timer.

• MySQL LoadDataLocal missing batches: Reader handler is now registered per batch in MySQL's LoadDataLocal, ensuring every batch is loaded instead of only the first.

• DuckDB Describe errors no longer masked: SELECT/WITH queries fail fast on Describe errors in DuckDB instead of swallowing the error and returning misleading results.

• Nil reader panic in ConsumeCsvReaderChl: Guards against a nil reader to prevent panics.

• Spurious HTTP-URL check in GenerateInsertStatement: Removed an unnecessary check that could misclassify insert targets.

• chunk() function return type: The pipeline chunk() function now returns *ChunkResult instead of a bare channel for consistent downstream consumption.

• Cleaner error/messages: Clearer row-count-mismatch error in WriteToDb/writeToDbDirectly, simplified pause error handling on those same paths, corrected DuckDB pipeline path and sling command output message, clarified the sling run CLI description.

v1.5.18

Compare

Choose a tag to compare

Sling v1.5.17 (2026-05-05T02:25:34Z)

New Features

• CDC shared-reader replication: New CDCGroupRunner orchestrates multi-stream CDC replications through a shared reader, with phased prepare/read-into-cache/write-and-commit execution and bounded-parallel writes.
• Connection retry in HookQuery: Automatically retries query hooks against dead database connections instead of failing the run.
• NewArrowWriter options: Arrow writer accepts additional options for finer control over output, with quieter logging in InferColumnTypes.
• sling agent service commands: Manage the Sling agent as a system service on Windows, macOS, and Linux (install/uninstall/start/stop).
• sling project CLI subcommands: New commands for managing project executions, files, and connections — including filters, output formatting, create/update/delete on jobs and files, and confirmation prompts on destructive actions.
• MCP file:// SQL support: MCP server can now execute queries from saved SQL files.
• LSP working-directory awareness: Language server resolves env and project files relative to the current working directory.

Bug Fixes

• Concurrent map iteration in API requests: Locked context in NewSingleRequest to prevent concurrent map iteration and write panics during API calls.
• Source options on zipped files: Source options (delimiter, header, etc.) are now propagated when reading from zip archives. (thanks @hermogenes)
• Standardized casing for file paths across scripts and documentation to avoid case-sensitivity issues on Linux.

v1.5.17

Compare

Choose a tag to compare