PyPI · #3252 most downloaded on PyPI

daft

0.7.251.6M downloads/moEventual-Inc/Daft ↗New releases in your daily digest, changelog included.

Distributed Dataframes for Multimodal Data

Last release 23 days ago
12 Sep 2026
Ships fairly regularly
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
8 versions withdrawn
withdrawn after publishing
14 years old
79 releases · first in 2012

Release timeline

79 releases since 2012

2012201420162018

Releases

0.7.2512 Sept 2026
feat(lerobot): support v2.0/v2.1 dataset layouts by @XuQianJin-Stars in #7381
What's Changed

✨ Features

feat(lerobot): support v2.0/v2.1 dataset layouts by @XuQianJin-Stars in #7381

feat(iceberg): add overwrite_filter for static partition overwrite by @atovk in #7419

feat(optimizer): push limit through row-preserving join sides by @hello-peter-tang in #7439

feat(io): add WebDataset reader by @rohankumardubey in #7493

🐛 Bug Fixes

fix: clean up unscheduled UDF actors by @everettVT in #7346

fix: IsIn/Decimal/UnresolvedColumn expressions fail to translate to SQL by @jiangxt2 in #7278

fix(functions): make coalesce() argument validation reachable by @DogerW666 in #7404

fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs by @kyo-tom in #7417

fix(distributed): surface background errors while input is pending by @zhouwenjiajia in #7396

fix(parquet): error instead of nulling columns when field ids are absent by @jackylee-ch in #7446

fix(catalog): avoid shared mutable default for from_postgres extensions by @DogerW666 in #7426

fix(sql): escape single quotes in SQL literal rendering by @chuqCTC in #7269

fix(sql): reject negative literals for unsigned table function options by @jackylee-ch in #7479

fix(utf8): preserve backslashes in regexp_replace replacement by @jonasdedden in #7472

fix: Fix startup of IO-Integration Docker Compose by @srilman in #7499

🚀 Performance

perf(file): avoid opening files for file_size by @yongster in #7481

♻️ Refactor

refactor(shuffle): unify distributed shuffle backend config onto daft-local-plan ShuffleBackend by @BABTUNA in #7402

📖 Documentation

docs(sql): document table function options by @jackylee-ch in #7355

docs: Update extras installation docs by @srilman in #7391

docs: remove stale TableState references by @MeteorSkyOne in #7457

✅ Tests

test: use runtime concurrency in morsel plan assertions by @ConeyLiu in #7454

👷 CI

ci(integration): build Iceberg test image from python:3.10-bookworm by @hello-peter-tang in #7489

New Contributors

@DogerW666 made their first contribution in #7404

@atovk made their first contribution in #7419

@MeteorSkyOne made their first contribution in #7457

@chuqCTC made their first contribution in #7269

@yongster made their first contribution in #7481

@NellyWhads made their first contribution in #7487

@rohankumardubey made their first contribution in #7493

@jonasdedden made their first contribution in #7472

Full Changelog: v0.7.24...v0.7.25

One column per quarter.

0.7.2415 Aug 2026

feat: add read_blob function for reading files as raw bytes by @hello-peter-tang in #7318

What's Changed

✨ Features

feat: add read_blob function for reading files as raw bytes by @hello-peter-tang in #7318
feat(utf8): add split_part string function by @ByJH in #7371

🐛 Bug Fixes

fix: ls pagination by @prayanshchh in #7372

🔧 Maintenance

chore: Update Rust deps for dependabot warnings by @srilman in #7363
chore(deps): bump opendal to 0.58.1 by @Xuanwo in #7358
chore: Remove some duplicate dependencies by @srilman in #7370
chore: Bump some dependencies by @srilman in #7378

New Contributors

@Xuanwo made their first contribution in #7358
@prayanshchh made their first contribution in #7372
@hello-peter-tang made their first contribution in #7318
@ByJH made their first contribution in #7371

Full Changelog: v0.7.23...v0.7.24

Open source →

0.7.235 Aug 2026

feat(iceberg): inject DataFile column stats into DataSourceTask for predicate short-circuit by @jiangxt2 in #7301

What's Changed

✨ Features

feat(iceberg): inject DataFile column stats into DataSourceTask for predicate short-circuit by @jiangxt2 in #7301

🐛 Bug Fixes

fix(dataframe): preserve explain() plan for all write sinks by @desmondcheongzx in #6580

🚀 Performance

perf: Optimize stats aggregation within Swordfish by @srilman in #7359

📖 Documentation

docs: migrate tutorials and examples to @daft.func / @daft.cls by @everettVT in #6729

✅ Tests

test(iceberg): cover read_iceberg against a local catalog by @jackylee-ch in #7357

🔧 Maintenance

chore: Update dashboard frontend dependencies by @srilman in #7360
chore: Ignore Dependabot warnings in benchmarks by @srilman in #7361

Full Changelog: v0.7.22...untagged-8f79647338339e50cd69

Open source →

0.7.224 Aug 2026

feat(catalog): support Gravitino Iceberg REST catalogs by @kyo-tom in #7245

What's Changed

✨ Features

feat(catalog): support Gravitino Iceberg REST catalogs by @kyo-tom in #7245
feat(io): add HDFS support via OpenDAL services-hdfs by @Lucas61000 in #7202
feat(io): export HdfsConfig from daft.io by @Lucas61000 in #7298
feat(sql): support CREATE TABLE statement by @Lucas61000 in #7186
feat(functions): add try_divide for PySpark parity by @BABTUNA in #7210
feat(file): trace media file reads by @everettVT in #7312
feat(file): trace video frame decoding by @everettVT in #7313
feat(sql): support read_parquet ignore_corrupt_files by @jackylee-ch in #7133
feat(sql): resolve read_iceberg io_config from table properties and default config by @jackylee-ch in #7273
feat(sql): support ignore_corrupt_files in read_csv by @jackylee-ch in #7264

🐛 Bug Fixes

fix(core): handle all-null list literals by @everySympathy in #7168
fix(parquet): honor row-group constraints in count_rows() count pushdown by @FANNG1 in #7288
fix: Gravitino catalog S3 credential key mismatch and _has_table error handling by @jiangxt2 in #7279
fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames by @AnayGarodia in #7229
fix(io): support writing to hdfs:// URLs that include a port by @Lucas61000 in #7307
fix(ai): route OpenAI chat completions parse through beta namespace by @colin-ho in #6995
fix: truncate default IOConfig fields in explain() output by @shreynath in #7327

♻️ Refactor

refactor(shuffle): migrate repartition onto shared backend and unify pipeline dispatch by @BABTUNA in #7275
refactor(io): migrate Delta Lake scans to DataSource by @ihopenre-eng in #7300
refactor(io): migrate Hudi scans to DataSource by @Lucas61000 in #7324
refactor: migrate IcebergScanOperator to DataSource by @Lucas61000 in #7311

📖 Documentation

docs: add motion trimming physical AI example by @ykdojo in #7341
docs: remove dead HDF5 example notebook link by @everettVT in #7352

✅ Tests

test(ai): mock beta chat completions parse by @everettVT in #7315

New Contributors

@jiangxt2 made their first contribution in #7279
@ihopenre-eng made their first contribution in #7300
@shreynath made their first contribution in #7327

Full Changelog: v0.7.21...v0.7.22

Open source →

0.7.2117 Jul 2026

fix(ai): keep UDF options out of OpenAI requests by @FANNG1 in #7277

What's Changed

🐛 Bug Fixes

fix(ai): keep UDF options out of OpenAI requests by @FANNG1 in #7277

📖 Documentation

docs(benchmark): add hand-tracking workload A/B benchmark for lerobot batched decode by @ykdojo in #7267
docs: add physical AI examples (hand tracking + reward scoring) by @ykdojo in #7271

Full Changelog: v0.7.20...v0.7.21

Open source →

0.7.2015 Jul 2026Deprecated

chore(config): deprecate min_cpu_per_task by @XiaoHongbo-Hope in #7125

What's Changed

✨ Features

feat(iceberg): push down starts_with and is_nan/not_nan filters by @jackylee-ch in #7266

🐛 Bug Fixes

fix(dashboard): support OUT_DIR on a different filesystem from the source tree by @alejandro-soto-franco in #7246
fix(daft-file): release GIL in PyDaftFile methods to prevent deadlock by @zhouwenjiajia in #7257
fix: fd and thread leak from tqdm progress bar in Jupyter by @ykdojo in #7262

♻️ Refactor

refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions by @BABTUNA in #7221

🔧 Maintenance

chore(deps): lower opt-level for opendal-service-oss release builds by @alejandro-soto-franco in #7249
chore(config): deprecate min_cpu_per_task by @XiaoHongbo-Hope in #7125

New Contributors

@alejandro-soto-franco made their first contribution in #7249
@zhouwenjiajia made their first contribution in #7257

Full Changelog: v0.7.19...v0.7.20

Open source →

0.7.1910 Jul 2026

feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220

What's Changed

✨ Features

feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220
feat(writers): add single_file kwarg to write_parquet by @aaron-ang in #7004

🐛 Bug Fixes

fix: normalize HF bucket tree paths by @everettVT in #7218
fix(session): allow set_provider to accept Provider instance for auto… by @DieselZhang in #7041
fix(telemetry): prevent interpreter-shutdown segfault from Scarf telemetry by @FANNG1 in #7203
fix(io): route oss:// through S3-compatible storage options for write_lance by @AnayGarodia in #7228
fix: Reuse file format config in Iceberg and Delta by @srilman in #7163

📖 Documentation

docs: add GooseFS connector documentation by @XuQianJin-Stars in #7193
docs: Clean up formatting in Common Crawl docs by @srilman in #7237
docs(checkpoint): add user guide and connector cross-references by @rohitkulshreshtha in #6985

👷 CI

ci: Include New Contributers in Automated Draft Releases by @srilman in #7234

New Contributors

@DieselZhang made their first contribution in #7041
@FANNG1 made their first contribution in #7203
@AnayGarodia made their first contribution in #7228

Full Changelog: v0.7.18...v0.7.19

Open source →

0.7.189 Jul 2026

feat: Remove dynamodb dependency for concurrent writes in deltalake on s3 @arpatilmh

What's Changed 🚀

✨ Features

feat: Remove dynamodb dependency for concurrent writes in deltalake on s3 @arpatilmh (#7187)
feat(lerobot): batch video-frame decode by shard @ykdojo (#7184)
feat(io): export GooseFSConfig from daft.io @XuQianJin-Stars (#7194)

🚀 Performance

perf(inline-agg): pack two-string-column keys into u64 for typed FNV grouping @BABTUNA (#6924)

Full Changelog: v0.7.17...v0.7.18

Open source →

0.7.176 Jul 2026

feat: streamline HDF5 file access @everettVT

What's Changed 🚀

✨ Features

feat: streamline HDF5 file access @everettVT (#7160)
feat: Use HF Bucket as as alternative for Common Crawl @lhoestq (#7103)
feat: Support Xet reads for HuggingFace IO @srilman (#7183)
feat(inline-agg): add AnyValue accumulator @BABTUNA (#7036)
feat(io): add GooseFS support via OpenDAL services-goosefs @XuQianJin-Stars (#7109)
feat(ai): support text file inputs in transformers prompter @YuangGao (#7174)
feat: asof_join_aligned distributed @euanlimzx (#7177)
feat: add HDF5 file support @everettVT (#7159)
feat(ai): add transformers provider for prompt() @YuangGao (#7152)
feat(lerobot): Add daft.datasets.lerobot for working with LeRobot v3 datasets @srilman (#7090)
feat(session): resolve catalog-qualified identifiers in namespace methods @Lucas61000 (#7171)

🐛 Bug Fixes

fix: add pillow to video extra @everettVT (#7213)
fix: Format nested Rust errors as a simplified backtrace in Python @srilman (#7178)
fix(iceberg): pass ignore_corrupt_files through table reads @jackylee-ch (#7147)
fix(io): default https scheme for scheme-less S3 endpoint_url @plusplusjiajia (#7111)

♻️ Refactor

refactor(inline-agg): consolidate Sum/Product/Min/Max into one macro @BABTUNA (#7151)

🔧 Maintenance

chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)
chore: Rename GoosefsConfig to GooseFSConfig @srilman (#7192)

⬆️ Dependencies

chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)

Full Changelog: v0.7.16...v0.7.17

Open source →

0.7.1626 Jun 2026

feat(droid): Add daft.datasets.droid @srilman

What's Changed 🚀

✨ Features

feat(droid): Add daft.datasets.droid @srilman (#7089)
feat(temporal): add Spark-style timezone conversions @BABTUNA (#6919)
feat: add top-level daft.concat() for concatenating multiple DataFrames @Liusixuuu (#7105)
feat(file): Add a file_exists expression @srilman (#7140)
feat: Add DataFrame.to_torch_dataloader @srilman (#6997)
feat(daft-ext): add scalar #[daft_func] proc macro @rchowell (#7043)
feat(iceberg): support SQL read_iceberg ignore_corrupt_files @jackylee-ch (#7130)
feat: add Spark-compatible string functions (translate, substring_index, soundex, ascii, chr, space) @XuQianJin-Stars (#7070)
feat(iceberg): support branch and tag reads in SQL @jackylee-ch (#7084)
feat: add support for range partition hints in clustering spec @euanlimzx (#7050)
feat: tests for aligned asof join @euanlimzx (#7072)
feat: add ignore_corrupt_files option to read_parquet, read_csv and read_iceberg @chenghuichen (#6520)
feat(ray): Implement dynamic scale-in for RaySwordfishActor @huleilei (#5903)
feat(delta-lake): support column mapping for reads @aaron-ang (#7005)
feat(functions): add string distance/similarity functions @nish2292 (#7068)

🐛 Bug Fixes

fix(scan): estimate scan size from materialized buffer, not encoded size @madvart (#7161)
fix(scan): use parquet metadata for scan task size estimates @desmondcheongzx (#6542)
fix(session): list_namespaces mirrors list_tables resolution @YuangGao (#7144)
fix(tracing): honor OTEL resource env config @RitwijParmar (#7061)
fix(parquet): Use chunk size for ArrayReader buffer size @srilman (#7149)
fix(ci): pin maturin <1.14 to keep --timings=html working @XiaoHongbo-Hope (#7145)
fix: Fix flaky dataloader test @srilman (#7142)
fix: detect unschedulable actor UDFs and warn on over-requested concurrency @XiaoHongbo-Hope (#7113)
fix(session): list_tables mirrors get_table resolution and returns Identifier @YuangGao (#7126)
fix: actor UDF followed by BlockingSink deadlocks due to held channel sender @XiaoHongbo-Hope (#7115)
fix(sql): allow read_parquet file options @jackylee-ch (#7128)
fix(parquet): cast map values to explicit schema @jackylee-ch (#7093)
fix(mcap): support http urls through fsspec handler @RitwijParmar (#7062)
fix(deps): bump ray floor to >=2.11.0 @rohitkulshreshtha (#7116)
fix(native-runner): assign unique plan-cache fingerprint when caller omits one @TechyMT (#7088)
fix(flotilla): tolerate node churn during worker startup @desmondcheongzx (#7100)
fix(deps): gate vllm to Linux so macOS/Windows resolve without CUDA wheels @rohitkulshreshtha (#7095)
fix: pass options in Gravitino PostgreSQL read method @qingfeng-occ (#7047)

🚀 Performance

perf(inline-agg): add BoolAnd and BoolOr accumulator types @BABTUNA (#6984)
perf(inline-agg): add Product accumulator type @BABTUNA (#6975)

📖 Documentation

docs: explain lazy execution in architecture flow @everettVT (#6994)
docs: collapse sidebar nav and group Guide sections @colin-ho (#7131)
docs: update uuid function docstring @everettVT (#7106)

✅ Tests

test(hf): retry transient network errors in HF integration tests @XuQianJin-Stars (#7120)
test: tolerate HuggingFace Hub HTTP 429 in classifier and IO tests @XuQianJin-Stars (#7080)
test(parquet): cover read_parquet edge cases @jackylee-ch (#7085)

🔧 Maintenance

chore: upgrade arrow-rs from 57.1.0 to 59.0.0 @mikedep333 (#7141)
chore: Replace TOS support with OpenDAL @srilman (#7135)
chore: upgrade opendal from 0.55 to 0.57.0 @XuQianJin-Stars (#7037)

👋 New Contributors

@Liusixuuu made their first contribution in #7105
@nish2292 made their first contribution in #7068
@TechyMT made their first contribution in #7088
@RitwijParmar made their first contribution in #7061
@XiaoHongbo-Hope made their first contribution in #7113

Full Changelog: v0.7.15...v0.7.16

Open source →

0.7.157 Jun 2026Deprecated

fix(integration): ray data deprecated api fix @kyuds

What's Changed 🚀

✨ Features

feat: add try_cast function for safe type conversion @XuQianJin-Stars (#6960)
feat(shuffle): default flight shuffle compression to lz4 @colin-ho (#7071)
feat(iceberg): support branch and tag reads @jackylee-ch (#7042)
feat: thread assume_sorted_and_aligned_partitions parameter through ASOF join @euanlimzx (#7067)
feat(checkpoint): distributed observability counters @rohitkulshreshtha (#7026)
feat(functions): add UUIDv7 timestamp-extraction partition transforms @jaychia (#7032)
feat: tests for range repartition hints in clustering spec @euanlimzx (#7044)
feat: allow custom datasources to specify clustering spec @jaychia (#7031)
feat(iceberg): accept PathLike metadata paths in read_iceberg @jackylee-ch (#7040)
feat(ext): native extension support on ray @rchowell (#6996)
feat: support PostgreSQL as a structured data source via Gravitino @qingfeng-occ (#6989)
feat: Better extension display @universalmind303 (#6802)
feat(gcs): implement delete for GCS object store @daiping8 (#6958)
feat(inline-agg): symbolize string group keys in multi-column grouped aggregation @BABTUNA (#6748)
feat: first_value / last_value aggs for window functions @euanlimzx (#6974)
feat(iceberg): auto-configure IOConfig for Alibaba Cloud OSS @plusplusjiajia (#6993)
feat: nearest asof joins @euanlimzx (#6953)
feat(functions): add sample_interval_seconds to video_frames @TheR1sing3un (#6832)
feat: wrappers for using Daft's FileIO / open from top-level ops @srilman (#6986)
feat: configurable flight shuffle compression @colin-ho (#6979)
feat(checkpoint): idempotent write_deltalake + cross-sink helpers @rohitkulshreshtha (#6932)

🐛 Bug Fixes

fix(asof-join): use unknown clustering spec instead of hash @euanlimzx (#7075)
fix(ray): configure worker startup timeout on runner @colin-ho (#7055)
fix(shuffle): concat recordbatches before repartition @colin-ho (#7064)
fix(flight-shuffle): reduce coordinator memory to O(map_tasks + partitions) @colin-ho (#7056)
fix(planner): propagate clustering through bound projections @jaychia (#7033)
fix(filesystem): fix pyarrow fs memory by caching by value, not identity @rchowell (#7025)
fix: strip trailing slash from resolved write/read paths @colin-ho (#6987)
fix(integration): ray data deprecated api fix @kyuds (#6871)
fix(io): use checked_shl and saturating_mul in retry jitter calculation @ARDA7787 (#6949)
fix: cast generic file arrays to media-specific types @everettVT (#6848)
fix(daft-distributed): remove unused hash_map_macro feature gate @mikedep333 (#6981)
fix: gate Self import on Python 3.11+ @everettVT (#6971)

🚀 Performance

perf: update jemalloc 5.3.0 → 5.3.1 to fix muzzy decay performance bug @everettVT (#7059)
perf(window): reduce unnecessary copies of data in finalize() step of window functions to reduce memory usage @euanlimzx (#7006)
perf(window): return Series from window ops to reduce intermediate RecordBatch copies @euanlimzx (#7011)
perf(flotilla): batch flight partition refs per finalize @cckellogg (#7008)
perf: Specialized Code Path for Hash Join on Ints @srilman (#6644)
perf: enable per-column compression configuration @rchowell (#6884)

♻️ Refactor

refactor(checkpoint): drop "seal" vocabulary from Rust API surface @rohitkulshreshtha (#7078)
refactor(file): rename File byte-range fields to position/size @aaron-ang (#6747)
refactor(distributed): rename needs_hash_repartition to can_skip_hash_repartition @euanlimzx (#7053)
refactor(checkpoint): bincode-encoded versioned FileMetadata blobs @rohitkulshreshtha (#7001)

📖 Documentation

docs: standardize Slack links to use daft.ai/slack @everettVT (#7066)
docs: reorder UDF API docs to prioritize new API and deprecate legacy @everettVT (#7028)
docs(paimon): clarify object-store IO config usage @jackylee-ch (#7029)
docs: add shuffle algorithms tuning guide @colin-ho (#7017)
docs(lance): fix daft-lance repository links @jackylee-ch (#7016)

✅ Tests

test(lance): use daft-lance APIs directly @jackylee-ch (#7010)
test(parquet): cover embedding and image roundtrips @jackylee-ch (#7021)
test(iceberg): cover partition field transform conversion @jackylee-ch (#7024)
test(lance): import test helpers from daft-lance @jackylee-ch (#7015)
test(checkpoint): optional real-S3 routing for checkpoint test suites @rohitkulshreshtha (#6934)

👷 CI

ci: catch undeclared runtime imports via TC strict + py version matrix @colin-ho (#6972)

🔧 Maintenance

chore(deps): bump the all group across 1 directory with 59 updates @dependabot[bot] (#7000)
chore: bump minimum PyArrow version to 16 @0xdeadd (#6868)

⬆️ Dependencies

chore(deps): bump the all group across 1 directory with 59 updates @dependabot[bot] (#7000)

Full Changelog: v0.7.14...v0.7.15

Open source →

0.7.1420 May 2026

feat(flotilla): hint users to switch to flight\_shuffle on large shuffles @colin-ho

What's Changed 🚀

✨ Features

feat(flotilla): hint users to switch to flight_shuffle on large shuffles @colin-ho (#6962)
feat: add uuidv7 generation @everettVT (#6909)
feat(flotilla): Distributed Limit Counter @colin-ho (#6942)
feat: add JSON functions json_array_length, json_object_keys, json_tuple @XuQianJin-Stars (#6945)
feat: add conv function for PySpark parity @YuangGao (#6910)
feat: ASOF join benchmarking scripts @euanlimzx (#6940)
feat(show): env defaults and auto alignment for preview output @BABTUNA (#6856)
feat(temporal): add Spark-style temporal aliases @BABTUNA (#6830)

🐛 Bug Fixes

fix: hash(-0.0) == hash(0.0) @rchowell (#6963)
fix: short-circuit on empty micropartitions @rchowell (#6956)

🚀 Performance

perf(parquet): rewrite reader with arrow-rs public decoder API @colin-ho (#6952)
perf(flotilla): Coalesce task outputs based on byte threshold @colin-ho (#6943)
perf(shuffle): Write one shuffle file per task instead of N partition files @colin-ho (#6948)

📖 Documentation

docs: Update API references and fix code examples @colin-ho (#6938)
docs(examples): migrate text-embeddings example to public bucket @colin-ho (#6941)

🔧 Maintenance

chore: migrate daft.io.lance to daft_lance @rchowell (#6957)
chore(deps): bump the all group across 1 directory with 4 updates @dependabot[bot] (#6863)

⬆️ Dependencies

chore(deps): bump the all group across 1 directory with 4 updates @dependabot[bot] (#6863)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.13...v0.7.14

Open source →

0.7.1315 May 2026

feat: add forward ASOF joins @euanlimzx

What's Changed 🚀

✨ Features

feat: add forward ASOF joins @euanlimzx (#6918)

🐛 Bug Fixes

fix: remove hallucinated lance-namespace code @rchowell (#6939)
fix(sql): omit connection URL from read_sql error messages and __repr__ @madvart (#6933)
fix(dashbaord): highlight failed operators for queries @cckellogg (#6930)
fix: handle broadcast of scalars when calling to_struct @euanlimzx (#6931)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.12...v0.7.13

Open source →

0.7.1213 May 2026

feat(iceberg): honor table-level write.target-file-size-bytes property @YuangGao

What's Changed 🚀

✨ Features

feat(iceberg): honor table-level write.target-file-size-bytes property @YuangGao (#6912)
feat(temporal): add Spark-style add_months and months_between @BABTUNA (#6913)

🐛 Bug Fixes

fix: revert "feat(daft-ext): scalar daft_func macro with overloading (#6844)" @rchowell (#6925)

📖 Documentation

docs: Adds daft-qdrant community extension to docs.daft.ai @Anush008 (#6921)

✅ Tests

test: re-enable Iceberg map-type round-trip test @YuangGao (#6911)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.11...v0.7.12

Open source →

0.7.1112 May 2026withdrawn: Breaking change in extensions

feat: Arrow PyCapsule Interface @aaron-ang

What's Changed 🚀

✨ Features

feat: Arrow PyCapsule Interface @aaron-ang (#6745)
feat: add Python UDAF support (@daft.udaf) @chenghuichen (#6790)
feat(dashboard): per-task rows/bytes stats with task-topology markers @samstokes (#6861)
feat(iceberg): idempotent commits via daft.idempotence-key snapshot marker @rohitkulshreshtha (#6905)
feat(lance): partitioned reads/writes @universalmind303 (#6898)
feat(dashboard): make header smaller in query details @cckellogg (#6900)
feat: streaming ASOF joins @euanlimzx (#6846)
feat: Add map_keys function to extract keys from Map type columns @qingfeng-occ (#6875)
feat: add event log config api @cckellogg (#6894)
feat(dashboard): surface query failure details in query view @cckellogg (#6897)
feat(dashboard): bound query state retention (DF-1970) @samstokes (#6896)
feat(dashboard): per-task progress updates from flotilla workers @samstokes (#6838)
feat(dashboard): plumb TaskScheduled events to distinguish pending vs running @samstokes (#6866)
feat(dashboard): show task sources in tasks sidebar @samstokes (#6879)
feat(ci): use larger runner for pr testing @cckellogg (#6890)
feat(native): emit TaskStart events from native executor @cckellogg (#6840)
feat(session): add non-materialized DataFrame temp views @BABTUNA (#6786)
feat(dtype): add Float16 (half-precision) support @universalmind303 (#6859)
feat: extension datatype casting @universalmind303 (#6803)
feat(file): add Image media type to File @chenghuichen (#6847)
feat(iceberg): idempotent snapshot commits via snapshot_properties @rohitkulshreshtha (#6841)
feat(daft-ext): scalar daft_func macro with overloading @universalmind303 (#6844)
feat(checkpoint): wire checkpoint= into csv, json, warc, iceberg, hudi, lance @rohitkulshreshtha (#6855)
feat(checkpoint): bundle source config into CheckpointConfig with tunable filter knobs @rohitkulshreshtha (#6853)
feat(checkpoint): decouple sealed-keys column name from source schema @rohitkulshreshtha (#6850)
feat: use scan task metadata to parallelize initial limit reads @samstokes (#6763)
feat(checkpoint): track in-flight writes and seal empty manifests @rohitkulshreshtha (#6835)
feat(file): add byte-range read support to FileReference @chenghuichen (#6842)
feat(list): add list_flatten expression @mytsvy (#6812)
feat(extensions): add proc macro for making writing extension functio… @universalmind303 (#6837)

🐛 Bug Fixes

fix: Handle dtype mismatch error in join_asof join keys @euanlimzx (#6904)
fix(distributed): emit empty downstream task for limit(0) @madvart (#6916)
fix(sql): redact password in SQL connection errors and repr @madvart (#6902)
fix(dashboard): split task groups by local-plan shape @samstokes (#6899)
fix(stats): close race in RuntimeStatsManager shutdown @samstokes (#6891)
fix(writers): strip partition columns from Hive-partitioned file bodies @aaron-ang (#6770)
fix(distributed): fire OperatorStart/End events per-node on Flotilla @samstokes (#6860)
fix(test): handle transient iceberg rest catalog startup failures @cckellogg (#6880)
fix(dashboard): handle offline builds in frontend @samstokes (#6867)
fix: infer audio file return dtype @everettVT (#6849)

🚀 Performance

perf(file): add buffer_size to File.open() to reduce wasteful pre-reads @chenghuichen (#6876)
perf: iceberg predicate pushdowns @rchowell (#6815)

♻️ Refactor

refactor(distributed): decouple shuffle cleanup from Ray @chenghuichen (#6809)
refactor(dashboard): remove result preview tab and ResultOut event @samstokes (#6878)

📖 Documentation

docs(dashboard): add user guide for Daft Dashboard @samstokes (#6882)
docs: change affiliation for maintainer @gweaverbiodev (#6851)

✅ Tests

test: restore fixed-size list parquet roundtrip @TuodiAunty (#6895)
test(iceberg): make predicate-pushdown limit test order-deterministic @samstokes (#6843)

👷 CI

ci(issues): auto-label issues from component selections @BABTUNA (#6820)
ci: fix dependency check on external prs @srilman (#6737)

🔧 Maintenance

chore: Revert "feat(lance): partitioned reads/writes" @universalmind303 (#6907)

👋 New Contributors

@mytsvy made their first contribution in https://github.com/Eventual-Inc/Daft/pull/6812
@TuodiAunty made their first contribution in https://github.com/Eventual-Inc/Daft/pull/6895

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.10...v0.7.11

Open source →

0.7.101 May 2026Deprecated

fix(docs): replace deprecated .struct.get() with .get() expression in examples @everettVT

What's Changed 🚀

✨ Features

feat(dashboard): Draft tasks view for Flotilla @samstokes (#6783)
feat(functions): add simhash and hamming_distance for near-duplicate detection @chenghuichen (#6821)
feat: add pmod function for PySpark parity @YuangGao (#6801)
feat: distributed range repartitioned asof joins v2 @euanlimzx (#6816)
feat(distributed): Add task metadata to task events @cckellogg (#6822)
feat: genericize RandomShuffle to support Flight shuffle backend @chenghuichen (#6808)
feat(paimon): enhance Paimon integration @YannByron (#6635)
feat(dashboard): render Repartition spec in human-readable form @BABTUNA (#6798)
feat(cli): show query id in CLI output @BABTUNA (#6755)
feat(functions): add hamming_distance function @gweaverbiodev (#6797)
feat(ext): add UDAF support to daft-ext @chenghuichen (#6789)
feat(list): add list_filter expression @aaron-ang (#6769)
feat(ext): add C++ hello extension example @universalmind303 (#6804)
feat(functions): add great_circle_distance function @gweaverbiodev (#6754)
feat: support Ray 2.55.0 @XuQianJin-Stars (#6767)
feat(distributed): Add env to enable task events @cckellogg (#6782)
feat(functions): add seq expression for per-row integer sequences @yuchen-ecnu (#6772)
feat(dataframe): add DataFrame.count_distinct() and GroupedDataFrame.count_distinct() methods @kerwin-zk (#6658)
feat(distributed): enable two-stage HLL aggregation for approx_count_distinct @desmondcheongzx (#6597)
feat(distributed): emit TaskSubmit and TaskEnd lifecycle events @cckellogg (#6759)
feat: introduce AggFn trait and two-stage UDAF aggregation pipeline @chenghuichen (#6704)
feat(cast): support cross-TimeUnit Duration casts @aaron-ang (#6766)
feat: introduce richer source stats @rchowell (#6723)
feat(dataframe): add DataFrame.product() and GroupedDataFrame.product() methods @kerwin-zk (#6655)
feat(checkpoint): distributed checkpoint filter via optimizer rule @rohitkulshreshtha (#6725)
feat(stats): Add num_tasks metric tracking across all runtime stats @samstokes (#6716)
feat: implement bin function @YuangGao (#6728)
feat: implement batch 3 temporal functions (make_date, make_timestamp, make_timestamp_ltz, last_day, next_day) @BABTUNA (#6672)
feat: percentile and median ops @aaron-ang (#6153)
feat(datatype): allow accessing simple type constructors as properties @Lucas61000 (#6620)
feat(sql): improve error reporting with source context and caret @Lucas61000 (#6572)
feat: better native execution for asof joins @euanlimzx (#6699)
feat(sql): add SAMPLE support for SQL queries @Lucas61000 (#6636)
feat(subscribers): add optional origin_node_id to py operator events @cckellogg (#6714)
feat: Union Type @PhysicsACE (#6497)
feat: implement e, pi, factorial, and hypot math functions @BABTUNA (#6682)
feat: support catalog-qualified identifiers in create_table, drop_table @BABTUNA (#6680)
feat: remote EventLogSubscriber @universalmind303 (#6701)
feat(dashboard): subscriber heartbeat and dead query detection @samstokes (#6676)
feat: add S3-backed CheckpointStore implementation @chenghuichen (#6599)
feat: distributed execution of asof joins @euanlimzx (#6667)

🐛 Bug Fixes

fix(test): update test_local_full_ls to expect canonical file:// URIs @rohitkulshreshtha (#6824)
fix: ensure paimon is installed for integration tests @rchowell (#6827)
fix(udf): resolve use_process=True subprocess deadlocks @rohitkulshreshtha (#6793)
fix(udf): handle UDF expressions with no column references (#6805) @rohitkulshreshtha (#6814)
fix(io): emit canonical file:// URIs for Windows drive paths @rohitkulshreshtha (#6817)
fix(docs): fix many docs issues @colin-ho (#6811)
fix(checkpoint): use strip_file_uri_to_path in put_bytes for Windows @rohitkulshreshtha (#6796)
fix(checkpoint): normalize Windows tempdir paths in s3_store tests @rohitkulshreshtha (#6791)
fix: respect per-method max_retries/on_error overrides @BABTUNA (#6784)
fix(core): normalize FixedSizeListArray inner field name to 'item' @veinkr-bot (#6733)
fix(tests): deflake test_sharding_with_file_scan @veinkr-bot (#6787)
fix: preserve identity partition predicates when combined with ScalarFn siblings @gavin9402 (#6695)
fix(flight-shuffle): handle all-empty inputs in shuffle cache and shu… @ohbh (#6780)
fix(flotilla): overreporting of bytes.read @universalmind303 (#6774)
fix: add missing serde feature to uuid workspace dependency @chenghuichen (#6773)
fix: respect proxy env vars (HTTP_PROXY, HTTPS_PROXY, etc.) in S3 client @BABTUNA (#6679)
fix: fix column not found when using count_rows() for sparse data @caican00 (#6703)
fix(scheduler): include dispatched tasks in autoscaling ratio @desmondcheongzx (#6388)
fix(io): Write metrics in close for the last batch @kvthr (#6606)
fix(docs): replace deprecated .struct.get() with .get() expression in examples @everettVT (#6709)
fix: incrementally ramp up Ray autoscaler resource requests to avoid exceeding cluster capacity @ohbh (#6653)
fix: Fix document example, unnest is not a param on prompt @colin-ho (#6712)
fix(stats): Track source execution time per message in pipeline stats @samstokes (#6715)
fix: add tenacity retries to Google Sheets upload in benchmarking @jeevb (#6713)
fix: skip schema pruning on Source node when can_absorb_select is false @helmanofer (#6501)
fix(io): support writing to local fs via GravitinoGvfs local issue @qingfeng-occ (#6579)

🚀 Performance

perf(optimizer): implement DP-ccp join ordering algorithm @desmondcheongzx (#6460)
perf(inline-agg): add Utf8/Binary single-column fast path @BABTUNA (#6656)

♻️ Refactor

refactor(flight): implement into_partitions for flight shuffle @ohbh (#6764)
refactor(flight): implement distributed gather for flight shuffle @ohbh (#6751)
refactor(partition-refs): consolidate partition ref types and trim unused derives @ohbh (#6742)
refactor(distributed): Remove node_origin_id self-ref in distributed @cckellogg (#6738)
refactor(inline-agg): macro-generate accumulator dispatch @BABTUNA (#6642)
refactor: flight-server passes PartitionRefs directly to the scheduler @ohbh (#6627)
refactor(daft-distributed): cleanup of statistics_manager @universalmind303 (#6718)

📖 Documentation

docs: add extension ecosystem pages @everettVT (#6836)
docs: add Tencent Cloud COS connector documentation @XuQianJin-Stars (#6818)
docs: fix critical documentation issues from audit @colin-ho (#6834)
docs: clarify audio file size description @ykdojo (#6792)
docs: Fix docsearch results to respect current documentation version @colin-ho (#6778)
docs: fix critical documentation issues 7-14 from audit @colin-ho (#6777)
docs(extensions): move authoring guide to dedicated page @everettVT (#6732)
docs(custom-code): document cpus/gpus/retries/ray_options and fix migration guide @everettVT (#6702)
docs(extensions): add community extensions subpage @desmondcheongzx (#6717)

✅ Tests

test(udf): cover combined large-stderr + no-newline use_process case @rohitkulshreshtha (#6825)
test: Add local worker execution and statistics testing infrastructure @samstokes (#6697)

🔧 Maintenance

chore: Simplify local Repartition operator @srilman (#6739)
chore(dashboard): harden npm/next.js supply chain via .npmrc @everettVT (#6727)
chore: upgrade deltalake to 1.5.0 @rchowell (#6736)
chore(parquet): lower opt-level for daft-parquet release builds @desmondcheongzx (#6690)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.9...v0.7.10

Open source →

0.7.914 Apr 2026

feat: Add support for writing Extension and Duration types to Parquet @srilman

What's Changed 🚀

✨ Features

feat: Add support for writing Extension and Duration types to Parquet @srilman (#6684)
feat(dashboard): Support event log import to dashboard @cckellogg (#6646)
feat(dashboard): display version metadata on query detail page @samstokes (#6671)
feat: implement batch 2 temporal functions @BABTUNA (#6563)
feat: add video_frames() for decoding all frames with metadata @everettVT (#6536)
feat: UUID Type @srilman (#6611)
feat(dashboard): improved dashboard metrics for bytes in/out and inflation/deflation @universalmind303 (#6640)
feat: Point-in-Time / ASOF Joins @euanlimzx (#6615)
feat(observability): track per-operator bytes_in / bytes_out @universalmind303 (#6612)
feat(file): add file_path() function @everettVT (#6621)
feat(stats): add phase-aware SortStats for distributed Sort operator @samstokes (#6632)

🐛 Bug Fixes

fix(docs): fix syntax errors in connectors documentation @colin-ho (#6683)
fix(docs): correct from_glob_paths to from_glob_path in audio-transcription workload @colin-ho (#6675)
fix(docs): correct mkdocstrings YAML indentation in file_types.md @colin-ho (#6668)
fix: Re-enable subscriber support on Flotilla @srilman (#6650)
fix: broadcast zero-field StructArray to requested length @universalmind303 (#6660)
fix: validate CheckpointId is path-safe for object stores @rohitkulshreshtha (#6641)
fix: Add Cargo Timing to CI @srilman (#6459)

🚀 Performance

perf: Parallelize post-groupby op @srilman (#6663)
perf: Do a 1-stage aggregation for distributed ListAgg @srilman (#6665)

♻️ Refactor

refactor: migrate streaming sink to use consolidated BatchManager @universalmind303 (#6568)

📖 Documentation

docs: Add documentation for missing connectors @everettVT (#6664)
docs: Add describe SQL statement, plus add orphan pages, plus see also @everettVT (#6649)
docs: fix incorrect accessor syntax in audio.md @everettVT (#6648)
docs: add missing write methods to I/O API reference @colin-ho (#6647)

👷 CI

ci: Add ClickBench to CodSpeed @srilman (#6669)
ci: Show crate diff in PRs @srilman (#6645)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.8...v0.7.9

Open source →

0.7.88 Apr 2026

chore!: Remove unused max_task_backlog parameter @srilman

What's Changed 🚀

💥 Breaking Changes

chore!: Remove unused max_task_backlog parameter @srilman (#6591)

✨ Features

feat: Expose Subscribers through nicer APIs @srilman (#6631)
feat(dashboard): heatmap coloring for nodes. @universalmind303 (#6628)
feat(dataframe): add DataFrame.skew() global aggregation method @kerwin-zk (#6619)
feat(dashboard): add arrows indicating pipeline direction @universalmind303 (#6625)
feat: wire CheckpointId through Flotilla execution pipeline @rohitkulshreshtha (#6567)
feat: add image_hash() for image deduplication @chenghuichen (#6485)
feat: support get function from catalog @gavin9402 (#6524)
feat(dataframe): add var() method to DataFrame and GroupedDataFrame @kerwin-zk (#6584)
feat: checkpoint based on distributed key-existence filter @everySympathy (#5931)
feat(distributed): make flotilla worker actor startup timeout configurable @desmondcheongzx (#6592)

🐛 Bug Fixes

fix: Fix nightly Daft version resolution in Ray runtime env @jeevb (#6630)
fix(dashboard): pass partition sets to repr_json so plan matches execution topology @samstokes (#6576)
fix(scan): skip getting bytes when range start equals end in daft async reader @gweaverbiodev (#6602)
fix(io): retry transient errors on initial GET request @desmondcheongzx (#6544)
fix(dashboard): use smart per-node stats aggregation for distributed execution @samstokes (#6574)
fix(io): handle schema-evolved Iceberg columns in Parquet predicate pushdown @sankarreddy-atlan (#6551)
fix(dashboard): prevent flotilla workers from sending spurious lifecycle events @samstokes (#6573)

🚀 Performance

perf: Optimize GroupBy Map Building & List-Agg @srilman (#6613)
perf(flight): Read local shuffle data directly from disk @srilman (#6436)
perf(inline-agg): add min/max accumulator types @BABTUNA (#6604)
perf: inline vectorized aggregation for grouped count/sum @desmondcheongzx (#6345)

♻️ Refactor

refactor: migrate PaimonScanOperator to DataSource API @chenghuichen (#6600)
refactor(subscriber): collapse trait to single sync on_event method @cckellogg (#6593)
refactor: consolidate BatchManager as single buffer abstraction @universalmind303 (#6566)
refactor: Use DAFT_REF_NAME and DAFT_SHA env vars for benchmark run metadata @jeevb (#6610)
refactor: Minor parameterization for benchmarking workflows @jeevb (#6609)
refactor(distributed): unify repartition exchange write flow across ray and flight @ohbh (#6499)

📖 Documentation

docs: add custom SQL filter documentation for Lance @Jay-ju (#5916)

👷 CI

ci(test): reduce Ray resource requests in chained skip_existing tests @desmondcheongzx (#6633)
ci(test): reduce Ray resource requests in skip_existing tests @desmondcheongzx (#6629)
ci(deps): fix CI failures from dependabot bump #6570 @desmondcheongzx (#6596)

🔧 Maintenance

chore(cargo): inherit edition/version from workspace in root package @yew1eb (#6329)
chore!: Remove unused max_task_backlog parameter @srilman (#6591)
chore(dashboard): add ?debug query param for SSE event console logging @samstokes (#6577)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.7...v0.7.8

Open source →

0.7.72 Apr 2026

feat(subscriber): add Event enum and on\_event dispatch for subscriber @cckellogg

What's Changed 🚀

✨ Features

feat(subscriber): add Event enum and on_event dispatch for subscriber @cckellogg (#6508)
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication @linguoxuan (#6338)
feat: add support for writing _SUCCESS files in parquet operations @Lucas61000 (#6090)
feat: add dual-send telemetry to osstelemetry.io @ykdojo (#6540)
feat(tracing): add DAFT_TRACE console tracing with configurable format @cckellogg (#6458)
feat: convert and replace timezone @aaron-ang (#6106)
feat: added concat_ws function @euanlimzx (#6543)
feat: Add a random shuffle operation @srilman (#6481)

🐛 Bug Fixes

fix(dataframe): explain plan does not make sense when using a write_ op @Abyss-lord (#6564)
fix(native): preserve df metrics after stats manager shutdown @cckellogg (#6555)
fix: hide dashboard result preview for unsupported queries (Flotilla) @samstokes (#6557)
fix: short-circuit evaluation for coalesce @Lucas61000 (#6525)
fix(ci): use content-hash mtimes to prevent stale cargo cache artifacts @desmondcheongzx (#6546)

🚀 Performance

perf(parquet): yield individual batches in streaming path to fix morsel cache locality @desmondcheongzx (#6558)
perf: reduce binary size by de-monomorphizing sort and dispatch paths @NikkeTryHard (#6541)
perf: Rewrite decode(..., "utf-8") as a cast @srilman (#6537)

📖 Documentation

docs: add UDF patterns tutorial notebook and example page @everettVT (#6539)

🔧 Maintenance

chore(deps): bump the all group with 9 updates @dependabot[bot] (#6569)
chore: harden supply chain — exclude-newer, pin actions and uv @everettVT (#6565)
chore(io): replace python source shim on the rust side @rchowell (#6556)

⬆️ Dependencies

chore(deps): bump the all group with 9 updates @dependabot[bot] (#6569)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.6...v0.7.7

Open source →

0.7.631 Mar 2026Deprecated

chore: deprecate all extra lance functions in favor of new daft-lance @universalmind303

What's Changed 🚀

💥 Breaking Changes

feat!: add iceberg table support with the gravitino catalog @rchowell (#6509)

✨ Features

feat!: add iceberg table support with the gravitino catalog @rchowell (#6509)
feat(swordfish): plan caching @colin-ho (#6278)
feat: add daft-checkpoint crate with CheckpointStore trait and in-mem… @rohitkulshreshtha (#6511)
feat: optionally keep the native progress bar visible after job completion @BABTUNA (#6449)
feat(build): add experimental cross-wheel build target @cckellogg (#6360)
feat: DataFrame.from_files @rchowell (#6507)
feat: Add Apache Paimon Lake Format Support @chenghuichen (#6450)
feat: add current_date, current_timestamp, current_timezone and SQL aliases @BABTUNA (#6495)
feat(subscribers): add typed event for operator and stats callbacks @cckellogg (#6479)
feat: support to_ray_dataset() from the native runner @desmondcheongzx (#6486)
feat: async data sources and tasks @rchowell (#6492)
feat: Add hex and unhex @Lucas61000 (#6373)
feat: implement _repr_mimebundle_ for DataFrame @Abyss-lord (#6385)
feat: Add string strip function @Lucas61000 (#6372)
feat: add plan fingerprinting for LocalPhysicalPlan @colin-ho (#6276)
feat: Add metrics for join operators @srilman (#6391)
feat(observability): add process-level memory and CPU monitoring @desmondcheongzx (#6428)
feat: define a single DataSource trait matching our python interface. @rchowell (#6427)
feat(io): Propagate storage_options to Lance backend to enable Apache Gravitino (gvfs://) support @qingfeng-occ (#6430)
feat(subscribers): add JSONL event log subscriber @cckellogg (#6420)
feat: Add support for reading whole text files to read_text @plotor (#6354)
feat(expressions): support count(mode='all') without expr @Abyss-lord (#6358)
feat: kafka bounded datasource @everySympathy (#5970)
feat(udf): support ray_options and resource overrides in UDF v2 @Jay-ju (#5982)
feat: Add global agg support for Dataframe select @caican00 (#6067)

🐛 Bug Fixes

fix: correct grid card indentation on modalities overview page @everettVT (#6530)
fix(series): unable to cast directly from string to fixed_size_binary @Abyss-lord (#6521)
fix: respect scheme aliases in Iceberg writes @rchowell (#6512)
fix(series): cast should not change input argument name @Abyss-lord (#6515)
fix: preserve show options without setting format @aaron-ang (#6102)
fix: strict inequalities must NOT be relaxed for Identity transform @gavin9402 (#6484)
fix: fix column missing issue for jsonl record schema when reading json/jsonl files @caican00 (#5681)
fix: remove use_safetensors=True to allow model fallback @rchowell (#6489)
fix: close connections in io tests and address other warnings @rchowell (#6470)
fix: deflake a probabilistic test that is causing release uploads to fail @rchowell (#6475)
fix(io): fix JSONL byte-range reading and make chunk_size byte-based @everySympathy (#6374)
fix(sql): fall back to single scan when partition bounds are degenerate @qingfeng-occ (#6482)
fix: clippy errors on iterators in tests @rchowell (#6476)
fix: handles schema inference errors when partition columns are NULL @rchowell (#6467)
fix: keep field metadata when converting series to ffi schema @universalmind303 (#6471)
fix: Track durations for all local operators in distributed @srilman (#6438)
fix(optimizer): allow column-alias projections in join reordering @desmondcheongzx (#6412)
fix: Expose json_target_filesize execution config @plotor (#6382)
fix: fix for bad merge @universalmind303 (#6419)
fix(sql): Fix error in TPC-DS q23 @Lucas61000 (#6317)
fix: Preserve materialized stats when splitting granular projections @plotor (#6376)
fix(dataframe): preserve field order in from_pylist @Abyss-lord (#6369)
fix: report bytes.read for local CSV scans @Abyss-lord (#6380)

🚀 Performance

perf: Optimize list_* aggregation functions @srilman (#6494)
perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch @universalmind303 (#6444)

♻️ Refactor

refactor: implement generic shuffle interface for flight shuffle @ohbh (#6456)
refactor(ext): use single, publishable crate for daft-ext types @rchowell (#6452)
refactor(optimizer): introduce bitmask-based RelationSet for join reordering @desmondcheongzx (#6421)
refactor(io): merge conditional arms in opendal_backends ls operator @qingfeng-occ (#6461)
refactor(io): move common source fields to own type @rchowell (#6451)
refactor: remove EmptyScan plan node @colin-ho (#6275)
refactor(swordfish): Streaming flight shuffle read @colin-ho (#6269)

📖 Documentation

docs: clean up roadmap wording @ykdojo (#6517)
docs: update roadmap for 2026 @everettVT (#6480)
docs: fix daft dashboard command example @qingfeng-occ (#6483)
docs: document get_or_infer_runner_type and get_or_create_runner @ykdojo (#6453)
docs(kafka): add dedicated connector docs for kafka @everySympathy (#6433)
docs: improve skills install instructions @ykdojo (#6422)
docs: add Daft Skills page and Claude Code plugin config @ykdojo (#6368)

👷 CI

ci: disable HF xet storage backend in unit tests @rchowell (#6493)
ci: refresh apt index before installing deps @rchowell (#6491)
ci: remove broken links @rchowell (#6490)

🔧 Maintenance

chore: exclude symlink files from end-of-file-fixer pre-commit hook @chenghuichen (#6531)
chore(ci): migrate BuildJet runners to Blacksmith @madvart (#6416)
chore: use dedicated OSS AWS account @jeevb (#6442)
chore: Collection of random Swordfish cleanups @srilman (#6454)
chore(swordfish): Small swordfish cleanups @colin-ho (#6488)
chore: use concurrent broken link checker @rchowell (#6474)
chore: remove _FixEmptyStructArrays workaround @universalmind303 (#6478)
chore: deprecate all extra lance functions in favor of new daft-lance @universalmind303 (#6487)
chore: remove deprecated catalog modules @rchowell (#6457)
chore: Disable cargo check in local dev @srilman (#6439)
chore: Abstract metrics meters for execution @srilman (#6384)
chore: Remove some extra & duplicate Rust packages @srilman (#6435)
chore: Make the partition threshold to use pre_shuffle_merge configurable @plotor (#6356)
chore: rename DataSource enum to ScanSource @rchowell (#6417)
chore: separates source types from file format configuration @rchowell (#6406)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.5...v0.7.6

Open source →

0.7.517 Mar 2026

perf!: lazy-load lance to speed up import daft @desmondcheongzx

What's Changed 🚀

💥 Breaking Changes

perf!: lazy-load lance to speed up import daft @desmondcheongzx (#6328)
feat!: add ddof parameter to stddev (default ddof=1) @aaron-ang (#6142)

✨ Features

feat(io): support numeric range expansion in glob patterns @singularityDLW (#6127)
feat(dashboard): add query result preview tab @samstokes (#6327)
feat: Add read_text API to support reading text files @plotor (#6111)
feat: Add protocol aliases for IOConfig @universalmind303 (#6252)
feat(observability): Improve distributed operator names @srilman (#6324)
feat(metrics): add node_plan_id for distributed stats attribution @cckellogg (#6332)
feat: adds support for daft native extensions via stable C ABI @rchowell (#6301)
feat(lance): refine distributed scalar index support with new types and parameters @huleilei (#5817)
feat!: add ddof parameter to stddev (default ddof=1) @aaron-ang (#6142)
feat: use human-readable query names instead of UUIDs @samstokes (#6310)
feat(metrics): add node phase attribute for multi-phase pipeline nodes @cckellogg (#6290)
feat(dashboard): add wall-clock and CPU duration to operator cards @samstokes (#6300)
feat: Add Python Function data source info in explain @plotor (#6119)
feat(dashboard): physical plan tree with live operator stats @samstokes (#6299)
feat(io): implement write_sql with SQLDataSink and explicit dtype support @huleilei (#5979)
feat(dashboard): add interactive tree visualizer for query plans @samstokes (#6295)

🐛 Bug Fixes

fix(windows): pin transitive aws deps to fix windows linking bug @universalmind303 (#6404)
fix(video): add missing pillow dependency for video keyframes @desmondcheongzx (#6355)
fix(dashboard): request notification permission before enabling notifications @samstokes (#6362)
fix(dashboard): make dashboard subscriber non blocking @cckellogg (#6336)
fix(dashboard): ensure query_end fires when generator is abandoned early @samstokes (#6326)
fix(tests): make Ray conda UDF tests robust to unavailable Python micro versions @cckellogg (#6341)
fix(sql): reject duplicate CTE names in WITH clause @singularityDLW (#6331)
fix: Use Tokio FS for Flight Server Reads @srilman (#6308)
fix(io): add connection pool semaphore to Azure Blob Storage backend @singularityDLW (#6305)
fix(dashboard): sort extra stats alphabetically to prevent flicker @samstokes (#6309)
fix: Support Union and Optional in DataType.infer_from_type @srilman (#6304)
fix: make max_concurrency control coroutine concurrency for async UDFs @colin-ho (#6302)
fix: Fix native executor closing @colin-ho (#6282)
fix(docs): Remove duplicate Voice AI analytics listing in SUMMARY.md @copilot-swe-agent[bot] (#6274)

🚀 Performance

perf: Support multi-task parallel glob scan for from_glob_path @plotor (#5897)
perf: Drop tasks early in distributed limits @srilman (#6350)
perf(image): make image ops run in parallel via rayon @universalmind303 (#6333)
perf(upload): use a zero copy approach for url_upload @universalmind303 (#6334)
perf(parquet): late materialization via RowFilter and RowSelection @desmondcheongzx (#6311)
perf!: lazy-load lance to speed up import daft @desmondcheongzx (#6328)

♻️ Refactor

refactor: decouple extension ABI from arrow-rs with Arrow C Data Interfaces @rchowell (#6337)
refactor(schema): change Field.name from String to Arc<str> @desmondcheongzx (#6351)
refactor(deps): remove 11 dependencies by inlining builder dependencies @universalmind303 (#6364)
refactor(arrow2): goodbye arrow2 @universalmind303 (#6363)
refactor(arrow2): migrate remaining parquet code to arrow-rs @universalmind303 (#6361)
refactor(arrow2): remove parquet2 from daft @desmondcheongzx (#6339)
refactor(arrow2): remove parquet2 read pipeline, make arrow-rs the default reader @desmondcheongzx (#6335)
refactor(arrow2): add arrow-rs parquet reader @desmondcheongzx (#6294)
refactor(arrow2): daft-csv crate @universalmind303 (#6298)
refactor(arrow2): daft-functions-serde @universalmind303 (#6303)
refactor(arrow2): introduce DaftParquetMetadata adapter to decouple crates from parquet2 @desmondcheongzx (#6292)
refactor(arrow2): refactor daft-json to arrow-rs @universalmind303 (#6297)
refactor(arrow2): move the auto conversion logic into array init @universalmind303 (#6289)

📖 Documentation

docs(optimization): add join strategies guide @desmondcheongzx (#6367)
docs: add custom catalog implementation guide @desmondcheongzx (#6366)
docs(skills): add Claude Code compatible guides and update README @Jay-ju (#6116)

🔧 Maintenance

chore: removes unnecessary pushdown builder code @rchowell (#6403)
chore: simplify data source implementation @rchowell (#6397)
chore: add claude/worktrees to gitignore @universalmind303 (#6402)
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 @singularityDLW (#6378)
chore: Enable write_json related test cases for Ray Runner @plotor (#6387)
chore: Upgrade PyO3 to 0.28 @srilman (#6383)
chore: update Slack invite links across docs and README @everettVT (#6377)
chore: Replace RepartitionSpec::IntoPartitions with custom path @srilman (#6349)
chore: Improve Common Crawl Benchmark @srilman (#6307)
chore(tests): migrate internal usages of daft.udf to cls/func @universalmind303 (#6348)
chore(dashboard): add README with build and architecture notes @samstokes (#6323)
chore(lints): add workspace lints inheritance to missing subcrates @yew1eb (#6330)
chore: Fix some typo issues in the code or docs @plotor (#6320)
chore(deps): bump the all group with 4 updates @dependabot[bot] (#6314)
chore: refactor pipeline node ownership in start @srilman (#6259)
chore: Move EXPLAIN ANALYZE to Python @srilman (#6254)

⬆️ Dependencies

chore(deps): bump the all group with 4 updates @dependabot[bot] (#6314)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.4...v0.7.5

Open source →

0.7.426 Feb 2026Deprecated

refactor(arrow2): misc deprecation warnings @universalmind303

What's Changed 🚀

💥 Breaking Changes

refactor(arrow2)!: remaining arrow2 from daft-core @universalmind303 (#6284)
refactor(arrow2)!: use arrow-rs casting for strftime function @universalmind303 (#6263)
refactor(arrow2)!: migrate interval arithmetic to arrow-rs @rohitkulshreshtha (#6186)

✨ Features

feat(ai): Add HTTP URL direct passthrough for images and videos in prompt function @huleilei (#6182)
feat(sql): add DATE_TRUNC function support @desmondcheongzx (#6258)
feat(swordfish): Streaming sources @colin-ho (#5978)
feat(metrics): add metrics docs @cckellogg (#6253)
feat(io/av): enhance time-interval sampling with comprehensive tests and improved @huleilei (#6088)
feat: Add Tencent Cloud COS (Cloud Object Storage) support @XuQianJin-Stars (#6140)
feat(metrics): consolidate naming and add node.type attribute @cckellogg (#6236)
feat: add Flight shuffle to Flotilla @srilman (#6123)
feat: Apache OpenDAL™ compatible backends @universalmind303 (#6177)
feat(observability): Split duration into separate column in metrics DF @srilman (#6235)
feat: add support for pyiceberg 0.11.0 @gweaverbiodev (#6200)
feat: add support for SQL ORDER BY column position @Lucas61000 (#6211)
feat: Supports running dashboard in daemon mode @plotor (#5993)
feat: json_write support for time stamps @gpathak128 (#6214)
feat: .as_T cast methods @aaron-ang (#6100)

🐛 Bug Fixes

fix: handle case where join keys are different for sort-merge multi-partition join @gweaverbiodev (#6243)
fix(arrow2): reinterpret physical array as logical type after cast fallback @desmondcheongzx (#6291)
fix(sql): resolve GROUP BY ambiguous column names for derived expressions @desmondcheongzx (#6286)
fix(flight): Add a check for flight_shuffle_dirs arg and change default @srilman (#6266)
fix: Map Literal <-> Python Dict conversion @w2ais (#6084)
fix: NaN-aware comparator for multi-column search_sorted and sort @desmondcheongzx (#6242)
fix: add ignore_empty_and_null parameter to .explode() @singularityDLW (#6047)
fix: Broadcast literal expressions in aggregations to match input length @desmondcheongzx (#6155)
fix: Cleanup imports from dashboard daemon PR @srilman (#6222)
fix: Fix compilation on main @srilman (#6221)
fix: canonicalize negative NaN in multi-column sort comparator @ykdojo (#6215)
fix: delta version parsing @aaron-ang (#6156)
fix: Register extension type before reading from Lance @ykdojo (#6058)
fix: use union instead of append_column in window agg to fix schema mismatch @ykdojo (#6178)
fix: into_batches should not allow downstream shuffle elision @desmondcheongzx (#6170)

♻️ Refactor

refactor(arrow2): refactor index_bitmap and time_unit @universalmind303 (#6287)
refactor(arrow2)!: remaining arrow2 from daft-core @universalmind303 (#6284)
refactor(arrow2): remove series::try_from(name, arrow2_arr) @universalmind303 (#6283)
refactor(arrow2): switch to arrow-rs backed arrays @universalmind303 (#6280)
refactor(arrow2): misc deprecation warnings @universalmind303 (#6270)
refactor(arrow2): remove from_arrow2 @universalmind303 (#6267)
refactor(arrow2): fully remove toarrow2 from arrays & series @universalmind303 (#6265)
refactor(arrow2)!: use arrow-rs casting for strftime function @universalmind303 (#6263)
refactor(arrow2): use arrow-rs for filtering on python arrays @universalmind303 (#6262)
refactor(arrow2): migrate cast.rs from arrow2 to arrow-rs @desmondcheongzx (#6239)
refactor(arrow2): migrate BooleanArray bitmap access to arrow-rs @desmondcheongzx (#6256)
refactor(arrow2): migrate concat.rs to arrow-rs @desmondcheongzx (#6255)
refactor(arrow2): remove misc arrow2 references in csv and json @universalmind303 (#6248)
refactor(arrow2): remove more arrow2 usages in daft-core @universalmind303 (#6249)
refactor(arrow2): remove arrow2 from daft-recordbatch @universalmind303 (#6231)
refactor(arrow2): replace arrow2 based buffer with custom impl @universalmind303 (#6247)
refactor(arrow2)!: migrate interval arithmetic to arrow-rs @rohitkulshreshtha (#6186)
refactor(arrow2): remove daft_arrow from arrow_growable @universalmind303 (#6251)
refactor(arrow2): migrate len.rs to arrow-rs @rohitkulshreshtha (#6171)
refactor(arrow2): migrate image.rs to arrow-rs @rohitkulshreshtha (#6191)
refactor(arrow2): migrate dyn_compare and probeable to arrow-rs @desmondcheongzx (#6227)
refactor(arrow2): migrate if_else kernel from arrow2 to arrow-rs @desmondcheongzx (#6240)
refactor(arrow2): migrate array serdes from arrow2 to arrow-rs @desmondcheongzx (#6238)
refactor(arrow2): migrate growable internals from arrow2 to arrow-rs @desmondcheongzx (#6228)
refactor(arrow-rs): Remove basic growable usages @srilman (#5779)
refactor(arrow2): misc arrow2 cleanups in daft-core @universalmind303 (#6232)
refactor(arrow2): remove daft_arrow from daft-functions-utf8 @universalmind303 (#6229)
refactor(arrow2): fully remove daft-arrow from functions-list @universalmind303 (#6230)
refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs @universalmind303 (#6166)
refactor(arrow2): replace arrow2 iterators with custom daft-native iterators @universalmind303 (#6220)
refactor(arrow2): list kernels @universalmind303 (#6219)
refactor(arrow2): migrate groups to arrow-rs @cckellogg (#6185)
refactor(arrow2): second attempt at offsets @universalmind303 (#6162)
refactor(arrow2): update image_array to not use arrow2 @universalmind303 (#6217)
refactor(arrow2): Migrate arithmetic kernels @desmondcheongzx (#6193)
refactor(arrow2): Migrate concat agg @desmondcheongzx (#6190)

📖 Documentation

docs: Fix broken links to the url modality and various datatypes @desmondcheongzx (#6212)

✅ Tests

test(benchmarking): add some read_json python benchmarks @universalmind303 (#6288)
test(parquet): add benchmarks for nested types, codecs, and filter pushdown @desmondcheongzx (#6285)
test: Filter null bytes from generated column names in property-based tests @desmondcheongzx (#6213)
test(postmerge): Minor fix to OpenAI integration tests @desmondcheongzx (#6209)
test: Fix incomplete metrics migration in OpenAI integration tests @desmondcheongzx (#6204)

👷 CI

ci: include dashboard assets in maturin sdist for manylinux builds @desmondcheongzx (#6281)
ci: fix restore-mtime exit code when last file is deleted on macos runners @desmondcheongzx (#6268)
ci: cache workspace crates and share rust caches across all workflows @desmondcheongzx (#6264)
ci: cache workspace crate artifacts in integration build @desmondcheongzx (#6261)
ci: share rust cache across branches and restore file mtimes @desmondcheongzx (#6246)
ci: bump all timeouts from 30 -> 45 @universalmind303 (#6250)
ci: increase integration-test-build timeout from 45 to 90 minutes @desmondcheongzx (#6244)
ci: increase integration-test-build timeout from 30 to 45 minutes @desmondcheongzx (#6241)
ci: increase integration-test-sql timeout from 30 to 45 minutes @desmondcheongzx (#6237)
ci: Reduce xdist workers on macOS to fix Ray actor timeout @desmondcheongzx (#6210)
ci: make repository secrets optional for CI workflows @jeevb (#6199)
ci: Improve CI reliability via disk space reclamation and coverage optimization @desmondcheongzx (#6205)

🔧 Maintenance

chore: Update codeowners @colin-ho (#6183)
chore: Adding Issue requirement for PRs, Updating maintainers @madvart (#6196)
chore(observability): Split dashboard cli into separate start / stop subcommands @srilman (#6234)
chore(deps): Resolve Dependabot security alerts @desmondcheongzx (#6226)
chore: address review feedback from #6208 @desmondcheongzx (#6225)
chore(deps): Bump dependency group with conflict resolution @desmondcheongzx (#6208)
chore: Replace Bun with Node / NPM @srilman (#6202)
chore(arrow2): simplify deprecation markers @universalmind303 (#6218)
chore: add .values method for utf8Array @universalmind303 (#6216)
chore: add mypy-boto3-glue to aws optional dependencies @Killua7163 (#6080)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.3...v0.7.4

Open source →

0.7.313 Feb 2026

feat(observability): Support exporting Flotilla metrics @srilman

What's Changed 🚀

✨ Features

feat(observability): Support exporting Flotilla metrics @srilman (#6122)
feat: Nightly installations under nightly.daft.ai @desmondcheongzx (#6175)
feat: Support additional OTEL_* configuration envs @srilman (#6148)
feat(lance): Support lance namespace read and write @shaofengshi (#5980)
feat: add unity oauth m2m access token support @cckellogg (#5839)
feat: Support snapshot properties for Iceberg writes @desmondcheongzx (#6139)
feat: delimiter for agg_concat @aaron-ang (#6099)
feat: add uuid function @everySympathy (#5983)
feat: add custom date and timestamp formatting for CSV writes @madvart (#6073)
feat(observability): Export metrics out as a table with the result @srilman (#6055)
feat: comparison ops for list and struct types @aaron-ang (#6104)
feat(frontend): enhance dashboard UI and fix Ray runner state reporting @Jay-ju (#6063)
feat(dashboard): backend implementation @Jay-ju (#6062)
feat: string casing functions @aaron-ang (#6096)
feat: list_contains expression @aaron-ang (#6095)
feat: specify text embedding dim @aaron-ang (#6097)
feat: Expr.var with ddof @aaron-ang (#6105)
feat(lance): add nearest vector search support @huleilei (#6025)
feat: distance and similarity functions @aaron-ang (#6098)
feat: add gravitino connector optional dependency @shaofengshi (#6083)
feat: Add support for ignoring null fields when writing json @gpathak128 (#6049)

🐛 Bug Fixes

fix(docs): Fix broken links in modalities documentation @everettVT (#6197)
fix: Render Map columns as python dicts instead of list[struct] in terminal prints @srilman (#6198)
fix: Fix merge conflict with OTEL configuration PR @srilman (#6187)
fix(observability): Truncate progress bar names by characters, not bytes @desmondcheongzx (#6180)
fix: fix concat.rs @colin-ho (#6174)
fix(test): bump actor UDF timeout from 10s to 60s to reduce flakiness @ykdojo (#6163)
fix(optimizer): Fix bug with pushing filters through anti-joinsI s @desmondcheongzx (#6150)
fix: Allow is_in to accept sets, tuples, and other iterables @desmondcheongzx (#6115)
fix(ci): restore Rust code coverage by pinning cargo-llvm-cov @desmondcheongzx (#6146)
fix: make df.into_partitions() work when input num == num_partitions @everySympathy (#6061)
fix(udf): ensure per-call kwargs in udf v2 are uniquely bound per call site @huleilei (#6079)
fix(rustfmt): ignore parquet directories @aaron-ang (#6101)
fix: No need extra 1 for OffsetBufferBuilder @colin-ho (#6057)
fix: use unique bucket names for running tests in parallel @rchowell (#6052)

🚀 Performance

perf: zero copy on from_vec @universalmind303 (#6172)
perf: Only Serialize Required Cols in Actor UDFs @plotor (#5884)

♻️ Refactor

refactor(arrow2): migrate time.rs temporal methods to arrow-rs @rohitkulshreshtha (#6160)
refactor(arrow2): migrate null checks to arrow-rs @rohitkulshreshtha (#6152)
refactor(arrow2): migrates sparse-tensor to arrow-rs @cckellogg (#6179)
refactor(swordfish): Separate concat and make streaming sink single input @colin-ho (#6059)
refactor(arrow2): migrates hll-sketch to arrow-rs @cckellogg (#6169)
refactor(arrow2): use arrow-rs for python conversions @universalmind303 (#6130)
refactor: Migrate hll_merge.rs from arrow2 to arrow-rs @rohitkulshreshtha (#6158)
refactor(arrow2): remove arrow2 from minhash @cckellogg (#6149)
refactor: Migrate product.rs from arrow2 to arrow-rs @rohitkulshreshtha (#6159)
refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs @rchowell (#6056)
refactor(arrow2): migrate the extra easy kernels to arrow-rs @universalmind303 (#6145)
refactor(arrow2): migrate BinaryArray iceberg_truncate to arrow-rs @rohitkulshreshtha (#6157)
refactor(arrow2): migrate DataArray full_null/empty to arrow-rs @rohitkulshreshtha (#6151)
refactor(arrow2): remove a bunch of from impls that used arrow2 @universalmind303 (#6137)
refactor(scalarudf): followup to uuid pr @universalmind303 (#6129)
refactor(arrow-rs): Remove arrow2 from the search_sorted kernel @desmondcheongzx (#6034)
refactor(arrow-rs): Migrate approx count distinct @desmondcheongzx (#6038)
refactor(arrow-rs): migrate is_in and get_lit ops to Arrow-rs arrays @huleilei (#6085)
refactor: Migrate utf8 left function to arrow-rs @huleilei (#6004)
refactor(arrow2): migrate daft-functions-utf8/split to arrow-rs @colin-ho (#6046)
refactor(arrow-rs): Migrate sketch_percentile kernel @desmondcheongzx (#6044)
refactor(swordfish): Separate joins @colin-ho (#6042)
refactor(arrow2): remove some arrow2 based from impls @universalmind303 (#6054)
refactor(arrow2): migrate sum, min, max agg kernels @kevinzwang (#6045)
refactor(arrow2): utf8 comparison kernels @kevinzwang (#6032)
refactor(flotilla): Use stream::iter instead of channel in flotilla source nodes @colin-ho (#6043)
refactor(arrow2): remove arrow2 *_array() methods from ImageArray @universalmind303 (#6050)
refactor(arrow-rs): Migrate binary from_iter methods @desmondcheongzx (#6040)
refactor(observability): Refactor StatSnapshot to be predefined structs @srilman (#6033)

📖 Documentation

docs: Add daft.File usage throughout modalities @everettVT (#6074)
docs: Governance proposed changes @jaychia (#6117)
docs: Clarify imagenet benchmark setup @desmondcheongzx (#6147)
docs: add TosConfig documentation reference in config.md @huleilei (#6068)
docs: Update daft.File API and add docstrings, improve error handling in tests @everettVT (#5877)
docs: add mm structured outputs tutorial @everettVT (#5816)
docs: Add end-to-end image pipeline example and regression test @huleilei (#6006)

✅ Tests

test: Add missing result parameter to on_query_end in Google AI test @desmondcheongzx (#6184)

👷 CI

ci: Exclude common-arrow-ffi from rust tests @desmondcheongzx (#6173)
ci: Accept pandas StringDtype in schema override tests @desmondcheongzx (#6165)
ci: Add pyarrow to wheel build test dependencies @desmondcheongzx (#6109)
ci: Add pytz and numpy to wheel build test dependencies @desmondcheongzx (#6108)
ci: Fix nightly workflow permissions @desmondcheongzx (#6037)

🔧 Maintenance

chore: Intermediate op single input @colin-ho (#6189)
chore: Revert "refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs" @universalmind303 (#6164)
chore: Remove ci/ folder @srilman (#6154)
chore: agg_concat kernel nits @colin-ho (#6136)
chore: Remove vendored parquet-format-safe, use upstream patched version @desmondcheongzx (#6118)
chore(deps): bump the all group with 5 updates @dependabot[bot] (#5908)
chore: Allows make clean to skip cleaning the python virtual environment @plotor (#6103)
chore: correct comment of enable_scan_task_split_and_merge in set_execution_config @everySympathy (#6077)
chore: Update lru dependency @desmondcheongzx (#6041)

⬆️ Dependencies

chore(deps): bump the all group with 5 updates @dependabot[bot] (#5908)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.2...v0.7.3

Open source →

0.7.216 Jan 2026Deprecated

refactor(arrow2): add more deprecation markers @universalmind303

What's Changed 🚀

✨ Features

feat: Add name and path properties to daft.File @everettVT (#6024)
feat(mcap): support topic_start_time_resolver and raw-bytes non-seekable reader @Jay-ju (#5886)
feat: Add configurable token limits to OpenAI text embedder @kyo-tom (#6017)
feat: Add guess_mime_type scalar expression for MIME type detection from bytes @copilot-swe-agent[bot] (#5883)
feat: Channel-less intermediate op @colin-ho (#5999)
feat: support dropping dimensions in /v1/embeddings requests @rchowell (#5988)
feat: Add Apache Gravitino virtual file system (gvfs://) write support in io module @shaofengshi (#5965)
feat: async embed image @colin-ho (#5833)
feat: support delimiter, quota, header options for csv writer @stayrascal (#5794)
feat(agg): support map_groups with v2 udf @Jay-ju (#5927)
feat: index_col option in explode @aaron-ang (#5842)
feat: support resumable stream @stayrascal (#5824)
feat(functions): add shell_op for distributed shell execution @huleilei (#5738)
feat: Add Apache Gravitino virtual file system (gvfs://) read support in io module @shaofengshi (#5766)
feat: Support configuring Conda Env for Class UDFs in Flotilla @plotor (#5117)
feat: image to tensor @aaron-ang (#5847)
feat: support native writer via tos @stayrascal (#5760)

🐛 Bug Fixes

fix(observability): Clean up progress bar naming @srilman (#6028)
fix(video): correct keyframe seek timestamp calculation for start_time @huleilei (#6005)
fix: ".*" not handled correctly in SQL planner @Lucas61000 (#5784)
fix: Optimize the small files issue of sink lance @caican00 (#5844)
fix: overriding dimensions for openai embedding models @kevinzwang (#6013)
fix: support externally-hosted models via OpenAI-compatible API when using embed_text func @caican00 (#5873)
fix: respect model dtype when overriding embedding dimensions @fenfeng9 (#5899)
fix: Pass csv option into native writer @colin-ho (#6003)
fix: Nonzero morsel upper bound @colin-ho (#5989)
fix: Clean up UDF display name in progress bar and plans @srilman (#5810)
fix(cast): handle whitespace in string-to-number casting @ykdojo (#5955)
fix: Incorrect buffer pool calculation strategy when reading CSV @plotor (#5857)
fix: Optimize the display information of Join nodes in query plan @plotor (#5617)
fix: Fast failure when dashboard is enabled in Ray Runner @plotor (#5867)
fix(ai): resolve intermittent meta tensor error in classify_text/classify_image @rohitkulshreshtha (#5977)
fix(ci): cargo machete error that slipped through ci somehow @universalmind303 (#5975)
fix: Daft.ai link checker to ignore X @everettVT (#5879)
fix: Supporting fractional gpu count on class udf @caican00 (#5840)
fix: remove flaky datasets from read_huggingface tests @everettVT (#5926)
fix: allows appending nulls to lists, null is compatible with all types @rchowell (#5921)
fix(test): read all splits in HuggingFace integration tests @ykdojo (#5878)
fix(iceberg): Correct test setup to ensure delete files are created @huleilei (#5864)
fix(ray): namespace flotilla actor per job to avoid plan id collisions @huleilei (#5855)

🚀 Performance

perf: dont eval empty recordbatches @universalmind303 (#5968)

♻️ Refactor

refactor(swordfish): Channel-less blocking sink @colin-ho (#6023)
refactor(arrow2): rename validity to nulls to align with arrow-rs @universalmind303 (#6027)
refactor(arrow-rs): remove usages of build_is_equal and replace with … @universalmind303 (#6018)
refactor(arrow2): arrow based take kernels @universalmind303 (#6022)
refactor(swordfish): Channel-less streaming sink @colin-ho (#6021)
refactor(arrow2): remove makegrowable from fsl array @universalmind303 (#6019)
refactor(flotilla): Swordfish task builder @colin-ho (#5976)
refactor(arrow-rs): remove makegrowable from concat_agg.rs @universalmind303 (#6020)
refactor(arrow2): refactor build_probe_table_without_nulls @universalmind303 (#6011)
refactor(arrow2): rest of comparison.rs @kevinzwang (#6000)
refactor: abstract TosRetrier to retry all tos operation @stayrascal (#5858)
refactor(arrow2): functions-utf8 utils @universalmind303 (#5996)
refactor(arrow2): parquet/read @universalmind303 (#5997)
refactor(arrow2): remove some arrow2 based from impls @universalmind303 (#5995)
refactor(arrow2): migrate apply and binaryapply functions @universalmind303 (#5994)
refactor(arrow2): DaftCompare between two DataArrays @kevinzwang (#5964)
refactor(arrow-rs): Remove arrow2 from daft-writers @srilman (#5985)
refactor(arrow2): add new from_iter_values and arange impl @universalmind303 (#5984)
refactor(arrow-rs): Remove arrow2 from daft-scan and related @srilman (#5974)
refactor(arrow-rs): Remove arrow2 from sort kernels @desmondcheongzx (#5963)
refactor(arrow-rs): Upgrade arrow-rs to 57.1.0 @srilman (#5969)
refactor(arrow2): migrate float.rs to arrow-rs @rohitkulshreshtha (#5953)
refactor(arrow2): use arrow for file arrays @universalmind303 (#5972)
refactor(arrow2): count and array/mod @universalmind303 (#5973)
refactor(arrow-rs): Remove arrow2 from daft-sketch @srilman (#5967)
refactor(arrow2): migrate repeat.rs to arrow-rs @universalmind303 (#5928)
refactor(arrow2): migrate left.rs to arrow-rs @universalmind303 (#5930)
refactor(arrow2): migrate daft-image/src/ops.rs to arrow-rs @universalmind303 (#5940)
refactor(arrow2): migrate daft-image/series.rs to arrow-rs @universalmind303 (#5939)
refactor(arrow2): migrate endswith.rs to arrow-rs @universalmind303 (#5933)
refactor(arrow2): migrate replace.rs to arrow-rs @universalmind303 (#5932)
refactor(arrow2): migrate find.rs to arrow-rs @universalmind303 (#5929)
refactor(arrow2): recordbatch ops @kevinzwang (#5962)
refactor(arrow-rs): Remove arrow2 from WARC reader @srilman (#5948)
refactor(arrow2): bool_agg @kevinzwang (#5959)
refactor(arrow2): migrate functions-utf8 repeat, replace, to_datetime @cckellogg (#5961)
refactor(arrow2): migrate functions-utf8 to_date.rs @cckellogg (#5960)
refactor(arrow-rs): Remove arrow2 use in jq kernel @colin-ho (#5957)
refactor(arrow2): migrate functions-utf8 substr.rs @cckellogg (#5958)
refactor(arrow-rs): remove arrow2 from PartitionedWriter @cckellogg (#5951)
refactor(arrow2): binary kernels @kevinzwang (#5956)
refactor(arrow-rs): use arrow-rs for delete map @colin-ho (#5950)
refactor(arrow2): Remove arrow2 from utf8 array ops @desmondcheongzx (#5954)
refactor(arrow2): Migrate utf8.right to use arrow-rs instead of arrow2 @huleilei (#5889)
refactor(arrow-rs): Move IPC conversion from arrow2 to arrow-rs @srilman (#5805)
refactor(arrow2): migrate streaming_sink/vllm @universalmind303 (#5922)
refactor(arrow2): add more deprecation markers @universalmind303 (#5917)
refactor(arrow2): array & series to/from arrow @universalmind303 (#5848)

📖 Documentation

docs: new section for openai compatible providers @everettVT (#5748)
docs: add tos config and a specific schema example in write lance @huleilei (#5992)

✅ Tests

test: Parametrize limit offset tests @colin-ho (#5971)
test: Don't skip all pyarrow 8.0.0 tests @colin-ho (#5944)
test: strip whitespace from process output for run_process test @kevinzwang (#5942)

🔧 Maintenance

chore(observability): Refactor progress bar to remove RuntimeStatsSubscriber @srilman (#6030)
chore: Use tokio channel for swordfish channels @colin-ho (#6035)
chore: optimizes slow tests in CI/CD @rchowell (#6029)
chore: Add workflow permissions @desmondcheongzx (#6014)
chore: Upgrade nextjs @desmondcheongzx (#6015)
chore: Update dependencies for dependabot alerts @desmondcheongzx (#6002)
chore: uv lock check @colin-ho (#6010)
chore: add some additional iterator methods for daft arrays @universalmind303 (#5937)
chore: add deprecation warnings on other arrow2 arrays @kevinzwang (#5946)
chore: ignore all markdown files inside .claude @universalmind303 (#5943)
chore: ignore all prompting agents inside .claude dir @universalmind303 (#5935)
chore: consolidates arrow-* crates as workspace dependencies @rchowell (#5923)
chore: bump mypy and ruff in pre-commit @aaron-ang (#5836)
chore: Add basic usage instructions for daft-dashboard in development guide @plotor (#5865)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.1...v0.7.2

Open source →

0.7.119 Dec 2025

feat: Support pattern filtering for SHOW TABLES @aaron-ang

What's Changed 🚀

✨ Features

feat: Support pattern filtering for SHOW TABLES @aaron-ang (#5423)
feat(docs): add copy page as markdown button @ykdojo (#5828)
feat: support overwrite files by native IO @stayrascal (#5728)
feat: Add support for Series[start:end] @everySympathy (#5815)
feat: Implement retry-after mechanism for model apis and udfs @colin-ho (#5769)

🐛 Bug Fixes

fix: using estimate memory bytes at first for display scan task source @stayrascal (#5845)
fix: Remove pop_all assertion @colin-ho (#5850)
fix: Check if deletion vector propagation is supported in deltalake @cckellogg (#5829)
fix: Set default ImageMode in decode_image to RGB @colin-ho (#5827)
fix: handle FileNotFoundError in read_huggingface fallback @ykdojo (#5831)
fix: Add overflow protection to memory estimation @yudduy (#5417)

♻️ Refactor

refactor(arrow2): get field and dtype roundtrips working @universalmind303 (#5849)

📖 Documentation

docs: improve docstrings of IO read methods for remote URLs @aaron-ang (#5841)
docs: fix broken links causing CI failure @ykdojo (#5832)

👷 CI

ci: enable Windows Rust tests on PRs @ykdojo (#5823)
ci: skip quickstart notebook in notebook-checker workflow @ykdojo (#5804)

🔧 Maintenance

chore: remove overwrite_files & write_empty_tabular method @stayrascal (#5838)
chore: Fix a minor ambiguity in the README docs @plotor (#5830)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.0...v0.7.1

Open source →

0.7.016 Dec 2025Security

fix: patch CVE-2025-66478 update next dependencies to 16.0.7 @everettVT

What's Changed 🚀

💥 Breaking Changes

chore!: remove spark connect @universalmind303 (#5743)

✨ Features

feat(lance): row-level schema evolution support @Jay-ju (#5749)
feat: Improve model api typing @colin-ho (#5809)
feat(deltalake): allow users to ignore deletion vectors on read @kevinzwang (#5758)
feat: Capture UDF argument names and values in structured logging. @rohitkulshreshtha (#5771)
feat: support split and merge jsonl/ndjson files @caican00 (#5695)
feat(tools): add markdown-to-notebook converter for documentation @ykdojo (#5691)
feat: support label_selector specification in ray actor/task creation @Jay-ju (#5042)
feat: support native csv writer @stayrascal (#5706)
feat: Update optional dependencies for daft[postgres] @desmondcheongzx (#5586)
feat(lance): add distributed compaction for Lance @huleilei (#5699)
feat: Use JSON Serialization for Plans in Subscribers @srilman (#5709)
feat: extended hash function to take and hash multiple inputs @rahulkodali (#5692)
feat: Add Apache Gravitino catalog in catalog module @shaofengshi (#5694)
feat: Better errors when lazy imports fail @samstokes (#5753)
feat: Added structured logging for UDF errors. @rohitkulshreshtha (#5688)
feat: Add statistics info to ScanTask when reading Lance dataset @plotor (#5727)
feat: dynamic batching per operator @universalmind303 (#5676)
feat: Allow users to disable the suffix range request @TheR1sing3un (#5188)
feat: Streaming sample by size @colin-ho (#5663)
feat: Allow dashboard to show query canceled/failed/dead information when query exited abnormally @VOID001 (#5576)
feat: Add Google AI provider with prompt @everettVT (#5640)
feat: Add pow expression @kliwongan (#5237)
feat: No truncate in .collect preview @colin-ho (#5632)
feat: sample api supports precise sampling by size params @caican00 (#5600)
feat: audio file subtype @universalmind303 (#5602)
feat(tos): enhance the retry logic to aware response @stayrascal (#5569)
feat: emit selectivity metric to OTel in swordfish filter op @samstokes (#5584)
feat: Adding otel logger collector for collecting UDF errors. @rohitkulshreshtha (#5624)

🐛 Bug Fixes

fix: handle Windows paths and query params in local_path_from_uri @ykdojo (#5819)
fix: use pytest.importorskip for lance in test_limit_offset @ykdojo (#5818)
fix: support skip empty json/jsonl files @caican00 (#5660)
fix: minor doc fix @yuchaoran2011 (#5814)
fix: CountRows with Limit returns unexpected result when reading Lance dataset @plotor (#5550)
fix: Check for missing dependencies in OpenAI provider @everettVT (#5747)
fix: fix btree index invalid issue when reading lance for point lookup @caican00 (#5673)
fix: Combine deltalake with unity extra @everettVT (#5785)
fix: enhance unit tests @caican00 (#5787)
fix: patch CVE-2025-66478 update next dependencies to 16.0.7 @everettVT (#5786)
fix: use single consolidated progress bar in Jupyter notebooks @ykdojo (#5774)
fix: CuPy → NumPy needs explicit conversion @Jay-ju (#5680)
fix: Fix Pydantic cloudpickle serialization in Google Colab @ykdojo (#5705)
fix: update AI integration tests for new Subscriber interface @ykdojo (#5763)
fix(optimizer): Prevent limits from being pushed below explodes in non-top-level projections @desmondcheongzx (#5292)
fix(io): load all splits in read_huggingface fallback path @ykdojo (#5757)
fix(test): use read_huggingface instead of read_parquet for HF test @ykdojo (#5755)
fix: add disk cleanup to nightly integration-test-io job @ykdojo (#5711)
fix: Postgres overwrite table should enable RLS and set up pgvector automatically @desmondcheongzx (#5657)
fix: make it easier to enable different logging levels @Abyss-lord (#5661)
fix: Dashboard logo animation. @j3nkii (#5672)
fix(ci): add disk cleanup to integration-test-ai job @ykdojo (#5733)
fix: update hypothesis test to use new expression API @ykdojo (#5723)
fix: Fix type annotation check on Python 3.14 @srilman (#5721)
fix: add fallback mechanism for HuggingFace datasets without parquet files @ykdojo (#5650)
fix: Import or skip lance @colin-ho (#5662)
fix: Add missing trailing slashes to S3-compatible endpoint urls @desmondcheongzx (#5575)
fix: Add outer try-finally block in executor generator @colin-ho (#5633)
fix: test_explain @universalmind303 (#5656)
fix: Unify the naming and type of URI parameter for Lance-related APIs @plotor (#5634)
fix: Fix blocked and oom issues for scan lance @caican00 (#5592)
fix: Executing explain will panic when ScanTask is empty @plotor (#5582)
fix: Embed text dropping texts @colin-ho (#5641)
fix: limit(n) return n rows directly @caican00 (#5597)
fix: Upgrade to deltalake 1.2.1 @colin-ho (#5580)
fix: add disk cleanup to integration-test-io-credentialed job @ykdojo (#5610)
fix: add disk cleanup to doctests job @ykdojo (#5609)
fix: Hashable identifier @colin-ho (#5598)

🚀 Performance

perf: Lazy udf worker @colin-ho (#5542)
perf: Use growable for build side @colin-ho (#5613)
perf: optimize setting lance schema @Jay-ju (#5704)

♻️ Refactor

refactor(arrow2): values_iter removals for primitive array @universalmind303 (#5802)
refactor(arrow2): remove arrow2 from daft-functions-binary @universalmind303 (#5799)
refactor(arrow2): remove deprecated usages from daft-functions-utf8 @universalmind303 (#5800)
refactor(arrow2): remove deprecated methods from daft-functions-uri crate @universalmind303 (#5798)
refactor(arrow2): remove arrow2 from daft-functions-tokenize @universalmind303 (#5797)
refactor(arrow2): rename and deprecate to_arrow and as_arrow functions @universalmind303 (#5796)
refactor: write empty dataframe to parquet/json files via native IO @stayrascal (#5682)
refactor(arrow-rs): Move temporal conversions from arrow2 to arrow-rs @srilman (#5782)
refactor(arrow-rs): Remove arrow2 Index generics usages @srilman (#5761)
refactor(arrow2): rename and deprecate .to_arrow methods @universalmind303 (#5789)
refactor(arrow): use arrow for ffi instead of arrow2 @universalmind303 (#5775)
refactor(arrow): remove daft-arrow from daft-sql & daft-context crates @universalmind303 (#5773)
refactor(arrow-rs): Move all validity daft_arrow::bitmap::Bitmaps to daft_arrow::buffer::NullBuffer @srilman (#5750)
refactor: abstract MultipartWriter to write data to object store @stayrascal (#5702)
refactor: Remove Unloaded MicroPartitions @srilman (#5710)
refactor(arrow-rs): Add daft-arrow middleman crate for Rust & Arrow usage @srilman (#5730)

📖 Documentation

docs: Update slack invite @everettVT (#5813)
docs: add logging settings @Jay-ju (#5671)
docs: fix broken Bodo benchmark link @ykdojo (#5762)
docs: add voice-analytics-example and update index @everettVT (#5737)
docs: fix broken Lance documentation link @ykdojo (#5724)
docs: remove redundant About Daft section from README @ykdojo (#5689)
docs: remove redundant Table of Contents from README @ykdojo (#5684)
docs: add Daft Cloud mentions to distributed execution docs @ykdojo (#5686)
docs: fix quickstart connector links formatting @ykdojo (#5687)
docs: update README to reflect AI/multimodal positioning @ykdojo (#5677)
docs: Improve mkdocstrings template for Python examples rendering @ykdojo (#5642)
docs: changed dev url to a live link to prevent 404 @j3nkii (#5669)
docs: add Python version requirement to README @ykdojo (#5655)
docs: update index overview page @ykdojo (#5627)
docs: remove Python tabs from quickstart @ykdojo (#5626)
docs: update contributor policy, add contributing section, remove old… @madvart (#5251)
docs: add tip to find your dylib @universalmind303 (#5625)
docs: add data persistence section to quickstart @ykdojo (#5607)
docs: revamp quickstart with Amazon product dataset example @ykdojo (#5585)

✅ Tests

test: fix flaky OpenAI test by using pattern constraint for hex color format @ykdojo (#5808)
test(io): remove flaky test_read_huggingface_http_urls test @ykdojo (#5795)

👷 CI

ci: increase unit-test timeout to 75 minutes for macOS @ykdojo (#5731)
ci: exclude Kaggle from link checker @ykdojo (#5725)

🔧 Maintenance

chore(deps): bump the minor group across 1 directory with 45 updates @dependabot[bot] (#5734)
chore: Provide query end state to RuntimeStatsManager on query end @colin-ho (#5791)
chore: update uvlock to remove tensorflow @kevinzwang (#5780)
chore: Codeowners @colin-ho (#5502)
chore: Don't install pytorch in iceberg test docker compose @colin-ho (#5781)
chore: remove arrow dep from common-image @universalmind303 (#5772)
chore: Pin dependencies @colin-ho (#5667)
chore!: remove spark connect @universalmind303 (#5743)
chore: remove ir and proto crates @universalmind303 (#5742)
chore(deps): bump actions/checkout from 5 to 6 in the all group @dependabot[bot] (#5713)
chore(deps): bump ctor from 0.5.0 to 0.6.1 @dependabot[bot] (#5717)
chore: Cleanup additional Ray runner artifacts @srilman (#5714)
chore: Remove the old Ray Runner @srilman (#5375)
chore: Remove expression namespaces @colin-ho (#5619)
chore: Remove runner from context @colin-ho (#5628)
chore: add deprecation to daft.udf @kevinzwang (#5665)
chore(deps): bump the all group with 13 updates @dependabot[bot] (#5480)
chore: update bug report template @universalmind303 (#5652)
chore: remove checklist from pr template @universalmind303 (#5653)
chore: Remove deprecated agg methods and series split @colin-ho (#5630)
chore: remove broken docpublish job from build-docs workflow @ykdojo (#5631)
chore: Hint users to use .collect when printing empty dataframes @colin-ho (#5616)
chore: reduce binary size by feature flagging derive(Debug) @universalmind303 (#5622)
chore: deprecate llm_generate function @universalmind303 (#5603)
chore: reduce sdist size @universalmind303 (#5606)

⬆️ Dependencies

<details> <summary>4 changes</summary>

chore(deps): bump the minor group across 1 directory with 45 updates @dependabot[bot] (#5734)
chore(deps): bump actions/checkout from 5 to 6 in the all group @dependabot[bot] (#5713)
chore(deps): bump ctor from 0.5.0 to 0.6.1 @dependabot[bot] (#5717)
chore(deps): bump the all group with 13 updates @dependabot[bot] (#5480) </details>

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.14...v0.7.0

Open source →

0.6.1417 Nov 2025

feat: embed text metrics @colin-ho

What's Changed 🚀

✨ Features

feat: embed text metrics @colin-ho (#5583)
feat: Add description and attributes to custom udf metrics @colin-ho (#5574)
feat(flotilla): Aggregate Completed Worker Metrics in StatsManager @srilman (#5531)
feat: add amplification metric for explode operator in native runner @samstokes (#5565)

🐛 Bug Fixes

fix: Fix empty dataframe show issue @caican00 (#5595)
fix: Fix openai test metrics fixture @colin-ho (#5593)
fix: resolve docgen disk space failures by removing unused tools @ykdojo (#5589)
fix: fix imports in explode.rs @colin-ho (#5573)
fix: Add support for parsing STRUCT with parentheses syntax @Lucas61000 (#5449)
fix: Dashboard Verbose Tracing Error @srilman (#5567)
fix: Skip prompt metrics tests on ray runner @colin-ho (#5564)

📖 Documentation

docs: Update AI functions usage patterns @everettVT (#5568)

🔧 Maintenance

chore: Enrich and unify the UDF properties displayed in the query plan @plotor (#5553)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.13...v0.6.14

Open source →

0.6.1313 Nov 2025

refactor!: Remove support for creating File objects from bytes @universalmind303

What's Changed 🚀

💥 Breaking Changes

refactor!: Remove support for creating File objects from bytes @universalmind303 (#5556)

✨ Features

feat: Prompt metrics @colin-ho (#5549)
feat: Async udf metrics @colin-ho (#5541)
feat: Support specifying dimensions for text embedding @samstokes (#5543)
feat: support customized retries error message of S3 request @stayrascal (#5447)
feat: UDF metrics @colin-ho (#5507)
feat: Support product @luoyuxia (#5515)
feat: OTEL Metrics from Swordfish @srilman (#5454)
feat: add tos object source @stayrascal (#5372)
feat: Bind the name of the running UDF to the UDFActor @plotor (#5514)
feat: Support text documents in prompt @colin-ho (#5520)

🐛 Bug Fixes

fix: sorting on a literal value and aggregation with order-by @kevinzwang (#5547)
fix: Limit with Offset returns unexpected result when reading Lance dataset @plotor (#5540)
fix: Add absolute diff threshold to embed_text integration test @colin-ho (#5527)
fix: test_embed_text_with_none_values with the OpenAI provider fails @desmondcheongzx (#5534)
fix: Check for numpy dependency in prompt @colin-ho (#5521)
fix: Handle Nones when embedding text with openai @desmondcheongzx (#5513)
fix: Fix broken benchmark blog link @colin-ho (#5522)

♻️ Refactor

refactor!: Remove support for creating File objects from bytes @universalmind303 (#5556)

📖 Documentation

docs: fix migration guide @kevinzwang (#5563)
docs: standardize key features casing to sentence case @ykdojo (#5559)
docs: legacy UDF migration guide @kevinzwang (#5562)
docs: simplify getting started tip in introduction @ykdojo (#5560)
docs: update blog icon from bookmark to blog @ykdojo (#5557)

🔧 Maintenance

chore: remove docs feedback widget @ykdojo (#5544)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.12...v0.6.13

Open source →

0.6.1210 Nov 2025

feat(flotilla): Add Query ID to OTEL Metrics @srilman

What's Changed 🚀

✨ Features

feat(flotilla): Add Query ID to OTEL Metrics @srilman (#5506)

🐛 Bug Fixes

fix: Support custom models in openai text embedder @colin-ho (#5525)

🚀 Performance

perf: Async embed text @colin-ho (#5524)

📖 Documentation

docs: change the provider name in embed_text example @huleilei (#5523)

👷 CI

ci: Skip Windows Rust Tests in PRs @srilman (#5511)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.11...v0.6.12

Open source →

0.6.117 Nov 2025

feat: PostgresCatalog and PostgresTable followups @desmondcheongzx

What's Changed 🚀

✨ Features

feat: PostgresCatalog and PostgresTable followups @desmondcheongzx (#5508)
feat: Add Catalog and Table implementations for PostgreSQL @desmondcheongzx (#5487)
feat: make maintain_order configurable @stayrascal (#5505)
feat: chat completions api for prompt function @colin-ho (#5497)

🐛 Bug Fixes

fix: Assert nonzero concurrency for udfs @colin-ho (#5500)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.10...v0.6.11

Open source →

0.6.105 Nov 2025

feat: add --addr flag to daft-dashboard cli @VOID001

What's Changed 🚀

✨ Features

feat: add --addr flag to daft-dashboard cli @VOID001 (#5444)
feat: Support multiple image and file inputs for prompt function @colin-ho (#5481)

🐛 Bug Fixes

fix: removes checking model directly for embedding dimensions @rchowell (#5445)
fix: return-dtype for embed_text/image @universalmind303 (#5496)
fix: Lower json inflation factor @colin-ho (#5461)

📖 Documentation

docs: adds daft.func and daft.cls usage with migration page @everettVT (#5475)

🔧 Maintenance

chore: Drop Python 3.9 @srilman (#5479)
chore: remove extra from build command @stayrascal (#5493)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.9...v0.6.10

Open source →

0.6.830 Oct 2025

feat: Allow images in prompt @colin-ho

What's Changed 🚀

✨ Features

feat: Allow images in prompt @colin-ho (#5466)
feat: add pre-existence checks for lance_data_sink @huleilei (#5381)
feat: Support setting actor_udf_ready_timeout via Env @plotor (#5426)
feat: retryable udfs @universalmind303 (#5392)
feat: Add a Bigtable data sink @desmondcheongzx (#5431)
feat: classify_image expression @universalmind303 (#5428)
feat: Add support for Metrics tab in quickstart Ray dashboard @jeevb (#5429)
feat(lance): distributed FTS index creation via Daft UDF with fragment-level parallelism @huleilei (#5236)
feat: add mimetype detection for daft.file @universalmind303 (#5411)

🐛 Bug Fixes

fix: report a more reasonable error message when select * from <some_keywords> @VOID001 (#5440)
fix: Fix async udf with use_process @colin-ho (#5457)
fix: Drop table error in current active session @plotor (#5439)
fix: add retry on "unable to open file" @kevinzwang (#5442)
fix: Allow publishing quickstart helm chart to GHCR @jeevb (#5437)
fix: convert num_rows to int when query count(*) from clickhouse @dujl (#5421)
fix: Actually clone the repo before publishing quickstart helm chart @jeevb (#5433)
fix: file reads for huggingface @universalmind303 (#5427)
fix: Make benchmarking Ray cluster setup commands idempotent @jeevb (#5425)
fix(lance): correct limit pushdown semantics with filters @huleilei (#5408)

🚀 Performance

perf: Call async udfs asynchronously @colin-ho (#5451)
perf: Elide shuffle for window if already partitioned @colin-ho (#5450)
perf: defer allocation when creating series from literals @universalmind303 (#5391)
perf: Double workers per udf actor handle @colin-ho (#5415)

♻️ Refactor

refactor: Make helper function for calling async python functions from rust @colin-ho (#5432)
refactor: combine sentence_transformers + transformers, and clean up … @universalmind303 (#5422)

📖 Documentation

docs: adds ai functions, ai providers, contributing, and docstrings with nav @everettVT (#5438)
docs: warning for Common Crawl dataset API instability @malcolmgreaves (#5436)
docs: update the example to access S3-compatible services @huleilei (#5405)
docs(connectors): add connector page for Lance format @huleilei (#5397)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.7...v0.6.8

Open source →

0.6.720 Oct 2025

feat!: Catch transient errors on turbopuffer writes @desmondcheongzx

What's Changed 🚀

💥 Breaking Changes

feat!: Catch transient errors on turbopuffer writes @desmondcheongzx (#5380)

✨ Features

feat: add viz for embedding @samster25 (#5419)
feat!: Catch transient errors on turbopuffer writes @desmondcheongzx (#5380)
feat(dashboard): Cleanup Queries Page @srilman (#5416)
feat: Extend hash variants for xxhash @srilman (#5276)
feat: prompt @colin-ho (#5394)
feat: Add case function for better SQL-style conditional expressions @rasanpreetsingh3 (#5383)

🐛 Bug Fixes

fix: Reduce number udfs by 1 in multi udf test @colin-ho (#5414)
fix: Wrap azure fsspec in pafs.FSSpecHandler @colin-ho (#5412)
fix(flotilla): Set flotilla actor cpu requests to 1 @colin-ho (#5404)
fix: Fix prompt integration tests @colin-ho (#5401)
fix: Fix Operator Finalization in Swordish Stat Manager @srilman (#5398)

🚀 Performance

perf(swordfish): Reuse scheduler runtime @colin-ho (#5387)

♻️ Refactor

refactor: Cleanup Dtype Names @srilman (#5400)

📖 Documentation

docs: Fix some document errors @plotor (#5409)
docs: update minhash example to use cc dataset @everettVT (#5390)
docs: fix daft.File usage examples @kevinzwang (#5403)

👷 CI

ci: Remove Tests for the Old Ray Runner @srilman (#5374)
ci: disable running tpch profiling on push @kevinzwang (#5384)

🔧 Maintenance

chore: bump pyo3 dependency @universalmind303 (#5410)
chore: revert #5383 @kevinzwang (#5396)
chore: optimize operator naming @Jay-ju (#5204)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.6...v0.6.7

Open source →

0.6.614 Oct 2025

docs!: update docstrings for various functions @universalmind303

What's Changed 🚀

💥 Breaking Changes

docs!: update docstrings for various functions @universalmind303 (#5344)

✨ Features

feat: Explicit AWS vs. HTTP mode for common crawl dataset @malcolmgreaves (#5379)
feat: pydantic model type conversion @kevinzwang (#5370)
feat(dashboard): Individual Query Page @srilman (#5367)
feat(flotilla): Flotilla sort merge join @colin-ho (#5369)
feat: batch UDF with @daft.func.batch @kevinzwang (#5362)
feat(dashboard): Queries Page @srilman (#5257)
feat: more tensor conversions @kevinzwang (#5357)
feat: @daft.cls decorator for new class UDFs @kevinzwang (#5350)
feat: Detect concurrency / num gpus for model apis @colin-ho (#5342)
feat: Flotilla linear scheduler @colin-ho (#4378)
feat: Lazy from_glob_path @colin-ho (#5235)

🐛 Bug Fixes

fix: Use sum supertype for list_sum type inference @colin-ho (#5366)
fix: Use default io config in read video if not passed in @colin-ho (#5364)
fix: support serialize and deserialize LazyImport @stayrascal (#5361)
fix(file): python expects bytes instead of None @universalmind303 (#5348)
fix: read_video_frames handles EOF gracefully @rchowell (#5343)

🚀 Performance

perf(flotilla): Throttle worker refresh and autoscaling @colin-ho (#5351)
perf: Elide shuffle for distinct if input is already partitioned @colin-ho (#5354)
perf: use bincode instead of python for io_conf serialization in FileArray @universalmind303 (#5340)
perf: Only Serialize Required Cols in Process UDFs @srilman (#5069)

📖 Documentation

docs: Update Common Crawl Dataset docs to make AWS region explicit @desmondcheongzx (#5373)
docs: add Flotilla blog post links to AI benchmarks @ykdojo (#5359)
docs: Revamp optimization docs @colin-ho (#5347)
docs!: update docstrings for various functions @universalmind303 (#5344)

👷 CI

ci: Fix ai benchmark workflow @colin-ho (#5363)
ci: Pin pydantic version in provision.py for iceberg tests @colin-ho (#5352)
ci: Add ai benchmarks ci @colin-ho (#5337)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.5...v0.6.6

Open source →

0.6.56 Oct 2025

refactor!: make daft.File immutable @universalmind303

What's Changed 🚀

💥 Breaking Changes

refactor!: make daft.File immutable @universalmind303 (#5288)

✨ Features

feat: add use_process flag for @daft.func(...) @universalmind303 (#5323)
feat: Dashboard Query Subscriber @srilman (#5266)
feat: Subscriber Framework @srilman (#5210)
feat: make file-array serializable @universalmind303 (#5304)
feat: add count() pushdown optimization in Iceberg datasource @huleilei (#5029)

🐛 Bug Fixes

fix: Fix the make docs warnings @colin-ho (#5328)
fix: Iterate on the patched azure_identity@0.21.0 @desmondcheongzx (#5322)
fix: decimal format for handling scientific notation @rchowell (#5303)
fix: Use patched azure_identity@0.21.0 for AKS Workload Identity credentials to continue working > 24 hours @desmondcheongzx (#5299)

🚀 Performance

perf: more literal optimizations @universalmind303 (#5314)
perf: Support parallel CSV parsing when files contain carriage returns @desmondcheongzx (#5319)

♻️ Refactor

refactor!: make daft.File immutable @universalmind303 (#5288)

📖 Documentation

docs: add casting matrix @kevinzwang (#5333)
docs: add daft.func docs page and APIs @kevinzwang (#5335)
docs: Update links for running Daft in distributed mode @desmondcheongzx (#5334)
docs: Fix broken links on minhash example @colin-ho (#5326)
docs: Add architecture docs @colin-ho (#5320)
docs: Add docs to broken link checker @colin-ho (#5324)
docs: Clean up AGENTS.md structure @ykdojo (#5321)
docs: Add Kubernetes quickstart to Daft docs @jeevb (#5318)
docs: Add docs and values reference for quickstart chart @jeevb (#5313)
docs: Fix broken link in Common Crawl dataset docs @desmondcheongzx (#5301)
docs: Document Common Crawl dataset @desmondcheongzx (#5300)

👷 CI

ci: Only fail broken link checker on 404s @colin-ho (#5327)
ci: fix property test column name @kevinzwang (#5325)

🔧 Maintenance

chore: Refactor DistributedPipelineNode to implement TreeDisplay @srilman (#5315)
chore: Enable interactive html for df.__repr_html__ @colin-ho (#5312)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.4...v0.6.5

Open source →

0.6.41 Oct 2025

feat!: unify Python -> Daft type conversions @kevinzwang

What's Changed 🚀

💥 Breaking Changes

feat!: unify Python -> Daft type conversions @kevinzwang (#5201)

✨ Features

feat: Add rows written stat for sinks @colin-ho (#5285)
feat: Add Common Crawl dataset @desmondcheongzx (#5244)
feat!: unify Python -> Daft type conversions @kevinzwang (#5201)
feat: when function @kevinzwang (#5283)
feat: Add first draft of k8s-quickstart helm chart @jeevb (#5272)
feat: add support for pyiceberg 0.10.0 @gmweaver (#5277)

🐛 Bug Fixes

fix: Pass io config when grabbing Common Crawl manifest @desmondcheongzx (#5294)
fix: Use {} for dashboard dynamic route @colin-ho (#5289)

📖 Documentation

docs: Update readme with new benchmarks @colin-ho (#5281)
docs: fixed typo on s3 section of docs @destroyer22719 (#5280)

✅ Tests

test: Temporarily remove Common Crawl integration test @desmondcheongzx (#5296)
test: Complete coverage for array comparisons @desmondcheongzx (#5286)

🔧 Maintenance

chore: Fix imports to not import pyarrow @colin-ho (#5290)
chore: remove ChanChan from PR checklist @kevinzwang (#5282)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.3...v0.6.4

Open source →

0.6.325 Sept 2025

feat: allow PythonArray to be serialized @kevinzwang

What's Changed 🚀

✨ Features

feat: allow PythonArray to be serialized @kevinzwang (#5270)
feat: support pyarrow.schema in write_lance api @huleilei (#5247)
feat(dashboard): Add verbose flag @srilman (#5267)
feat: file_size expr @universalmind303 (#5243)
feat: Use separate runtime and port for detached dashboard @srilman (#5263)
feat: expression avg alias @destroyer22719 (#5252)
feat: add df constructors for daft.File @universalmind303 (#5074)
feat: Add WARC-Target-URI as a top-level column for WARC reads @desmondcheongzx (#5254)
feat: Force build side for broadcast joins @colin-ho (#5238)

🐛 Bug Fixes

fix: Increase actor udf readiness timeout @colin-ho (#5258)
fix: Ensure super extension type is registered in to_arrow_dtype @ConeyLiu (#5265)

🚀 Performance

perf: Use take for join probes @colin-ho (#5160)

♻️ Refactor

refactor: add fragment_group_size to reduce lance scan task @Jay-ju (#5261)

📖 Documentation

docs: improve copy and consistency on index page @ykdojo (#5271)
docs: Update AI benchmarks @colin-ho (#5264)
docs: AI benchmarks @colin-ho (#5245)

🔧 Maintenance

chore(deps): Upgrade aws lc rs @colin-ho (#5274)
chore(deps): bump the minor group across 1 directory with 51 updates @dependabot[bot] (#5262)
chore: Add warning for smj @colin-ho (#5268)
chore(flotilla): Remove stages @srilman (#5222)
chore: Fix more dependabot security warnings @desmondcheongzx (#5260)
chore: Fix dependabot h11 security warning @desmondcheongzx (#5259)
chore: remove old documentation about daft cli @universalmind303 (#5241)

⬆️ Dependencies

chore(deps): bump the minor group across 1 directory with 51 updates @dependabot[bot] (#5262)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.2...v0.6.3

Open source →

0.6.219 Sept 2025Deprecated

chore: remove expr.str.regexp\_split since we will immediately deprecate it @kevinzwang

What's Changed 🚀

✨ Features

feat: add File.to_tempfile method and optimize range requests @universalmind303 (#5226)
feat: migrate nested constructors, get, slice, to_unix_epoch, partitioning functions @kevinzwang (#5233)
feat: implement Lance filter+count pushdown optimization @huleilei (#5152)
feat: migrate remaining basic expressions @kevinzwang (#5234)
feat: Add regexp_count @srilman (#5191)
feat: retry eof error in S3 multipart upload @kevinzwang (#5179)
feat(flotilla): Distributed Pivot @srilman (#5199)
feat: allow for functions module to be accessed from daft import @kevinzwang (#5219)
feat: migrate first batch of daft functions @kevinzwang (#5086)
fix: remove io_config from global scan operator @Jay-ju (#4676)
feat: implement cross join in distributed flotilla engine @ohbh (#5180)
feat: make multipart part size configurable @stayrascal (#5051)

🐛 Bug Fixes

fix: write_iceberg should handle write.data.path @eric-maynard (#5153)
fix: Cast to table schema in lance write @colin-ho (#5221)
fix: rename index function to find @kevinzwang (#5218)
fix: remove unnecessary field renaming in ListMap scalar UDF @ltdthanhdat (#5205)
fix: remove io_config from global scan operator @Jay-ju (#4676)

🚀 Performance

perf: Drop local limit tasks global limit reached @colin-ho (#5194)

📖 Documentation

docs: add file datatype to urls&files docs page @universalmind303 (#5240)
docs: add pipeline visualization and fix grammar in batch inference docs @ykdojo (#5239)
docs: add Examples link to getting started sections @ykdojo (#5202)
docs: fix additional incorrect references to Iceberg as a catalog @ykdojo (#5198)
docs: clarify data catalog vs table format in README @ykdojo (#5197)
docs: remove outdated engine comparison page @ykdojo (#5196)
docs: Add embed_image to the user guide @desmondcheongzx (#5195)
docs: update minhash dedupe example and tutorial @everettVT (#5192)
docs: Add Minhash Dedupe Example @everettVT (#5165)

👷 CI

ci: enable AI integration tests in PR @kevinzwang (#5217)
ci: move transformers tests to integration tests @rchowell (#5186)

🔧 Maintenance

chore: Don't log failures in url download if on_error="null" @colin-ho (#5231)
chore: Make the runner a separate global singleton @srilman (#5185)
chore: remove expr.str.regexp_split since we will immediately deprecate it @kevinzwang (#5216)
chore: Rewrite the dashboard server to use Axum @srilman (#5212)
chore: Remove reset_runner option @srilman (#5184)
chore: Split .str.split into .str.split and .str.regexp_split @srilman (#5211)
chore: enable logging configuration during launch udf worker @stayrascal (#5168)

⏪ Reverts

revert: "fix: remove io_config from global scan operator" @universalmind303 (#5203)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.1...v0.6.2

Open source →

0.6.110 Sept 2025

feat: expose image attribute as expression @Jay-ju

What's Changed 🚀

✨ Features

feat: expose image attribute as expression @Jay-ju (#4848)
feat(flotilla): no shuffle for hash join if conditions are met @colin-ho (#5135)
feat: .list.append Expression @srilman (#5159)
feat: Base64 Encoding @srilman (#5158)
feat: adds support for classify_text @rchowell (#5113)
feat: Add Arrow IPC conversion for RecordBatches @srilman (#5143)
feat: unnest param on @daft.func @kevinzwang (#5132)

🐛 Bug Fixes

fix: Account for unschedulable udf actors @colin-ho (#4987)
fix: Cleanup CLI Progress Bar Output @srilman (#5157)
fix: flaky test test_transformers_image_embedder_other @kevinzwang (#5130)

🚀 Performance

perf: Use count_rows on fragment to reduce lance scans with limit pushdowns only @colin-ho (#5120)

📖 Documentation

docs: improve text readability on examples page @ykdojo (#5182)
docs: add TrendShift badge to README @ykdojo (#5181)
docs: improve explode method documentation with null/empty list examples @ykdojo (#5164)
docs: fix broken tutorial links and remove redundant file @ykdojo (#5154)

👷 CI

ci: Reduce free disk space time @colin-ho (#5178)
ci: re-add mac os unit tests on main @kevinzwang (#5163)
ci: fix TPC-H benchmark workflows @kevinzwang (#5123)
ci: Pipe unit test failures through duration aggregator @colin-ho (#5161)
ci: remove macos from PR test suite @kevinzwang (#5142)
ci: Aggregate test durations @colin-ho (#5129)

🔧 Maintenance

chore: add PR conventions note to AGENTS.md @ykdojo (#5166)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.0...v0.6.1

Open source →

0.6.04 Sept 2025Deprecated

SQLCatalog was deprecated in v0.5 and is now removed, in favor of the bindings kwargs.

What's Changed 🚀

v0.6.0 marks the official release of our new ray-based distributed engine, Flotilla! If you are already using the ray runner, you do not need to change anything. Setting the DAFT_RUNNER=ray environment variable, or within your python program via daft.context.set_runner_ray(), will use Flotilla by default.

All operations except cross join, sort merge join, and pivot are currently supported. We will be working on adding support for them soon! If you need to use the legacy ray runner, please set daft.set_execution_config(use_legacy_ray_runner=True)

💥 Breaking Changes

SQLCatalog was deprecated in v0.5 and is now removed, in favor of the bindings kwargs.

Before:

catalog = SQLCatalog({"test_data": df})
result = daft.sql("SELECT * FROM test_data", catalog=catalog)

After:

bindings = {"test_data": df}
result = daft.sql("SELECT * FROM test_data", **bindings)

feat!: revert daft.func behavior on literal arguments @kevinzwang (#5087)
revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (#5084)

✨ Features

feat(embed_text): Support LM Studio as a provider @desmondcheongzx (#5103)
feat: Implement embed_image() @desmondcheongzx (#5101)
feat!: revert daft.func behavior on literal arguments @kevinzwang (#5087)
feat: Automatically grab embedding dimensions for sentence transformers @desmondcheongzx (#5078)
feat: add mcap datasource reader @Jay-ju (#4727)

🐛 Bug Fixes

fix: Undo skipcheck change @srilman (#5131)
fix: fix youtube video reading @rchowell (#5126)
fix: Remove flotilla fallback @colin-ho (#5114)
fix: Add nulls in json reads if a line doesn't contain the field from the schema @colin-ho (#4993)
fix: Check if UDFs are Serializable @srilman (#5091)
fix: nightly property test @malcolmgreaves (#5076)
fix: Handle Unserializable Errors in Process UDFs @srilman (#5075)
fix: Implement Multi-Column Aggregations with List-like columns @srilman (#5017)

🚀 Performance

perf: Implement count pushdown for parquet @desmondcheongzx (#5038)
perf(flotilla): Use Worker Affinity with Pre-Shuffle Merge @srilman (#5112)
perf: Split UDFs from Filters @srilman (#5070)
perf(embed_text): Let Sentence Transformers select the best available device @desmondcheongzx (#5082)

♻️ Refactor

refactor: proper organization for providers and protocols in daft.ai @rchowell (#5125)

📖 Documentation

docs: fix navigation labels to match section names @ykdojo (#5121)
docs: fix flickering typewriter animation on overview page @ykdojo (#5118)
docs: Add batch inference use case @desmondcheongzx (#5116)
docs: Add docs for custom data sources and sinks @desmondcheongzx (#5115)
docs: add dark mode support for Algolia DocSearch @ykdojo (#5109)
docs: add noindex tag to non-stable pages @jaychia (#5105)
docs: Add text guide @desmondcheongzx (#5102)
docs: Improve installation instructions @desmondcheongzx (#5094)
docs: More fixes to the overview page in light mode @desmondcheongzx (#5095)
docs: Document write_turbopuffer in the user guide @desmondcheongzx (#5092)

👷 CI

ci: fix test-wheels job in build-wheel.yml @kevinzwang (#5134)
ci: Truncate the # of concurrent jobs in PR CI @srilman (#5122)
ci: Run tests before publish @colin-ho (#5009)
ci: Always run the unit-tests required check @colin-ho (#5119)
ci: Do not skip postmerge tests @desmondcheongzx (#5096)

🔧 Maintenance

chore: Add AGENTS.md @srilman (#5124)
chore: Remove docs codeowners @desmondcheongzx (#5111)
chore: Clean up write_turbopuffer guide @desmondcheongzx (#5093)

⏪ Reverts

revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (#5084)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.22...v0.5.23

Open source →

0.5.2228 Aug 2025Deprecated

chore: Remove deprecated APIs for 0.6 @colin-ho

What's Changed 🚀

💥 Breaking Changes

refactor!: use struct datatype as daft representation of tuples @universalmind303 (#5030)

✨ Features

feat: Add uv.lock to git @desmondcheongzx (#5065)
feat: Add Hash Function Support for Decimal128, Time, Timestamp, Timestamptz Datatypes @Zyiqin-Miranda (#5026)
feat: pushdown for lance scan @Jay-ju (#4710)
feat: add lance merge_column task @Jay-ju (#5008)
feat: Make the max parallel of scan tasks configurable for Native Runner @plotor (#5018)
feat: basic generator udf @kevinzwang (#5036)
feat: implements an openai provider with embed_text @rchowell (#4997)
feat: daft.File object store support @universalmind303 (#5002)

🐛 Bug Fixes

fix: Fix venv command for windows build @colin-ho (#5073)
fix: add setuptools_scm to build wheel requirements @colin-ho (#5072)
fix: Use cachebusting and range request fallback for HTTP requests to Hugging Face CDNs @desmondcheongzx (#5061)
fix: Use async for starting and calling udf actors in flotilla @colin-ho (#5000)
fix: Always refresh tqdm when updating total @colin-ho (#5033)
fix: Fix docs build @desmondcheongzx (#5066)
fix: require uv as prerequisite for development setup @ykdojo (#5059)
fix: Add missing source command in Makefile install-docs-deps target @ykdojo (#5060)
fix: Mermaid syntax error when enable explain analyze for Native Runner @plotor (#5052)
fix: clean notebook output before running tests & tweak doc proc notebook @malcolmgreaves (#5055)
fix: correct Modin query optimizer value in comparison tables @ykdojo (#4983)
fix: skip credentialed tests if not from main @rchowell (#5048)
fix: subprocess UDF inherits current process env @rchowell (#5047)
fix: sql/spark read_iceberg and read_deltalake @kevinzwang (#5035)
fix(blc): Disabled pipefail @rohitkulshreshtha (#5031)

♻️ Refactor

refactor!: use struct datatype as daft representation of tuples @universalmind303 (#5030)

📖 Documentation

docs: Make overview page legible for light mode @desmondcheongzx (#5067)
docs: Move custom python code higher up in docs @desmondcheongzx (#5064)
docs: Add better description in overview page @jaychia (#5063)
docs: remove core_concepts.md and broken anchor link references @ykdojo (#5062)
docs: fix formatting @rchowell (#4994)
docs: remove runllm widget @ccmao1130 (#5056)
docs: add reo script to docs @ccmao1130 (#5049)
docs: fix broken UDF link due to core_concepts.md redirect @ykdojo (#5022)
docs: fix typo "Github" --> "GitHub" @metonym (#5025)
docs: fix df.limit link in quickstart.md @rockokw (#5013)

👷 CI

ci: Don't run pr test suite on non-code changes fr @desmondcheongzx (#5057)

🔧 Maintenance

chore: Remove deprecated APIs for 0.6 @colin-ho (#5050)
chore: disable hugging face library progress bars @kevinzwang (#5040)
chore: relax assertion in flaky sharding distribution test @Jay-ju (#5053)
chore(dev): use pyproject.toml to manage the dev dependencies @xy-xin (#4849)
chore: random the counter during creating DistributedActorPoolProject… @stayrascal (#5039)

⏪ Reverts

revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (#5068)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.21...v0.5.22

Open source →

0.5.2121 Aug 2025

feat: Propagate morsel size top-down in swordfish @colin-ho

What's Changed 🚀

✨ Features

feat: Propagate morsel size top-down in swordfish @colin-ho (#4894)
feat: DataFrame.write_huggingface @kevinzwang (#5015)

🐛 Bug Fixes

fix(blc): Attempt to fix the broken link checker. @rohitkulshreshtha (#5010)
fix: Print UDF stdout and Daft logs above the progress bar @srilman (#4861)

📖 Documentation

docs: Add audio transcription example card @desmondcheongzx (#5020)
docs: improve audio transcription example @universalmind303 (#4990)
docs: Spice up the examples page @desmondcheongzx (#5019)

🔧 Maintenance

chore: Only test MacOS on Python 3.11 in CI @srilman (#5014)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.20...v0.5.21

Open source →

0.5.2020 Aug 2025

feat!: RowWiseUdf.eval for eager evaluation @kevinzwang

What's Changed 🚀

💥 Breaking Changes

feat!: RowWiseUdf.eval for eager evaluation @kevinzwang (#4998)

✨ Features

feat: support count(1) in dataframe and choose the cheap column @huleilei (#4977)
feat: add clickhouse data sink @huleilei (#4850)
feat: implement distributed sort in flotilla engine @ohbh (#4991)
feat!: RowWiseUdf.eval for eager evaluation @kevinzwang (#4998)
feat: basic read_huggingface functionality @kevinzwang (#4996)
feat: support using max() and min() on list of boolean values @varun117 (#4989)
feat: Flotilla pre-shuffle merge @colin-ho (#4873)
feat: Flotilla into partitions @colin-ho (#4963)
feat(optimizer): Add Lance count() pushdown optimization @huleilei (#4969)
feat: adds video frame streaming source @rchowell (#4979)
feat: Add offset support to Spark Connect @plotor (#4962)
feat: new daft.File datatype @universalmind303 (#4959)
feat: unify all Daft type to Python type conversions @kevinzwang (#4972)

🐛 Bug Fixes

fix: Can translate sort in flotilla @colin-ho (#5005)
fix: Lazily import pil in infer dtype @colin-ho (#5004)
fix: Lazily import pyarrow when importing daft @colin-ho (#4999)
fix: lance schema does not work @ddupg (#4940)
fix: correct possessive apostrophe typo in README @ykdojo (#4984)
fix: correct GitHub capitalization and add missing period in README @ykdojo (#4985)
fix: ignore NotFound error of the non-first list during iter dir @stayrascal (#4891)
fix: S3 multipart upload redirect to correct region @kevinzwang (#4865)

♻️ Refactor

refactor: make DaftExtension class definition static @Jay-ju (#4968)

📖 Documentation

docs: fix grammar in CONTRIBUTING.md setup instructions @ykdojo (#4986)

👷 CI

ci: Don't run pr test suite on non-code changes @desmondcheongzx (#4992)
ci: No progress bar in CI @colin-ho (#4988)

🔧 Maintenance

chore: Snapshot Testing Optimizations @srilman (#4995)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.19...v0.5.20

Open source →

0.5.1915 Aug 2025

We have a pretty crazy release this time around. Some especially notable features include:

What's Changed 🚀

We have a pretty crazy release this time around. Some especially notable features include:

Interactive DataFrames in Jupyter Notebooks, with special support for some multimodal types
An async API for LLM text generation, particularly with OpenAI
A new .into_batches DataFrame API, the modern alternative to .into_partitions
Adding support for .offset / OFFSET operator across the engine. Thanks @plotor for the great work!
Various Flotilla performance and reliability improvements
Various casting improvements

✨ Features

feat: Async open ai llm generate @colin-ho (#4879)
feat: Add offset syntax support to SQL @plotor (#4707)
feat: adds support for SQL GROUP BY column position @rchowell (#4955)
feat: better dtype type inference @universalmind303 (#4973)
feat: Casting from Python into struct or list types @srilman (#4957)
feat: support creating partitioned tables in Iceberg via the Catalog interface. @redpheonixx (#4951)
feat: implement into_batches operator on flotilla distrubted engine @ohbh (#4958)
feat: literal variants for (pretty much) all types @kevinzwang (#4947)
feat: Add offset support to Flotilla Engine @plotor (#4918)
feat: implement into_batches on the swordfish native daft runner @ohbh (#4935)
feat: Flotilla broadcast join @colin-ho (#4867)

🐛 Bug Fixes

fix: Always just use actor for flotilla scheduler @colin-ho (#4978)
fix: Add handle for swordfish runtime stats manager @colin-ho (#4970)
fix: Dudep lance read required columns @xloya (#4967)
fix: Don't use wildcard for logical plan match in pushdown rules @colin-ho (#4945)
fix: Coerce arrow schema for parquet decoding @colin-ho (#4948)
fix: use associate type for swordfish into_batches operator state @ohbh (#4956)
fix: raise error on invalid cross join parameters @rchowell (#4952)
fix: interactive html fixes @colin-ho (#4943)

♻️ Refactor

refactor: Use associate type for swordfish operator state @colin-ho (#4921)

📖 Documentation

docs: update links in document processing example @ccmao1130 (#4946)
docs: improve daft.func documentation and type inference @universalmind303 (#4942)
docs: fix link for pandas @universalmind303 (#4941)

👷 CI

ci: add 3.11 to matrix and always include typing_extensions @rchowell (#4950)

🔧 Maintenance

chore: Remove Err Case for Series and RecordBatch size_bytes @srilman (#4954)

👋 New Contributors

@redpheonixx made their first contribution in https://github.com/Eventual-Inc/Daft/pull/4951

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.18...v0.5.19

Open source →

0.5.188 Aug 2025

feat: adds column set visitor and use in pushdowns @rchowell

What's Changed 🚀

✨ Features

feat: adds column set visitor and use in pushdowns @rchowell (#4929)
feat: async @daft.func @universalmind303 (#4908)
feat: Add offset operator support to DataFrame for Ray Runner @plotor (#4706)
feat: model resource plumbing for inference expressions @rchowell (#4902)
feat: Flotilla eager limit @colin-ho (#4887)
feat: implement random repartition in flotilla distributed engine @ohbh (#4893)
feat: Add support for specifying hash algorithm used in expression.hash() func. @Zyiqin-Miranda (#4640)

🐛 Bug Fixes

fix: Batch RuntimeSubscriber updates for all nodes @srilman (#4932)
fix: Column Ordering in UDF & Project Optimizations @srilman (#4923)
fix: Refactor Progress Bar to be a RuntimeStatSubscriber @srilman (#4837)

🚀 Performance

perf: scalar udfs use same optimizations as legacy udfs @universalmind303 (#4931)
perf: improve @daft.func performance by ~27% @universalmind303 (#4920)

♻️ Refactor

refactor: move literal to daft-core @kevinzwang (#4928)

📖 Documentation

docs: Generate files for llms.txt @desmondcheongzx (#4937)
docs: Pull colab examples into pages @desmondcheongzx (#4936)
docs: Add embeddings generation example @desmondcheongzx (#4934)
docs: improve navigation for functions doc pages @kevinzwang (#4924)
docs: getdaft.io → daft.ai @ccmao1130 (#4926)
docs: add video to examples @ccmao1130 (#4915)
docs: adds coalesce to docs @rchowell (#4909)
docs: fix doctest formatting errors @rchowell (#4911)
docs: add docstrings to I/O & DataFrame methods (issue Eventual-Inc#4124) @TheOphige (#4854)

🔧 Maintenance

chore: config isort known_third_party to fix import formatting errors @Jay-ju (#4840)
chore: add warning on repartition in native runner @kevinzwang (#4910)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.17...v0.5.18

Open source →

0.5.175 Aug 2025

docs: add examples to docs @ccmao1130

What's Changed 🚀

📖 Documentation

docs: add examples to docs @ccmao1130 (#4903)
docs: Fixing Link to Resource Requests from Managing Memory Usage Page @madvart (#4901)

🔧 Maintenance

chore: change series literal to list @kevinzwang (#4896)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.16...v0.5.17

Open source →

0.5.162 Aug 2025

feat: Interactive jupyter display @colin-ho

What's Changed 🚀

✨ Features

feat: Interactive jupyter display @colin-ho (#4835)
feat: supports passing a projection kwargs in select @rchowell (#4884)
feat: Add offset operator support to DataFrame for Native Runner @plotor (#4582)

🐛 Bug Fixes

fix: Return from streaming sink if channel closed @colin-ho (#4885)
fix: No empty turbopuffer write @colin-ho (#4897)
fix: Use planning config instead of env variables during filter pushdowns @desmondcheongzx (#4888)
fix: azure storage resource url @kevinzwang (#4892)

🚀 Performance

perf: Add parallel execution for Python UDFs with batched GIL acquisitions @universalmind303 (#4886)

📖 Documentation

docs: Remove redirect for installation page @desmondcheongzx (#4895)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.15...v0.5.16

Open source →

0.5.151 Aug 2025

feat: add openai provider in llm\_generate function @huleilei

What's Changed 🚀

✨ Features

feat: add openai provider in llm_generate function @huleilei (#4809)
feat: Use shuffle_aggregation_default_partitions in flotilla aggregate @colin-ho (#4869)
feat: abstract the interface of scan pushdown @Jay-ju (#4772)
feat: Add get_or_infer_runner_type to support getting runner type from context @plotor (#4810)
feat: support glob multiple path @stayrascal (#4811)

🐛 Bug Fixes

fix(runtime): Reduce thread name length for compute and I/O threadpools @rohitkulshreshtha (#4877)
fix: import ParamSpec from typing_extensions only for python < 3.10 @kevinzwang (#4878)
fix: Allow file:/ schemes in read_iceberg @colin-ho (#4843)

📖 Documentation

docs: Restructure docs to target users @desmondcheongzx (#4875)

🔧 Maintenance

chore: Cleanup MicroPartition::concat_or_get @srilman (#4864)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.14...v0.5.15

Open source →

0.5.1429 Jul 2025

feat: Include UDF Names in Progress Bar @srilman

What's Changed 🚀

✨ Features

feat: Include UDF Names in Progress Bar @srilman (#4813)

🐛 Bug Fixes

fix: Fix list sorting when operating on a morsel slice @desmondcheongzx (#4863)

📖 Documentation

docs: add more docs for UDF concurrency @kevinzwang (#4856)

🔧 Maintenance

chore: new code path for scalar UDF @kevinzwang (#4814)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.13...v0.5.14

Open source →

0.5.1328 Jul 2025

feat: Support multiple namespaces in the turbopuffer writer @desmondcheongzx

What's Changed 🚀

✨ Features

feat: Support multiple namespaces in the turbopuffer writer @desmondcheongzx (#4845)
feat: add support for reading json arrays @universalmind303 (#4844)
feat: Use daft-decoding for hive value deserialization @colin-ho (#4836)
feat: use min size column as count pushdown column @Jay-ju (#4681)
feat: optimize read parquet metadata logic via suffix-range get @stayrascal (#4775)
feat(turbopuffer): Add client and write kwargs @desmondcheongzx (#4825)

🐛 Bug Fixes

fix: add cachebusting parameter to huggingface urls @universalmind303 (#4853)
fix: Handle unserializable exceptions from data sinks @desmondcheongzx (#4858)
fix: concat chunked array safety @stayrascal (#4852)
fix: Allow anonymous mode to be used for S3 uploads @desmondcheongzx (#4855)
fix: read/write embeddings for Parquet and Lance @malcolmgreaves (#4834)
fix: Fix File writer must be created before bytes_written can be called bug in native parquet writer @colin-ho (#4817)
fix: read/write embeddings for Parquet and Lance @malcolmgreaves (#4812)
fix: expand ~ to home directory in deltatable read and write @r3stl355 (#4831)

📖 Documentation

docs: categorize functions and make each function its own page @kevinzwang (#4838)

👷 CI

ci: update daft domain @ccmao1130 (#4678)
ci: fail on timeout @colin-ho (#4833)

🔧 Maintenance

chore: upgrade py-spy version @Jay-ju (#4830)

⏪ Reverts

revert: "fix: read/write embeddings for Parquet and Lance" @malcolmgreaves (#4832)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.12...v0.5.13

Open source →

0.5.1223 Jul 2025

feat: struct Expression.unnest() @kevinzwang

What's Changed 🚀

✨ Features

feat: struct Expression.unnest() @kevinzwang (#4815)

🐛 Bug Fixes

fix: add custom robots.txt @ccmao1130 (#4823)
fix: update a new expression example in the contributing guide to work @r3stl355 (#4818)
fix: LSP handling for daft.context and daft.io @srilman (#4805)

🚀 Performance

perf: Use processes for non-actor UDFs @srilman (#4477)

📖 Documentation

docs: fix typo in scaling description @ykdojo (#4808)

🔧 Maintenance

chore: add vscode debug example with env @Jay-ju (#4819)
chore(deps-dev): bump boto3-stubs[essential,glue,s3,s3tables] from 1.36.20 to 1.38.46 @dependabot[bot] (#4661)
chore(deps-dev): bump pyarrow from 19.0.1 to 20.0.0 @dependabot[bot] (#4647)
chore: add DAFT_LOG env indicates log level @Jay-ju (#4807)
chore: Support specify extra arguments when running make test @plotor (#4801)
chore: remove dbg from tests @rchowell (#4804)

⬆️ Dependencies

chore(deps-dev): bump boto3-stubs[essential,glue,s3,s3tables] from 1.36.20 to 1.38.46 @dependabot[bot] (#4661)
chore(deps-dev): bump pyarrow from 19.0.1 to 20.0.0 @dependabot[bot] (#4647)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.11...v0.5.12

Open source →

0.5.1118 Jul 2025Deprecated

fix: Clarify json.query deprecation @colin-ho

What's Changed 🚀

✨ Features

feat: Spread actor udf tasks @colin-ho (#4802)
feat: Allow flotilla to autoscale with existing workers @colin-ho (#4765)
feat: Add a DataFrame.write_turbopuffer method @desmondcheongzx (#4798)
feat(flotilla): Non-actor UDFs with gpu requirements @colin-ho (#4731)
feat: Add experimental turbopuffer sink @desmondcheongzx (#4779)
feat: support suffix range get @stayrascal (#4688)

🐛 Bug Fixes

fix: Propagate worker failures to scheduler @colin-ho (#4800)
fix: mark all pending nodes completed when plan completes @joyceerhl (#4799)
fix(flotilla): Add logical node ids to task context @colin-ho (#4755)
fix: Improve UDF Error Messaging @srilman (#4788)
fix: use pc.field in turbopuffer sink @colin-ho (#4790)
fix(repr): avoid crashing notebook UI when visualizing large strings @joyceerhl (#4789)
fix: use / for objectstorage instead of os.path.join when reading deltalake @universalmind303 (#4785)
fix: Filter out null ids for turbopuffer sink @colin-ho (#4784)
fix: Don't recompute df for count rows if results exist @colin-ho (#4778)
fix: Clarify json.query deprecation @colin-ho (#4774)
fix: document processing notebook and explode expression docs @kevinzwang (#4758)

📖 Documentation

docs: fix docstrings for map and shift_right @ccmao1130 (#4797)
docs: add docstrings to Schema methods (issue #4770) @TheOphige (#4792)
docs: update slack link in runllm widget @ccmao1130 (#4769)

👷 CI

ci: enable building wheels without LTO @joyceerhl (#4787)

🔧 Maintenance

chore: Refactor flotilla pipeline node @colin-ho (#4793)
chore: add docstring template to contributing @ccmao1130 (#4794)
chore: add note about test cluster usage @colin-ho (#4795)
chore: add bucket into endpoint for virtual host style @stayrascal (#4740)
chore: add doc format and install dependency @Jay-ju (#4741)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.10...v0.5.11

Open source →

0.5.1014 Jul 2025

feat(flotilla): Sort \& Top-N @srilman

What's Changed 🚀

✨ Features

feat(flotilla): Sort & Top-N @srilman (#4734)
feat(flotilla): Support Gather Variants of Aggs @srilman (#4701)
feat: explode expression @kevinzwang (#4751)
feat(flotilla): Added concat @rohitkulshreshtha (#4748)
feat: support for deltalake v1.0 @kevinzwang (#4718)
feat(flotilla): Hash Join @srilman (#4700)
feat(flotilla): Add monotonically_increasing_id. @rohitkulshreshtha (#4717)

🐛 Bug Fixes

fix: Ignore partition refs if they are empty @colin-ho (#4766)
fix: Change dashboard warning to info @colin-ho (#4767)
fix(flotilla): Add multiple notify tokens to task @colin-ho (#4757)
fix: Put to_partition_tasks call in try block @colin-ho (#4762)
fix: fix explode expr doc test @colin-ho (#4764)
fix: Swap arg order for list_fill @colin-ho (#4761)
fix: Accept embedding types in cosine_distance @colin-ho (#4753)
fix: populate query graph nodes even if they lack task progress @joyceerhl (#4752)

🚀 Performance

perf: Concat partitions in flotilla repartition @colin-ho (#4756)

📖 Documentation

docs: Add document processing notebook tutorial @malcolmgreaves (#4750)
docs: add some s3config on tos @Jay-ju (#4742)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.9...v0.5.10

Open source →

0.5.910 Jul 2025

feat(flotilla): GroupBy-style Ops @srilman

What's Changed 🚀

✨ Features

feat(flotilla): GroupBy-style Ops @srilman (#4699)
feat: Customizable Progress Bar Statistics @srilman (#4541)
feat: rowwise (scalar) udfs @universalmind303 (#4723)
feat: add some spec for some params @Jay-ju (#4642)
feat: add list_map function @universalmind303 (#4529)
feat(flotilla): support task progress for logical plan graph @joyceerhl (#4686)
feat(flotilla): Repartition @srilman (#4626)

🐛 Bug Fixes

fix(docs): Updated docs to correct calculate max rows per partition @rohitkulshreshtha (#4744)
fix: flush pending task updates in HttpSubscriber on plan complete event @joyceerhl (#4736)
fix: DAFT_FLOTILLA env var should check for false conditions now @colin-ho (#4738)
fix: update readme banner image link @ccmao1130 (#4733)
fix(flotilla): Disambiguate StatisticsEvent variant names to reflect events @rohitkulshreshtha (#4722)
fix: Uncomment skipped iceberg tables in tests @colin-ho (#4593)
fix: Use large PyArrow types for casting from Python series @srilman (#4625)
fix(flotilla): Change MaterializedOutput to return a Vec<Partitions> @srilman (#4715)
fix(flotilla): Add back autoscaling function @colin-ho (#4709)
fix: Picking MicroPartitions with duplicate column names @srilman (#4696)
fix: Fix flaky notebook that fails if image decode fails. @rohitkulshreshtha (#4693)

📖 Documentation

docs: Add debugging udfs docs @colin-ho (#4714)

🔧 Maintenance

chore(flotilla): Refactor task naming for clarity @rohitkulshreshtha (#4730)
chore: support pass virtual host style to pyarrow s3 fs @stayrascal (#4726)
chore: scalarudf post refactor cleanup @universalmind303 (#4480)
chore: disable another flaky test @universalmind303 (#4721)
chore: Disabling flaky test until resolution. @rohitkulshreshtha (#4698)
chore: Add build-whl make command for building wheel package @plotor (#4689)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.8...v0.5.9

Open source →

0.5.83 Jul 2025

feat: move lakehouse source to io directory @Jay-ju

What's Changed 🚀

✨ Features

feat: move lakehouse source to io directory @Jay-ju (#4635)
feat: read_lance with datasource api and move lance w/r to connector directory @Jay-ju (#4622)
feat: Add support for JSON writes in the native runner @desmondcheongzx (#4540)

🐛 Bug Fixes

fix: keep the same behavior of local source for invalid range get ope… @stayrascal (#4638)
fix: Change the Limit operator type from i64 to u64 @plotor (#4637)
fix: Fix setting min_cpu_per_task when using daft.context.set_execution_config @xloya (#4668)
fix: Dependabot tweaks @srilman (#4659)

📖 Documentation

docs: Minor Dev Workflow & README Updates @malcolmgreaves (#4645)

🔧 Maintenance

chore: keep same ruff version with pre-commit-config @stayrascal (#4673)
chore(deps-dev): bump tiktoken from 0.7.0 to 0.9.0 @dependabot[bot] (#4652)

⬆️ Dependencies

chore(deps-dev): bump tiktoken from 0.7.0 to 0.9.0 @dependabot[bot] (#4652)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.7...v0.5.8

Open source →

0.5.71 Jul 2025

This release makes our new distributed execution engine, nicknamed Flotilla, the default. Try it out!

What's Changed 🚀

This release makes our new distributed execution engine, nicknamed Flotilla, the default. Try it out!

✨ Features

feat(flotilla): Enable flotilla by default @colin-ho (#4641)
feat(flotilla): Logging @colin-ho (#4629)
feat: Add more lance.dataset kwargs to daft.read_lance() @huleilei (#4621)
feat(flotilla): Fault tolerance @colin-ho (#4628)
feat: Enable native remote writer by default @rohitkulshreshtha (#4630)
feat(flotilla): Concurrent flotilla tasks @colin-ho (#4607)
feat: adds daft expr to pyarrow compute translator @rchowell (#4616)
feat: Support nulls first/last ordering for window functions @desmondcheongzx (#4546)
feat(flotilla): Flotilla autoscaling @colin-ho (#4606)
feat: support version to read lance table @huleilei (#4585)

🐛 Bug Fixes

fix: Typing of DataFrame.__getitem__ @srilman (#4572)
fix: missing libpython when using pyo3 in cargo test @Jay-ju (#4576)

🚀 Performance

perf: Split projections after optimizer passes @desmondcheongzx (#3992)

📖 Documentation

docs: update site description @ccmao1130 (#4632)

👷 CI

ci: Deflake series iter memory test @desmondcheongzx (#4608)

🔧 Maintenance

chore: Add testing section to contributing guide @colin-ho (#4604)
chore: Improve codecov @colin-ho (#4601)

⬆️ Dependencies

chore(deps-dev): bump memray from 1.13.4 to 1.17.2 @dependabot[bot] (#4651)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.6...v0.5.7

Open source →

0.5.624 Jun 2025

feat: support daft schema to arrow schema with metadata @Jay-ju

What's Changed 🚀

✨ Features

feat: support daft schema to arrow schema with metadata @Jay-ju (#4578)
feat(flotilla): Flotilla progress bar @colin-ho (#4584)
feat: Commit write sink @colin-ho (#4569)
feat: resolve some unused warning @Jay-ju (#4583)
feat: Support file-based sharding @desmondcheongzx (#4483)
feat: support reading from AWS Athena @ptdtan (#4573)
feat: add some throttling error for tos @Jay-ju (#4563)
feat: creates daft-ir and daft-proto @rchowell (#4539)
feat: Add DISTINCT ON / drop_duplicates on a subset of columns @srilman (#4513)
feat(flotilla): Make one node per physical operator @colin-ho (#4555)

🐛 Bug Fixes

fix: Pin optional pyspark dependency to 3.5.5 @desmondcheongzx (#4596)
fix: daft spark-connect server on windows hangs @SparkApplicationMaster (#4581)
fix: correct the throttle_time to fix flakey test @rchowell (#4579)
fix: Fix daft.range partitioning logic @colin-ho (#4570)
fix: Fix get_or_create_runner deadlock @colin-ho (#4561)
fix: fix pypi image url @colin-ho (#4562)
fix: Increase runtime stats subscriber test intervals @colin-ho (#4571)
fix: fix ray connection issues with ray:// protocol @jaychia (#4567)
fix: add missing distinct field @rchowell (#4558)
fix: url download retry test io config @kevinzwang (#4548)

🚀 Performance

perf: Improve efficiency of to_arrow for struct / fixed size lists @colin-ho (#4560)

🔧 Maintenance

chore: Use scarf only for analytics @colin-ho (#4595)
chore: Update various Rust dependencies @srilman (#4574)
chore: improve the stability of unit test @stayrascal (#4565)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.5...v0.5.6

Open source →

0.5.519 Jun 2025

chore!: upgrade AWS sdk and remove vendored OpenSSL @kevinzwang

What's Changed 🚀

💥 Breaking Changes

chore!: upgrade AWS sdk and remove vendored OpenSSL @kevinzwang (#4508)

✨ Features

feat: add new "dashboard" metrics subscriber @universalmind303 (#4527)
feat: add some code style check cmd @Jay-ju (#4505)
feat: HTTPConfig settings for timeouts and retries @srilman (#4537)
feat(flotilla): Add cpu profiling / tracing @colin-ho (#4542)
feat: Url parse function @jordandelbar (#4533)

🐛 Bug Fixes

fix: Resolve mismatch with Thrift compact protocol @desmondcheongzx (#4545)
fix: Enable chrome trace to flush on native executor cancel @colin-ho (#4534)

📖 Documentation

docs: Fix docstring for DataFrame.sort() @desmondcheongzx (#4551)

🔧 Maintenance

chore: add @publicapi decorators to all public expr methods @universalmind303 (#4473)
chore!: upgrade AWS sdk and remove vendored OpenSSL @kevinzwang (#4508)
chore(flotilla): PipelineNode .start accept Arc<Self> to simplify cloning @colin-ho (#4543)
chore: change unity volume prefix to "vol+dbfs:/" @kevinzwang (#4519)
chore: Vendor parquet-format-safe dependency @desmondcheongzx (#4544)
chore: Refactor storage backends for native writers @desmondcheongzx (#4536)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.4...v0.5.5

Open source →

0.5.414 Jun 2025

feat(flotilla): Use max sources config to determine files per partition @colin-ho

What's Changed 🚀

✨ Features

feat(flotilla): Use max sources config to determine files per partition @colin-ho (#4535)
feat(flotilla): Plan explain @colin-ho (#4517)
feat: Add a native remote parquet writer @desmondcheongzx (#4493)
feat(flotilla): Actor pool project @colin-ho (#4497)
feat: add clean for target files @Jay-ju (#4507)
feat(connect): add support for spark when/otherwise @universalmind303 (#4524)
feat(sql): support count(1) @universalmind303 (#4512)
feat: regexp replace posix groups and single regex speedup @SparkApplicationMaster (#4504)
feat: make min_cpu of ray runner configurable @Jay-ju (#4506)
feat: adds .serialize('json') to expressions @rchowell (#4516)
feat: f"" string style string concatenation @Farzan-Hashmi (#4500)
feat: add id and plan_id to otel subscribers, and add method for getting tree hierarchy for a pipeline @universalmind303 (#4485)
feat: adds .jq expression with complex use-case examples @rchowell (#4470)

🐛 Bug Fixes

fix: Fix chrome trace @colin-ho (#4531)
fix: Use PYTHON_VERSION instead of python3 in Makefile @petern48 (#4499)

🚀 Performance

perf: add harness for benchmarking parquet writes @desmondcheongzx (#4523)

📖 Documentation

docs: reorganize integrations @ccmao1130 (#4488)
docs: fix bad markdown links @rchowell (#4528)

👷 CI

ci: Add a dbg check in precommit @jordandelbar (#4509)
ci: Fix style checks @colin-ho (#4511)

🔧 Maintenance

chore: add additional context to the runtime stats events @universalmind303 (#4520)
chore: add better messaging for missing spark functions @universalmind303 (#4521)
chore: update dashboard backend and disable query viz temporarily @universalmind303 (#4475)
chore: add test case over #4180 @flaneur2020 (#4484)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.3...v0.5.4

Open source →

0.5.35 Jun 2025

feat: url download for unity catalog volumes @kevinzwang

What's Changed 🚀

✨ Features

feat: url download for unity catalog volumes @kevinzwang (#4476)
feat(flotilla): Flotilla runner @colin-ho (#4459)

♻️ Refactor

refactor(exprs): move uri functions into own crate @universalmind303 (#4455)
refactor(exprs): refactor tokenize, coalesce, struct @universalmind303 (#4454)
refactor(exprs): refactor hash,minhash, and cosine_distance @universalmind303 (#4452)

🔧 Maintenance

chore: rename block_on method to convey intent @rohitkulshreshtha (#4474)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.2...v0.5.3

Open source →

Your coding agent can read these notes before it upgrades. Set up the MCP server →