datafusion

This project publishes no release notes. Between 53.0.0 and 54.0.0 there were 53 commits, 49 of them substantive:

• ci: disable symbol export on Windows verification (#1486)
• Add Python bindings for accessing ExecutionMetrics (#1381)
• Support None comparisons for null expressions (#1489)
• Fix error on show() with an explain plan (#1492)
• Add SKILL.md and enrich package docstring (#1497)
• Skills require the header to be the first thing in the file which conflicts with the RAT check. Make an exception for this file. (#1501)
• docs: enrich module docstrings and add doctest examples (#1498)
• docs: add README section for AI coding assistants (#1503)
• tpch examples: rewrite queries idiomatically and embed reference SQL (#1504)
• feat: add AI skill to find and improve the Pythonic interface to functions (#1484)
• Move public skills to a directory to avoid downloading the whole repo (#1519)
• Update user documentation for AI agent skill usage (#1505)
• docs: add upstream sync process documentation (#1524)
• build(deps): combined dependabot bumps (Cargo + workflows) (#1534)
• Add support for logical and physical codecs (#1541)
• Update datafusion dependency to latest in preparation for DF54 (#1532)
• feat: enable pickling of most Expr except udaf and udwf (#1544)
• feat: enable pickling for Python aggregate and window UDFs (#1545)
• feat: expose variety of features from DF54 update (#1554)
• Add details on caching to skill (#1521)

…and 29 more.

This project publishes no release notes. Between 52.3.0 and 53.0.0 there were 53 commits, 49 of them substantive:

• Merge release 52.0.0 into main (#1389)
• Add workflow to verify release candidate on multiple systems (#1388)
• Allow running "verify release candidate" github workflow on Windows (#1392)
• fix: satisfy rustfmt check in lib.rs re-exports (#1406)
• Add check for crates.io patches to CI (#1407)
• Enable doc tests in local and CI testing (#1409)
• Upgrade to DataFusion 53 (#1402)
• Catch warnings in FFI unit tests (#1410)
• feat: feat: add to_time, to_local_time, to_date functions (#1387)
• Add docstring examples for Scalar trigonometric functions (#1411)
• Create workspace with core and util crates (#1414)
• feat: Add FFI_TableProviderFactory support (#1396)
• Add docstring examples for Scalar regex, crypto, struct and other (#1422)
• Add docstring examples for Scalar math functions (#1421)
• Add docstring examples for Common utility functions (#1419)
• Add docstring examples for Aggregate basic and bitwise/boolean functions (#1416)
• Fix CI errors on main (#1432)
• Add docstring examples for Scalar temporal functions (#1424)
• Add docstring examples for Aggregate statistical and regression functions (#1417)
• Add docstring examples for Scalar array/list functions (#1420)

…and 29 more.

This project publishes no release notes. Between 52.0.0 and 52.3.0 there were 3 commits, 3 of them substantive:

• Update datafusion dependency to 52.3.0
• Add changelog for 52.3.0
• minor: cargo fmt nightly update

This project publishes no release notes. Between 51.0.0 and 52.0.0 there were 27 commits, 18 of them substantive:

• Use explicit timer in unit test (#1338)
• Add use_fabric_endpoint parameter to MicrosoftAzure class (#1357)
• Prepare for DF52 release (#1337)
• Pass Field information back and forth when using scalar UDFs (#1299)
• Cargo update to prepare for DF52 release (#1368)
• Improve displayed error by using DataFusionError's Display trait (#1370)
• Enforce DataFrame display memory limits with max_rows + min_rows constraint (deprecate repr_rows) (#1367)
• Implement all CSV reader options (#1361)
• chore: add confirmation before tarball is released (#1372)
• feat: add CatalogProviderList support (#1363)
• Build in debug mode for PRs (#1375)
• Remove the FFI test wheel from the distribution artifact (#1378)
• fix: mangled errors (#1377)
• docs: Clarify first_value usage in select vs aggregate (#1348)
• feat: add support for generating JSON formatted substrait plan (#1376)
• Fix Python UDAF list-of-timestamps return by enforcing list-valued scalars and caching PyArrow types (#1347)
• feat: add regexp_instr function (#1382)
• minor: update cargo dependencies (#1383)

This project publishes no release notes. Between 50.1.0 and 51.0.0 there were 26 commits, 23 of them substantive:

• feat: expose select_exprs method on DataFrame (#1271)
• feat: allow DataFrame.filter to accept SQL strings\ (#1276)
• Add temporary view option for into_view (#1267)
• feat: support session token parameter for AmaoznS3 (#1275)
• with_column supports SQL expression (#1284)
• feat: Add SQL expression for repartition_by_hash (#1285)
• feat: Add SQL expression support for with_columns (#1286)
• Update python minimum version to 3.10 (#1296)
• Enable remaining pylints (#1298)
• Add Arrow C streaming, DataFrame iteration, and OOM-safe streaming execution (#1222)
• Add PyCapsule Type Support and Type Hint Enhancements for AggregateUDF in DataFusion Python Bindings (#1277)
• feat: reduce duplicate fields on join (#1184)
• Add function collect_column to dataframe (#1302)
• chore: apply cargo fmt with import organization (#1303)
• Feat/parameterized sql queries (#964)
• Upgrade to Datafusion 51 (#1311)
• Minor build errors (#1325)
• Update build workflow link (#1330)
• fix: use coalesce instead of drop_duplicate_keys for join (#1318)
• Do not convert pyarrow scalar values to plain python types when passing as lit (#1319)

…and 3 more.

This project publishes no release notes. Between 50.0.0 and 50.1.0 there were 13 commits, 10 of them substantive:

• Fix drop() method to handle quoted column names consistently (#1242)
• Make Session Context pyclass frozen so interior mutability is only managed by rust (#1248)
• CHANGES to review
• REVERT CHANGES to commit
• macos-13 is deprecated (#1259)
• Freeze PyO3 wrappers & introduce interior mutability to avoid PyO3 borrow errors (#1253)
• Unify Table representations (#1256)
• feat: expose DataFrame.write_table (#1264)
• feat: expose DataFrame.parse_sql_expr (#1274)
• Update cargo lock

This project publishes no release notes. Between 49.0.0 and 50.0.0 there were 13 commits, 12 of them substantive:

• \49.0.0 release (#1211)
• Update development guide in README.md (#1213)
• Add benchmark script and documentation for maximizing CPU usage in DataFusion Python (#1216)
• Fixing a few Typos (#1220)
• Set fail on warning for documentation generation (#1218)
• docs: fix CaseBuilder documentation example (#1225)
• fix: update link to user example for custom table provider (#1224)
• feat: allow passing a slice to and expression with the [] indexing (#1215)
• chore: remove redundant error transformation (#1232)
• Support string column identifiers for sort/aggregate/window and stricter Expr validation (#1221)
• docs: add apache iceberg as datafusion data source (#1240)
• Prepare for DF50 (#1231)

This project publishes no release notes. Between 48.0.0 and 49.0.0 there were 18 commits, 10 of them substantive:

• Prepare for 48 release (#1175)
• Change merge rule to require branches up to date but do not require a committer approval. This matches the arrow-rs repo. (#1188)
• Fix Python UDAF Accumulator Interface example to Properly Handle State and Updates with List[Array] Types (#1192)
• chore: Upgrade datafusion to version 49 (#1200)
• Update how to dev instructions (#1179)
• Allow users to pass a single expression instead of a list of expressions for partition_by and order_by (#1187)
• suggestions (#1183)
• Improve show() output for empty DataFrames (#1208)
• fix(build): Include build.rs in published crates (#1199)
• Update cargo lock file and changelog for release

This project publishes no release notes. Between 47.0.0 and 48.0.0 there were 16 commits, 16 of them substantive:

• Release DataFusion 47.0.0 (#1130)
• Add a documentation build step in CI (#1139)
• Add DataFrame API Documentation for DataFusion Python (#1132)
• Add Interruptible Query Execution in Jupyter via KeyboardInterrupt Support (#1141)
• feat: upgrade df48 dependency (#1143)
• Support types other than String and Int for partition columns (#1154)
• feat: Support Parquet writer options (#1123)
• Fix signature of __arrow_c_stream__ (#1168)
• Consolidate DataFrame Docs: Merge HTML Rendering Section as Subpage (#1161)
• Add compression_level support to ParquetWriterOptions and enhance write_parquet to accept full options object (#1169)
• feat: dataframe string formatter (#1170)
• feat: collect once during display() in jupyter notebooks (#1167)
• feat: python based catalog and schema provider (#1156)
• feat: add FFI support for user defined functions (#1145)
• refactor: style loading logic in DataFrameHtmlFormatter (#1177)
• Prepare for 48 release