polars-ops

✨ Enhancements

• Add adaptive HTTP rate-limiter for cloud IO (#28591)

🐞 Bug fixes

• Unsoundness in rayon block_on (#28709)
• Check for (shallow) IR node equality on top of hashing in CSPE (#28506)

📖 Documentation

• Document that arg_extremum functions return any index on ties (#28691)

Thank you to all our contributors for making this release possible!
@dancsi, @dsprenkels, @kdn36, @orlp and @ritchie46

This Releases' DSL matches Python 1.43.2.

🚀 Performance improvements

• Pushdown len() to concat/union inputs (#28570)
• Optimize not(bool_f) to not_bool_f (#28474)
• Split multiplexers that directly scan from in-memory DataFrame (#28376)
• Make DNS cache global (#28352)
• Do not remove cache if predicates not pushed to all inputs (#28341)
• Environment variable for logging slow DNS lookup (#28211)
• Pre-partition on left, right and semi joins on hive partitioned data (#28374)
• Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
• Tighten redundant comparisons and propagate filter constraints (#27999)
• Canonicalize predicates (#28212)
• Remove excess memory copy in streaming IPC sink (#28141)
• Parallelize scan_iceberg resolve (#28168)
• Chunk-aware morsel splitting on single-file IO sink (#27950)
• Don't allocate for full_null() with small lengths (#28181)
• Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)
• Add a sampled resolve mode for multi-file parquet metadata (#28111)
• Speed up small dtype series sums with a upcast sum kernel (#27958)
• Don't block on path expansion (#28073)
• Add bytes-based concurrency control for cloud IO (#27924)
• Stop flushing multiplexer on phase change (#28013)
• Prevent empty chunks from triggering copy in rechunk (#27953)
• Improve performance of struct unnest inside list eval on streaming (#27905)

✨ Enhancements

• Add struct.drop() (#28666)
• Improve error message when CSV name de-duplication fails (#28658)
• Always keep first metadata per source for Parquet (#28661)
• Improve plan-time row estimates for multi-file parquet scans (#28380)
• Serialize and deserialize SinkTypeIR::Partitioned (#28112)
• Add infer_schema_files parameter to scan_csv (#28440)
• Support reading IEEE 754 total order Parquet column order (#27896)
• Expose more Expr nodes for cudf_polars (pt 2) (#28404)
• Allow callback sinks on cloud (#28458)
• Pre-partition group-by on hive keys (#28444)
• Add ewm_sum and ewm_sum_by #28151 (#28215)
• Serve stale records from object_store DNS cache (#28256)
• Expose more expr nodes for cudf-polars (#28117)
• Make partition_hive an opt flag (#28345)
• When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
• Show only differing dtypes in assert_frame_equal error (#27816)
• Add Clone to SpillFrame/Token (#28302)
• Add Series.degrees/radians and clarify trig functions use radians (#22961)
• Merge sorted with multikey (#28067)
• Add list expression which consistently packs elements together into new List type (#27990)
• Construct pyarrow predicates via PyO3 instead of eval str (#27271)
• Optimize joins with redundant equi-join keys (#28017)
• Implement missing datetime options for CastColumnsPolicy (#28084)
• Add Expr.cat.to and Expr.cat.physical (#28096)
• Add POLARS_OOC_DISK_BUDGET_MB (#28045)
• Close ResourceWarning leaks in database tests (#28066)
• Expose IRFunctionExpr::Fused in the python node visitor (#27889)
• Deprecate casts from string to temporal dtypes (#28056)
• Add naive out-of-core spilling to Polars (#27998)
• Add experimental strict mode (#28023)
• Optimize non-negative len and null_count comparisons (#28002)
• Optimize len and null_count comparisons (#27727)
• Support SQL's implicit JOIN syntax (#27890)
• Sample from bivariate distribution for spill context Thompson sampling (#27973)
• Add Expr.is_sorted (#26708)
• Add suggestions in wrong api usage or mismatched column names (#27954)
• Add external object_store support for non-native schemes (#27871)
• Skip Delta files and parquet row groups using per-field struct statistics (#27887)
• Add re-registration of OOC SpillTokens (#27909)
• Improve IR sortedness detection for with_row_index and explode (#27875)
• Add DataFrame.is_sorted() (#27870)

🐞 Bug fixes

• Don't hand IO plugins a predicate they cannot evaluate (#28643)
• Don't hang or crash on sqrt/cbrt/pct_change/ewm_* with a Struct column (#28696)
• Raise on overflow in Decimal sum instead of silently wrapping (#28688)
• Preserve nulls when importing Arrow maps (#28680)
• Propagate null by column values in rolling_*_by (#27367)
• Fix self-referencing field in struct.with_fields with over (#28678)
• Fix Arrow buffer offset for Utf8 and Binary (#28662)
• Clamp group-by slice offset (#28579)
• Ensure categorical fill_null(strategy="min"/"max") uses lexical min/max (#28636)
• Release the GIL while resolving the schema in __arrow_c_stream__ (#28642)
• Fix data corruption on Series from nested Arrow LargeList (#28632)
• Properly propagate enum metadata for iceberg column mapping (#28580)
• Correctly write parquet field IDs for enums and categoricals (#28581)
• Fix arrow C interop binview offset (#28623)
• Fix duplicate hive values in hive pre-partition rewrite (#28621)
• Double-counted offset for Arrow export of sliced Series(Array) (#28602)
• Struct/List/Array equality handling in sort and join (#28601)
• Use try_new in StructArray construction in polars-json (#27489)
• Raise ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX (#27892)
• Run type coercion pass on pivot's internally generated group_by (#27897)
• More careful slice pushdown into joins (#28578)
• Preserve ordering in sliced unions (#28576)
• Drop input sortedness when casting to a string (#28574)
• Clear sortedness flags in StringChunked substring kernels (#28573)
• Bad mask handling when reading optional parquet column (#28547)
• Any operation on Unknown(Int) and Unknown(Float) should result in Unknown(Float) (#28545)
• Flip nulls_last after Expr.reverse() (#28572)
• Fix high blocking thread use in sink_parquet with async local path (#28543)
• Serialize LazyFrames backed by bytes (#28568)
• Release GIL in SQLContext.execute() (#28549)
• Propagate nulls_last in function_expr_sortedness (#28544)
• Incorrect slicing when a join requires sorting (#28541)
• Fix panic on self join of scan_delta / scan_iceberg (#28468)
• Decrement ParkGroup worker count when worker exits (#28492)
• Add missing Send bounds in Connector (#28497)
• Fix UB on first/last_non_null with empty chunk (#28495)
• Do not CSE non-column height expr on streaming engine (#28480)
• Propagate adjusted sortedness through Gather (#28493)
• Deduplicate the expression equality checks (#28483)
• Address 3VL issue with SQL NOT IN interaction with NULL values and joins (#28484)
• Use total comparison in Expr literal values (#28482)
• Ensure SQL SUM and CORR aggregates return NULL for all-null inputs, add TOTAL (#28475)
• Share null_count_dtype helper between Delta and Iceberg, fixing SchemaError (#28479)
• Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
• Fix dropped slice on multiple unions (#28477)
• Fix in-memory engine incorrect slice on maintain order join (#28478)
• Check join schema by position (#28455)
• Block predicate pushdown past overwritten window keys (#28429)
• Avoid panic when union slice skips all rows (#28420)
• Solve panic in dt.replace when there were multiple chunks (#28437)
• Propagate is_scalar from the input to the output of .sort() and .sort_by() (#28438)
• Insert missing coercions from Unknown(_) in list/array arithmetic (#28411)
• Resolve CSV column names overwrite in DSL->IR conversion (#28383)
• Avoid IEJoin rewrite for Categorical comparisons (#28427)
• Do not rewrite sort().reverse() to sort(descending=True) when maintain_order=True (#28403)
• Propagate bigidx to polars-plan from polars (#28396)
• Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
• Invalid offset in strptime (#28388)
• Incorrect schema type for decimal <-> primitive division (#28373)
• Panic in in-memory CSEE handling (#28371)
• Apply same type coercion to IsBetween as binary comparisons (#28300)
• Fix offset in arrow ffi export of sliced struct arrays (#28369)
• Fix cross filter not applied with sink and CSE (#28297)
• Fix panic on projection pushdown with caches (#28280)
• Fix write_json() null values in Array columns being written incorrectly as null (#28330)
• Fix EntityTooSmall on sink_ipc to S3 (#28255)
• Raise error instead of silent wrapping for select(len()) (#28355)
• Float16 groupby aggregates (#28361)
• Respect lexical ordering of Categorical in top_k/bottom_k (#28359)
• Support dtype-list schema overrides in scan_csv (#28226)
• Resolve unknown type in fused multiply add (#28321)
• Honor aws_checksum_algorithm when provided (#28328)
• Fix regression in sourcing HF_TOKEN (#28317)
• Raise on addition/subtraction between temporal and non-temporal Series (#28266)
• Rewrite with_columns on empty df to select during DSL -> IR (#28298)
• Fix sink deadlock regression (#28296)
• Avoid panic when consumer of vertical concat is done (#28272)
• Ensure predicate simplification is deterministically ordered (#28269)
• Don't attribute untimed execution to optimization in profile (#28273)
• Harden IPC mmap API (#28258)
• Classify S3 region request failures as IO errors (#28225)
• Incorrect inner dtype in ArrayChunked::from_aligned_values (#28249)
• Unsoundness in arg_sort_ascending (#28245)
• Add column length check to row encoding (#28248)
• Raise on implode of Object dtype instead of creating invalid List(Object) (#28246)
• Don't panic in qcut on degenerate float input (all-NaN, infinite, include_breaks) (#28220)
• Fix streaming Strptime feature flags (#28242)
• Return null not 0 for the first unbiased ewm_var/ewm_std value (#28235)
• Use after free in file cache (#28236)
• Incorrect length calculation in parallel hashmap (#28227)
• Avoid slice pushdown panic on shared cache inputs (#28157)
• Incorrect dtype for Unknown right literal in arithmetic (#28184)
• Remove unnecessary unsafe Python object downcast (#28189)
• Use-after-free in numpy ufunc FFI (#28195)
• Use-after-free in plugin registry (#28194)
• Unpickling for pl.when().then() (#28165)
• Reduce unknown casts (#28145)
• Consider a rolling_rank position invalid if ost.len() < min_samples (#28142)
• Incorrect broadcast with empty len in pl.concat_list (#28134)
• Incorrect sorted fast-path returning nan for grouped max (#28129)
• Fix panic on scan_parquet filter of fixed-size binary column (#28122)
• Fix decimal dynamic float cast (#28126)
• Fix pyiceberg key dot HDFS prefix filter (#28109)
• Fix projection pushdown panic on select(len()) after groupby (#28108)
• Close ResourceWarning leaks in database tests (#28066)
• Bad array len in UnionArray (#28079)
• Avoid panic on temporal extraction for datetime columns with nulls (#28054)
• Prevent panic in replace when old/new contain Expr or object dtype values (#27433)
• Fix qcut panic and incorrect bins on NaN input (#28020)
• Remove TrustedLen from Scan (#28055)
• Add a deprecation warning for .explode() without empty_as_null argument (#28040)
• Fix select(len()) incorrectly returned 0 when using scan_iceberg with pyiceberg as reader override (#28044)
• Ensure slice expr always observes input order (#28041)
• Fix projection pushdown with row index out of order (#28043)
• Fix cross-join predicates not applied when using sink_* functions (#28042)
• Grammar corrections (recommend to → recommend, allows to → allows using) (#28038)
• Ensure optimisations aren't omitted by cached IR nodes (#28004)
• Computed GROUP keys in SQL projections (#27989)
• Handle empty DataFrames in dt.truncate and dt.round (#26120)
• Address join alias state leakage in SQL CTEs (#27985)
• Support scanning file:// URIs with percent-encoded paths (#27876)
• Lower SQL [NOT] IN (subquery) to semi/anti join (#27888)
• Check for continuity across morsels in AsOf join node (#27461)
• Add test for projection pushdown with hconcat regression (#27937)
• Traverse caches only once in explain (#27939)
• Preserve null placement when collapsing sort.reverse() into single sort (#27918)
• Incorrect schema order for hstack in projection pushdown optimizer (#27920)
• Fix duplicated map_elements warning on streaming engine (

Note truncated.

🏆 Highlights

• Add LazyFrame.gather (#27501)
• Nested common subplan elimination (#27340)
• Stabilize streaming engine (#27497)
• Speed up parquet metadata decode with hand-written Thrift (#27427)
• Add streaming support for grouped AsOf join (#27293)

🚀 Performance improvements

• Eliminate filters with contradictory predicates (#27775)
• Update to new jemalloc (#27797)
• Do not materialize ScalarColumn in Column split_at (#27782)
• Avoid materializing broadcast in array.shift (#27740)
• Avoid materializing broadcast list in list.sample(n) and list.sample(frac) (#27679)
• Adaptive size dispatch to hashset or radix sort + capacity-aware reset in agg_n_unique (#27719)
• Dispatch {list,arr}.{unique,n_unique,reverse} to group_by engine (#27278)
• Improve in-memory grouped non-null count (#27702)
• Factor shared conjuncts out of OR-of-ANDs predicates (#27627)
• Skip downloading IPC batches exceeding slice bounds (#27683)
• Faster Series::is_sorted for logical / non-primitive types (#27567)
• Avoid materializing broadcast list in list.shift (#27628)
• Optimise json_decode Datetime string parsing (#27559)
• Speed up to_numpy C-order via cache-blocked transpose (#27522)
• Optimize select(len()) for non-strict horizontal concat (#27516)
• Pushdown slices to inputs on left/right/full join (#27508)
• Don't infer CSV schema if schema is set (#27507)
• Nested common subplan elimination (#27340)
• Make is_in row-group pruning precise on null-containing haystacks (#27495)
• Don't do fused-multiply-add on scalars (#27479)
• List full fast path (#27477)
• Make is_in row-group pruning precise on multi-value lists (#27475)
• Add streaming GatherNode (#27465)
• Lower non-elementwise FunctionExprIR to ColumnarFunctionNode (#27462)
• Speed up parquet metadata decode with hand-written Thrift (#27427)
• Skip validity mask processing in __array_ufunc__ when no inputs have nulls (#27358)
• Create IR slice from expr slice pushdown (#27200)
• Add streaming support for grouped AsOf join (#27293)
• Avoid unnecessary rechunk when sorting already sorted DataFrame (#27264)
• Lower basic over() to streaming primitives (#27303)
• Lower drop_{nulls,nans} in streaming group_by aggregations (#27296)
• Lower entropy to streaming reductions (#27174)
• Add native streaming interpolate (#27185)
• Streaming strptime with format=None (#27056)
• Lower skew / kurtosis to streaming aggregations (#27176)
• Post apply pyarrow filter in Polars' engine instead of pyarrow (#27192)
• Optimize drop_nulls().{first,last}() to {first,last}(ignore_nulls=True) (#27187)
• Always process pyarrow scan in batches (#27183)
• Make cut output Enum and mark as elementwise (#27173)
• Remove unused expression sorts (#27075)
• Use delta stats for mixed hive and non-hive predicate pushdown (#27102)
• Take into account size per row in join sampling (#27098)
• Streaming is_first_distinct and unique(maintain_order=True) (#27052)
• Streaming cov and corr (#27008)
• Add sorted unique node to streaming engine (#26990)
• Ensure Expr.append is lowered in streaming engine (#27022)
• Collapse consecutive Sort nodes (#26965)
• Drop maintain_order=True requirement in sink_delta (#27007)
• Lower index_of to streaming engine (#26923)
• Streaming native backward_fill (#26967)
• Native streaming forward_fill (#26922)
• Drop unused filter column above cache (#26955)
• Optimize .replace() from a single value (#26948)
• Add a streaming range-join (#26790)
• Lower arg_{min,max} to streaming engine (#26845)
• Additional IR slice pushdown after filter pushdown (#26815)
• Streaming first/last on Enum through physical (#26783)
• Fast filter for scalar predicates (#26745)
• Allow SimpleProjection in streaming engine to rename (#26709)
• Streaming cloud download for scan_csv (#26637)
• Drop columns only needed for predicates after the predicate is applied (#26703)
• Run projection pushdown after predicate pushdown (#26688)
• Comparison literal downcasting (#26663)
• Add dynamic predicates for TopK (#26495)
• Increase minimum default parquet row group prefetch to 8 (#26632)
• Partial predicate conversion to PyArrow (#26567)
• Streaming cloud download for scan_ndjson / scan_lines (#26563)
• Grab GIL fewer times during Object join materialization (#26587)
• Improve CSV and NDJSON cloud sink performance (#26545)
• Tune cloud writer performance (#26518)
• Allow parallel InMemorySinks in streaming engine (#26501)
• Add streaming AsOf join node (#26398)

✨ Enhancements

• Expose fixed-size rolling window expressions in Python visitor (#27108)
• Expose IR::Scan hive parts in the python node visitor (#27829)
• Expose IRFunctionExpr::DynamicPred in the python visitor (#27616)
• Fix SchemaError using lazy HConcat->Sink (#27770)
• Add pinning and queuing logic to polars-ooc (#27791)
• Add tiered multi-file parquet metadata resolver (#27720)
• Cache and shuffle DNS for cloud object_store (#27659)
• Update to new jemalloc (#27797)
• Allow deeper expressions (#27768)
• Add is_inherently_nondeterministic helper for AExpr (#27687)
• Use true division for the / operator in Polars SQL (#27391)
• Add Rust backend for Expr.has_nulls (#27590)
• Add block_in_place to Polars' async executor (#27612)
• Stabilize float16 (#27607)
• Add Expr.is_empty (#27583)
• Add support for the SQL FILTER clause for aggregate functions, and STRING_AGG (#27564)
• Make parquet FileMetadata prunable for IR-plan dispatch (#27535)
• Broadcast scalar input for list.slice (#27487)
• Add LazyFrame.gather (#27501)
• Add null_on_oob in {Expr/Series}.gather (#27327)
• Stabilize streaming engine (#27497)
• Process batched arr.eval on overflow boundaries (#27496)
• Process batched list.eval on overflow boundaries (#27483)
• Print SLICED UNION in LazyFrame explain (#27467)
• Cargo deny (#27363)
• Add maintain_order parameter to merge_sorted (#27263)
• Add ignore_nulls to {list,arr}.{any,all} (#27186)
• Lock-free memory manager with spill-to-disk and fully OOC multiplexer (#26774)
• Add is_unique to list/array dtypes (#27290)
• Add pl.merge_sorted operating on multiple frames (#27014)
• Add fast_alloc feature flag, remove default_alloc (#27206)
• Add a GPU slot to OptFlags so we can control CSE (similar to streaming) (#27026)
• Allow group_by() without key exprs (#27141)
• Collapse consecutive Sort nodes (#26965)
• Use UUIDv7 for sink_iceberg directory name generation (#26958)
• Truncate large binary/utf8 Parquet statistics values (#26764)
• Error if PartitionBy path provider returns absolute path that does not begin with base path, or contains '..' (#26894)
• Support Delta deletion vectors in scan_delta (#26867)
• Support Decimal32/64 in scan_parquet (#26941)
• Support casting Duration to String in ISO 8601 format (#26860)
• Add a streaming range-join (#26790)
• Support Expr for holidays in business day calculations (#26193)
• Parameter for pivot to always include value column name (#26730)
• Raise error in .collect_schema() when arr.get() is out-of-bounds (#26866)
• Extend Expr.reinterpret to all numeric types of the same size (#26401)
• Add missing_columns parameter to scan_csv (#26787)
• Clear no-op scan projections (#26858)
• Support nested datatypes for {min,max}_by (#26849)
• Support SQL ARRAY init from typed literals (#26622)
• Accept table identifier string in scan_iceberg() (#26826)
• Add a convenience make fresh command to the Makefile (#26809)
• Add unstable LazyFrame.sink_iceberg (#26799)
• Add maintain order argument on implode (#26782)
• Implement predicate pushdown for aliased groupby keys (#26597)
• Speed up casting primitive to bool by at least 2x (#26823)
• Enable rowgroup skipping for float columns (#26805)
• Add expression context to errors (#26716)
• Add Decimal support for product reduction (#26725)
• Support all Iceberg V2 arrow types in sink_parquet arrow_schema parameter (#26669)
• Re-work behavior of arrow_schema parameter on sink_parquet (#26621)
• Add contains_dtype() method for Schema (#26661)
• Implement truncate as a "to_zero" rounding mode (#26677)
• Expose AExpr::Rolling in the python visitor (#26715)
• More generic streaming GroupBy lowering (#26696)
• Add basic MemoryManager to track buffered dataframes for out-of-core support later (#26443)
• Add truncate Expression for numeric values (#26666)
• Better error messages for hex literal conversion issues in the SQL interface (#26657)
• Add SQL support for LPAD and RPAD string functions (#26631)
• Support SQL "FROM-first" SELECT query syntax (#26598)
• Speed up any() and all() for nulls (#26615)
• Bump Chrono to 0.4.24, enabling stricter parsing of %.3f/%.6f/%.9f specifiers (#26075)
• Expose unstable assert_schema_equal in py-polars (#24869)
• Allow parsing of compact ISO 8601 strings (#24629)
• Streaming cloud download for scan_ndjson / scan_lines (#26563)
• Configuration to cast integers to floats in cast_options for scan_parquet (#26492)
• Add escaping to quotes and newlines when reading JSON object into string (#26578)
• Standardise on RFC-5545 when doing datetime arithmetic on timezone-aware datetimes (#26425)
• Support sas_token in Azure credential provider (#26565)
• Expose HConcat options in the python node visitor (#26551)
• Relax SQL requirement for derived tables and subqueries to have aliases (#26543)
• Add polars-config and pl.Config.reload_env_vars() (#26524)
• Record path for object store error raised from sinks (#26541)
• Use CRC64NVME for checksum in aws sinks (#26522)
• Add get() for binary Series (#26514)
• Add streaming AsOf join node (#26398)

🐞 Bug fixes

• Fix skip_batches not handling negation of bool dtype with None values (#27452)
• Use block_in_place_on for calls which can come from executor thread (#27855)
• Mismatch in max_threads -> pipeline configuration (#27854)
• Keep maintain_order on sliced unique (#27852)
• Fix SchemaError using lazy HConcat->Sink (#27770)
• Fix incorrect projection height when selecting only literals (#27825)
• Fix rolling aggregations with window_size=0 (#27812)
• Select with expr slice and len gave incorrect len (#27824)
• Prevent import panic when environment variable set to unexpected value (#27831)
• Broken link to AI Policy corrected (#27793)
• Update to new jemalloc (#27797)
• Swap PlHashMap for PlIndexMap to make Multiplexer insertion order stable (#27785)
• Compare length for inline slice as usize (#27779)
• Raise length mismatch in multiple sort_by in group_by (#27772)
• Respect min_samples for rolling_by ops with nulls (#27706)
• Fix memory usage regression affecting TPCH Q22 (#27758)
• Add POLARS_ALLOW_NESTED_CSPE env var and make nested CSPE opt-in (#27765)
• Post-apply residual pyarrow predicates (#27764)
• Fix loss of precision for smaller floating types(#27662) (#27732)
• Filter at scan dropped in CSPE filter pushdown (#27763)
• Fix portstate assertion error on is_in (#27757)
• Fix incorrect when/then after forward fill / reverse in groupby (#27745)
• Accept empty Thrift list encoded as bare 0x00 byte in parquet metadata (#27754)
• Stabilize object store credentialprovider cache key (#27712)
• Panic in scan of empty IPC with slice (#27708)
• Persist object_store rebuild state in cache (#27707)
• Sort flag on GroupsType only applies to first element (#27684)
• Invalid unwrap_unchecked when length isn't exact (#27685)
• Logic error in async executor block_in_place (#27698)
• Don't unwrap channel send in streaming join_asof (

Note truncated.