polars

⚠️ Deprecations

• Fix group keys in partition_by(as_dict=True) / GroupBy.__iter__ in some cases (#13646)
• Rename row_count_name/row_count_offset parameters in IO functions to row_index_* (#13563)
• Deprecate dt.datetime in favor of dt.replace_time_zone(None) (#13520)
• Rename with_row_count to with_row_index (#13494)
• Deprecate Expr.where in favor of filter (#13440)
• Allow drop with no inputs as a no-op (#13460)

🚀 Performance improvements

• elide parallelism restriction on generic rolling expressions (#13662)
• ensure time groups are parallelized (#13660)
• do not eagerly compute bitcount (#13562)
• optimise SQL engine string concat (#13499)
• Refactor expression parsing logic of predicates/constraints (#13468)
• Represent Enum categories as Series (#13434)
• remove lifetime requirement from CategoricalChunkedBuilder (#13319)

✨ Enhancements

• write parquet ColumnOrder (#13672)
• Impl contains for ArrayNameSpace (#13638)
• improve rolling() expression formatting (#13657)
• Implement is_between in Rust (#11945)
• Add base PolarsError and PolarsWarning class (#13615)
• typing overloads for Series operator methods ge, gt, ... (#13167)
• Expressify pattern of str.extract (#13607)
• Impl join for ArrayNameSpace (#13586)
• add SQL engine support for string cast to json (#13624)
• add SQL engine support for EXTRACT and DATE_PART (#13603)
• Allow drop with no inputs as a no-op (#13460)
• add SQL engine support for POSITION and STRPOS (#13585)
• additional multi-column support for pl.<function> entries (#13336)
• is_in support for array dtype (#13559)
• add new str.find expression, returning the index of a regex pattern or literal substring (#13561)
• Impl and dispatch arr.first/last to get (#13536)
• Implement from_dataframe natively (interchange protocol) (#10701)
• add SQL engine support for LIKE and ILIKE pattern matching (#13522)
• improve hive partition pruning (#13358) (#13426)
• Add compact syntax for int_range starting from 0 (#13530)
• don't rechunk by default in lazy scans (#13518)
• Add cum_count expression function (#13478)
• add SQL engine support for IF control flow function (#13491)
• add SQL engine support for MOD function (#13502)
• return datetime for datetime mean & median (#13417)
• add SQL engine support for CONCAT_WS string function (#13483)
• Allow map_batches to auto-convert output NumPy arrays to Series (#13277)
• add SQL engine support for RIGHT and REVERSE string functions (#13461)
• implement BinaryView and Utf8View in polars-arrow (#13243)
• add SQL engine support for variadic string CONCAT function (#13428)
• add support for AND in SQL join-clause context (#13242)
• Impl ordering ops for array namespace (#13414)
• add SQL engine support for REPLACE string function (#13431)
• add SQL engine support for SIGN function (#13429)
• add SQL engine support for IFNULL function (#13432)
• additional SQL support for bytes, bit, and hex literals (#13389)

🐞 Bug fixes

• gather.get schema (#13679)
• Fix group keys in partition_by(as_dict=True) / GroupBy.__iter__ in some cases (#13646)
• ensure we hit proper cache in nested rolling expressions (#13666)
• Allow av_buffer cast numeric record to temporal type (#13661)
• streaming cross join if swapped is hit (#13656)
• Make sure rolling key is projected when process projection (#13622)
• fix schema inference for json (#13637)
• Improve parsing of inputs for Expr dunders (#13635)
• Empty series of AggregatedList should also have list dtype (#13620)
• Series.eq_missing should return an Expr when the input is an Expr (#13628)
• fallback to cast kernel if inline_cast AnyValue raise (#13595)
• Fix formatting in describe for precise quantiles (#13593)
• fix reverse variable row decoding (#13587)
• Fix scatter for null values (#13578)
• Fix cum_count with regards to start value / null values (#13535)
• Fix precision/scale handling and invalid numbers in string-to-decimal conversions. (#13548)
• Treat Python None as null value for Object dtype (#13564)
• Fix scatter to allow single temporal inputs (#13577)
• Fix interchange protocol data buffer dtype (#10787)
• Expr.replace to single value did not replace NULLs (#13551)
• improve hive partition pruning (#13358) (#13426)
• fix projection pushdown for new outer join schema (#13527)
• dont raise when partial function is passed to map_elements (#13524)
• improve reading of mixed string/other dtype column data from spreadsheets with openpyxl and pyxlsb engines (#13495)
• ensure size-hint of TrueIdxIter is correct (#13508)
• correct 'outer_coalesce' logic in case of duplicate names (#13501)
• raise for out-of-range datetimes in to_datetime/strptime (#13403)
• Fix Series equality for List/Array types (#13477)
• Keep logical type when getting values from list (#13456)
• Handle duplicate/ambiguous inputs for replace (#13217)
• Handle empty inputs to Enum constructor (#13446)
• Fix group_by iteration when grouping by certain selectors (#13437)
• Fix to_pandas for 0x0 dataframe (#13420)
• Fix offsets for numeric types in from_buffer (#13398)

📖 Documentation

• Clarify documentation for the agg_list argument in Expr.map_batches (#13625)
• fix linking to feature flags in user guide (#13644)
• bring sink_ndjson docstring in line with other sink docstrings (#13636)
• Update then and otherwise docstrings with "strings are parsed as column names" (#13630)
• Add sink_ndjson to API reference. (#13627)
• Improve documentation on broadcasting (#13394)
• Add note about toolchain issue under native Windows (#13590)
• Hint about ruff setting in VSCode (#13421)
• Clarify examples for .transpose() (#13581)
• Add additional Series docstring examples (#13558)
• Doc example for read_csv (#13161) (#13545)
• Add more doc examples on how to create an index column (#13532)
• update SQL section of the README (#13529)
• Add note to int_range docs for creating an index column (#13516)
• add a note to the read_database_uri docstring about escaping special characters in the connection string (#13514)
• update polars-business > polars-xdt link (#13509)
• Fix various typos, grammar and formatting in docstrings and user guide (#13506)
• Doc examples for threadpool_size and get_index_type (#13496)
• Add missing datetime examples to docs (#13487)
• add polars-distance to plugins page (#13454)
• define file-like object in read_parquet docstring (#13463)
• Move Series.struct.json_encode to methods in Sphinx autosummary (#13443)
• Add missing examples of series/list.py (#13423)
• show datetime.date import in code block (#13419)
• clarify documentation for rle and rle_id (#13397)
• use named series in Series.plot example (#13407)
• fix alphabetical order of documentation entries (#13396)

🛠️ Other improvements

• Auto-add 'needs triage' label to bugs (#13671)
• make rolling index column visible to optimizer (#13658)
• Enable new error message lint to improve stack trace display (#13596)
• Add Documentation / Build system sections to the changelog (#13594)
• Filter unhelpful messages in make build (#13579)
• Remove extra line break between checkboxes in GitHub bug report issues (#13576)
• Narrow type hint for get_index_type util (#13556)
• Fix some test failures/slowdowns (#13504)
• pandas 2.2 compat (#13467)
• Increase timeout for gevent async test (#13448)
• Do not end docstrings with a blank line (#13193)

Thank you to all our contributors for making this release possible! @Bromeon, @MarcNuebel, @MarcoGorelli, @ShivMunagala, @Wainberg, @aaarrti, @alexander-beedie, @bchalk101, @c-peters, @cgevans, @cmdlineluser, @collinprince, @deanm0000, @hamishs, @henryharbeck, @ion-elgreco, @jcrozum, @mcrumiller, @nameexhaustion, @orlp, @petrosbar, @r-brink, @reswqa, @ritchie46, @s-banach, @shritesh, @stinodego, @tim-stephenson and @wjandrea

🏆 Highlights

• add plot namespace (which defers to hvplot) (#13238)

🚀 Performance improvements

• optimize set bit count (#13317)
• speed up .dt.truncate for large numbers of years (#13310)
• don't eagerly evaluate error branches (#13311)
• don't trigger internal borrwing in numpy memmap (#13304)
• don't needlessly allocate validity in concat/rechunk (#13288)
• add fast path to count_bits_set_by_offsets (#13253)
• make .dt.truncate('*mo') more than 3x faster (#13192)

✨ Enhancements

• support negative indices in gather in group_by context (#13373)
• validate Enum categories (#13356)
• improve Series/DataFrame init from existing Series/DataFrame objects (#13344)
• support negative indexing in gather (select context) (#13343)
• support min_periods for temporal rolling aggregations (#13342)
• support REGEXP and RLIKE pattern matching in SQL engine (#13359)
• emit suggestion for how to replace map_elements sigmoid function with expressions (#13347)
• Support Enum types in interchange protocol (#13368)
• add plot namespace (which defers to hvplot) (#13238)
• gracefully handle panics in plugins (#13329)
• rework pl.exclude as a pure selector, allowing other selectors as input (#13301)
• Implement unique/n_unique/unique_counts/is_unique/is_duplicated for Null series (#13307)
• support common variant spelling STDEV in the SQL engine (in addition to STDDEV) (#13303)
• enhance expression-level filter syntax with support for multiple predicates and kwargs (#12689)
• change doc links to new url docs.pola.rs (#13290)
• support horizontal concatenation of LazyFrames (#13139)
• Rename Utf8 data type to String, keep Utf8 as alias (#13257)
• dispatch strict_cast via cast (#13255)
• Impl any/all for array type (#13250)
• add cancellable queries (#13178)
• add offset parameter to gather_every (#13156)
• Support Array dtype AnyValue Series construction (#12817)
• Allow step parameter in int_ranges to take an expression (#13148)
• make python map_batches safer (#13181)
• Implement count for DataFrame/LazyFrame (#13153)

🐞 Bug fixes

• don't lose track of ones and zeros dtype, improve use with Array, raise error if dtype invalid (#13326)
• updated Display trait for enum categoricals (#13331)
• nested dtypes: export logical type in plugins (#13325)
• fix invalid dtype setting in array (#13327)
• fix csv parser error when commented-out rows precede the header row (#13318)
• invalid schema outer join after projection pd (#13315)
• invalid predicate optimization (#13313)
• Account for null values in categorical unique/n_unique (#13308)
• fix schema when subtracting (#13309)
• broadcasting of unit LHS in string operations (#12737)
• sorting categorical lexically bugs on null values (#13271)
• improve replace on categoricals (#13223)
• round trip to JSON and back should preserve Enum type (#13267)
• fix return type hint of list series any/all (#13265)
• sink_csv deadlock (#13239)
• Correctly use read_parquet for all binary inputs (#13218)
• is_in operator for categoricals (#13205)
• Better handle mismatched dtypes in replace (#13213)
• Fix replace fast path by casting old input to the right data type (#13176)
• ndjson nested null schema inference (#13206)
• don't cast to unknown dtypes (#13197)
• maintain old join behavior in window expression (#13179)

🛠️ Other improvements

• reverse condtion order in udfs _expr function (#13348)
• Update release workflow for new upload/download artifact versions (#13355)
• Allow construction of Series from memory buffers (#13323)
• add 'pipe littering' to 'coming from pandas' section (#13335)
• Refactor functionality related to Series buffers (#13291)
• Restore light/darkmode switch in API reference (#13312)
• Copy Makefile build commands to top level (#13293)
• Fix release flags (#13298)
• Re-enable consortium standard tests (#13296)
• Update CODEOWNERS (#13292)
• Add CPU compatibility check (#13134)
• Change base url of docs/guide to docs.pola.rs (#13281)
• Fix source link for dev docs (#13279)
• fix return type hint of list series any/all (#13265)
• Fix display of overloaded signatures (#13258)
• clean up bytecode parsing a bit (#13221)
• Add a couple of docstring examples to Series methods (#13244)
• remove unnecessary arg unpacking (#13241)
• update rustc (#13219)
• fix horizontal concatenation documentation (#13141)
• Replace blackdoc by ruff's new docstring formatter (#13182)
• Update ruff & ruff settings (#13126)
• Link to latest object_store docs in api doc (#13180)
• Fix failing test (#13171)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @TNieuwdorp, @adamreeve, @alexander-beedie, @c-peters, @cjfuller, @dependabot, @dependabot[bot], @mcrumiller, @nameexhaustion, @orlp, @petrosbar, @r-brink, @reswqa, @ritchie46, @robvanmieghem and @stinodego

🚀 Performance improvements

• don't needlessly allocate validity in concat/rechunk (#13288)
• add fast path to count_bits_set_by_offsets (#13253)
• make .dt.truncate('*mo') more than 3x faster (#13192)

✨ Enhancements

• change doc links to new url docs.pola.rs (#13290)
• support horizontal concatenation of LazyFrames (#13139)
• Rename Utf8 data type to String, keep Utf8 as alias (#13257)
• dispatch strict_cast via cast (#13255)
• Impl any/all for array type (#13250)
• add cancellable queries (#13178)
• add offset parameter to gather_every (#13156)
• Support Array dtype AnyValue Series construction (#12817)
• Allow step parameter in int_ranges to take an expression (#13148)
• make python map_batches safer (#13181)
• Implement count for DataFrame/LazyFrame (#13153)

🐞 Bug fixes

• sorting categorical lexically bugs on null values (#13271)
• improve replace on categoricals (#13223)
• round trip to JSON and back should preserve Enum type (#13267)
• fix return type hint of list series any/all (#13265)
• sink_csv deadlock (#13239)
• Correctly use read_parquet for all binary inputs (#13218)
• is_in operator for categoricals (#13205)
• Better handle mismatched dtypes in replace (#13213)
• Fix replace fast path by casting old input to the right data type (#13176)
• ndjson nested null schema inference (#13206)
• don't cast to unknown dtypes (#13197)
• maintain old join behavior in window expression (#13179)

🛠️ Other improvements

• Copy Makefile build commands to top level (#13293)
• Fix release flags (#13298)
• Re-enable consortium standard tests (#13296)
• Update CODEOWNERS (#13292)
• Add CPU compatibility check (#13134)
• Change base url of docs/guide to docs.pola.rs (#13281)
• Fix source link for dev docs (#13279)
• fix return type hint of list series any/all (#13265)
• Fix display of overloaded signatures (#13258)
• clean up bytecode parsing a bit (#13221)
• Add a couple of docstring examples to Series methods (#13244)
• remove unnecessary arg unpacking (#13241)
• update rustc (#13219)
• fix horizontal concatenation documentation (#13141)
• Replace blackdoc by ruff's new docstring formatter (#13182)
• Update ruff & ruff settings (#13126)
• Link to latest object_store docs in api doc (#13180)
• Fix failing test (#13171)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @TNieuwdorp, @adamreeve, @alexander-beedie, @c-peters, @cjfuller, @dependabot, @dependabot[bot], @mcrumiller, @orlp, @petrosbar, @r-brink, @reswqa, @ritchie46, @robvanmieghem and @stinodego

🚀 Performance improvements

• add fast path to count_bits_set_by_offsets (#13253)
• make .dt.truncate('*mo') more than 3x faster (#13192)

✨ Enhancements

• Rename Utf8 data type to String, keep Utf8 as alias (#13257)
• dispatch strict_cast via cast (#13255)
• Impl any/all for array type (#13250)
• add cancellable queries (#13178)
• add offset parameter to gather_every (#13156)
• Support Array dtype AnyValue Series construction (#12817)
• Allow step parameter in int_ranges to take an expression (#13148)
• make python map_batches safer (#13181)
• Implement count for DataFrame/LazyFrame (#13153)

🐞 Bug fixes

• sorting categorical lexically bugs on null values (#13271)
• improve replace on categoricals (#13223)
• round trip to JSON and back should preserve Enum type (#13267)
• fix return type hint of list series any/all (#13265)
• sink_csv deadlock (#13239)
• Correctly use read_parquet for all binary inputs (#13218)
• is_in operator for categoricals (#13205)
• Better handle mismatched dtypes in replace (#13213)
• Fix replace fast path by casting old input to the right data type (#13176)
• ndjson nested null schema inference (#13206)
• don't cast to unknown dtypes (#13197)
• maintain old join behavior in window expression (#13179)

🛠️ Other improvements

• Add CPU compatibility check (#13134)
• Change base url of docs/guide to docs.pola.rs (#13281)
• Fix source link for dev docs (#13279)
• fix return type hint of list series any/all (#13265)
• Fix display of overloaded signatures (#13258)
• clean up bytecode parsing a bit (#13221)
• Add a couple of docstring examples to Series methods (#13244)
• remove unnecessary arg unpacking (#13241)
• update rustc (#13219)
• fix horizontal concatenation documentation (#13141)
• Replace blackdoc by ruff's new docstring formatter (#13182)
• Update ruff & ruff settings (#13126)
• Link to latest object_store docs in api doc (#13180)
• Fix failing test (#13171)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @TNieuwdorp, @adamreeve, @alexander-beedie, @c-peters, @cjfuller, @dependabot, @dependabot[bot], @mcrumiller, @orlp, @petrosbar, @r-brink, @reswqa, @ritchie46, @robvanmieghem and @stinodego

🚀 Performance improvements

• ensure single expression evaluation for replace (#13147)
• drop the pyarrow conversion path in iter_rows; we can now do fully native conversion ~2-3x faster (#13122)

✨ Enhancements

• Move from GA to more privacy friendly framework (#13155)
• prune all/any_horizontals with single inputs (#13146)
• ensure we get cleaner logical plans with any/all_horizontal (#13144)

🐞 Bug fixes

• Fix comparison of categoricals (#13137)
• Use the name of the leftmost expression in horizontal operations (#13143)
• any_value should supports cast to boolean (#13125)
• Update offsets of null value correctly for all from_iter_xxx_trusted_len (#13132)
• fix neq for series cmp str (#13128)
• Fix off-by-one error in lit dtype determination for integers (#13129)
• fix category list builder append series with multiple chunks (#13116)

🛠️ Other improvements

• Fix release LTS CPU step (#13160)
• Use the name of the leftmost expression in horizontal operations (#13143)
• ensure we get cleaner logical plans with any/all_horizontal (#13144)
• Minor cleanup of PyO3 bindings (#13067)
• Update auto_explode param name to returns_scalar (#13119)
• Mark whether the current package is the LTS-CPU version (#13068)

Thank you to all our contributors for making this release possible! @alexander-beedie, @c-peters, @orlp, @reswqa, @ritchie46 and @stinodego

🐞 Bug fixes

• repeat_by should not raise if by contains nulls (#13105)
• [csv] raise on single quote char (#13104)
• Raise if scan zstd compressed csv file (#13102)
• allow timeunit-less dtype in pl.lit creation (#12997)
• Don't check map length if input is literal (#13098)
• rolling_quantile can get incorrect state (#13088)

🛠️ Other improvements

• Fix column name in contains_any example (#13090)
• update user-defined-functions for 0.19.x (#13071)
• Fix some links, and make map_batches warning more evident (#13081)
• Linting updates (#13069)
• take pl.concat out of StringCache context manager in "mismatched string cache" error message (#13076)
• add Enum to dtype list (#13080)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @mcrumiller, @reswqa, @ritchie46 and @stinodego

This version includes quite a few breaking changes. We are preparing for the 1.0 release and aim to make the upgrade from 0.20 to 1.0 as smooth as possible. Therefore, we prioritized getting any breaking changes in now rather than with 1.0.

Check out the upgrade guide for help navigating the upgrade to this version.

Please bear with us while we continue to make Polars the best tool it can be!

🏆 Highlights

• Add new Enum categorical data type which allows a fixed set of categories (#11822)

💥 Breaking changes

• Use Object Store instead of fsspec for read_parquet (#13044)
• Reimplement replace expression on the Rust side (#13002)
• Preserve left and right join keys in outer joins (#12963)
• Update update signature (#12986)
• Update Expr.count to ignore null values by default (#12934)
• Scheduled removal of previously deprecated functionality (#12885)
• Allow all DataType objects to be instantiated (#12470)
• Change value_counts resulting column name from counts to count (#12506)
• Change default join behavior with regard to nulls, add join_nulls parameter to keep existing behavior (#12840)
• Default to exact checking for integers in assertion utils (#12331)
• Set default dtype for Series to Null when no data is present (#12807)
• Update lit behavior for list/tuple inputs (#12559)
• Change DataType.is_nested from property to classmethod (#12453)
• Update constructors for Array and Decimal (#12837)
• Smaller integer data types for datetime components (#12070)
• Fix NaN ordering to make NaNs compare greater than any other float, and equal to themselves (#12721)

⚠️ Deprecations

• Rename write_database parameter if_exists to if_table_exists (#12783)

🚀 Performance improvements

• Avoid dispatching to expression engine for various Series methods (#13010)
• Elide allocation in outer join materialization (#12992)
• Avoid dispatching Series.head/tail to the expression engine (#12946)
• Ensure we reduce for any/all_horizontal (#12976)
• Add fast paths for UTC in truncate (#12965)
• Use select_seq for expression dispatch (#12962)
• Improve rolling_median algorithm (#12704)
• Use fast path for non-null data in new SQL-like null matching (#12874)
• Optimize DataFrame.iter_rows for smaller buffer sizes (#12804)
• Speed up initializing Series from a list of NumPy arrays (#12785)

✨ Enhancements

• Add str.contains_any and str.replace_many (Aho-Corasick algorithms) (#13073)
• Auto-infer credentials from .aws folder (#13062)
• Support private cloud S3 storage in scan_parquet (#13060)
• Use Object Store instead of fsspec for read_parquet (#13044)
• Avoid dispatching to expression engine for various Series methods (#13010)
• Allow order operators (<,>,>=,<=) on Enum types (#12982)
• Reimplement replace expression on the Rust side (#13002)
• Expand set of NumPy functions which emit inefficient map_* warning (#13039)
• Use tokio semaphore for concurrency handling (#13026)
• Improve and expressify hist (#13014)
• Update describe to use new count implementation (#12990)
• Add default to_struct Series name consistent with the usual default Series name (empty string) (#12998)
• Preserve left and right join keys in outer joins (#12963)
• Clarify "inefficient map_elements" warning message (#12978)
• Allow end before start in date/time_range (#12964)
• Update update signature (#12986)
• Minor update to Array data type repr (#12973)
• Implement group-tuples for Null dtype (#12975)
• Cast to an enum from int (#12954)
• Move categorical ordering into dtype (#12911)
• Avoid importing interchange module by default (#12927)
• Update Expr.count to ignore null values by default (#12934)
• Raise if expression passed as scalar to DataFrame constructor (#12916)
• Update repr of Struct data type class (#12922)
• Enable partial predicate pushdown past window expressions (#12710)
• Add merge mode to write_delta and remove pyarrow to delta conversions (#12392)
• Add str.reverse (#12878)
• Allow all DataType objects to be instantiated (#12470)
• Specific performance warnings from Rust to Python (#12802)
• Change value_counts resulting column name from counts to count (#12506)
• Implement std and var for Duration columns (#12865)
• Change default join behavior with regard to nulls, add join_nulls parameter to keep existing behavior (#12840)
• Enhance write_database return (indicate the number of rows affected by the operation) (#12830)
• Add dedicated Decimal selector (#12852)
• Preserve base dtype when raising to UInt power (#10446)
• Default to exact checking for integers in assertion utils (#12331)
• Improve __repr__ implementation for Expr (#12770)
• Support SQL subqueries for JOIN and FROM (#12819)

🐞 Bug fixes

• Fix off-by-one error in quantile(method="nearest") (#13058)
• Fix incorrect schema inference on nested columns (#13057)
• Don't raise for datetime_range if starting on ambiguous datetime and earliest was specified (#13050)
• Parse json_decode per max buffer length (#13029)
• Parse 00:00 time zone as UTC (#13034)
• Fix timeout errors in concurrent downloads (#13023)
• Streamline align_frames and fix edge-case where the identical frame object appears more than once (#13007)
• Fix SQL substring indexing (#13016)
• Allow broadcasting in ranges (#11900)
• Prevent deadlock in sink_csv (#12991)
• Don't get mutable if buffer is sliced (#12979)
• Support parameterized read_database calls against cursors that only take positional args (#12967)
• Fix truncate when truncating by multiple weeks (#12948)
• Fix segfault / memory corruption after plugins return Err result (#12953)
• Raise a proper python typed exception when IO writers try to write to an non existent folder (#12936)
• Don't panic when ambiguous parameter is not Utf8 (#12913)
• Raise a proper python typed exception when the CSV writer tries to write to an non existent folder (#12919)
• Patch rolling_var/rolling_std numerical stability (#12909)
• Fix incorrect Int16 min/max due to incorrect SIMD mask construction (#12908)
• Improve handling of decimal conversion with to_numpy in the absence of pyarrow (#12888)
• Fix OOB error in list set operations on empty frame (#12845)
• Fix error message for uninstantiated Enum types (#12886)
• Fix repr of Expr.gather (which was still showing deprecated take) (#12864)
• Fix Array dtype equality (#12853)
• Fix nan_min/max incorrectly aggregating chunks with addition (#12848)
• Revert type hint change on expression inputs (#12792)
• More accurate type hinting for collect_all functions (#12796)
• Use total float ordering in is_in (#12800)
• Handle aggregation for all-NaN groups in group_by (#12304)

🛠️ Other improvements

• Update version switcher for 0.20 (#12844)
• Add upgrade guide for Python Polars 0.20 (#12872)
• Run doctests before other tests (#13047)
• Update describe calculation of min/max (#13027)
• Minor typo fix (#13003)
• Resolve two interchange tests failing locally (#12999)
• Update outdated links to API in Expressions/Functions page (#12981)
• Expand docstrings for count (#12960)
• Fix issue with docs for group_by_dynamic (#12906)
• Prefer explicit --no-cov flag for py3.12/ubuntu test workflow (vs implicit/omitted) (#12889)
• Scheduled removal of previously deprecated functionality (#12885)
• Fix references in deprecation notes (#12877)
• Fix typo in hash docstring (#12879)
• Fix docstring for deprecated list.take (#12873)
• Note that list.take is deprecated (#12867)
• Fix failing tests (#12859)
• Add quotes to pip install with dependencies (#12799)
• Fix parameter name reference in update docstring #12797

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Object905, @Yerachmiel-Feltzman, @alexander-beedie, @c-peters, @ion-elgreco, @jankislinger, @mcrumiller, @nameexhaustion, @oli-clive-griffin, @orlp, @rancomp, @ritchie46, @romanovacca, @stinodego and @xuestrange

✨ Enhancements

• Parquet support required deltabyte encoding (#12836)

🐞 Bug fixes

• Fix incorrect values from parquet RLE decoding (#12818)
• Write only one dict page per row rowgroup (#12831)

Thank you to all our contributors for making this release possible! @nameexhaustion, @ritchie46 and @stinodego

✨ Enhancements

• support nested null in vstack/append/extend/concat (#12771)
• Improve error messages on attempted Arrow conversions involving incompatible/unknown dtypes (#12421)
• determine mode parallelism depending on current tasks (#12764)
• enable slice push down past with_columns (#12742)
• Improve write_database, accounting for latest adbc fixes/updates (#12713)

🐞 Bug fixes

• don't use streaming engine if aggregate is unknown (#12769)
• Enable special casing of sequence in list_to_struct (#12759)
• hold align_chunks_invariant (#12738)
• allow leading zero and plus in integer parsing (#12744)
• csv lines iter, always return remainder (#12739)
• fix oob in set operations (#12736)
• undo regression in ability to read certain parquet files (#12731)

🛠️ Other improvements

• Use latest atoi_simd release (#12748)
• Fix invalid references to xlsx2csv dependency (#12741)
• Remove pinned aiohttp dependency (#12733)

Thank you to all our contributors for making this release possible! @0siride, @PierreAttard, @RoDmitry, @alexander-beedie, @dependabot, @dependabot[bot], @eitsupi, @kszlim, @nameexhaustion, @orlp, @ritchie46 and @stinodego

✨ Enhancements

• Automatically wrap NumPy array as lit (#12709)
• Add DataFrame.iter_columns (#12653)
• favour showing "adbc_driver_manager" over "adbc_driver_sqlite" in show_versions (#12690)

🐞 Bug fixes

• corr return nan if denominator is invalid (#12708)
• parquet decimal statistics and schema (#12705)
• support append/extend with null series (#11824) (#12686)
• address a numpy ndarray init regression (#12701)
• fix carrying over infinity into other windows (#12685)

🛠️ Other improvements

• Update URI prefix in examples (prefer "postgresql" to "postgres") (#12707)
• now that scan_parquet supports hive partitioning, remove note pointing to scan_pyarrow_dataset (#12706)
• Minor docstring fixes (#12688)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @c-peters, @ritchie46, @stinodego and @tkarabela

⚠️ Deprecations

• Rename series_equal/frame_equal to equals (#12618)
• Rename map_dict to replace and change default behavior (#12599)

🚀 Performance improvements

• order(s) of magnitude speedup when initialising List dtype Series from 2D numpy array (#12672)
• improve merge_local_rhs_categorical traversal (#12660)
• make values_size estimate correct for sliced arrays (#12658)
• improve parquet utf8 validation (#12655)
• parquet pre-allocate buffer in binary plain encode (#12652)
• optimize dict binary decoding in parquet (#12648)
• ensure we only check the values within bounds (#12633)
• parquet; elide recursion in hot path (#12625)
• improve cov/corr algorithm (#12590)

✨ Enhancements

• Join operations on local categoricals (#12657)
• Implement PySeries.from_buffer for boolean buffers (#12654)
• Implement PySeries.from_buffer for numeric types (#12646)
• use RLE_DICTIONARY for integers in parquet (#12647)
• extend recent filter syntax upgrades to when/then construct (#12603)
• implement RLE_DICT encoding for utf8/binary columns (reduced parquet file size) (#12623)
• implement 'DeltaByteArray' decoding for parquet (#12602)

🐞 Bug fixes

• json null inference (#12677)
• cov/corr respect f32 type (#12676)
• fix ternary zip_with null broadcast (#12668)
• support negative slice on eager frame (#12644)
• fix concurrency budget assertion (#12641)
• fix oob in set operations (#12640)
• panic reading parquet nested struct column (#12614)
• Fix deprecation message for DataFrame.sum (#12619)
• features: performant,lazy,random (#12600)

🛠️ Other improvements

• Use range instead of np.arange in constructors (#12621)
• update custom allocator instructions to include macOS (#12593)

Thank you to all our contributors for making this release possible! @alexander-beedie, @c-peters, @cardoso, @dmitrybugakov, @nameexhaustion, @orlp, @ritchie46 and @stinodego

⚠️ Deprecations

• Rename str.json_extract to str.json_decode (#12586)

🚀 Performance improvements

• apply left side predicate pushdown also to right side on semi join (#12565)
• ensure streaming parquet download remains concurrent ~7x (#12552)

✨ Enhancements

• warn if by column is not sorted in rolling aggregations (as opposed to raising), add warn_if_unsorted argument (#12398)
• struct -> json encoding expression (#12583)
• Implement support for multi-character comments in read_csv (#12519)
• Implement LazyFrame.sink_ndjson (#10786)
• use JEMALLOC on all unix architectures (#12568)
• improve concurrency parameters (#12567)
• In explain(), rename PIPELINE to STREAMING so it's clearer what it means (#12547)

🐞 Bug fixes

• error when invalid list to array is given (#12584)
• parquet: do not extend existing nested that is already complete (#12569)
• accidental panic if predicate selects no files (#12575)
• fix lazy parquet slice with nested columns (#12558)
• ensure stats-evalutor exists (#12566)
• list schema of list eval (#12563)
• ensure concurrency budget never locks (#12555)
• Fix lazy schema for group_by_dynamic and rolling (#12551)
• address overflow on vec capacity calculation for int_ranges with negative step (#12548)

🛠️ Other improvements

• convert all recursive parquet deserialize to iterative (#12560)
• Minor cleanup in Expr class (#12549)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Qqwy, @alexander-beedie, @dmitrybugakov, @fernandocast, @gab23r, @itamarst, @nameexhaustion, @ritchie46, @stinodego and @uchiiii

🏆 Highlights

• Support Python 3.12 (#12094)
• make 1D numpy to polars conversion zero-copy for numeric data (#12403)

⚠️ Deprecations

• Rename DataFrame column index methods (#12542)
• Rename Series.set_at_idx to scatter (#12540)
• Deprecate Series.view (#12539)
• Rename cumulative functions cumsum -> cum_sum and similar (#12513)
• Rename take to gather (#12528)
• Add dedicated horizontal aggregation methods to DataFrame (#12492)
• Rename take_every to gather_every (#12531)
• Deprecate Series.inner_dtype property (#12494)
• Deprecate parse_int in favor of to_integer (#12464)
• Deprecate DataType method is_not (#12458)
• Deprecate Series methods is_boolean and is_utf8 (#12457)
• Add DataType.is_integer and other dtype groups (#12200)

🚀 Performance improvements

• speed up parquet download of streaming engine (#12544)
• speed up cov/corr with SIMD + strength-reduction ~3x 0.19.13/ ~2x numpy (#12471)
• apply predicates and statistics of parquet files in streaming mode (#12439)
• use online algorithm for cov/corr ~2x (#12412)
• make 1D numpy to polars conversion zero-copy for numeric data (#12403)

✨ Enhancements

• Add dedicated horizontal aggregation methods to DataFrame (#12492)
• support http scan_parquet (#12517)
• Add support for UTF-8 BOM option in write_csv and sink_csv (#12253)
• remove lexical (replace with atoi_simd, ryu, and itao). (#12512)
• more changes for versioned plugins (#12504)
• plugins add version and context (#12433)
• Add DataType.is_integer and other dtype groups (#12200)
• include i128 in more primitive functions (#12413)
• write rolling functions as private expressions. (#12379)

🐞 Bug fixes

• fix incorrect ternary agg states (#12538)
• fix and improve ternary evaluation on groups (#12529)
• saturating sub in debug msg (#12525)
• fix panic when writing Decimal type to parquet (#12532)
• pre-fefetch struct columns in async projection pd (#12514)
• rechunk cross join output in streaming (#12511)
• Ensure behaviour ofSeries comparison with timedelta matches that of other types (#12497)
• fix as_list logical types (#12507)
• fix streaming cross join on empty df (#12491)
• dont overflow when calculating date range over very long periods (#12479)
• Allow append/zip_with/extend on local categoricals (#12369)
• Do not panic if time is invalid (#12466)
• ensure explicit "return_dtype" is respected by map_dicts (#12436)
• empty csv no-raise (#12434)
• Fix scan_csv error type (#12355)
• binary operations in aggregation context on literals (#12430)
• raw HTML output alignment was incorrect for dtype in header (#12422)
• update groups state after binary aggregation (#12415)
• Remove extra \n when reading file-like object wi… (#12333)
• Issue correct PolarsInefficientMapWarning for lshift/rshift operations (#12385)
• revert ternary special broadcast, ensure broadcast is always to max height (#12395)
• ensure first/last return null if empty (#12401)

🛠️ Other improvements

• fix and improve ternary evaluation on groups (#12529)
• Add polars-ds to list of community plugins (#12527)
• Future-proof consortium standard test (#12524)
• add schema test (#12523)
• remove lexical (replace with atoi_simd, ryu, and itao). (#12512)
• add test for previous commit (#12510)
• Update polars-hash reference (#12505)
• Add note on hash stability and mention polars-hash (#12496)
• Support Python 3.12 (#12094)
• Improved import polars timing test; now much more consistent/reliable (#12478)
• Use .with_columns() in all .list namespace examples (#12475)
• update rustc (#12468)
• Fix docs trigger (#12449)
• Update for new maturin release (#12437)
• Remove 'experimental' tag for auto-structify setting (#12435)
• make "DataFrame" and "Series" case more consistent across docs/comments/errors (#12428)
• dprint/markdown link checker minor updates (#12409)
• Use manylinux_2_17 for building x86-64 wheel (#12408)
• Use manylinux 2.24 instead of 2.28 for compatibility reasons (#12397)
• use with_columns in is_in example, and fix some bullet points not rendering (#12383)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @abstractqqq, @alexander-beedie, @c-peters, @cmdlineluser, @hirohira9119, @ion-elgreco, @jerome3o, @nameexhaustion, @reswqa, @ritchie46, @stinodego and @uchiiii

🏆 Highlights

• improve join performance through radix partitioned join (#12270)

⚠️ Deprecations

• Rename write_csv parameter has_header to include_header (#12351)
• Deprecate _saturating in duration string language, make it the default (#12301)
• Switch args for Decimal and set default scale=0 (#12224)
• Rename dt.seconds to dt.total_seconds (likewise for days, hours, minutes, milliseconds, microseconds, and nanoseconds) (#12179)
• Deprecate DataFrame.as_dict positional input (#12131)

🚀 Performance improvements

• indexvec in group-by (#12371)
• Reduce allocations in hash join (#12368)
• Change concurrency parameters (#12321)
• Improve join performance through radix partitioned join (#12270)
• Remove extra multiplication in hash_to_partition (#12233)
• Allow non-power-of-two partitions (#12225)
• Reduce compute in error message for failed datetime parsing (#12147)

✨ Enhancements

• Updated BytecodeParser for Python 3.12 (#12348)
• Add round_sig_figs expression for rounding to significant figures (#11959)
• Change concurrency parameters (#12321)
• Deprecate _saturating in duration string language, make it the default (#12301)
• Auto-infer ambiguous for truncate and round (#12204)
• Allow construction of Datetime series from datetime.date array (#12175)
• New Config options for numeric formatting: digit grouping and thousands/decimal separator (#12099)
• Allow non-aggregation predicate in ternary groupby (#12286)
• Add name= in .write_avro to set schema name (#12255)
• Update write_delta to write large arrow types without casting (#12260)
• Add support for reading zstd compressed files (no-options) in read_csv (#12214)
• Start prefetching all files immediately (#12201)
• Expose more options to plugin registration (#12197)
• Add .list.to_array expression (#12192)
• Consolidate & improve all casting failure error messages (#12168)
• Add Binary dtype to hypothesis tests (#12140)
• Tunable concurrency (#12171)
• Support reverse sort in streaming (#12169)
• Add .arr.to_list expression (#12136)
• Support decimals in assert utils (#12119)
• Add concurrency budget (#12117)
• Improved support for use of file-like objects with DataFrame "write" methods (#12113)
• Introduce ignore_nulls for str.concat (#12108)

🐞 Bug fixes

• Do not cast lit if has same dtype (#12342)
• Fix index column name of rolling/dynamic group by (#12365)
• Ternary broadcasting with empty truthy or falsy and agg predicate (#12357)
• UInt64 should be correctly extracted from python object (#12338)
• Ignore IDE-mediated DeprecationWarning when debugging tests under 3.12 (#12343)
• expr_output_name include literal (#12335)
• Fix Decimal dtype table repr (#12318)
• Fix behavior of month intervals in date_range (#12317)
• Scan empty csv miss row_count (#12316)
• zip_with also broadcast mask (#12309)
• respect hive_partitioning flag when dealing with multiple files (#12315)
• parquet, add row_count to empty file materialization (#12310)
• Fix invalid DeprecationWarning generated from date_range defined with 'saturating' interval (#12311)
• fix download ranges in parquet (#12313)
• object store path derivation for local URL (#12308)
• don't move right endpoint of windows in rolling in default offset==-period case (#12267)
• Raise more informative error on invalid reshape input (#12288)
• incorrect super type for literals in nested binary exprs (#12238)
• typo in exception message (#12278)
• fix ambiguous aggregation type (#12269)
• return frames from read_excel in the originally specified order (#12243)
• Consistently propagate nulls for numpy ufuncs (#12212)
• respect return_scalar of list scalars (#12251)
• fix plugins system on Windows (#12230)
• potential overflow (#12206)
• always start a new thread if the thread is already blocking (#12202)
• with_row_count should block predicate push down for lazy csv (#12187)
• rechunk failed-list series before iterate (#12189)
• Fix interchange protocol boolean buffer size (#12177)
• fix incorrect desc sort behavior (#12141)
• take should block predicate pushdown (#12130)
• use null type when read from unknown row (#12128)
• boundary predicate to block all accumulated predicates in push down (#12105)
• make python schema_overrides information available to the rust-side inference code when initialising from records/dicts (#12045)
• fix panic when initializing Series with array of list dtype (#12148)
• Fix schema of arr.min/max (#12127)
• ensure filter predicate inputs exist in schema (#12089)
• Update null_count after arithmetic (#12280)

🛠️ Other improvements

• Workaround for maturin issue (#12370)
• Fix incorrect boundary column name in group_by_dynamic docstrings (#12366)
• Fix typo in rolling_* docstrings (#12362)
• Fix ruff linting invocation (#12350)
• Clean up conversion utils (#11789)
• Organize Cargo.toml (#12323)
• Consolidate "getting started" and "user guide" sections (#12246)
• Minor updates to prepare for Python 3.12 support (#12314)
• Move script for testing map warning (#12306)
• simplify expr checking in predicate push down (#12287)
• Remove external link (#12223)
• Fix rebase issue breaking CI (#12296)
• Add top-level make clippy, simplify Rust linting workflows (#12290)
• ensure we git-ignore ALL .venv dirs (#12289)
• incorrect super type for literals in nested binary exprs (#12238)
• Remove recommended setting from IDE docs (#12275)
• Clean up Python test workflow (#12261)
• clarify contains selector (#12265)
• Add py-polars to Cargo workspace (#12256)
• Use .with_columns in some docstrings (#12250)
• Add test for scan_csv plus slice (#12239)
• Fix emphasis formatting in docstring (#12240)
• Fix emphasis formatting in docstring (#12237)
• add deprecation notices to the docs for expressions moved into the new name namespace (#12236)
• update Cargo.lock (#12226)
• make sort test work with unstable sort (#12221)
• Build Python wheels on manylinux_2_28 (#12211)
• Include rust-toolchain.toml with sdist/wheels (#12184)
• Standardize project name formatting across docs (#12185)
• Update sqlparser to 0.39 (#12173)
• pin ring (#12176)
• Improve strip_{prefix, suffix} & strip_chars_{start, end} (#12161)
• Fix tests for pyarrow 14 (#12170)
• Fix rendering of note in DataFrame.fold (#12164)
• Fix triggers for docs deployment (#12159)
• Refactor some tests (#12121)
• Consolidate contributing info (#12109)
• Fix typo in user-guide/expressions/plugins.md (#12115)
• Render docstring text in single backticks as code (#12096)
• use more ergonomic syntax in select/with_columns where possible (#12101)
• Update CODEOWNERS (#12107)
• visualize plugin directory layout in user guide (#12092)
• Minor tweak in code example in section Expressions/Aggregation (#12033)
• Minor tweak in code example in section Expressions/Missing data (#12080)
• Minor improvements to the docs website (#12084)

Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Priyansh121096, @alexander-beedie, @cmdlineluser, @daviskirk, @dependabot, @dependabot[bot], @dgilman, @hirohira9119, @ion-elgreco, @jrycw, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @owrior, @rancomp, @reswqa, @ritchie46, @rob-sil, @stefmolin, @stinodego and @wsyxbcl

⚠️ Deprecations

• Deprecate DataFrame.as_dict positional input (#12131)

🚀 Performance improvements

• Reduce compute in error message for failed datetime parsing (#12147)

✨ Enhancements

• tunable concurrency (#12171)
• support reverse sort in streaming (#12169)
• Add .arr.to_list expression (#12136)
• Support decimals in assert utils (#12119)
• add concurrency budget (#12117)
• improved support for use of file-like objects with DataFrame "write" methods (#12113)
• Introduce ignore_nulls for str.concat (#12108)

🐞 Bug fixes

• fix incorrect desc sort behavior (#12141)
• take should block predicate pushdown (#12130)
• use null type when read from unknown row (#12128)
• boundary predicate to block all accumulated predicates in push down (#12105)
• make python schema_overrides information available to the rust-side inference code when initialising from records/dicts (#12045)
• fix panic when initializing Series with array of list dtype (#12148)
• Fix schema of arr.min/max (#12127)
• ensure filter predicate inputs exist in schema (#12089)

🛠️ Other improvements

• pin ring (#12176)
• Improve strip_{prefix, suffix} & strip_chars_{start, end} (#12161)
• Fix tests for pyarrow 14 (#12170)
• Fix rendering of note in DataFrame.fold (#12164)
• Fix triggers for docs deployment (#12159)
• Refactor some tests (#12121)
• Consolidate contributing info (#12109)
• Fix typo in user-guide/expressions/plugins.md (#12115)
• Render docstring text in single backticks as code (#12096)
• use more ergonomic syntax in select/with_columns where possible (#12101)
• Update CODEOWNERS (#12107)
• visualize plugin directory layout in user guide (#12092)
• Minor tweak in code example in section Expressions/Aggregation (#12033)
• Minor tweak in code example in section Expressions/Missing data (#12080)
• Minor improvements to the docs website (#12084)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Priyansh121096, @alexander-beedie, @dependabot, @dependabot[bot], @jrycw, @moritzwilksch, @nameexhaustion, @reswqa, @ritchie46, @stefmolin and @stinodego

⚠️ Deprecations

• Deprecate nans_compare_equal parameter in assert utils (#12019)
• Rename ljust/rjust to pad_end/pad_start (#11975)
• Deprecate shift_and_fill in favor of shift (#11955)
• Deprecate clip_min/clip_max in favor of clip (#11961)

🚀 Performance improvements

• improve parquet downloading (#12061)
• fix regression non-null asof join (#11984)
• drasticly improve performance of limit on async parquet datasets (#11965)

✨ Enhancements

• Add supertype for List/Array (#12016)
• enable eq and neq for array dtype (#12020)
• Expressify n of shift (#12004)
• add dedicated name namespace for operations that affect expression names (#11973)
• optimize asof_join and allow null/string keys (#11712)
• limit concurrent downloads in async parquet (#11971)
• sample fraction can take an expr (#11943)
• Add infer_schema_length to pl.read_json (#11724)

🐞 Bug fixes

• Fix get_index/iteration for Array types (#12047)
• improved xlsx2csv defaults for read_excel (#12081)
• str.concat on empty list (#12066)
• fix issue with invalid Mapping objects used as schema being silently ignored (#12027)
• improve ingest from numpy scalar values (#12025)
• binary agg should group aware if literal not a scalar (#12043)
• Use Arrow schema for file readers (#12048)
• Error on duplicates in hive partitioning (#12040)
• display fmt for str split (#12039)
• sum_horizontal should not always cast to int (#12031)
• fix apply_to_inner's dtype (#12010)
• Allow inexact checking of nested integers (#12037)
• Fix padding for non-ASCII strings (#12008)
• fix dot visualization of anonymous scans (#12002)
• SQL table aliases (#11988)
• fix streaming multi-column/multi-dtype sort (#11981)
• ensure streaming parquet datasets deal with limits (#11977)
• implement proper hash for identifier in cse (#11960)
• fix take return dtype in group context. (#11949)
• fix panic in format of anonymous scans (#11951)
• sql In should work without specific ops (#11947)
• construct list series from any values subject to dtype (#11944)

🛠️ Other improvements

• minor updates to lint-related dependencies (#12073)
• Add Excel page to user guide (#12055)
• Direct CONTRIBUTING to the docs website (#12042)
• Replace black by ruff format (#11996)
• Further assert utils refactor (#12015)
• Remove stacklevels checker utility script (#11962)
• Disable type checking for dataframe_api_compat dependency (#11997)
• Fix release tag (#11994)
• optimize asof_join and allow null/string keys (#11712)
• Add Development and Releases sections to the documentation (#11932)
• include the "build" dir when running make clean for docs (#11970)
• make cloning PyExpr consistent (#11956)
• fix take return dtype in group context. (#11949)
• warn about scan_pyarrow_dataset's limitations and suggest scan_parquet instead (if possible) (#11952)
• Add set_fmt_table_cell_list_len to API docs (#11942)

Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Rohxn16, @alexander-beedie, @braaannigan, @brayanjuls, @messense, @nameexhaustion, @orlp, @reswqa, @ritchie46, @squnit, @stinodego and @universalmind303

⚠️ Deprecations

• Deprecate shift_and_fill in favor of shift (#11955)
• Deprecate clip_min/clip_max in favor of clip (#11961)

🚀 Performance improvements

• fix regression non-null asof join (#11984)
• drasticly improve performance of limit on async parquet datasets (#11965)

✨ Enhancements

• optimize asof_join and allow null/string keys (#11712)
• limit concurrent downloads in async parquet (#11971)
• sample fraction can take an expr (#11943)
• Add infer_schema_length to pl.read_json (#11724)

🐞 Bug fixes

• fix streaming multi-column/multi-dtype sort (#11981)
• ensure streaming parquet datasets deal with limits (#11977)
• implement proper hash for identifier in cse (#11960)
• fix take return dtype in group context. (#11949)
• fix panic in format of anonymous scans (#11951)
• sql In should work without specific ops (#11947)
• construct list series from any values subject to dtype (#11944)

🛠️ Other improvements

• optimize asof_join and allow null/string keys (#11712)
• Add Development and Releases sections to the documentation (#11932)
• include the "build" dir when running make clean for docs (#11970)
• make cloning PyExpr consistent (#11956)
• fix take return dtype in group context. (#11949)
• warn about scan_pyarrow_dataset's limitations and suggest scan_parquet instead (if possible) (#11952)
• Add set_fmt_table_cell_list_len to API docs (#11942)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Rohxn16, @alexander-beedie, @messense, @orlp, @reswqa, @ritchie46, @squnit and @stinodego

⚠️ Deprecations

• Rename shift parameter from periods to n (#11923)
• Fix Array data type initialization (#11907)

🚀 Performance improvements

• support multiple files in a single scan parquet node. (#11922)

✨ Enhancements

• improve error handling in scan_parquet and deal with file limits (#11938)
• support multiple files in a single scan parquet node. (#11922)
• error instead of panic in unsupported sinks (#11915)
• upcast int->float and date->datetime for certain Series comparisons (#11779)

🐞 Bug fixes

• avoid integer overflow in offsets_to_groups when bigidx is enabled (#11901)
• read_csv for empty lines (#11924)
• raise suitable error on invalid predicates passed to filter method (#11928)
• Fix Array data type initialization (#11907)
• set null_count on categorical append (#11914)
• predicate push-down remove predicate refers to alias for more branch (#11887)
• address DataFrame construction error with lists of numpy arrays (#11905)
• address issue with inadvertently shared options dict in read_excel (#11908)
• raise a suitable error from read_excel and/or read_ods when target sheet does not exist (#11906)

🛠️ Other improvements

• Fix typo in read_excel docstring (#11934)
• Fix docstring for diff methods (#11921)
• fix some typos and add polars-business to curated plugin list (#11916)
• add missing 'diagonal_relaxed' to pl.concat "how" param docstring signature (#11909)

Thank you to all our contributors for making this release possible! @LaurynasMiksys, @alexander-beedie, @mcrumiller, @reswqa, @ritchie46, @romanovacca, @shenker, @stinodego and @uchiiii

⚠️ Deprecations

• Deprecate DataType.is_nested (#11844)

🚀 Performance improvements

• fix accidental quadratic behavior; cache null_count (#11889)
• fix quadratic behavior in append sorted check (#11893)
• optimise read_database Databricks queries made using SQLAlchemy connections (#11885)
• properly push down slice before left/asof join (#11854)

✨ Enhancements

• Introduce list.sample (#11845)
• don't require empty config for cloud scan_parquet (#11819)

🐞 Bug fixes

• use physical append (#11894)
• Add include_nulls parameter to update (#11830)
• recursively apply cast_unchecked in lists (#11884)
• recursively check allowed streaming dtypes (#11879)
• Frame slicing single column (#11825)
• fix project pushdown for double projection contains count (#11843)
• Propagate validity when cast primitive to list (#11846)
• Edge cases for list count formatting (#11780)

🛠️ Other improvements

• Further assert utils refactor (#11888)
• load 40x40 avatar from github and add loading=lazy attribute. (#11886)
• Fix Cargo warning for parquet2 dependency (#11882)
• Allow manual trigger for docs deployment (#11881)
• add section about plugins (#11855)
• fix incorrect example of valid time zones (#11873)
• fix typo in code example in section Expressions - Basic operators (#11848)
• Bump docs dependencies (#11852)
• add missing polars-ops tests to CI (#11859)
• Assert utils refactor (#11813)

Thank you to all our contributors for making this release possible! @Walnut356, @alexander-beedie, @dannyvankooten, @dependabot, @dependabot[bot], @ewoolsey, @jrycw, @mcrumiller, @nameexhaustion, @orlp, @reswqa, @ritchie46, @rjthoen, @romanovacca and @stinodego

🏆 Highlights

• extend filter capabilities with new support for *args predicates, **kwargs constraints, and chained boolean masks (#11740)

⚠️ Deprecations

• Deprecate non-keyword args for ewm methods (#11804)
• Deprecate use_pyarrow param for Series.to_list (#11784)
• Rename group_by_rolling to rolling (#11761)

🚀 Performance improvements

• Improve DataFrame.get_column performance by ~35% (#11783)
• rechunk before grouping on multiple keys (#11711)
• process parquet statistics before downloading row-group (#11709)
• push down predicates that refer to group_by keys (#11687)
• slightly faster float equality (#11652)

✨ Enhancements

• Expressify pct_change and move to ops (#11786)
• primitive kwargs in plugins (#11268)
• add DATE function for SQL (#11541)
• Add config setting to control how many List items are printed (#11409)
• Use OrderedDict for schemas (#11742)
• allow specifying schema in pl.scan_ndjson (#10963)
• add support for "outer" mode to frame update method (#11688)
• transparently support "qmark" parameterisation of SQLAlchemy queries in read_database (#11700)
• support multiple sources in scan_file (#11661)
• support batched frame iteration over read_database queries (#11664)
• column selector support for DataFrame.melt and LazyFrame.unnest (#11662)

🐞 Bug fixes

• ensure projections containing only hive columns are projected (#11803)
• patch broken aHash AES intrinsics on ARM (#11801)
• fix key in object-store cache (#11790)
• handle logical types in plugins (#11788)
• Fix values printed by assert_*_equal AssertionError when exact=False (#11781)
• make PyLazyGroupby reusable (#11769)
• only exclude final output names of group_by key expressions (#11768)
• Fix subsecond parsing in timedelta conversions (#11759)
• fix ambiguity wrt list aggregation states (#11758)
• Correctly process subseconds in pl.duration (#11748)
• use actual number of read rows for hive materialization (#11690)
• return float dtype in interpolate (for method="linear") for numeric dtypes (#11624)
• fix seg fault in concat_str of empty series (#11704)
• fix sort_by regression (#11679)
• Fix match on last item for join_asof with strategy="nearest" (#11673)

🛠️ Other improvements

• Bump lint dependencies (#11802)
• Minor updates to assertion utils and docstrings (#11798)
• Remove unused _to_rust_syntax util (#11795)
• Minor tweak in code example in section Coming from Pandas (#11764)
• Fix Exception module paths (#11785)
• Rename IntegralType to IntegerType (#11773)
• more granular polars-ops imports (#11760)
• Link to expand_selector in user guide (#11722)
• Add parametric test for df.to_dict/series.to_list (#11757)
• Minor fix in code example in section Coming from Pandas (#11745) (#11745)
• Move tests for group_by_dynamic into one module (#11741)
• Update group_by_dynamic example (#11737)
• reorder pl.duration arguments (#11641)
• remove default features from some crates (#11680)
• *_horizontal dependent on reduce_expr to expression architecture (#11685)
• clarify that median is equivalent to the 50% percentile shown in describe metrics (#11694)
• update rustc and fix future (#11696)
• Publish release after uploading assets (#11686)
• upgrade pyo3 to 0.20.0 (#11683)
• better align help command output following addition of some longer options (#11681)
• sum_horizontal to expression architecture (#11659)
• add note about use of polars-lts-cpu for macOS x86-64/rosetta (#11660)
• improve rank implementation, especially around nulls (#11651)

Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Walnut356, @aberres, @alexander-beedie, @alicja-januszkiewicz, @cmdlineluser, @jrycw, @mcrumiller, @messense, @nameexhaustion, @orlp, @petrosbar, @rancomp, @reswqa, @ritchie46, @romanovacca, @sd2k, @stinodego, @svaningelgem and @thomasjpfan

🏆 Highlights

• Enable additional flags for x86-64 wheels (#11487)

⚠️ Deprecations

• Rename .list.lengths and .str.lengths (#11613)
• Deprecate default value for radix in parse_int (#11615)
• Rename write_csv parameter quote to quote_char (#11583)

🚀 Performance improvements

• actually use projection information in async parquet reader (#11637)
• improve performance and fix panic in async parquet reader (#11607)
• use try_binary_elementwise over try_binary_elementwise_values (#11596)
• skip empty chunks in concat (#11565)
• improve sparse sample performance (#11544)

✨ Enhancements

• Standardize error message format (#11598)
• allow coalesce in streaming (#11633)
• Implement schema, schema_override for pl.read_json with array-like input (#11492)
• add SQL support for UNION [ALL] BY NAME, add "diagonal_relaxed" strategy for pl.concat (#11597)
• improve performance and fix panic in async parquet reader (#11607)
• add time_unit argument to duration, default to "us" (#11586)
• support read_database options passthrough to the underlying connection's execute method (enables parameterised SQL queries, etc) (#11562)
• elide overflow checks on i64 (#11563)
• add INITCAP string function for SQL (#9884)

🐞 Bug fixes

• Fix input replacement logic for slice (#11631)
• slice expr can be taken in cse (#11628)
• ensure nested logical types are converted to physical (#11621)
• correctly convert nullability of nested parquet fields to arrow (#11619)
• improve performance and fix panic in async parquet reader (#11607)
• normalize filepath in sink_parquet (#11605)
• parse time unit properly in pl.lit (#11573)
• expand all literals before group_by (#11590)
• fix as_dict with include_key=False for partition_by (#9865)
• mark take_group_last function as unsafe (#11587)
• handle unary operators applied to numbers used in SQL IN clauses (#11574)
• Align new_columns argument for scan_csv and read_csv (#11575)
• Add initialization support for python Timedeltas (#11566)
• incomplete reading of list types from parquet (#11578)
• respect identity in horizontal sum (#11559)
• bug in BitMask::get_u32 (#11560)
• take slice into account in parallel unions (#11558)
• correct schema empty df in hive partitioning read (#11557)
• ensure ListChunked::full_null uses physical types (#11554)
• respect 'hive_partitioning' argument in parquet (#11551)
• fix parquet deserialization Overflow error by using i64 offset types when promoting Arrow Lists to LargeLists (#11549)
• streamline is_in handling of mismatched dtypes and fix a minor regression (#11533)
• fix comparing tz-aware series with stdlib datetime (#11480)
• catch use of non equi-joins in SQL interface and raise appropriate error (#11526)
• rework SQL join constraint processing to properly account for all USING columns (#11518)

🛠️ Other improvements

• Improved user guide for cloud functionality (#11646)
• Improve some docstrings (#11644)
• Disable clippy lint "too many arguments" for py-polars (#11616)
• Make backwardfill and forwardfill function expr non-anonymous (#11630)
• Make all expr in dt namespace non-anonymous (#11627)
• Fix changelog for language-specific breaking changes (#11617)
• Make value_counts and unique_counts function expr non-anonymous (#11601)
• Make arg_min(max), diff in list namespace non-anonymous (#11602)
• Rename write_csv parameter quote to quote_char (#11583)
• improve struct documentation (#11585)
• Remove **kwargs from LazyFrame.collect() (#11567)
• use a generic consistent total ordering, also for floats (#11468)
• fix lints (#11555)
• Remove toolchain specification workaround (#11552)
• Trigger Python release from Actions workflow dispatch (#11538)
• Enable additional flags for x86-64 wheels (#11487)

Thank you to all our contributors for making this release possible! @ByteNybbler, @MarcoGorelli, @TheDataScientistNL, @alexander-beedie, @andysham, @c-peters, @jhorstmann, @mcrumiller, @nameexhaustion, @orlp, @reswqa, @ritchie46, @romanovacca, @stinodego and @svaningelgem

🏆 Highlights

• Postfix rolling expression as a special case of window functions. (#11445)
• Use IPC for (un)pickling dataframes/series (#11507)

🚀 Performance improvements

• early return in replace_time_zone if target and source time zones match (#11478)
• greatly improve parquet cloud reading (#11479)
• ensure we download row-groups concurrently. (#11464)

✨ Enhancements

• support left and right anti/semi joins from the SQL interface (#11501)
• Add left_on and right_on parameters to df.update (#11277)
• expressify peak_min/peak_max (#11482)
• IN(subquery) and SQL Subquery Infrastructure (#11218)
• add ODBC connection string support to read_database (#11448)
• postfix rolling expression as a special case of window functions. (#11445)
• allow for "by" column to be of dtype Date in rolling_* functions (#11004)
• rework ColumnFactory to additionally support tab-complete for col in IPython (#11435)

🐞 Bug fixes

• literal hash (#11508)
• Fix lazy schema for cut/qcut when allow_breaks=True (#11287)
• correct output schema of hive partition and projection at scan (#11499)
• correct projection pushdown in hive partitioned read (#11486)
• fix for write_csv when using non-default "quote" char (#11474)
• fix deserialization of parquets with large string list columns causing stack overflow (#11471)
• enable read_database fallback for Snowflake warehouses/connections that don't support Arrow resultsets (#11447)
• Fix SQL ANY and ALL behaviour (#10879)
• partially address some PyCharm tooltip/signature issues with decorated methods (#11428)
• address multiple issues caused by implicit casting of is_in values to the column dtype being searched (#11427)

🛠️ Other improvements

• minor changes in peak-min/max (#11491)
• align cloud url regex in rust and python (#11481)
• Test sdist before releasing (#11494)
• Unpin maturin version, fix release workflow (#11483)
• More release workflow refactor (#11472)
• Set some env vars for release (#11463)
• move repeat_by to polars-ops (#11461)
• upgrade to nightly-10-02 (#11460)
• Update contributing guide to include memory requirement (#11458)
• add missing docs entry for rolling (#11456)
• use with_columns in shift examples (#11453)
• Add wheels as assets to GitHub release (#11452)
• Build more wheels for polars-lts-cpu/polars-u64-idx (#11430)

Thank you to all our contributors for making this release possible! @ByteNybbler, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @mcrumiller, @orlp, @ritchie46, @romanovacca, @stinodego, @svaningelgem and Romano Vacca

🚀 Performance improvements

• don't load N metadata files when globbing N files (#11422)

🐞 Bug fixes

• raise on invalid sort_by group lengths (#11423)
• fix outer join on bools (#11417)
• fix categorical collect (#11414)
• fix opaque python reader schema (#11412)
• async parquet. (#11403)
• Fix edge-case where the Array dtype could (internally) be considered numeric (#11398)
• handle ambiguous datetimes in pl.lit (#11386)
• fix panic in hive read of booleans (#11376)

🛠️ Other improvements

• Split Python release into build / release jobs (#11421)
• Refactor Python release workflow (#11382)
• clarify use of "batch_size" for read_database (#11377)
• large windows runner for release (#11370)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @bowlofeggs, @c-peters, @jonashaag, @orlp, @ritchie46 and @stinodego

🚀 Performance improvements

• remove double memcopy (#11365)
• adress perf regression (#11354)

🐞 Bug fixes

• revert invalid runtime check (#11363)
• more cloud urls (#11361)
• ensure cloud globbing can deal with spaces (#11360)
• recognize more cloud urls (#11357)

🛠️ Other improvements

• Disable version warning banner for now (#11359)
• Fix error message reference to infer_schema_length (#11358)
• Mark some tests as slow (#11350)
• improve parametric tests for group_by_rolling by skipping overflowing cases (#11286)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @jonashaag, @orlp, @ritchie46 and @stinodego

🏆 Highlights

• support 'hive partitioning' aware readers (#11284)
• natively support reading parquet for aws, gcp and azure (#11210)
• Add support for Iceberg (#10375)
• The great expressification by @reswqa (#11320, #11344, #11313, #11257, #11288, #11275, #11197, #11167, #11155)

⚠️ Deprecations

• Add disable_string_cache (#11020)

🚀 Performance improvements

• improve dynamic_groupby_iter (#11341)
• improve and fix rolling windows by linear scanning (#11326)
• faster init from pydantic models that have a small number of fields, and support direct init from SQLModel data (often used with FastAPI) (#11263)
• improve outer join materialization (#11241)
• use ryu and itoa for primitive serialization (#11193)
• use try-binary-elementwise instead of try-binary-elementwise-values in dt_truncate (#11189)
• Using cache for str.contains regex compilation (#11183)

✨ Enhancements

• introduce 'label' instead of 'truncate' in group_by_dynamic, which can take label='right' (#11337)
• Expressify list.shift (#11320)
• top_k and bottom_k supports pass an expr (#11344)
• add "pyxlsb" engine support to read_excel (for excel binary workbook files) (#11248)
• support 'hive partitioning' aware readers (#11284)
• str.strip_chars supports take an expr argument (#11313)
• sample n can take an expr (#11257)
• Add disable_string_cache (#11020)
• clip supports expr arguments and physical numeric dtype (#11288)
• Introduce list.drop_nulls (#11272)
• str.splitn and split_exact can take an expr argument by (#11275)
• introduce ambiguous option for dt.round (#11269)
• Adds NULLIF and COALESCE SQL functions (#11124)
• better tree-formatting representation (#11176)
• natively support reading parquet for aws, gcp and azure (#11210)
• Expressify str.strip_prefix & suffix (#11197)
• Add support for Iceberg (#10375)
• list.join's separator can be expression (#11167)
• argument every of datetime.truncate can be expression (#11155)

🐞 Bug fixes

• Fix Series.__contains__ for None values and implement is_in for null Series (#11345)
• don't panic on multi-nodes in streaming conversion (#11343)
• ensure trailing quote is written for temporal data when CSV quote_style is non-numeric (#11328)
• clarify has_validity docstring and fix several cases where the presence of a bitmask was used to incorrectly infer the existence of null values (#11319)
• fix empty Series construction edge-case with Struct dtype (#11301)
• DataFrame init from collections.namedtuple values (#11314)
• Exclude functools wrapper frames in find_stacklevel (#11292)
• set partitions independent of thread pool (#11304)
• address VSCode issue with autocomplete on selector expressions in editor/console (#11235)
• consume duplicates in rolling_by window (#11261)
• handle url encoded paths in objectpath creation (#11240)
• use POOL when writing csv (#11222)
• don't conflate saved Config JSON string with file path (#11098)
• is_in for bool evaluate has_false incorrectly (#11217)
• improve handling of database drivers that can return arrow data (#11201)
• fix nullable filter mask in group_by (#11207)
• replace n-th in filter (#11206)
• fix translation of Series-nested datetime/date values for scan_pyarrow predicates (#11195)
• address unexpected expression name from use of unary - or + operators (#11158)
• impl hash for more function expr (#11182)
• list.join's separator can be expression (#11167)
• Add some missing expr type hint for series (#11171)
• consistently use negative every as the default for offset in group_by_dynamic (#11164)
• Make pl.struct serializable (#11169)
• only raise on actual parameter collision when "dtypes" specified in read_excel "read_csv_options" (#11162)
• propagate null value for str/binary starts/ends_with and contains (#11141)

🛠️ Other improvements

• simplify/clarify group_by_dynamic examples (#11335)
• tighten assert_frame_equal for LazyFrames (don't collect until after the schema has been checked) (#11331)
• unify display for namespaced function expr (#11342)
• add lazy pivot example (#11325)
• Use GITHUB_TOKEN to get contributor information for docs (#11321)
• Enable version warning banner (#11322)
• cross-reference null_count from has_validity (clarifies the correct way to check for nulls) (#11323)
• Pin pydantic in dev requirements <2.4.0 (#11312)
• remove default auto-explode for map_many_private (#11270)
• Add type alias IntoExprColumn (#11296)
• update a few dependencies (#11283)
• Properly skip ADBC test (#11282)
• Fix some minor Makefile issues (#11276)
• update sponsors (#11271)
• parametric tests for group_by_rolling (#11262)
• Make some list function expr non-anonymous (#11230)
• Mention the performant feature only once (#11223)
• remove unneeded indirection (#11233)
• remove unneeded mutex around object-store (#11224)
• clarify every/period/offset in group_by_dynamic (#11175)
• Fix read_database batch_size docstring (#11132)

Thank you to all our contributors for making this release possible! @ByteNybbler, @Cheukting, @Fokko, @Hofer-Julian, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @billylanchantin, @jonashaag, @mcrumiller, @orlp, @ptiza, @reswqa, @ritchie46, @stinodego and @universalmind303

🏆 Highlights

• Polars plugins (#10924)

⚠️ Deprecations

• Rename is_first/last to is_first/last_distinct (#11130)
• Rename count_match to count_matches (#11028)
• Rename strip to strip_chars (#10813)
• Add datetime_range expression function (#10213)

🚀 Performance improvements

• optimize _unpack_schema() (#11080)
• optimize polars.utils._post_apply_columns() (#11086)
• optimize polars.utils._post_apply_columns() (#11041)
• optimize _unpack_schema() (#10960)
• improve performance of fast projection (#10945)

✨ Enhancements

• Expressify str.split argument. (#11117)
• Polars plugins (#10924)
• better async_collect (#10912)
• Expressify argument of binary contains (#11091)
• dt.offset_by supports broadcasting lhs (#11095)
• Expressify argument of binary starts_with and ends_with (#11076)
• add OpenOffice spreadsheet support via new pl.read_ods function (#11011)
• json_extract supports extract static and string value to list dtype (#11057)
• add quote_style="never" option for write_csv (#11015)
• Add literal for str count_match (#10996)
• More dtypes supports cast to list (#11025)
• Add strip_prefix and strip_suffix to the string namespace (#10958)
• improve read_excel table data identification (#10953)
• Add from_dataframe fast path and improve typing (#10979)
• add openpyxl as a new/optional engine for read_excel (#6183)
• Add datetime_range expression function (#10213)

🐞 Bug fixes

• Correct hash and fmt for struct expr (#11119)
• enforce sortedness of by argument in rolling_* functions (#11002)
• Make Series.__getitem__ raise an IndexError (#11061)
• Filter on empty objectChunked should not throw error (#11073)
• ensure null_count statistics accounts for null array (#11070)
• toggle off cse if ext_context is used (#11051)
• Correct field dtype of string concat (#11055)
• fix partial schema init with read_dicts and reduce latency of small-frame creation (#11047)
• pushed-down expr should be considered when evaluating ExternalContext (#11023)
• fix rolling_* functions when "by" has nanosecond resolution (#11005)
• Don't reuse member for Selector::Add (#11026)
• ensure series_equal properly accounts for dtypes when strict=True (#11012)
• fix the construction of List<Null> (#10969)
• write_excel "hidden_columns" parameter fails when taking a selector (#10987)
• allow singular null in regex pattern (#10948)
• compute length of null array in explode (#10946)

🛠️ Other improvements

• remove low contrast coloring from visited links (#11133)
• Ignore matplotlib warning (#11129)
• Do not run user guide examples by default (#11128)
• Ignore matplotlib mypy warnings (#11126)
• Add deprecation message in groupby docs (#11121)
• Removed duplicated example (#11109)
• Add CODEOWNERS for docs folder (#11107)
• Refactor starts_with and ends_with for string (#11085)
• Integrate user guide (#11089)
• remove mentions of the deprecated random module (#11087)
• simplify SchemaDefinition type alias (#11077)
• put fetch explanation in a "notes" block to better highlight it in the docs (#11058)
• remove feature gate join/groupby in polars-core (#10965)
• Add Documentation issue type (#11042)
• warn that "by" argument must be sorted for results to be correct in rolling_* functions (#11013)
• Adds missing method refs in LazyDataFrame API docs (#11027)
• Add lint for boolean trap (#11010)
• Add private LazyFrame method for setting sink optimizations (#10988)
• Enable a few more ruff lints (#10998)
• document polars string duration language in temporal range functions (#10978)
• Additional tests for interchange get_data_buffer (#10966)
• genericize PolarsDataType (#10952)
• Document that filter, drop_nulls, left join preserve order (#10955)
• add note about adbc flight sql driver (#10949)
• Revert pydantic >= 2.0.0 requirement (#10944)
• note that pl.duration represents fixed durations, point to offset_by for non-fixed (#10927)
• Test S3 functionality using moto server (#10164)

Thank you to all our contributors for making this release possible! @I8dNLo, @KacpiW, @MarcoGorelli, @Object905, @Qqwy, @TNieuwdorp, @alexander-beedie, @antoniocali, @bvanelli, @cjackal, @henrikig, @jakob-keller, @mrogowski11, @nameexhaustion, @orlp, @reswqa, @ritchie46, @s-banach, @stinodego, @svaningelgem and @thomasjpfan

🏆 Highlights

• Add syntactic sugar for col("foo") -> col.foo (#10874)

⚠️ Deprecations

• Rename Expr.is_not() to not_() (#10838)

✨ Enhancements

• allow individual Config options to be easily reset to their default value (#10922)
• accept expr in str.count_match (#10900)
• allow additional glimpse customisation, fix strings repr (#10895)
• accept expressions in .offset_by (#9967)
• support schema overrides for frames created from databases (#10884)
• Add syntactic sugar for col("foo") -> col.foo (#10874)
• support negative indexing in set_at_idx (#10891)
• implement drop as special case of select (#10885)
• raise a more helpful error when non-query statements passed to read_database (#10851)

🐞 Bug fixes

• Allow exactly one value in start/end for int_range (#10914)
• fix(rust, python): raise error when function didn't receive any inputs (#8635)
• count was falsy tagged as cse in group by (#10917)
• CSE don't accept opaque functions (#10905)
• Make int_range(s) exclusive on the upper bound when step is negative (#10898)
• don't overflow length before checking limit (#10883)
• fix bug where datetimes were not parsed in read_csv when pattern had no hour or minute (#10877)
• use pool in dataframe arithmetic (#10864)
• repair polars_err string interpolation (#10863)
• make count_match docs and extract_all docs/impl consistent around zero matches (#10854)

🛠️ Other improvements

• Set minimum version for pydantic to 2.0.0 (#10923)
• fix and clarify docs for Expr.map_elements (#10647)
• fix rendering of bullet points in dt.round (#10911)
• add test for 10875 (#10913)
• apply with_name in more places (#10899)
• never compare opaque functions (#10906)
• eliminate repetition in utf8 datetime functions (#10860)
• Fix issue templates for bug reports (#10896)
• request verbose logging output of minimal reproducable examples (#10882)
• add a note about read_database connection/cursor behaviour (#10873)
• introduce with_name for Series/ChunkedArray (#10859)

Thank you to all our contributors for making this release possible! @Barsik-sus, @MarcoGorelli, @alexander-beedie, @c-peters, @cmdlineluser, @dependabot, @dependabot[bot], @drgif, @jeroenjanssens, @orlp, @ritchie46, @stinodego and @wdoppenberg

💥 Breaking changes

• empty product returns identity and product ignores nulls (#10842)

✨ Enhancements

• add binary, boolean, categorical, date, object, and time selectors (#10806)
• Supports is_last operation (#10760)
• minor typing improvement for DataFrame.__iter__ (#10825)
• Add custom error for allow_copy=False (#10822)

🐞 Bug fixes

• empty product returns identity (#10842)
• never panic in hash/equality doesn't hold in cse (#10836)
• Improve bound checks on temporal ranges (#10837)
• var/std behavior around few elements (#10828)
• Fix divided by zero error when read empty csv in streaming mode (#10819)
• behaviour of reversed(df) (#10823)
• fix equality of quantile aggregation node (#10816)
• Reading an only-header csv file in streaming mode should not panic (#10810)

🛠️ Other improvements

• Refactor range related functions (#10830)
• map-related docstring updates (#10779)
• Move sink tests to streaming module (#10821)

Thank you to all our contributors for making this release possible! @alexander-beedie, @orlp, @reswqa, @ritchie46 and @stinodego

An upgrade guide is available on our website.

🏆 Highlights

• implementing sink_csv for LazyFrame (#10682)
• Support DataFrame init from queries against users' existing database connections (#10649)
• Rename groupby to group_by (#10656)

💥 Breaking changes

• return f64 for rank when method="average" (#10734)
• Update a lot of error types (#10637)
• Remove deprecated behavior from vertical aggregations (#10602)
• Read/write support for IPC streams in DataFrames (#10606)
• Change behavior of all - fix Kleene logic implementation for all/any (#10564)
• Improve consistency of parsing expression input (#9512)
• allow from_arrow to take a generator of RecordBatches, change error type to TypeError (#10529)
• remove fixed_seed and add pl.set_random_seed (#10388)
• Make arange an alias for int_range (#9983)
• date_range/time_range no longer return a List type (#10526)
• Remove various functionalities deprecated before 0.18 (#10527)
• Improve some error types and messages (#10470)

⚠️ Deprecations

• Rename map to map_batches (#10801)
• Rename GroupBy.apply to map_groups (#10799)
• Rename DataFrame.apply to map_rows (#10797)
• Rename Series/Expr.rolling_apply to rolling_map (#10750)
• Rename Series/Expr.apply to map_elements (#10678)
• Rename groupby to group_by (#10656)
• Deprecate some parameters of cut/qcut (#10484)

🚀 Performance improvements

• parse time zones outside of downcast_iter() in replace_time_zone (#10713)
• use binary abstraction for atan2 (#10588)
• use binary abstraction in pow (#10562)

✨ Enhancements

• activate cse for group_by (again) (#10749)
• implementing sink_csv for LazyFrame (#10682)
• Supports series unique & arg_unique & n_unique for list (#10743)
• repeat_by should also support broadcasting of LHS (#10735)
• deprecate 'use_earliest' argument in favour of 'ambiguous', which can take expressions (#10719)
• is_first also supports numeric list type. (#10727)
• improve slice pushdown in unions (#10723)
• Explicitly implement Protocol for interchange classes (#10688)
• Support min and max strategy for binary & str columns fill null (#10673)
• support broadcasting in list set operations (#10668)
• csv: add schema argument (#10665)
• Support DataFrame init from queries against users' existing database connections (#10649)
• add truncate_ragged_lines (#10660)
• supports cast to list (#10623)
• Update a lot of error types (#10637)
• preserve whitespace in notebook output (#10644)
• Remove deprecated behavior from vertical aggregations (#10602)
• support selector usage in write_excel arguments (#10589)
• Add LazyFrame.collect_async and pl.collect_all_async (#10616)
• Read/write support for IPC streams in DataFrames (#10606)
• propagate null is in is_in and more generic array construction (#10614)
• Change behavior of all - fix Kleene logic implementation for all/any (#10564)
• frame-level cast support (#10504)
• Improve consistency of parsing expression input (#9512)
• Add failed column to cast exception (#10507)
• allow from_arrow to take a generator of RecordBatches, change error type to TypeError (#10529)
• Remove deprecated get_idx_type - use get_index_type instead (#10556)
• Make arange an alias for int_range (#9983)
• date_range/time_range no longer return a List type (#10526)
• Remove various functionalities deprecated before 0.18 (#10527)
• Improve some error types and messages (#10470)
• suggest str.to_datetime instead of apply and stdlib strptime (#10266)

🐞 Bug fixes

• get_single_leaf can't handle Expr::Count (#10790)
• support groupby literal in streaming (#10771)
• ORDER BY on unselected columns (#10752)
• Fix is_in cannot cast list type for float (#10769)
• whitespace CSS in Notebook HTML updated to use pre-wrap instead of pre (#10739)
• only preserve sortedness flag in replace_time_zone when safe (#10738)
• Error on value_counts on column named "counts" (#10737)
• return f64 for rank when method="average" (#10734)
• Keep min/max and arg_min/arg_max consistent. (#10716)
• use time zone from dtype to overwrite output time zone when initialising Series (#10689)
• Cast small int type when scan csv in streaming mode. (#10679)
• raise exception with invalid on arg type for join_asof (#10690)
• Reused input series in rolling_apply should not be orderly (#10694)
• re-sort buffer when update window swap the whole buffer (#10696)
• Set the correct fast_explode flag for ListUtf8ChunkedBuilder (#10684)
• Sorted Utf8Chunked max_str and min_str should consider null value (#10675)
• Correctly handle time zones in write_delta (#10633)
• fix apply for empty series in threading mode (#10651)
• respect 'ignore_errors=False' in csv parser (#10641)
• fix rename + projection pushdown (#10624)
• fix int/float downcast in is_in (#10620)
• Change behavior of all - fix Kleene logic implementation for all/any (#10564)
• Fix serialization for categorical chunked. (#10609)
• Take input_schema to create physical expr for Selection (#10571)
• Clear window cache after evaluate predication expr (#10505)
• Parsing regex col in Expr::Columns (#10551)
• sanitize column naming in boolean ops (#10531)
• Fix write_delta with schema in delta_write_options (#10541)
• remove fixed_seed and add pl.set_random_seed (#10388)
• respect pl.Config options relating to shape, column names, and types when rendering HTML (#10449)

🛠️ Other improvements

• update cargo.lock (#10800)
• Create .venv in repo root (#10789)
• refactored write_database unit tests to properly separate concerns (#10773)
• Fix some broken links / formatting (#10772)
• Document chained when-then behaviour more prominently (#10759)
• Fix test failing due to new adbc release (#10763)
• Unpin connectorx and bump other Python dependencies (#10753)
• add note to testing docs about module import (#10741)
• Clear GitHub Actions caches weekly (#10715)
• Update for new pyarrow 13.0.0 behavior (#10691)
• Fix minor issue with sink_parquet docs (#10669)
• Remove deprecate_renamed_methods util (#10537)
• add "see also" entries to ne/eq_missing and update related examples (#10667)
• fix potential memory leak from usage of inspect.currentframe (#10630)
• give more relevant example for polars.apply (#10631)
• Bump ruff and enable new setting (#10626)
• Add docstrings for Expr.meta namespace (#10617)
• Enforce up-to-date Cargo.lock (#10555)
• deprecate DataFrame.replace (#10600)
• ensure that make requirements fully refreshes unpinned packages/deps (#10591)
• fix out-of-date explain default parameter (#10566)
• Fix expr_dispatch decorator to work on methods with decorators (#10549)
• Fix link to source code (#10542)
• Add title to index page (#10539)
• Disable SIM108 lint (#10519)
• Keep versioned docs (#10500)
• switch to pyo3/maturin-action (#10503)
• Update URLs for dev documentation (#10495)
• Skip failing test (#10496)
• Add version switcher to API reference (#10488)

Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Object905, @OndrejSlamecka, @SeanTroyUWO, @VasanthakumarV, @alexander-beedie, @aminalaee, @braaannigan, @c-peters, @ion-elgreco, @lorepozo, @marki259, @mcrumiller, @messense, @orlp, @owrior, @rben01, @reswqa, @ritchie46, @sdamashek, @stinodego, @svaningelgem, @titoeb, @trueb2, @washcycle and @zundertj

🐞 Bug fixes

• rollback cse in groupby: python 0.18.15 (#10491)

🛠️ Other improvements

• Mark import timing check as slow (#10487)
• Gather all streaming tests (#10485)
• Bump maturin to version 1.2.1 (#10479)

Thank you to all our contributors for making this release possible! @ritchie46 and @stinodego

🏆 Highlights

• Native implementation of dataframe interchange protocol (#10267)

⚠️ Deprecations

• Deprecate behavior of list/tuple inputs for lit (#10461)

🚀 Performance improvements

• optimise retrieval of values from df.item (~4-5x speedup) (#10411)
• pre-alloc int_ranges (#10399)
• use hash as CSE Identifier (#10385)

✨ Enhancements

• quote style option for csv writer (#10422)
• add "raise_if_empty" flag to read_excel, read_csv, scan_csv, and read_csv_batched (#10409)
• add use_earliest to to_datetime / strptime (#10426)
• add new "header_format" option for write_excel (#10392)
• {any/all}_horizontal to expression architecture (#10412)
• Native implementation of dataframe interchange protocol (#10267)
• allow unaligned pointers in arrow FFI (#10403)
• add line_terminator option to write_csv (#10373)
• add explicit selector variants for signed/unsigned integers (#10384)
• Add is_local and to_local to categorical namespace (#10372)
• enhance selectors expansion function, so it can operate on a schema as well as a frame (#10341)
• Order percentiles in describe (#10378)
• cse for groupby.agg and reduced cse collisions (#10381)
• improve take_every(0) exception (#10352)
• add offset and length to get_ptr (#10361)

🐞 Bug fixes

• fix pyarrow write_to_dataset wrt check_not_directory parameter (#10471)
• fix negative decimal parsing (#10444)
• ensure sorted_sink hash equals the default path (#10464)
• address inconsistency in init from square numpy arrays with/without an explicit schema (#10445)
• ensure last aggregation deals with default chunk (#10453)
• fix cse input schema (#10450)
• Fix by argument handling in join_asof (#10447)
• fix potential OverflowError in testing asserts with huge UInt64 diffs (#10437)
• Create delta compatible schema during writing (#10165)
• fix list groupby of array dtype (#10408)
• correct AnyValue::hash (#10391)
• finalize cast in partitioned groupby (#10359)

🛠️ Other improvements

• add vertical_relaxed example for pl.concat (#10472)
• Run all streaming tests on the same test runner (#10469)
• Organize OOC tests (#10463)
• add test for 10417 (#10420)
• Clean up some Sphinx settings (#10400)
• add test for 10401 (#10405)
• Address Ruff per file ignores (#10258)
• Small improvement for PySeries.get_buffer (#10363)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @OndrejSlamecka, @alexander-beedie, @c-peters, @cmdlineluser, @drgif, @ion-elgreco, @lfn3, @orlp, @potzenhotz, @rea1bacon, @reswqa, @ritchie46, @stinodego and @zundertj

⚠️ Deprecations

• Rename LazyFrame.read/write_json to de/serialize (#10238)
• Add categorical_as_str parameter to testing utils (#10350)

🚀 Performance improvements

• don't parallelize literal expressions (#10321)

✨ Enhancements

• support selectors in additional frame methods (#10255)
• Add Series.cat.uses_lexical_ordering (#10325)
• utility to get buffers and pointers (#10331)
• improve datetime parsing error message (#10332)
• add ptr for small integer types (#10330)
• add offsets utility (#10328)
• allow sequential runners in select/with_columns (#10322)
• warn about inefficient apply json.loads if json is local import (#10310)
• improve err msg parsing time, date, datetime (#10298)
• Add categorical_as_str parameter to testing utils

🐞 Bug fixes

• fix oob in 'last' (#10329)
• show inefficient apply warning in ipython (#10312)
• add cse to no_optimization in profile (#10317)
• fix categorical lexical sort (#10318)
• Fix join validation (#10257)
• Set correct dtype for .extract_groups() (#10306)

Thank you to all our contributors for making this release possible! @CanglongCl, @JulianCologne, @MarcoGorelli, @alexander-beedie, @cmdlineluser, @eltociear, @orlp, @ritchie46 and @stinodego

⚠️ Deprecations

• renaming approx_unique as approx_n_unique (#10290)
• Rename first qcut parameter to quantiles (#10253)
• Deprecate avg alias for mean (#10236)

🚀 Performance improvements

• fix O(n^2) in sorted check during append (#10241)

✨ Enhancements

• Add str.extract_groups (#10179)
• raise TypeError for all LazyFrame comparison operators (#10275)
• support bytecode translation to map_dict where the lookup key is an expression (#10265)
• add entry point to the Consortium DataFrame API (#10244)
• Extend datetime expression function with time zone/time unit parameters (#10235)
• add "batch_size" to scan_pyarrow_dataset parameters (#10249)

🐞 Bug fixes

• clear window cache and run windows on proper runners (#10303)
• fix sorted fast path in streaming groupby wrt nulls (#10289)
• Fix interchange protocol allowing copy even when allow_copy was set to False (#10262)
• fix nan aggregation in groupby (#10287)
• don't panic on cse if function hasn't implemented __eq__ (#10286)
• fix empty streaming parquet file (#10252)
• fix logical columns of streaming multi-column sort (#10250)
• fix date/datetime parsing for short inputs with exact=False (#10231)
• don't panic in wildcard apply (#10240)
• fix cse profile (#10239)

🛠️ Other improvements

• Update CODEOWNERS (#10261)
• add note about pyarrow partitioning (#10297)
• Do not keep history in gh-pages branch (#10282)
• make an explicit note in read_parquet and scan_parquet about hive-style partitioning (point to scan_pyarrow_dataset instead) (#10277)
• Fix typo in error message (#10281)
• Replace "question" issues with link to Stack Overflow (#10230)
• Use sphinx' maximum_signature_line_length (#10228)
• add warning about parallel eval of .then(..) branches (#10229)
• Update Sphinx to 7.1.1 and bump related dependencies (#10221)
• Update dependabot config (#10222)

Thank you to all our contributors for making this release possible! @0xbe7a, @MarcoGorelli, @TLouf, @alexander-beedie, @cmdlineluser, @dependabot, @dependabot[bot], @duvenagep, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego

🐞 Bug fixes

• correct struct null counts (#10142)
• no cse in groupby until fixed (#10216)
• avoid false positives from multiple RETURN_VALUE ops when checking apply lambdas/functions (#10211)

🛠️ Other improvements

• Improve deprecation utils (#10167)

Thank you to all our contributors for making this release possible! @alexander-beedie, @magarick, @ritchie46, @stinodego and @varunmittal91

✨ Enhancements

• raise a better error message from read_database if not passed a string URI (#10191)
• Add pyarrow write_to_dataset to write_parquet function (#9835)

🐞 Bug fixes

• fix is_in on empty series (#10195)
• fix cse windows (#10197)
• block predicate pushdown is_in and null producing … (#10194)
• prevent re-ordering of dict keys inside .apply (#10172)
• initialize fixed null values (#10192)
• Don't pickle _scan_impl (#10175)
• ensure window function run partitioned when cse is hit (#10170)

🛠️ Other improvements

• prepend set_ to set operations on lists (#10182)
• Track version in deprecation utils (#10147)
• Add a simple util issue_deprecation_warning (#10146)
• more precise checks for inefficient apply warnings (#10135)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @cjackal, @cmdlineluser, @potzenhotz, @ritchie46 and @stinodego

🏆 Highlights

• common subexpression elemination (#9632)

⚠️ Deprecations

• Deprecate parsing string inputs as literals for when-then-otherwise (#10122)
• deprecate "connection_uri" → "connection" param in read/write database methods (#10134)
• remove/deprecate cache and its logic (#10066)
• Add date_ranges/time_ranges expression functions (#10005)

🚀 Performance improvements

• speedup mode on sorted data (#10084)
• speedup boolean apply (#10073)
• shrink alp/lp ~2.5x (#10039)

✨ Enhancements

• suggest map_dict instead of lambda x: DICT[x] (#10123)
• enable "inefficient apply" warnings from Series (#10104)
• support writing duration type in json (#10112)
• BytecodeParser can now handle mixed/nested and/or control flow (#10085)
• inline lit(Series).cast(..) to -> lit(Series.cast(..)) (#10092)
• Add ArcTan2 to SQLContext (#9571)
• cse in groupby's (#10062)
• Adds sql CASE statement expressions (#10065)
• Add date_ranges/time_ranges expression functions (#10005)
• comm_subexpr_elim in streaming 'select/with_columns' (#10050)
• add dataframe.flags property (#10037)
• common subexpression elemination (#9632)
• detect and warn about usage of str/int/float python-based casts with apply (#10026)
• detect and warn about usage of json.loads in conjunction with apply (#10023)
• detect and warn about bare numpy functions passed to apply (#10021)
• support bytecode identification/mapping of python string-case functions in UDFs (#10007)
• support bytecode identification of numpy functions in UDFs that we can map to native expressions (#10003)

🐞 Bug fixes

• adjust for null values in str.replace fast path (#10132)
• clear bit settings in list iteration (#10131)
• use row-encoded for struct::is_sorted (#10129)
• fix(rust, python): don't run file-caching in streaming mode (#10117)
• Allow initialize of pl.Array in Dataframe using schema alone (#10100)
• silence Series.apply inefficient apply warning when calling Expr.apply (#10116)
• don't panic if masked out values are invalid in temporal kernels (#10114)
• Fix struct get field by index out of bounds error. (#10097)
• fix ub in simd-json (#10093)
• fix invalid access when groupby rolling produces empty sets (#10109)
• respect null_on_oob=False in list.take when pa… (#10105)
• undo regression in scan_parquet from s3 (#10098)
• fix is_sorted for structs (#10099)
• add file path to io error in scan_csv (#10076)
• fix false positive in parquet stats evaluation (#10087)
• Address .col(regex).exclude() operations not executing. (#10025)
• address an inadvertently shallow-copy issue on underlying PySeries (#10086)
• fix Boolean::isin(null values) (#10074)
• predicate pushdown #10058 (#10071)
• map 'postgres' URI prefix to ADBC 'postgresql' module (#10018)
• Fix weighted quantile for 0 weights (#10051)
• eager time_range/date_range dimensions fix (#9996)

🛠️ Other improvements

• get test_udfs running on all python versions again (#10136)
• temporarily turn off fail-fast so that ubuntu tests run (#10133)
• clarify "clones data" in to_numpy (#10095)
• Refactor when/then/otherwise internals (#9922)
• Properly format Returns sections of docstrings (#10064)
• much-improved Instruction matching for BytecodeParser (#10040)
• add pure-python tests and CI for bytecodeparser (#10027)
• split-out expression translation and instruction-rewrite logic from BytecodeParser (#10012)
• cleans api sections in docs (#10004)
• Bump some dependencies (#9997)
• Add patchelf extra to maturin (#9995)
• restructure all UDF parsing/translation methods into a new BytecodeParser class (#9993)
• Clean up date_range/time_range (#9985)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @c-peters, @cmdlineluser, @jonashaag, @magarick, @mcrumiller, @rikkaka, @ritchie46 and @stinodego

⚠️ Deprecations

• Add Series.extend (#9901)
• Deprecate functions series input (#9878)

🚀 Performance improvements

• Rolling min/max for partially sorted data (#9819)
• Use pyo3::intern to avoid needlessly recreating PyString (#9853)

✨ Enhancements

• Name transpose from column (#9846)
• adds SQRT, CBRT, PI functions to SQLContext (#9936)
• Let qcut create evenly spaced probabilities (#9960)
• add freeze_panes option to write_excel (#9974)
• initial support for parsing the set of jump bytecode instructions required to reconstruct and/or logic (#9972)
• suggest more efficient expression if user passes simple lambda to Expr.apply or DataFrame.apply (#9918)
• sorted flag on singletons (#9933)
• maintain sorted flag after partition_by (#9944)
• keep sorted flag in streaming left join (#9932)
• Add cloudpickle for serializing python UDFs (#9921)
• Optional three-valued logic for any/all (#9848)
• Add Series.extend (#9901)
• pass through unknown schema in unnest (#9896)
• convenience support for parsing a list of SQL strings with sql_expr (#9881)
• respect and allow more options in eager json parsing (#9882)
• allow set_sorted in streaming (#9876)
• Expr.cat.get_categories expression (#9869)
• add LENGTH and OCTET_LENGTH string functions for SQL (#9860)
• polars_warn! macro (#9868)

🐞 Bug fixes

• fix incorrect state in projection pushdown with joins (#9987)
• don't pass predicates referring to renamed literal… (#9965)
• fix regression in regex expansion (#9952)
• potential SO in csv infer schema (#9950)
• raise on unsupported transpose and object types (#9946)
• Fix as-of join when by groups are interleaved (#9938)
• Handle DataFrame.extend extending by itself (#9897)
• don't SO on align_frames (#9911)
• respect original series dtype when constructing LitIter (#9886)
• Handle DataFrame.vstack stacking itself (#9895)
• sum aggregation empty set is 0, not null (#9894)
• preserve expression aliases when parsing SQL with pl.sql_expr (#9875)
• fmt unknown dtype (#9872)

🛠️ Other improvements

• Update autolabeler again (#9984)
• use param_name more in udfs for greater defensiveness (#9969)
• fix or/and docstrings to say bitwise, not logical (#9964)
• minor fix for apply docstring example text (#9953)
• add note that collect_all returns result frames in the same order as input (#9951)
• Improve docstrings for renaming operations (#9942)
• Move sink_* methods to IO chapter (#9939)
• Add 'nearest' in Expr.interpolation docstring with an example (#9935)
• fix hyperlinks to pandas (#9937)
• Address ignored Ruff doc rules (#9919)
• improve weekday, day, ordinal_day examples (#9926)
• deprecate bins argument and rename to breaks in Series.cut (#9913)
• Use Pathlib everywhere (#9914)
• Add various unit tests (#9903)
• add big warnings about using apply (#9906)
• Update autolabeler (#9885)
• Workaround for PyCharm deprecation warning (#9907)
• Mention func_horizontal on deprecated func docstrings (#9863)
• note ordering guarantee for groupby (#9879)
• add logo link entry to sphinx conf and factor-out website root paths (#9864)

Thank you to all our contributors for making this release possible! @0xbe7a, @JulianCologne, @MarcoGorelli, @OneRaynyDay, @SeanTroyUWO, @StefanBRas, @alexander-beedie, @c-peters, @fsimkovic, @ion-elgreco, @magarick, @mcrumiller, @messense, @ritchie46, @sorhawell, @stinodego, @thomasaarholt and @zundertj

🚀 Performance improvements

• speed up python object to AnyValue construction (#9840)
• use hash set in drop_many (#9807)
• speed up in series 10x (#9794)
• Faster is_sorted when no flag set (#9777)

✨ Enhancements

• Add Run-length Encoding functions (#9826)
• add include_key parameter to partition_by (#9750)
• add LEFT string function for SQL (#9836)
• add REGEXP_LIKE function for SQL (both two and three parameter version) (#9838)
• add maintain_order argument to sort/top_k/bottom_k (#9672)
• Dedicated horizontal aggregation functions (#9752)
• support numpy datetime64 units (from 'ns' to 'D') in polars.from_numpy (#9783)
• implement with_row_count as private function (#9810)
• add support for SQL SUBSTR function (#9803)
• add SQL support for binary data and expand recognised SQL dtype strings (#9802)
• add new duration selector and improve selector typing (#9772)
• reworked comfy-table layout constraints, improving table wrapping/repr (#9744)

🐞 Bug fixes

• fix row-encode of 32 byte payloads (#9843)
• shrink_type on all-null columns (#9811)
• don't go into streaming engine when groupby by list (#9834)
• fix regex + exclude (#9827)
• add maintain_order argument to sort/top_k/bottom_k (#9672)
• fix array concat and Series::fill_null (#9825)
• dont preserve sortedness in offset_by for tz-aware non-constant durations (#9818)
• Remove stray arr.eval references (#9821)
• fix row-encode of null data (#9813)
• allow +00:00 when loading from arrow (#9747)
• improve/fix write_database handling of db schema and quoted table names (#9788)
• fix row-count schema (#9797)
• fix supertype detection (#9787)
• fix import error when writing parquet with pyarrow (#9760)

🛠️ Other improvements

• Refactor failing test (#9823)
• Remove stray arr.eval references (#9821)
• Remove old cut/qcut (#9763)
• improve note about the behaviour when converting from ns-precision temporal values to python-native types (#9798)
• Small updates to issue templates (#9789)
• More cleanup around arange (#9769)
• add missing last entry (#9782)
• Add rows_by_key docs (#9766)

Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @avimallu, @jonashaag, @magarick, @mcrumiller, @ritchie46 and @stinodego

✨ Enhancements

• allow qcut in window expressions (#9745)

🐞 Bug fixes

• merge rev-maps when building list arrays of categoricals. (#9742)
• Loosen restrictions on cut expressions and add docs (#9730)
• Fix list symmetric difference (#9732)
• Fix list intersection (#9735)

Thank you to all our contributors for making this release possible! @magarick and @ritchie46

🏆 Highlights

• drop Python 3.7 support (#9679)

🚀 Performance improvements

• optimize n_unique for integers (#9568)
• remove sort columns on multiple-key OOC sort (#9545)
• don't needlessly trigger bitcount (#9561)
• optimize _datetime_to_pl_timestamp (#9533)

✨ Enhancements

• Improve cut and allow use in expressions (#9580)
• clearer message when stringcache-related errors occur (#9715)
• improve expression formatting (#9704)
• set string cache in window functions (#9705)
• raise on both sides of datetime/str comparison (#9692)
• support deserializing struct json into df (#9688)
• add tree formatter for expressions (#9684)
• streamline adbc connectivity, adding snowflake support (#9600)
• improve selector utility functions with better docstrings/examples (#9683)
• add .list.any() and .list.all() (#9573)
• extend dtype/selector matching for Datetime with a "*" wildcard for timezones (#9641)
• add symmetric difference to list set operations (#9655)
• Pass through stdin/stderr buffer in to_csv (#9624)
• add dt.base_utc_offset (#9636)
• add dt.dst_offset feature (#9629)
• allow to specify index order in to_numpy (#9592)
• accept expressions in repeat (#9614)
• set operations for list (#9599)
• make LazyFrame.map pickle (#9597)
• add a new rows_by_key method, returning a keyed-dictionary of row data (#9567)
• implement apply object -> struct (#9578)

🐞 Bug fixes

• don't clear rev_map when categorical series is cle… (#9720)
• fix(rust, python) improve glob pattern testing (#9721)
• don't run hstack checks when using cached names (#9709)
• fix result dtype in date_range(..., eager=True) if duration contains "1s1d" (#9670)
• increment seed between samples (#9694)
• fix cse_plan invalid projection removal (#9700)
• fix ne_missing for booleans vs lit (#9693)
• raise if to_datetime would have parsed input incorrectly (#9675)
• respect time_zone in lazy date_range (#8591)
• Align dependency versions (#9661)
• redo weighted rolling var (#9609)
• Correct weighted rolling quantile definition (#9608)
• clear hashes buffer in generic streaming joins (#9612)
• stable list namespace ouput when all elements are … (#9610)
• address schema edge-case with scalar-expanded data that resolves to an empty frame (#9593)
• handle dictionary init with unsized iterators that also hits the scalar-expansion fast path (#9594)
• validate time zone in cast and from_arrow operations (#9598)
• ensure from_dicts drops columns explicitly omitted from schema (#9581)
• fix join suffix collision (#9579)
• fix sum consistency (#9576)
• fix take of array dtype (#9575)
• fix predicate pushdown case before sort (#9574)
• fix lazy schema of temporal_range functions when no alias is provided (#9543)
• fix join validation when swapped (#9534)

🛠️ Other improvements

• More cleanup for arange (#9681)
• Fix some more type hints (#9716)
• Added trivial examples for the aggregation of columns in groupby (#9708)
• Fix some type hints (#9695)
• additional ADBC examples and docstring information for read_database (inc snowflake) (#9686)
• drop Python 3.7 support (#9679)
• improve selector utility functions with better docstrings/examples (#9683)
• refactor arange and add int_range/int_ranges (#9666)
• Clarify Dataframe.corr operates on columns (#9678)
• remove false "eager=True" from date_range tests (#9663)
• Add examples to .merge_sorted (#9664)
• bump maturin from 1.0.1 to 1.1.0 in /py-polars (#9646)
• remove deprecation warning of already-enforced valid timezones change (#9639)
• fix failing ci test (#9638)
• fix inconsistency in .list.difference() example (#9615)
• Clean up doctests for rolling (#9626)
• fix faulty test of to_numpy (#9619)
• examples for .list.union(), .list.difference(), .list.intersection() (#9602)
• fix see also broken links (#9607)
• clarify sortedness condition of groupby_dynamic and groupby_rolling (#9606)
• clean up inconsistencies in duration string language (#9551)
• Adding examples to binary functions (#9553)
• Minor cleanup of arange (#9544)
• Remove outdated badges from README (#9532)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @borchero, @datapythonista, @dependabot, @dependabot[bot], @eitsupi, @guanqun, @jeroenjanssens, @jorisSchaller, @kljensen, @magarick, @mcrumiller, @messense, @mishpat, @moritzwilksch, @ritchie46, @stinodego, @ttencate, @universalmind303 and @zundertj

🚀 Performance improvements

• don't initialize memory before row-encoding (#9435)
• further optimize datetime conversion (#9452)
• speedup datetime conversion (#9432)
• reduce page faults in q1 ~-30% (#9423)
• reduce rayon/idle time in streaming (#9416)

✨ Enhancements

• add drop_first parameter for to_dummies (issue #8246) (#9143)
• raise if window size in rolling functions isn't strictly positive (#9465)
• serializable python functions in expressions (#9462)
• add infer schema len to json_extract (#9478)
• Adds (Most) Remaining Trig Functions to SQLContext (#9453)
• update error handling msg for sql functions (#9474)
• Update LazyFrame.__repr__ (#9460)
• support inversion of first & last selectors, additional minor repr improvements (#9456)
• add str.titlecase (#9457)
• raise if period is negative in groupby_rolling (#9445)
• enhanced polars.selectors repr and implicit application of as_expr when broadcasting (#9450)
• add SQL round support (#9330)
• dont error for time-zone-aware parsing if time zone is UTC (#9414)

🐞 Bug fixes

• ensure that trying to use a string as a dtype raises a consistent error on both DataFrame and Series init (#9493)
• fix race condition in out-of-core sort (#9521)
• unset sortedness for local date and local datetime (#9515)
• maintain sortedness flags on append/extend (#9496)
• fix serde for small integer dtypes (#9495)
• raise if window size in rolling functions isn't strictly positive (#9465)
• Fix empty list or Series selections on DF or Series (#8660)
• groupby rolling with negative offset (#9428)
• pl.lit with datetime was producing slightly incorrect results (#9438)
• read_csv was parsing dates incorrectly when the dtype was overridden (#9420)
• Compute Spearman rank correlations using average ra… (#9415)
• Fix rolling min/max when window is empty (#9406)

🛠️ Other improvements

• don't pickle pyarrow dataset (#9523)
• fix rendering of examples (#9482)
• Warn for future change of closed default value in rolling functions (#9470)
• Document aggregate_function=None in pivot (#9473)
• Docstrings for expressions and dtypes (#9351)
• fix typo in rolling_* docstrings (#9449)
• Deprecate some expr input parsing behavior (#9455)
• improve date-range docs (#9451)
• Improve docstrings rolling functions (#9215)
• Remove _tempdir module references (#9427)
• fix typo in Series.qcut (#9421)
• Add some documentation on the CI workflows (#9404)

Thank you to all our contributors for making this release possible! @EdmundsEcho, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @baggiponte, @braaannigan, @datapythonista, @magarick, @mcrumiller, @messense, @mgperry, @mishpat, @ritchie46, @stinodego, @tarrafil, @universalmind303 and @zundertj

🚀 Performance improvements

• use row format in streaming join ~15% (#9379)
• row encode buffer reuse (#9371)
• bytes row format for streaming groupby/unique keys >3.5x (#9346)
• push slices down map functions (#9350)

✨ Enhancements

• support all numeric dtypes in serde (#9393)
• allow easy load/save of polars Config options to/from file (#9391)
• ensure part of the plan is streaming if aggregati… (#9387)
• add relaxed concatenation (#9382)
• add sql DROP TABLE (#9355)
• support ternary expressions in streaming (#9343)
• add SQL support for null-aware equality checks (#9332)
• add SQL support for regular expression operators (~, !~, ~*, and !~*) (#9327)
• support // integer floordiv operator in the SQL engine (#9324)

🐞 Bug fixes

• fix bug when comparing series (#9359)
• list zip with (#9367)
• parquet + categorical (#9363)
• respect startby in groupby_dynamic when every is greater than 1d (#9362)
• raise groupby apply on empty frame (#9360)
• raise more informative error on string arguments (#9352)
• Allow for tolerance when comparing nested dtype columns (#9272)
• avoid is_in TypeError with sets of values containing 'None' (#9323)

🛠️ Other improvements

• add top-k test for #9385 (#9388)
• document apply 'return_dtype' requirement (#9361)
• clarify when day of week takes effect in groupby_dynamic (#9342)
• add "if you're coming from pandas" tip to groupby_dynamic (#9336)
• fix string language formatting (#9341)
• add doc entries for eq_missing and ne_missing expressions (#9331)
• fixup options for validate arg in join (#9319)

Thank you to all our contributors for making this release possible! @0xbe7a, @AnatolyBuga, @MarcoGorelli, @alexander-beedie, @dkrako, @durandtibo, @ritchie46 and @universalmind303

🚀 Performance improvements

• increase streaming groupby spill size from 256 to 10_000 (#9312)
• perf(rust, python) Improve rolling min and max for nonulls (#9277)

✨ Enhancements

• allow use of StringCache object as a function decorator (#9309)
• allow use of Config object as a function decorator (#9307)
• serde for 'to_physical' expr (#9294)

🐞 Bug fixes

• fix rolling weighted mean (#9292)
• fix overly-broad string matching in selectors (#9303)
• fix when loading model data from upcoming pydantic 2.x release (#9296)

🛠️ Other improvements

• fix extraneous indent in examples block (#9297)
• Fix typo in Selectors documentation (#9295)

Thank you to all our contributors for making this release possible! @alexander-beedie, @magarick, @ritchie46, @stinodego and @thomascamminady

🏆 Highlights

• add dedicated selectors module, consolidating/expanding existing selector capabilities (#9204)

🚀 Performance improvements

• slightly improve n_unique performance (#9286)
• use ciborium in Expression pickling (#9235)

✨ Enhancements

• add join cardinality validation (#9278)
• implement set operations for selector API (#9276)
• keep sorted flag after Expr::truncate (#9275)
• add "sql_expr" function (#9248)
• rewrite correlation functions to expression architecture (#9258)
• keep sorted flag on offset_by (#9253)
• add expression json serde (#9236)
• add intersection primitive for selector API (#9240)
• building blocks for expression expansion sets (#9231)
• Add ddof option to rolling_var and rolling_std (#8957)
• immediately flatten nested unions (#9220)
• Allow empty select/with_columns/groupby (#9205)
• add a datetime selector (#9212)
• support float expression on integers (#9210)
• add dedicated selectors module, consolidating/expanding existing selector capabilities (#9204)
• add binary to list<u8> cast (#9161)
• groupby_dynamic by quarter. (#6842)
• add arr.unique expression (#9159)
• implement explode for DataType::Array (#9157)
• Decimal type: sum, min, max aggregations in select and agg context. (#9135)
• Decimal arithmetic (#9123)
• support decimals as cast types in csv parser (#9121)
• Improve error handling for repeat (#9117)

🐞 Bug fixes

• fix pyarrow dataset literal filter (#9274)
• raise on invalid sort_by (#9262)
• match missing Array and Struct classes in FromPyObject (#9271)
• correct ne/e_missing schema (#9257)
• fix cached reproject offsets (#9254)
• delay opening files in streaming engine (#9251)
• ensure agg(F(lit)) == lit (#9222)
• don't SO on concat(expressions) (#9214)
• df.apply first rechunk (#9211)
• clip window_size to length in rolling_apply (#9209)
• raise error on invalid df.apply return (#9207)
• Handle edge cases of named select input (#9198)
• rolling_apply window_size == len (#9181)
• respect time zone in strptime/to_datetime when exact=False (#9171)
• make null chunking behavior equal to other dtypes (#9176)
• return single numpy array in Array dtype -> numpy (#9164)
• fix regression in boolean nulls comparison (#9142)
• fix struct null_count if fields are null arrays (#9151)
• Fix DataFrame.to_arrow() for 0x0 dataframes (#9144)
• categorical construction from null values (#9145)
• let apply caller determine if length needs to be checked. (#9140)
• struct is_in should upcast numeric types (#9110)
• Restore functionality of name arg for date_range (#9107)
• bubble up dtype when converting from arrow (#9120)

🛠️ Other improvements

• Fix grammar and add periods in Expr.over docs (#9244)
• Update linting for py-polars crate (#9242)
• Deprecate exprs=... input for select/with_columns/agg/struct (#9219)
• Enable parallelization in Python Windows tests (#9232)
• Use pytest tmp_path (#9206)
• Build docs in parallel (#9229)
• Unify Python docs workflows (#9228)
• add docstring to __array__ methods (#8055)
• Update expr parsing util to return PyExpr (#9166)
• update pyo3 requirement from 0.18 to 0.19 (#9155)
• clarify how the windows are formed in the rolling_* functions (#9192)
• stabilise polars importtime check (#9196)
• fix "to_decimal" docstring (#9197)
• note that exact=False is a performance footgun (#9186)
• change decimal inference and argument order (#9133)
• Cache Rust build on main branch (#9130)
• Improve df.clear() docs (#8809)
• Bump maturin to 1.0.1 (#9115)
• Bump lint dependency versions (#9116)

Thank you to all our contributors for making this release possible! @DeflateAwning, @MarcoGorelli, @alexander-beedie, @ankane, @avimallu, @bfeif, @dependabot, @dependabot[bot], @jonashaag, @josh, @lorentzenchr, @magarick, @ritchie46, @stinodego, @universalmind303 and @zundertj

🏆 Highlights

• Rename list namespace accesor from .arr to .list (#8999)

⚠️ Breaking changes

• propagate null in equality comparisons (#9053)
• formalize implode -> explode relation (#9038)
• Drop subclassing support for DataFrame/LazyFrame (#9008)
• consistently return list of date/datetime from lazy date_range (#8513)
• Default date_range/ones/zeros to eager=False (#9007)
• Rename list namespace accesor from .arr to .list (#8999)
• disallow time zones other than those in zoneinfo.available_timezones() (#8993)
• remove window expression magic (#8992)
• raise error when sorted flag not set (#8994)
• Drop subclassing support for GroupBy (#7746)
• in Series constructor, if inputs are time-zone-aware datetimes, convert to UTC (#8881)
• parse offset-naive date time strings as Timestamp(time_unit), offset-aware datetime strings as Timestamp(time_unit, "UTC"), and remove the utc argument (#8714)
• Remove deprecated tz_aware argument (#8696)

🚀 Performance improvements

• speed up write_csv for time-zone-aware columns (#9093)
• parallelize rolling_window group materialization (#9095)
• elide hot loop in hash joins (#9075)

✨ Enhancements

• conversion from Utf8 to Decimal. (#9090)
• default to checking sortedness in groupby_rolling… (#9063)
• propagate null in equality comparisons (#9053)
• warn if constructing Series with time-zone-aware datetimes (#9058)
• implement apply for rolling/dynamic_groupby (#9049)
• Support more data types in lazy repeat (#9046)
• implement strategy=nearest for join_asof (#9024)
• arr.sum expression (#9041)
• formalize implode -> explode relation (#9038)
• add array namespace and min/max expression (#9032)
• improve error message on row-wise overflow (#9021)
• properly apply slice at UNION level (#9018)
• consistently return list of date/datetime from lazy date_range (#8513)
• Default date_range/ones/zeros to eager=False (#9007)
• disallow time zones other than those in zoneinfo.available_timezones() (#8993)
• raise error when sorted flag not set (#8994)
• in Series constructor, if inputs are time-zone-aware datetimes, convert to UTC (#8881)
• parse offset-naive date time strings as Timestamp(time_unit), offset-aware datetime strings as Timestamp(time_unit, "UTC"), and remove the utc argument (#8714)

🐞 Bug fixes

• rolling_groupy was returning incorrect results when offset was positive (#9082)
• don't underflow on list.tail (#9089)
• fix null/empty in List::take_unchecked (#9074)
• repeat by (#9023)
• raise in to_datetime/strptime if format contains hour but not minute directive (#9044)
• Order of pl.Array arguments in docstring (#9059)
• propagate nulls in broadcasting of order comparisons (#9050)
• Improve read_parquet missing column error message (#8961)
• fix apply with passed date/datetime return_dtype (#9035)
• respect inner type in Array construction (#9020)
• raise error on invalid aggregation (#9013)
• fix fused arithmetic in window functions (#9012)
• don't allow silent init of Series declared as int/temporal with floating point values (#9004)
• deprecate time_unit property from Series (#8990)

🛠️ Other improvements

• Improve expression parsing utils (#9094)
• Refactor expression input parsing util (#9085)
• Organize "as_datatype" functions (#9080)
• Change eager path for repeat (#9048)
• Clean up arange/date_range/time_range (#9027)
• Drop subclassing support for DataFrame/LazyFrame (#9008)
• minor SQLContext docstring cleanups (#9005)
• Rename list namespace accesor from .arr to .list (#8999)
• remove window expression magic (#8992)
• Drop subclassing support for GroupBy (#7746)
• refactor!(python): Remove old deprecated functionality (#8995)
• Remove deprecated tz_aware argument (#8696)

Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @charliegallop, @jonashaag, @mcrumiller, @raymead, @ritchie46, @sorhawell, @stinodego, @tim-habitat and @universalmind303

🏆 Highlights

• Array (backed by arrow::FixedSizeList datatype (#8943)
• Write dataframes as delta tables (#7616)

🚀 Performance improvements

• improve list explode perf (#8974)
• Improve explodes: offsets_to_indexes performance (#8964)
• avoid quadratic exclude behaviour when selecting against dtypes and/or wildcards (#8953)
• use simd-json for all json parsing (#8922)
• improve performance of align_frames, and add new alignment option (#8899)

✨ Enhancements

• error on invalid sortby expr (#8986)
• Pushdown is_in to pyarrow dataset (#8930)
• allow set column list input to 'drop' and 'drop_nulls' (#8962)
• Array (backed by arrow::FixedSizeList datatype (#8943)
• Add dtype argument for repeat (#8946)
• Use schema keys to define the columns if only the schema is provided to pl.struct (#8952)
• multiple enhancements for SQLContext (#8944)
• add sql UNION, UNION ALL & UNION DISTINCT (#8936)
• add sql compound identifiers (#8934)
• add sql EXCLUDE (#8913)
• add sql CASE (#8911)
• add sql EXPLAIN (#8897)
• Write dataframes as delta tables (#7616)
• improve performance of align_frames, and add new alignment option (#8899)
• improved inference from type annotations (#8895)

🐞 Bug fixes

• Fix erroneous warning in hist (#8982)
• don't modify Series with empty names in-place on DataFrame init (#8956)
• respect rechunk in parquet (#8935)
• Add hint on PyArrow to ADBC import error (#8898)
• Simplify offsets_to_indexes, fix empty offsets edge cases (#8920)
• sql qualified wildcards (#8916)
• address edge cases with in-place modification of Series objects (#8915)
• don't check sortedness in asof by (#8906)
• check for object type in csv writer (#8894)
• improve performance of align_frames, and add new alignment option (#8899)
• window function with filtered groups (#8880)

🛠️ Other improvements

• deprecate rename "in_place" parameter (#8960)
• Clean up tests for repeat (#8979)
• Deprecate name argument for repeat (#8977)
• simplify take_every (#8971)
• Clean up repeat/ones/zeros (#8963)
• further enhance SQLContext docstrings (#8948)
• docs(python) Fix typo in lazygroupby.rs error message (#8937)
• fix docstring for time() (#8939)
• refactor tzinfo-related tests (#8883)

Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @avimallu, @cbowdon, @chitralverma, @jonashaag, @kpberry, @mcrumiller, @petar-savov, @ritchie46, @stinodego and @universalmind303

🚀 Performance improvements

• optimise align_frames and properly handle the case where the alignment key has duplicate values (#8825)

✨ Enhancements

• add an align option to pl.concat (#8835)
• add support for sql DISTINCT ON (#8824)
• add LazyFrame null_count (#8837)
• check categorical cache on transpose (#8836)
• add support for OFFSET keyword in SQL queries (#8833)
• optimise align_frames and properly handle the case where the alignment key has duplicate values (#8825)

🐞 Bug fixes

• parse offset-aware strings as UTC in read_csv when try_parse_dates=True (#8864)
• handle InitVar typing declarations on dataclass objects (#8856)
• free buffer, but not its contents (#8848)
• improve agg expr field types (#8834)
• optimise align_frames and properly handle the case where the alignment key has duplicate values (#8825)
• sql BETWEEN bounds should be inclusive (#8818)

🛠️ Other improvements

• add examples for Config "set_tbl_formatting" and "set_fmt_str_lengths" methods (#8859)
• Convert between Vec of Series/Pyseries using trait (#8846)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @stinodego and @universalmind303

🚀 Performance improvements

• add optimizer passes and change initial order (#8811)
• fused multiply sub / sub multiply (#8799)
• improve parallel work distribution of sort expression ~4x (#8775)
• change default row-group size (#8758)
• elide function calls in AnyValue::eq (#8725)

✨ Enhancements

• add a new time_range utility function (#8776)
• Add hint to use _saturating on overflow (#8805)
• add a "restore_defaults" kwarg to Config init (#8797)
• add lazy time expression (#8785)
• support boolean addition (#8778)
• support SQLContext registration of DataFrames (#8762)
• support automatic SQLContext frame/table registration from local variables (#8749)
• improved detail in several error messages (#8747)
• support frame registration at SQLContext init time, and add an "unregister" method (#8744)
• support repeat for all types (#8741)
• add support for DISTINCT keyword in SQL select clauses (#8740)
• support any day of the week in 'start_by' in groupby_dynamic (#8720)
• add support for USING clause in SQL join operations (#8731)
• add unit tests for extend_constant Expr (#8734)
• add clean multi-frame registration to SQLContext (#8724)
• add support for HAVING clause to SQL GROUP BY operations (#8704)
• improved numpy string interop (#8703)

🐞 Bug fixes

• sort cached window groups (#8813)
• check null data before take (#8812)
• fix broadcasting on integer bitwise (#8798)
• Fix incorrect type hint for arange (#8796)
• correct aggregation of overlapping groups (#8794)
• don't parallelize sort within rayon job (#8774)
• fix deadlock in cache and improve parallelism/work… (#8765)
• check offset before doing owned mutation (#8760)
• don't persist temporary column in disjoint calls to update (#8763)
• validate data on successful deserialization (#8757)
• improve supertype coercion of functions (#8755)
• groupby_dynamic was unnecessarily failing on ambiguous local datetime (#8737)
• ensure count aggregation has proper length when spilling (#8735)
• fix return value of std for single-element sequence with ddof=1 (#8730)
• don't take logical plan during streaming fmt (#8711)
• Don't upcast in round() for f32 when decimal is 0 (#8706)

🛠️ Other improvements

• add entry for lazy time func (#8786)
• add unit tests for extend_constant Expr (#8734)
• add rounding coverage for 32/64 bit floats (#8715)
• Add warning to count methods on null (#8698)

Thank you to all our contributors for making this release possible! @DeflateAwning, @MarcoGorelli, @alexander-beedie, @mcrumiller, @ritchie46, @stinodego, @uchiiii, @universalmind303 and @zundertj

🚀 Performance improvements

• add fused multiply add optimization for expressions (#8690)
• use expression for dot product (#8686)

✨ Enhancements

• streaming unions (#8676)
• allow arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)
• expression cache (#8674)
• rolling covariance and correlation (#8671)
• .to_physical() for List(Categorical) (#8499)
• allow from_repr to handle parsing of table reprs with no dtype row (#8640)
• Add dt.to_string alias for dt.strftime (#8290)
• support DataFrame export to numpy structured/record arrays (#8628)
• support transparent DataFrame init from numpy structured/record arrays. (#8620)
• Prettify show_versions (#8627)

🐞 Bug fixes

• allow arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)
• block predicate containing shifts and windows after sort (#8670)
• ensure perfect hash table processes the nulls (#8668)
• Reading more tiny CSVs than workers in parallel will deadlock (#8441)
• respect maintain_order in partitioned groupby (#8653)
• fix explode null series (#8654)
• fix categorical agg type (#8645)
• allow list<null> -> list<cat> (#8636)

🛠️ Other improvements

• add notes/examples on use of inline regex flags to replace docstrings (#8685)
• Add "See Also" sections for alias, map_alias, prefix, s… (#8682)
• add notes/examples on use of inline regex flags to extract_all docstrings (#8675)
• allow arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)
• add notes on the use of inline regex flags to extract docstrings (#8669)
• Add missing implode to internal functions (#8667)
• Clean up type checking imports (#8666)
• Organize PySeries impl blocks (#8665)
• clean-up some examples, extend pipe docstring (#8658)
• add notes on the use of inline regex flags to contains docstrings (#8657)
• fix/improve from_repr example/doctest (#8642)
• Improve some bindings imports (#8630)
• Move functions in Rust bindings to functions module (#8629)
• only require typing_extensions before Python 3.8 (#8623)
• Set up separate modules for lazy classes (#8624)
• Remove duplicate util in Rust bindings (#8622)
• Move Python version to env in release workflow (#8621)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @dependabot, @dependabot[bot], @ghuls, @jonashaag, @josh, @mcrumiller, @ritchie46 and @stinodego

🚀 Performance improvements

• improve nested grouptuples related code (#8618)
• buffer spill partitions in ooc sort. ~10/20% (#8616)
• avoid potentially redundant casts on Series init (#8613)

✨ Enhancements

• add Expr.meta namespace eq and ne methods (#8599)
• avoid potentially redundant casts on Series init (#8613)
• use temp dir for ooc spills (#8614)
• add strict dtype equality comparison methods (is_ and is_not) (#8600)
• automatically convert series <op> expr to pl.lit(series) <op> expr (#8549)

🐞 Bug fixes

• maintain sorted info on top-k and empty sort (#8615)
• fix ooc sort regression; don't take IO-thread before init (#8607)
• maintain sortedness in date -> datetime cast (#8606)

🛠️ Other improvements

• document sortedness of return value of upsample (#8612)
• Set up functions module in Rust bindings (#8598)
• Split PyExpr impl block into modules (#8596)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @dependabot, @dependabot[bot], @mcrumiller, @ritchie46 and @stinodego

🏆 Highlights

• Out-of-core unique (#8573)

🚀 Performance improvements

• improve OOC sort performance during partition phase (#8590)
• significant speedup for python iteration over Series data (#8501)

✨ Enhancements

• make ooc-sort resilient against chunk_size (#8588)
• Out-of-core unique (#8573)
• Add to_date, to_datetime, to_time to String namespace (#8579)
• enhance parametric strategy retrieval, enable List strategy by default (#8571)
• Add default value for round (#8566)
• don't trigger unreachable code if no dtype is set (#8532)
• Ergonomic inputs for all, any, sum, and cumsum (#8541)
• accept expressions in groupby_dynamic/rolling (#8528)
• add is_nested property to dtypes (#8514)

🐞 Bug fixes

• fix determining of supertype for tz-aware and tz-naive datetimes (#8585)
• correct for nested offsets in json serialization (#8584)
• fix wrong dtype init in streaming groupby (#8574)
• fix edge-case with NamedTuple input that contains unhashable field data (#8578)
• temporarily disable List dtype in parametric tests (#8581)
• fix categorical/string_cache fill_null panic (#8562)
• fix testing asserts for NaN values in Struct data (#8557)
• fix window function contention in binary expression (#8544)
• fix struct pyarrow ffi (#8543)
• don't trigger unreachable code if no dtype is set (#8532)
• fix testing asserts for NaN values in List data (#8537)
• keep sorted info on agg_first and simple singleton… (#8526)
• don't downcast Decimal to Float64 in truediv (#8523)
• unset fast_unique coming from arrow (#8521)
• correct sign-reversed scale on DecimalChunked to Python Decimal conversion (fixes #8423) (#8508)
• Clarify and fix behaviour in pl.min/max (#8509)

🛠️ Other improvements

• warn about changing date_range default from lazy=False to eager=False (#8593)
• Rename internals module to _reexport (#8554)
• change partition strategy (#8561)
• fix testing asserts for NaN values in Struct data (#8557)
• note sortedness of results from groupby ops (#8540)
• better type signature for set_sorted (#8529)
• add test for categorical input that is not fast_unique (#8527)
• Improvements to the Python release workflow (#8121)
• Update docs requirements (#8200)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @cgevans, @ritchie46, @stinodego and @uchiiii

Migration guide.

Operation that require columns to be sorted will now give a warning if they are not explicitly sorted, or tagged as sorted.

# 1. inform polars that a column is sorted on the DataFrame / LazyFrame.
(
    df.set_sorted("foo")
    .groupby_dynamic(..)
)

# 2. inform polars inline via the `set_sorted` expression
df.join_asof(df2, on=pl.col("foo").set_sorted())

# 3. explicitly sort first 
# this is expensive if the data is already sorted
df.sort("foo")

✨ Enhancements

• expose quantile/mean for duration (#8491)
• require explicitly sorted flag for upsample (#8488)
• allow for _saturating suffix in duration strings (#8479)

🐞 Bug fixes

• don't error on cast if column is not projected (#8495)
• ensure window function succeeds on empty frame (#8492)
• don't set verbose on union (#8487)
• check literal/group length before claiming agg sta… (#8486)

🛠️ Other improvements

• Remove unneeded operation in strptime (#8496)
• additional parametric testing docs/examples (#8485)
• improve sorted warning/ fix tests (#8484)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46 and @stinodego

🚀 Performance improvements

• less naive count (#8473)
• parallelise dataframe describe method (#8465)
• parallelize almost all flattens (#8468)
• optimize horizontal min/max (#8463)
• reinstate old behavior in numeric group-tuples (#8445)

✨ Enhancements

• apply thousand-separators to "shape" html output, consi… (#8472)
• let duration string accept "1mo_saturating" (#8469)
• add dt.month_start and dt.month_end (#8435)
• add SQL support for cumulative functions (#8457)
• improve utility of dtype groups (#8453)
• improved parametric Decimal strategy (#8444)
• improved hypothesis/parametric testing profile registration (#8433)

🐞 Bug fixes

• fix error message of offset_by if offsetting by negative number of months (#8464)
• fix sorted warning (#8462)
• improve utility of dtype groups (#8453)

🛠️ Other improvements

• bubble up time_iter errors (#8467)
• additional test coverage for dtype groups (#8458)
• integrate live refresh/reload facility while writing docs (#8452)
• add a series of parametric/hypothesis example tests to the main testing docs page (#8454)
• parametric testing docs improvements (#8447)
• improved parametric Decimal strategy (#8444)
• improved hypothesis/parametric testing profile registration (#8433)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @universalmind303 and @utkarshgupta137

🚀 Performance improvements

• remove false sharing in perfect hash table >2x (#8432)
• further optimised conversions to python date/datetime (#8417)

✨ Enhancements

• initial parametric/hypothesis Decimal dtype testing strategy (note: disabled by default) (#8430)
• add Series support to pl.from_repr (#8429)
• Allow %f in strptime format strings (#8404)

🐞 Bug fixes

• raise upon invalid use of zero_copy_only (#8418)
• respect dtype in anonymous list builder in case of… (#8428)
• str.strptime error message: utf -> utc (#8422)

🛠️ Other improvements

• initial parametric/hypothesis Decimal dtype testing strategy (note: disabled by default) (#8430)

Thank you to all our contributors for making this release possible! @alexander-beedie, @ayemjay, @jonashaag, @mzjp2, @pgimalac, @ritchie46 and @stinodego

🚀 Performance improvements

• optimize join inner materialization of single keys (#8405)
• parallelize sorted group tuple materialization (#8387)
• improve materialization of huge cardinality group tuples (#8382)
• improve group_tuples materialization (#8375)
• conversion speedups from polars int64 timestamps to python temporal types:
  • ~35% faster → python date (#8339)
  • ~15% faster → python time (#8352)
  • ~10% faster → python datetime (#8339)

✨ Enhancements

• allow existing item method to optionally take row/col indices (#8412)
• allow negative 'arange' expression (#8413)
• warn if argument is not explicitly sorted (#8409)
• .to_numpy(use_pyarrow=False) for Object and Boolean (#8397)
• new hypothesis strategy that can generate data for List dtypes (#8400)
• offer cleaner usage pattern for Config object in context-manager context (#8394)
• add support for SQL "IN" expr (#8396)
• add a "signed" param to Series.is_integer (#8383)
• add is_integer (#8373)
• raise error on invalid dict aggregation (#8371)
• cli output mode & sql read_json (#8336)
• more informative keyerror on invalid getitem (#8320)

🐞 Bug fixes

• infer supertype in json serde (#8411)
• duration on empty df (#8403)
• don't inadvertently set Series initialised with nested tuple data as Object dtype (#8401)
• use physical in streaming unique global table (#8390)
• recursively bubble up all dtypes in list cast (#8386)
• is_in struct logical types (#8378)
• fix nested null parquet read (#8372)
• fix logical type in ListChunked::new_from_index (#8367)
• fix unintentional loading of hypothesis profile (#8362)
• bubble up logical type in recursive list cast (#8356)
• ensure that iter_rows doesn't return nested Timestamp values (#8359)
• implement clone_inner for all series (#8357)
• add missing __hash__ support to Field, include "time_zone" in Datetime hash, fix Struct hash (#8354)
• fix fill_null for categorical (#8353)
• time.cast(str) as strftime (#8351)
• fix logical dtypes in parallel list collection (#8349)
• improve logical types of explode operation (#8348)
• logical type in anonymous list builders (#8346)
• address potential error caused by float division on time_unit scaling (#8337)
• escape csv header names if they contain special chars (#8331)
• nested struct/list/categorical logical/physical (#8334)
• fix struct schema argument (#8327)
• fix precision issue when converting pl.Datetime("ms") to Python datetime (#8332)
• fix deserialize empty list (#8326)
• List<Null> consistency (#8325)
• fix coalesce schema (#8324)
• don't do null propagation (#8322)
• validate window_size user input in rolling_expr (#8318)
• ensure invalid list eval raises (#8317)
• fix typing overloads of read_excel (#8300)

🛠️ Other improvements

• new hypothesis strategy that can generate data for List dtypes (#8400)
• update duration docstring/example (#8392)
• Upgrade ruff (#8380)
• enhanced parametric testing for temporal dtypes (#8347)
• Minor update to strptime (#8345)
• adjust pytest config so as not to inadvertently prevent test debugging in IPython consoles (#8308)
• add newline in pl.DataFrame.pivot docs (#8307)

Thank you to all our contributors for making this release possible! @JoonHong-Kim, @MarcoGorelli, @StefanBRas, @alexander-beedie, @avimallu, @grantmcdermott, @jonashaag, @rben01, @ritchie46, @stinodego and @universalmind303

🚀 Performance improvements

• use online variance kernel for aggregation (#8306)

Thank you to all our contributors for making this release possible! @ritchie46

🚀 Performance improvements

• add specialized boolean aggregation for min/max (#8294)

✨ Enhancements

• preserve time zone in combine (#8263)

🐞 Bug fixes

• pass name to struct construction in aggregation (#8299)
• improve nested list construction (#8278)
• Truncate long column name in glimpse (#8281)
• Fix DataFrame.sum returning empty column names (#8283)
• always sort in top_k fast path (#8275)
• don't use fast paths for sorted join if there are … (#8272)

🛠️ Other improvements

• use concat_owned_array_unchecked when possible (#8274)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @stinodego, @zaynetro and @zundertj

🏆 Highlights

• support DataFrame init from pydantic model data (#8178)

🚀 Performance improvements

• fail fast on non-inferable strings in strptime if no fmt is provided (#8111)
• make chunks search more resilient (#8229)
• SIMD accelerated arg_min/arg_max (via argminmax) (#8074)
• speed up csv parsing for slower datetimes formats (#8213)
• improve datetime interpret perf (#8209)
• arr.eval run on groupby expression engine when possible (#8199)
• ~2-3x speedup for DataFrame init from pydantic models (#8181)

✨ Enhancements

• add use_earliest argument to replace_time_zone for dealing with ambiguous datetimes (#8087)
• fail loudly on .%f directive, as it differs from the Python standard library (#8237)
• SQL CTE's (#8208)
• automatically convert series OP expr -> pl.lit(series) OP expr where OP is arithmetic (#8225)
• add pickle support for LazyFrame (#8220)
• add duration cumsum and remainder (#8219)
• support DataFrame init from nested dataclass, pydantic, and NamedTuple objects (#8185)
• better algorithm for streaming unique (#8003)
• Add approx distinct count via approx_unique() (#7937)
• add percentiles to describe methods (#8169)
• support DataFrame init from pydantic model data (#8178)
• display skipped row if same number of rows (#8170)

🐞 Bug fixes

• add special numpy float branch in anyvalue conversion (#8259)
• fix boolean par materialization (#8257)
• improve null/empty list construction (#8255)
• fix offsets in parallel utf8 materialization (#8254)
• nested struct logical type consistency (#8249)
• keep literal state if elementwise function is applied (#8195)
• decimal ensure backed arrow arrays have correct dtype (#8193)

🛠️ Other improvements

• parametric/hypothesis testing code cleanups (#8253)
• Rename strptime/strftime args (#8221)
• change sampling ratio for groupby strategy (#8223)
• Rename Expr.list to implode (#8165)
• don't panic on err in offset_by (#8210)
• re-enable test parallization for Windows tests (#8214)
• Fix small typo: "im memory" -> "in memory" (#8187)
• remove unused dtype_to_arrow_type (#8177)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @avimallu, @borchero, @chitralverma, @clickingbuttons, @ghuls, @josh, @jvdd, @rben01, @ritchie46, @stinodego and @universalmind303

✨ Enhancements

• make unique expr serde and cmp (#8153)
• Enhanced parametric testing DataFrame generation (#8149)
• support negative index in pct_change (#8137)
• add log1p to list of mathematical functions (#8102)

🐞 Bug fixes

• object conversion in anyvalue (#8155)
• Address a ~15% regression in import polars speed (#8151)
• validate map lenghts (#8147)
• fix row-wise init of UInt64 values that exceed Int64 upper bound (#8146)
• implement list<null> constructor (#8143)
• add all primitives to av_buffer builder (#8140)
• struct is_in (#8139)
• fix wrong display name of binary expressions (#8131)

🛠️ Other improvements

• Enhanced parametric testing DataFrame generation (#8149)

Thank you to all our contributors for making this release possible! @alexander-beedie, @borchero, @dependabot, @dependabot[bot], @jonashaag, @ritchie46 and @stinodego