NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #396 most downloaded on PyPI
Blazingly fast DataFrame library
Latest stable
2.1.0
Last release 24 days ago
25 Oct 2026
Ships fairly regularly
a new release about every 3 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
67 versions withdrawn
withdrawn after publishing
6 years old
436 releases · first in 2021
Drag across the ruler to compare, or tap a release to open it.
This version includes quite a few breaking changes. We are preparing for the 1.0 release and aim to make the upgrade from 0.20 to 1.0 as smooth as pos…
This version includes quite a few breaking changes. We are preparing for the 1.0 release and aim to make the upgrade from 0.20 to 1.0 as smooth as possible. Therefore, we prioritized getting any breaking changes in now rather than with 1.0.
Check out the upgrade guide for help navigating the upgrade to this version.
Please bear with us while we continue to make Polars the best tool it can be!
Enum categorical data type which allows a fixed set of categories (#11822)read_parquet (#13044)replace expression on the Rust side (#13002)update signature (#12986)Expr.count to ignore null values by default (#12934)DataType objects to be instantiated (#12470)value_counts resulting column name from counts to count (#12506)join behavior with regard to nulls, add join_nulls parameter to keep existing behavior (#12840)Null when no data is present (#12807)lit behavior for list/tuple inputs (#12559)DataType.is_nested from property to classmethod (#12453)NaN ordering to make NaNs compare greater than any other float, and equal to themselves (#12721)write_database parameter if_exists to if_table_exists (#12783)Series methods (#13010)Series.head/tail to the expression engine (#12946)any/all_horizontal (#12976)truncate (#12965)select_seq for expression dispatch (#12962)rolling_median algorithm (#12704)DataFrame.iter_rows for smaller buffer sizes (#12804)Series from a list of NumPy arrays (#12785)str.contains_any and str.replace_many (Aho-Corasick algorithms) (#13073).aws folder (#13062)scan_parquet (#13060)read_parquet (#13044)Series methods (#13010)replace expression on the Rust side (#13002)inefficient map_* warning (#13039)hist (#13014)describe to use new count implementation (#12990)to_struct Series name consistent with the usual default Series name (empty string) (#12998)map_elements" warning message (#12978)end before start in date/time_range (#12964)update signature (#12986)Array data type repr (#12973)Null dtype (#12975)Expr.count to ignore null values by default (#12934)repr of Struct data type class (#12922)merge mode to write_delta and remove pyarrow to delta conversions (#12392)str.reverse (#12878)DataType objects to be instantiated (#12470)value_counts resulting column name from counts to count (#12506)std and var for Duration columns (#12865)join behavior with regard to nulls, add join_nulls parameter to keep existing behavior (#12840)write_database return (indicate the number of rows affected by the operation) (#12830)Decimal selector (#12852)UInt power (#10446)__repr__ implementation for Expr (#12770)JOIN and FROM (#12819)quantile(method="nearest") (#13058)datetime_range if starting on ambiguous datetime and earliest was specified (#13050)json_decode per max buffer length (#13029)00:00 time zone as UTC (#13034)align_frames and fix edge-case where the identical frame object appears more than once (#13007)ranges (#11900)sink_csv (#12991)read_database calls against cursors that only take positional args (#12967)truncate when truncating by multiple weeks (#12948)Err result (#12953)ambiguous parameter is not Utf8 (#12913)rolling_var/rolling_std numerical stability (#12909)min/max due to incorrect SIMD mask construction (#12908)to_numpy in the absence of pyarrow (#12888)Enum types (#12886)Expr.gather (which was still showing deprecated take) (#12864)Array dtype equality (#12853)nan_min/max incorrectly aggregating chunks with addition (#12848)collect_all functions (#12796)group_by (#12304)0.20 (#12844)describe calculation of min/max (#13027)count (#12960)group_by_dynamic (#12906)--no-cov flag for py3.12/ubuntu test workflow (vs implicit/omitted) (#12889)hash docstring (#12879)list.take (#12873)list.take is deprecated (#12867)pip install with dependencies (#12799)update docstring #12797Thank you to all our contributors for making this release possible! @MarcoGorelli, @Object905, @Yerachmiel-Feltzman, @alexander-beedie, @c-peters, @ion-elgreco, @jankislinger, @mcrumiller, @nameexhaustion, @oli-clive-griffin, @orlp, @rancomp, @ritchie46, @romanovacca, @stinodego and @xuestrange
Your coding agent can read these notes before it upgrades. Set up the MCP server →
join behavior with regard to null valuesPreviously, null values in the join key were considered a value like any other value. This meant that null values in the left frame would be joined with null values in the right frame. This is expensive and does not match default behavior in SQL.
Default behavior has now been changed to ignore null values in the join key. The previous behavior
can be retained by setting join_nulls=True.
Example
Before:
>>> df1 = pl.DataFrame({"a": [1, 2, None], "b": [4, 4, 4]})
>>> df2 = pl.DataFrame({"a": [None, 2, 3], "c": [5, 5, 5]})
>>> df1.join(df2, on="a", how="inner")
shape: (2, 3)
┌──────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞══════╪═════╪═════╡
│ null ┆ 4 ┆ 5 │
│ 2 ┆ 4 ┆ 5 │
└──────┴─────┴─────┘
After:
>>> df1.join(df2, on="a", how="inner")
shape: (1, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 2 ┆ 4 ┆ 5 │
└─────┴─────┴─────┘
>>> df1.join(df2, on="a", how="inner", nulls_equal=True) # Keeps previous behavior
shape: (2, 3)
┌──────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞══════╪═════╪═════╡
│ null ┆ 4 ┆ 5 │
│ 2 ┆ 4 ┆ 5 │
└──────┴─────┴─────┘
Previously, the result of an outer join did not contain the join keys of the left and right frames. Rather, it contained a coalesced version of the left key and right key. This loses information and does not conform to default SQL behavior.
The behavior has been changed to include the original join keys. Name clashes are solved by
appending a suffix (_right by default) to the right join key name. The previous behavior can be
retained by setting how="outer_coalesce".
Example
Before:
>>> df1 = pl.DataFrame({"L1": ["a", "b", "c"], "L2": [1, 2, 3]})
>>> df2 = pl.DataFrame({"L1": ["a", "c", "d"], "R2": [7, 8, 9]})
>>> df1.join(df2, on="L1", how="outer")
shape: (4, 3)
┌─────┬──────┬──────┐
│ L1 ┆ L2 ┆ R2 │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ a ┆ 1 ┆ 7 │
│ c ┆ 3 ┆ 8 │
│ d ┆ null ┆ 9 │
│ b ┆ 2 ┆ null │
└─────┴──────┴──────┘
After:
>>> df1.join(df2, on="L1", how="outer")
shape: (4, 4)
┌──────┬──────┬──────────┬──────┐
│ L1 ┆ L2 ┆ L1_right ┆ R2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ str ┆ i64 │
╞══════╪══════╪══════════╪══════╡
│ a ┆ 1 ┆ a ┆ 7 │
│ b ┆ 2 ┆ null ┆ null │
│ c ┆ 3 ┆ c ┆ 8 │
│ null ┆ null ┆ d ┆ 9 │
└──────┴──────┴──────────┴──────┘
>>> df1.join(df2, on="a", how="outer_coalesce") # Keeps previous behavior
shape: (4, 3)
┌─────┬──────┬──────┐
│ L1 ┆ L2 ┆ R2 │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ a ┆ 1 ┆ 7 │
│ c ┆ 3 ┆ 8 │
│ d ┆ null ┆ 9 │
│ b ┆ 2 ┆ null │
└─────┴──────┴──────┘
count now ignores null valuesThe count method for Expr and Series now ignores null values. Use len to get the count with
null values included.
Note that pl.count() and group_by(...).count() are unchanged. These count the number of rows in
the context, so nulls are not applicable in the same way.
This brings behavior more in line with the SQL standard, where COUNT(col) ignores null values but
COUNT(*) counts rows regardless of null values.
Example
Before:
>>> df = pl.DataFrame({"a": [1, 2, None]})
>>> df.select(pl.col("a").count())
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 3 │
└─────┘
After:
>>> df.select(pl.col("a").count())
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
>>> df.select(pl.col("a").len()) # Mirrors previous behavior
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 3 │
└─────┘
NaN values are now considered equalFloating point NaN values were treated as unequal across Polars operations. This has been
corrected to better match user expectation and existing standards.
While this is considered a bug fix, it is included in this guide in order to draw attention to
possible impact on user workflows that may contain NaN values.
Example
Before:
>>> s = pl.Series([1.0, float("nan"), float("inf")])
>>> s == s
shape: (3,)
Series: '' [bool]
[
true
false
true
]
After:
>>> s == s
shape: (3,)
Series: '' [bool]
[
true
true
true
]
NaN equalityThe assertion utility functions assert_frame_equal and assert_series_equal would use the
tolerance parameters atol and rtol to do approximate checking, unless check_exact was set to
True. This could lead to some surprising behavior, as integers are generally thought of as exact
values. Integer values are now always checked exactly. To do inexact checking, convert to float
first.
Additionally, the nans_compare_equal parameter has been removed and NaN values are now always
considered equal, which was the previous default behavior. This parameter had previously been
deprecated but has been removed before the end of the standard deprecation period to facilitate the
change to NaN equality.
Example
Before:
>>> from polars.testing import assert_frame_equal
>>> df1 = pl.DataFrame({"id": [123456]})
>>> df2 = pl.DataFrame({"id": [123457]})
>>> assert_frame_equal(df1, df2) # Passes
After:
>>> assert_frame_equal(df1, df2)
...
AssertionError: DataFrames are different (value mismatch for column 'id')
[left]: [123456]
[right]: [123457]
DataType objects to be instantiatedPolars data types are subclasses of the DataType class. We had a 'hack' in place that
automatically converted data types instantiated without any arguments to the class, rather than
actually instantiating it. The idea was to allow specifying data types as Int64 rather than
Int64(), which is more succinct. However, this caused some unexpected behavior when working
directly with data type objects, especially as there was a discrepancy with data types like
Datetime which will be instantiated in many cases.
Going forward, instantiating a data type will always return an instance of that class. Both classes
and instances are handled by Polars, so the previous short syntax is still available. Methods that
return data types like Series.dtype and DataFrame.schema now always return instantiated data
types objects.
You may have to update some of your data type checks if you were not already using the equality
operator (==), as well as update some type hints.
Example
Before:
>>> s = pl.Series([1, 2, 3], dtype=pl.Int8)
>>> s.dtype == pl.Int8
True
>>> s.dtype is pl.Int8
True
>>> isinstance(s.dtype, pl.Int8)
False
After:
>>> s.dtype == pl.Int8
True
>>> s.dtype is pl.Int8
False
>>> isinstance(s.dtype, pl.Int8)
True
Decimal and Array data typesThe data types Decimal and Array have had their parameters switched around. The new constructors
should more closely match user expectations.
Example
Before:
>>> pl.Array(2, pl.Int16)
Array(Int16, 2)
>>> pl.Decimal(5, 10)
Decimal(precision=10, scale=5)
After:
>>> pl.Array(pl.Int16, 2)
Array(Int16, width=2)
>>> pl.Decimal(10, 5)
Decimal(precision=10, scale=5)
DataType.is_nested changed from a property to a class methodThis is a minor change, but a very important one to properly update. Failure to update accordingly
may result in faulty logic, as Python will evaluate the method to True. For example,
if dtype.is_nested will now evaluate to True regardless of the data type, because it returns the
method, which Python considers truthy.
Example
Before:
>>> pl.List(pl.Int8).is_nested
True
After:
>>> pl.List(pl.Int8).is_nested()
True
dt.month, dt.weekMost datetime components such as month and week would previously return a UInt32 type. This
has been updated to the smallest appropriate signed integer type. This should reduce memory
consumption.
| Method | Dtype old | Dtype new |
|---|---|---|
| year | i32 | i32 |
| iso_year | i32 | i32 |
| quarter | u32 | i8 |
| month | u32 | i8 |
| week | u32 | i8 |
| day | u32 | i8 |
| weekday | u32 | i8 |
| ordinal_day | u32 | i16 |
| hour | u32 | i8 |
| minute | u32 | i8 |
| second | u32 | i8 |
| millisecond | u32 | i32* |
| microsecond | u32 | i32 |
| nanosecond | u32 | i32 |
*Technically, millisecond can be an i16. This may be updated in the future.
Example
Before:
>>> from datetime import date
>>> s = pl.Series([date(2023, 12, 31), date(2024, 1, 1)])
>>> s.dt.month()
shape: (2,)
Series: '' [u32]
[
12
1
]
After:
>>> s.dt.month()
shape: (2,)
Series: '' [u8]
[
12
1
]
Null data type when no data is presentThis replaces the previous behavior of initializing as a Float32 type.
Example
Before:
>>> pl.Series("a", [None])
shape: (1,)
Series: 'a' [f32]
[
null
]
After:
>>> pl.Series("a", [None])
shape: (1,)
Series: 'a' [null]
[
null
]
replace reimplemented with slightly different behaviorThe new implementation is mostly backwards compatible. Please do note the following:
return_dtype to override the inferred data type, or take advantage of the new function
signature (separate old and new parameters) to influence the return type.Example
Before:
>>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]}, schema={"a": pl.Int8, "b": pl.Float64})
>>> df.select(pl.col("a").replace({2: 100}))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ i8 │
╞═════╡
│ 1 │
│ 100 │
│ 100 │
│ 3 │
└─────┘
>>> df.select(pl.struct("a", "b").replace({2: 100}, default=pl.col("b")))
shape: (4, 1)
┌───────┐
│ a │
│ --- │
│ f64 │
╞═══════╡
│ 1.5 │
│ 100.0 │
│ 100.0 │
│ 1.0 │
└───────┘
After:
>>> df.select(pl.col("a").replace({2: 100}))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 100 │
│ 100 │
│ 3 │
└─────┘
>>> df.select(pl.col("a").replace({2: 100}, default=pl.col("b"))) # No struct needed
shape: (4, 1)
┌───────┐
│ a │
│ --- │
│ f64 │
╞═══════╡
│ 1.5 │
│ 100.0 │
│ 100.0 │
│ 1.0 │
└───────┘
value_counts resulting column renamed from counts to countThe resulting struct field for the value_counts method has been renamed from counts to count.
Example
Before:
>>> s = pl.Series("a", ["x", "x", "y"])
>>> s.value_counts()
shape: (2, 2)
┌─────┬────────┐
│ a ┆ counts │
│ --- ┆ --- │
│ str ┆ u32 │
╞═════╪════════╡
│ x ┆ 2 │
│ y ┆ 1 │
└─────┴────────┘
After:
>>> s.value_counts()
shape: (2, 2)
┌─────┬───────┐
│ a ┆ count │
│ --- ┆ --- │
│ str ┆ u32 │
╞═════╪═══════╡
│ x ┆ 2 │
│ y ┆ 1 │
└─────┴───────┘
read_parquet to use Object Store rather than fsspecIf you were using read_parquet, installing fsspec as an optional dependency is no longer
required. The new Object Store implementation was already in use for scan_parquet. It may have
slightly different behavior in certain cases, such as how credentials are detected and how downloads
are performed.
The resulting DataFrame should be identical between versions.
cum* to cum_*Technically, this deprecation was introduced in version 0.19.14, but many users will first
encounter it when upgrading to 0.20. It's a relatively impactful change, which is why we mention
it here.
| Old name | New name |
|---|---|
cumfold |
cum_fold |
cumreduce |
cum_reduce |
cumsum |
cum_sum |
cumprod |
cum_prod |
cummin |
cum_min |
cummax |
cum_max |
cumcount |
cum_count |
Parquet support required deltabyte encoding
Thank you to all our contributors for making this release possible! @nameexhaustion, @ritchie46 and @stinodego
support nested null in vstack/append/extend/concat
with_columns (#12742)write_database, accounting for latest adbc fixes/updates (#12713)atoi_simd release (#12748)xlsx2csv dependency (#12741)aiohttp dependency (#12733)Thank you to all our contributors for making this release possible! @0siride, @PierreAttard, @RoDmitry, @alexander-beedie, @dependabot, @dependabot[bot], @eitsupi, @kszlim, @nameexhaustion, @orlp, @ritchie46 and @stinodego
Automatically wrap NumPy array as lit
DataFrame.iter_columns (#12653)show_versions (#12690)append/extend with null series (#11824) (#12686)scan_parquet supports hive partitioning, remove note pointing to scan_pyarrow_dataset (#12706)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @c-peters, @ritchie46, @stinodego and @tkarabela
Fix deprecation message for DataFrame.sum
series_equal/frame_equal to equals (#12618)map_dict to replace and change default behavior (#12599)List dtype Series from 2D numpy array (#12672)merge_local_rhs_categorical traversal (#12660)PySeries.from_buffer for boolean buffers (#12654)PySeries.from_buffer for numeric types (#12646)filter syntax upgrades to when/then construct (#12603)DataFrame.sum (#12619)performant,lazy,random (#12600)range instead of np.arange in constructors (#12621)Thank you to all our contributors for making this release possible! @alexander-beedie, @c-peters, @cardoso, @dmitrybugakov, @nameexhaustion, @orlp, @ritchie46 and @stinodego
Rename str.json_extract to str.json_decode
str.json_extract to str.json_decode (#12586)~7x (#12552)by column is not sorted in rolling aggregations (as opposed to raising), add warn_if_unsorted argument (#12398)read_csv (#12519)LazyFrame.sink_ndjson (#10786)eval (#12563)group_by_dynamic and rolling (#12551)int_ranges with negative step (#12548)Thank you to all our contributors for making this release possible! @MarcoGorelli, @Qqwy, @alexander-beedie, @dmitrybugakov, @fernandocast, @gab23r, @itamarst, @nameexhaustion, @ritchie46, @stinodego and @uchiiii
Deprecate Series.inner_dtype property
Series.set_at_idx to scatter (#12540)Series.view (#12539)cumsum -> cum_sum and similar (#12513)take to gather (#12528)DataFrame (#12492)take_every to gather_every (#12531)Series.inner_dtype property (#12494)parse_int in favor of to_integer (#12464)is_not (#12458)is_boolean and is_utf8 (#12457)DataType.is_integer and other dtype groups (#12200)~3x 0.19.13/ ~2x numpy (#12471)~2x (#12412)DataFrame (#12492)write_csv and sink_csv (#12253)DataType.is_integer and other dtype groups (#12200)Decimal type to parquet (#12532)Series comparison with timedelta matches that of other types (#12497)map_dicts (#12436)scan_csv error type (#12355)\n when reading file-like object wi… (#12333)PolarsInefficientMapWarning for lshift/rshift operations (#12385)polars-ds to list of community plugins (#12527)polars-hash reference (#12505)polars-hash (#12496)import polars timing test; now much more consistent/reliable (#12478).with_columns() in all .list namespace examples (#12475)manylinux_2_17 for building x86-64 wheel (#12408)Thank you to all our contributors for making this release possible! @MarcoGorelli, @abstractqqq, @alexander-beedie, @c-peters, @cmdlineluser, @hirohira9119, @ion-elgreco, @jerome3o, @nameexhaustion, @reswqa, @ritchie46, @stinodego and @uchiiii
Deprecate _saturating in duration string language, make it the default
write_csv parameter has_header to include_header (#12351)_saturating in duration string language, make it the default (#12301)Decimal and set default scale=0 (#12224)dt.seconds to dt.total_seconds (likewise for days, hours, minutes, milliseconds, microseconds, and nanoseconds) (#12179)DataFrame.as_dict positional input (#12131)BytecodeParser for Python 3.12 (#12348)round_sig_figs expression for rounding to significant figures (#11959)_saturating in duration string language, make it the default (#12301)ambiguous for truncate and round (#12204)Datetime series from datetime.date array (#12175)Config options for numeric formatting: digit grouping and thousands/decimal separator (#12099)name= in .write_avro to set schema name (#12255)write_delta to write large arrow types without casting (#12260).list.to_array expression (#12192).arr.to_list expression (#12136)DataFrame "write" methods (#12113)UInt64 should be correctly extracted from python object (#12338)date_range (#12317)date_range defined with 'saturating' interval (#12311)offset==-period case (#12267)reshape input (#12288)read_excel in the originally specified order (#12243)numpy ufuncs (#12212)take should block predicate pushdown (#12130)schema_overrides information available to the rust-side inference code when initialising from records/dicts (#12045)null_count after arithmetic (#12280)group_by_dynamic docstrings (#12366)rolling_* docstrings (#12362)make clippy, simplify Rust linting workflows (#12290).venv dirs (#12289)py-polars to Cargo workspace (#12256).with_columns in some docstrings (#12250)scan_csv plus slice (#12239)name namespace (#12236)manylinux_2_28 (#12211)rust-toolchain.toml with sdist/wheels (#12184)sqlparser to 0.39 (#12173)strip_{prefix, suffix} & strip_chars_{start, end} (#12161)DataFrame.fold (#12164)Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Priyansh121096, @alexander-beedie, @cmdlineluser, @daviskirk, @dependabot, @dependabot[bot], @dgilman, @hirohira9119, @ion-elgreco, @jrycw, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @owrior, @rancomp, @reswqa, @ritchie46, @rob-sil, @stefmolin, @stinodego and @wsyxbcl
Deprecate DataFrame.as_dict positional input
DataFrame.as_dict positional input (#12131).arr.to_list expression (#12136)DataFrame "write" methods (#12113)take should block predicate pushdown (#12130)schema_overrides information available to the rust-side inference code when initialising from records/dicts (#12045)strip_{prefix, suffix} & strip_chars_{start, end} (#12161)DataFrame.fold (#12164)Thank you to all our contributors for making this release possible! @MarcoGorelli, @Priyansh121096, @alexander-beedie, @dependabot, @dependabot[bot], @jrycw, @moritzwilksch, @nameexhaustion, @reswqa, @ritchie46, @stefmolin and @stinodego
Deprecate nans_compare_equal parameter in assert utils
nans_compare_equal parameter in assert utils (#12019)ljust/rjust to pad_end/pad_start (#11975)shift_and_fill in favor of shift (#11955)clip_min/clip_max in favor of clip (#11961)List/Array (#12016)name namespace for operations that affect expression names (#11973)infer_schema_length to pl.read_json (#11724)get_index/iteration for Array types (#12047)read_excel (#12081)Mapping objects used as schema being silently ignored (#12027)numpy scalar values (#12025)black by ruff format (#11996)dataframe_api_compat dependency (#11997)Development and Releases sections to the documentation (#11932)make clean for docs (#11970)PyExpr consistent (#11956)set_fmt_table_cell_list_len to API docs (#11942)Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Rohxn16, @alexander-beedie, @braaannigan, @brayanjuls, @messense, @nameexhaustion, @orlp, @reswqa, @ritchie46, @squnit, @stinodego and @universalmind303
Deprecate shift_and_fill in favor of shift
shift_and_fill in favor of shift (#11955)clip_min/clip_max in favor of clip (#11961)infer_schema_length to pl.read_json (#11724)Development and Releases sections to the documentation (#11932)make clean for docs (#11970)PyExpr consistent (#11956)set_fmt_table_cell_list_len to API docs (#11942)Thank you to all our contributors for making this release possible! @MarcoGorelli, @Rohxn16, @alexander-beedie, @messense, @orlp, @reswqa, @ritchie46, @squnit and @stinodego
Rename shift parameter from periods to n
shift parameter from periods to n (#11923)Array data type initialization (#11907)read_csv for empty lines (#11924)filter method (#11928)Array data type initialization (#11907)numpy arrays (#11905)read_excel (#11908)read_excel and/or read_ods when target sheet does not exist (#11906)read_excel docstring (#11934)diff methods (#11921)pl.concat "how" param docstring signature (#11909)Thank you to all our contributors for making this release possible! @LaurynasMiksys, @alexander-beedie, @mcrumiller, @reswqa, @ritchie46, @romanovacca, @shenker, @stinodego and @uchiiii
fix accidental quadratic behavior; cache null\_count
DataType.is_nested (#11844)read_database Databricks queries made using SQLAlchemy connections (#11885)include_nulls parameter to update (#11830)cast_unchecked in lists (#11884)Thank you to all our contributors for making this release possible! @Walnut356, @alexander-beedie, @dannyvankooten, @dependabot, @dependabot[bot], @ewoolsey, @jrycw, @mcrumiller, @nameexhaustion, @orlp, @reswqa, @ritchie46, @rjthoen, @romanovacca and @stinodego
Deprecate non-keyword args for ewm methods
filter capabilities with new support for *args predicates, **kwargs constraints, and chained boolean masks (#11740)ewm methods (#11804)use_pyarrow param for Series.to_list (#11784)group_by_rolling to rolling (#11761)DataFrame.get_column performance by ~35% (#11783)DATE function for SQL (#11541)OrderedDict for schemas (#11742)pl.scan_ndjson (#10963)update method (#11688)read_database (#11700)read_database queries (#11664)DataFrame.melt and LazyFrame.unnest (#11662)assert_*_equal AssertionError when exact=False (#11781)PyLazyGroupby reusable (#11769)pl.duration (#11748)join_asof with strategy="nearest" (#11673)_to_rust_syntax util (#11795)IntegralType to IntegerType (#11773)expand_selector in user guide (#11722)df.to_dict/series.to_list (#11757)group_by_dynamic into one module (#11741)describe metrics (#11694)help command output following addition of some longer options (#11681)polars-lts-cpu for macOS x86-64/rosetta (#11660)Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Walnut356, @aberres, @alexander-beedie, @alicja-januszkiewicz, @cmdlineluser, @jrycw, @mcrumiller, @messense, @nameexhaustion, @orlp, @petrosbar, @rancomp, @reswqa, @ritchie46, @romanovacca, @sd2k, @stinodego, @svaningelgem and @thomasjpfan
Fix changelog for language-specific breaking changes
.list.lengths and .str.lengths (#11613)radix in parse_int (#11615)write_csv parameter quote to quote_char (#11583)schema, schema_override for pl.read_json with array-like input (#11492)UNION [ALL] BY NAME, add "diagonal_relaxed" strategy for pl.concat (#11597)read_database options passthrough to the underlying connection's execute method (enables parameterised SQL queries, etc) (#11562)INITCAP string function for SQL (#9884)IN clauses (#11574)scan_csv and read_csv (#11575)is_in handling of mismatched dtypes and fix a minor regression (#11533)USING columns (#11518)py-polars (#11616)write_csv parameter quote to quote_char (#11583)**kwargs from LazyFrame.collect() (#11567)Thank you to all our contributors for making this release possible! @ByteNybbler, @MarcoGorelli, @TheDataScientistNL, @alexander-beedie, @andysham, @c-peters, @jhorstmann, @mcrumiller, @nameexhaustion, @orlp, @reswqa, @ritchie46, @romanovacca, @stinodego and @svaningelgem
Postfix rolling expression as a special case of window functions.
rolling expression as a special case of window functions. (#11445)left_on and right_on parameters to df.update (#11277)IN(subquery) and SQL Subquery Infrastructure (#11218)read_database (#11448)rolling expression as a special case of window functions. (#11445)ColumnFactory to additionally support tab-complete for col in IPython (#11435)cut/qcut when allow_breaks=True (#11287)write_csv when using non-default "quote" char (#11474)read_database fallback for Snowflake warehouses/connections that don't support Arrow resultsets (#11447)ANY and ALL behaviour (#10879)is_in values to the column dtype being searched (#11427)repeat_by to polars-ops (#11461)polars-lts-cpu/polars-u64-idx (#11430)Thank you to all our contributors for making this release possible! @ByteNybbler, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @mcrumiller, @orlp, @ritchie46, @romanovacca, @stinodego, @svaningelgem and Romano Vacca
don't load N metadata files when globbing N files
read_database (#11377)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @bowlofeggs, @c-peters, @jonashaag, @orlp, @ritchie46 and @stinodego
ensure cloud globbing can deal with spaces
infer_schema_length (#11358)Thank you to all our contributors for making this release possible! @MarcoGorelli, @jonashaag, @orlp, @ritchie46 and @stinodego
support 'hive partitioning' aware readers
disable_string_cache (#11020)pydantic models that have a small number of fields, and support direct init from SQLModel data (often used with FastAPI) (#11263)label='right' (#11337)read_excel (for excel binary workbook files) (#11248)disable_string_cache (#11020)NULLIF and COALESCE SQL functions (#11124)tree-formatting representation (#11176)Series.__contains__ for None values and implement is_in for null Series (#11345)quote_style is non-numeric (#11328)has_validity docstring and fix several cases where the presence of a bitmask was used to incorrectly infer the existence of null values (#11319)collections.namedtuple values (#11314)find_stacklevel (#11292)selector expressions in editor/console (#11235)Config JSON string with file path (#11098)scan_pyarrow predicates (#11195)- or + operators (#11158)read_excel "read_csv_options" (#11162)assert_frame_equal for LazyFrames (don't collect until after the schema has been checked) (#11331)GITHUB_TOKEN to get contributor information for docs (#11321)null_count from has_validity (clarifies the correct way to check for nulls) (#11323)<2.4.0 (#11312)IntoExprColumn (#11296)performant feature only once (#11223)read_database batch_size docstring (#11132)Thank you to all our contributors for making this release possible! @ByteNybbler, @Cheukting, @Fokko, @Hofer-Julian, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @billylanchantin, @jonashaag, @mcrumiller, @orlp, @ptiza, @reswqa, @ritchie46, @stinodego and @universalmind303
Add deprecation message in groupby docs
is_first/last to is_first/last_distinct (#11130)count_match to count_matches (#11028)strip to strip_chars (#10813)datetime_range expression function (#10213)_unpack_schema() (#11080)polars.utils._post_apply_columns() (#11086)polars.utils._post_apply_columns() (#11041)_unpack_schema() (#10960)pl.read_ods function (#11011)write_csv (#11015)literal for str count_match (#10996)strip_prefix and strip_suffix to the string namespace (#10958)read_excel table data identification (#10953)from_dataframe fast path and improve typing (#10979)openpyxl as a new/optional engine for read_excel (#6183)datetime_range expression function (#10213)Series.__getitem__ raise an IndexError (#11061)read_dicts and reduce latency of small-frame creation (#11047)series_equal properly accounts for dtypes when strict=True (#11012)SchemaDefinition type alias (#11077)fetch explanation in a "notes" block to better highlight it in the docs (#11058)get_data_buffer (#10966)pydantic >= 2.0.0 requirement (#10944)Thank you to all our contributors for making this release possible! @I8dNLo, @KacpiW, @MarcoGorelli, @Object905, @Qqwy, @TNieuwdorp, @alexander-beedie, @antoniocali, @bvanelli, @cjackal, @henrikig, @jakob-keller, @mrogowski11, @nameexhaustion, @orlp, @reswqa, @ritchie46, @s-banach, @stinodego, @svaningelgem and @thomasjpfan
Add syntactic sugar for col("foo") -> col.foo
col("foo") -> col.foo (#10874)Expr.is_not() to not_() (#10838)Config options to be easily reset to their default value (#10922)str.count_match (#10900)glimpse customisation, fix strings repr (#10895).offset_by (#9967)col("foo") -> col.foo (#10874)select (#10885)read_database (#10851)int_range (#10914)int_range(s) exclusive on the upper bound when step is negative (#10898)2.0.0 (#10923)Expr.map_elements (#10647)read_database connection/cursor behaviour (#10873)Thank you to all our contributors for making this release possible! @Barsik-sus, @MarcoGorelli, @alexander-beedie, @c-peters, @cmdlineluser, @dependabot, @dependabot[bot], @drgif, @jeroenjanssens, @orlp, @ritchie46, @stinodego and @wdoppenberg
empty product returns identity and product ignores nulls
binary, boolean, categorical, date, object, and time selectors (#10806)allow_copy=False (#10822)reversed(df) (#10823)range related functions (#10830)Thank you to all our contributors for making this release possible! @alexander-beedie, @orlp, @reswqa, @ritchie46 and @stinodego
Remove deprecated behavior from vertical aggregations
An upgrade guide is available on our website.
DataFrame init from queries against users' existing database connections (#10649)groupby to group_by (#10656)f64 for rank when method="average" (#10734)all - fix Kleene logic implementation for all/any (#10564)from_arrow to take a generator of RecordBatches, change error type to TypeError (#10529)arange an alias for int_range (#9983)date_range/time_range no longer return a List type (#10526)0.18 (#10527)map to map_batches (#10801)GroupBy.apply to map_groups (#10799)DataFrame.apply to map_rows (#10797)Series/Expr.rolling_apply to rolling_map (#10750)Series/Expr.apply to map_elements (#10678)groupby to group_by (#10656)cut/qcut (#10484)Protocol for interchange classes (#10688)DataFrame init from queries against users' existing database connections (#10649)truncate_ragged_lines (#10660)write_excel arguments (#10589)LazyFrame.collect_async and pl.collect_all_async (#10616)is_in and more generic array construction (#10614)all - fix Kleene logic implementation for all/any (#10564)cast support (#10504)from_arrow to take a generator of RecordBatches, change error type to TypeError (#10529)get_idx_type - use get_index_type instead (#10556)arange an alias for int_range (#9983)date_range/time_range no longer return a List type (#10526)0.18 (#10527)ORDER BY on unselected columns (#10752)pre-wrap instead of pre (#10739)value_counts on column named "counts" (#10737)f64 for rank when method="average" (#10734)on arg type for join_asof (#10690)write_delta (#10633)is_in (#10620)all - fix Kleene logic implementation for all/any (#10564)write_delta with schema in delta_write_options (#10541)pl.Config options relating to shape, column names, and types when rendering HTML (#10449).venv in repo root (#10789)write_database unit tests to properly separate concerns (#10773)adbc release (#10763)connectorx and bump other Python dependencies (#10753)testing docs about module import (#10741)13.0.0 behavior (#10691)sink_parquet docs (#10669)deprecate_renamed_methods util (#10537)inspect.currentframe (#10630)Expr.meta namespace (#10617)Cargo.lock (#10555)make requirements fully refreshes unpinned packages/deps (#10591)expr_dispatch decorator to work on methods with decorators (#10549)pyo3/maturin-action (#10503)Thank you to all our contributors for making this release possible! @JulianCologne, @MarcoGorelli, @Object905, @OndrejSlamecka, @SeanTroyUWO, @VasanthakumarV, @alexander-beedie, @aminalaee, @braaannigan, @c-peters, @ion-elgreco, @lorepozo, @marki259, @mcrumiller, @messense, @orlp, @owrior, @rben01, @reswqa, @ritchie46, @sdamashek, @stinodego, @svaningelgem, @titoeb, @trueb2, @washcycle and @zundertj
This impacts aggregation functions like sum, min, and max. These functions were overloaded to
support both vertical and horizontal computation. Recently, new dedicated functionality for
horizontal computation was released, and horizontal computation was deprecated.
Restore the old behavior by using the horizontal variant, e.g. sum_horizontal.
Example
Before:
>>> df = pl.DataFrame({'a': [1, 2], 'b': [11, 12]})
>>> df.select(pl.sum('a', 'b')) # horizontal computation
shape: (2, 1)
┌─────┐
│ sum │
│ --- │
│ i64 │
╞═════╡
│ 12 │
│ 14 │
└─────┘
After:
>>> df = pl.DataFrame({'a': [1, 2], 'b': [11, 12]})
>>> df.select(pl.sum('a', 'b')) # vertical computation
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 23 │
└─────┴─────┘
all / anyall will now ignore null values by default, rather than treat them as False.
For both any and all, the drop_nulls parameter has been renamed to ignore_nulls and is now
keyword-only. Also fixed an issue when setting this parameter to False would erroneously result in
None output in some cases.
To restore the old behavior, set ignore_nulls to False and check for None output.
Example
Before:
>>> pl.Series([True, None]).all()
False
After:
>>> pl.Series([True, None]).all()
True
Improving our error messages is an ongoing effort. We did a sweep of our Python code base and made
many improvements to error messages and error types. Most notably, many ValueErrors were changed
to TypeErrors.
If your code relies on handling Polars exceptions, you may have to make some adjustments.
Example
Before:
>>> pl.Series(values=15)
...
ValueError: Series constructor called with unsupported type; got 'int'
After:
>>> pl.Series(values=15)
...
TypeError: Series constructor called with unsupported type 'int' for the `values` parameter
Methods like select and with_columns accept one or more expressions. But they also accept
strings, integers, lists, and other inputs that we try to interpret as expressions. We updated our
internal logic to parse inputs more consistently.
Example
Before:
>>> pl.DataFrame({'a': [1, 2]}).with_columns(None)
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
└─────┘
After:
>>> pl.DataFrame({'a': [1, 2]}).with_columns(None)
shape: (2, 2)
┌─────┬─────────┐
│ a ┆ literal │
│ --- ┆ --- │
│ i64 ┆ null │
╞═════╪═════════╡
│ 1 ┆ null │
│ 2 ┆ null │
└─────┴─────────┘
shuffle / sample now use an internal Polars seedIf you used the built-in Python random.seed function to control the randomness of Polars
expressions, this will no longer work. Instead, use the new set_random_seed function.
Example
Before:
import random
random.seed(1)
After:
import polars as pl
pl.set_random_seed(1)
Creating a consistent and intuitive API is hard; finding the right name for each function, method,
and parameter might be the hardest part. The new version comes with several naming changes, and you
will most likely run into deprecation warnings when upgrading to 0.19.
If you want to upgrade without worrying about deprecation warnings right now, you can add the following snippet to your code:
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
groupby renamed to group_byThis is not a change we make lightly, as it will impact almost all our users. But "group by" is really two different words, and our naming strategy dictates that these should be separated by an underscore.
Most likely, a simple search and replace will be enough to take care of this update:
.groupby(.group_by(apply renamed to map_*apply is probably the most misused part of our API. Many Polars users come from pandas, where
apply has a completely different meaning.
We now consolidate all our functionality for user-defined functions under the name map. This
results in the following renaming:
| Before | After |
|---|---|
Series/Expr.apply |
map_elements |
Series/Expr.rolling_apply |
rolling_map |
DataFrame.apply |
map_rows |
GroupBy.apply |
map_groups |
apply |
map_groups |
map |
map_batches |
rollback cse in groupby: python 0.18.15
maturin to version 1.2.1 (#10479)Thank you to all our contributors for making this release possible! @ritchie46 and @stinodego
Deprecate behavior of list/tuple inputs for lit
lit (#10461)df.item (~4-5x speedup) (#10411)read_excel, read_csv, scan_csv, and read_csv_batched (#10409)use_earliest to to_datetime / strptime (#10426)write_excel (#10392)selector variants for signed/unsigned integers (#10384)is_local and to_local to categorical namespace (#10372)selectors expansion function, so it can operate on a schema as well as a frame (#10341)describe (#10378)OverflowError in testing asserts with huge UInt64 diffs (#10437)vertical_relaxed example for pl.concat (#10472)Sphinx settings (#10400)Thank you to all our contributors for making this release possible! @MarcoGorelli, @OndrejSlamecka, @alexander-beedie, @c-peters, @cmdlineluser, @drgif, @ion-elgreco, @lfn3, @orlp, @potzenhotz, @rea1bacon, @reswqa, @ritchie46, @stinodego and @zundertj
Rename LazyFrame.read/write_json to de/serialize
LazyFrame.read/write_json to de/serialize (#10238)categorical_as_str parameter to testing utils (#10350)selectors in additional frame methods (#10255)Series.cat.uses_lexical_ordering (#10325)time, date, datetime (#10298)categorical_as_str parameter to testing utils.extract_groups() (#10306)Thank you to all our contributors for making this release possible! @CanglongCl, @JulianCologne, @MarcoGorelli, @alexander-beedie, @cmdlineluser, @eltociear, @orlp, @ritchie46 and @stinodego
renaming approx_unique as approx_n_unique
approx_unique as approx_n_unique (#10290)qcut parameter to quantiles (#10253)avg alias for mean (#10236)str.extract_groups (#10179)TypeError for all LazyFrame comparison operators (#10275)map_dict where the lookup key is an expression (#10265)datetime expression function with time zone/time unit parameters (#10235)scan_pyarrow_dataset parameters (#10249)allow_copy was set to False (#10262)gh-pages branch (#10282)read_parquet and scan_parquet about hive-style partitioning (point to scan_pyarrow_dataset instead) (#10277)maximum_signature_line_length (#10228).then(..) branches (#10229)Thank you to all our contributors for making this release possible! @0xbe7a, @MarcoGorelli, @TLouf, @alexander-beedie, @cmdlineluser, @dependabot, @dependabot[bot], @duvenagep, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego
avoid false positives from multiple RETURN_VALUE ops when checking apply lambdas/functions
RETURN_VALUE ops when checking apply lambdas/functions (#10211)Thank you to all our contributors for making this release possible! @alexander-beedie, @magarick, @ritchie46, @stinodego and @varunmittal91
Track version in deprecation utils
read_database if not passed a string URI (#10191)is_in on empty series (#10195).apply (#10172)_scan_impl (#10175)issue_deprecation_warning (#10146)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @cjackal, @cmdlineluser, @potzenhotz, @ritchie46 and @stinodego
Deprecate parsing string inputs as literals for when-then-otherwise
when-then-otherwise (#10122)date_ranges/time_ranges expression functions (#10005)~2.5x (#10039)Series (#10104)and/or control flow (#10085)lit(Series).cast(..) to -> lit(Series.cast(..)) (#10092)SQLContext (#9571)CASE statement expressions (#10065)date_ranges/time_ranges expression functions (#10005)apply (#10026)json.loads in conjunction with apply (#10023)numpy functions passed to apply (#10021)numpy functions in UDFs that we can map to native expressions (#10003)null_on_oob=False in list.take when pa… (#10105).col(regex).exclude() operations not executing. (#10025)time_range/date_range dimensions fix (#9996)when/then/otherwise internals (#9922)Returns sections of docstrings (#10064)Instruction matching for BytecodeParser (#10040)BytecodeParser (#10012)BytecodeParser class (#9993)date_range/time_range (#9985)Thank you to all our contributors for making this release possible! @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @c-peters, @cmdlineluser, @jonashaag, @magarick, @mcrumiller, @rikkaka, @ritchie46 and @stinodego
Deprecate functions series input
Series.extend (#9901)pyo3::intern to avoid needlessly recreating PyString (#9853)SQRT, CBRT, PI functions to SQLContext (#9936)jump bytecode instructions required to reconstruct and/or logic (#9972)Series.extend (#9901)sql_expr (#9881)LENGTH and OCTET_LENGTH string functions for SQL (#9860)polars_warn! macro (#9868)by groups are interleaved (#9938)DataFrame.extend extending by itself (#9897)LitIter (#9886)DataFrame.vstack stacking itself (#9895)pl.sql_expr (#9875)apply docstring example text (#9953)collect_all returns result frames in the same order as input (#9951)sink_* methods to IO chapter (#9939)weekday, day, ordinal_day examples (#9926)bins argument and rename to breaks in Series.cut (#9913)link entry to sphinx conf and factor-out website root paths (#9864)Thank you to all our contributors for making this release possible! @0xbe7a, @JulianCologne, @MarcoGorelli, @OneRaynyDay, @SeanTroyUWO, @StefanBRas, @alexander-beedie, @c-peters, @fsimkovic, @ion-elgreco, @magarick, @mcrumiller, @messense, @ritchie46, @sorhawell, @stinodego, @thomasaarholt and @zundertj
speed up python object to AnyValue construction
in series 10x (#9794)include_key parameter to partition_by (#9750)LEFT string function for SQL (#9836)REGEXP_LIKE function for SQL (both two and three parameter version) (#9838)maintain_order argument to sort/top_k/bottom_k (#9672)SUBSTR function (#9803)duration selector and improve selector typing (#9772)maintain_order argument to sort/top_k/bottom_k (#9672)arr.eval references (#9821)write_database handling of db schema and quoted table names (#9788)arr.eval references (#9821)arange (#9769)last entry (#9782)rows_by_key docs (#9766)Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @avimallu, @jonashaag, @magarick, @mcrumiller, @ritchie46 and @stinodego
allow qcut in window expressions
Thank you to all our contributors for making this release possible! @magarick and @ritchie46
remove deprecation warning of already-enforced valid timezones change
_datetime_to_pl_timestamp (#9533)adbc connectivity, adding snowflake support (#9600)selector utility functions with better docstrings/examples (#9683).list.any() and .list.all() (#9573)Datetime with a "*" wildcard for timezones (#9641)to_numpy (#9592)repeat (#9614)rows_by_key method, returning a keyed-dictionary of row data (#9567)from_dicts drops columns explicitly omitted from schema (#9581)arange (#9681)read_database (inc snowflake) (#9686)selector utility functions with better docstrings/examples (#9683)arange and add int_range/int_ranges (#9666).list.difference() example (#9615)to_numpy (#9619).list.union(), .list.difference(), .list.intersection() (#9602)arange (#9544)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @borchero, @datapythonista, @dependabot, @dependabot[bot], @eitsupi, @guanqun, @jeroenjanssens, @jorisSchaller, @kljensen, @magarick, @mcrumiller, @messense, @mishpat, @moritzwilksch, @ritchie46, @stinodego, @ttencate, @universalmind303 and @zundertj
Deprecate some expr input parsing behavior
~-30% (#9423)SQLContext (#9453)first & last selectors, additional minor repr improvements (#9456)polars.selectors repr and implicit application of as_expr when broadcasting (#9450)round support (#9330)Series.qcut (#9421)Thank you to all our contributors for making this release possible! @EdmundsEcho, @MarcoGorelli, @SeanTroyUWO, @alexander-beedie, @baggiponte, @braaannigan, @datapythonista, @magarick, @mcrumiller, @messense, @mgperry, @mishpat, @ritchie46, @stinodego, @tarrafil, @universalmind303 and @zundertj
use row format in streaming join ~15%
~15% (#9379)>3.5x (#9346)Config options to/from file (#9391)~, !~, ~*, and !~*) (#9327)// integer floordiv operator in the SQL engine (#9324)is_in TypeError with sets of values containing 'None' (#9323)eq_missing and ne_missing expressions (#9331)validate arg in join (#9319)Thank you to all our contributors for making this release possible! @0xbe7a, @AnatolyBuga, @MarcoGorelli, @alexander-beedie, @dkrako, @durandtibo, @ritchie46 and @universalmind303
increase streaming groupby spill size from 256 to 10\_000
StringCache object as a function decorator (#9309)Config object as a function decorator (#9307)pydantic 2.x release (#9296)Thank you to all our contributors for making this release possible! @alexander-beedie, @magarick, @ritchie46, @stinodego and @thomascamminady
Deprecate exprs=... input for select/with_columns/agg/struct
selectors module, consolidating/expanding existing selector capabilities (#9204)offset_by (#9253)select/with_columns/groupby (#9205)datetime selector (#9212)selectors module, consolidating/expanding existing selector capabilities (#9204)Decimal type: sum, min, max aggregations in select and agg context. (#9135)repeat (#9117)select input (#9198)apply caller determine if length needs to be checked. (#9140)is_in should upcast numeric types (#9110)name arg for date_range (#9107)Expr.over docs (#9244)py-polars crate (#9242)exprs=... input for select/with_columns/agg/struct (#9219)tmp_path (#9206)PyExpr (#9166)exact=False is a performance footgun (#9186)maturin to 1.0.1 (#9115)Thank you to all our contributors for making this release possible! @DeflateAwning, @MarcoGorelli, @alexander-beedie, @ankane, @avimallu, @bfeif, @dependabot, @dependabot[bot], @jonashaag, @josh, @lorentzenchr, @magarick, @ritchie46, @stinodego, @universalmind303 and @zundertj
Remove deprecated tz\_aware argument
.arr to .list (#8999)DataFrame/LazyFrame (#9008)date_range/ones/zeros to eager=False (#9007).arr to .list (#8999)Utf8 to Decimal. (#9090)repeat (#9046)date_range/ones/zeros to eager=False (#9007)Series declared as int/temporal with floating point values (#9004)time_unit property from Series (#8990)repeat (#9048)arange/date_range/time_range (#9027)DataFrame/LazyFrame (#9008)SQLContext docstring cleanups (#9005).arr to .list (#8999)Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @charliegallop, @jonashaag, @mcrumiller, @raymead, @ritchie46, @sorhawell, @stinodego, @tim-habitat and @universalmind303
deprecate rename "in\_place" parameter
Array (backed by arrow::FixedSizeList datatype (#8943)offsets_to_indexes performance (#8964)exclude behaviour when selecting against dtypes and/or wildcards (#8953)align_frames, and add new alignment option (#8899)is_in to pyarrow dataset (#8930)Array (backed by arrow::FixedSizeList datatype (#8943)dtype argument for repeat (#8946)pl.struct (#8952)SQLContext (#8944)align_frames, and add new alignment option (#8899)hist (#8982)Series with empty names in-place on DataFrame init (#8956)Series objects (#8915)align_frames, and add new alignment option (#8899)rename "in_place" parameter (#8960)repeat (#8979)name argument for repeat (#8977)take_every (#8971)repeat/ones/zeros (#8963)SQLContext docstrings (#8948)lazygroupby.rs error message (#8937)time() (#8939)Thank you to all our contributors for making this release possible! @CloseChoice, @MarcoGorelli, @alexander-beedie, @avimallu, @cbowdon, @chitralverma, @jonashaag, @kpberry, @mcrumiller, @petar-savov, @ritchie46, @stinodego and @universalmind303
optimise align_frames and properly handle the case where the alignment key has duplicate values
align_frames and properly handle the case where the alignment key has duplicate values (#8825)align option to pl.concat (#8835)null_count (#8837)OFFSET keyword in SQL queries (#8833)align_frames and properly handle the case where the alignment key has duplicate values (#8825)InitVar typing declarations on dataclass objects (#8856)align_frames and properly handle the case where the alignment key has duplicate values (#8825)BETWEEN bounds should be inclusive (#8818)Config "set_tbl_formatting" and "set_fmt_str_lengths" methods (#8859)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @stinodego and @universalmind303
add optimizer passes and change initial order
~4x (#8775)time_range utility function (#8776)Config init (#8797)time expression (#8785)SQLContext registration of DataFrames (#8762)SQLContext frame/table registration from local variables (#8749)SQLContext init time, and add an "unregister" method (#8744)DISTINCT keyword in SQL select clauses (#8740)USING clause in SQL join operations (#8731)extend_constant Expr (#8734)SQLContext (#8724)HAVING clause to SQL GROUP BY operations (#8704)numpy string interop (#8703)arange (#8796)update (#8763)time func (#8786)extend_constant Expr (#8734)Thank you to all our contributors for making this release possible! @DeflateAwning, @MarcoGorelli, @alexander-beedie, @mcrumiller, @ritchie46, @stinodego, @uchiiii, @universalmind303 and @zundertj
add fused multiply add optimization for expressions
arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)from_repr to handle parsing of table reprs with no dtype row (#8640)dt.to_string alias for dt.strftime (#8290)DataFrame export to numpy structured/record arrays (#8628)DataFrame init from numpy structured/record arrays. (#8620)arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)replace docstrings (#8685)extract_all docstrings (#8675)arr.to_struct to take a list of field names, fix it for Series, improve related docstrings (#8673)extract docstrings (#8669)implode to internal functions (#8667)impl blocks (#8665)pipe docstring (#8658)contains docstrings (#8657)from_repr example/doctest (#8642)functions module (#8629)typing_extensions before Python 3.8 (#8623)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @dependabot, @dependabot[bot], @ghuls, @jonashaag, @josh, @mcrumiller, @ritchie46 and @stinodego
improve nested grouptuples related code
~10/20% (#8616)Series init (#8613)Expr.meta namespace eq and ne methods (#8599)Series init (#8613)is_ and is_not) (#8600)series <op> expr to pl.lit(series) <op> expr (#8549)functions module in Rust bindings (#8598)impl block into modules (#8596)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @dependabot, @dependabot[bot], @mcrumiller, @ritchie46 and @stinodego
improve OOC sort performance during partition phase
Series data (#8501)to_date, to_datetime, to_time to String namespace (#8579)List strategy by default (#8571)round (#8566)all, any, sum, and cumsum (#8541)groupby_dynamic/rolling (#8528)is_nested property to dtypes (#8514)NamedTuple input that contains unhashable field data (#8578)List dtype in parametric tests (#8581)NaN values in Struct data (#8557)NaN values in List data (#8537)Decimal to Float64 in truediv (#8523)pl.min/max (#8509)internals module to _reexport (#8554)NaN values in Struct data (#8557)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @cgevans, @ritchie46, @stinodego and @uchiiii
Operation that require columns to be sorted will now give a warning if they are not explicitly sorted, or tagged as sorted.
Operation that require columns to be sorted will now give a warning if they are not explicitly sorted, or tagged as sorted.
# 1. inform polars that a column is sorted on the DataFrame / LazyFrame.
(
df.set_sorted("foo")
.groupby_dynamic(..)
)
# 2. inform polars inline via the `set_sorted` expression
df.join_asof(df2, on=pl.col("foo").set_sorted())
# 3. explicitly sort first
# this is expensive if the data is already sorted
df.sort("foo")
strptime (#8496)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46 and @stinodego
parallelise dataframe describe method
describe method (#8465)Decimal strategy (#8444)Decimal strategy (#8444)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @universalmind303 and @utkarshgupta137
remove false sharing in perfect hash table >2x
>2x (#8432)Decimal dtype testing strategy (note: disabled by default) (#8430)Series support to pl.from_repr (#8429)%f in strptime format strings (#8404)str.strptime error message: utf -> utc (#8422)Decimal dtype testing strategy (note: disabled by default) (#8430)Thank you to all our contributors for making this release possible! @alexander-beedie, @ayemjay, @jonashaag, @mzjp2, @pgimalac, @ritchie46 and @stinodego
optimize join inner materialization of single keys
item method to optionally take row/col indices (#8412)List dtypes (#8400)Config object in context-manager context (#8394)Series.is_integer (#8383)Series initialised with nested tuple data as Object dtype (#8401)iter_rows doesn't return nested Timestamp values (#8359)__hash__ support to Field, include "time_zone" in Datetime hash, fix Struct hash (#8354)window_size user input in rolling_expr (#8318)read_excel (#8300)List dtypes (#8400)duration docstring/example (#8392)strptime (#8345)Thank you to all our contributors for making this release possible! @JoonHong-Kim, @MarcoGorelli, @StefanBRas, @alexander-beedie, @avimallu, @grantmcdermott, @jonashaag, @rben01, @ritchie46, @stinodego and @universalmind303
use online variance kernel for aggregation
Thank you to all our contributors for making this release possible! @ritchie46
add specialized boolean aggregation for min/max
top_k fast path (#8275)concat_owned_array_unchecked when possible (#8274)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ritchie46, @stinodego, @zaynetro and @zundertj
support DataFrame init from pydantic model data
DataFrame init from pydantic model data (#8178)fmt is provided (#8111)arg_min/arg_max (via argminmax) (#8074)arr.eval run on groupby expression engine when possible (#8199)DataFrame init from pydantic models (#8181)use_earliest argument to replace_time_zone for dealing with ambiguous datetimes (#8087)series OP expr -> pl.lit(series) OP expr where OP is arithmetic (#8225)LazyFrame (#8220)DataFrame init from nested dataclass, pydantic, and NamedTuple objects (#8185)approx_unique() (#7937)describe methods (#8169)DataFrame init from pydantic model data (#8178)strptime/strftime args (#8221)Expr.list to implode (#8165)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @avimallu, @borchero, @chitralverma, @clickingbuttons, @ghuls, @josh, @jvdd, @rben01, @ritchie46, @stinodego and @universalmind303
Enhanced parametric testing DataFrame generation
DataFrame generation (#8149)pct_change (#8137)log1p to list of mathematical functions (#8102)import polars speed (#8151)map lenghts (#8147)UInt64 values that exceed Int64 upper bound (#8146)is_in (#8139)DataFrame generation (#8149)Thank you to all our contributors for making this release possible! @alexander-beedie, @borchero, @dependabot, @dependabot[bot], @jonashaag, @ritchie46 and @stinodego
fix stacklevel of some deprecation warnings
n expression passed to Expr.head/tail (#8098)show_versions util (#8096)scan_parquet/ipc and fsspec (#8071)Thank you to all our contributors for making this release possible! @MarcoGorelli, @StefanBRas, @alexander-beedie, @josh, @n8henrie, @rben01, @ritchie46, @stinodego, @universalmind303 and @zundertj
don't create duplicate pivot names
toggle_string_cache to enable_string_cache (#7970)sort, top_k, sort_by, and arg_sort_by, raise if descending is a sequence and its length doesn't match the number of columns to sort by (#7957)time_unit/time_zone instead of tu/tz (#7910)struct, concat_str, and arg_sort_by (#7308)shift_and_fill and add default… (#7192)func to function (#7139)Series/Expr methods to keyword-only (#7860)FromParalleIter<Option<str>> for Utf8Chunked ~1.9x (#8058)~2.5x (#8057)~2x. (#8053)into_groups materialization ~-25% (#8036)~25% (#7980)DataFrame init from pyarrow RecordBatch objects, and improve init from Array (#8011)write_ipc to take file=None (returning BytesIO) (#7997)Config methods, reference POLARS_MAX_THREADS in threadpool_size docstring (#7965)struct, concat_str, and arg_sort_by (#7308)sort, top_k, sort_by, and arg_sort_by, raise if descending is a sequence and its length doesn't match the number of columns to sort by (#7957)toggle_string_cache to enable_string_cache (#7970)time_unit/time_zone instead of tu/tz (#7910)shift_and_fill and add default… (#7192)func to function (#7139)Series/Expr methods to keyword-only (#7860)Thank you to all our contributors for making this release possible! @MarcoGorelli, @StefanBRas, @alexander-beedie, @ghuls, @rben01, @ritchie46, @stinodego and @universalmind303
improve group\_tuples of high cardinality data ~10%
~10% (#7938)f to function in reduce docstring (#7925)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @alonme, @ankane, @dependabot, @dependabot[bot], @lorentzenchr, @rben01, @ritchie46 and @zundertj
auto-infer detecting time-zone-awareness of fmt argument in strptime; deprecate tz\_aware argument
Series.pow() (#7898)lit values (#7879)write_excel (#7871)Unknown dtype to proceed as if dtype is None, to allow inference (#7830)to_repr methods to DataFrame and Series (#7802)DataFrame (#7775)map_dict. (#7797)from_repr function that reconstructs a DataFrame from its table repr (#7781)aggregation_function being 'first' in pivot. In a future version, it will default to None (#7784)check_exact for temporal types in assert_series_equal (#7896)_repr_html_ escapes column names in addition to data/body elements (#7877)is_between (#7835)venv folder to .venv (#7790)make requirements option to install/refresh dependencies without having to recreate the venv (#7792)ruff target version (#7791)Thank you to all our contributors for making this release possible! @LdRoW, @MarcoGorelli, @Newtoniano, @advoet, @alexander-beedie, @duskmoon314, @foxcroftjn, @ghuls, @jonashaag, @ritchie46, @stinodego and @zundertj
raise error on invalid categorical cast
Datetime or Duration dtype timeunit (#7768)Thank you to all our contributors for making this release possible! @alexander-beedie, @ghuls, @ritchie46 and @universalmind303
runtime SIMD target detection for min/max/sum and impl SIMD mean ~2-5x
min/max/sum and impl SIMD mean ~2-5x (#7702)write_excel that adds a row-wise total column using structured references (#7751)min/max (#7742)concat_list (#7745)Series.hist (#7727)qcut (#7724)maintain_order option to Series.cut (#7723)maintain_order in arr.unique (#7721)DataFrame.top_k/ LazyFrame.top_k (#7720)set_fmt_float value in Config load/save state (#7696)add operator-equivalent expression (#7667)is_leap_year to temporal expressions (#7618)scan_csv to take a list of column names in a new_columns param (#7642)groupby/unique of groupby on integer keys (#7604)is_in expressions (#7613)Series init regression from list of np.arange objects (#7692)__version__ attribute (#7680)Series init with integer 1/0 values (#7619)Expr.pipe API docs link (#7734)wrap_x utils to utils module (#7672)expr parsing to utils (#7661)internals (#7650)internals (#7649)Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @borchero, @chitralverma, @didriksg, @ghuls, @jakob-keller, @minimav, @ritchie46, @stinodego, @universalmind303 and @zundertj
optimize string kernels, (elide redundant allocs)
str_replace for same length replacements ~2x (#7580)DataFrame init by implementing dynamic singledispatch registration (#7559)str.replace_n and add n argument ~10x (#7575)replace_literal_all of single byte replacements ~15x. (#7565)DataFrame init by implementing dynamic singledispatch registration (#7559)head/tail (#7554)BatchedCsvReader from public API (#7546)internals (#7597)lru_cache to the apply docstrings (#7593)pli in type hints (part 2) (#7587)pli in type hints (part 1) (#7586)fmt tests to test_fmt (#7555)sep arg to separator (#7533)Thank you to all our contributors for making this release possible! @MarcoGorelli, @Vincenthays, @alexander-beedie, @ritchie46, @stinodego, @universalmind303 and @vincev
One column per quarter.