NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
crates.io
crates.io, #4294 most downloaded on crates.io
Unofficial implementation of Apache Arrow spec in safe Rust
Latest stable
0.18.0
Last release 3 years ago
no release in 18 months
Ships fairly regularly
a new release about every 6 weeks
Most releases are documented
notes for 28 of 38 stable releases
Nothing withdrawn
no release was ever pulled
5 years old
38 releases · first in 2021
Drag across the ruler to compare, or tap a release to open it.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Fixed writing nested parquet by @jorgecarleitao in https://github.com/jorgecarleitao/arrow2/pull/1390
bloom_filter module from parquet2 crate by @ozgrakkurt in https://github.com/jorgecarleitao/arrow2/pull/1420set_len method to Buffer by @haixuanTao in https://github.com/jorgecarleitao/arrow2/pull/1374num_values public by @b41sh in https://github.com/jorgecarleitao/arrow2/pull/1431len/len_proxy consistent with Offsets by @ritchie46 in https://github.com/jorgecarleitao/arrow2/pull/1434&[u8] as BooleanArray by @ritchie46 in https://github.com/jorgecarleitao/arrow2/pull/1436Full Changelog: https://github.com/jorgecarleitao/arrow2/compare/v0.16.0...v0.17.0
Breaking changes:
len/len_proxy consistent with Offsets #1434 (ritchie46)New features:
Fixed bugs:
parquet_read panics when working with date64s #1400List<u32> #1368Enhancements:
bloom_filter module from parquet2 crate #1420 (ozgrakkurt)set_len method to Buffer #1374 (haixuanTao)Documentation updates:
A new release is here! Thank you everyone that contributed to it! 🙇
A new release is here! Thank you everyone that contributed to it! 🙇
Breaking changes:
GrowableUnion #1360 (jleibs)Fixed bugs:
estimated_size_bytes #1351 (jleibs)Enhancements:
take of FixedSizeListArray #1386 (kylebarron)factory argument on parquet read functions to reader_factory #1380 (ozgrakkurt)Utf8Array::apply_validity #1367 (Arty-Maly)Documentation updates:
OffsetsBuffer docs #1373 (DzenanJupic)Testing updates:
A new release is here, adding a number of new features and improvements to arrow2. Thank you to everyone that contributed to it!
A new release is here, adding a number of new features and improvements to arrow2. Thank you to everyone that contributed to it!
This release adds support to a new format, the "record" JSON format, contributed by @AnIrishDuck, a new trait TryExtendFromSelf to efficiently concatenate an array into an existing mutable array, and multiple improvements by @sundy-li and @ritchie46 to performance. Finally, we have a new API OffsetsBuffer and Offsets proposed by @ritchie46 to allow creating variable sized-arrays without having to check for offsets.
This release also features a number of contributions from first contributors:
Thank you everyone for the great work this year, and happy festivities everyone!
Breaking changes:
MutableBinaryArray::reserve #1277from_data from all arrays #1328 (jorgecarleitao)Offsets and OffsetsBuffer #1316 (jorgecarleitao)New features:
TryExtendFromSelf #1278 (jorgecarleitao)Fixed bugs:
Enhancements:
UnionArray #1331 (jorgecarleitao)~-64-73% #1305 (ritchie46)reserve to pushable containers in parquet extend_from_decoder #1301 (ritchie46)MutableBinaryValuesArray #1276 (jorgecarleitao)Documentation updates:
Added MutableUtf8ValuesArray \#1260 (jorgecarleitao)
New features:
Fixed bugs:
Testing updates:
MutableUtf8Array::as_box #1266 (jorgecarleitao)A couple of backward-compatible bug fixes and improvements that everyone benefits from :)
A couple of backward-compatible bug fixes and improvements that everyone benefits from :)
Thank you @cjermain, @shaeqahmed and @ozgrakkurt! 🙇
Fixed bugs:
try_push_valid for nested avro deserialization #1248 (shaeqahmed)Enhancements:
BinaryArray::into_mut and double-ended support for its iterator #1255 (ozgrakkurt)Testing updates:
Fixed bugs:
try_push_valid for nested avro deserialization #1248 (shaeqahmed)Enhancements:
BinaryArray::into_mut and double-ended support for its iterator #1255 (ozgrakkurt)Testing updates:
Another release of arrow2 is here!
Another release of arrow2 is here!
Besides API improvements to reading IPC and parquet, there are two main new features, the ability to memory map arrow files (check out https://jorgecarleitao.github.io/arrow2/v0.14.0/guide/io/ipc_mmap.html) and support for decimal 256.
The following had their first time contribution to their crate:
Thank you everyone for all the issues, PRs and ideas!
Breaking changes:
Count (parquet statistics) #1217 (jorgecarleitao)FileReader #1216 (jorgecarleitao)New features:
FileReader #1215 (jorgecarleitao)MutableStructArray #1196 (hohav)Fixed bugs:
FileSink now closes the underlying writer. #1213 (samkaufman)Enhancements:
MutableStruct::push #1223 (hohav)mmap #1205 (ritchie46)MutableArray::reserve #1202 (jorgecarleitao)Documentation updates:
TimeUnit. #1238 (TurnOfACard)Testing updates:
Thanks @daniel-martinez-maqueda-sap!
Thanks @daniel-martinez-maqueda-sap!
Fixed bugs:
A new version (0.13) is now available on crates.io! 🎉🎉🎉🎉
A new version (0.13) is now available on crates.io! 🎉🎉🎉🎉
This is another large release of arrow2. Among the many, many changes (see below), it is worth noting:
(a + b) * 2 by a factor of 2-10xf16Thank you to the numerous contributors, both via PRs and issues, that resulted in this fantastic release 🙇
Breaking changes:
nested argument of array_to_pages non-owning #1174Result by panic in boolean comparison #1159 (jorgecarleitao)NativeType::to_ne_bytes #1112 (jorgecarleitao)with_validity #1104 (jorgecarleitao)unsafe in FFI #1100 (jorgecarleitao)Buffer::into_mut and make_mut functions #1089 (jorgecarleitao)Bitmap::null_count to Bitmap::unset_bits #1087 (jorgecarleitao)chunk_size optional in parquet's column_iter_to_arrays #1055 (jorgecarleitao)Arc<dyn Array> to Box<dyn Array> #1042 (jorgecarleitao)New features:
FixedSizeBinary to Avro #1118 (jorgecarleitao)_metadata sidecar #1063 (jorgecarleitao)Fixed bugs:
FnOnce for PrimitiveArray::apply_validity #1176 (ritchie46)Struct<List<...>> from parquet #1150 (jorgecarleitao)FixedSizeBinaryArray::values_iter #1117 (jorgecarleitao)FixedSizeListArray::new_null #1114 (jorgecarleitao)Enhancements:
MutableUtf8Array::into_data #1170 (ritchie46)Default for Utf8Array #1169 (ritchie46)MutableFixedSizeListArray::new_from #1161 (hohav)Default constraint #1157 (hohav)Buffer #1152 (sundy-li)Bytes to own crate #1141 (jorgecarleitao)O(N) clone in writing CSV #1128 (jorgecarleitao)row_group::get_field_columns public #1110 (ritchie46)~12%) #1105 (ritchie46)io::flight #1094 (jorgecarleitao)i64 into u32 written by pyarrow #1090 (jorgecarleitao)assign_ops #1085 (jorgecarleitao)Bitmap::make_mut with offset #1079 (jorgecarleitao)Default for PrimitiveArray #1073 (ritchie46)Buffer #1072 (ritchie46)compute::arity_assign #1070 (jorgecarleitao)Null from/to every type #1066 (jorgecarleitao)Documentation updates:
BooleanArray #1083 (jorgecarleitao)Testing updates:
u32 in example #1162 (jorgecarleitao)Bytes (internal) #1099 (jorgecarleitao)A new version of arrow2 is now available in crates.io. 🎉🎉🎉
A new version of arrow2 is now available in crates.io. 🎉🎉🎉
See below all great things that were released 🚀. But before that, thank you so much to everyone that contributed to this release: 🙇
@ahmedriza, @dexterduck, @GPSnoopy, @HaoYang670, @SimonSchneider, @TurnOfACard, @aptr322, @arxra, @b41sh, @cjermain, @dbr, @jorgecarleitao, @ritchie46
Breaking changes:
from_trusted_len_* from Buffer #1020 (jorgecarleitao)fn Offset::is_large() as const Offset::IS_LARGE #1002 (HaoYang670)ArrowError to Error #993 (jorgecarleitao)New features:
MapArray from parquet #1045 (jorgecarleitao)build_compare_fn #1016 (b41sh)Fixed bugs:
ListArray does not preserve all values #1008FixedSizedBinary from c data interface (FFI) #1026 (jorgecarleitao)NaN to json #990 (SimonSchneider)ListArray to parquet #984 (jorgecarleitao)Enhancements:
Debug and PartialEq for MapArray #1043 (jorgecarleitao).arced/.boxed to arrays #1040 (jorgecarleitao)parquet_to_arrow_schema public #1006 (martingallagher)min_max_boolean for the case where all values are null #1005 (HaoYang670)min_max_string and min_max_binary #1004 (HaoYang670)build_compare #998 (GPSnoopy)Documentation updates:
MutableBitmap #1025 (jorgecarleitao)bitmap #1022 (jorgecarleitao)PrimitiveArray and Utf8Array #1017 (jorgecarleitao)Testing updates:
cargo-llvm-cov #1028 (jorgecarleitao)tests #1001 (jorgecarleitao)Added support to append to existing IPC Arrow file \#972 (jorgecarleitao)
New features:
Fixed bugs:
async read of IPC metadata #969 (jorgecarleitao)Enhancements:
FileWriter without writing to the file #970 (jorgecarleitao)Fixed bug in writing csv with buffer resizing \#965 (ritchie46)
This release is mainly focus on improving upon the previous one on better parquet support. In particular, we have the main ingredients to read indexed
Arrow2 v0.11.0 is out!! 🎉🎉🎉
This release is mainly focus on improving upon the previous one on better parquet support. In particular, we have the main ingredients to read indexed parquet pages, which allow skipping deserializing individual pages, and since this version parquet files are written with page indexes. There is still some work to improve the frontend API to skip pages via statistics, which will be left for the next version.
This version also contains multiple bug fixes.
Thanks everyone that contributed to this release (individual PRs below)! 🙇
Breaking changes:
Send + Sync #947 (jorgecarleitao)New features:
Fixed bugs:
exceptions.ArrowErrorException: NotYetImplemented("Can't read Dictionary(UInt32, LargeUtf8, false) from parquet") #955lz4 compression fails with OutOfSpec("Thrift out of range") #940Enhancements:
Documentation updates:
Testing updates:
Added support to write StructArray to Avro \#909 (jorgecarleitao)
New features:
StructArray to Avro #909 (jorgecarleitao)ListArray to Avro #908 (jorgecarleitao)Fixed bugs:
FixedSizeBinaryArray::new_null #914 (jorgecarleitao)Enhancements:
capacity to some mutable arrays and tests #913 (jorgecarleitao)sum, min and max for extension and decimal #907 (jorgecarleitao)Testing updates:
Continuing breaking ground, this constitutes one of the most feature rich releases of this crate so far!
Arrow2 0.10.0 is out! 🚀🚀🚀🚀🚀
Continuing breaking ground, this constitutes one of the most feature rich releases of this crate so far!
Thank you to everyone for the impressive work over the past 2.5 months that make arrow2 so feature rich, safe, fast, and easy to use! :bow:
Here are the main headlines:
So far, whenever we applied a transformation to an array, we had to create a new array. When multiple operations were used (e.g. c1 x 2 + 1), it lead to the following compute pattern:
1. allocate new region
2. compute
3. allocate new region
4. compute
This was identified by @sundy-li on #741 and addressed by @ritchie46 on #794.
Users can now re-use Arced arrays, just like std::sync::Arc::get_mut. As expected, if the array is being used in multiple places, it will return a None and users do need to allocate a new region (exclusive mutability).
This is being used in Polars to further re-use allocated regions and therefore reduce both memory pressure and wasted compute cycles allocating new regions.
This release now supports reading from, and write to, any ODBC driver.
This builds on top of the superb odbc-api created by @pacman82, that allows this crate to use the columnar format provided by ODBC specification.
Given a performant ODBC driver, this is expected to be the fastest way to load data to the Arrow format, as many operations are simple memcopies.
Check out the example and guide for details on how to use it!
async support for writing to Arrow's IPCUntil now, we had limited support to writing to Arrow IPC asynchronously. @dexterduck closed this gap on #878, offering complete async support for both Arrow files and Arrow streams, including implementations of futures::Stream and futures::Sink for them!
std::simdAfter some back and forth with the working group of the project portable simd, this release replaces packed_simd2 by std::simd. This resulted in no performance difference but allow us to leverage the great work that is happening on std::simd.
A common pain point in using arrow2's logical types is that they are quite rich, making them sometimes difficult
to visualize or represent in e.g. JSON. @houqp closed this with #858, that adds compatibility with Serde for
schema-related structs in this crate (PhysicalType DataType, Field, Schema).
Arrow has an experimental specification for an FFI to iterators of arrow arrays. This release now fully supports this interface.
deny(missing_docs)This makes us developers more conscious about documenting APIs, thereby allowing users more context about them. We have also start documenting IO-related APIs over whether they are CPU or IO-bounded, so that users know which ones block async contexts.
Breaking changes:
Ffi_ArrowArray and Ffi_ArrowSchema #859New features:
GrowableUnion #902 (jorgecarleitao)months_days_ns #900 (jorgecarleitao)hash of month_day_ns arrays #899 (jorgecarleitao)futures::Sink for parquet async writer #877 (dexterduck)try_new and new to all arrays #873 (jorgecarleitao)Decimal128 to Avro #837 (potter420)LargeUtf8 and LargeBinary to Avro #828 (illumination-k)BooleanArray::from_trusted_len_values_iter_unchecked #799 (ritchie46)MutableUtf8Array::extend_values #798 (ritchie46)Buffer, Bitmap and some arrays #794 (ritchie46)Fixed bugs:
#derive(Clone) for FFI structs #882 (jorgecarleitao)Enhancements:
FixedSizeBinaryScalar #782FixedSizeList::try_push_valid public and added new_with_field #887 (ncpenke)MutableFixedList::mut_values #886 (jorgecarleitao)try_new #879 (jorgecarleitao)ListValuesIter #874 (ritchie46)into_mut implementations #801 (ritchie46)FixedSizeListScalar and FixedSizeBinaryScalar #786 (illumination-k)Documentation updates:
deny(missing_docs) #808 (jorgecarleitao)Bitmap::set_bit #802 (yjshen)dyn Array::slice docstring #792 (ritchie46)Testing updates:
unsafe #843 (jorgecarleitao)Nothing published for this version
Added support for compare dictionary-encoded with scalar \#686 (jorgecarleitao)
New features:
Fixed bugs:
None as ipc_fields in flight API #780 (jorgecarleitao)Enhancements:
Testing updates:
A new release is here! 🎉🎉🎉🎉 This release has four major improvements:
A new release is here! 🎉🎉🎉🎉 This release has four major improvements:
Vec, thus making it
unsafesync and asyncplanus, a re-implementation of the flatbuffers specification in Rust (you should check out that project, awesome work by @kristoff3r and @TethysSvensson)
unsound#[forbid(unsafe)]unsafe via bytemuck's dependencypanic-free, to reduce risks of DOS from untrusted dataA big thanks to all contributors (listed below) and our users for all the dedication, hard work, and patience. 🙇
Breaking changes:
nullif #753 (jorgecarleitao)Schema and Field #728 (jorgecarleitao)RecordBatch by Chunk #717 (jorgecarleitao)Option from fields' metadata #715 (jorgecarleitao)Field to DataType::Dictionary #711 (jorgecarleitao)Block and CompressedBlock #698 (jorgecarleitao)Display by Debug for Array #694 (jorgecarleitao)MutableBuffer by std::Vec #693 (jorgecarleitao)Utf8Scalar and BinaryScalar #660 (jorgecarleitao)New features:
and_scalar and or_scalar for boolean_kleene #662lower and upper support for string #635any and all kernel #739 (ritchie46)and_scalar and or_scalar for boolean kleene #723 (silathdiir)and_scalar and or_scalar for boolean #707 (silathdiir)upper #664 (Xuanwo)lower #641 (Xuanwo)async read of Avro #620 (jorgecarleitao)Fixed bugs:
Scalar: Send+Sync #644 (jorgecarleitao)Enhancements:
FixedBinaryArray #757 (jorgecarleitao)unsafe #756 (jorgecarleitao)unsafe #755 (jorgecarleitao)#[forbid(unsafe)] #749 (jorgecarleitao)use Array to access primitives' data_type #697 (jorgecarleitao)avro-rs to avro-schema #692 (jorgecarleitao)MutablePrimitiveArray::extend_constant #689 (jorgecarleitao)dyn Array and Scalar usable in #[derive(PartialEq)] #680 (jorgecarleitao)Buffer::offset public #652 (ritchie46)Filter: Send+Sync #645 (jorgecarleitao)String #643 (jorgecarleitao)Documentation updates:
Bitmap::as_slice #654 (ritchie46)Testing updates:
Nothing published for this version
Fixed compilation with individual features activated \#642 (ritchie46)
This release has so many important new features and bug fixes that will be summarized as: thank you everyone for all the issues and PRs that resulted
A new release is here 🚀🚀🚀
This release has so many important new features and bug fixes that will be summarized as: thank you everyone for all the issues and PRs that resulted in this release (in order of appearance) 🙇🙇🙇🙇:
Breaking changes:
compression to IpcWriteOptions #570cast accept CastOptions parameter #569ArrowError #640 (jorgecarleitao)DynComparator for lexsort and partition #637 (yjshen)Operator #604 (jorgecarleitao)UnionMode enum #557 (simonvandel)New features:
StructArray #613 (illumination-k)NullArray to all other types #589 (flaneur2020)FixedSizeList and FixedSizeBinary #565 (jorgecarleitao)async csv reading. #562 (jorgecarleitao)bitwise operations #553 (1aguna)StructArray from parquet #547 (jorgecarleitao)Fixed bugs:
io::ipc::read::reader::FileReader #596 (illumination-k)Enhancements:
take with null values (2-3x) #633 (jorgecarleitao)to conversion to FixedSizeBinary #622 (ritchie46)confy-table #618 (jorgecarleitao)MutableArray Send + Sync #617 (jorgecarleitao)is_not_null. #600 (jorgecarleitao)len to every array #599 (jorgecarleitao)NullArray at FFI. #598 (jorgecarleitao)MutableBinaryArray #597 (jorgecarleitao)bitmap::from_trusted (3x) #578 (jorgecarleitao)Utf8Array from MutableUtf8Array #560 (jorgecarleitao)Documentation updates:
Testing updates:
RUSTSEC-2020-0159 #595 (jorgecarleitao)Nothing published for this version
As usual, a bunch of optimizations as well as some work in two main fronts:
Another release is here 🚀🚀🚀
As usual, a bunch of optimizations as well as some work in two main fronts:
Thank you to all contributors (names below) for the amazing contributions!
Breaking changes:
FileReader to own the underlying reader #518 (blakesmith)arrow_format crate #517 (jorgecarleitao)New features:
checked_negate and wrapping_negate for PrimitiveArray #506 (yjhmelody)Fixed bugs:
StructArray to json #511 (VasanthakumarV)Enhancements:
BinaryValueIter #526 (yjhmelody)Documentation updates:
Testing updates:
utf8 #543 (jorgecarleitao)Small release with two minor but relevant bug fixes and a new feature.
Small release with two minor but relevant bug fixes and a new feature.
New features:
Fixed bugs:
try_push_valid public again #509 (ritchie46)Enhancements:
(in crates as 0.6.1: I made a mistake in publishing). Anyways, another big release is here!
(in crates as 0.6.1: I made a mistake in publishing). Anyways, another big release is here!
There are just too many improvements for a 22 days release - let's try to capture important mentions:
Buffer and MutableBuffer are now compatible with Rust's std::Vec with no strings attached: everything continues to work, including FFI with the rest of the ecosystem! You can recover the previous behavior (of using cached-aligned allocations), via feature cache_alignedio_avroBreaking changes:
MutableFixedSizeListArray to the spec used by the rest of the Mutable API #475ALIGNMENT invariant from [Mutable]Buffer #449compute::arithemtics::basic #461 (jorgecarleitao)&Option<Bitmap> to Option<&Bitmap>. #431 (jorgecarleitao)FileWriter to own the writer. #420 (yjshen)DynComparator Send+Sync #414 (yjshen)New features:
List,Enum, Duration and Fixed. #493 (jorgecarleitao)Decimal from parquet #489 (potter420)BitXor trait for Bitmap #485 (houqp)extend/extend_unchecked for MutableBooleanArray #478 (VasanthakumarV)shrink_to_fit to mutable arrays #467 (ritchie46)DataType::Map and MapArray #464 (jorgecarleitao)std::Vec. #385 (jorgecarleitao)Fixed bugs:
unsafe #458 (jorgecarleitao)MutableBitmap #393 (jorgecarleitao)Enhancements:
compare function are not exported #349hash work for extension type #487 (jorgecarleitao)extend/extend_unchecked for MutableBinaryArray #486 (VasanthakumarV)GrowableFixedSizeList and improved MutableFixedSizeListArray #470 (jorgecarleitao)MutableBitmap::shrink_to_fit #468 (jorgecarleitao)MutableArray::as_box #450 (sd2k)assert from MutableBuffer::set_len #443 (ritchie46)null_count #442 (ritchie46)PrimitiveGrowable for nulls (-10%) #434 (jorgecarleitao)Array #432 (jorgecarleitao)ahash and multiversion (-30%) #428 (Dandandan)Utf8Array and BinaryArray (3-4x) #427 (jorgecarleitao)simdutf8 (-40%) #426 (Dandandan)extend/extend_unchecked for MutableUtf8Array #413 (VasanthakumarV)io_csv_read and io_csv_write feature #408 (ritchie46)comparison docs and re-export the array-comparing function #404 (HagaiHargil)Array::with_validity #399 (ritchie46)Documentation updates:
io module has no submodules #390full #391 (jorgecarleitao)Testing updates:
compute #474 (yjhmelody)src/compute/ to tests/ #423 (VasanthakumarV)unsafe #403 (jorgecarleitao)Nothing published for this version
A new release is here, containing bug fixes and backward-compatible enhancements.
A new release is here, containing bug fixes and backward-compatible enhancements.
Thank you to all involved in the testing and development that resulted in this version!
New features:
Fixed bugs:
MutableBitmap #393 (jorgecarleitao)Enhancements:
Array::with_validity #399 (ritchie46)Testing updates:
Hot fix release to make the API docs contain all optional features.
Hot fix release to make the API docs contain all optional features.
Documentation updates:
io module has no submodules #390full #391 (jorgecarleitao)\[0.5\] The docs io module has no submodules \#390
Documentation updates:
io module has no submodules #390full #391 (jorgecarleitao)As usual, there is a small number of backward incompatible changes. See associated issues below, which include the migration paths to each of them.
A new release is here! 🎉🎉🎉
This one marked by further alignment with the arrow specification. Of special mention:
async parquet write (by @GrandChaman)extend_*values to MutablePrimitiveArray (by @ritchie46)BinaryArray(by @zhyass)MONTH_DAY_NANO interval type (by @jorgecarleitao)cargo features not default (by @jorgecarleitao)As usual, there is a small number of backward incompatible changes. See associated issues below, which include the migration paths to each of them.
Breaking changes:
Extension to DataType #361MonthDayNano added to enum IntervalUnit #360io::parquet::write::write_* return size of file in bytes #354bitmap::utils::null_count to bitmap::utils::count_zeros #342GroupFilter optional in parquet'sRecordReader and added method to set it. #386 (jorgecarleitao)PartialOrd and Ord of all enums in datatypes #379 (jorgecarleitao)cargo features not default #369 (jorgecarleitao)New features:
async parquet write #372 (GrandChaman)Extension (logical) type #359 (jorgecarleitao)BinaryArray #346 (zhyass)MONTH_DAY_NANO interval type #268 (jorgecarleitao)Fixed bugs:
parquet_read fails when a column has too many rows with string values #366parquet_read panics with index_out_of_bounds #351MutableBitmap::push_unchecked #384 (jorgecarleitao)Enhancements:
extend_*values to MutablePrimitiveArray #383 (ritchie46)lexical-core #378 (jorgecarleitao)Send+Sync to MutableBuffer #368 (jorgecarleitao)num #353 (jorgecarleitao)Documentation updates:
Testing updates:
tests/ #389 (jorgecarleitao)As usual, there is a small number of backward incompatible changes. The associated issues include the migration paths.
A new release is here! 🎉🎉🎉
This one marked by a lot of enhancements to existing functionality. Of special mention:
multiversion, lexical-core and serde-derive dependencies optionalUnionArray (including FFI and IPC tests)Field(full list below)
As usual, there is a small number of backward incompatible changes. The associated issues include the migration paths.
Finally, thank you to all contributors and reporters 🙇 In particular, thank you to polars and datafuse teams for the 🐛 reports. They help tremendously 💯
Breaking changes:
Arrays of one element to Scalars #335*_compare_scalar not return Result #316io::print, get_value_display and get_display not return Result #286MetadataVersion to IPC interfaces #282DataType::Union to enable round trips in IPC #281StructArray -> RecordBatch #307 (jorgecarleitao)BitmapIter #296 (jorgecarleitao)New features:
Field #321 (jorgecarleitao)compare_scalar #317 (jorgecarleitao)UnionArray #283 (jorgecarleitao)Fixed bugs:
compute::cast and compute::can_cast #295 (jorgecarleitao)days_ms / Interval(DayTime) #285 (jorgecarleitao)Enhancements:
multiversion and lexical-core optional #324 (jorgecarleitao)TryPush #314 (jorgecarleitao)date32 -> i64 and date64 -> i32 #308 (ritchie46)BinaryArray #304 (jorgecarleitao)MutableFixedSizeBinaryArray #303 (jorgecarleitao)MutablePrimitiveArray and MutableUtf8Array #299 (jorgecarleitao)MutableBooleanArray #297 (jorgecarleitao)io::print::write #287 (jorgecarleitao)serde_derive dependency #279 (jorgecarleitao)wasm on io_print feature. #276 (jorgecarleitao)rem_scalar/div_scalar for integer types (4x-10x) #275 (ritchie46)Documentation updates:
Testing updates:
Closed issues:
This is the first Rust implementation to support reading parquet on wasm, which opens up a whole new range of possibilities by allowing reading parque
This is the first Rust implementation to support reading parquet on wasm, which opens up a whole new range of possibilities by allowing reading parquet directly on a browser without having to communicate data to a server.
This is also the first time an implementation of arrow in Rust offers native async support to read parquet, enabling consumers to perform ranged queries against blob storage without blocking. Check out an example here
Finally, this implementation now offers a Scalar API that we can build upon to support arrow's more exotic types such as maps and unions.
Thanks a lot to @sundy-li , @PsiACE , @ritchie46, @ghuls and @Dandandan for the contributions and discussions and to @Darksonn for all the patience and help that unblocked me when working on the async parquet support.
Breaking changes:
sum to sum_primitive #273Index from array::Index to types::Index #272projection to IPC FileReader #271page_filter to parquet's RecordReader and get_page_iterator #270CompressionCodec to Compression #269New features:
Fixed bugs:
Enhancements:
sort::sort_to_indices with limit option in arrow2 #245Documentation updates:
Closed issues:
Simplified new signature of growable API \#238 (jorgecarleitao)
Breaking changes:
new signature of growable API #238 (jorgecarleitao)New features:
Fixed bugs:
Enhancements:
unsafe #225 (jorgecarleitao)Documentation updates:
* This Changelog was automatically generated by github_changelog_generator
Nothing published for this version
One column per quarter.