NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
crates.io · #3953 most downloaded on crates.io
Unofficial implementation of Apache Arrow spec in safe Rust
Last release 3 years ago
no release in 18 months
Ships fairly regularly
a new release about every 6 weeks
Most releases are documented
notes for 27 of 38 stable releases
Nothing withdrawn
no release was ever pulled
5 years old
38 releases · first in 2021
One column per quarter.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fixed writing nested parquet by @jorgecarleitao in #1390
bloom_filter module from parquet2 crate by @ozgrakkurt in #1420set_len method to Buffer by @haixuanTao in #1374num_values public by @b41sh in #1431len/len_proxy consistent with Offsets by @ritchie46 in #1434&[u8] as BooleanArray by @ritchie46 in #1436Full Changelog: v0.16.0...v0.17.0
Breaking changes:
len/len_proxy consistent with Offsets #1434 (ritchie46)New features:
Fixed bugs:
parquet_read panics when working with date64s #1400List<u32> #1368Enhancements:
bloom_filter module from parquet2 crate #1420 (ozgrakkurt)set_len method to Buffer #1374 (haixuanTao)Documentation updates:
A new release is here! Thank you everyone that contributed to it! 🙇
A new release is here! Thank you everyone that contributed to it! 🙇
Breaking changes:
GrowableUnion #1360 (jleibs)Fixed bugs:
estimated_size_bytes #1351 (jleibs)Enhancements:
take of FixedSizeListArray #1386 (kylebarron)factory argument on parquet read functions to reader_factory #1380 (ozgrakkurt)Utf8Array::apply_validity #1367 (Arty-Maly)Documentation updates:
OffsetsBuffer docs #1373 (DzenanJupic)Testing updates:
A new release is here, adding a number of new features and improvements to arrow2. Thank you to everyone that contributed to it!
A new release is here, adding a number of new features and improvements to arrow2. Thank you to everyone that contributed to it!
This release adds support to a new format, the "record" JSON format, contributed by @AnIrishDuck, a new trait TryExtendFromSelf to efficiently concatenate an array into an existing mutable array, and multiple improvements by @sundy-li and @ritchie46 to performance. Finally, we have a new API OffsetsBuffer and Offsets proposed by @ritchie46 to allow creating variable sized-arrays without having to check for offsets.
This release also features a number of contributions from first contributors:
Thank you everyone for the great work this year, and happy festivities everyone!
Breaking changes:
MutableBinaryArray::reserve #1277from_data from all arrays #1328 (jorgecarleitao)Offsets and OffsetsBuffer #1316 (jorgecarleitao)New features:
TryExtendFromSelf #1278 (jorgecarleitao)Fixed bugs:
Enhancements:
UnionArray #1331 (jorgecarleitao)~-64-73% #1305 (ritchie46)reserve to pushable containers in parquet extend_from_decoder #1301 (ritchie46)MutableBinaryValuesArray #1276 (jorgecarleitao)Documentation updates:
Added MutableUtf8ValuesArray \#1260 (jorgecarleitao)
New features:
Fixed bugs:
Testing updates:
MutableUtf8Array::as_box #1266 (jorgecarleitao)A couple of backward-compatible bug fixes and improvements that everyone benefits from :)
A couple of backward-compatible bug fixes and improvements that everyone benefits from :)
Thank you @cjermain, @shaeqahmed and @ozgrakkurt! 🙇
Fixed bugs:
try_push_valid for nested avro deserialization #1248 (shaeqahmed)Enhancements:
BinaryArray::into_mut and double-ended support for its iterator #1255 (ozgrakkurt)Testing updates:
Fixed bugs:
try_push_valid for nested avro deserialization #1248 (shaeqahmed)Enhancements:
BinaryArray::into_mut and double-ended support for its iterator #1255 (ozgrakkurt)Testing updates:
Another release of arrow2 is here!
Another release of arrow2 is here!
Besides API improvements to reading IPC and parquet, there are two main new features, the ability to memory map arrow files (check out https://jorgecarleitao.github.io/arrow2/v0.14.0/guide/io/ipc_mmap.html) and support for decimal 256.
The following had their first time contribution to their crate:
Thank you everyone for all the issues, PRs and ideas!
Breaking changes:
Count (parquet statistics) #1217 (jorgecarleitao)FileReader #1216 (jorgecarleitao)New features:
FileReader #1215 (jorgecarleitao)MutableStructArray #1196 (hohav)Fixed bugs:
FileSink now closes the underlying writer. #1213 (samkaufman)Enhancements:
MutableStruct::push #1223 (hohav)mmap #1205 (ritchie46)MutableArray::reserve #1202 (jorgecarleitao)Documentation updates:
TimeUnit. #1238 (TurnOfACard)Testing updates:
Thanks @daniel-martinez-maqueda-sap !
Thanks @daniel-martinez-maqueda-sap!
Fixed bugs:
A new version (0.13) is now available on crates.io! 🎉🎉🎉🎉
A new version (0.13) is now available on crates.io! 🎉🎉🎉🎉
This is another large release of arrow2. Among the many, many changes (see below), it is worth noting:
(a + b) * 2 by a factor of 2-10xf16Thank you to the numerous contributors, both via PRs and issues, that resulted in this fantastic release 🙇
Breaking changes:
nested argument of array_to_pages non-owning #1174Result by panic in boolean comparison #1159 (jorgecarleitao)NativeType::to_ne_bytes #1112 (jorgecarleitao)with_validity #1104 (jorgecarleitao)unsafe in FFI #1100 (jorgecarleitao)Buffer::into_mut and make_mut functions #1089 (jorgecarleitao)Bitmap::null_count to Bitmap::unset_bits #1087 (jorgecarleitao)chunk_size optional in parquet's column_iter_to_arrays #1055 (jorgecarleitao)Arc<dyn Array> to Box<dyn Array> #1042 (jorgecarleitao)New features:
FixedSizeBinary to Avro #1118 (jorgecarleitao)_metadata sidecar #1063 (jorgecarleitao)Fixed bugs:
FnOnce for PrimitiveArray::apply_validity #1176 (ritchie46)Struct<List<...>> from parquet #1150 (jorgecarleitao)FixedSizeBinaryArray::values_iter #1117 (jorgecarleitao)FixedSizeListArray::new_null #1114 (jorgecarleitao)Enhancements:
MutableUtf8Array::into_data #1170 (ritchie46)Default for Utf8Array #1169 (ritchie46)MutableFixedSizeListArray::new_from #1161 (hohav)Default constraint #1157 (hohav)Buffer #1152 (sundy-li)Bytes to own crate #1141 (jorgecarleitao)O(N) clone in writing CSV #1128 (jorgecarleitao)row_group::get_field_columns public #1110 (ritchie46)~12%) #1105 (ritchie46)io::flight #1094 (jorgecarleitao)i64 into u32 written by pyarrow #1090 (jorgecarleitao)assign_ops #1085 (jorgecarleitao)Bitmap::make_mut with offset #1079 (jorgecarleitao)Default for PrimitiveArray #1073 (ritchie46)Buffer #1072 (ritchie46)compute::arity_assign #1070 (jorgecarleitao)Null from/to every type #1066 (jorgecarleitao)Documentation updates:
BooleanArray #1083 (jorgecarleitao)Testing updates:
u32 in example #1162 (jorgecarleitao)Bytes (internal) #1099 (jorgecarleitao)A new version of arrow2 is now available in crates.io . 🎉🎉🎉
A new version of arrow2 is now available in crates.io. 🎉🎉🎉
See below all great things that were released 🚀. But before that, thank you so much to everyone that contributed to this release: 🙇
@ahmedriza, @dexterduck, @GPSnoopy, @HaoYang670, @SimonSchneider, @TurnOfACard, @aptr322, @arxra, @b41sh, @cjermain, @dbr, @jorgecarleitao, @ritchie46
Breaking changes:
from_trusted_len_* from Buffer #1020 (jorgecarleitao)fn Offset::is_large() as const Offset::IS_LARGE #1002 (HaoYang670)ArrowError to Error #993 (jorgecarleitao)New features:
MapArray from parquet #1045 (jorgecarleitao)build_compare_fn #1016 (b41sh)Fixed bugs:
ListArray does not preserve all values #1008FixedSizedBinary from c data interface (FFI) #1026 (jorgecarleitao)NaN to json #990 (SimonSchneider)ListArray to parquet #984 (jorgecarleitao)Enhancements:
Debug and PartialEq for MapArray #1043 (jorgecarleitao).arced/.boxed to arrays #1040 (jorgecarleitao)parquet_to_arrow_schema public #1006 (martingallagher)min_max_boolean for the case where all values are null #1005 (HaoYang670)min_max_string and min_max_binary #1004 (HaoYang670)build_compare #998 (GPSnoopy)Documentation updates:
MutableBitmap #1025 (jorgecarleitao)bitmap #1022 (jorgecarleitao)PrimitiveArray and Utf8Array #1017 (jorgecarleitao)Testing updates:
cargo-llvm-cov #1028 (jorgecarleitao)tests #1001 (jorgecarleitao)Added support to append to existing IPC Arrow file #972 ( jorgecarleitao )
New features:
Fixed bugs:
async read of IPC metadata #969 (jorgecarleitao)Enhancements:
FileWriter without writing to the file #970 (jorgecarleitao)Fixed bug in writing csv with buffer resizing \#965 (ritchie46)
Refactored parquet statistics deserialization \#962 (jorgecarleitao)
Breaking changes:
Send + Sync #947 (jorgecarleitao)New features:
Fixed bugs:
exceptions.ArrowErrorException: NotYetImplemented("Can't read Dictionary(UInt32, LargeUtf8, false) from parquet") #955lz4 compression fails with OutOfSpec("Thrift out of range") #940Enhancements:
Documentation updates:
Testing updates:
Added support to write StructArray to Avro \#909 (jorgecarleitao)
New features:
StructArray to Avro #909 (jorgecarleitao)ListArray to Avro #908 (jorgecarleitao)Fixed bugs:
FixedSizeBinaryArray::new_null #914 (jorgecarleitao)Enhancements:
capacity to some mutable arrays and tests #913 (jorgecarleitao)sum, min and max for extension and decimal #907 (jorgecarleitao)Testing updates:
Renamed Ffi_ArrowArray and Ffi_ArrowSchema \#859
Breaking changes:
Ffi_ArrowArray and Ffi_ArrowSchema #859New features:
GrowableUnion #902 (jorgecarleitao)months_days_ns #900 (jorgecarleitao)hash of month_day_ns arrays #899 (jorgecarleitao)futures::Sink for parquet async writer #877 (dexterduck)try_new and new to all arrays #873 (jorgecarleitao)Decimal128 to Avro #837 (potter420)LargeUtf8 and LargeBinary to Avro #828 (illumination-k)BooleanArray::from_trusted_len_values_iter_unchecked #799 (ritchie46)MutableUtf8Array::extend_values #798 (ritchie46)Buffer, Bitmap and some arrays #794 (ritchie46)Fixed bugs:
#derive(Clone) for FFI structs #882 (jorgecarleitao)Enhancements:
FixedSizeBinaryScalar #782FixedSizeList::try_push_valid public and added new_with_field #887 (ncpenke)MutableFixedList::mut_values #886 (jorgecarleitao)try_new #879 (jorgecarleitao)ListValuesIter #874 (ritchie46)into_mut implementations #801 (ritchie46)FixedSizeListScalar and FixedSizeBinaryScalar #786 (illumination-k)Documentation updates:
deny(missing_docs) #808 (jorgecarleitao)Bitmap::set_bit #802 (yjshen)dyn Array::slice docstring #792 (ritchie46)Testing updates:
unsafe #843 (jorgecarleitao)Nothing published for this version
Added support for compare dictionary-encoded with scalar \#686 (jorgecarleitao)
New features:
Fixed bugs:
None as ipc_fields in flight API #780 (jorgecarleitao)Enhancements:
Testing updates:
Added number of rows read in CSV inference \#765 (jorgecarleitao)
Breaking changes:
nullif #753 (jorgecarleitao)Schema and Field #728 (jorgecarleitao)RecordBatch by Chunk #717 (jorgecarleitao)Option from fields' metadata #715 (jorgecarleitao)Field to DataType::Dictionary #711 (jorgecarleitao)Block and CompressedBlock #698 (jorgecarleitao)Display by Debug for Array #694 (jorgecarleitao)MutableBuffer by std::Vec #693 (jorgecarleitao)Utf8Scalar and BinaryScalar #660 (jorgecarleitao)New features:
and_scalar and or_scalar for boolean_kleene #662lower and upper support for string #635any and all kernel #739 (ritchie46)and_scalar and or_scalar for boolean kleene #723 (silathdiir)and_scalar and or_scalar for boolean #707 (silathdiir)upper #664 (Xuanwo)lower #641 (Xuanwo)async read of Avro #620 (jorgecarleitao)Fixed bugs:
Scalar: Send+Sync #644 (jorgecarleitao)Enhancements:
FixedBinaryArray #757 (jorgecarleitao)unsafe #756 (jorgecarleitao)unsafe #755 (jorgecarleitao)#[forbid(unsafe)] #749 (jorgecarleitao)use Array to access primitives' data_type #697 (jorgecarleitao)avro-rs to avro-schema #692 (jorgecarleitao)MutablePrimitiveArray::extend_constant #689 (jorgecarleitao)dyn Array and Scalar usable in #[derive(PartialEq)] #680 (jorgecarleitao)Buffer::offset public #652 (ritchie46)Filter: Send+Sync #645 (jorgecarleitao)String #643 (jorgecarleitao)Documentation updates:
Bitmap::as_slice #654 (ritchie46)Testing updates:
Nothing published for this version
Fixed compilation with individual features activated \#642 (ritchie46)
Made CSV write options use chrono formatting by default \#624
Breaking changes:
compression to IpcWriteOptions #570cast accept CastOptions parameter #569ArrowError #640 (jorgecarleitao)DynComparator for lexsort and partition #637 (yjshen)Operator #604 (jorgecarleitao)UnionMode enum #557 (simonvandel)New features:
StructArray #613 (illumination-k)NullArray to all other types #589 (flaneur2020)FixedSizeList and FixedSizeBinary #565 (jorgecarleitao)async csv reading. #562 (jorgecarleitao)bitwise operations #553 (1aguna)StructArray from parquet #547 (jorgecarleitao)Fixed bugs:
io::ipc::read::reader::FileReader #596 (illumination-k)Enhancements:
take with null values (2-3x) #633 (jorgecarleitao)to conversion to FixedSizeBinary #622 (ritchie46)confy-table #618 (jorgecarleitao)MutableArray Send + Sync #617 (jorgecarleitao)is_not_null. #600 (jorgecarleitao)len to every array #599 (jorgecarleitao)NullArray at FFI. #598 (jorgecarleitao)MutableBinaryArray #597 (jorgecarleitao)bitmap::from_trusted (3x) #578 (jorgecarleitao)Utf8Array from MutableUtf8Array #560 (jorgecarleitao)Documentation updates:
Testing updates:
RUSTSEC-2020-0159 #595 (jorgecarleitao)Nothing published for this version
Simplified reading parquet \#532 (jorgecarleitao)
Breaking changes:
FileReader to own the underlying reader #518 (blakesmith)arrow_format crate #517 (jorgecarleitao)New features:
checked_negate and wrapping_negate for PrimitiveArray #506 (yjhmelody)Fixed bugs:
StructArray to json #511 (VasanthakumarV)Enhancements:
BinaryValueIter #526 (yjhmelody)Documentation updates:
Testing updates:
utf8 #543 (jorgecarleitao)Added wrapping version arithmetics for PrimitiveArray \#496 (yjhmelody)
New features:
Fixed bugs:
try_push_valid public again #509 (ritchie46)Enhancements:
Bring MutableFixedSizeListArray to the spec used by the rest of the Mutable API \#475
Breaking changes:
MutableFixedSizeListArray to the spec used by the rest of the Mutable API #475ALIGNMENT invariant from [Mutable]Buffer #449compute::arithemtics::basic #461 (jorgecarleitao)&Option<Bitmap> to Option<&Bitmap>. #431 (jorgecarleitao)FileWriter to own the writer. #420 (yjshen)DynComparator Send+Sync #414 (yjshen)New features:
List,Enum, Duration and Fixed. #493 (jorgecarleitao)Decimal from parquet #489 (potter420)BitXor trait for Bitmap #485 (houqp)extend/extend_unchecked for MutableBooleanArray #478 (VasanthakumarV)shrink_to_fit to mutable arrays #467 (ritchie46)DataType::Map and MapArray #464 (jorgecarleitao)std::Vec. #385 (jorgecarleitao)Fixed bugs:
unsafe #458 (jorgecarleitao)MutableBitmap #393 (jorgecarleitao)Enhancements:
compare function are not exported #349hash work for extension type #487 (jorgecarleitao)extend/extend_unchecked for MutableBinaryArray #486 (VasanthakumarV)GrowableFixedSizeList and improved MutableFixedSizeListArray #470 (jorgecarleitao)MutableBitmap::shrink_to_fit #468 (jorgecarleitao)MutableArray::as_box #450 (sd2k)assert from MutableBuffer::set_len #443 (ritchie46)null_count #442 (ritchie46)PrimitiveGrowable for nulls (-10%) #434 (jorgecarleitao)Array #432 (jorgecarleitao)ahash and multiversion (-30%) #428 (Dandandan)Utf8Array and BinaryArray (3-4x) #427 (jorgecarleitao)simdutf8 (-40%) #426 (Dandandan)extend/extend_unchecked for MutableUtf8Array #413 (VasanthakumarV)io_csv_read and io_csv_write feature #408 (ritchie46)comparison docs and re-export the array-comparing function #404 (HagaiHargil)Array::with_validity #399 (ritchie46)Documentation updates:
io module has no submodules #390full #391 (jorgecarleitao)Testing updates:
compute #474 (yjhmelody)src/compute/ to tests/ #423 (VasanthakumarV)unsafe #403 (jorgecarleitao)Nothing published for this version
Added support to read and write extension types to and from parquet \#396 (jorgecarleitao)
New features:
Fixed bugs:
MutableBitmap #393 (jorgecarleitao)Enhancements:
Array::with_validity #399 (ritchie46)Testing updates:
Nothing published for this version
\[0.5\] The docs io module has no submodules \#390
Documentation updates:
io module has no submodules #390full #391 (jorgecarleitao)Added Extension to DataType \#361
Breaking changes:
Extension to DataType #361MonthDayNano added to enum IntervalUnit #360io::parquet::write::write_* return size of file in bytes #354bitmap::utils::null_count to bitmap::utils::count_zeros #342GroupFilter optional in parquet'sRecordReader and added method to set it. #386 (jorgecarleitao)PartialOrd and Ord of all enums in datatypes #379 (jorgecarleitao)cargo features not default #369 (jorgecarleitao)New features:
async parquet write #372 (GrandChaman)Extension (logical) type #359 (jorgecarleitao)BinaryArray #346 (zhyass)MONTH_DAY_NANO interval type #268 (jorgecarleitao)Fixed bugs:
parquet_read fails when a column has too many rows with string values #366parquet_read panics with index_out_of_bounds #351MutableBitmap::push_unchecked #384 (jorgecarleitao)Enhancements:
extend_*values to MutablePrimitiveArray #383 (ritchie46)lexical-core #378 (jorgecarleitao)Send+Sync to MutableBuffer #368 (jorgecarleitao)num #353 (jorgecarleitao)Documentation updates:
Testing updates:
tests/ #389 (jorgecarleitao)Change dictionary iterator of values from Arrays of one element to Scalars \#335
Breaking changes:
Arrays of one element to Scalars #335*_compare_scalar not return Result #316io::print, get_value_display and get_display not return Result #286MetadataVersion to IPC interfaces #282DataType::Union to enable round trips in IPC #281StructArray -> RecordBatch #307 (jorgecarleitao)BitmapIter #296 (jorgecarleitao)New features:
Field #321 (jorgecarleitao)compare_scalar #317 (jorgecarleitao)UnionArray #283 (jorgecarleitao)Fixed bugs:
compute::cast and compute::can_cast #295 (jorgecarleitao)days_ms / Interval(DayTime) #285 (jorgecarleitao)Enhancements:
multiversion and lexical-core optional #324 (jorgecarleitao)TryPush #314 (jorgecarleitao)date32 -> i64 and date64 -> i32 #308 (ritchie46)BinaryArray #304 (jorgecarleitao)MutableFixedSizeBinaryArray #303 (jorgecarleitao)MutablePrimitiveArray and MutableUtf8Array #299 (jorgecarleitao)MutableBooleanArray #297 (jorgecarleitao)io::print::write #287 (jorgecarleitao)serde_derive dependency #279 (jorgecarleitao)wasm on io_print feature. #276 (jorgecarleitao)rem_scalar/div_scalar for integer types (4x-10x) #275 (ritchie46)Documentation updates:
Testing updates:
Closed issues:
Renamed sum to sum_primitive \#273
Breaking changes:
sum to sum_primitive #273Index from array::Index to types::Index #272projection to IPC FileReader #271page_filter to parquet's RecordReader and get_page_iterator #270CompressionCodec to Compression #269New features:
Fixed bugs:
Enhancements:
sort::sort_to_indices with limit option in arrow2 #245Documentation updates:
Closed issues:
Simplified new signature of growable API \#238 (jorgecarleitao)
Breaking changes:
new signature of growable API #238 (jorgecarleitao)New features:
Fixed bugs:
Enhancements:
unsafe #225 (jorgecarleitao)Documentation updates:
* This Changelog was automatically generated by github_changelog_generator
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →