crates.io · #974 most downloaded on crates.io

arrow-string

60.0.084M downloads/moapache/arrow-rs ↗New releases in your daily digest, changelog included.

String kernels for arrow arrays

Last release 21 days ago
15 Sep 2026
Ships fairly regularly
a new release about every 4 weeks
Nearly every release is documented
notes for 56 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
62 releases · first in 2022

Release timeline

62 releases since 2022

20222023202420252026

20222026

One column per quarter.

Releases

55.2.026 Jun 2025Deprecated
Minor: Add version to deprecation notice for ParquetMetaDataReader::decode_footer \#7639 [parquet] (etseidl)
Add a strong_count method to Buffer #7568 [arrow]

Create version of LexicographicalComparator that compares fixed number of columns #7531 [arrow]

parquet-show-bloom-filter should work with integer typed columns #7528 [parquet]

Allow merging primitive dictionary values in concat and interleave kernels #7518 [arrow]

Add efficient concatenation of StructArrays #7516 [arrow]

Rename flight-sql-experimental to flight-sql #7498 [arrow] [arrow-flight]

Consider moving from ryu to lexical-core for string formatting / casting floats to string. #7496

Arithmetic kernels can be safer and faster #7494 [arrow]

Speedup filter_bytes by precalculating capacity #7465 [arrow]

[Variant]: Rust API to Create Variant Values #7424 [parquet] [arrow]

[Variant] Rust API to Read Variant Values #7423 [arrow]

Release arrow-rs / parquet Minor version 55.1.0 (May 2025) #7393 [parquet]

Support create_random_array for Decimal data types #7343 [arrow]

Truncate Parquet page data page statistics #7555 [parquet] (etseidl)

Fixed bugs:

In arrow_json, Decoder::decode can panic if it encounters two high surrogates in a row. #7712

FlightSQL "GetDbSchemas" and "GetTables" schemas do not fully match the protocol #7637 [arrow] [arrow-flight]

Cannot read encrypted Parquet file if page index reading is enabled #7629 [parquet]

encoding_stats not present in Parquet generated by parquet-rewrite #7616 [parquet]

When writing parquet plaintext footer files footer_signing_key_metadata is not included, encryption alghoritm is always written in footer #7599 [parquet]

new_null_array panics when constructing a struct of a dictionary #7571

Parquet derive fails to build when Result is aliased #7547

Unable to read Dictionary(u8, FixedSizeBinary(_)) using datafusion. #7545 [parquet]

filter_record_batch panics with empty struct array. #7538 [arrow]

Panic in pretty_format function when displaying DurationSecondsArray with i64::MIN / i64::MAX #7533 [arrow]

Record API unable to parse TIME_MILLIS when encoded as INT32 #7510 [parquet]

The read_record_batch func of the RecordBatchDecoder does not respect the skip_validation property #7508 [arrow]

arrow-55.1.0 breaks filter_record_batch #7500

Files containing binary data with >=8_388_855 bytes per row written with arrow-rs can't be read with pyarrow #7489 [parquet]

[Bug] Ingestion with Arrow Flight Sql panic when the input stream is empty or fallible #7329 [arrow] [arrow-flight]

Ensure page encoding statistics are written to Parquet file #7643 [parquet] (etseidl)

Documentation updates:

arrow_reader_row_filter benchmark doesn't capture page cache improvements #7460 [parquet] [arrow]

chore: fix a typo in ExtensionType::supports_data_type docs #7682 [arrow] (mbrobbel)

[Variant] Add variant docs and examples #7661 [parquet] (alamb)

Minor: Add version to deprecation notice for ParquetMetaDataReader::decode_footer #7639 [parquet] (etseidl)

Add references for defaults in WriterPropertiesBuilder #7558 [parquet] (etseidl)

Clarify Docs: NullBuffer::len is in bits #7556 [arrow] (alamb)

docs: fix typo for Decimal128Array #7525 [arrow] (burmecia)

Minor: Add examples to ProjectionMask documentation #7523 [parquet] (alamb)

Improve documentation for Parquet WriterProperties #7491 [parquet] (alamb)

Closed issues:

[Variant] More efficient determination of String vs ShortString #7700

[Variant] Improve API for iterating over values of a VariantList #7685 [parquet]

[Variant] Consider validating variants on creation (rather than read) #7684 [parquet]

Miri test_native_type_pow test failing #7641 [arrow]

Improve performance of coalesce and concat for views #7615 [arrow]

Bad min value in row group statistics in some special cases #7593

Feature Request: BloomFilter Position Flexibility in parquet-rewrite #7552 [parquet]

Merged pull requests:

arrow-array: Implement PartialEq for RunArray #7727 [arrow] (brancz)

fix: Do not add null buffer for NullArray in MutableArrayData #7726 [arrow] (comphead)

fix JSON decoder error checking for UTF16 / surrogate parsing panic #7721 [arrow] (nicklan)

[Variant] Introduce new type over &str for ShortString #7718 [parquet] (friendlymatthew)

Split out variant code into several new sub-modules #7717 [parquet] (scovich)

Support write to buffer api for SerializedFileWriter #7714 [parquet] (zhuqi-lucas)

Make variant iterators safely infallible #7704 [parquet] (scovich)

Speedup interleave_views (4-7x faster) #7695 [arrow] (Dandandan)

Define a "arrow-pyrarrow" crate to implement the "pyarrow" feature. #7694 [arrow] (brunal)

Document REE row format and add some more tests #7680 [arrow] (alamb)

feat: add min max aggregate support for FixedSizeBinary #7675 [arrow] (alexwilcoxson-rel)

arrow-data: Add REE support for build_extend and build_extend_nulls #7671 [arrow] (brancz)

Remove lazy_static dependency #7669 [arrow] (Expyron)

Finish implementing Variant::Object and Variant::List #7666 [parquet] (scovich)

Add RecordBatch::schema_metadata_mut and Field::metadata_mut #7664 [arrow] (emilk)

[Variant] Simplify creation of Variants from metadata and value #7663 [parquet] (alamb)

chore: group prost dependabot updates #7659 (mbrobbel)

Initial Builder API for Creating Variant Values #7653 [parquet] (PinkCrow007)

Add BatchCoalescer::push_filtered_batch and docs #7652 [arrow] (alamb)

Optimize coalesce kernel for StringView (10-50% faster) #7650 [arrow] (alamb)

arrow-row: Add support for REE #7649 [arrow] (brancz)

Use approximate comparisons for pow tests #7646 [arrow] (adamreeve)

[Variant] Implement read support for remaining primitive types #7644 [parquet] (superserious-dev)

Add pretty_format_batches_with_schema function #7642 [arrow] (lewiszlw)

Deprecate old Parquet page index parsing functions #7640 [parquet] (etseidl)

Update FlightSQL GetDbSchemas and GetTables schemas to fully match the protocol #7638 [arrow] [arrow-flight] (sgrebnov)

Minor: Remove outdated FIXME from ParquetMetaDataReader #7635 [parquet] (etseidl)

Fix the error info of StructArray::try_new #7634 [arrow] (xudong963)

Fix reading encrypted Parquet pages when using the page index #7633 [parquet] (adamreeve)

[Variant] Add commented out primitive test casees #7631 [parquet] (alamb)

Improve coalesce kernel tests #7626 [arrow] (alamb)

Revert "Revert "Improve coalesce and concat performance for views… #7625 [arrow] (Dandandan)

Revert "Improve coalesce and concat performance for views (#7614)" #7623 [arrow] (Dandandan)

Improve coalesce_kernel benchmark to capture inline vs non inline views #7619 [arrow] (alamb)

Improve coalesce and concat performance for views #7614 [arrow] (Dandandan)

feat: add constructor to help efficiently upgrade key for GenericBytesDictionaryBuilder #7611 [arrow] (albertlockett)

feat: support append_nulls on additional builders #7606 [arrow] (albertlockett)

feat: add AsyncArrowWriter::into_inner #7604 [parquet] (jpopesculian)

Move variant interop test to Rust integration test #7602 [parquet] (alamb)

Include footer key metadata when writing encrypted Parquet with a plaintext footer #7600 [parquet] (rok)

Add coalesce kernel andBatchCoalescer for statefully combining selected b…atches: #7597 [arrow] (alamb)

Add FixedSizeBinary to take_kernel benchmark #7592 [arrow] (alamb)

Fix GenericBinaryArray docstring. #7588 [arrow] (brunal)

fix: error reading multiple batches of Dict(_, FixedSizeBinary(_)) #7585 [parquet] (albertlockett)

Revert "Minor: remove filter code deprecated in 2023 (#7554)" #7583 [arrow] (alamb)

Fixed a warning build build: function never used. #7577 [parquet] (JigaoLuo)

Adding Encoding argument in parquet-rewrite #7576 [parquet] (JigaoLuo)

feat: add row_group_is_[max/min]_value_exact to StatisticsConverter #7574 [parquet] (CookiePieWw)

[array] Remove unwrap checks from GenericByteArray::value_unchecked #7573 [arrow] (ctsk)

[benches/row_format] fix typo in array lengths #7572 [arrow] (ctsk)

Add a strong_count method to Buffer #7569 [arrow] (westonpace)

Minor: Enable byte view for clickbench benchmark #7565 [parquet] (zhuqi-lucas)

Optimize length calculation in row encoding for fixed-length columns #7564 [arrow] (ctsk)

Use PR title and description for commit message #7563 (kou)

Use apache/arrow-{go,java,js} in integration test #7561 (kou)

Implement Array Decoding in arrow-avro #7559 [arrow] (jecsand838)

Minor: remove filter code deprecated in 2023 #7554 [arrow] (alamb)

fix: Correct docs for WriterPropertiesBuilder::set_column_index_truncate_length #7553 [parquet] (etseidl)

Adding Bloom Filter Position argument in parquet-rewrite #7550 [parquet] (JigaoLuo)

Fix Result name collision in parquet_derive #7548 (jspaezp)

Fix: Converted feature flight-sql-experimental to flight-sql #7546 [arrow] [arrow-flight] (kunalsinghdadhwal)

Fix CI on main due to logical conflict #7542 [arrow] (alamb)

Fix filter_record_batch panics with empty struct array #7539 [arrow] (thorfour)

[Variant] Initial API for reading Variant data and metadata #7535 (mkarbo)

fix: Panic in pretty_format function when displaying DurationSecondsA… #7534 [arrow] (zhuqi-lucas)

Create version of LexicographicalComparator that compares fixed number of columns (~ -15%) #7530 [arrow] (Dandandan)

Make parquet-show-bloom-filter work with integer typed columns #7529 [parquet] (adamreeve)

chore(deps): update criterion requirement from 0.5 to 0.6 #7527 [parquet] [arrow] (mbrobbel)

Minor: Add a parquet row_filter test, reduce some test boiler plate #7522 [parquet] (alamb)

Refactor build_array_reader into a struct #7521 [parquet] (alamb)

arrow: add concat structs benchmark #7520 [arrow] (asubiotto)

arrow-select: add support for merging primitive dictionary values #7519 [arrow] (asubiotto)

arrow-select: add support for optimized concatenation of struct arrays #7517 [arrow] (asubiotto)

Fix Clippy in CI for Rust 1.87 release #7514 [parquet] [arrow] [arrow-flight] (alamb)

Simplify ParquetRecordBatchReader::next control logic #7512 [parquet] (alamb)

Fix record API support for reading INT32 encoded TIME_MILLIS #7511 [parquet] (njaremko)

RecordBatchDecoder: skip RecordBatch validation when skip_validation property is enabled #7509 [arrow] (nilskch)

Introduce ReadPlan to encapsulate the calculation of what parquet rows to decode #7502 [parquet] (alamb)

Update documentation for ParquetReader #7501 [parquet] (alamb)

Improve Field docs, add missing Field::set_* methods #7497 [arrow] (alamb)

Speed up arithmetic kernels, reduce unsafe usage #7493 [arrow] (Dandandan)

Prevent FlightSQL server panics for do_put when stream is empty or 1st stream element is an Err #7492 [arrow] [arrow-flight] (superserious-dev)

arrow-ipc: add StreamDecoder::schema #7488 [arrow] (lidavidm)

arrow-select: Implement concat for RunArrays #7487 [arrow] (brancz)

[Variant] Add (empty) parquet-variant crate, update parquet-testing pin #7485 (alamb)

Improve error messages if schema hint mismatches with parquet schema #7481 [parquet] [arrow] (alamb)

Add arrow_reader_clickbench benchmark #7470 [parquet] (alamb)

Speedup filter_bytes ~-20-40%, filter_native low selectivity (~-37%) #7463 [arrow] (Dandandan)

55.1.013 May 2025

refactor!: do not default the struct array length to 0 in Struct::try\_new \#7247 [arrow] (westonpace)

Full Changelog

Breaking changes:

refactor!: do not default the struct array length to 0 in Struct::try_new #7247 [arrow] (westonpace)

Implemented enhancements:

Add a way to get max usize from OffsetSizeTrait #7474 [arrow]
Deterministic metadata encoding #7448 [arrow]
Support Arrow type Dictionary with value FixedSizeBinary in Parquet #7445
Parquet: Add ability to project rowid in parquet reader #7444
Move parquet::file::metadata::reader::FooterTail to parquet::file::metadata so that it is public #7438 [parquet]
Speedup take_bytes by precalculating capacity #7432 [arrow]
Improve performance of interleave_primitive and interleave_bytes #7421 [arrow]
Implement Eq and Default for ScalarBuffer #7411 [arrow]
Add decryption support for column index and offset index #7390 [parquet]
Support writing encrypted Parquet files with plaintext footers #7320 [parquet]
Support Parquet key management tools #7256 [parquet]
Verify footer tags when reading encrypted Parquet files with plaintext footers #7255 [parquet]
StructArray::try_new behavior can be unexpected when there are no child arrays #7246 [arrow]
Parquet performance: improve performance of reading int8/int16 #7097 [parquet]

Fixed bugs:

StructArray::try_new validation incorrectly returns an error when logical_nulls() returns Some() && null_count == 0 #7435
Reading empty DataPageV2 fails with snappy: corrupt input (empty) #7388 [parquet]

Documentation updates:

Improve documentation and add examples for ArrowPredicateFn #7480 [parquet] (alamb)
Document Arrow <--> Parquet schema conversion better #7479 [parquet] (alamb)
Fix a typo in arrow/examples/README.md #7473 [arrow] (Mottl)

Closed issues:

Refactor Parquet DecryptionPropertiesBuilder to fix use of unreachable #7476 [parquet]
Implement Eq and Default for OffsetBuffer #7417 [arrow]

Merged pull requests:

Add Parquet arrow_reader benchmarks for {u}int{8,16} columns #7484 [parquet] (alamb)
fix: rustdoc::unportable_markdown was removed #7483 [arrow] [arrow-flight] (crepererum)
Support round trip reading / writing Arrow Duration type to parquet #7482 [parquet] (Liyixin95)
Add const MAX_OFFSET to OffsetSizeTrait #7478 [arrow] (thinkharderdev)
Refactor Parquet DecryptionPropertiesBuilder #7477 [parquet] (adamreeve)
Support parsing and display pretty for StructType #7469 [arrow] (goldmedal)
chore(deps): update sysinfo requirement from 0.34.0 to 0.35.0 #7462 [parquet] (dependabot[bot])
Verify footer tags when reading encrypted Parquet files with plaintext footers #7459 [parquet] (rok)
Improve comments for avro #7449 [arrow] (kumarlokesh)
feat: Support round trip reading/writing Arrow type Dictionary(_, FixedSizeBinary(_)) to Parquet #7446 [parquet] (albertlockett)
Fix out of bounds crash in RleValueDecoder #7441 [parquet] (apilloud)
Make FooterTail public #7440 [parquet] (masonh22)
Support writing encrypted Parquet files with plaintext footers #7439 [parquet] (rok)
feat: deterministic metadata encoding #7437 [arrow] (timsaucer)
Fix validation logic in StructArray::try_new to account for array.logical_nulls() returning Some() and null_count == 0 #7436 [arrow] (phillipleblanc)
Minor: Fix typo in async_reader comment #7433 [parquet] (amoeba)
feat: coerce fixed size binary to binary view #7431 [arrow] (chenkovsky)
chore(deps): update brotli requirement from 7.0 to 8.0 #7430 [parquet] (dependabot[bot])
Speedup take_bytes (-35% -69%) by precalculating capacity #7422 [arrow] (Dandandan)
Improve performance of interleave_primitive (-15% - 45%) / interleave_bytes (-10-25%) #7420 [arrow] (Dandandan)
Implement Eq and Default for OffsetBuffer #7418 [arrow] (kylebarron)
Implement Default for Buffer & ScalarBuffer #7413 [arrow] (emilk)
Implement Eq for ScalarBuffer when T: Eq #7412 [arrow] (emilk)
Skip page should also support skip dict page #7409 [parquet] (zhuqi-lucas)
Replace RecordBatch::with_schema_unchecked with RecordBatch::new_unchecked #7405 [arrow] (tustvold)
feat: Adding with_schema_unchecked method for RecordBatch #7402 [arrow] (comphead)
Add benchmark for parquet reader with row_filter and project settings #7401 [parquet] (zhuqi-lucas)
Parquet: Expose accessors from ArrowReaderOptions #7400 (kylebarron)
Support decryption of Parquet column and offset indexes #7399 [parquet] (adamreeve)
Handle compressed empty DataPage v2 #7389 [parquet] (EnricoMi)
Improve performance of reading int8/int16 Parquet data #7055 [parquet] (etseidl)

Open source →

55.0.011 Apr 2025Deprecated

MINOR: fix incorrect method name in deprecate node \#7306 [arrow] (waynexia)

Full Changelog

Breaking changes:

Change Parquet API interaction to use u64 (support files larger than 4GB in WASM) #7371 [parquet] (kylebarron)
Remove AsyncFileReader::get_metadata_with_options, add options to AsyncFileReader::get_metadata #7342 [parquet] (corwinjoy)
Parquet: Support reading Parquet metadata via suffix range requests #7334 [parquet] (kylebarron)
Upgrade to object_store to 0.12.0 #7328 [parquet] (mbrobbel)
Upgrade pyo3 to 0.24 #7324 [arrow] (mbrobbel)
Reapply Box FlightErrror::tonic to reduce size (fixes nightly clippy) #7277 [arrow] [arrow-flight] (alamb)
Improve parquet gzip compression performance using zlib-rs #7200 [parquet] (psvri)
Fix: date_part to extract only the requested part (not the overall interval) #7189 [arrow] (delamarch3)
chore: upgrade flatbuffer version to 25.2.10 #7134 [arrow] (tisonkun)
Add hooks to json encoder to override default encoding or add support for unsupported types #7015 [arrow] (adriangb)

Implemented enhancements:

Improve the performance of concat #7357 [arrow]
Pushdown predictions to Parquet in-memory row group fetches #7348 [parquet]
Improve CSV parsing errors: Print the row that makes csv parsing fails #7344 [arrow]
Support ColumnMetaData encoding_stats in Parquet Writing #7341 [parquet]
Support writing Parquet with modular encryption #7327 [parquet]
Parquet Use U64 Instead of Usize (wasm support for files greater than 4GB) #7238 [parquet]
Support different TimeUnits and timezones when reading Timestamps from INT96 #7220 [parquet]

Fixed bugs:

New clippy failures in code base with release of rustc 1.86 #7381 [parquet] [arrow]
Fix bug in ParquetMetaDataReader and add test of suffix metadata reads with encryption #7372 [parquet] (etseidl)

Documentation updates:

Improve documentation on ArrayData::offset #7385 [arrow] (alamb)
Improve documentation for AsyncFileReader::get_metadata #7380 [parquet] (alamb)
Improve documentation on implementing Parquet predicate pushdown #7370 [parquet] (alamb)
Add documentation and examples for pretty printing, make pretty_format_columns_with_options pub #7346 [arrow] (alamb)
Improve documentation on writing parquet, including multiple threads #7321 [parquet] (alamb)

Merged pull requests:

chore: apply clippy suggestions newly introduced in rust 1.86 #7382 [parquet] [arrow] (westonpace)
bench: add more {boolean, string, int} benchmarks for concat kernel #7376 [arrow] (rluvaton)
Add more examples of using Parquet encryption #7374 [parquet] (adamreeve)
Clean up ArrowReaderMetadata::load_async #7369 [parquet] (etseidl)
bump pyo3 for RUSTSEC-2025-0020 #7368 [arrow] (onursatici)
Test int96 Parquet file from Spark #7367 [parquet] (mbutrovich)
fix: respect offset/length when converting ArrayData to StructArray #7366 [arrow] (westonpace)
Print row, data present, expected type, and row number in error messages for arrow-csv #7361 [arrow] (psiayn)
Use rust builtins for round_upto_multiple_of_64 and ceil #7358 [arrow] (psvri)
Write parquet PageEncodingStats #7354 [parquet] (jhorstmann)
Move sysinfo to dev-dependencies #7353 [parquet] (mbrobbel)
chore(deps): update sysinfo requirement from 0.33.0 to 0.34.0 #7352 [parquet] (dependabot[bot])
Add additional benchmarks for utf8view comparison kernels #7351 [arrow] (zhuqi-lucas)
Upgrade to twox-hash 2.0 #7347 [parquet] (alamb)
refactor: apply borrowed chunk reader to Sbbf::read_from_column_chunk #7345 [parquet] (ethe)
Merge changelog and version from 54.3.1 into main #7340 [parquet] [arrow] (timsaucer)
Remove object-store label from .asf.yaml #7339 (mbrobbel)
Encapsulate encryption code more in readers #7337 [parquet] (alamb)
Bump MSRV to 1.81 #7336 [parquet] [arrow] [arrow-flight] (mbrobbel)
Add an option to show column type #7335 [arrow] (blaginin)
Add missing type annotation #7326 [parquet] (mbrobbel)
Minor: Improve parallel parquet encoding example #7323 [parquet] (alamb)
feat: allow if expressions for fallbacks in downcast macro #7322 [arrow] (rluvaton)
Minor: rename ParquetRecordBatchStream::reader to ParquetRecordBatchStream::reader_factory #7319 [parquet] (alamb)
bugfix: correct offsets when serializing a list of fixed sized list and non-zero start offset #7318 [arrow] (timsaucer)
Remove object_store references in Readme.md #7317 (alamb)
Adopt MSRV policy #7314 (psvri)
fix: correct array length validation error message #7313 [arrow] (wkalt)
chore: remove trailing space in debug print #7311 [arrow] (xxchan)
Improve concat performance, and add append_array for some array builder implementations #7309 [arrow] (rluvaton)
feat: add append_buffer for NullBufferBuilder #7308 [arrow] (rluvaton)
MINOR: fix incorrect method name in deprecate node #7306 [arrow] (waynexia)
Allow retrieving Parquet decryption keys using the key metadata #7286 [parquet] (adamreeve)
Support different TimeUnits and timezones when reading Timestamps from INT96 #7285 [parquet] (mbutrovich)
Add Parquet Modular encryption support (write) #7111 [parquet] (rok)

Open source →

54.3.021 Mar 2025

Using column chunk offset index in InMemoryRowGroup::fetch \#7300

Full Changelog

Implemented enhancements:

Using column chunk offset index in InMemoryRowGroup::fetch #7300
Support reading parquet with modular encryption #7296 [parquet]
Add example for how to read/write encrypted parquet files #7281 [parquet]
Have writer return parsed ParquetMetadata #7254 [parquet]
feat: Support Utf8View in JSON reader #7244 [arrow]
StructBuilder should provide a way to get a &dyn ArrayBuilder of a field builder #7193 [arrow]
Support div_wrapping/rem_wrapping for numeric arithmetic kernels #7158 [arrow]
Improve RleDecoder performance #7195 [parquet] (Dandandan)
Improve arrow-json deserialization performance by 30% #7157 [arrow] (mwylde)
Add with_skip_validation flag to IPC StreamReader, FileReader and FileDecoder #7120 [arrow] (alamb)

Fixed bugs:

Archery integration CI test is failing on main: error: package half v2.5.0 cannot be built because it requires rustc 1.81 or newer, while the currently active rustc version is 1.77.2 #7291
MSRV CI check is failing on main #7289
Incorrect IPC schema encoding for multiple dictionaries #7058 [arrow] [arrow-flight]

Documentation updates:

Add example for how to read encrypted parquet files #7283 [parquet] (rok)
Update the relative path of the test data in docs #7221 (Ziy1-Tan)
Minor: fix doc and remove unused code #7194 [arrow] (lewiszlw)
doc: modify wrong comment #7190 [arrow] (YichiZhang0613)
doc: fix IPC file reader/writer docs #7178 [arrow] (Jefffrey)

Merged pull requests:

chore: require ffi feature in arrow-schema benchmark #7298 [arrow] (ethe)
Fix archery integration test #7292 (alamb)
Minor: run test_decimal_list again #7282 [parquet] (alamb)
Move Parquet encryption tests into the arrow_reader integration tests #7279 [parquet] (adamreeve)
Include license and notice files in published crates, part 2 #7275 [arrow] (ankane)
feat: Support Utf8View in JSON reader #7263 [arrow] (zhuqi-lucas)
feat: use force_validate feature flag when creating an arrays #7241 [arrow] (rluvaton)
fix: take on empty struct array returns empty array #7224 [arrow] (westonpace)
fix: correct bloom_filter_position description #7223 [parquet] (romanz)
Minor: Move make_builder into mod.rs #7218 (lewiszlw)
Expose field_builders in StructBuilder #7217 [arrow] (lewiszlw)
Minor: Fix json StructMode docs links #7215 [arrow] (gstvg)
[main] Bump arrow version to 54.2.1 (#7207) #7212 (alamb)
feat: add downcast_integer_array macro helper #7211 [arrow] (rluvaton)
Remove zstd pin #7199 [parquet] (tustvold)
fix: Use chrono's quarter() to avoid conflict #7198 [arrow] (yutannihilation)
Fix some Clippy 1.85 warnings #7167 [parquet] [arrow] (mbrobbel)
feat: add to concat different data types error message the data types #7166 [arrow] (rluvaton)
Add Week ISO, Year ISO computation #7163 [arrow] (kosiew)
fix: create_random_batch fails with timestamp types having a timezone #7162 [arrow] (niebayes)
Avoid overflow of remainder #7159 [arrow] (wForget)
fix: Data type inference for NaN, inf and -inf in csv files #7150 [arrow] (Mottl)
Preserve null dictionary values in interleave and concat kernels #7144 [arrow] (kawadakk)
Support casting Date to a time zone-specific timestamp #7141 [arrow] (friendlymatthew)
Minor: Add doctest to ArrayDataBuilder::build_unchecked #7139 [arrow] (gstvg)
arrow-ord: add support for nested types to partition #7131 [arrow] (asubiotto)
Update prost-build requirement from =0.13.4 to =0.13.5 #7127 [arrow] [arrow-flight] (dependabot[bot])
Avoid use of flatbuffers::size_prefixed_root, fix validation error in arrow-flight #7109 [arrow] [arrow-flight] (bkietz)
Optimise decimal casting for infallible conversions #7021 [arrow] (aweltsch)

Open source →

54.1.02 Feb 2025Deprecated

Improve Buffer documentation, deprecate Buffer::from_bytes add From and From impls \#6939 [parquet] [arrow] [arrow-flight] (alamb)

Full Changelog

Implemented enhancements:

Create GitHub releases automatically on tagging #7041
Add required methods to access inner builder for NullBufferBuilder #7002 [arrow]
Re-export NullBufferBuilder in the arrow crate #6975 [arrow]
arrow-string function should support binary input as well #6923 [arrow]
MMap support for IPC files #6709 [arrow]
fix: mark (Large)ListView as nested and support in equal data type #6995 [arrow] (rluvaton)
Expose min/max values for Decimal128/256 and improve docs #6992 [arrow] (alamb)
[Parquet] Improve speed of dictionary encoding NaN float values #6953 [parquet] (adamreeve)
Optimize BooleanBufferBuilder for non nullable columns #6973 [arrow]
arrow::compute::concat should merge dictionary type when concatenating list of dictionaries #6888 [arrow]
Improve error message for unsupported cast between struct and other types #6724 [arrow]
implement regexp_match, regexp_scalar_match and regexp_array_match for StringViewArray #6717 [arrow]
Speed up Parquet utf8 validation #6667 [parquet]

Fixed bugs:

Regression: Concatenating sliced ListArrays is broken #7034
PrimitiveDictionaryBuilder with specific value data type and capacity #7011 [arrow]
Arrow IPC Writer Panics for sliced nested arrays #6997 [arrow]
RecordBatch with no columns cannot be roundtripped through Parquet #6988 [parquet]
StringView: Using the Interleave kernel (and potentially others) results in many repeated buffers in variadic_buffers #6780 [arrow]
fix prefetch of page index #6999 [parquet] (adriangb)
fix: Parquet column writer Dictionary(_, Decimal128) and Dictionary(_, Decimal256) #6987 [parquet] (korowa)
Writing floating point values containing NaN to Parquet is slow when using dictionary encoding #6952 [parquet] [arrow]
Public API using private types: Buffer::from_bytes takes unexported Bytes #6754 [parquet] [arrow] [arrow-flight]
Some MSRVs are inaccurate #6741 [parquet] [arrow] [arrow-flight]

Documentation updates:

docs: add to bit slice iterator docs that the start value is inclusive and end value is exclusive #7022 [arrow] (rluvaton)
Fix duplicate link references in README #7020 (Jefffrey)
Enhance ListViewArray related docs #7007 [arrow] (Jefffrey)
Document data type support and examples to predicates *like, starts_with, ends_with, contains #7003 [arrow] (alamb)
Minor: improve documentation on timezone representations #7000 [arrow] (alamb)
Add additional documentation for UTC representation of timestamps #6994 [arrow] (Abdullahsab3)
Improve ParquetRecordBatchStreamBuilder docs / examples #6948 [parquet] (alamb)
Document the ParquetRecordBatchStream buffering #6947 [parquet] (alamb)
Minor: improve zip kernel docs, add examples #6928 [arrow] (alamb)
Add doctest example for Buffer::from_bytes #6920 [arrow] (kylebarron)
[object store] Add planned object_store release schedule to crate readme #6904 (alamb)
Avoid panics? #6737 [parquet]

Merged pull requests:

Create GitHub releases automatically on tagging #7042 (kou)
Fix concat for sliced ListArrays #7037 [arrow] (alamb)
Minor: Clarify NullBufferBuilder::new capacity parameter #7016 [arrow] (alamb)
Add is_valid and truncate methods to NullBufferBuilder #7013 [arrow] (Chen-Yuan-Lai)
fix: use the values builder capacity for the hash map in PrimitiveDictionaryBuilder::new_from_builders #7012 [arrow] (rluvaton)
Refactor ipc reading code into methods on ArrayReader #7006 [arrow] (alamb)
Minor: make it clear Predicate is crate private #7001 [arrow] (alamb)
fix: Panic on reencoding offsets in arrow-ipc with sliced nested arrays #6998 [arrow] (HawaiianSpork)
Add check for empty schema in parquet::schema::types::from_thrift_helper #6990 [parquet] (etseidl)
Add example reading data from an mmaped IPC file #6986 [arrow] (alamb)
Improve arrow-ipc documentation #6983 [arrow] (alamb)
Add simdutf8 feature to make simdutf8 optional, consolidate check_valid_utf8 #6979 [parquet] (alamb)
Export NullBufferBuilder along with BooleanBufferBuilder in arrow crate #6976 [arrow] (alamb)
Minor: improve the documentation of NullBuffer and BooleanBuffer #6974 [arrow] (alamb)
Simplify Validation/Alignment APIs of ArrayDataBuilder: validate and align #6966 [arrow] (alamb)
Fix WASM CI for Rust 1.84 release #6963 (alamb)
[Parquet] Add benchmark and test for writing NaNs to Parquet #6955 [parquet] [arrow] (adamreeve)
Add peek_next_page_offset to SerializedPageReader #6945 [parquet] (XiangpengHao)
Improve Buffer documentation, deprecate Buffer::from_bytes add From<Bytes> and From<bytes::Bytes> impls #6939 [parquet] [arrow] [arrow-flight] (alamb)
minor: fix test and remove println in tests #6935 [arrow] (himadripal)
Document how to use Extend for generic methods on ArrayBuilders #6932 [arrow] (wiedld)
[Parquet] Add projection utility functions #6931 [parquet] (XiangpengHao)
[Parquet] Reuse buffer in ByteViewArrayDecoderPlain #6930 [parquet] (XiangpengHao)
Support Binary arrays in starts_with, ends_with and contains #6926 [arrow] (rluvaton)
Improve the error message for casting between struct and non-struct types #6919 [arrow] (takaebato)
Fix error message typos with Parquet compression #6918 [parquet] (orf)
Expose arrow-schema methods, for use when writing parquet outside of ArrowWriter #6916 [parquet] (wiedld)
feat(arrow-ord): support boolean in rank and add tests for sorting lists of booleans #6912 [arrow] (rluvaton)
chore(arrow-ord): move can_rank to the rank file #6910 [arrow] (rluvaton)
feat(parquet): Add next_row_group API for ParquetRecordBatchStream #6907 [parquet] (Xuanwo)
feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries #6893 [arrow] (rluvaton)
add extend_dictionary in dictionary builder for improved performance #6875 [arrow] (rluvaton)
[arrow-string] Implement string view support for regexp_match #6849 [arrow] (tlm365)
Add support StringView / BinaryView in interleave kernel #6779 [arrow] (onursatici)
RecordBatch normalization (flattening) #6758 [arrow] (ngli-me)

Open source →

54.0.023 Dec 2024Deprecated

Remove some very old deprecated functions \#6774 [parquet] [arrow] (alamb)

Full Changelog

Breaking changes:

avoid redundant parsing of repeated value in RleDecoder #6834 [parquet] (jp0317)
Handling nullable DictionaryArray in CSV parser #6830 [arrow] (edmondop)
fix(flightsql): remove Any encoding of DoPutUpdateResult #6825 [arrow] [arrow-flight] (davisp)
arrow-ipc: Default to not preserving dict IDs #6788 [arrow] (brancz)
Remove some very old deprecated functions #6774 [parquet] [arrow] (alamb)
update to pyo3 0.23.0 #6745 [arrow] (psvri)
Remove APIs deprecated since v 4.4.0 #6722 [arrow] [arrow-flight] (findepi)
Return None when Parquet page indexes are not present in file #6639 [parquet] (etseidl)
Add ParquetError::NeedMoreData mark ParquetError as non_exhaustive #6630 [parquet] (etseidl)
Remove APIs deprecated since v 2.0.0 #6609 [arrow] (findepi)

Implemented enhancements:

Parquet schema hint doesn't support integer types upcasting #6891 [parquet]
Parquet UTF-8 max statistics are overly pessimistic #6867 [parquet]
Add builder support for Int8 keys #6844 [arrow]
Formalize the name of the nested Field in a list #6784 [parquet] [arrow] [arrow-flight]
Allow disabling the writing of Parquet Offset Index #6778 [parquet]
parquet::record::make_row is not exposed to users, leaving no option to users to manually create Row objects #6761 [parquet]
Avoid from_num_days_from_ce_opt calls in timestamp_s_to_datetime if we don't need #6746 [arrow]
Support Temporal -> Utf8View casting #6734 [arrow]
Add Option To Coerce List Type on Parquet Write #6733 [parquet] [arrow]
Support Numeric -> Utf8View casting #6714 [arrow]
Support Utf8View <=> boolean casting #6713 [arrow]

Fixed bugs:

Buffer::bit_slice loses length with byte-aligned offsets #6895 [arrow]
parquet arrow writer doesn't track memory size correctly for fixed sized lists #6839 [parquet]
Casting Decimal128 to Decimal128 with smaller precision produces incorrect results in some cases #6833 [arrow]
Should empty nullable dictionary be parsed as null from arrow-csv? #6821 [arrow]
Array take doesn't make fields nullable #6809
Arrow Flight Encodes a Slice's List Offsets If the slice offset is starts with zero #6803 [arrow]
Parquet readers incorrectly interpret legacy nested lists #6756 [parquet]
filter_bits under-allocates resulting boolean buffer #6750 [arrow]
Multi-language support issues with Arrow FlightSQL client's execute_update and execute_ingest methods #6545 [arrow] [arrow-flight]

Documentation updates:

Should we document at what rate deprecated APIs are removed? #6851 [parquet] [arrow]
Fix docstring for Format::with_header in arrow-csv #6856 [arrow] (kylebarron)
Add deprecation / API removal policy #6852 [parquet] [arrow] (alamb)
Minor: add example for creating SchemaDescriptor #6841 [parquet] (alamb)
chore: enrich panic context when BooleanBuffer fails to create #6810 [arrow] (tisonkun)

Closed issues:

[FlightSQL] GetCatalogsBuilder does not sort the catalog names #6807 [arrow] [arrow-flight]
Add a lint to automatically check for unused dependencies #6796 [arrow] [arrow-flight]

Merged pull requests:

doc: add comment for timezone string #6899 [arrow] (xxchan)
docs: fix typo #6890 [arrow] (rluvaton)
Minor: Fix deprecation notice for arrow_to_parquet_schema #6889 [parquet] (etseidl)
Add Field::with_dict_is_ordered #6885 [arrow] (alamb)
Deprecate "max statistics size" property in WriterProperties #6884 [parquet] (etseidl)
Add deprecation warnings for everything related to dict_id #6873 [parquet] [arrow] [arrow-flight] (brancz)
Enable matching temporal as from_type to Utf8View #6872 [arrow] (Kev1n8)
Enable string-based column projections from Parquet files #6871 [parquet] (etseidl)
Improvements to UTF-8 statistics truncation #6870 [parquet] (etseidl)
fix: make GetCatalogsBuilder sort catalog names #6864 [arrow] [arrow-flight] (niebayes)
add buffered data_pages to parquet column writer total bytes estimation #6862 [parquet] (onursatici)
Update prost-build requirement from =0.13.3 to =0.13.4 #6860 [arrow] [arrow-flight] (dependabot[bot])
Minor: add comments explaining bad MSRV, output in json #6857 (alamb)
perf: Use Cow in get_format_string in FFI_ArrowSchema #6853 [arrow] (andygrove)
chore: add cast_decimal benchmark #6850 [arrow] (andygrove)
arrow-array::builder: support Int8, Int16 and Int64 keys #6845 [arrow] (ajwerner)
Add ArrowToParquetSchemaConverter, deprecate arrow_to_parquet_schema #6840 [parquet] (alamb)
Remove APIs deprecated in 50.0.0 #6838 [arrow] (findepi)
fix: decimal conversion looses value on lower precision #6836 [arrow] (himadripal)
Update sysinfo requirement from 0.32.0 to 0.33.0 #6835 [parquet] (dependabot[bot])
Optionally coerce names of maps and lists to match Parquet specification #6828 [parquet] (etseidl)
Remove deprecated unary_dyn and try_unary_dyn #6824 [arrow] (findepi)
Remove deprecated flight_data_from_arrow_batch #6823 [arrow] [arrow-flight] (findepi)
[arrow-cast] Support cast boolean from/to string view #6822 [arrow] (tlm365)
Hook up Avro Decoder #6820 [arrow] (tustvold)
Fix arrow-avro compilation without default features #6819 [arrow] (findepi)
Support shrink to empty #6817 [arrow] (tustvold)
[arrow-cast] Support cast numeric to string view (alternate) #6816 [arrow] (alamb)
Hide implicit optional dependency features in arrow-flight #6806 [arrow] [arrow-flight] (findepi)
fix: Encoding of List offsets was incorrect when slice offsets begin with zero #6805 [arrow] (HawaiianSpork)
Enable unused_crate_dependencies Rust lint, remove unused dependencies #6804 [arrow] [arrow-flight] (findepi)
Minor: Fix docstrings for ColumnProperties::statistics_enabled property #6798 [parquet] (etseidl)
Add option to disable writing of Parquet offset index #6797 [parquet] (etseidl)
Remove unused dependencies #6792 [arrow] [arrow-flight] (findepi)
Add Array::shrink_to_fit(&mut self) #6790 [arrow] (emilk)
Formalize the default nested list field name to item #6785 [parquet] [arrow] [arrow-flight] (gruuya)
Improve UnionArray logical_nulls tests #6781 [arrow] (gstvg)
Improve list builder usage example in docs #6775 [arrow] (findepi)
Update proc-macro2 requirement from =1.0.89 to =1.0.92 #6772 [arrow] [arrow-flight] (dependabot[bot])
Allow NullBuffer construction directly from array #6769 [parquet] [arrow] (findepi)
Include license and notice files in published crates #6767 [parquet] [arrow] [arrow-flight] (ankane)
fix: remove redundant bit_util::ceil #6766 [arrow] (miroim)
Remove 'make_row', expose a 'Row::new' method instead. #6763 [parquet] (jonded94)
Read nested Parquet 2-level lists correctly #6757 [parquet] (etseidl)
Split timestamp_s_to_datetime to date and time to avoid unnecessary computation #6755 [arrow] (jayzhan211)
More trivial implementation of Box<dyn AsyncArrowWriter> and Box<dyn AsyncArrowReader> #6748 [parquet] (ethe)
Update cache action to v4 #6744 (findepi)
Remove redundant implementation of StringArrayType #6743 [arrow] (tlm365)
Fix Dictionary logical nulls for RunArray/UnionArray Values #6740 [arrow] (findepi)
Allow reading Parquet maps that lack a values field #6730 [parquet] (etseidl)
Improve default implementation of Array::is_nullable #6721 [arrow] (findepi)
Fix Buffer::bit_slice losing length with byte-aligned offsets #6707 [arrow] [arrow-flight] (itsjunetime)

Open source →

53.3.020 Nov 2024Breaking

Update PR template section about API breaking changes \#6723 (findepi)

Full Changelog

Signed decimal e-notation parsing bug #6728 [arrow]
Add support for Utf8View -> numeric in can_cast_types #6715
IPC file writer produces incorrect footer when not preserving dict ID #6710 [arrow]
parquet from_thrift_helper incorrectly checks index #6693 [parquet]
Primitive REPEATED fields not contained in LIST annotated groups aren't read as lists by record reader #6648 [parquet]
DictionaryHandling does not recurse into Map fields #6644 [arrow] [arrow-flight]
Array writer output empty when no record is written #6613 [arrow]
Archery Integration Test with c# failing on main #6577 [arrow]
Potential unsoundness in filter_run_end_array #6569 [arrow]
Parquet reader can generate incorrect validity buffer information for nested structures #6510 [parquet]
arrow-array ffi: FFI_ArrowArray.null_count is always interpreted as unsigned and initialized during conversion from C to Rust. #6497 [arrow]

Documentation updates:

Minor: Document pattern for accessing views in StringView #6673 [arrow] (alamb)
Improve Array::is_nullable documentation #6615 [arrow] (findepi)
Minor: improve docs for ByteViewArray->ByteArray From impl #6610 [arrow] (alamb)

Performance improvements:

Speed up filter_run_end_array #6712 [arrow] (Dandandan)

Closed issues:

Incorrect like results for pattern starting/ending with % percent and containing escape characters #6702 [arrow]

Merged pull requests:

Fix signed decimal e-notation parsing #6729 [arrow] (gruuya)
Clean up some arrow-flight tests and duplicated code #6725 [arrow] [arrow-flight] (itsjunetime)
Update PR template section about API breaking changes #6723 (findepi)
Support for casting StringViewArray to DecimalArray #6720 [arrow] (tlm365)
File writer preserve dict bug #6711 [arrow] (brancz)
Add filter_kernel benchmark for run array #6706 [arrow] (delamarch3)
Fix string view ILIKE checks with NULL values #6705 [arrow] (findepi)
Implement logical_null_count for more array types #6704 [arrow] (findepi)
Fix LIKE with escapes #6703 [arrow] (findepi)
Speed up filter_bytes #6699 [arrow] (Dandandan)
Minor: fix misleading comment in byte view #6695 [arrow] (jayzhan211)
minor fix on checking index #6694 [parquet] (jp0317)
Undo run end filter performance regression #6691 [arrow] (delamarch3)
Reimplement PartialEq of GenericByteViewArray compares by logical value #6689 [arrow] (tlm365)
feat: expose known_schema from FlightDataEncoder #6688 [arrow] [arrow-flight] (nathanielc)
Update hashbrown requirement from 0.14.2 to 0.15.1 #6684 [parquet] [arrow] (dependabot[bot])
Support Duration in JSON Reader #6683 [arrow] (simonvandel)
Check predicate and values are the same length for run end array filter safety #6675 [arrow] (delamarch3)
[ffi] Fix arrow-array null_count error during conversion from C to Rust #6674 [arrow] (adbmal)
Support Utf8View for bit_length kernel #6671 [arrow] (austin362667)
Fix string view LIKE checks with NULL values #6662 [arrow] (findepi)
Improve documentation for nullif kernel #6658 [arrow] (alamb)
Improve test_auth error message when contains() fails #6657 [arrow] [arrow-flight] (findepi)
Let std::fmt::Debug for StructArray output Null/Validity info #6655 [arrow] (XinyuZeng)
Include offending line number when processing CSV file fails #6653 [arrow] (findepi)
feat: add write_bytes for GenericBinaryBuilder #6652 [arrow] (tisonkun)
feat: Support Utf8View in JSON serialization #6651 [arrow] (jonmmease)
fix: include chrono-tz in flight sql cli #6650 [arrow] [arrow-flight] (crepererum)
Handle primitive REPEATED field not contained in LIST annotated group #6649 [parquet] (zeevm)
Implement append_n for BooleanBuilder #6646 [arrow] (delamarch3)
fix: recurse into Map datatype when hydrating dictionaries #6645 [arrow] [arrow-flight] (nathanielc)
fix: enable TLS roots for flight CLI client #6640 [arrow] [arrow-flight] (crepererum)
doc: Clarify take kernel semantics #6632 [arrow] (viirya)
Return error rather than panic when too many row groups are written #6629 [parquet] (etseidl)
Fix test feature selection so all feature combinations work as expected #6626 [parquet] (itsjunetime)
Add Parquet RowSelection benchmark #6623 [parquet] (XiangpengHao)
Optimize take_bits to optimize take_boolean / take_primitive / take_byte_view: up to -25% #6622 [arrow] (Dandandan)
Make downcast macros hygenic (#6400) #6620 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.88 to =1.0.89 #6618 [arrow] [arrow-flight] (dependabot[bot])
Fix arrow-json writer empty #6614 [arrow] (gwik)
Add ParquetObjectReader::with_runtime #6612 [parquet] [arrow] (itsjunetime)
Re-enable C# arrow flight integration test #6611 [arrow] (alamb)

Open source →

53.2.024 Oct 2024

Implement arrow\_json encoder for Decimal128 & Decimal256 DataTypes \#6605 [arrow]

Full Changelog

Implemented enhancements:

Implement arrow_json encoder for Decimal128 & Decimal256 DataTypes #6605 [arrow]
Support DataType::FixedSizeList in make_builder within struct_builder.rs #6594 [arrow]
Support DataType::Dictionary in make_builder within struct_builder.rs #6589 [arrow]
Interval parsing from string - accept "mon" and "mons" token #6548 [arrow]
AsyncArrowWriter API to get the total size of a written parquet file #6530 [parquet]
append_many for Dictionary builders #6529 [arrow]
Missing tonic GRPC_STATUS with tonic 0.12.1 #6515 [arrow] [arrow-flight]
Add example of how to use parquet metadata reader APIs for a local cache #6504 [parquet]
Remove reliance on raw-entry feature of Hashbrown #6498 [parquet] [arrow] [arrow-flight]
Improve page index metadata loading in SerializedFileReader::new_with_options #6491 [parquet]
Release arrow-rs / parquet minor version 53.1.0 (October 2024) #6340 [arrow]

Fixed bugs:

Compilation fail where c_char = u8 #6571 [arrow]
Arrow flight CI test failing on master #6568 [arrow] [arrow-flight]

Documentation updates:

Minor: Document SIMD rationale and tips #6554 [arrow] (alamb)

Closed issues:

Casting to and from unions #6247 [arrow]

Merged pull requests:

Minor: more comments for RecordBatch.get_array_memory_size() #6607 [arrow] (2010YOUY01)
Implement arrow_json encoder for Decimal128 & Decimal256 #6606 [arrow] (phillipleblanc)
Add support for building FixedSizeListBuilder in struct_builder's mak… #6595 [arrow] (kszlim)
Add limited support for dictionary builders in make_builders for stru… #6593 [arrow] (kszlim)
Fix CI with new valid certificates and add script for future usage #6585 [arrow] [arrow-flight] (itsjunetime)
Update proc-macro2 requirement from =1.0.87 to =1.0.88 #6579 [arrow] [arrow-flight] (dependabot[bot])
Fix clippy complaints #6573 [parquet] [arrow] [arrow-flight] (itsjunetime)
Use c_char instead of i8 to compile on platforms where c_char = u8 #6572 [arrow] (itsjunetime)
Bump pyspark from 3.3.1 to 3.3.2 in /parquet/pytest #6564 [parquet] (dependabot[bot])
unsafe improvements #6551 [arrow] (ssbr)
Update README.md #6550 [arrow] [arrow-flight] (Abdullahsab3)
Fix string '0' cast to decimal with scale 0 #6547 [arrow] (findepi)
Add finish to AsyncArrowWriter::finish #6543 [parquet] (etseidl)
Add append_nulls to dictionary builders #6542 [arrow] (adriangb)
Improve UnionArray::is_nullable #6540 [arrow] (tustvold)
Allow to read parquet binary column as UTF8 type #6539 [parquet] (goldmedal)
Use HashTable instead of raw_entry_mut #6537 [parquet] [arrow] (tustvold)
Add append_many to dictionary arrays to allow adding repeated values #6534 [arrow] (adriangb)
Adds documentation and example recommending Vec<ArrayRef> over ChunkedArray #6527 [arrow] (efredine)
Update proc-macro2 requirement from =1.0.86 to =1.0.87 #6526 [arrow] [arrow-flight] (dependabot[bot])
Add ColumnChunkMetadataBuilder clear APIs #6523 [parquet] (alamb)
Update sysinfo requirement from 0.31.2 to 0.32.0 #6521 [parquet] (dependabot[bot])
Update Tonic to 0.12.3 #6517 [arrow] [arrow-flight] (cisaacson)
Detect missing page indexes while reading Parquet metadata #6507 [parquet] (etseidl)
Use ParquetMetaDataReader to load page indexes in SerializedFileReader::new_with_options #6506 [parquet] (etseidl)
Improve parquet MetadataFetch and AsyncFileReader docs #6505 [parquet] (alamb)
fix arrow-json encoding with dictionary including nulls #6503 [arrow] (samuelcolvin)
Update brotli requirement from 6.0 to 7.0 #6499 [parquet] (dependabot[bot])
Benchmark both scenarios, with records skipped and without skipping, for delta-bin-packed primitive arrays with half nulls. #6489 [parquet] (wiedld)
Add round trip tests for reading/writing parquet metadata #6463 [parquet] (alamb)

Open source →

53.1.06 Oct 2024Breaking

Add breaking change from \#6043 to CHANGELOG \#6354 (mbrobbel)

Full Changelog

Implemented enhancements:

Write null counts in Parquet statistics when they are known to be zero #6502 [parquet]
Make it easier to find / work with ByteView #6478 [arrow]
Update lexical-core version due to soundness issues with current version #6468
Add builder style API for manipulating ParquetMetaData #6465 [parquet]
ArrayData.align_buffers should support Struct data type / child data #6461 [arrow]
Add a method to return the number of skipped rows in a RowSelection #6428 [parquet]
Bump lexical-core to 1.0 #6397 [arrow]
Add union_extract kernel #6386 [arrow]
implement regexp_is_match_utf8 and regexp_is_match_utf8_scalar for StringViewArray #6370 [arrow]
Add support for BinaryView in arrow_string::length #6358 [arrow]
Add as_union to AsArray #6351
Ability to append non contiguous strings to StringBuilder #6347 [arrow]
Add Catalog DB Schema subcommands to flight_sql_client #6331 [arrow] [arrow-flight]
Add support for Utf8View in arrow_string::length #6305 [arrow]
Reading FIXED_LEN_BYTE_ARRAY columns with nulls is inefficient #6296 [parquet]
Optionally verify 32-bit CRC checksum when decoding parquet pages #6289 [parquet]
Speed up pad_nulls for FixedLenByteArrayBuffer #6297 [parquet] (etseidl)
Improve performance of set_bits by avoiding to set individual bits #6288 [arrow] (kazuyukitanimura)

Fixed bugs:

BitIterator panics when retrieving length #6480 [arrow]
Flight data retrieved via Python client (wrapping C++) cannot be used by Rust Arrow #6471 [arrow]
CI integration test failing: Archery test With other arrows #6448 [parquet] [arrow] [arrow-flight]
IPC not respecting not preserving dict ID #6443 [parquet] [arrow] [arrow-flight]
Failing CI: Prost requires Rust 1.71.1 #6436 [arrow] [arrow-flight]
Invalid struct arrays in IPC data causes panic during read #6416 [arrow]
REE Dicts cannot be encoded/decoded with streaming IPC #6398 [arrow]
Reading json map with non-nullable value schema doesn't error if values are actually null #6391
StringViewBuilder with deduplication does not clear observed values #6384 [arrow]
Cast from Decimal(p, s) to dictionary-encoded Decimal(p, s) loses precision and scale #6381 [arrow]
LocalFileSystem list operation returns objects in wrong order #6375
compute::binary_mut returns Err(PrimitiveArray<T>) only with certain arrays #6374 [arrow]
Exporting Binary/Utf8View from arrow-rs to pyarrow fails #6366 [arrow]
warning: methods as_any and next_batch are never used in parquet crate #6143 [parquet]

Documentation updates:

chore: add docs, part of #37 #6496 [parquet] [arrow] [arrow-flight] (ByteBaker)
Minor: improve ChunkedReader docs #6477 [parquet] (alamb)
Minor: Add some missing documentation to fix CI errors #6445 [arrow] (etseidl)
Fix doc "bit width" to "byte width" #6434 [arrow] (kylebarron)
chore: add docs, part of #37 #6433 [arrow] (ByteBaker)
chore: add docs, part of #37 #6424 [arrow] (ByteBaker)
Rephrase doc comment #6421 [parquet] [arrow] [arrow-flight] (waynexia)
Remove "NOT YET FULLY SUPPORTED" comment from DataType::Utf8View/BinaryView #6380 [arrow] (alamb)
Improve GenericStringBuilder documentation #6372 [arrow] (alamb)

Closed issues:

Columnar json writer for arrow-json #6411
Primitive binary/unary are not as fast as they could be #6364 [arrow]
Different numeric type may be able to compare #6357

Merged pull requests:

fix: override size_hint for BitIterator to return the exact remaining size #6495 [arrow] (Beihao-Zhou)
Minor: Fix path in format command in CONTRIBUTING.md #6494 (etseidl)
Write null counts in Parquet statistics when they are known #6490 [parquet] (etseidl)
Add configuration option to StatisticsConverter to control interpretation of missing null counts in Parquet statistics #6485 [parquet] (etseidl)
fix: check overflow numbers while inferring type for csv files #6481 [arrow] (CookiePieWw)
Add better documentation, examples and builer-style API to ByteView #6479 [arrow] (alamb)
Add take_arrays util for getting entries from 2d arrays #6475 [arrow] (akurmustafa)
Deprecate MetadataLoader #6474 [parquet] (etseidl)
Update tonic-build requirement from =0.12.2 to =0.12.3 #6473 [arrow] [arrow-flight] (dependabot[bot])
Align buffers from Python (FFI) #6472 [arrow] (EnricoMi)
Add ParquetMetaDataBuilder #6466 [parquet] (alamb)
Make ArrayData.align_buffers align child data buffers recursively #6462 [arrow] (EnricoMi)
Minor: Silence compiler warnings for parquet::file::metadata::reader #6457 [parquet] (etseidl)
Minor: Error rather than panic for unsupported for dictionary casting #6456 [arrow] (goldmedal)
Support cast between Durations + between Durations all numeric types #6452 [arrow] (tisonkun)
Deprecate methods from footer.rs in favor of ParquetMetaDataReader #6451 [parquet] (etseidl)
Workaround for missing Parquet page indexes in ParquetMetadaReader #6450 [parquet] (etseidl)
Fix CI by disabling newly failing rust <> nanoarrow integration test in CI #6449 (alamb)
Add IpcSchemaEncoder, deprecate ipc schema functions, Fix IPC not respecting not preserving dict ID #6444 [parquet] [arrow] [arrow-flight] (brancz)
Add additional documentation and builder APIs to SortOptions #6441 [arrow] (alamb)
Update prost-build requirement from =0.13.2 to =0.13.3 #6440 [arrow] [arrow-flight] (dependabot[bot])
Bump arrow-flight MSRV to 1.71.1 #6437 [arrow] [arrow-flight] (gstvg)
Silence warnings that as_any and next_batch are never used #6432 [parquet] (etseidl)
Add ParquetMetaDataReader #6431 [parquet] (etseidl)
Add RowSelection::skipped_row_count #6429 [parquet] (progval)
perf: Faster decimal precision overflow checks #6419 [arrow] (andygrove)
fix: don't panic in IPC reader if struct child arrays have different lengths #6417 [arrow] (alexwilcoxson-rel)
Reduce integration test matrix #6407 (kou)
Move lifetime of take_iter from iterator to its items #6403 [arrow] (dariocurr)
Update lexical-core requirement from 0.8 to 1.0 (to resolve RUSTSEC-2023-0086) #6402 [arrow] (dariocurr)
Fix encoding/decoding REE Dicts when using streaming IPC #6399 [arrow] (brancz)
fix: binary_mut should work if only one input array has null buffer #6396 [arrow] (viirya)
Add set_bits fuzz test #6394 [arrow] (alamb)
impl From<ScalarBuffer<T>> for Buffer #6389 [arrow] (mbrobbel)
Add union_extract kernel #6387 [arrow] (gstvg)
Clear string-tracking hash table when ByteView deduplication is enabled #6385 [arrow] (shanesveller)
fix: Stop losing precision and scale when casting decimal to dictionary #6383 [arrow] (andygrove)
Add ARROW_VERSION const #6379 [arrow] (samuelcolvin)
parquet writer: Raise an error when the row_group_index overflows i16 #6378 [parquet] (progval)
Implement native support StringViewArray for regexp_is_match and regexp_is_match_scalar function, deprecate regexp_is_match_utf8 and regexp_is_match_utf8_scalar #6376 [arrow] (tlm365)
Update chrono-tz requirement from 0.9 to 0.10 #6371 [arrow] (dependabot[bot])
Support StringViewArray interop with python: fix lingering C Data Interface issues for *ViewArray #6368 [arrow] (a10y)
stop panic in MetadataLoader on invalid data #6367 [parquet] (samuelcolvin)
Add support for BinaryView in arrow_string::length #6359 [arrow] (Omega359)
impl From<Vec<T>> for Buffer #6355 [arrow] (mbrobbel)
Add breaking change from #6043 to CHANGELOG #6354 (mbrobbel)
Benchmark for bit_mask (set_bits) #6353 [arrow] (kazuyukitanimura)
Update prost-build requirement from =0.13.1 to =0.13.2 #6350 [arrow] [arrow-flight] (dependabot[bot])
fix: clippy warnings from nightly rust 1.82 #6348 [parquet] [arrow] (waynexia)
Add support for Utf8View in arrow_string::length #6345 [arrow] (Omega359)
feat: add catalog/schema subcommands to flight_sql_client. #6332 [arrow] [arrow-flight] (nathanielc)
Manually run fmt on all files under parquet #6328 [parquet] (etseidl)
Implement UnionArray logical_nulls #6303 [arrow] (gstvg)
Parquet: Verify 32-bit CRC checksum when decoding pages #6290 [parquet] (xmakro)

Open source →

53.0.03 Sept 2024Deprecated

parquet Statistics - deprecate has_* APIs and add _opt functions that return Option \#6216 [parquet] (Michael-J-Ward)

Full Changelog

Breaking changes:

parquet_derive: Match fields by name, support reading selected fields rather than all #6269 (double-free)
Update parquet object_store dependency to 0.11.0 #6264 [parquet] (alamb)
parquet Statistics - deprecate has_* APIs and add _opt functions that return Option<T> #6216 [parquet] (Michael-J-Ward)
Expose bulk ingest in flight sql client and server #6201 [arrow] [arrow-flight] (djanderson)
Upgrade protobuf definitions to flightsql 17.0 (#6133) #6169 [arrow-flight] (alamb)
Remove automatic buffering in ipc::reader::FileReader for for consistent buffering #6132 [arrow] (V0ldek)
No longer write Parquet column metadata after column chunks *and* in the footer #6117 [parquet] (etseidl)
Remove impl<T: AsRef<[u8]>> From<T> for Buffer that easily accidentally copies data #6043 [arrow] (XiangpengHao)

Implemented enhancements:

Derive PartialEq and Eq for parquet::arrow::ProjectionMask #6329 [parquet]
Allow converting empty pyarrow.RecordBatch to arrow::RecordBatch #6318 [arrow]
Parquet writer should not write any min/max data to ColumnIndex when all values are null #6315 [parquet]
Parquet: Add union method to RowSelection #6307 [parquet]
Support writing UTC adjusted time arrow array to parquet #6277 [parquet]
A better way to resize the buffer for the snappy encode/decode #6276 [parquet]
parquet_derive: support reading selected columns from parquet file #6268
Tests for invalid parquet files #6261 [parquet]
Implement date_part for Duration #6245 [arrow]
Avoid unnecessary null buffer construction when converting arrays to a different type #6243 [parquet] [arrow]
Add parquet_opendal in related projects #6235
Look into optimizing reading FixedSizeBinary arrays from parquet #6219 [parquet] [arrow]
Add benchmarks for BYTE_STREAM_SPLIT encoded Parquet FIXED_LEN_BYTE_ARRAY data #6203 [parquet]
Make it easy to write parquet to object_store -- Implement AsyncFileWriter for a type that implements obj_store::MultipartUpload for AsyncArrowWriter #6200 [parquet]
Remove test duplication in parquet statistics tets #6185 [parquet]
Support BinaryView Types in C Schema FFI #6170 [arrow]
speedup take_byte_view kernel #6167 [arrow]
Add support for StringView and BinaryView statistics in StatisticsConverter #6164 [parquet]
Support casting BinaryView --> Utf8 and LargeUtf8 #6162 [arrow]
Implement filter kernel specially for FixedSizeByteArray #6153 [arrow]
Use LevelHistogram throughout Parquet metadata #6134 [parquet]
Support DoPutStatementIngest from Arrow Flight SQL 17.0 #6124 [arrow] [arrow-flight]
ColumnMetaData should no longer be written inline with data #6115 [parquet]
Implement date_part for Interval #6113 [arrow]
Implement Into<Arc<dyn Array>> for ArrayData #6104
Allow flushing or non-buffered writes from arrow::ipc::writer::StreamWriter #6099 [arrow]
Default block_size for StringViewArray #6094 [arrow]
Remove Statistics::has_min_max_set and ValueStatistics::has_min_max_set and use Option instead #6093 [parquet]
Upgrade arrow-flight to tonic 0.12 #6072
Improve speed of row converter by skipping utf8 checks #6058 [arrow]
Extend support for BYTE_STREAM_SPLIT to FIXED_LEN_BYTE_ARRAY, INT32, and INT64 primitive types #6048 [parquet]
Release arrow-rs / parquet minor version 52.2.0 (August 2024) #5998 [parquet] [arrow]

Fixed bugs:

Invalid ColumnIndex written in parquet #6310 [parquet]
comparison_kernels benchmarks panic #6283 [arrow]
Printing schema metadata includes possibly incorrect compression level #6270 [parquet]
Don't panic when creating Field from FFI_ArrowSchema with no name #6251 [arrow]
lexsort_to_indices should not fallback to non-lexical sort if the datatype is not supported #6226 [arrow]
Parquet Statistics null_count does not distinguish between 0 and not specified #6215 [parquet]
Using a take kernel on a dense union can result in reaching "unreachable" code #6206 [arrow]
Adding sub day seconds to Date64 is ignored. #6198 [arrow]
mismatch between parquet type is_optional codes and comment #6191 [parquet]

Documentation updates:

Minor: improve filter documentation #6317 [arrow] (alamb)
Minor: Improve comments on GenericByteViewArray::bytes_iter(), prefix_iter() and suffix_iter() #6306 [arrow] (alamb)
Minor: improve RowFilter and ArrowPredicate docs #6301 [parquet] (alamb)
Improve documentation for MutableArrayData #6272 [arrow] (alamb)
Add examples to StringViewBuilder and BinaryViewBuilder #6240 [arrow] (alamb)
minor: enhance document for ParquetField #6239 [parquet] (mapleFU)
Minor: Improve Type documentation #6224 [arrow] (alamb)
Minor: Update DateType::Date64 docs #6223 [arrow] (alamb)
Add (more) Parquet Metadata Documentation #6184 [parquet] (alamb)
Add additional documentation and examples to ArrayAccessor #6141 [arrow] (alamb)
Minor: improve comments in temporal.rs tests #6140 [arrow] (alamb)
Minor: Update release schedule in README #6125 (alamb)

Closed issues:

Simplify take octokit workflow #6279
Make the bearer token visible in FlightSqlServiceClient #6253 [arrow] [arrow-flight]
Port take workflow to use oktokit #6242
Remove SchemaBuilder dependency from StructArray constructors #6138 [arrow]

Merged pull requests:

Derive PartialEq and Eq for parquet::arrow::ProjectionMask #6330 [parquet] (thinkharderdev)
Support zero column RecordBatches in pyarrow integration (use RecordBatchOptions when converting a pyarrow RecordBatch) #6320 [arrow] (Michael-J-Ward)
Fix writing of invalid Parquet ColumnIndex when row group contains null pages #6319 [parquet] (adriangb)
Pass empty vectors as min/max for all null pages when building ColumnIndex #6316 [parquet] (etseidl)
Update tonic-build requirement from =0.12.0 to =0.12.2 #6314 [arrow] [arrow-flight] (dependabot[bot])
Parquet: add union method to RowSelection #6308 [parquet] (sdd)
Specialize filter for structs and sparse unions #6304 [arrow] (gstvg)
Err on try_from_le_slice #6295 [parquet] (samuelcolvin)
fix reference in doctest to size_of which is not imported by default #6286 [arrow] (rtyler)
Support writing UTC adjusted time arrays to parquet #6278 [parquet] (aykut-bozkurt)
Minor: pub use ByteView in arrow and improve documentation #6275 [arrow] (alamb)
Fix accessing name from ffi schema #6273 [arrow] (kylebarron)
Do not print compression level in schema printer #6271 [parquet] (ttencate)
ci: use octokit to add assignee #6267 (dsgibbons)
Add tests for bad parquet files #6262 [parquet] (alamb)
Add Statistics::distinct_count_opt and deprecate Statistics::distinct_count #6259 [parquet] (alamb)
Minor: move FallibleRequestStream and FallibleTonicResponseStream to a module #6258 [arrow] [arrow-flight] (alamb)
Make the bearer token visible in FlightSqlServiceClient #6254 [arrow] [arrow-flight] (ccciudatu)
Use unary() for array conversion in Parquet array readers, speed up Decimal128, Decimal256 and Float16 #6252 [parquet] [arrow] (etseidl)
Update tower requirement from 0.4.13 to 0.5.0 #6250 [arrow] [arrow-flight] (dependabot[bot])
Implement date_part for durations #6246 [arrow] (nrc)
Remove unnecessary null buffer construction when converting arrays to a different type #6244 [parquet] [arrow] (etseidl)
Implement PartialEq for GenericByteViewArray #6241 [arrow] (alamb)
Minor: Remove non standard footer from LICENSE.txt / reference to Apache Aurora #6237 (alamb)
docs: Add parquet_opendal in related projects #6236 (Xuanwo)
Avoid infinite loop in bad parquet by checking the number of rep levels #6232 [parquet] (jp0317)
Specialize Prefix/Suffix Match for Like/ILike between Array and Scalar for StringViewArray #6231 [arrow] (xinlifoobar)
fix: lexsort_to_indices should not fallback to non-lexical sort if the datatype is not supported #6225 [arrow] (viirya)
Modest improvement to FixedLenByteArray BYTE_STREAM_SPLIT arrow decoder #6222 [parquet] (etseidl)
Improve performance of FixedLengthBinary decoding #6220 [parquet] (etseidl)
Update documentation for Parquet BYTE_STREAM_SPLIT encoding #6212 [parquet] (etseidl)
Improve interval parsing #6211 [arrow] (samuelcolvin)
minor: Suggest take on interleave docs #6210 [arrow] (gstvg)
fix: Correctly handle take on dense union of a single selected type #6209 [arrow] (gstvg)
Add time dictionary coercions #6208 [arrow] (adriangb)
fix(arrow): restrict the range of temporal values produced via data_gen #6205 [arrow] (kyle-mccarthy)
Add benchmarks for BYTE_STREAM_SPLIT encoded Parquet FIXED_LEN_BYTE_ARRAY data #6204 [parquet] (etseidl)
Move ParquetMetadataWriter to its own module, update documentation #6202 [parquet] (alamb)
Add ThriftMetadataWriter for writing Parquet metadata #6197 [parquet] (adriangb)
Update zstd-sys requirement from >=2.0.0, <2.0.13 to >=2.0.0, <2.0.14 #6196 [parquet] (dependabot[bot])
fix parquet type is_optional comments #6192 [parquet] (jp0317)
Remove duplicated statistics tests in parquet #6190 [parquet] (Kev1n8)
Benchmarks for bool_and #6189 [arrow] (simonvandel)
Fix typo in documentation of Float64Array #6188 [arrow] (mesejo)
Make it clear that StatisticsConverter can not panic #6187 [parquet] (alamb)
add filter benchmark for FixedSizeBinaryArray #6186 [arrow] (chloro-pn)
Update sysinfo requirement from 0.30.12 to 0.31.2 #6182 [parquet] (dependabot[bot])
Add support for StringView and BinaryView statistics in StatisticsConverter #6181 [parquet] (Kev1n8)
Support casting between BinaryView <--> Utf8 and LargeUtf8 #6180 [arrow] (xinlifoobar)
Implement specialized filter kernel for FixedSizeByteArray #6178 [arrow] (chloro-pn)
Support StringView and BinaryView in CDataInterface #6171 [arrow] (a10y)
Optimize take kernel for BinaryViewArray and StringViewArray #6168 [arrow] (a10y)
Support Parquet BYTE_STREAM_SPLIT for INT32, INT64, and FIXED_LEN_BYTE_ARRAY primitive types #6159 [parquet] (etseidl)
Fix comparison kernel benchmarks #6147 [arrow] (samuelcolvin)
improve LIKE regex performance up to 12x #6145 [arrow] (samuelcolvin)
Optimize min_boolean and bool_and #6144 [arrow] (simonvandel)
Reduce bounds check in RowIter, add unsafe Rows::row_unchecked #6142 [arrow] (XiangpengHao)
Minor: Simplify StructArray constructors #6139 [arrow] (Rafferty97)
Implement exponential block size growing strategy for StringViewBuilder #6136 [arrow] (XiangpengHao)
Use LevelHistogram in PageIndex #6135 [parquet] (etseidl)
Add ArrowError::ArithmeticError #6130 [arrow] (andygrove)
Improve LIKE performance for "contains" style queries #6128 [arrow] (samuelcolvin)
Add BooleanArray::new_from_packed and BooleanArray::new_from_u8 #6127 [arrow] (chloro-pn)
improvements to (i)starts_with and (i)ends_with performance #6118 [arrow] (samuelcolvin)
Fix Clippy for the Rust 1.80 release #6116 [parquet] [arrow] [arrow-flight] (alamb)
added a flush method to IPC writers #6108 [arrow] (V0ldek)
Add support for level histograms added in PARQUET-2261 to ParquetMetaData #6105 [parquet] (etseidl)
Implement date_part for intervals #6071 [arrow] (nrc)
feat(parquet): Implement AsyncFileWriter for object_store::buffered::BufWriter #6013 [parquet] (Xuanwo)

Open source →

52.2.028 Jul 2024

Faster min/max for string/binary view arrays \#6088 [arrow]

Full Changelog

Implemented enhancements:

Faster min/max for string/binary view arrays #6088 [arrow]
Support casting to/from Utf8View #6076 [arrow]
Min/max support for String/BinaryViewArray #6052 [arrow]
Improve performance of constructing ByteViews for small strings #6034 [parquet] [arrow]
Fast UTF-8 validation when reading StringViewArray from Parquet #5995 [parquet]
Optimize StringView row decoding #5945 [arrow]
Implementing deduplicate / intern functionality for StringView #5910 [arrow]
Add FlightSqlServiceClient::new_from_inner #6003 [arrow] [arrow-flight] (lewiszlw)
Complete StringViewArray and BinaryViewArray parquet decoder: #6004 [parquet] (XiangpengHao)
Add begin/end_transaction methods in FlightSqlServiceClient #6026 [arrow] [arrow-flight] (lewiszlw)
Read Parquet statistics as arrow Arrays #6046 [parquet] (efredine)

Fixed bugs:

Panic in ParquetMetadata::memory_size if no min/max set #6091 [parquet]
BinaryViewArray doesn't roundtrip a single Some(&[]) through parquet #6086 [parquet]
Parquet ColumnIndex for null columns is written even when statistics are disabled #6010 [parquet]

Documentation updates:

Fix typo in GenericByteViewArray documentation #6054 [arrow] (progval)
Minor: Improve parquet PageIndex documentation #6042 [parquet] (alamb)

Closed issues:

Potential performance improvements for reading Parquet to StringViewArray/BinaryViewArray #5904 [parquet] [arrow]

Merged pull requests:

Faster GenericByteView construction #6102 [parquet] [arrow] (XiangpengHao)
Add benchmark to track byte-view construction performance #6101 [parquet] (XiangpengHao)
Optimize bool_or using max_boolean #6100 [arrow] (simonvandel)
Optimize max_boolean by operating on u64 chunks #6098 [arrow] (simonvandel)
fix panic in ParquetMetadata::memory_size: check has_min_max_set before invoking min()/max() #6092 [parquet] (Fischer0522)
Implement specialized min/max for GenericBinaryView (StringView and BinaryView) #6089 [arrow] (XiangpengHao)
Add PartialEq to ParquetMetaData and FileMetadata #6082 [parquet] (adriangb)
Enable casting from Utf8View #6077 [arrow] (a10y)
StringView support in arrow-csv #6062 [arrow] (2010YOUY01)
Implement min max support for string/binary view types #6053 [arrow] (XiangpengHao)
Minor: clarify the relationship between file::metadata and format in docs #6049 [parquet] (alamb)
Minor API adjustments for StringViewBuilder #6047 [arrow] (XiangpengHao)
Add parquet StatisticsConverter for arrow reader #6046 [parquet] (efredine)
Directly decode String/BinaryView types from arrow-row format #6044 [arrow] (XiangpengHao)
Clean up unused code for view types in offset buffer #6040 [parquet] (XiangpengHao)
Avoid using Buffer api that accidentally copies data #6039 [parquet] [arrow] [arrow-flight] (XiangpengHao)
MINOR: Fix hashbrown version in arrow-array, remove from arrow-row #6035 [arrow] (mbrobbel)
Improve performance reading ByteViewArray from parquet by removing an implicit copy #6031 [parquet] (XiangpengHao)
Add begin/end_transaction methods in FlightSqlServiceClient #6026 [arrow] [arrow-flight] (lewiszlw)
Unsafe improvements: core parquet crate. #6024 [parquet] (veluca93)
Additional tests for parquet reader utf8 validation #6023 [parquet] (alamb)
Update zstd-sys requirement from >=2.0.0, <2.0.12 to >=2.0.0, <2.0.13 #6019 [parquet] (dependabot[bot])
fix doc ci in latest rust nightly version #6012 [arrow] [arrow-flight] (Rachelint)
Do not write ColumnIndex for null columns when not writing page statistics #6011 [parquet] (etseidl)
Fast utf8 validation when loading string view from parquet #6009 [parquet] (XiangpengHao)
Deduplicate strings/binarys when building view types #6005 [arrow] (XiangpengHao)
Complete StringViewArray and BinaryViewArray parquet decoder: implement delta byte array and delta length byte array encoding #6004 [parquet] (XiangpengHao)
Add FlightSqlServiceClient::new_from_inner #6003 [arrow] [arrow-flight] (lewiszlw)
Rename Schema::all_fields to flattened_fields #6001 [parquet] [arrow] [arrow-flight] (lewiszlw)
Refine documentation and examples for DataType #5997 [arrow] (alamb)
implement DataType::try_form(&str) #5994 [arrow] (samuelcolvin)
Implement dictionary support for reading ByteView from parquet #5973 [parquet] (XiangpengHao)

Open source →

52.1.06 Jul 2024Breaking

Document process for PRs with breaking changes \#5953 (alamb)

Full Changelog

Implemented enhancements:

Implement eq comparison for StructArray #5960 [arrow]
A new feature as a workaround hack to unavailable offset support in Arrow Java #5959 [arrow]
Add min_bytes and max_bytes to PageIndex #5949 [parquet]
Error message in ArrowNativeTypeOp::neg_checked doesn't include the operation #5944 [arrow]
Add object_store_opendal as related projects #5925
Opaque retry errors make debugging difficult #5923
Implement arrow-row en/decoding for GenericByteView types #5921 [arrow]
The arrow-rs repo is very large #5908
[DISCUSS] Release arrow-rs / parquet patch release 52.0.1 #5906 [arrow]
Implement compare_op for GenericBinaryView #5897 [arrow]
New null with view types are not supported #5893 [arrow]
Cleanup ByteView construction #5878 [parquet] [arrow]
cast kernel support for StringViewArray and BinaryViewArray \<--\> DictionaryArray` #5861 [arrow]
parquet::ArrowWriter show allow writing Bloom filters before the end of the file #5859 [parquet]
API to get memory usage for parquet ArrowWriter #5851 [parquet]
Support writing IntervalMonthDayNanoArray to parquet via Arrow Writer #5849 [parquet]
Write parquet statistics for IntervalDayTimeArray , IntervalMonthDayNanoArray and IntervalYearMonthArray #5847 [parquet]
Make RowSelection::from_consecutive_ranges public #5846 [parquet]
Schema::try_merge should be able to merge List of any data type with List of Null data type #5843 [arrow]
Add a way to move fields out of parquet Row #5841 [parquet]
Make TimeUnit and IntervalUnit Copy #5839 [arrow]
Limit Parquet Page Row Count By Default to reduce writer memory requirements with highly compressable columns #5797 [parquet]
Report / blog on parquet metadata sizes for "large" (1000+) numbers of columns #5770 [parquet] [arrow]
Structured ByteView Access (underlying StringView/BinaryView representation) #5736 [arrow]
[parquet_derive] support OPTIONAL (def_level = 1) columns by default #5716
Maps cast to other Maps with different Elements, Key and Value Names #5702 [arrow]
Provide Arrow Schema Hint to Parquet Reader #5657 [parquet] [arrow]

Fixed bugs:

Wrong error type in case of invalid amount in Interval components #5986 [arrow]
Empty and Null structarray fails to IPC roundtrip #5920
FixedSizeList got out of range when the total length of the underlying values over i32::MAX #5901 [arrow]
Out of range when extending on a slice of string array imported through FFI #5896 [arrow]
cargo msrv test is failing on main for object_store #5864 [parquet]

Documentation updates:

chore: update RunArray reference in run_iterator.rs #5892 [arrow] (Weijun-H)
Minor: Clarify when page index structures are read #5886 [parquet] (alamb)
Improve Parquet reader/writer properties docs #5863 [parquet] (alamb)
Refine documentation for unary_mut and binary_mut #5798 [arrow] (alamb)

Closed issues:

Implement benchmarks for compare_op for GenericBinaryView #5903 [arrow]

Merged pull requests:

fix: error in case of invalid amount interval component #5987 [arrow] (DDtKey)
Minor: fix clippy complaint in parquet_derive #5984 (alamb)
Reduce repo size by removing accumulative commits in CI job #5982 (Owen-CH-Leung)
Add operation in ArrowNativeTypeOp::neg_check error message (#5944) #5980 [arrow] (zhao-gang)
Implement directly build byte view array on top of parquet buffer #5972 [parquet] (XiangpengHao)
Handle flight dictionary ID assignment automatically #5971 [arrow] [arrow-flight] (thinkharderdev)
Add view buffer for parquet reader #5970 [parquet] [arrow] (XiangpengHao)
Add benchmark for reading binary/binary view from parquet #5968 [parquet] (XiangpengHao)
feat(5851): ArrowWriter memory usage #5967 [parquet] (wiedld)
Add ParquetMetadata::memory_size size estimation #5965 [parquet] (alamb)
Fix FFI array offset handling #5964 [arrow] (tustvold)
Implement sort for String/BinaryViewArray #5963 [arrow] (XiangpengHao)
Improve error message for unsupported nested comparison #5961 [arrow] (alamb)
chore(5797): change default parquet data_page_row_limit to 20k #5957 [parquet] (wiedld)
Document process for PRs with breaking changes #5953 (alamb)
Minor: fixup contribution guide about clippy #5952 (alamb)
feat: add max_bytes and min_bytes on PageIndex #5950 [parquet] (tshauck)
test: Add unit test for extending slice of list array #5948 [arrow] (viirya)
minor: row format benches for bool & nullable int #5943 [arrow] (korowa)
Better document support for nested comparison #5942 [arrow] (tustvold)
Provide Arrow Schema Hint to Parquet Reader - Alternative 2 #5939 [parquet] (efredine)
like benchmark for StringView #5936 [arrow] (alamb)
Fix typo in benchmark name egexp --> regexp #5935 [arrow] (alamb)
Revert "Write Bloom filters between row groups instead of the end " #5932 [parquet] (alamb)
Implement like/ilike etc for StringViewArray #5931 [arrow] (XiangpengHao)
docs: Fix broken links of object_store_opendal README #5929 (Xuanwo)
Expose IntervalMonthDayNano and IntervalDayTime and update docs #5928 [arrow] (alamb)
Update proc-macro2 requirement from =1.0.85 to =1.0.86 #5927 [arrow] [arrow-flight] (dependabot[bot])
docs: Add object_store_opendal as related projects #5926 (Xuanwo)
Add eq benchmark for StringArray/StringViewArray #5924 [arrow] (XiangpengHao)
Implement arrow-row encoding/decoding for view types #5922 [arrow] (XiangpengHao)
fix(ipc): set correct row count when reading struct arrays with zero fields #5918 [arrow] (kawadakk)
Update zstd-sys requirement from >=2.0.0, <2.0.10 to >=2.0.0, <2.0.12 #5913 [parquet] (dependabot[bot])
fix: prevent potential out-of-range access in FixedSizeListArray #5902 [arrow] (BubbleCal)
Implement compare operations for view types #5900 [arrow] (XiangpengHao)
minor: use as_primitive replace downcast_ref #5898 [arrow] (Kikkon)
fix: Adjust FFI_ArrowArray offset based on the offset of offset buffer #5895 [arrow] (viirya)
implement new_null_array for view types #5894 [arrow] (XiangpengHao)
chore: add view type single column tests #5891 [parquet] (ariesdevil)
Minor: expose timestamp_tz_format for csv writing #5890 [arrow] (tmi)
chore: implement parquet error handling for object_store #5889 [parquet] (abhiaagarwal)
Document when the ParquetRecordBatchReader will re-read metadata #5887 [parquet] (alamb)
Add simple GC for view array types #5885 [arrow] (XiangpengHao)
Update for new clippy rules #5881 [parquet] [arrow] (XiangpengHao)
clean up ByteView construction #5879 [parquet] [arrow] (XiangpengHao)
Avoid copy/allocation when read view types from parquet #5877 [parquet] (XiangpengHao)
Document parquet ArrowWriter type limitations #5875 [parquet] (alamb)
Benchmark for casting view to dict arrays (and the reverse) #5874 [arrow] (XiangpengHao)
Implement Take for Dense UnionArray #5873 [arrow] (gstvg)
Improve performance of casting StringView/BinaryView to DictionaryArray #5872 [arrow] (XiangpengHao)
Improve performance of casting DictionaryArray to StringViewArray #5871 [arrow] (XiangpengHao)
fix: msrv CI for object_store #5866 (korowa)
parquet: Fix warning about unused import #5865 [parquet] (progval)
Preallocate for FixedSizeList in concat #5862 [arrow] (judahrand)
Faster primitive arrays encoding into row format #5858 [arrow] (korowa)
Added panic message to docs. #5857 [arrow] (SeeRightThroughMe)
feat: call try_merge recursively for list field #5852 [arrow] (mnpw)
Minor: refine row selection example more #5850 [parquet] (alamb)
Make RowSelection's from_consecutive_ranges public #5848 [parquet] (advancedxy)
Add exposing fields from parquet row #5842 [parquet] (SHaaD94)
Derive Copy for TimeUnit and IntervalUnit #5840 [arrow] (mbrobbel)
feat: support reading OPTIONAL column in parquet_derive #5717 (double-free)
Add the ability for Maps to cast to another case where the field names are different #5703 [arrow] (HawaiianSpork)

Open source →

52.0.06 Jun 2024Deprecated

Remove deprecated JSON writer \#5651 [arrow] (tustvold)

Full Changelog

Breaking changes:

chore: Make binary_mut kernel accept different type for second arg #5833 [arrow] (viirya)
fix(flightsql): remove Any encoding of DoPutPreparedStatementResult #5817 [arrow] [arrow-flight] (erratic-pattern)
Encode UUID as FixedLenByteArray in parquet_derive #5773 (conradludgate)
Structured interval types for IntervalMonthDayNano or IntervalDayTime (#3125) (#5654) #5769 [parquet] [arrow] (tustvold)
Fallible stream for arrow-flight do_exchange call (#3462) #5698 [arrow] [arrow-flight] (opensourcegeek)
Update object_store dependency in arrow to 0.10.0 #5675 [parquet] (tustvold)
Remove deprecated JSON writer #5651 [arrow] (tustvold)
Change UnionArray constructors #5623 [arrow] [arrow-flight] (mbrobbel)
Update py03 from 0.20 to 0.21 #5566 [arrow] (Jefffrey)
Optionally require alignment when reading IPC, respect alignment when writing #5554 [arrow] [arrow-flight] (hzuo)

Implemented enhancements:

Serialize Binary and LargeBinary as HEX with JSON Writer #5783 [arrow]
Some optimizations in arrow_buffer::util::bit_util do more harm than good #5771 [arrow]
Support skipping comments in CSV files #5758 [arrow]
parquet-derive should be included in repository README. #5751
proposal: Make AsyncArrowWriter accepts AsyncFileWriter trait instead #5738 [parquet]
Nested nullable fields do not get treated as nullable in data_gen #5712 [arrow]
Optionally support flexible column lengths #5678 [arrow]
Arrow Flight SQL example server: do_handshake should include auth header #5665 [arrow] [arrow-flight]
Add support for the "r+" datatype in the C Data interface / RunArray #5631 [arrow]
Serialize FixedSizeBinary as HEX with JSON Writer #5620 [arrow]
Cleanup UnionArray Constructors #5613 [arrow] [arrow-flight]
Zero Copy Support #5593
ObjectStore bulk delete #5591
Retry on Broken Connection #5589
StreamReader is not zero-copy #5584 [arrow]
Create ArrowReaderMetadata from externalized metadata #5582 [parquet]
Make filter in filter_leaves API propagate error #5574 [arrow]
Support List in compare_op #5572
Make FixedSizedList Json serializable #5568 [arrow]
arrow-ord: Support sortting StructArray #5559 [arrow]
Add scientific notation decimal parsing in parse_decimal #5549 [arrow]
take kernel support for StringViewArray and BinaryViewArray #5511 [arrow]
filter kernel support for StringViewArray and BinaryViewArray #5510 [arrow]
Display support for StringViewArray and BinaryViewArray #5509 [arrow]
Arrow Flight format support for StringViewArray and BinaryViewArray #5507 [arrow] [arrow-flight]
IPC format support for StringViewArray and BinaryViewArray #5506 [parquet] [arrow]

Fixed bugs:

List Row Encoding Sorts Incorrectly #5807 [arrow]
Schema Root Message Name Ignored by parquet-fromcsv #5804 [parquet]
Compute data buffer length by using start and end values in offset buffer #5756 [arrow]
parquet: ByteArrayEncoder allocates large unused FallbackEncoder for Parquet 2 #5755 [parquet]
The CI pipeline Archery test With other arrow is broken #5742 [arrow]
Unable to parse scientific notation string to decimal when scale is 0 #5739 [arrow]
Stateless prepared statements wrap DoPutPreparedStatementResult with Any which is differs from Go implementation #5731 [arrow] [arrow-flight]
"Rustdocs are clean (amd64, nightly)" CI check is failing #5725 [parquet] [arrow]
"Archery test With other arrows" integration tests are failing #5719 [parquet] [arrow]
parquet_derive: invalid examples/documentation #5687
Arrow FLight SQL: invalid location in get_flight_info_prepared_statement #5669 [arrow] [arrow-flight]
Rust Interval definition incorrect #5654 [parquet] [arrow]
DECIMAL regex in csv reader does not accept positive exponent specifier #5648 [arrow]
panic when casting ListArray to FixedSizeList #5642 [arrow]
FixedSizeListArray::try_new Errors on Entirely Null Array With Size 0 #5614 [arrow]
parquet / Build wasm32 (pull_request) CI check failing on main #5565 [parquet] [arrow]
Documentation fix: example in parquet/src/column/mod.rs is incorrect #5560 [parquet]
IPC code writes data with insufficient alignment #5553 [arrow] [arrow-flight]
Cannot access example Flight SQL Server from dbeaver #5540 [arrow] [arrow-flight]
parquet: "not yet implemented" error when codec is actually implemented but disabled #5520 [parquet]

Documentation updates:

Minor: Improve arrow_cast documentation #5825 [arrow] (alamb)
Minor: Improve ArrowReaderBuilder::with_row_selection docs #5824 [parquet] (alamb)
Minor: Add examples for ColumnPath::from #5813 [parquet] (alamb)
Minor: Clarify docs on EnabledStatistics #5812 [parquet] (alamb)
Add parquet-derive to repository README #5795 (konjac)
Refine ParquetRecordBatchReaderBuilder docs #5774 [parquet] (alamb)
docs: add sizing explanation to bloom filter docs in parquet #5705 [parquet] (hiltontj)

Closed issues:

binary_mut kernel requires both args to be the same type (which is inconsistent with binary) #5818 [arrow]
Panic when displaying debug the results via log::info in the browser. #5599 [arrow]

Merged pull requests:

feat: impl *Assign ops for types in arrow-buffer #5832 [arrow] (waynexia)
Relax zstd-sys Version Pin #5829 [parquet] (waynexia)
Minor: Document timestamp with/without cast behavior #5826 [arrow] (alamb)
fix: invalid examples/documentation in parquet_derive doc #5823 (Weijun-H)
Check length of FIXED_LEN_BYTE_ARRAY for uuid logical parquet type #5821 [parquet] (mbrobbel)
Allow overriding the inferred parquet schema root #5814 [parquet] (tustvold)
Revisit List Row Encoding (#5807) #5811 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.83 to =1.0.84 #5805 [arrow] [arrow-flight] (dependabot[bot])
Fix typo continuation maker -> marker #5802 [arrow] (djanderson)
fix: serialization of decimal #5801 [arrow] (yjshen)
Allow constructing ByteViewArray from existing blocks #5796 [arrow] (tustvold)
Push SortOptions into DynComparator Allowing Nested Comparisons (#5426) #5792 [arrow] (tustvold)
Fix incorrect URL to Parquet CPP types.h #5790 [parquet] (viirya)
Update proc-macro2 requirement from =1.0.82 to =1.0.83 #5789 [arrow] [arrow-flight] (dependabot[bot])
Update prost-build requirement from =0.12.4 to =0.12.6 #5788 [arrow] [arrow-flight] (dependabot[bot])
Refine parquet documentation on types and metadata #5786 [parquet] (alamb)
feat(arrow-json): encode Binary and LargeBinary types as hex when writing JSON #5785 [arrow] (hiltontj)
fix broken link to ballista crate in README.md #5784 (navicore)
feat(arrow-csv): support encoding of binary in CSV writer #5782 [arrow] (hiltontj)
Fix documentation for parquet parse_metadata, decode_metadata and decode_footer #5781 [parquet] (alamb)
Support casting a FixedSizedList<T>[1] to T #5779 [arrow] (sadboy)
[parquet] Set the default size of BitWriter in DeltaBitPackEncoder to 1MB #5776 [parquet] (AdamGS)
Remove harmful table lookup optimization for bitmap operations #5772 [arrow] (HadrienG2)
Remove deprecated comparison kernels (#4733) #5768 [arrow] (tustvold)
Add environment variable definitions to run the nanoarrow integration tests #5764 (paleolimbot)
Downgrade to Rust 1.77 in integration pipeline to fix CI (#5719) #5761 (tustvold)
Expose boolean builder contents #5760 [arrow] (HadrienG2)
Allow specifying comment character for CSV reader #5759 [arrow] (bbannier)
Expose the null buffer of every builder that has one #5754 [arrow] (HadrienG2)
feat: Make AsyncArrowWriter accepts AsyncFileWriter #5753 [parquet] (Xuanwo)
Improve repository readme #5752 (alamb)
Document object store release cadence #5750 (alamb)
Compute data buffer length by using start and end values in offset buffer #5741 [arrow] (viirya)
fix: parse string of scientific notation to decimal when the scale is 0 #5740 [arrow] (yjshen)
Minor: avoid (likely unreachable) panic in FlightClient #5734 [arrow] [arrow-flight] (alamb)
Update proc-macro2 requirement from =1.0.81 to =1.0.82 #5732 [arrow] [arrow-flight] (dependabot[bot])
Improve error message for timestamp queries outside supported range #5730 [arrow] (Abdi-29)
Refactor to share code between do_put and do_exchange calls #5728 [arrow] [arrow-flight] (opensourcegeek)
Update brotli requirement from 5.0 to 6.0 #5726 [parquet] (dependabot[bot])
Fix GenericListBuilder test typo #5724 [arrow] (Kikkon)
Deprecate NullBuilder capacity, as it behaves in a surprising way #5721 [arrow] (HadrienG2)
Fix nested nullability when randomly generating arrays #5713 [arrow] (alexwilcoxson-rel)
Fix up clippy for Rust 1.78 #5710 [parquet] [arrow] (alamb)
Support casting StringView/BinaryView --> StringArray/BinaryArray. #5704 [arrow] (RinChanNOWWW)
Fix documentation around handling of nulls in cmp kernels #5697 [arrow] (Jefffrey)
Support casting StringArray/BinaryArray --> StringView / BinaryView #5686 [arrow] (RinChanNOWWW)
Add support for flexible column lengths #5679 [arrow] (Posnet)
Move ffi stream and utils from arrow to arrow-array #5670 [arrow] (alexandreyc)
Arrow Flight SQL example JDBC driver incompatibility #5666 [arrow] [arrow-flight] (istvan-fodor)
Add ListView & LargeListView basic construction and validation #5664 [arrow] (Kikkon)
Update proc-macro2 requirement from =1.0.80 to =1.0.81 #5659 [arrow] [arrow-flight] (dependabot[bot])
Modify decimal regex to accept positive exponent specifier #5649 [arrow] (jdcasale)
feat: JSON encoding of FixedSizeList #5646 [arrow] (hiltontj)
Update proc-macro2 requirement from =1.0.79 to =1.0.80 #5644 [arrow] [arrow-flight] (dependabot[bot])
fix: panic when casting ListArray to FixedSizeList #5643 [arrow] (jonahgao)
Add more invalid utf8 parquet reader tests #5639 [parquet] (alamb)
Update brotli requirement from 4.0 to 5.0 #5637 [parquet] (dependabot[bot])
Update flatbuffers requirement from 23.1.21 to 24.3.25 #5636 [arrow] (dependabot[bot])
Increase BinaryViewArray test coverage #5635 [arrow] (alamb)
PrettyPrint support for StringViewArray and BinaryViewArray #5634 [arrow] (alamb)
feat(ffi): add run end encoded arrays #5632 [arrow] (notfilippo)
Accept parquet schemas without explicitly required Map keys #5630 [parquet] (jupiter)
Implement filter kernel for byte view arrays. #5624 [arrow] (RinChanNOWWW)
feat: encode FixedSizeBinary in JSON as hex string #5622 [arrow] (hiltontj)
Update Flight crate README version #5621 [arrow] [arrow-flight] (phillipleblanc)
feat: support reading and writingStringView and BinaryView in parquet (part 1) #5618 [parquet] [arrow] (alamb)
Use FixedSizeListArray::new in FixedSizeListBuilder #5612 [arrow] (tustvold)
String to decimal conversion written using E/scientific notation #5611 [arrow] (Nekit2217)
Account for Timezone when Casting Timestamp to Date32 #5605 [arrow] (Lordworms)
Update prost-build requirement from =0.12.3 to =0.12.4 #5604 [arrow] [arrow-flight] (dependabot[bot])
Fix panic when displaying dates on 32-bit platforms #5603 [arrow] (ivanceras)
Implement take kernel for byte view array. #5602 [arrow] (RinChanNOWWW)
Add tests for Arrow Flight support for StringViewArray and BinaryViewArray #5601 [arrow] [arrow-flight] (XiangpengHao)
test: Add a test for RowFilter with nested type #5600 [parquet] (viirya)
Minor: Add docs for GenericBinaryBuilder, links to GenericStringBuilder #5597 [arrow] (alamb)
Bump chrono-tz from 0.8 to 0.9 #5596 [arrow] (Jefffrey)
Update brotli requirement from 3.3 to 4.0 #5586 [parquet] (dependabot[bot])
Add UnionArray::into_parts #5585 [arrow] (mbrobbel)
Expose ArrowReaderMetadata::try_new #5583 [parquet] (kylebarron)
Add try_filter_leaves to propagate error from filter closure #5575 [arrow] (viirya)
filter for run end array #5573 [arrow] (fabianmurariu)
Pin zstd-sys to v2.0.9 in parquet #5567 [parquet] (Jefffrey)
Split arrow_cast::cast::string into it's own submodule #5563 [arrow] (monkwire)
Correct example code for column (#5560) #5561 [parquet] (zgershkoff)
Split arrow_cast::cast::dictionary into it's own submodule #5555 [arrow] (monkwire)
Split arrow_cast::cast::decimal into it's own submodule #5552 [arrow] (monkwire)
Fix new clippy lints for Rust 1.77 #5544 [parquet] [arrow] (alamb)
fix: correctly encode ticket #5543 [arrow] [arrow-flight] (freddieptf)
feat: implemented with_field() for FixedSizeListBuilder #5541 [arrow] (istvan-fodor)
Split arrow_cast::cast::list into it's own submodule #5537 [arrow] (monkwire)
Bump black from 22.10.0 to 24.3.0 in /parquet/pytest #5535 [parquet] (dependabot[bot])
Add OffsetBufferBuilder #5532 [arrow] (tustvold)
Add IPC StreamDecoder #5531 [arrow] (tustvold)
IPC format support for StringViewArray and BinaryViewArray #5525 [arrow] (XiangpengHao)
parquet: Use specific error variant when codec is disabled #5521 [parquet] (progval)
impl From<ScalarBuffer<T>> for Vec<T> #5518 [arrow] (mbrobbel)

Open source →

51.0.018 Mar 2024Deprecated

Deprecate array\_to\_json\_array \#5515 [arrow] (tustvold)

Full Changelog

Breaking changes:

Remove internal buffering from AsyncArrowWriter (#5484) #5485 [parquet] (tustvold)
Make ArrayBuilder also Sync #5353 [arrow] (dvic)
Raw JSON writer (~10x faster) (#5314) #5318 [arrow] (tustvold)

Implemented enhancements:

Prototype Arrow over HTTP in Rust #5496 [arrow]
Add DataType::ListView and DataType::LargeListView #5492 [parquet] [arrow]
Improve documentation around handling of dictionary arrays in arrow flight #5487 [arrow] [arrow-flight]
Better memory limiting in parquet ArrowWriter #5484 [parquet]
Support Creating Non-Nullable Lists and Maps within a Struct #5482 [arrow]
[DISCUSSION] Better borrow propagation (e.g. RecordBatch::schema() to return &SchemaRef vs SchemaRef) #5463 [parquet] [arrow] [arrow-flight]
Build Scalar with ArrayRef #5459
AsyncArrowWriter doesn't limit underlying ArrowWriter to respect buffer-size #5450 [parquet]
Refine Display implementation for FlightError #5438 [arrow] [arrow-flight]
Better ergonomics for FixedSizeList and LargeList #5372 [arrow]
Update Flight proto #5367 [arrow] [arrow-flight]
Support check similar datatype but with different magnitudes #5358 [arrow]
Buffer memory usage for custom allocations is reported as 0 #5346 [arrow]
Can the ArrayBuilder trait be made Sync? #5344 [arrow]
support cast 'UTF8' to FixedSizeList #5339 [arrow]
Support Creating Non-Nullable Lists with ListBuilder #5330 [arrow]
ParquetRecordBatchStreamBuilder::new() panics instead of erroring out when opening a corrupted file #5315 [parquet]
Raw JSON Writer #5314 [arrow]
Add support for more fused boolean operations #5297 [arrow]
parquet: Allow disabling embed ARROW_SCHEMA_META_KEY added by the ArrowWriter #5296 [parquet]
Support casting strings like '2001-01-01 01:01:01' to Date32 #5280 [arrow]
Temporal Extract/Date Part Kernel #5266 [arrow]
Support for extracting hours/minutes/seconds/etc. from Time32/Time64 type in temporal kernels #5261 [arrow]
parquet: add method to get both the inner writer and the file metadata when closing SerializedFileWriter #5253 [parquet]
Release arrow-rs version 50.0.0 #5234

Fixed bugs:

Empty String Parses as Zero in Unreleased Arrow #5504 [arrow]
Unused import in nightly rust #5476 [parquet] [arrow] [arrow-flight]
Error The data type type List .. has no natural order when using arrow::compute::lexsort_to_indices with list and more than one column #5454 [arrow]
Wrong size assertion in arrow_buffer::builder::NullBufferBuilder::new_from_buffer #5445 [arrow]
Inconsistency between comments and code implementation #5430 [arrow]
OOB access in Buffer::from_iter #5412 [arrow]
Cast kernel doesn't return null for string to integral cases when overflowing under safe option enabled #5397 [arrow]
Make ffi consume variable layout arrays with empty offsets #5391 [arrow]
RecordBatch conversion from pyarrow loses Schema's metadata #5354 [arrow]
Debug output of Time32/Time64 arrays with invalid values has confusing nulls #5336 [arrow]
Removing a column from a RecordBatch drops schema metadata #5327 [arrow]
Panic when read an empty parquet file #5304 [parquet]
How to enable statistics for string columns? #5270 [parquet]
concat::tests::test_string_dictionary_merge failure fails on Mac / has different results in different platforms #5255 [arrow]

Documentation updates:

Minor: Add doc comments to GenericByteViewArray #5512 [arrow] (alamb)
Improve docs for logical and physical nulls even more #5434 [arrow] (alamb)
Add example of converting RecordBatches to JSON objects #5364 [arrow] (alamb)

Performance improvements:

improve float to string cast by ~20%-40% #5401 [arrow] (psvri)

Closed issues:

Add StringViewArray implementation and layout and basic construction + tests #5469 [parquet] [arrow]
Add DataType::Utf8View and DataType::BinaryView #5468 [parquet] [arrow]

Merged pull requests:

Deprecate array_to_json_array #5515 [arrow] (tustvold)
Fix integer parsing of empty strings (#5504) #5505 [arrow] (tustvold)
feat: clarifying comments in struct_builder.rs #5494 #5499 [arrow] (istvan-fodor)
Update proc-macro2 requirement from =1.0.78 to =1.0.79 #5498 [arrow] [arrow-flight] (dependabot[bot])
Add DataType::ListView and DataType::LargeListView #5493 [parquet] [arrow] (Kikkon)
Better document parquet pushdown #5491 [parquet] (tustvold)
Fix NullBufferBuilder::new_from_buffer wrong size assertion #5489 [arrow] (Kikkon)
Support dictionary encoding in structures for FlightDataEncoder, add documentation for arrow_flight::encode::Dictionary #5488 [arrow] [arrow-flight] (thinkharderdev)
Add MapBuilder::with_values_field to support non-nullable values (#5482) #5483 [arrow] (lasantosr)
feat: initial support string_view and binary_view, supports layout and basic construction + tests #5481 [arrow] (ariesdevil)
Add more comprehensive documentation on testing and benchmarking to CONTRIBUTING.md #5478 (monkwire)
Remove unused import detected by nightly rust #5477 [parquet] [arrow] [arrow-flight] (XiangpengHao)
Add RecordBatch::schema_ref #5474 [parquet] [arrow] [arrow-flight] (monkwire)
Provide access to inner Write for parquet writers #5471 [parquet] (tustvold)
Add DataType::Utf8View and DataType::BinaryView #5470 [parquet] [arrow] (XiangpengHao)
Update base64 requirement from 0.21 to 0.22 #5467 [parquet] [arrow] [arrow-flight] (dependabot[bot])
Minor: Fix formatting typo in Field::new_list_field #5464 [arrow] (alamb)
Fix test_string_dictionary_merge (#5255) #5461 [arrow] (tustvold)
Use Vec::from_iter in Buffer::from_iter #5460 [arrow] (Kikkon)
Document parquet writer memory limiting (#5450) #5457 [parquet] (tustvold)
Document UnionArray Panics #5456 [arrow] (Kikkon)
fix: lexsort_to_indices unsupported mixed types with list #5455 [arrow] (alamb)
Refine Display and Source implementation for error types #5439 [arrow] [arrow-flight] (BugenZhao)
Improve debug output of Time32/Time64 arrays #5428 [arrow] (monkwire)
Miri fix: Rename invalid_mut to without_provenance_mut #5418 [arrow] (Jefffrey)
Ensure addition/multiplications in when allocating buffers don't overflow #5417 [arrow] (Jefffrey)
Update Flight proto: PollFlightInfo & expiration time #5413 [arrow] [arrow-flight] (Jefffrey)
Add tests for serializing lists of dictionary encoded values to json #5399 [arrow] (jhorstmann)
Return null for overflow when casting string to integer under safe option enabled #5398 [arrow] (viirya)
Propagate error instead of panic for take_bytes #5395 [arrow] (viirya)
Improve like kernel by ~2% #5390 [arrow] (psvri)
Enable running arrow-array and arrow-arith with miri and avoid strict provenance warning #5387 [arrow] (jhorstmann)
Update to chrono 0.4.34 #5385 [arrow] (tustvold)
Return error instead of panic when reading invalid Parquet metadata #5382 [parquet] (mmaitre314)
Update tonic requirement from 0.10.0 to 0.11.0 #5380 [arrow] [arrow-flight] (dependabot[bot])
Update tonic-build requirement from =0.10.2 to =0.11.0 #5379 [arrow] [arrow-flight] (dependabot[bot])
Fix latest clippy lints #5376 [arrow] (tustvold)
feat: utility functions for creating FixedSizeList and LargeList dtypes #5373 [arrow] (universalmind303)
Minor(docs): update master to main for DataFusion/Ballista #5363 (caicancai)
Return an error instead of a panic when reading a corrupted Parquet file with mismatched column counts #5362 [parquet] (mmaitre314)
feat: support casting FixedSizeList with new child type #5360 [arrow] (wjones127)
Add more debugging info to StructBuilder validate_content #5357 [arrow] (viirya)
pyarrow: Preserve RecordBatch's schema metadata #5355 [arrow] (atwam)
Mark Encoding::BIT_PACKED as deprecated and document its compatibility issues #5348 [parquet] (jhorstmann)
Track the size of custom allocations for use via Array::get_buffer_memory_size #5347 [arrow] (jhorstmann)
fix: Return an error on type mismatch rather than panic (#4995) #5341 [parquet] (carols10cents)
Minor: support cast values to fixedsizelist #5340 [arrow] (Weijun-H)
Enhance Time32/Time64 support in date_part #5337 [arrow] (Jefffrey)
feat: add take_record_batch. #5333 [arrow] (RinChanNOWWW)
Add ListBuilder::with_field to support non nullable list fields (#5330) #5331 [arrow] (tustvold)
Don't omit schema metadata when removing column #5328 [arrow] (kylebarron)
Update proc-macro2 requirement from =1.0.76 to =1.0.78 #5324 [arrow] [arrow-flight] (dependabot[bot])
Enhance Date64 type documentation #5323 [arrow] (Jefffrey)
fix panic when decode a group with no child #5322 [parquet] (Liyixin95)
Minor/Doc Expand FlightSqlServiceClient::handshake doc #5321 [arrow] [arrow-flight] (devinjdangelo)
Refactor temporal extract date part kernels #5319 [arrow] (Jefffrey)
Add JSON writer benchmarks (#5314) #5317 [arrow] (tustvold)
Bump actions/cache from 3 to 4 #5308 (dependabot[bot])
Avro block decompression #5306 [arrow] (tustvold)
Result into error in case of endianness mismatches #5301 [arrow] (pangiole)
parquet: Add ArrowWriterOptions to skip embedding the arrow metadata #5299 [parquet] (evenyag)
Add support for more fused boolean operations #5298 [arrow] (RTEnzyme)
Support Parquet Byte Stream Split Encoding #5293 [parquet] (mwlon)
Extend string parsing support for Date32 #5282 [arrow] (gruuya)
Bring some methods over from ArrowWriter to the async version #5251 [parquet] (AdamGS)

Open source →

50.0.012 Jan 2024Deprecated

Fix deprecated note for Buffer::from_raw_parts \#5190 [arrow] (mbrobbel)

Full Changelog

Breaking changes:

Make regexp_match take scalar pattern and flag #5245 [arrow] (viirya)
Use Vec in ColumnReader (#5177) #5193 [parquet] (tustvold)
Remove SIMD Feature #5184 [arrow] (tustvold)
Use Total Ordering for Aggregates and Refactor for Better Auto-Vectorization #5100 [arrow] (jhorstmann)
Allow the zip compute function to operator on Scalar values via Datum #5086 [arrow] (Nathan-Fenner)
Improve C Data Interface and Add Integration Testing Entrypoints #5080 [arrow] (pitrou)
Parquet: read/write f16 for Arrow #5003 [parquet] (Jefffrey)

Implemented enhancements:

Support get offsets or blocks info from arrow file. #5252 [arrow]
Make regexp_match take scalar pattern and flag #5246 [arrow]
Cannot access pen state website on arrow-row #5238 [arrow]
RecordBatch with_schema's error message is hard to read #5227 [arrow]
Support cast between StructArray. #5219 [arrow]
Remove nightly-only simd feature and related code in ArrowNumericType #5185 [arrow]
Use Vec instead of Slice in ColumnReader #5177 [parquet]
Request to Memmap Arrow IPC files on disk #5153 [arrow]
GenericColumnReader::read_records Yields Truncated Records #5150 [parquet]
Nested Schema Projection #5148 [parquet] [arrow]
Support specifying quote and escape in Csv WriterBuilder #5146 [arrow]
Support casting of Float16 with other numeric types #5138 [arrow]
Parquet: read parquet metadata with page index in async and with size hints #5129 [parquet]
Cast from floating/timestamp to timestamp/floating #5122 [arrow]
Support Casting List To/From LargeList in Cast Kernel #5113 [arrow]
Expose a path for converting bytes::Bytes into arrow_buffer::Buffer without copy #5104 [arrow]
API inconsistency of ListBuilder make it hard to use as nested builder #5098 [arrow]
Parquet: don't truncate min/max statistics for float16 and decimal when writing file #5075 [parquet]
Parquet: derive boundary order when writing columns #5074 [parquet]
Support new Arrow PyCapsule Interface for Python FFI #5067 [arrow]
48.0.1 arrow patch release #5050 [parquet] [arrow]
Binary columns do not receive truncated statistics #5037 [parquet]
Re-evaluate Explicit SIMD Aggregations #5032 [arrow]
Min/Max Kernels Should Use Total Ordering #5031 [arrow]
Allow zip compute kernel to take Scalar / Datum #5011 [arrow]
Add Float16/Half-float logical type to Parquet #4986 [parquet]
feat: cast (Large)List to FixedSizeList #5081 [arrow] (wjones127)
Update Parquet Encoding Documentation #5051 [parquet]

Fixed bugs:

json schema inference can't handle null field turned into object field in subsequent rows #5215 [arrow]
Invalid trailing content after Z in timezone is ignored #5182 [arrow]
Take panics on a fixed size list array when given null indices #5169 [arrow]
EnabledStatistics::Page does not take effect on ByteArrayEncoder #5162 [parquet]
Parquet: ColumnOrder not being written when writing parquet files #5152 [parquet]
Parquet: Interval columns shouldn't write min/max stats #5145 [parquet]
cast Utf8 to decimal failure #5127 [arrow]
coerce_primitive not honored when decoding from serde object #5095 [arrow]
Unsound MutableArrayData Constructor #5091 [arrow]
RowGroupReader.get_row_iter() fails with Path ColumnPath not found #5064 [parquet]
cast format 'yyyymmdd' to Date32 give a error #5044 [arrow]

Performance improvements:

ArrowArrayStreamReader imports FFI_ArrowSchema on each iteration #5103 [arrow]

Closed issues:

Working example of list_flights with ObjectStore #5116
(object_store) Error broken pipe on S3 multipart upload #5106

Merged pull requests:

Update parquet object_store dependency to 0.9.0 #5290 [parquet] (tustvold)
Update proc-macro2 requirement from =1.0.75 to =1.0.76 #5289 [arrow] [arrow-flight] (dependabot[bot])
Enable JS tests again #5287 (domoritz)
Update proc-macro2 requirement from =1.0.74 to =1.0.75 #5279 [arrow] [arrow-flight] (dependabot[bot])
Update proc-macro2 requirement from =1.0.73 to =1.0.74 #5271 [arrow] [arrow-flight] (dependabot[bot])
Update proc-macro2 requirement from =1.0.71 to =1.0.73 #5265 [arrow] [arrow-flight] (dependabot[bot])
Update docs for datatypes #5260 [arrow] (Jefffrey)
Don't suppress errors in ArrowArrayStreamReader #5256 [arrow] (tustvold)
Add IPC FileDecoder #5249 [arrow] (tustvold)
optimize the next function of ArrowArrayStreamReader #5248 [arrow] (doki23)
ci: Fail Miri CI on first failure #5243 (Jefffrey)
Remove 'unwrap' from Result #5241 [parquet] (zeevm)
Update arrow-row docs URL #5239 [arrow] (thomas-k-cameron)
Improve regexp kernels performance by avoiding cloning Regex #5235 [arrow] (viirya)
Update proc-macro2 requirement from =1.0.70 to =1.0.71 #5231 [arrow] [arrow-flight] (dependabot[bot])
Minor: Improve comments and errors for ArrowPredicate #5230 [parquet] (alamb)
Bump actions/upload-pages-artifact from 2 to 3 #5229 (dependabot[bot])
make with_schema's error more readable #5228 [arrow] (shuoli84)
Use try_new when casting between structs to propagate error #5226 [arrow] (viirya)
feat(cast): support cast between struct #5221 [arrow] (my-vegetable-has-exploded)
Add entries to MapBuilder to return both key and value array builders #5218 [arrow] (viirya)
fix(json): fix inferring object after field was null #5216 [arrow] (kskalski)
Support MapBuilder in make_builder #5210 [arrow] (viirya)
impl From<OffsetBuffer<T>> for ScalarBuffer<T> #5203 [arrow] (mbrobbel)
impl From<BufferBuilder<T>> for Buffer #5202 [arrow] (mbrobbel)
impl From<BufferBuilder<T>> for ScalarBuffer<T> #5201 [arrow] (mbrobbel)
feat: Support quote and escape in Csv WriterBuilder #5196 [arrow] (my-vegetable-has-exploded)
chore: simplify cast_string_to_interval #5195 [arrow] (jackwener)
Clarify interval comparison behavior with documentation and tests #5192 [arrow] (alamb)
Add BooleanArray::into_parts method #5191 [arrow] (mbrobbel)
Fix deprecated note for Buffer::from_raw_parts #5190 [arrow] (mbrobbel)
Fix: Ensure Timestamp Parsing Rejects Characters After 'Z #5189 [arrow] (razeghi71)
Simplify parquet statistics generation #5183 [parquet] (tustvold)
Parquet: Ensure page statistics are written only when conifgured from the Arrow Writer #5181 [parquet] (AdamGS)
Blockwise IO in IPC FileReader (#5153) #5179 [arrow] (tustvold)
Replace ScalarBuffer in Parquet with Vec (#1849) (#5177) #5178 [parquet] (tustvold)
Bump actions/setup-python from 4 to 5 #5175 (dependabot[bot])
Add LargeListBuilder to make_builder #5171 [arrow] (viirya)
fix: ensure take_fixed_size_list can handle null indices #5170 (westonpace)
Removing redundant as casts in parquet #5168 [parquet] (psvri)
Bump actions/labeler from 4.3.0 to 5.0.0 #5167 (dependabot[bot])
improve: make RunArray displayable #5166 [arrow] (yukkit)
ci: Add cargo audit CI action #5160 [arrow] (Jefffrey)
Parquet: write column_orders in FileMetaData #5158 [parquet] (Jefffrey)
Adding is_null datatype shortcut method #5157 [arrow] (comphead)
Parquet: don't truncate f16/decimal min/max stats #5154 [parquet] (Jefffrey)
Support nested schema projection (#5148) #5149 [arrow] (tustvold)
Parquet: omit min/max for interval columns when writing stats #5147 [parquet] (Jefffrey)
Deprecate Fields::remove and Schema::remove #5144 [arrow] (tustvold)
Support casting of Float16 with other numeric types #5139 [arrow] (viirya)
Parquet: Make MetadataLoader public #5137 [parquet] (AdamGS)
Add FileReaderBuilder for arrow-ipc to allow reading large no. of column files #5136 [arrow] (Jefffrey)
Parquet: clear metadata and project fields of ParquetRecordBatchStream::schema #5135 [parquet] (Jefffrey)
JSON: write struct array nulls as null #5133 [arrow] (Jefffrey)
Update proc-macro2 requirement from =1.0.69 to =1.0.70 #5131 [arrow] [arrow-flight] (dependabot[bot])
Fix negative decimal string #5128 [arrow] (viirya)
Cleanup list casting and support nested lists (#5113) #5124 [arrow] (tustvold)
Cast from numeric/timestamp to timestamp/numeric #5123 [arrow] (viirya)
Improve cast docs #5114 [arrow] (tustvold)
Update prost-build requirement from =0.12.2 to =0.12.3 #5112 [arrow] [arrow-flight] (dependabot[bot])
Parquet: derive boundary order when writing #5110 [parquet] (Jefffrey)
Implementing ArrayBuilder for Box<dyn ArrayBuilder> #5109 [arrow] (viirya)
Fix 'ColumnPath not found' error reading Parquet files with nested REPEATED fields #5102 [parquet] (mmaitre314)
fix: coerce_primitive for serde decoded data #5101 [arrow] (fansehep)
Extend aggregation benchmarks #5096 [arrow] (jhorstmann)
Expand parquet crate overview doc #5093 [parquet] (mmaitre314)
Ensure arrays passed to MutableArrayData have same type (#5091) #5092 [arrow] (tustvold)
Update prost-build requirement from =0.12.1 to =0.12.2 #5088 [arrow] [arrow-flight] (dependabot[bot])
Add FFI from_raw #5082 [arrow] (tustvold)
[fix #5044] Support converting 'yyyymmdd' format to date #5078 [arrow] (Tangruilin)
Enable truncation of binary statistics columns #5076 [parquet] (emcake)

Open source →

48.0.023 Oct 2023

Evaluate null\_regex for string type in csv \(now such values will be parsed as Null rather than ""\) \#4942 [arrow] (haohuaijin)

Full Changelog

Breaking changes:

Evaluate null_regex for string type in csv (now such values will be parsed as Null rather than "") #4942 [arrow] (haohuaijin)
fix(csv)!: infer null for empty column. #4910 [arrow] (kskalski)
feat: log headers/trailers in flight CLI (+ minor fixes) #4898 [arrow] [arrow-flight] (crepererum)
fix(arrow-json)!: include null fields in schema inference with a type of Null #4894 [arrow] (kskalski)
Mark OnCloseRowGroup Send #4893 [parquet] (devinjdangelo)
Specialize Thrift Decoding (~40% Faster) (#4891) #4892 [parquet] (tustvold)
Make ArrowRowGroupWriter Public and SerializedRowGroupWriter Send #4850 [parquet] (devinjdangelo)

Implemented enhancements:

Allow schema fields to merge with Null datatype #4901 [arrow]
Add option to FlightDataEncoder to always send dictionaries #4895 [arrow] [arrow-flight]
Rework Thrift Encoding / Decoding of Parquet Metadata #4891 [parquet]
Plans for supporting Extension Array to support Fixed shape tensor Array #4890
Implement Take for UnionArray #4882 [arrow]
Check precision overflow for casting floating to decimal #4865 [arrow]
Replace lexical #4774 [arrow]
Add read access to settings in csv::WriterBuilder #4735 [arrow]
Improve the performance of "DictionaryValue" row encoding #4712 [arrow] [arrow-flight]

Fixed bugs:

Should we make blank values and empty string to None in csv? #4939 [arrow]
[FlightSQL] SubstraitPlan structure is not exported #4932 [arrow] [arrow-flight]
Loading page index breaks skipping of pages with nested types #4921 [parquet]
CSV schema inference assumes Utf8 for empty columns #4903 [arrow]
parquet: Field Ids are not read from a Parquet file without serialized arrow schema #4877 [parquet]
make_primitive_scalar function loses DataType Internal information #4851 [arrow]
StructBuilder doesn't handle nulls correctly for empty structs #4842 [arrow]
NullArray::is_null() returns false incorrectly #4835 [arrow]
cast_string_to_decimal should check precision overflow #4829 [arrow]
Null fields are omitted by infer_json_schema_from_seekable #4814 [arrow]

Closed issues:

Support for reading JSON Array to Arrow #4905 [arrow]

Merged pull requests:

Assume Pages Delimit Records When Offset Index Loaded (#4921) #4943 [parquet] (tustvold)
Update pyo3 requirement from 0.19 to 0.20 #4941 [arrow] (crepererum)
Add FileWriter schema getter #4940 [arrow] (haixuanTao)
feat: support parsing for parquet writer option #4938 [parquet] (fansehep)
Export SubstraitPlan structure in arrow_flight::sql (#4932) #4933 [arrow] [arrow-flight] (amartins23)
Update zstd requirement from 0.12.0 to 0.13.0 #4923 [parquet] [arrow] (dependabot[bot])
feat: add method for async read bloom filter #4917 [parquet] (hengfeiyang)
Minor: Clarify rationale for FlightDataEncoder API, add examples #4916 [arrow] [arrow-flight] (alamb)
Update regex-syntax requirement from 0.7.1 to 0.8.0 #4914 [arrow] (dependabot[bot])
feat: document & streamline flight SQL CLI #4912 [arrow] [arrow-flight] (crepererum)
Support Arbitrary JSON values in JSON Reader (#4905) #4911 [arrow] (tustvold)
Cleanup CSV WriterBuilder, Default to AutoSI Second Precision (#4735) #4909 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.68 to =1.0.69 #4907 [arrow] [arrow-flight] (dependabot[bot])
chore: add csv example #4904 [arrow] (fansehep)
feat(schema): allow null fields to be merged with other datatypes #4902 [arrow] (kskalski)
Update proc-macro2 requirement from =1.0.67 to =1.0.68 #4900 [arrow] [arrow-flight] (dependabot[bot])
Add option to FlightDataEncoder to always resend batch dictionaries #4896 [arrow] [arrow-flight] (alexwilcoxson-rel)
Fix integration tests #4889 (tustvold)
Support Parsing Avro File Headers #4888 (tustvold)
Support parquet bloom filter length #4885 [parquet] (letian-jiang)
Replace lz4 with lz4_flex Allowing Compilation for WASM #4884 [parquet] [arrow] (tustvold)
Implement Take for UnionArray #4883 [arrow] (avantgardnerio)
Update tonic-build requirement from =0.10.1 to =0.10.2 #4881 [arrow] [arrow-flight] (dependabot[bot])
parquet: Read field IDs from Parquet Schema #4878 [parquet] (Samrose-Ahmed)
feat: improve flight CLI error handling #4873 [arrow] [arrow-flight] (crepererum)
Support Encoding Parquet Columns in Parallel #4871 [parquet] (tustvold)
Check precision overflow for casting floating to decimal #4866 [arrow] (viirya)
Make align_buffers as public API #4863 [arrow] (viirya)
Enable new integration tests (#4828) #4862 (tustvold)
Faster Serde Integration (~80% faster) #4861 [arrow] (tustvold)
fix: make_primitive_scalar bug #4852 [arrow] (JasonLi-cn)
Update tonic-build requirement from =0.10.0 to =0.10.1 #4846 [arrow] [arrow-flight] (dependabot[bot])
Allow Constructing Non-Empty StructArray with no Fields (#4842) #4845 [arrow] (tustvold)
Refine documentation to Array::is_null #4838 [arrow] (alamb)
fix: add missing precision overflow checking for cast_string_to_decimal #4830 [arrow] (jonahgao)

Open source →

47.0.022 Sept 2023Deprecated

More chrono deprecations \#4822 [arrow] (tustvold)

Full Changelog

Breaking changes:

Make FixedSizeBinaryArray value_data return a reference #4820 [arrow]
Update prost to v0.12.1 #4825 [arrow] [arrow-flight] (tustvold)
feat: FixedSizeBinaryArray::value_data return reference #4821 [arrow] (wjones127)
Stateless Row Encoding / Don't Preserve Dictionaries in RowConverter (#4811) #4819 [arrow] [arrow-flight] (tustvold)
fix: entries field is non-nullable #4808 [arrow] (wjones127)
Fix flight sql do put handling, add bind parameter support to FlightSQL cli client #4797 [arrow] [arrow-flight] (suremarc)
Remove unused dyn_cmp_dict feature #4766 [arrow] (tustvold)
Add underlying std::io::Error to IoError and add IpcError variant #4726 [arrow] [arrow-flight] (alexandreyc)

Implemented enhancements:

Row Format Adapative Block Size #4812 [arrow]
Stateless Row Conversion #4811 [arrow] [arrow-flight]
Add option to specify custom null values for CSV reader #4794 [arrow]
parquet::record::RowIter cannot be customized with batch_size and defaults to 1024 #4782 [parquet]
DynScalar abstraction (something that makes it easy to create scalar Datums) #4781 [arrow]
Datum is not exported as part of arrow (it is only exported in arrow_array) #4780 [arrow]
Scalar is not exported as part of arrow (it is only exported in arrow_array) #4779 [arrow]
Support IntoPyArrow for impl RecordBatchReader #4730 [arrow]
Datum Based String Kernels #4595 [arrow] [arrow-flight]

Fixed bugs:

MapArray::new_from_strings creates nullable entries field #4807 [arrow]
pyarrow module can't roundtrip tensor arrays #4805 [arrow]
concat_batches errors with "schema mismatch" error when only metadata differs #4799 [arrow]
panic in cmp kernels with DictionaryArrays: Option::unwrap() on a None value' #4788 [arrow]
stream ffi panics if schema metadata values aren't valid utf8 #4750 [arrow]
Regression: Incorrect Sorting of *ListArray in 46.0.0 #4746 [arrow]
Row is no longer comparable after reuse #4741 [arrow]
DoPut FlightSQL handler inadvertently consumes schema at start of Request<Streaming<FlightData>> #4658
Return error when converting schema #4752 [arrow] (wjones127)
Implement PyArrowType for Box<dyn RecordBatchReader + Send> #4751 [arrow] (wjones127)

Closed issues:

Building arrow-rust for target wasm32-wasi falied to compile packed_simd_2 #4717

Merged pull requests:

Respect FormatOption::nulls for NullArray #4836 [arrow] (tustvold)
Fix merge_dictionary_values in selection kernels #4833 [arrow] (tustvold)
Fix like scalar null #4832 [arrow] (tustvold)
More chrono deprecations #4822 [arrow] (tustvold)
Adaptive Row Block Size (#4812) #4818 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.66 to =1.0.67 #4816 [arrow] [arrow-flight] (dependabot[bot])
Do not check schema for equality in concat_batches #4815 [arrow] (alamb)
fix: export record batch through stream #4806 [arrow] (wjones127)
Improve CSV Reader Benchmark Coverage of Small Primitives #4803 [arrow] (tustvold)
csv: Add option to specify custom null values #4795 [arrow] (vrongmeal)
Expand docstring and add example to Scalar #4793 [arrow] (alamb)
Re-export array crate root (#4780) (#4779) #4791 [arrow] (tustvold)
Fix DictionaryArray::normalized_keys (#4788) #4789 [arrow] (tustvold)
Allow custom tree builder for parquet::record::RowIter #4783 [parquet] (YuraKotov)
Bump actions/checkout from 3 to 4 #4767 (dependabot[bot])
fix: avoid panic if offset index not exists. #4761 [parquet] (RinChanNOWWW)
Relax constraints on PyArrowType #4757 (tustvold)
Chrono deprecations #4748 [arrow] (tustvold)
Fix List Sorting, Revert Removal of Rank Kernels #4747 [arrow] (tustvold)
Clear row buffer before reuse #4742 [arrow] (yjshen)
Datum based like kernels (#4595) #4732 [arrow] [arrow-flight] (tustvold)
feat: expose DoGet response headers & trailers #4727 [arrow] [arrow-flight] (crepererum)
Cleanup length and bit_length kernels #4718 [arrow] (tustvold)

Open source →

46.0.024 Aug 2023Deprecated

Remove deprecated limit kernel \#4597 [arrow] (tustvold)

Full Changelog

Breaking changes:

API improvement: batches_to_flight_data forces clone #4656 [arrow]
Add AnyDictionary Abstraction and Take ArrayRef in DictionaryArray::with_values #4707 [arrow] (tustvold)
Cleanup parquet type builders #4706 [parquet] (tustvold)
Take kernel dyn Array #4705 [arrow] (tustvold)
Improve ergonomics of Scalar #4704 [arrow] (tustvold)
Datum based comparison kernels (#4596) #4701 [parquet] [arrow] [arrow-flight] (tustvold)
Improve Array Logical Nullability #4691 [parquet] [arrow] (tustvold)
Validate ArrayData Buffer Alignment and Automatically Align IPC buffers (#4255) #4681 [arrow] (tustvold)
More intuitive bool-to-string casting #4666 [arrow] (fsdvh)
enhancement: batches_to_flight_data use a schema ref as param. #4665 [arrow] [arrow-flight] (jackwener)
fix: from_thrift avoid panic when stats in invalid. #4642 [parquet] (jackwener)
bug: Add some missing field in row group metadata: ordinal, total co… #4636 [parquet] (liurenjie1024)
Remove deprecated limit kernel #4597 [arrow] (tustvold)

Implemented enhancements:

parquet: support setting the field_id with an ArrowWriter #4702 [parquet]
Support references in i256 arithmetic ops #4694 [arrow]
Precision-Loss Decimal Arithmetic #4664 [arrow]
Faster i256 Division #4663 [arrow]
Support concat_batches for 0 columns #4661 [arrow]
filter_record_batch should support filtering record batch without columns #4647 [arrow]
Improve speed of lexicographical_partition_ranges #4614 [arrow]
object_store: multipart ranges for HTTP #4612
Add Rank Function #4606 [arrow]
Datum Based Comparison Kernels #4596 [parquet] [arrow] [arrow-flight]
Convenience method to create DataType::List correctly #4544 [arrow]
Remove Deprecated Arithmetic Kernels #4481 [arrow]
Equality kernel where null==null gives true #4438 [arrow]

Fixed bugs:

Parquet ArrowWriter Ignores Nulls in Dictionary Values #4690 [parquet] [arrow]
Schema Nullability Validation Fails to Account for Dictionary Nulls #4689 [parquet] [arrow]
Comparison Kernels Ignore Nulls in Dictionary Values #4688 [parquet] [arrow]
Casting List to String Ignores Format Options #4669 [arrow]
Double free in C Stream Interface #4659 [arrow]
CI Failing On Packed SIMD #4651 [arrow]
RowInterner::size() much too low for high cardinality dictionary columns #4645 [arrow]
Decimal PrimitiveArray change datatype after try_unary #4644
Better explanation in docs for Dictionary field encoding using RowConverter #4639 [arrow]
List(FixedSizeBinary) array equality check may return wrong result #4637 [arrow]
arrow::compute::nullif panics if NullArray is provided #4634 [arrow]
Empty lists in FixedSizeListArray::try_new is not handled #4623 [arrow]
Bounds checking in MutableBuffer::set_null_bits can be bypassed #4620 [arrow]
TypedDictionaryArray Misleading Null Behaviour #4616 [parquet] [arrow]
bug: Parquet writer missing row group metadata fields such as compressed_size, file offset. #4610 [parquet]
new_null_array generates an invalid union array #4600 [arrow]
Footer parsing fails for very large parquet file. #4592 [parquet]
bug(parquet): Disabling global statistics but enabling for particular column breaks reading #4587 [parquet]
arrow::compute::concat panics for dense union arrays with non-trivial type IDs #4578 [arrow]

Closed issues:

[object_store] when Create a AmazonS3 instance work with MinIO without set endpoint got error MissingRegion #4617

Merged pull requests:

Add distinct kernels (#960) (#4438) #4716 [arrow] (tustvold)
Update parquet object_store 0.7 #4715 [parquet] (tustvold)
Support Field ID in ArrowWriter (#4702) #4710 [parquet] (tustvold)
Remove rank kernels #4703 [arrow] (tustvold)
Support references in i256 arithmetic ops #4692 [arrow] (viirya)
Cleanup DynComparator (#2654) #4687 [arrow] (tustvold)
Separate metadata fetch from ArrowReaderBuilder construction (#4674) #4676 [parquet] (tustvold)
cleanup some assert() with error propagation #4673 [parquet] (zeevm)
Faster i256 Division (2-100x) (#4663) #4672 [arrow] (tustvold)
Fix MSRV CI #4671 (tustvold)
Fix equality of nested nullable FixedSizeBinary (#4637) #4670 [arrow] (tustvold)
Use ArrayFormatter in cast kernel #4668 [arrow] (tustvold)
Minor: Improve API docs for FlightSQL metadata builders #4667 [arrow] [arrow-flight] (alamb)
Support concat_batches for 0 columns #4662 [arrow] (Dandandan)
fix ownership of c stream error #4660 [arrow] (wjones127)
Minor: Fix illustration for dict encoding #4657 [arrow] (JayjeetAtGithub)
minor: move comment to the correct location #4655 [arrow] (jackwener)
Update packed_simd and run miri tests on simd code #4654 [arrow] (jhorstmann)
impl From<Vec<T>> for BufferBuilder and MutableBuffer #4650 [arrow] (mbrobbel)
Filter record batch with 0 columns #4648 [arrow] (Dandandan)
Account for child Bucket size in OrderPreservingInterner #4646 [arrow] (alamb)
Implement Default,Extend and FromIterator for BufferBuilder #4638 [arrow] (mbrobbel)
fix(select): handle NullArray in nullif #4635 [arrow] (kawadakk)
Move BufferBuilder to arrow-buffer #4630 [arrow] (mbrobbel)
allow zero sized empty fixed #4626 [arrow] (smiklos)
fix: compute_dictionary_mapping use wrong offsetSize #4625 [arrow] (jackwener)
impl FromIterator for MutableBuffer #4624 [arrow] (mbrobbel)
expand docs for FixedSizeListArray #4622 [arrow] (smiklos)
fix(buffer): panic on end index overflow in MutableBuffer::set_null_bits #4621 [arrow] (kawadakk)
impl Default for arrow_buffer::buffer::MutableBuffer #4619 [arrow] (mbrobbel)
Minor: improve docs and add example for lexicographical_partition_ranges #4615 [arrow] (alamb)
Cleanup sort #4613 [arrow] (tustvold)
Add rank function (#4606) #4609 [arrow] (tustvold)
Add more docs and examples for ListArray and OffsetsBuffer #4607 [arrow] (alamb)
Simplify dictionary sort #4605 [arrow] (tustvold)
Consolidate sort benchmarks #4604 [arrow] (tustvold)
Don't Reorder Nulls in sort_to_indices (#4545) #4603 [arrow] (tustvold)
fix(data): create child arrays of correct length when building a sparse union null array #4601 [arrow] (kawadakk)
Use u32 metadata_len when parsing footer of parquet. #4599 [parquet] (Berrysoft)
fix(data): map type ID to child index before indexing a union child array #4598 [arrow] (kawadakk)
Remove deprecated arithmetic kernels (#4481) #4594 [arrow] (tustvold)
Test Disabled Page Statistics (#4587) #4589 [parquet] (tustvold)
Cleanup ArrayData::buffers #4583 [arrow] (tustvold)
Use contains_nulls in ArrayData equality of byte arrays #4582 [arrow] (tustvold)
Vectorized lexicographical_partition_ranges (~80% faster) #4575 [arrow] (tustvold)
chore: add datatype new_list #4561 [arrow] (fansehep)

Open source →

42.0.020 Jun 2023Deprecated

deprecate: as\_decimal\_array \#4363 [arrow] (izveigor)

Full Changelog

Breaking changes:

Remove 64-bit to 32-bit Cast from IPC Reader #4412 [arrow] (ming08108)
Truncate Min/Max values in the Column Index #4389 [parquet] (AdamGS)
feat(flight): harmonize server metadata APIs #4384 [arrow] [arrow-flight] (roeap)
Move record delimiting into ColumnReader (#4365) #4376 [parquet] (tustvold)
Changed array_to_json_array to take &dyn Array #4370 [arrow] (dadepo)
Make PrimitiveArray::with_timezone consuming #4366 [parquet] [arrow] (tustvold)

Implemented enhancements:

Add doc example of constructing a MapArray #4385 [arrow]
Support millisecond and microsecond functions #4374 [arrow]
Changed array_to_json_array to take &dyn Array #4369 [arrow]
compute::ord kernel for getting min and max of two scalar/array values #4347 [arrow]
Release 41.0.0 of arrow/arrow-flight/parquet/parquet-derive #4346
Refactor CAST tests to use new cast array syntax #4336 [arrow]
pass bytes directly to parquet's KeyValue #4317
PyArrow conversions could return TypeError if provided incorrect Python type #4312 [arrow]
Have array_to_json_array support Map #4297 [arrow]
FlightSQL: Add helpers to create CommandGetXdbcTypeInfo responses (XdbcInfoValue and builders) #4257 [arrow] [arrow-flight]
Have array_to_json_array support FixedSizeList #4248 [arrow]
Truncate ColumnIndex ByteArray Statistics #4126 [parquet]
Arrow compute kernel regards selection vector #4095 [arrow]

Fixed bugs:

Wrongly calculated data compressed length in IPC writer #4410 [arrow]
Take Kernel Handles Nullable Indices Incorrectly #4404 [arrow]
StructBuilder::new Doesn't Validate Builder DataTypes #4397 [arrow]
Parquet error: Not all children array length are the same! when using RowSelection to read a parquet file #4396
RecordReader::skip_records Is Incorrect for Repeated Columns #4368 [parquet]
List-of-String Array panics in the presence of row filters #4365 [parquet]
Fail to read block compressed gzip files with parquet-fromcsv #4173 [parquet]

Closed issues:

Have a parquet file not able to be deduped via arrow-rs, complains about Decimal precision? #4356
Question: Could we move dict_id, dict_is_ordered into DataType? #4325

Merged pull requests:

Fix reading gzip file with multiple gzip headers in parquet-fromcsv. #4419 [parquet] (ghuls)
Cleanup nullif kernel #4416 [arrow] (tustvold)
Fix bug in IPC logic that determines if the buffer should be compressed or not #4411 [arrow] (lwpyr)
Faster unpacking of Int32Type dictionary #4406 [arrow] (tustvold)
Improve take kernel performance on primitive arrays, fix bad null index handling (#4404) #4405 [arrow] (tustvold)
More take benchmarks #4403 [arrow] (tustvold)
Add BooleanBuffer::new_unset and BooleanBuffer::new_set and BooleanArray::new_null constructors #4402 [arrow] (tustvold)
Add PrimitiveBuilder type constructors #4401 [arrow] (tustvold)
StructBuilder Validate Child Data (#4397) #4400 [arrow] (tustvold)
Faster UTF-8 truncation #4399 [parquet] (tustvold)
Minor: Derive Hash impls for CastOptions and FormatOptions #4395 [arrow] (alamb)
Fix typo in README #4394 [arrow] [arrow-flight] (okue)
Improve parquet WriterProperites and ReaderProperties docs #4392 [parquet] (alamb)
Cleanup downcast macros #4391 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.59 to =1.0.60 #4388 [arrow] [arrow-flight] (dependabot[bot])
Consolidate ByteArray::from_iterator #4386 [arrow] (tustvold)
Add MapArray constructors and doc example #4382 [arrow] (tustvold)
Documentation Improvements #4381 [arrow] (tustvold)
Add NullBuffer and BooleanBuffer From conversions #4380 [arrow] (tustvold)
Add more examples of constructing Boolean, Primitive, String, and Decimal Arrays, and From impl for i256 #4379 [arrow] (alamb)
Add ListArrayReader benchmarks #4378 [parquet] (tustvold)
Update comfy-table requirement from 6.0 to 7.0 #4377 [arrow] (dependabot[bot])
feat: Addmicrosecond and millisecond kernels #4375 [arrow] (izveigor)
Update hashbrown requirement from 0.13 to 0.14 #4373 [parquet] [arrow] (dependabot[bot])
minor: use as_boolean to resolve TODO #4367 [arrow] (jackwener)
Have array_to_json_array support MapArray #4364 [arrow] (dadepo)
deprecate: as_decimal_array #4363 [arrow] (izveigor)
Add support for FixedSizeList in array_to_json_array #4361 [arrow] (dadepo)
refact: use as_primitive in cast.rs test #4360 [arrow] (Weijun-H)
feat(flight): add xdbc type info helpers #4359 [arrow] [arrow-flight] (roeap)
Minor: float16 to json #4358 [arrow] (izveigor)
Raise TypeError on PyArrow import #4316 [arrow] (wjones127)
Arrow Cast: Fixed Point Arithmetic for Interval Parsing #4291 [arrow] (mr-brobot)

Open source →

41.0.06 Jun 2023Deprecated

Use page_size consistently, deprecate pagesize in parquet WriterProperties \#4313 [parquet] (alamb)

Full Changelog

Breaking changes:

Rename list contains kernels to in_list (#4289) #4342 [parquet] [arrow] (tustvold)
Move BooleanBufferBuilder and NullBufferBuilder to arrow_buffer #4338 [arrow] (tustvold)
Add separate row_count and level_count to PageMetadata (#4321) #4326 [parquet] (tustvold)
Treat legacy TIMSETAMP_X converted types as UTC #4309 [parquet] (sergiimk)
Simplify parquet PageIterator #4306 [parquet] (tustvold)
Add Builder style APIs and docs for FlightData, FlightInfo, FlightEndpoint, Locaation and Ticket #4294 [arrow] [arrow-flight] (alamb)
Make GenericColumnWriter Send #4287 [parquet] (tustvold)
feat: update flight-sql to latest specs #4250 [arrow] [arrow-flight] (roeap)
feat(api!): make ArrowArrayStreamReader Send #4232 [arrow] (wjones127)

Implemented enhancements:

Make SerializedRowGroupReader::new() Public #4330 [parquet]
Speed up i256 division and remainder operations #4302 [arrow]
export function parquet_to_array_schema_and_fields #4298 [parquet]
FLightSQL: add helpers to create CommandGetCatalogs, CommandGetSchemas, and CommandGetTables requests #4295 [arrow] [arrow-flight]
Make ColumnWriter Send #4286 [parquet]
Add Builder for FlightInfo to make it easier to create new requests #4281 [arrow] [arrow-flight]
Support Writing/Reading Decimal256 to/from Parquet #4264 [parquet]
FlightSQL: Add helpers to create CommandGetSqlInfo responses (SqlInfoValue and builders) #4256 [arrow] [arrow-flight]
Update flight-sql implementation to latest specs #4249 [arrow] [arrow-flight]
Make ArrowArrayStreamReader Send #4222 [arrow]
Support writing FixedSizeList to Parquet #4214 [parquet]
Cast between Intervals #4181 [arrow]
Splice Parquet Data #4155 [parquet]
CSV Schema More Flexible Timestamp Inference #4131 [arrow]

Fixed bugs:

Doc for arrow_flight::sql is missing enums that are Xdbc related #4339 [arrow] [arrow-flight]
concat_batches panics with total_len <= bit_len assertion for records with lists #4324 [arrow]
Incorrect PageMetadata Row Count returned for V1 DataPage #4321 [parquet]
[parquet] Not following the spec for TIMESTAMP_MILLIS legacy converted types #4308 [parquet]
ambiguous glob re-exports of contains_utf8 #4289 [parquet] [arrow]
flight_sql_client --header "key: value" yields a value with a leading whitespace #4270 [arrow] [arrow-flight]
Casting Timestamp to date is off by one day for dates before 1970-01-01 #4211 [arrow]

Merged pull requests:

Don't infer 16-byte decimal as decimal256 #4349 [parquet] (tustvold)
Fix MutableArrayData::extend_nulls (#1230) #4343 [arrow] (tustvold)
Update FlightSQL metadata locations, names and docs #4341 [arrow] [arrow-flight] (alamb)
chore: expose Xdbc related FlightSQL enums #4340 [arrow] [arrow-flight] (appletreeisyellow)
Update pyo3 requirement from 0.18 to 0.19 #4335 [arrow] (dependabot[bot])
Skip unnecessary null checks in MutableArrayData #4333 [arrow] (tustvold)
feat: add read parquet by custom rowgroup examples #4332 [parquet] (sundy-li)
Make SerializedRowGroupReader::new() public #4331 [parquet] (burmecia)
Don't split record across pages (#3680) #4327 [parquet] (tustvold)
fix date conversion if timestamp below unixtimestamp #4323 [arrow] (comphead)
Short-circuit on exhausted page in skip_records #4320 [parquet] (tustvold)
Handle trailing padding when skipping repetition levels (#3911) #4319 [parquet] (tustvold)
Use page_size consistently, deprecate pagesize in parquet WriterProperties #4313 [parquet] (alamb)
Add roundtrip tests for Decimal256 and fix issues (#4264) #4311 [parquet] (tustvold)
Expose page-level arrow reader API (#4298) #4307 [parquet] (tustvold)
Speed up i256 division and remainder operations #4303 [arrow] (viirya)
feat(flight): support int32_to_int32_list_map in sql infos #4300 [arrow] [arrow-flight] (roeap)
feat(flight): add helpers to handle CommandGetCatalogs, CommandGetSchemas, and CommandGetTables requests #4296 [arrow] [arrow-flight] (roeap)
Improve docs and tests for `SqlInfoList #4293 [arrow] [arrow-flight] (alamb)
minor: fix arrow_row docs.rs links #4292 [arrow] (roeap)
Update proc-macro2 requirement from =1.0.58 to =1.0.59 #4290 [arrow] [arrow-flight] (dependabot[bot])
Improve ArrowWriter memory usage: Buffer Pages in ArrowWriter instead of RecordBatch (#3871) #4280 [parquet] (tustvold)
Minor: Add more docstrings in arrow-flight #4279 [arrow] [arrow-flight] (alamb)
Add Debug impls for ArrowWriter and SerializedFileWriter #4278 [parquet] (alamb)
Expose RecordBatchWriter to arrow crate #4277 [arrow] (alexandreyc)
Update criterion requirement from 0.4 to 0.5 #4275 [parquet] [arrow] (dependabot[bot])
Add parquet-concat #4274 [parquet] (tustvold)
Convert FixedSizeListArray to GenericListArray #4273 [arrow] (tustvold)
feat: support 'Decimal256' for parquet #4272 [parquet] (Weijun-H)
Strip leading whitespace from flight_sql_client custom header values #4271 [arrow] [arrow-flight] (mkmik)
Add Append Column API (#4155) #4269 [parquet] (tustvold)
Derive Default for WriterProperties #4268 [parquet] (tustvold)
Parquet Reader/writer for fixed-size list arrays #4267 [parquet] (dexterduck)
feat(flight): add sql-info helpers #4266 [arrow] [arrow-flight] (roeap)
Convert parquet metadata back to builders #4265 [parquet] (tustvold)
Add constructors for FixedSize array types (#3879) #4263 [arrow] (tustvold)
Extract IPC ArrayReader struct #4259 [arrow] (tustvold)
Update object_store requirement from 0.5 to 0.6 #4258 [parquet] (dependabot[bot])
Support Absolute Timestamps in CSV Schema Inference (#4131) #4217 [arrow] (tustvold)
feat: cast between Intervals #4182 [arrow] (izveigor)

Open source →

39.0.09 May 2023Deprecated

Remove deprecated parquet ArrowReader \#4125 [parquet] (tustvold)

Full Changelog

Breaking changes:

Allow creating unbuffered streamreader #4165 [arrow] (ming08108)
Cleanup ChunkReader (#4118) #4156 [parquet] (tustvold)
Remove Type from NativeIndex #4146 [parquet] (tustvold)
Don't Duplicate Offset Index on RowGroupMetadata #4142 [parquet] (tustvold)
Return BooleanBuffer from BooleanBufferBuilder #4140 [parquet] [arrow] (tustvold)
Cleanup CSV schema inference (#4129) (#4130) #4133 [parquet] [arrow] (tustvold)
Remove deprecated parquet ArrowReader #4125 [parquet] (tustvold)
refactor: construct StructArray w/ FieldRef #4116 [parquet] [arrow] (crepererum)
Ignore Field Metadata in equals_datatype for Dictionary, RunEndEncoded, Map and Union #4111 [arrow] (izveigor)
Add StructArray Constructors (#3879) #4064 [arrow] (tustvold)

Implemented enhancements:

Release 39.0.0 of arrow/arrow-flight/parquet/parquet-derive (next release after 38.0.0) #4170 [arrow] [arrow-flight]
Fixed point decimal multiplication for DictionaryArray #4135 [arrow]
Remove Seek Requirement from CSV ReaderBuilder #4130 [parquet] [arrow]
Inconsistent CSV Inference and Parsing DateTime Handling #4129 [parquet] [arrow]
Support accessing ipc Reader/Writer inner by reference #4121
Add Type Declarations for All Primitive Tensors and Buffer Builders #4112 [arrow]
Support Interval + Timestamp and Interval + Date in addition to Timestamp + Interval and Interval + Date #4094 [arrow]
Enable setting FlightDescriptor on FlightDataEncoderBuilder #3855 [arrow] [arrow-flight]

Fixed bugs:

Parquet Page Index Reader Assumes Consecutive Offsets #4149 [parquet]
Equality of nested data types #4110 [arrow]

Documentation updates:

Improve Documentation of Parquet ChunkReader #4118

Closed issues:

add specific error log for empty JSON array #4105 [arrow]

Merged pull requests:

Prep for 39.0.0 #4171 [arrow] [arrow-flight] (iajoiner)
Support Compression in parquet-fromcsv #4160 [parquet] (suxiaogang223)
feat: support bitwise shift left/right with scalars #4159 [arrow] (izveigor)
Cleanup reading page index (#4149) (#4090) #4151 [parquet] (tustvold)
feat: support bitwise shift left/right #4148 [arrow] (Weijun-H)
Don't hardcode port in FlightSQL tests #4145 [arrow] [arrow-flight] (tustvold)
Better flight SQL example codes #4144 [arrow] [arrow-flight] (sundy-li)
chore: clean the code by using as_primitive #4143 [arrow] (Weijun-H)
docs: fix the wrong ln command in CONTRIBUTING.md #4139 (SteveLauC)
Infer Float64 for JSON Numerics Beyond Bounds of i64 #4138 [arrow] (SteveLauC)
Support fixed point multiplication for DictionaryArray of Decimals #4136 [arrow] (viirya)
Make arrow_json::ReaderBuilder method names consistent #4128 [arrow] (tustvold)
feat: add get_{ref, mut} to arrow_ipc Reader and Writer #4122 (sticnarf)
feat: support Interval + Timestamp and Interval + Date #4117 [arrow] (Weijun-H)
Support NullArray in JSON Reader #4114 [arrow] (jiangzhx)
Add Type Declarations for All Primitive Tensors and Buffer Builders #4113 [arrow] (izveigor)
Update regex-syntax requirement from 0.6.27 to 0.7.1 #4107 [arrow] (dependabot[bot])
feat: set FlightDescriptor on FlightDataEncoderBuilder #4101 [arrow] [arrow-flight] (Weijun-H)
optimize cast for same decimal type and same scale #4088 [arrow] (liukun4515)

Open source →

38.0.025 Apr 2023

Remove DataType from PrimitiveArray constructors \#4098 [arrow] (tustvold)

Full Changelog

Breaking changes:

Remove DataType from PrimitiveArray constructors #4098 [arrow] (tustvold)
Use Into<Arc<str>> for PrimitiveArray::with_timezone #4097 [arrow] (tustvold)
Store StructArray entries in MapArray #4085 [parquet] [arrow] (tustvold)
Add DictionaryArray Constructors (#3879) #4068 [arrow] [arrow-flight] (tustvold)
Relax JSON schema inference generics #4063 [arrow] (tustvold)
Remove ArrayData from Array (#3880) #4061 [arrow] (tustvold)
Add CommandGetXdbcTypeInfo to Flight SQL Server #4055 [arrow] [arrow-flight] (c-thiel)
Remove old JSON Reader and Decoder (#3610) #4052 [parquet] [arrow] (tustvold)
Use BufRead for JSON Schema Inference #4041 [arrow] (WenyXu)

Implemented enhancements:

Support dyn_compare_scalar for Decimal256 #4083 [arrow]
Better JSON Reader Error Messages #4076 [arrow]
Additional data type groups #4056 [arrow]
Async JSON reader #4043 [arrow]
Field::contains Should Recurse into DataType #4029 [arrow]
Prevent UnionArray with Repeated Type IDs #3982 [parquet] [arrow]
Support Timestamp +/- Interval types #3963 [arrow]
First-Class Array Abstractions #3880 [parquet] [arrow] [arrow-flight]

Fixed bugs:

Update readme to remove reference to Jira #4091
OffsetBuffer::new Rejects 0 Offsets #4066 [arrow]
Parquet AsyncArrowWriter not shutting down inner async writer. #4058 [parquet]
Flight SQL Server missing command type.googleapis.com/arrow.flight.protocol.sql.CommandGetXdbcTypeInfo #4054 [arrow] [arrow-flight]
RawJsonReader Errors with Empty Schema #4053 [parquet] [arrow]
RawJsonReader Integer Truncation #4049 [arrow]
Sparse UnionArray Equality Incorrect Offset Handling #4044 [arrow]

Documentation updates:

Write blog about improvements in JSON and CSV processing #4062 [arrow]

Closed issues:

Parquet reader of Int96 columns and coercion to timestamps #4075
Serializing timestamp from int (json raw decoder) #4069 [arrow]
Support casting to/from Interval and Duration #3998 [arrow]

Merged pull requests:

Fix Docs Typos #4100 [parquet] (rnarkk)
Update tonic-build requirement from =0.9.1 to =0.9.2 #4099 [arrow] [arrow-flight] (dependabot[bot])
Increase minimum chrono version to 0.4.24 #4093 [arrow] (alamb)
Simplify reference to GitHub issues #4092 (bkmgit)
[Minor]: Add Hash trait to SortOptions. #4089 [arrow] (mustafasrepo)
Include byte offsets in parquet-layout #4086 [parquet] (tustvold)
feat: Support dyn_compare_scalar for Decimal256 #4084 [arrow] (izveigor)
Add ByteArray constructors (#3879) #4081 [arrow] (tustvold)
Update prost-build requirement from =0.11.8 to =0.11.9 #4080 [arrow] [arrow-flight] (dependabot[bot])
Improve JSON decoder errors (#4076) #4079 [arrow] (tustvold)
Fix Timestamp Numeric Truncation in JSON Reader #4074 [arrow] (tustvold)
Serialize numeric to tape (#4069) #4073 [arrow] (tustvold)
feat: Prevent UnionArray with Repeated Type IDs #4070 [arrow] (Weijun-H)
Add PrimitiveArray::try_new (#3879) #4067 [arrow] (tustvold)
Add ListArray Constructors (#3879) #4065 [arrow] (tustvold)
Shutdown parquet async writer #4059 [parquet] (kindly)
feat: additional data type groups #4057 [arrow] (izveigor)
Fix precision loss in Raw JSON decoder (#4049) #4051 [arrow] (tustvold)
Use lexical_core in CSV and JSON parser (~25% faster) #4050 [arrow] (tustvold)
Add offsets accessors to variable length arrays (#3879) #4048 [arrow] (tustvold)
Document Async decoder usage (#4043) (#78) #4046 [arrow] (tustvold)
Fix sparse union array equality (#4044) #4045 [arrow] (tustvold)
feat: DataType::contains support nested type #4042 [arrow] (Weijun-H)
feat: Support Timestamp +/- Interval types #4038 [arrow] (Weijun-H)
Fix object_store CI #4037 (tustvold)
feat: cast from/to interval and duration #4020 [arrow] (Weijun-H)

Open source →

37.0.010 Apr 2023Deprecated

Deprecate Array::data \(\#3880\) \#4019 [arrow] (tustvold)

Full Changelog

Breaking changes:

Fix timestamp handling in cast kernel (#1936) (#4033) #4034 [arrow] (tustvold)
Update tonic 0.9.1 #4011 [arrow] [arrow-flight] (tustvold)
Use FieldRef in DataType (#3955) #3983 [parquet] [arrow] (tustvold)
Store Timezone as Arc<str> #3976 [parquet] [arrow] (tustvold)
Panic instead of discarding nulls converting StructArray to RecordBatch - (#3951) #3953 [parquet] [arrow] (tustvold)
Fix(flight_sql): PreparedStatement has no token for auth. #3948 [arrow] [arrow-flight] (youngsofun)
Add Strongly Typed Array Slice (#3929) #3930 [parquet] [arrow] (tustvold)
Add Zero-Copy Conversion between Vec and MutableBuffer #3920 [arrow] (tustvold)

Implemented enhancements:

Support Decimals cast to Utf8/LargeUtf #3991 [arrow]
Support Date32/Date64 minus Interval #3962 [arrow]
Reduce Cloning of Field #3955 [parquet] [arrow] [arrow-flight]
Support Deserializing Serde DataTypes to Arrow #3949 [arrow]
Add multiply_fixed_point #3946 [arrow]
Strongly Typed Array Slicing #3929 [parquet] [arrow]
Make it easier to match FlightSQL messages #3874 [arrow] [arrow-flight]
Support Casting Between Binary / LargeBinary and FixedSizeBinary #3826 [arrow]

Fixed bugs:

Incorrect Overflow Casting String to Timestamp #4033
f16::ZERO and f16::ONE are mixed up #4016 [arrow]
Handle overflow precision when casting from integer to decimal #3995 [arrow]
PrimitiveDictionaryBuilder.finish should use actual value type #3971 [arrow]
RecordBatch From StructArray Silently Discards Nulls #3952 [parquet] [arrow]
I256 Checked Subtraction Overflows for i256::MINUS_ONE #3942 [arrow]
I256 Checked Multiply Overflows for i256::MIN #3941 [arrow]

Closed issues:

Remove non-existent js feature from README #4000 [arrow]
Support take on MapArray #3875 [arrow]

Merged pull requests:

Prep for 37.0.0 #4031 [arrow] [arrow-flight] (iajoiner)
Add RecordBatch::with_schema #4028 [arrow] (tustvold)
Only require compatible batch schema in ArrowWriter #4027 [parquet] (tustvold)
Add Fields::contains #4026 [arrow] (tustvold)
Minor: add methods "is_positive" and "signum" to i256 #4024 [arrow] (izveigor)
Deprecate Array::data (#3880) #4019 [arrow] (tustvold)
feat: add tests for ArrowNativeTypeOp #4018 [arrow] (izveigor)
fix: f16::ZERO and f16::ONE are mixed up #4017 [arrow] (izveigor)
Minor: Float16Tensor #4013 [arrow] (izveigor)
Add FlightSQL module docs and links to arrow-flight crates #4012 [arrow] [arrow-flight] (alamb)
Update proc-macro2 requirement from =1.0.54 to =1.0.56 #4008 [arrow] [arrow-flight] (dependabot[bot])
Cleanup Primitive take #4006 [arrow] (tustvold)
Deprecate combine_option_bitmap #4005 [arrow] (tustvold)
Minor: add tests for BooleanBuffer #4004 [arrow] (izveigor)
feat: support to read/write customized metadata in ipc files #4003 [arrow] (framlog)
Cleanup more uses of Array::data (#3880) #4002 [parquet] [arrow] (tustvold)
Remove js feature from README #4001 [arrow] (akazukin5151)
feat: add the implementation BitXor to BooleanBuffer #3997 [arrow] (izveigor)
Handle precision overflow when casting from integer to decimal #3996 [arrow] (viirya)
Support CAST from Decimal datatype to String #3994 [arrow] (comphead)
Add Field Constructors for Complex Fields #3992 [parquet] [arrow] [arrow-flight] (tustvold)
fix: remove unused type parameters. #3986 [arrow] (youngsofun)
Add UnionFields (#3955) #3981 [parquet] [arrow] (tustvold)
Cleanup Fields Serde #3980 [arrow] (tustvold)
Support Rust structures --> RecordBatch by adding Serde support to RawDecoder (#3949) #3979 [arrow] (tustvold)
Convert string_to_timestamp_nanos to doctest #3978 [arrow] (tustvold)
Fix documentation of string_to_timestamp_nanos #3977 [arrow] (byteink)
add Date32/Date64 support to subtract_dyn #3974 [arrow] (SinanGncgl)
PrimitiveDictionaryBuilder.finish should use actual value type #3972 [arrow] (viirya)
Update proc-macro2 requirement from =1.0.53 to =1.0.54 #3968 [arrow] [arrow-flight] (dependabot[bot])
Async writer tweaks #3967 [parquet] (tustvold)
Fix reading ipc files with unordered projections #3966 [arrow] (framlog)
Add Fields abstraction (#3955) #3965 [parquet] [arrow] [arrow-flight] (tustvold)
feat: cast between Binary/LargeBinary and FixedSizeBinary #3961 [arrow] (Weijun-H)
feat: support async writer (#1269) #3957 [parquet] (ShiKaiWi)
Add ListBuilder::append_value (#3949) #3954 [arrow] (tustvold)
Improve array builder documentation (#3949) #3951 [arrow] (tustvold)
Faster i256 parsing #3950 [arrow] (tustvold)
Add multiply_fixed_point #3945 [arrow] (viirya)
feat: enable metadata import/export through C data interface #3944 [arrow] (wjones127)
Fix checked i256 arithmetic (#3942) (#3941) #3943 [arrow] (tustvold)
Avoid memory copies in take_list #3940 [arrow] (tustvold)
Faster decimal parsing (30-60%) #3939 [arrow] (spebern)
Fix: FlightSqlClient panic when execute_update. #3938 [arrow] [arrow-flight] (youngsofun)
Cleanup row count handling in JSON writer #3934 [arrow] (tustvold)
Add typed buffers to UnionArray (#3880) #3933 [arrow] (tustvold)
feat: add take for MapArray #3925 [arrow] (wjones127)
Deprecate Array::data_ref (#3880) #3923 [arrow] (tustvold)
Zero-copy conversion from Vec to PrimitiveArray #3917 [arrow] (tustvold)
feat: Add Commands enum to decode prost messages to strong type #3887 [arrow] [arrow-flight] (stuartcarnie)

Open source →

36.0.028 Mar 2023

Use dyn Array in sort kernels \#3931 [arrow] (tustvold)

Full Changelog

Breaking changes:

Use dyn Array in sort kernels #3931 [arrow] (tustvold)
Enforce struct nullability in JSON raw reader (#3900) (#3904) #3906 [arrow] (tustvold)
Return ScalarBuffer from PrimitiveArray::values (#3879) #3896 [arrow] (tustvold)
Use BooleanBuffer in BooleanArray (#3879) #3895 [arrow] (tustvold)
Seal ArrowPrimitiveType #3882 [arrow] (tustvold)
Support compression levels #3847 [parquet] (spebern)

Implemented enhancements:

Improve speed of parsing string to Times #3919 [arrow]
feat: add comparison/sort support for Float16 #3914
Pinned version in arrow-flight's build-dependencies are causing conflicts #3876
Add compression options (levels) #3844 [parquet] [arrow]
Use Unsigned Integer for Fixed Size DataType #3815
Common trait for RecordBatch and StructArray #3764 [arrow]
Allow precision loss on multiplying decimal arrays #3689 [arrow]

Fixed bugs:

Raw JSON Reader Allows Non-Nullable Struct Children to Contain Nulls #3904
Nullable field with nested not nullable map in json #3900
parquet_derive doesn't support Vec<u8> #3864 [parquet]
[REGRESSION] Parsing timestamps with lower case time separator #3863 [arrow]
[REGRESSION] Parsing timestamps with leap seconds #3861 [arrow]
[REGRESSION] Parsing timestamps with fractional seconds / microseconds / milliseconds / nanoseconds #3859 [arrow]
CSV Reader Doesn't set Timezone #3841
PyArrowConvert Leaks Memory #3683 [arrow]

Merged pull requests:

Derive RunArray Clone #3932 [arrow] (tustvold)
Move protoc generation to binary crate, unpin prost/tonic build (#3876) #3927 [arrow] [arrow-flight] (tustvold)
Fix JSON Temporal Encoding of Multiple Batches #3924 [arrow] (doki23)
Cleanup uses of Array::data_ref (#3880) #3918 [parquet] [arrow] (tustvold)
Support microsecond and nanosecond in interval parsing #3916 [arrow] (alamb)
feat: add comparison/sort support for Float16 #3915 [arrow] (izveigor)
Add AsArray trait for more ergonomic downcasting #3912 [parquet] [arrow] (tustvold)
Add OffsetBuffer::new #3910 [arrow] (tustvold)
Add PrimitiveArray::new (#3879) #3909 [arrow] (tustvold)
Support timezones in CSV reader (#3841) #3908 [arrow] (tustvold)
Improve ScalarBuffer debug output #3907 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.52 to =1.0.53 #3905 [arrow] [arrow-flight] (dependabot[bot])
Re-export parquet compression level structs #3903 [parquet] (tustvold)
Fix parsing timestamps of exactly 32 characters #3902 [arrow] (tustvold)
Add iterators to BooleanBuffer and NullBuffer #3901 [arrow] (tustvold)
Array equality for &dyn Array (#3880) #3899 [arrow] (tustvold)
Add BooleanArray::new (#3879) #3898 [arrow] (tustvold)
Revert structured ArrayData (#3877) #3894 (tustvold)
Fix pyarrow memory leak (#3683) #3893 [arrow] (tustvold)
Minor: add examples for ListBuilder and GenericListBuilder #3891 [arrow] (alamb)
Update syn requirement from 1.0 to 2.0 #3890 (dependabot[bot])
Use of mul_checked to avoid silent overflow in interval arithmetic #3886 [arrow] (Weijun-H)
Flesh out NullBuffer abstraction (#3880) #3885 [parquet] [arrow] (tustvold)
Implement Bit Operations for i256 #3884 [arrow] (tustvold)
Flatten arrow_buffer #3883 [arrow] (tustvold)
Add Array::to_data and Array::nulls (#3880) #3881 [arrow] (tustvold)
Added support for byte vectors and slices to parquet_derive (#3864) #3878 [parquet] (waymost)
chore: remove LevelDecoder #3872 [parquet] (Weijun-H)
Parse timestamps with leap seconds (#3861) #3862 [arrow] (tustvold)
Faster time parsing (~93% faster) #3860 [arrow] (tustvold)
Parse timestamps with arbitrary seconds fraction #3858 [arrow] (tustvold)
Add BitIterator #3856 [arrow] (tustvold)
Improve decimal parsing performance #3854 [arrow] (spebern)
Update proc-macro2 requirement from =1.0.51 to =1.0.52 #3853 [arrow] [arrow-flight] (dependabot[bot])
Update bitflags requirement from 1.2.1 to 2.0.0 #3852 [arrow] (dependabot[bot])
Add offset pushdown to parquet #3848 [parquet] (tustvold)
Add timezone support to JSON reader #3845 [arrow] (tustvold)
Allow precision loss on multiplying decimal arrays #3690 [arrow] (viirya)

Open source →

35.0.013 Mar 2023

Add RunEndBuffer \(\#1799\) \#3817 [arrow] (tustvold)

Full Changelog

Breaking changes:

Add RunEndBuffer (#1799) #3817 [arrow] (tustvold)
Restrict DictionaryArray to ArrowDictionaryKeyType #3813 [arrow] (tustvold)
refactor: assorted FlightSqlServiceClient improvements #3788 [arrow] [arrow-flight] (crepererum)
minor: make Parquet CLI input args consistent #3786 [parquet] (XinyuZeng)
Return Buffers from ArrayData::buffers instead of slice (#1799) #3783 [arrow] (tustvold)
Use NullBuffer in ArrayData (#3775) #3778 [parquet] [arrow] (tustvold)

Implemented enhancements:

Support timestamp/time and date types in json decoder #3834 [arrow]
Support decoding decimals in new raw json decoder #3819 [arrow]
Timezone Aware Timestamp Parsing #3794 [arrow]
Preallocate buffers for FixedSizeBinary array creation #3792 [arrow]
Make Parquet CLI args consistent #3785 [parquet]
Creates PrimitiveDictionaryBuilder from provided keys and values builders #3776 [arrow]
Use NullBuffer in ArrayData #3775 [parquet] [arrow]
Support unary_dict_mut in arth #3710 [arrow]
Support cast <> String to interval #3643 [arrow]
Support Zero-Copy Conversion from Vec to/from MutableBuffer #3516 [arrow]

Fixed bugs:

Timestamp Unit Casts are Unchecked #3833 [arrow]
regexp_match skips first match when returning match #3803 [arrow]
Cast to timestamp with time zone returns timestamp #3800 [arrow]
Schema-level metadata is not encoded in Flight responses #3779 [arrow] [arrow-flight]

Closed issues:

FlightSQL CLI client: simple test #3814 [arrow] [arrow-flight]

Merged pull requests:

refactor: timestamp overflow check #3840 [arrow] (Weijun-H)
Prep for 35.0.0 #3836 [parquet] [arrow] [arrow-flight] (iajoiner)
Support timestamp/time and date json decoding #3835 [arrow] (spebern)
Make dictionary preservation optional in row encoding #3831 [arrow] (tustvold)
Move prettyprint to arrow-cast #3828 [arrow] [arrow-flight] (tustvold)
Support decoding decimals in raw decoder #3820 [arrow] (spebern)
Add ArrayDataLayout, port validation (#1799) #3818 [arrow] (tustvold)
test: add test for FlightSQL CLI client #3816 [arrow] [arrow-flight] (crepererum)
Add regexp_match docs #3812 [arrow] (tustvold)
fix: Ensure Flight schema includes parent metadata #3811 [arrow] [arrow-flight] (stuartcarnie)
fix: regexp_match skips first match #3807 [arrow] (Weijun-H)
fix: change uft8 to timestamp with timezone #3806 [arrow] (Weijun-H)
Support reading decimal arrays from json #3805 [arrow] (spebern)
Add unary_dict_mut #3804 [arrow] (viirya)
Faster timestamp parsing (~70-90% faster) #3801 [arrow] (tustvold)
Add concat_elements_bytes #3798 [arrow] (tustvold)
Timezone aware timestamp parsing (#3794) #3795 [arrow] (tustvold)
Preallocate buffers for FixedSizeBinary array creation #3793 [arrow] (maxburke)
feat: simple flight sql CLI client #3789 [arrow] [arrow-flight] (crepererum)
Creates PrimitiveDictionaryBuilder from provided keys and values builders #3777 [arrow] (viirya)
ArrayData Enumeration for Remaining Layouts #3769 [arrow] (tustvold)
Update prost-build requirement from =0.11.7 to =0.11.8 #3767 [arrow] [arrow-flight] (dependabot[bot])
Implement concat_elements_dyn kernel #3763 [arrow] (Weijun-H)
Support for casting Utf8 and LargeUtf8 --> Interval #3762 [arrow] (doki23)
into_inner() for CSV Writer #3759 [arrow] (Weijun-H)
Zero-copy Vec conversion (#3516) (#1176) #3756 [arrow] (tustvold)
ArrayData Enumeration for Primitive, Binary and UTF8 #3749 [arrow] (tustvold)
Add into_primitive_dict_builder to DictionaryArray #3715 [arrow] (viirya)

Open source →

33.0.013 Feb 2023

Use ArrayFormatter in Cast Kernel \#3668 [arrow] (tustvold)

Full Changelog

Breaking changes:

Use ArrayFormatter in Cast Kernel #3668 [arrow] (tustvold)
Use dyn Array in cast kernels #3667 [arrow] (tustvold)
Return references from FixedSizeListArray and MapArray #3652 [parquet] [arrow] (tustvold)
Lazy array display (#3638) #3647 [parquet] [arrow] (tustvold)
Use array_value_to_string in arrow-csv #3514 [arrow] (JayjeetAtGithub)

Implemented enhancements:

Support UTF8 cast to Timestamp with timezone #3664
Add modulus_dyn and modulus_scalar_dyn #3648 [arrow]
A trait for append_value and append_null on ArrayBuilders #3644
Improve error message "batches[0] schema is different with argument schema" #3628 [arrow]
Specified version of helper function to cast binary to string #3623 [arrow]
Casting generic binary to generic string #3606 [arrow]
Use array_value_to_string in arrow-csv #3483 [arrow]

Fixed bugs:

ArrowArray::try_from_raw Misleading Signature #3684 [arrow]
PyArrowConvert Leaks Memory #3683 [arrow]
Arrow-csv reader cannot produce RecordBatch even if the bytes are necessary #3674
FFI Fails to Account For Offsets #3671 [arrow]
Regression in CSV reader error handling #3656 [arrow]
UnionArray Child and Value Fail to Account for non-contiguous Type IDs #3653 [arrow]
Panic when accessing RecordBatch from pyarrow #3646 [arrow]
Multiplication for decimals is incorrect #3645
Inconsistent output between pretty print and CSV writer for Arrow #3513 [arrow]

Closed issues:

Release 33.0.0 of arrow/arrow-flight/parquet/parquet-derive (next release after 32.0.0) #3682
Release 32.0.0 of arrow/arrow-flight/parquet/parquet-derive (next release after 31.0.0) #3584 [parquet] [arrow] [arrow-flight]

Merged pull requests:

Move FFI to sub-crates #3687 [arrow] (tustvold)
Update to 33.0.0 and update changelog #3686 [parquet] [arrow] [arrow-flight] (iajoiner)
Cleanup FFI interface (#3684) (#3683) #3685 [arrow] (tustvold)
fix: take_run benchmark parameter #3679 [arrow] (askoa)
Minor: Add some examples to Date*Array and Time*Array #3678 [arrow] (alamb)
Add CSV Decoder::capacity (#3674) #3677 [arrow] (tustvold)
Add ArrayData::new_null and DataType::primitive_width #3676 [arrow] (tustvold)
Fix FFI which fails to account for offsets #3675 [arrow] (viirya)
Support UTF8 cast to Timestamp with timezone #3673 [arrow] (comphead)
Fix Date64Array docs #3670 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.50 to =1.0.51 #3669 [arrow] [arrow-flight] (dependabot[bot])
Add timezone accessor for Timestamp*Array #3666 [arrow] (tustvold)
Faster timezone cast #3665 [arrow] (tustvold)
feat + fix: IPC support for run encoded array. #3662 [arrow] (askoa)
Implement std::fmt::Write for StringBuilder (#3638) #3659 [arrow] (tustvold)
Include line and field number in CSV UTF-8 error (#3656) #3657 [arrow] (tustvold)
Handle non-contiguous type_ids in UnionArray (#3653) #3654 [arrow] (tustvold)
Add modulus_dyn and modulus_scalar_dyn #3649 [arrow] (viirya)
Improve error message with detailed schema #3637 [arrow] (Veeupup)
Add limit to ArrowReaderBuilder to push limit down to parquet reader #3633 [parquet] (thinkharderdev)
chore: delete wrong comment and refactor set_metadata in Field #3630 [arrow] (chunshao90)
Fix typo in comment #3627 [parquet] (kjschiroo)
Minor: Update doc strings about Page Index / Column Index #3625 [parquet] (alamb)
Specified version of helper function to cast binary to string #3624 [arrow] (viirya)
feat: take kernel for RunArray #3622 [arrow] (askoa)
Remove BitSliceIterator specialization from try_for_each_valid_idx #3621 [arrow] (tustvold)
Reduce PrimitiveArray::try_unary codegen #3619 [arrow] (tustvold)
Reduce Dictionary Builder Codegen #3616 [arrow] (tustvold)
Minor: Add test for dictionary encoding of batches #3608 [arrow-flight] (alamb)
Casting generic binary to generic string #3607 [arrow] (viirya)
Add ArrayAccessor, Iterator, Extend and benchmarks for RunArray #3603 [arrow] (askoa)

Open source →

32.0.030 Jan 2023Deprecated

Write Deprecated Min Max Statistics When ColumnOrder Signed \#3526 [parquet]

Full Changelog

Breaking changes:

Allow StringArray construction with Vec<Option<String>> #3602 [arrow] (sinistersnare)
Use native types in PageIndex (#3575) #3578 [parquet] (tustvold)
Add external variant to ParquetError (#3285) #3574 [parquet] (tustvold)
Return reference from ListArray::values #3561 [arrow] (tustvold)
feat: Add RunEndEncodedArray #3553 [parquet] [arrow] (askoa)

Implemented enhancements:

There should be a From<Vec<Option<String>>> impl for GenericStringArray<OffsetSize> #3599 [arrow]
FlightDataEncoder Optionally send Schema even when no record batches #3591 [arrow-flight]
Use Native Types in PageIndex #3575 [parquet]
Packing array into dictionary of generic byte array #3571 [arrow]
Implement Error::Source for ArrowError and FlightError #3566 [arrow] [arrow-flight]
[FlightSQL] Allow access to underlying FlightClient #3551 [arrow-flight]
Arrow CSV writer should not fail when cannot cast the value #3547 [arrow]
Write Deprecated Min Max Statistics When ColumnOrder Signed #3526 [parquet]
Improve Performance of JSON Reader #3441
Support footer kv metadata for IPC file #3432
Add External variant to ParquetError #3285 [parquet]

Fixed bugs:

Nullif of NULL Predicate is not NULL #3589
BooleanBufferBuilder Fails to Clear Set Bits On Truncate #3587 [arrow]
nullif incorrectly calculates null_count, sometimes panics with subtraction overflow error #3579 [arrow]
Meet warning when use pyarrow #3543 [arrow]
Incorrect row group total_byte_size written to parquet file #3530 [parquet]
Overflow when casting timestamps prior to the epoch #3512 [arrow]

Closed issues:

Panic on Key Overflow in Dictionary Builders #3562 [parquet] [arrow]
Bumping version gives compilation error (arrow-array) #3525

Merged pull requests:

Add Push-Based CSV Decoder #3604 [arrow] (tustvold)
Update to flatbuffers 23.1.21 #3597 [arrow] (tustvold)
Faster BooleanBufferBuilder::append_n for true values #3596 [arrow] (tustvold)
Support sending schemas for empty streams #3594 [arrow-flight] (alamb)
Faster ListArray to StringArray conversion #3593 [arrow] (tustvold)
Add conversion from StringArray to BinaryArray #3592 [arrow] (tustvold)
Fix nullif null count (#3579) #3590 [arrow] (tustvold)
Clear bits in BooleanBufferBuilder (#3587) #3588 [arrow] (tustvold)
Iterate all dictionary key types in cast test #3585 [arrow] (viirya)
Propagate EOF Error from AsyncRead #3576 [parquet] (Sach1nAgarwal)
Show row_counts also for (FixedLen)ByteArray #3573 [parquet] (bmmeijers)
Packing array into dictionary of generic byte array #3572 [arrow] (viirya)
Remove unwrap on datetime cast for CSV writer #3570 [arrow] (comphead)
Implement std::error::Error::source for ArrowError and FlightError #3567 [arrow] [arrow-flight] (alamb)
Improve GenericBytesBuilder offset overflow panic message (#139) #3564 [arrow] (tustvold)
Implement Extend for ArrayBuilder (#1841) #3563 [arrow] (tustvold)
Update pyarrow method call with kwargs #3560 [arrow] (Frankonly)
Update pyo3 requirement from 0.17 to 0.18 #3557 [arrow] (viirya)
Expose Inner FlightServiceClient on FlightSqlServiceClient (#3551) #3556 [arrow-flight] (tustvold)
Fix final page row count in parquet-index binary #3554 [parquet] (tustvold)
Parquet Avoid Reading 8 Byte Footer Twice from AsyncRead #3550 [parquet] (Sach1nAgarwal)
Improve concat kernel capacity estimation #3546 [arrow] (tustvold)
Update proc-macro2 requirement from =1.0.49 to =1.0.50 #3545 [arrow-flight] (dependabot[bot])
Update pyarrow method call to avoid warning #3544 [arrow] (Frankonly)
Enable casting between Utf8/LargeUtf8 and Binary/LargeBinary #3542 [arrow] (viirya)
Use GHA concurrency groups (#3495) #3538 (tustvold)
set sum of uncompressed column size as row group size for parquet files #3531 [parquet] (sidred)
Minor: Add documentation about memory use for ArrayData #3529 [arrow] (alamb)
Upgrade to clap 4.1 + fix test #3528 [parquet] (tustvold)
Write backwards compatible row group statistics (#3526) #3527 [parquet] (tustvold)
No panic on timestamp buffer overflow #3519 [arrow] (comphead)
Support casting from binary to dictionary of binary #3482 [arrow] (viirya)
Add Raw JSON Reader (~2.5x faster) #3479 [arrow] (tustvold)

Open source →

31.0.016 Jan 2023

support RFC3339 style timestamps in arrow-json \#3449 [arrow] (JayjeetAtGithub)

Full Changelog

Breaking changes:

support RFC3339 style timestamps in arrow-json #3449 [arrow] (JayjeetAtGithub)
Improve arrow flight batch splitting and naming #3444 [arrow-flight] (alamb)
Parquet record API: timestamp as signed integer #3437 [parquet] (ByteBaker)
Support decimal int32/64 for writer #3431 [parquet] (liukun4515)

Implemented enhancements:

Support casting Date32 to timestamp #3504 [arrow]
Support casting strings like '2001-01-01' to timestamp #3492 [arrow]
CLI to "rewrite" parquet files #3476 [parquet]
Add more dictionary value type support to build_compare #3465
Allow concat_batches to take non owned RecordBatch #3456 [arrow]
Release Arrow 30.0.1 (maintenance release for 30.0.0) #3455
Add string comparisons (starts_with, ends_with, and contains) to kernel #3442 [arrow]
make_builder Loses Timezone and Decimal Scale Information #3435 [arrow]
Use RFC3339 style timestamps in arrow-json #3416 [arrow]
ArrayDataget_slice_memory_size or similar #3407 [arrow] [arrow-flight]

Fixed bugs:

Unable to read CSV with null boolean value #3521 [arrow]
Make consistent behavior on zeros equality on floating point types #3509
Sliced batch w/ bool column doesn't roundtrip through IPC #3496 [arrow] [arrow-flight]
take kernel on List array introduces nulls instead of empty lists #3471 [arrow]
Infinite Loop If Skipping More CSV Lines than Present #3469 [arrow]

Merged pull requests:

Fix reading null booleans from CSV #3523 [arrow] (tustvold)
minor fix: use the unified decimal type builder #3522 [parquet] (liukun4515)
Update version to 31.0.0 and add changelog #3518 [parquet] [arrow] [arrow-flight] (iajoiner)
Additional nullif re-export #3515 [arrow] (tustvold)
Make consistent behavior on zeros equality on floating point types #3510 (viirya)
Enable cast Date32 to Timestamp #3508 [arrow] (comphead)
Update prost-build requirement from =0.11.5 to =0.11.6 #3507 [arrow-flight] (dependabot[bot])
minor fix for the comments #3505 [arrow] (liukun4515)
Fix DataTypeLayout for LargeList #3503 [arrow] (viirya)
Add string comparisons (starts_with, ends_with, and contains) to kernel #3502 [arrow] (snmvaughan)
Add a function to get memory size of array slice #3501 [arrow] (askoa)
Fix IPCWriter for Sliced BooleanArray #3498 [arrow] (crepererum)
Fix: Added support to cast string without time #3494 [arrow] (gaelwjl)
Fix negative interval prettyprint #3491 [arrow] (Jefffrey)
Fixes a broken link in the arrow lib.rs rustdoc #3487 [arrow] (AdamGS)
Refactoring build_compare for decimal and using downcast_primitive #3484 (viirya)
Add tests for record batch size splitting logic in FlightClient #3481 [arrow-flight] (alamb)
change concat_batches parameter to non owned reference #3480 [arrow] (askoa)
feat: add parquet-rewrite CLI #3477 [parquet] (crepererum)
Preserve empty list array elements in take kernel #3473 [arrow] (jonmmease)
Add a test for stream writer for writing sliced array #3472 [arrow] (viirya)
Fix CSV infinite loop and improve error messages #3470 [arrow] (tustvold)
Add more dictionary value type support to build_compare #3466 (viirya)
Add tests for FlightClient::{list_flights, list_actions, do_action, get_schema} #3463 [arrow-flight] (alamb)
Minor: add ticket links to failing ipc integration tests #3461 (alamb)
feat: column_name based index access for RecordBatch and StructArray #3458 [arrow] (askoa)
Support Decimal256 in FFI #3453 [arrow] (viirya)
Remove multiversion dependency #3452 [arrow] (tustvold)
Re-export nullif kernel #3451 [arrow] (tustvold)
Meaningful error message for map builder with null keys #3450 [arrow] (Jefffrey)
Parquet writer v2: clear buffer after page flush #3447 [parquet] (askoa)
Verify ArrayData::data_type compatible in PrimitiveArray::from #3440 [arrow] (tustvold)
Preserve DataType metadata in make_builder #3438 [arrow] (tustvold)
Consolidate arrow ipc tests and increase coverage #3427 [arrow] (alamb)
Generic bytes dictionary builder #3426 [arrow] (viirya)
Minor: Improve docs for arrow-ipc, remove clippy ignore #3421 [arrow] (alamb)
refactor: convert *like_dyn, *like_utf8_scalar_dyn and *like_dict functions to macros #3411 [arrow] (askoa)
Add parquet-index binary #3405 [parquet] (tustvold)
Complete mid-level FlightClient #3402 [arrow-flight] (alamb)
Implement RecordBatch <--> FlightData encode/decode + tests #3391 [arrow] [arrow-flight] (alamb)
Provide into_builder for bytearray #3326 [arrow] (viirya)

Open source →

30.0.03 Jan 2023Deprecated

Document / Deprecate arrow\_flight::flight\_data\_from\_arrow\_batch \#3312 [arrow] [arrow-flight]

Full Changelog

Breaking changes:

Infer Parquet JSON Logical and Converted Type as UTF-8 #3376 [parquet] (tustvold)
Use custom Any instead of prost_types #3360 [arrow-flight] (tustvold)
Use bytes in arrow-flight #3359 [arrow-flight] (tustvold)

Implemented enhancements:

Add derived implementations of Clone and Debug for ParquetObjectReader #3381 [parquet]
Speed up TrackedWrite #3366 [parquet]
Is it possible for ArrowWriter to write key_value_metadata after write all records #3356 [parquet]
Add UnionArray test to arrow-pyarrow integration test #3346
Document / Deprecate arrow_flight::utils::flight_data_from_arrow_batch #3312 [arrow] [arrow-flight]
[FlightSQL] Support HTTPs #3309 [arrow-flight]
Support UnionArray in ffi #3304 [arrow]
Add support for Azure Data Lake Storage Gen2 (aka: ADLS Gen2) in Object Store library #3283
Support casting from String to Decimal #3280 [arrow]
Allow ArrowCSV writer to control the display of NULL values #3268 [arrow]

Fixed bugs:

FlightSQL example is broken #3386 [arrow-flight]
CSV Reader Bounds Incorrectly Handles Header #3364 [arrow]
Incorrect output string from try_to_type #3350
Decimal arithmetic computation fails to run because decimal type equality #3344 [arrow]
Pretty print not implemented for Map #3322 [arrow]
ILIKE Kernels Inconsistent Case Folding #3311 [arrow]

Documentation updates:

minor: Improve arrow-flight docs #3372 [arrow] [arrow-flight] (alamb)

Merged pull requests:

Version 30.0.0 release notes and changelog #3406 [parquet] [arrow] [arrow-flight] (alamb)
Ends ParquetRecordBatchStream when polling on StreamState::Error #3404 [parquet] (viirya)
fix clippy issues #3398 (Jimexist)
Upgrade multiversion to 0.7.1 #3396 (viirya)
Make FlightSQL Support HTTPs #3388 [arrow-flight] (viirya)
Fix broken FlightSQL example #3387 [arrow-flight] (viirya)
Update prost-build #3385 [arrow-flight] (tustvold)
Split out arrow-arith (#2594) #3384 [arrow] (tustvold)
Add derive for Clone and Debug for ParquetObjectReader #3382 [parquet] (kszlim)
Initial Mid-level FlightClient #3378 [arrow-flight] (alamb)
Document all features on docs.rs #3377 [arrow] [arrow-flight] (tustvold)
Split out arrow-row (#2594) #3375 [arrow] (tustvold)
Remove unnecessary flush calls on TrackedWrite #3374 [parquet] (viirya)
Update proc-macro2 requirement from =1.0.47 to =1.0.49 #3369 [arrow-flight] (dependabot[bot])
Add CSV build_buffered (#3338) #3368 [arrow] (tustvold)
feat: add append_key_value_metadata #3367 [parquet] (jiacai2050)
Add csv-core based reader (#3338) #3365 [arrow] (tustvold)
Put BufWriter into TrackedWrite #3361 [parquet] (viirya)
Add CSV reader benchmark (#3338) #3357 [arrow] (tustvold)
Use ArrayData::ptr_eq in DictionaryTracker #3354 [arrow] (tustvold)
Deprecate flight_data_from_arrow_batch #3353 [arrow] [arrow-flight] (Dandandan)
Fix incorrect output string from try_to_type #3351 (viirya)
Fix unary_dyn for decimal scalar arithmetic computation #3345 [arrow] (viirya)
Add UnionArray test to arrow-pyarrow integration test #3343 (viirya)
feat: configure null value in arrow csv writer #3342 [arrow] (askoa)
Optimize bulk writing of all blocks of bloom filter #3340 [parquet] (viirya)
Add MapArray to pretty print #3339 [arrow] (askoa)
Update prost-build 0.11.4 #3334 [arrow-flight] (tustvold)
Faster Parquet Bloom Writer #3333 (tustvold)
Add bloom filter benchmark for parquet writer #3323 [parquet] (viirya)
Add ASCII fast path for ILIKE scalar (90% faster) #3306 [arrow] (tustvold)
Support UnionArray in ffi #3305 [arrow] (viirya)
Support casting from String to Decimal #3281 [arrow] (viirya)
add more integration test for parquet bloom filter round trip tests #3210 [parquet] (Jimexist)

Open source →

29.0.012 Dec 2022Deprecated

Deprecate limit kernel \#3200 [arrow] (tustvold)

Full Changelog

Breaking changes:

Minor: Allow Field::new and Field::new_with_dict to take existing String as well as &str #3288 [arrow] (alamb)
update &Option<T> to Option<&T> #3249 [parquet] [arrow] (Jimexist)
Hide *_dict_scalar kernels behind *_dyn kernels #3202 [arrow] (viirya)

Implemented enhancements:

Support writing BloomFilter in arrow_writer #3275 [parquet]
Support casting from unsigned numeric to Decimal256 #3272 [arrow]
Support casting from Decimal256 to float types #3266 [arrow]
Make arithmetic kernels supports DictionaryArray of DecimalType #3254 [arrow]
Casting from Decimal256 to unsigned numeric #3239 [arrow]
precision is not considered when cast value to decimal #3223 [arrow]
Use RegexSet in arrow_csv::infer_field_schema #3211 [arrow]
Implement FlightSQL Client #3206 [arrow-flight]
Add binary_mut and try_binary_mut #3143 [arrow]
Add try_unary_mut #3133 [arrow]

Fixed bugs:

Skip null buffer when importing FFI ArrowArray struct if no null buffer in the spec #3290 [arrow]
using ahash compile-time-rng kills reproducible builds #3271 [parquet]
Decimal128 to Decimal256 Overflows #3265 [arrow]
nullif panics on empty array #3261 [arrow]
Some more inconsistency between can_cast_types and cast_with_options #3250 [arrow]
Enable casting between Dictionary of DecimalArray and DecimalArray #3237 [arrow]
new_null_array Panics creating StructArray with non-nullable fields #3226 [arrow]
bool should cast from/to Float16Type as can_cast_types returns true #3221 [arrow]
Utf8 and LargeUtf8 cannot cast from/to Float16 but can_cast_types returns true #3220 [arrow]
Re-enable some tests in arrow-cast crate #3219 [arrow]
Off-by-one buffer size error triggers Panic when constructing RecordBatch from IPC bytes (should return an Error) #3215 [arrow]
arrow to and from pyarrow conversion results in changes in schema #3136 [arrow]

Documentation updates:

better document when we need LargeUtf8 instead of Utf8 #3228 [arrow]

Merged pull requests:

Use BufWriter when writing bloom filters and limit tests (#3318) #3319 [parquet] (tustvold)
Use take for dictionary like comparisons #3313 [arrow] (tustvold)
Update versions to 29.0.0 and update CHANGELOG #3315 [parquet] [arrow] [arrow-flight] (alamb)
refactor: Merge similar functions ilike_scalar and nilike_scalar #3303 [arrow] (askoa)
Split out arrow-ord (#2594) #3299 [arrow] (tustvold)
Split out arrow-string (#2594) #3295 [arrow] (tustvold)
Skip null buffer when importing FFI ArrowArray struct if no null buffer in the spec #3293 [arrow] (viirya)
Don't use dangling NonNull as sentinel #3289 [arrow] (tustvold)
Set bloom filter on byte array #3284 [parquet] (viirya)
Fix ipc schema custom_metadata serialization #3282 [arrow] (Jefffrey)
Disable const-random ahash feature on non-WASM (#3271) #3277 [parquet] (tustvold)
fix(ffi): handle null data buffers from empty arrays #3276 [arrow] (wjones127)
Support casting from unsigned numeric to Decimal256 #3273 [arrow] (viirya)
Add parquet-layout binary #3269 [parquet] (tustvold)
Support casting from Decimal256 to float types #3267 [arrow] (viirya)
Simplify decimal cast logic #3264 [arrow] (tustvold)
Fix panic on nullif empty array (#3261) #3263 [arrow] (tustvold)
Add BooleanArray::from_unary and BooleanArray::from_binary #3258 [arrow] (tustvold)
Minor: Remove parquet build script #3257 [parquet] (tustvold)
Make arithmetic kernels supports DictionaryArray of DecimalType #3255 [arrow] (viirya)
Support List and LargeList in Row format (#3159) #3251 [arrow] (tustvold)
Don't recurse to children in ArrayData::try_new #3248 [arrow] (tustvold)
Validate dictionaries read over IPC #3247 [arrow] (tustvold)
Fix MapBuilder example #3246 [arrow] (tustvold)
Loosen nullability restrictions added in #3205 (#3226) #3244 [arrow] (tustvold)
Better document implications of offsets (#3228) #3243 [arrow] (tustvold)
Add new API to validate the precision for decimal array #3242 [arrow] (liukun4515)
Move nullif to arrow-select (#2594) #3241 [arrow] (tustvold)
Casting from Decimal256 to unsigned numeric #3240 [arrow] (viirya)
Enable casting between Dictionary of DecimalArray and DecimalArray #3238 [arrow] (viirya)
Remove unwraps from 'create_primitive_array' #3232 [arrow] (aarashy)
Fix CI build by upgrading tonic-build to 0.8.4 #3231 [arrow-flight] (viirya)
Remove negative scale check #3230 [arrow] (viirya)
Update prost-build requirement from =0.11.2 to =0.11.3 #3225 [arrow-flight] (dependabot[bot])
Get the round result for decimal to a decimal with smaller scale #3224 [arrow] (liukun4515)
Move tests which require chrono-tz feature from arrow-cast to arrow #3222 [arrow] (viirya)
add test cases for extracting week with/without timezone #3218 [arrow] (waitingkuo)
Use RegexSet for matching DataType #3217 [arrow] (askoa)
Update tonic-build to 0.8.3 #3214 [arrow-flight] (tustvold)
Support StructArray in Row Format (#3159) #3212 [arrow] (tustvold)
Infer timestamps from CSV files #3209 [arrow] (Jefffrey)
fix bug: cast decimal256 to other decimal with no-safe #3208 [arrow] (liukun4515)
FlightSQL Client & integration test #3207 [arrow-flight] (avantgardnerio)
Ensure StructArrays check nullability of fields #3205 [arrow] (Jefffrey)
Remove special case ArrayData equality for decimals #3204 [arrow] (tustvold)
Add a cast test case for decimal negative scale #3203 [arrow] (viirya)
Move zip and shift kernels to arrow-select #3201 [arrow] (tustvold)
Deprecate limit kernel #3200 [arrow] (tustvold)
Use SlicesIterator for ArrayData Equality #3198 [arrow] (viirya)
Add _dyn kernels of like, ilike, nlike, nilike kernels for dictionary support #3197 [arrow] (viirya)
Adding scalar nlike_dyn, ilike_dyn, nilike_dyn kernels #3195 [arrow] (psvri)
Use self capture in DataType #3190 [arrow] (tustvold)
To pyarrow with schema #3188 [arrow] (doki23)
Support Duration in array_value_to_string #3183 [arrow] (psvri)
Support FixedSizeBinary in Row format #3182 [arrow] (tustvold)
Add binary_mut and try_binary_mut #3144 [arrow] (viirya)
Add try_unary_mut #3134 [arrow] (viirya)

Open source →