parquet2

Full Changelog

Fixed bugs:

• Fixed aggregation of max statistics for primitive fields #217 (tjwilson90)
• Fix reimport problem and clippy. #216 (RinChanNOWWW)
• Bump minimum async-stream version to 0.3.3 #212 (garrisonhess)
• Fixed error in rle decoding #207 (ritchie46)

Enhancements:

• Update dependencies + a little cleanup #211 (aldanor)
• Make some struct de/serializable. #209 (RinChanNOWWW)
• Removed unnecessary flushes while writing #206 (cyr)
• Enbaled setting selected_rows in the runtime. #205 (RinChanNOWWW)

Full Changelog

Fixed bugs:

• Fixed error in rle decoding #207 (ritchie46)

A new release is out there!

Thanks everyone for the fixes and improvements resulting in this stabler, easier to use and faster version of parquet2!

Breaking changes:

• Improved hybrid rle decoding performance ~-40% #203 (ritchie46)
• Improved API to read column chunks #195 (jorgecarleitao)
• Removed EncodedPage #191 (jorgecarleitao)

New features:

• Added serde support for RowGroupMetaData. #202 (youngsofun)

Fixed bugs:

• Removed un-necessary conversion #197 (jorgecarleitao)
• Avoid OOM on page streams #194 (jorgecarleitao)
• Fixed error requiring stats #193 (jorgecarleitao)

Enhancements:

• elide bound check in RLE decoder #201 (ritchie46)
• Replaced panics by errors on invalid pages #188 (evanrichter)

Full Changelog

Breaking changes:

• Improved hybrid rle decoding performance ~-40% #203 (ritchie46)
• Improved API to read column chunks #195 (jorgecarleitao)
• Removed EncodedPage #191 (jorgecarleitao)

New features:

• Added serde support for RowGroupMetaData. #202 (youngsofun)

Fixed bugs:

• Removed un-necessary conversion #197 (jorgecarleitao)
• Avoid OOM on page streams #194 (jorgecarleitao)
• Fixed error requiring stats #193 (jorgecarleitao)

Enhancements:

• elide bound check in RLE decoder #201 (ritchie46)
• Replaced panics by errors on invalid pages #188 (evanrichter)

Full Changelog

Fixed bugs:

• Fixed edge cases in writing delta encoder #189 (jorgecarleitao)

Documentation updates:

• Improved doc generation to include features #190 (jorgecarleitao)

Full Changelog

Fixed bugs:

• Fixed error in FilteredHybridBitmapIter's trait bounds #187 (jorgecarleitao)

Yet another release of parquet2, mostly focused on avoiding panics and oom. No impact on performance, but improves reliability.

v0.16.1 (2022-08-17)

Full Changelog

Fixed bugs:

• Fixed error in FilteredHybridBitmapIter's trait bounds #187 (jorgecarleitao)

v0.16.0 (2022-08-17)

Full Changelog

Breaking changes:

• Improved Error #181 (jorgecarleitao)
• Made decoding fallible #178 (jorgecarleitao)
• Improved bitpacking #176 (jorgecarleitao)

New features:

• Added DELTA_BYTE_ARRAY encoder #183 (jorgecarleitao)

Fixed bugs:

• FixedLenByteArray max_precision integer overflow #184 (evanrichter)

Documentation updates:

• enable doc_cfg feature #186 (ritchie46)
• Improved decoding documentation #180 (jorgecarleitao)

Full Changelog

Breaking changes:

• Improved Error #181 (jorgecarleitao)
• Made decoding fallible #178 (jorgecarleitao)
• Improved bitpacking #176 (jorgecarleitao)

New features:

• Added DELTA_BYTE_ARRAY encoder #183 (jorgecarleitao)

Fixed bugs:

• FixedLenByteArray max_precision integer overflow #184 (evanrichter)

Documentation updates:

• enable doc_cfg feature #186 (ritchie46)
• Improved decoding documentation #180 (jorgecarleitao)

Full Changelog

Fixed bugs:

• Fixed error in encoding large bitpacked deltas #179 (jorgecarleitao)

We have a new release of parquet2 available!

Breaking changes:

• Add max_size to get_page_stream and get_page_iterator #173
• Optional async #174 (jorgecarleitao)
• Privatized CompressionLevel #170 (jorgecarleitao)
• Delay deserialization of dictionary pages #160 (jorgecarleitao)

New features:

• Added feature flag to use zlib-ng backend for gzip #165 (ritchie46)

Fixed bugs:

• Fixed OOM on malicious/malformed thrift #172 (jorgecarleitao)

Enhancements:

• Made compute_page_row_intervals public #171 (jorgecarleitao)
• Simplified interal code #168 (jorgecarleitao)
• cargo fmt #166 (ritchie46)

Testing updates:

• Improved coverage report #175 (jorgecarleitao)

Full Changelog

Breaking changes:

• Add max_size to get_page_stream and get_page_iterator #173
• Optional async #174 (jorgecarleitao)
• Privatized CompressionLevel #170 (jorgecarleitao)
• Delay deserialization of dictionary pages #160 (jorgecarleitao)

New features:

• Added feature flag to use zlib-ng backend for gzip #165 (ritchie46)

Fixed bugs:

• Fixed OOM on malicious/malformed thrift #172 (jorgecarleitao)

Enhancements:

• Made compute_page_row_intervals public #171 (jorgecarleitao)
• Simplified interal code #168 (jorgecarleitao)
• cargo fmt #166 (ritchie46)

Testing updates:

• Improved coverage report #175 (jorgecarleitao)

A couple of bug fixes, by @jhorstmann and @v0y4g3r

Full Changelog

Fixed bugs:

• Fixed FileStreamer's end method to flush Parquet magic #163 (v0y4g3r)
• Fix compilation of parquet-tools #161 (jhorstmann)

Enhancements:

• Added Compressor::into_inner #158 (jorgecarleitao)

Full Changelog

Fixed bugs:

• Fixed FileStreamer's end method to flush Parquet magic #163 (v0y4g3r)
• Fix compilation of parquet-tools #161 (jhorstmann)

Enhancements:

• Added Compressor::into_inner #158 (jorgecarleitao)

Full Changelog

New features:

• Added support for legacy lz4 decompression #151 (dantengsky)

Enhancements:

• Improved performance of reading #157 (jorgecarleitao)

Full Changelog

Breaking changes:

• split_buffer should return Result #156

Fixed bugs:

• Removed panics on read #150 (jorgecarleitao)

Enhancements:

• Reduced reallocations #153 (jorgecarleitao)
• Removed AsyncSeek requirement from page stream #149 (medwards)

Full Changelog

Fixed bugs:

• Fixed missing re-export of FileMetaData to allow using side-car API #148 (jorgecarleitao)

Full Changelog

New features:

• Added support to write sidecar #147 (jorgecarleitao)

Full Changelog

Breaking changes:

• Removed unused cargo feature #145 (jorgecarleitao)
• Fix potential misuse of FileWriter API's (sync + async) #138 (TurnOfACard)

New features:

• Added new_with_page_meta to PageReader #136 (ygf11)
• Added compression options/levels for GZIP and BROTLI codecs. #132 (TurnOfACard)

Fixed bugs:

• Async FileStreamer does not write statistics #139
• Fixed error in compressing lz4raw with large offsets #140 (jorgecarleitao)

Enhancements:

• Improved read of metadata #143 (jorgecarleitao)
• Simplified async metadata read #137 (jorgecarleitao)

Testing updates:

• Lifted duplicated code to a function #141 (jorgecarleitao)
• Improved Integration test documentation and expanded tests #133 (TurnOfACard)

Full Changelog

Enhancements:

• Pass only necessary data when create PageReader #135

Full Changelog

Breaking changes:

• Add CompressionOptions, which allows for zstd compression levels. #128 (TurnOfACard)

Enhancements:

• Improved performance of RLE decoding (-18%) #130 (jorgecarleitao)
• Improved perf of bitpacking decoding (3.5x) #129 (jorgecarleitao)

Full Changelog

Breaking changes:

• Renamed ParquetError to Error #109
• Made .end not consume the parquet FileWriter #127 (jorgecarleitao)
• Removed compression from WriteOptions #125 (kornholi)
• Simplified API and converted some panics on read to errors #112 (jorgecarleitao)
• Improved typing to reduce clones and use of unwraps #106 (jorgecarleitao)
• Simplified PageIterator #103 (jorgecarleitao)

New features:

• Added support for page-level filter pushdown (indexes) #102
• Added support for bloom filters #98
• Added optional support for LZ4 via LZ4-flex crate (thus enabling wasm) #124 (jorgecarleitao)
• Added support for page-level filter pushdown (column and offset indexes) #107 (jorgecarleitao)
• Added support to read column and page indexes #100 (jorgecarleitao)

Fixed bugs:

• Fixed minimum version for LZ4 #122 (kornholi)
• Fixed Lz4Raw compression error (if input is tiny) #118 (dantengsky)
• Fixed LZ4 #95 (jorgecarleitao)

Enhancements:

• Made offsets be always written #123 (jorgecarleitao)
• Added specialized deserialization of one-level filtered pages #120 (jorgecarleitao)
• Added support to read and use bloom filters #99 (jorgecarleitao)
• Added ordinal and total_compressed_size to column meta #96 (jorgecarleitao)
• Added non-consuming function to get values of delta-decoder #94 (jorgecarleitao)
• Disabled bitpacking default-features and upgraded to edition 2021 #93 (light4)

Documentation updates:

• Fix deployment of guide #115 (jorgecarleitao)

Testing updates:

• Added tests for reducing statistics #116 (jorgecarleitao)
• Simplified tests #104 (jorgecarleitao)

Full Changelog

Fixed bugs:

• write ColumnMetaData instead of ColumnChunk after pages. #90 (youngsofun)

Full Changelog

Fixed bugs:

• Raise error when writing a page that is too large #84 (jorgecarleitao)

Testing updates:

• fix fmt and typo #83 (youngsofun)

Full Changelog

Enhancements:

• Update zstd dependency #82 (jhorstmann)
• Added file offset #81 (barrotsteindev)

Full Changelog

Breaking changes:

• Simplified API to write files #78 (jorgecarleitao)

Fixed bugs:

• Fixed panic in reading empty values in hybrid-RLE #80 (jorgecarleitao)

Full Changelog

Fixed bugs:

• Fixed panic in reading empty values in hybrid-RLE #80 (jorgecarleitao)

Full Changelog

Breaking changes:

• Changed stream of groups to stream of futures of groups #71 (jorgecarleitao)

Enhancements:

• bitpacking: use stack-allocated temporary buffer #76 (danburkert)
• Added constructor to RowGroupMetaData and ColumnChunkMetaData #74 (yjshen)
• Improved performance of reading multiple pages #73 (jorgecarleitao)

Fixed bugs:

• Fixed error in declaring size of compressed dict page. #72 (jorgecarleitao)

Full Changelog

Fixed bugs:

• Fixed error in declaring size of compressed dict page. #72 (jorgecarleitao)

Full Changelog

Breaking changes:

• Improved error message when a feature is not active #69 (jorgecarleitao)

Fixed bugs:

• Fixed error in finishing iterator. #68 (jorgecarleitao)

Full Changelog

Breaking changes:

• Use i64s for delta-bitpacked's interface #67 (kornholi)

New features:

• Added basic support to read nested types #64 (jorgecarleitao)

Fixed bugs:

• Fix off-by-one error in delta-bitpacked decoder #66 (kornholi)

Full Changelog

Breaking changes:

• Improved performance of codec initialization #63 (jorgecarleitao)
• Made PageFilter Send #62 (dantengsky)
• Alowed reusing compression buffer #60 (jorgecarleitao)

Fixed bugs:

• Fixed delta-bitpacked mini-block decoding #56 (kornholi)
• Add descriptor to FixedLenStatistics #54 (potter420)
• Fixed error in reading zero-width bit from hybrid RLE. #53 (jorgecarleitao)

Enhancements:

• Added writing reduced statistics for FixedLenByteArray #55 (potter420)

Full Changelog

Fixed bugs:

• Fixed delta-bitpacked mini-block decoding #56 (kornholi)
• Add descriptor to FixedLenStatistics #54 (potter420)

Enhancements:

• Added writing reduced statistics for FixedLenByteArray #55 (potter420)

Full Changelog

Fixed bugs:

• Fixed error in reading zero-width bit from hybrid RLE. #53 (jorgecarleitao)

Full Changelog

Breaking changes:

• Renamed Compression::Zsld to Compression::Zstd (typo) #48 (vincev)

Enhancements:

• Add null_count method to trait Statistics #49 (yjshen)

Full Changelog

Breaking changes:

• Make write_* return the number of written bytes. #45
• move HybridRleDecoder from read::levels to encoding::hybrid_rle #41
• Simplified split of page buffer #37 (jorgecarleitao)
• Simplified API to get page iterator #36 (jorgecarleitao)

New features:

• Added support to write to async writers. #35 (jorgecarleitao)

Fixed bugs:

• Fixed edge case of a small bitpacked. #43 (jorgecarleitao)
• Fixed error in decoding RLE-hybrid. #40 (jorgecarleitao)

Enhancements:

• Removed requirement of "Seek" on write. #44 (jorgecarleitao)

Documentation updates:

• Added guide to read #38 (jorgecarleitao)

Testing updates:

• Made tests deserializer use the correct decoder. #46 (jorgecarleitao)

Full Changelog

Breaking changes:

• Added option to apply filter pushdown to data pages. #34 (jorgecarleitao)
• Add support to read async #33 (jorgecarleitao)

New features:

• Add support for async read #32
• Added option to apply filter pushdown to data pages. #34 (jorgecarleitao)

Full Changelog

Enhancements:

• Add support to write dictionary-encoded pages #29
• Upgrade zstd to ^0.9 #31 (Dandandan)

* This Changelog was automatically generated by github_changelog_generator