kafka-protocol

What's Changed

Breaking changes ⚠️

• feat: add KIP-534 Batch attribute delete_horizon by @zac-torkelson in #157

Fixes

• fix: snappy decompress corrupts data when Xerial magic check fails by @StevenLeRoux in #149
• fix: decode/encode record timestampDelta as VarLong (i64) instead of VarInt (i32) by @hadiezatpanah in https://github.com/
  /pull/159

Features

• Make ResponseError fn const by @allevo in #153
• Expose batch decode info by @jakubdyszkiewicz in #150

Dependencies

• Bump serde_json from 1.0.145 to 1.0.151 by @dependabot[bot] in #166
• Bump serde from 1.0.228 to 1.0.229 by @dependabot[bot] in #165
• Bump anyhow from 1.0.100 to 1.0.104 by @dependabot[bot] in #168
• Bump bytes from 1.11.0 to 1.12.1 by @dependabot[bot] in #169
• Bump flate2 from 1.1.5 to 1.1.9 by @dependabot[bot] in #167
• Bump indexmap from 2.12.1 to 2.14.0 by @dependabot[bot] in #174
• Bump testcontainers from 0.25.2 to 0.27.3 by @dependabot[bot] in #172
• Bump snap from 1.1.1 to 1.1.2 by @dependabot[bot] in #170
• Bump uuid from 1.18.1 to 1.24.0 by @dependabot[bot] in #173
• Bump git2 from 0.18.3 to 0.21.0 by @dependabot[bot] in #171
• Bump testcontainers from 0.27.3 to 0.28.0 by @dependabot[bot] in #176
• chore(deps): drop unmaintained paste dependency by @michaelbeaumont in #160
• Bump parse-display from 0.9.1 to 0.11.0 by @dependabot[bot] in #175
• Bump serde_with from 3.16.0 to 3.22.0 by @dependabot[bot] in #180
• Bump rustls-webpki from 0.103.8 to 0.103.14 by @dependabot[bot] in #179
• chore(deps): update h2 to v0.4.16 by @michaelbeaumont in #183

New Contributors

• @StevenLeRoux made their first contribution in #149
• @allevo made their first contribution in #153
• @zac-torkelson made their first contribution in #157
• @hadiezatpanah made their first contribution in #159
• @jakubdyszkiewicz made their first contribution in #150
• @dependabot[bot] made their first contribution in #166

Full Changelog: 0.17.0...0.18.0

Breaking changes ⚠️

• feat: add KIP-534 Batch attribute delete_horizon by @zac-torkelson in https://github.com/kafka-protocol-rs/kafka-protocol-rs/pull/157

Fixes

• fix: snappy decompress corrupts data when Xerial magic check fails by @StevenLeRoux in https://github.com/kafka-protocol-rs/kafka-protocol-rs/pull/149
• fix: decode/encode record timestampDelta as VarLong (i64) instead of VarInt (i32) by @hadiezatpanah in https://github.com/ kafka-protocol-rs/kafka-protocol-rs/pull/159

Features

• Make ResponseError fn const by @allevo in https://github.com/kafka-protocol-rs/kafka-protocol-rs/pull/153
• Expose batch decode info by @jakubdyszkiewicz in https://github.com/kafka-protocol-rs/kafka-protocol-rs/pull/150

What's Changed

• Fix ApiKey::iter by @rukai in #139
• feat: impl Hash for ApiKey by @michaelbeaumont in #141
• feat(StrBytes): add some const functions by @michaelbeaumont in #142
• fix: snappy compression by @michaelbeaumont in #143
• feat: fallback to non-Kafka snappy algorithm on decompress by @michaelbeaumont in #144
• Release 0.17 by @rukai in #145
• Bump to rust 1.88 for compatibility with dependencies by @rukai in #146

Full Changelog: 0.16.0...0.17.0

• Breaking change:
  • change snappy compression to:
    • encode using the upstream java algorithm
    • and decode by autodetecting the upstream java algorithm or true snappy algorithm (previous kafka-protocol-rs encoding implementation)
  • This change makes it possible to decompress bytes compressed by the upstream java
• Fix ApiKey::iter which would terminate early in 0.16.0

What's Changed

• Update Kafka error codes to 4.1.0 by @belltoy in #129
• Support kafka 4.0 [breaking change] by @rukai in #120
• Add decode_request_header_from_buffer and encode_request_header_to_buffer helper methods by @aovestdipaperino in #131
• fix: handle records with headers with no value by @michaelbeaumont in #134
• Update README.md to mention the correct version of Kafka supported by @aovestdipaperino in #132
• Update to kafka 4.1.0 by @rukai in #135
• Release kafka-protocol 0.16.0 by @rukai in #136

New Contributors

• @aovestdipaperino made their first contribution in #131

Full Changelog: 0.15.1...0.16.0

• Added support for kafka 4.0
  • deprecated message types were removed in kafka 4.0 and those types have also been removed from this crate.
  • removed support for message sets v0 and v1

What's Changed

• Fix bytes version by @belltoy in #126

Full Changelog: 0.15.0...0.15.1

• Ensure bytes dependency is at least 0.10.1

What's Changed

• [Breaking Change] Remove unneeded trait bounds from Records::encode/Records::decode by @rukai in #102
• #107 Explicitly use trait impl in types by @phughk in #108
• Update messages to kafka 3.9 by @rukai in #105
• feat: impl Borrow for StrBytes by @michaelbeaumont in #115
• fix: impl Borrow<[u8]> for StrBytes, not Borrow<str> by @michaelbeaumont in #117
• feat: return compression information when decoding if possible by @michaelbeaumont in #118
• Remove ByteBuf try_get_ methods [breaking change] by @rukai in #121
• Add version validation to encode/decode by @rukai in #123
• Release kafka-protocol 0.15.0 by @rukai in #124

New Contributors

• @phughk made their first contribution in #108
• @michaelbeaumont made their first contribution in #115

Full Changelog: 0.14.0...0.15.0

• Records::encode/Records::decode no longer has unneeded trait bounds.
• ByteBuf::try_get_* methods that already exist on Buf are removed in favor of the Buf method

What's Changed

• Explicitly use trait impl in types (branch V0.14) by @rukai in #109
• Release V0.14.1 by @rukai in #110

Full Changelog: 0.14.0...0.14.1

What's Changed

• [Breaking change] Fix calling Records::encode / Records::decode with None by @rukai in #100
• [Breaking change] remove "Key" suffix from ApiKey variants by @rukai in #97
• VersionRange additions by @rukai in #99
• ApiKey additions by @rukai in #98
• Release 0.14 by @rukai in #101

Full Changelog: 0.13.0...0.14.0

• Records::encode/Records::decode is now reverted back to their original API, instead encode_with_custom_compression and decode_with_custom_compression is provided for custom compression.
• ApiKey variants are renamed to remove the Key suffix.
• Add iterate_all and valid_versions methods to ApiKey
• Implement PartialEq and Display for VersionRange

What's Changed

• Make ApiKey non_exhaustive by @rukai in #76
• Update to kafka 3.8 by @rukai in #75
• Make Debug impl for newtypes passthrough to inner type by @rukai in #77
• Cleanup generate messages by @rukai in #82
• user provided compression/decompression for record batches by @pdeva in #81
• Cache CI builds by @rukai in #87
• make record compression into default features by @rukai in #86
• Update edition to 2021 by @rukai in #91
• update pinned rust + fix clippy lints by @rukai in #92
• Reduce allocations when decoding maps by @rukai in #94
• Replace indexmap with vec by @rukai in #90
• Improve error handling on encode version violation by @rukai in #96
• Release 0.13 by @rukai in #95

Full Changelog: 0.12.0...0.13.0

• All "map" types in the protocol that were previously of type IndexMap<K, V> are now of type Vec<V>, the value of K is stored as a field within V.
  • This was done to resolve https://github.com/tychedelia/kafka-protocol-rs/issues/84 and improve decoding speed.
  • If you were previously calling .get() on the map, the best way to migrate is to refactor your code to avoid the need to lookup by iterating over the items in the response instead.
  • Alternatively, you could replace responses.get(name) with something like responses.iter().find(|x| x.name == name) to achieve the same result. But note that this access is now O(N) instead of O(1).
  • Alternatively, you could use an intermediate hashmap before converting to a Vec to retain the O(1) lookup.
• Update protocol to kafka 3.8.0
• The Debug impl for new type wrappers now passes directly to the inner type. The full list of new type wrappers is BrokerId, GroupId, ProducerId, TopicName and TransactionalId. For example GroupId was previously GroupId("some group") but is now "some group".
• ApiKey is now non_exhaustive.
• Added gzip, zstd, snappy and lz4 features to enable the different compression algorithms for records (All enabled by default)

What's Changed

• Add helper methods to RequestKind and ResponseKind by @rukai in #70
• Client/Broker cargo features by @rukai in #71
• Rename messages_enum -> messages_enums by @rukai in #72
• Release 0.12.0 by @rukai in #73

Full Changelog: 0.11.0...0.12.0

• Add client feature to enable encoding of requests and decoding of responses (enabled by default)
• Add broker feature to enable encoding of responses and decoding of requests (enabled by default)
• Add messages_enums feature to enable ResponseKind and RequestKind enums
• Add encode, decode and header_version methods to ResponseKind and RequestKind

What's Changed

• [Breaking change] RequestKind/ResponseKind - Avoid redundant variant names by @rukai in #61
• Add basic integration test by @rukai in #63
• Add Display and more From<T> implementations for StrBytes by @Hackzzila in #64
• use CRC-32 ISO/HDLC instead of CRC-32 CKSUM by @Hackzzila in #62
• Add integration tests to produce and fetch messages by @Hackzzila in #65
• Use IntoIterator instead of Iterator for RecordBatchEncoder::encode by @Hackzzila in #67
• Remove panic from RecordBatchEncoder::encode by @Hackzzila in #68
• Remove EncodeError/DecodeError aliases by @Hackzzila in #69
• WIP: Replace builder to improve ergonomics by @Hackzzila in #66

Full Changelog: 0.10.2...0.11.0

• Replace derive_builder with custom impl.
• Remove EncodeError/DecodeError aliases
• Remove panic from RecordBatchEncoder::encode
• Use IntoIterator instead of Iterator for RecordBatchEncoder::encode
• Use CRC-32 ISO/HDLC instead of CRC-32 CKSUM.
• Add Display and more From<T> implementations for StrBytes.
• Avoid redundant variant names in RequestKind/ResponseKind.

• Implement From<T> for RequestKind and ResponseKind.

• Fix lz4 compression.

• Use anyhow for public error types. In general, these errors reflect non-recoverable programmer error and should not occur when interacting with a well-formed Kafka implementation.
• Rename StrBytes::from_str to from_static_str to avoid overlap with the FromStr trait.
• Improve codegen, including filtering out unstable messages, improving ordering of structs in generated code, and other misc improvements.
• Upgrade message version to upstream v3.7.0 Kafka.

• Remove the string crate.
• Update derive-builder.

• Expose protocol::buf::gap module.

• Re-export indexmap as part of the public api.

• Implement Zstd and Lz4 compression.

• Switch to crc32c crate, providing hardware acceleration for crc operations on supported platforms.
• Formatting fixes.
• Miscellaneous dependency updates.
• Fix issue with derive_builder implementation in which default fields were not being used correctly.

• Miscellaneous dependency updates.

• Add ApiKey::request_header_version and ApiKey::response_header_version to assist deserializing headers without reference to message type.

Bump uuid crate version.

• Clarify versioning support for upstream Kafka, tracking the latest Kafka stable release. Improve docs.

Enhancements

• Builder::builder trait for retrieving builder instances from messages without requiring extra builder imports.
• #[non_exhaustive] added to all items to ensure forward compatability with protocol upgrades.

• Add utilities for dealing with response error codes.

Enhancements

• Build messages from crate root.
• Upgrade to latest version of Kafka protocol.

Enhancements

• Add derive builder for all messages.

Initial release.