2.16.0-rc1 (#2668)

Enhancements

• References librdkafka.redist 2.15.1. Refer to the librdkafka v2.15.1 release notes for more information.

Fixes

• Fix early return in FindMessageByName (#2644)
• Preserve HTTP status for retriable SR errors (#2647)
• Security patch in examples and tests using old NJsonSchema versions, test framework upgrade in Confluent.Kafka tests (#2656)

• Revert "DGS-24721 Add support for saving Azure key version with DEK (#2641)"

This reverts commit de9b874.

• Revert "DGS-24778 Pass context when clients make KEK calls to DEK Registry (#2642)"

This reverts commit a81057f.

• Revert "Add alicloud kms provider (#2648)"

This reverts commit 01095e2.

• Revert "Pack Confluent.SchemaRegistry.Encryption.AliCloud (#2649)"

This reverts commit 1ed651d.

• Revert "DGS-24222 Add support for inline validation rules (#2651)"

This reverts commit e31a347.

• v2.15.1-RC1

• Use librdkafka 2.15.1 final version

Upgrade considerations

Ensure you're not calling StoreOffset with an EOF message.
As when calling Commit with an EOF message, it will throw an InvalidOperationException.
Previously this could cause skipping an offset (<2.1.0) or message reprocessing (>=2.1.0).

New features

• New optional package Confluent.Kafka.OAuthBearer.Aws provides AWS IAM-based
  OAUTHBEARER authentication via AWS STS GetWebIdentityToken. Activate by
  setting SaslOauthbearerMethod = SaslOauthbearerMethod.Oidc,
  SaslOauthbearerMetadataAuthenticationType.AwsIam, and supplying
  region and audience in SaslOauthbearerConfig. See the package's README
  for details.

Enhancements

• References librdkafka.redist 2.15.0. Refer to the librdkafka v2.15.0 release notes for more information.
• Added support for .NET 10 (LTS) and dropped support for .NET 6 (end of support since November 2024). The library now multi-targets net8.0 and net10.0 alongside netstandard2.0 and net462. Builds on the approach proposed in #2545 by Fernando Luiz de Lima (@ffernandolima), adapted for the centralized Directory.Build.props structure on current master.
• Adds support for union-of-pools/ auto pool mapping for Schema Registry (#2634)

Fixes

• Throw an InvalidOperationException when calling StoreOffset with an EOF consume result, to match Commit behavior (#2621)
• Handle non-http errors during retries (#2633)

v2.15.0-RC2 (#2637)

Enhancements

• References librdkafka.redist 2.14.2. Refer to the librdkafka v2.14.2 release notes for more information.
• Add IHeader overload to Headers.Add() method (#2604)

Fixes

• Handle anyOf/allOf in JSON transforms (#2611)
• Allow encrypting enum for JObject in dotnet (#2615)
• Fix 'occured' -> 'occurred' typo in ConsumeException XML doc comments (#2609)

• v2.14.2-RC1

• Address review: keep README at last final, note skipped 2.14.1

Enhancements

• References librdkafka.redist 2.14.0. Refer to the librdkafka v2.14.0 release notes for more information.
• Add AssociatedNameStrategy (#2577)
• Add enableAt to RuleSet (#2593)
• Add config to validate before domain rules for JSON Schema (#2596)
• Cache FieldAccessor to address CSFLE perf with JSON (#2601)

Fixes

• Ensure dispose methods clean up properly (#2595)

v2.14.0-RC1 (#2600)

Enhancements

• Preserve inner exceptions in RestService (#2571)

Fixes

• Support multi-certificate PEM bundles in Schema Registry SslCaLocation (#2576)

• Release version v2.13.2-RC1

• comments fixed

Enhancements

• Unify build configurations and remove redundant properties (#2464)
• JSONSchema Validation Errors: Include paths to nested properties (#2554)
• Add Confluent-Client-Version header to requests to SR, C# client (#2549)

Fixes

• Fix DeriveKey in Hkdf implementation (#2559)
• Ensure single use of HttpRequestMessage for token fetch (#2544)
• Fix race conditions when retrieving KEK/DEK (#2565)
• Ensure JSON schema access is thread-safe (#2567)
• Use lock on root schema when protecting access to subschemas (#2568)
• Processing arrays and maps sequentially to avoid race conditions (#2569)

Enhancements

• References librdkafka.redist 2.13.0. Refer to the librdkafka v2.13.0 release notes for more information.
• Support Avro schema references for C# client (#2534)
• Add Accept-Version header (#2536)
• Handle evolution during field transformation (#2541)
• Re-enable deserialization optimization by removing unnecessary array conversions (#2531)

Fixes

• Ensure schemaId initialization is thread-safe (#2540)
• Ensure all deps are strongly named (#2548)

KIP-848 – General Availability

Starting with Confluent.Kafka 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Please refer to the following migration guide for moving from Classic to Consumer protocol.

Note: The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. GroupProtocol configuration property dictates whether to use the new Consumer protocol or older Classic protocol. It defaults to Classic if not provided.

Enhancements

• References librdkafka.redist 2.12.0. Refer to the librdkafka v2.12.0 release notes for more information.
• OAuth OIDC method example for Kafka metadata based authentication with an Azure IMDS endpoint using an attached managed identity as principal (#2526).

Enhancements

• References librdkafka.redist 2.11.1. Refer to the librdkafka v2.11.1 release notes for more information.

Enhancements

• References librdkafka.redist 2.11.0. Refer to the librdkafka v2.11.0 release notes for more information.

Enhancements

• References librdkafka.redist 2.10.1. Refer to the librdkafka v2.10.1 release notes for more information.
• Refactor serializer implementations to remove unnecessary ContinueWith calls (#2448)
• Remove shared mutex from CachedSchemaRegistryClient (#2449)
• Protobuf serialization perf improvements (#2459)
• Refactor AuthenticationHeaderValueProvider construction (#2463)
• Remove redundant ConfigureAwait (#2469)
• Support for schema id in header (#2465)
• Use vectorized null-terminated C-string parsing in PtrToStringUTF8 for .NET 6+ (#2466)
• Rework SchemaId encoding in headers to minimize heap allocations (#2480)
• Raise an error if Protobuf deprecated format is specified (#2479)
• Avoid array copy on Avro deserialization (#2483)

Fixes

• Fix JSON Schema validation to use latest schema if needed (#2406)
• Minor check for null client (#2462)
• Fix possible NPE in CSFLE executor (#2467)
• Add missing ConfigureAwait calls (#2477)
• Fix rule transformation of list properties (#2475)

Enhancements

• References librdkafka.redist 2.10.0. Refer to the librdkafka v2.10.0 release notes for more information.
• [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#2366)

Enhancements

• Add utilities to convert decimals from/to Protobuf (#2424)
• Add client credentials OAuth support (#2426)
• Add support for passing schema ID during serialization (#2428)
• Optimize schema registry client by caching results of LookupSchemaAsync (#2429)
• Use ConcurrentDictionary as default cache for Schema Registry client (#2433)

Fixes

• Fix JSON Schema validation to use latest schema if needed (#2406)
• Support Protobuf oneof fields in Data Contract rules (#2413)

Enhancements

• References librdkafka.redist 2.8.0. Refer to the librdkafka v2.8.0 release notes for more information.
• Add retry logic to RestService for Schema Registry (#2353)
• Add config for validating payloads against JSON Schema (#2350)
• Add ability to override disable flag and actions on a rule (#2377)
• Add AWS AssumeRole support to AWS KMS (#2379)

Fixes

• Retrieve version for writer schema when getting migrations (#2358)
• Add missing JSON Schema validate check (#2372)
• Ensure different key ids use different client instances (#2374)
• DGS-19409 Ensure Avro serde caches per subject (#2387)
• Update hash code for schema references so they can be retrieved correctly (#2390)

Enhancements

• References librdkafka.redist 2.6.1. Refer to the librdkafka v2.6.1 release notes for more information.

Fixes

• Fix to continue supporting .NET Framework 4.6.2+ in core client library (#2342).
• Fix JSON Schema handling to not require use of $id (#2339).
• Update Caching.Memory to 8.0.1 to address CVE (#23440.
• Added Qualified and Custom reference name strategy approaches for protobuf references (#2345).
• Fix validate of SSL CA certs in Schema Registry client (#2346).
• Skip SSL certs validation when configured in Schema Registry client (#2347).
• Allow proxy to be specified in Schema Registry client (#2348).

Enhancements

• References librdkafka.redist 2.6.0. Refer to the librdkafka v2.6.0 release notes for more information.
• KIP-848 EA: Admin API for listing consumer groups now has an optional filter to return only groups of given types (#2323).
• KIP-460 Admin Leader Election RPC (#2320)
• .NET 8 support with NJsonSchema 11 (#2314)

Enhancements

• References librdkafka.redist 2.5.3. Refer to the librdkafka v2.5.3 release notes for more information.

Fixes

• Properly handle messages with well-known types in Protobuf serializer
• Use AES128_GCM in the Local KMS client, for consistency with Java/go
• Include deleted schemas when getting schemas by subject and version
• Handle signed ints when transforming Protobuf payloads
• Allow null SchemaRegistryClient in AsyncSerde constructor

[!WARNING] Versions 2.5.0, 2.5.1 and 2.5.2 have a regression in which an assert is triggered during PushTelemetry call. This happens when no metric is matched on the client side among those requested by broker subscription.

You won't face any problem if:

• Broker doesn't support KIP-714.
• KIP-714 feature is disabled on the broker side.
• KIP-714 feature is disabled on the client side. This is enabled by default. Set configuration enable.metrics.push to false.
• If KIP-714 is enabled on the broker side and there is no subscription configured there.
• If KIP-714 is enabled on the broker side with subscriptions that match the KIP-714 metrics defined on the client.

Having said this, we strongly recommend using v2.5.3 and above to not face this regression at all.

Fixes

• Fix CSFLE (client-side field-level encryption) to use the Google Tink format for DEKs for interoperability with clients in other languages (Java, go, etc.).
• Improve error when specifying an invalid KMS type for CSFLE
• Enhance CSFLE examples with KMS configuration settings

Fixes

• Fix CSFLE (client-side field-level encryption) when using Azure Key Vault by specifying RsaOaep256 (instead of RsaOaep) for interoperability with clients in other languages (Java, go, etc.).
• Fix AvroSerializer configuration to allow using schema normalization.
• Upgrade Azure Identity library to 1.11.4 to address a vulnerability in previous versions.

Enhancements

• References librdkafka.redist 2.5.0. Refer to the librdkafka v2.5.0 release notes for more information.
• Add support for metadata and ruleSet in the schema registry client, which together support data contracts.
• Add support for CSFLE (client-side field-level encryption) for AWS, Azure, GCP, and HashiCorp Vault. See the encryption examples in the examples directory.
• Add support for CEL, CEL_FIELD, and JSONata rules.

Fixes

• Switch license expression and other repo information. (#2192, @thompson-tomo)

Enhancements

• References librdkafka.redist 2.4.0. Refer to the librdkafka v2.4.0 release notes for more information.
• KIP-848 EA: Added KIP-848 based new consumer group rebalance protocol. Integration tests running with the new consumer group protocol. The feature is an Early Access: not production ready. Please refer detailed doc for more information. (#2212).

Enhancements

• References librdkafka.redist 2.3.0. Refer to the librdkafka v2.3.0 release notes for more information.
• KIP-430: Return authorized operations in describe responses (#2021, @jainruchir).
• KIP-396: Added support for ListOffsets Admin API (#2086).
• Add Rack to the Node type, so AdminAPI calls can expose racks for brokers (currently, all Describe Responses) (#2021, @jainruchir).
• Added support for external JSON schemas in JsonSerializer and JsonDeserializer (#2042).
• Added compatibility methods to CachedSchemaRegistryClient (ISBronny, #2097).
• Add support for AdminAPI DescribeCluster() and DescribeTopics() (#2021, @jainruchir).

Enhancements

• References librdkafka.redist 2.2.0. Refer to the librdkafka v2.2.0 release notes for more information.
• KIP-339 IncrementalAlterConfigs API (#2005).
• KIP-554: User SASL/SCRAM credentials alteration and description (#2070).

Fixes

• Fix backwards compatability of TopicPartitionOffset constructor. (drinehimer, #2066)
• Fix IConsumer breaking change. (ttd2089, #2071)

Enhancements

• References librdkafka.redist 2.1.1. Refer to the librdkafka v2.1.1 release notes for more information.
• Less heap allocations when calling Produce (bjornbouetsmith, #2020)

Enhancements

• References librdkafka.redist 2.1.0. Refer to the librdkafka v2.1.0 release notes for more information.
• Added SetSaslCredentials. This new method (on the Producer, Consumer, and AdminClient) allows modifying the stored SASL PLAIN/SCRAM credentials that will be used for subsequent (new) connections to a broker (#1980).
• Changed the way the _SCHEMA filed is accessed internally from reflecting the static field to accessing it from the instance (AlexeyRaga).
• KIP-320: add offset leader epoch fields to the TopicPartitionOffset, TopicPartitionOffsetError and ConsumeResult classes (#2027).

Fixes

• Fixed OverflowException thrown intermittently when using the ListGroup method (#2003).

Upgrade considerations

OpenSSL 3.0.x upgrade in librdkafka requires a major version bump, as some legacy ciphers need to be explicitly configured to continue working, but it is highly recommended NOT to use them. The rest of the API remains backward compatible.

Enhancements

• References librdkafka.redist 2.0.2. Refer to the librdkafka v2.0.0 release notes and later ones for more information.
• Upgraded NJsonSchema to v10.6.3
• Added LatestCompatibilityStrict configuration property to JsonSerializerConfig to check the compatibility with latest schema when UseLatestVersion is set to true.
• Added DeleteConsumerGroupOffset to AdminClient.
• KIP-222 Finish remaining implementation: Add Consumer Group operations to Admin API (DeleteGroups is already present).
• KIP-518 Allow listing consumer groups per state.
• KIP-396 Partially implemented: support for AlterConsumerGroupOffsets.
• As result of the above KIPs, added (#1981)
  • ListConsumerGroups Admin operation. Supports listing by state.
  • DescribeConsumerGroups Admin operation. Supports multiple groups.
  • ListConsumerGroupOffsets Admin operation. Currently, only supports 1 group with multiple partitions. Supports the requireStable option.
  • AlterConsumerGroupOffsets Admin operation. Currently, only supports 1 group with multiple offsets.

Fixes

• During a group rebalance, partitions are now always revoked as a side effect of a call to Consume, whether or not a partitions revoked handler has been specified. Previously, if no handler was specified, the timing of when the consumer lost ownership of partitions during a rebalance was arbitrarily, frequently resulting in an erroneous state exception when committing or storing offsets.
• Fixed 100% CPU usage with DependentAdminClientBuilder.

Note: There were no 2.0.0 and 2.0.1 releases.

Enhancements

• Added NormalizeSchemas configuration property to the Avro, Json and Protobuf serdes.

Fixes

• Schema Registry authentication now works with passwords that contain the ':' character (luismedel).
• Added missing librdkafka internal and broker error codes to the ErrorCode enum.

Enhancements

• References librdkafka.redist 1.9.2 which includes an Apple M1 librdkafka build.
• Added ACL AdminClient operations (CreateAcls, DescribeAcls, DeleteAcls) (emasab).
• Added DeleteGroups to AdminClient (3schwartz).
• Enhanced the Avro Specific Deserializer to ignore the type namespace (sergemat).
• Improved efficiency of the statistics handler (VladimirTyrin).

Fixes

• The AdminClient poll loop no longer terminates when a request results in an error (emasab).
• Upgraded Newtonsoft.Json to 13.0.1 to address a security vulnerability in 9.0.1.

Enhancements

• References librdkafka.redist 1.9.0. Refer to the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
• References Apache.Avro 1.11.0. Refer to the release notes for further information (JanReimerD).
• Added support for serializing and deserializing null in Avro serdes (YairHalberstadt).
• Enhanced CachedSchemaRegistryClient to allow for user-implemented authentication schemes (henrydaly).
• Reduced memory use when producing with delivery reports disabled (TrickyCat).

Fixes

• Resolved incompatibility with Apple M1 processors (dkaukov).
• No longer crashes on Alpine Linux when MUSL is installed (shurivich).
• JSON validation exception messages now properly include failing paths (drinehimer).
• Upgraded Google.Protobuf dependency to 3.15.0 (CVE-2021-22570).
• Resolved memory leak in AdminClient response handler (emasab).
• Resolved memory leak in Producer.SendOffsetsToTransaction.

Upgrade Considerations

• The earliest supported .NET Framework version is now 4.6.2, previously this was 4.5 (bjornbouetsmith).