Confluent.SchemaRegistry

Enhancements

• References librdkafka.redist 1.6.1. Refer to the 1.6.0 and 1.6.1 release notes for more information. Headline features:
  • KIP-429: Incremental rebalancing.
  • KIP-447: Producer scalability for exactly once semantics.
  • KIP-480: Sticky partitioner.
• KIP-22: Support for custom partitioners.
• Confluent.Kafka can now be used with Mono on Linux and MacOS. Note: Mono is not a supported runtime.
• The debian9-librdkafka.so build of librdkafka has been replaced with a more portable one: centos6-librdkafka.so (note: Debian 9 is still supported).
• Exceptions thrown by Producer.Produce now include an inner exception with additional context on the error (joostas).
• Added ConfigureAwait(false) to async methods in the Avro Serdes.
• Added IsInvalid property to Handle class (volgunin).

Fixes

• Fixed race condition in ProtobufSerializer (yurii-hunter).

Enhancements

• References librdkafka 1.5.3. Refer to the release notes for more information.
• References Apache.Avro v1.10.1, which adds support for enum defaults AVRO-2750.

Enhancements

• References librdkafka 1.5.2. Refer to the release notes for more information.
• Avro serializer now supports generic parameter ISpecificRecord. In this case, data is serialized according to the per-message concrete type (@ni-mi).

Enhancements

• Added support for OAuth Authentication via SASL/OAUTHBEARER (KIP-255) (thtp).

Enhancements

• References librdkafka 1.5.0 which brings many small improvements and bug fixes (and no new large features). Refer to the release notes for more information.
• Added support for Schema Registry SSL Authentication (@dinegri).

Fixes

• Resolved a stack overflow issue in Error(IntPtr error) #1249 (@midnightriot)
• References librdkafka 1.4.4. Refer to the release notes for more information.

Fixes

• Subject names are now URL encoded when used in Schema Registry URLs.
• Fixed a memory leak that occured when passing a CancellationToken to Producer.ProduceAsync.

Fixes

• Maintenance release. Refer to the librdkafka release notes for more information.
• Fixed incorrect content-type header in Schema Registry HTTP requests (@jeremy001181).

Enhancements

• References librdkafka v1.4.0. Refer to the release notes for more information. Headline features:
  • KIP-98: Producer support for transactions (@edenhill). This is the final piece in the puzzle required to enable exactly once stream processing (EOS) in .NET.
  • KIP-345: Static consumer group membership (@rnpridgeon).
  • KIP-511: Client name and version are now provided to brokers.
• Added Protobuf and JSON serdes including integration with Schema Registry.
• Switched to the official Apache Avro nuget package, which includes support for logical types, and all fixes from the Confluent fork, which has now been discontinued.
• Message headers are now exposed to serdes via SerializationContext (@pascalconfluent).
• Added a CancellationToken parameter to the ProduceAsync methods.
• Uncaught exceptions thrown in handler methods are now propagated to the initiating function, or in the case of error or log events, ignored. Previously, they would cause the application to terminate.
• Added a WordCount example demonstrating a streaming map-reduce application with exactly-once processing.

Changes

• Some internal improvements to the Consmer (thanks to @andypook).
• BREAKING CHANGE: net452 is no longer a target framework of Confluent.SchemaRegistry or Confluent.SchemaRegistry.Serdes due to the switch to the official Apache Avro package which only targets netstandard2.0.
• Marked properties on ConsumeResult that simply delegate to the corresponding properties on ConsumeResult.Message as obsolete.

Fixes

• Fixed an ArgumentNullException regression in ListGroups (thanks to @andypook).

Enhancements

• Added support for Subject Name Strategies to Confluent.SchemaRegistry (thanks to @fipil, @alexpedrero and @eroyal for their input).
• ConsumeResult now throws MessageNullException, not NullReferenceException when a message property is accessed but no message exists (thanks to @enzian for this change).
• References librdkafka v1.3.0. Refer to the release notes for more information. Headline feature is support for fetch from follower (KIP-392).

Changes

• Deprecated properties of SchemaRegistryConfig with the (superfluous) prefix SchemaRegistry. Added corresponding properties without this prefix.

Fixes

• Resolved issue 993 whereby RestService was unable to communicate with Schema Registry hosted on a non-root path. Thanks to @jonathansant for this fix.

• References librdkafka v1.2.2 which upgrades the lz4 dependency to v1.9.2.

Fixes

• References librdkafka v1.2.1 which resolves an issue that broke GSSAPI authentication on Windows.

Bugs

WARNING: There is an issue with SASL GSSAPI authentication on Windows with this release. This is resolved in v1.2.1.

Enhancements

• References librdkafka v1.2.0. Refer to the release notes for more information. Headline feature is consumer side support for transactions.
• Added IDictionary overload to Config constructors (contribution by @AndyPook).
• Confluent.Kafka, Confluent.SchemaRegistry and Confluent.SchemaRegistry.Serdes are now all signed, and Confluent.Kafka.StrongName deprecated.

Fixes

• Updated the librdkafka build load order so that the most featureful version is used on any given platform.

Enhancements

• References librdkafka v1.1.0. Refer to the release notes for more informtion. Notable improvement for Windows SSL users: You no longer need to specify a CA certificate file/directory (SslCaLocation) - librdkafka will load the CA certs by default from the Windows Root Certificate Store.

Enhancements

• Support for Alpine Linux.
• New LogLevelType enum and functions to convert between different log level type levels.
• Added netstandard20 as a target.
• References librdkafka 1.0.1.

Summary

1.0.0 is a major update of the API, introducing many new features and enhancements. Note: The 1.0 API is not compatible with earlier versions of the library.

Feature highlights:

• Inherits all of the new features in librdkafka v1.0.0
• General improvements to client classes:
  • Strongly typed configuration.
  • Construction is via builder classes:
    • Allows/enforces that event handlers are specified at construction time.
    • More extensible.
  • Header support.
  • New Message class abstraction and related changes.
  • Consistency in error reporting across library (via exceptions).
  • Support for fatal errors.
• Added AdminClient:
  • CreateTopics, DeleteTopics, CreatePartitions, AlterConfigs, and DescribeConfigs methods.
  • Moved ListGroups and GetMetadata methods from the Producer and Consumer classes to AdminClient.
• Producer specific improvements:
  • New serialization interface:
    • Non-blocking support for async serializers.
    • Very flexible:
      • e.g. can be easily extended to support header serialization.
  • Capability to specify custom timestamps when producing messages.
  • Message persistence status support.
  • Renamed ProduceAsync variants with a callback to Produce.
• Consumer improvements:
  • A new rebalance API.
  • New deserialization API analogous to the new serialization API.
  • PartitionEOF notification is via ConsumeResult, not events.
    • EOF notification is now disabled by default. To enable, set the EnablePartitionEof config property to true.
• Confluent Schema Registry integration
  • Added support for basic authentication.
  • Added GET subject versions to the cached schema registry client.
  • Renamed Confluent.Kafka.Avro to Confluent.SchemaRegistry.Serdes in preparation for support for additional serialization formats.

New Features

• Revamped producer and consumer serialization functionality.
  • There are now two types of serializer and deserializer: ISerializer<T> / IAsyncSerializer<T> and IDeserializer<T> / IAsyncDeserializer<T>.
    • ISerializer<T>/IDeserializer<T> are appropriate for most use cases.
    • IAsyncSerializer<T>/IAsyncDeserializer<T> are async friendly, but less performant (they return Tasks).
  • Changed the name of Confluent.Kafka.Avro to Confluent.SchemaRegistry.Serdes (Schema Registry may support other serialization formats in the future).
  • Added an example demonstrating working with protobuf serialized data.
• Consumers, Producers and AdminClients are now constructed using builder classes.
  • This is more verbose, but provides a sufficiently flexible and future proof API for specifying serdes and other configuration information.
  • All events on the client classes have been replaced with corresponding Set...Handler methods on the builder classes.
    • This allows (enforces) handlers are set on librdkafka initialization (which is important for some handlers, particularly the log handler).
    • events allow for more than one handler to be set, but this is often not appropriate (e.g. OnPartitionsAssigned), and never necessary. This is no longer possible.
    • events are also not async friendly (handlers can't return Task). The Set...Handler appropach can be extend in such a way that it is.
• Avro serdes no longer make blocking calls to ICachedSchemaRegistryClient - everything is awaited.
  • Note: The Consumer implementation still calls async deserializers synchronously because the Consumer API is still otherwise fully synchronous.
• Reference librdkafka.redist 1.0.0-RC7
  • Notable features: idempotent producer, sparse connections, KIP-62 (max.poll.interval.ms).
  • Note: End of partition notification is now disabled by default (enable using the EnablePartitionEof config property).
• Removed the Consumer.OnPartitionEOF event in favor notifying of partition eof via ConsumeResult.IsPartitionEOF.
• Removed ErrorEvent class and added IsFatal to Error class.
  • The IsFatal flag is now set appropriately for all errors (previously it was always set to false).
• Added PersistenceStatus property to DeliveryResult, which provides information on the persitence status of the message.

Fixes

• Added Close method to IConsumer interface.
• Changed the name of ProduceException.DeliveryReport to ProduceException.DeliveryResult.
• Fixed bug where enum config property couldn't be read after setting it.
• Added SchemaRegistryBasicAuthCredentialsSource back into SchemaRegistryConfig (#679).
• Fixed schema registry client failover connection issue (#737).
• Improvements to librdkafka dependnecy discovery (#743).

New Features

• References librdkafka 1.0.0-PRE1. Highlights:
  • Idempotent producer.
  • Sparse connections (broker connections are only held open when in use).

Enhancements / Fixes

• Fixed a memory leak in ProduceAsync #640 (regression from 0.11.x).

New Features

• Added an AdminClient, providing CreateTopics, DeleteTopics, CreatePartitions, DescribeConfigs and AlterConfigs.
• Can now produce / consume message headers.
• Can now produce user defined timestamps.
• Added IClient, IProducer and IConsumer interfaces (useful for dependency injection and mocking when writing tests).
• Added a Handle property to all clients classes:
  • Producers can utilize the underlying librdkafka handle from other Producers (replaces the 0.11.x GetSerializingProducer method on the Producer class).
  • AdminClient can utilize the underlying librdkafka handle from other AdminClients, Producers or Consumers.
• IDeserializer now exposes message data via ReadOnlySpan<byte>, directly referencing librdkafka allocated memory. This results in a considerable (up to 2x) performance increase and reduced memory.
• Most blocking operations now accept a CancellationToken parameter.
  • TODO: in some cases there is no backing implementation yet.
• .NET Specific configuration parameters are all specified/documented in the ConfigPropertyNames class.

Major Breaking API Changes

• The Message class has been re-purposed and now encapsulates specifically the message payload only.
  • ProduceAsync / BeginProduce now return a DeliveryReport object and Consumer.Consume returns a ConsumeResult object.
• The methods used to produce messages have changed:
  • Methods that accept a callback are now named BeginProduce (not ProduceAsync), analogous to similar methods in the standard library.
  • Callbacks are now specified as Action<DeliveryReportResult<TKey, TValue>> delegates, not implementations of IDeliveryHandler.
  • The IDeliveryHandler interface has been deprecated.
  • There are two variants of ProduceAsync and BeginProduce, the first takes a topic name and a Message. The second takes a TopicPartition and a message.
    • i.e. when producing, there is now clear separation between what is produced and where it is produced to.
  • The new API is more future proof.
  • ProduceAsync now calls SetException instead of SetResult on the returned Task, making error checking more convenient and less prone to developer mistakes.
• The feature to block ProduceAsync calls on local queue full has been removed (result in Local_QueueFull error). This should be implemented at the application layer if required.
• The non-serializing Producer and non-deserializing Consumer types have been removed (use generic types with byte[] instead), considerably reducing API surface area.
• The ISerializingProducer interface has been removed - you can achieve the same functionality by sharing client handles instead.
• The Consumer.Poll method and corresponding OnMessage event have been removed. You should use Consumer.Consume instead.
• The Consumer.OnConsumeError has been removed. Consume errors are now exposed via a ConsumeException.
• The Consumer.Consume method now returns a ConsumeResult object, rather than a Message via an out parameter.
• CommitAsync has been removed (use Commit instead).
• Commit errors are reported via an exception and method return values have correspondingly changed.
• ListGroups, ListGroup, GetWatermarkOffsets, QueryWatermarkOffsets, and GetMetadata have been removed from Producer and Consumer and exposed only via AdminClient.
• Added Consumer.Close.
• Various methods that formerly returned TopicPartitionOffsetError / TopicPartitionError now return TopicPartitionOffset / TopicPartition and throw an exception in case of error (with a Result property of type TopicPartitionOffsetError / TopicPartitionError).

Minor Breaking API Changes

• Removed cast from Error to bool.
• Consumer.OffsetsForTimes if provided an empty collection will return an empty collection (not throw an exception).
• manualPoll argument has been removed from the Producer constructor and is now a configuration option.
• enableDeliveryReports argument has been removed from the Producer constructor and is now a configuration option.
• Removed methods with a millisecondsTimeout parameter (always preferring a TimeSpan parameter).
• Added Consumer.Consume variants with a CancellationToken parameter.
• Added A Producer.Flush method variant without a timeout parameter (but with a CancellationToken parameter that is observed).
• Added the SyslogLevel enumeration, which is used by the log handler delegate.

Minor Enhancements / Fixes

• When delivery reports are disabled, ProduceAsync will return completed Tasks rather than Tasks that will never complete.
• Avro serializers / deserializer now handle null values.
• Examples upgraded to target 2.1.
• Changed name of HasError to IsError
• Configuration options have been added to allow fine-grained control over of marshalling of values to/from librdkafka (for high performance usage).
  • headers, message keys and values, timestamps and the topic name.
• Improved XML API documentation.

Changes

• Moved SyncOverAsync functionality to the Confluent.Kafka.SyncOverAsync namespace.
• Marked DependentProducerBuilder as API-SUBJECT-TO-CHANGE.
• No-op handlers are no longer registered with librdkafka if corresponding handlers are not specified in client builder classes.
• Renamed AsyncAvroSerializer to AvroSerializer and AsyncAvroDeserializer to AvroDeserializer

New Features

• Added DependentAdminClientBuilder class.

Changes

• Reverted RC4 changes.
• Renamed AvroSerializer to AsyncAvroSerializer and AvroDeserializer to AsyncAvroDeserializer
• Added SyncOverAsyncSerializer and SyncOverAsyncDeserializer adapter classes.
• Added AsSyncOverAsync factory method to AsyncAvroSerializer and AsyncAvroDeserializer.
• Removed IAsyncDeserializer setter overloads from the ConsumerBuilder class.
• Renamed Producer.BeginProduce to Producer.Produce.
• Produce throws an exception if used when async serializers are configured.
• Made AdminClient, Producer, and Consumer classes internal.

Changes

• Removed SerializationContext from non-async serde interfaces.
• Replaced ISerializer interface with Serializer delegate.
• Replaced IDeserializer interface with Deserializer delegate.

New Features

• Producer.Poll can now be used with producer instances that are in background polling mode.
  • Typically use: Block for a minimal period of time following a ErrorCode.Local_QueueFull error.

Changes

• Removed the Confluent.Kafka.Serdes namespace.

Fixes

• Added CompressionType property to ProducerConfig class.

New Features

• References librdkafka.redist v1.0.0

Changes

• Moved API docs from the client classes to their respective interfaces.
• Update formatting of client API docs so they display well in Visual Studio Code intellisense.