Azure.Messaging.EventHubs.Processor

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• tovyhnal (GitHub)

Features Added

• Support for the Event Hubs geographic data replication feature has been enabled. Checking for whether or not this feature is enabled for your namespace can be done by querying for Event Hub properties using EventHubProducerClient or EventHubConsumerClient and referencing the the IsGeoReplicationEnabled property of the result.

  As part of this feature, the type of offset-related data has been changed from long to string to align with changes to the Event Hubs service API. To preserve backwards compatibility, the existing offset-related members have not been changed, and new members with names similar to OffsetString and string-based parameters for method overloads have been introduced.

  The long-based offset members will continue to work for Event Hubs namespaces that do not have GeoDR replication enabled, but are discouraged for use and have been marked as obsolete.

  Obsoleted properties:

  • EventData.Offset
  • LastEnqueuedEventProperties.Offset
  • PartitionProperties.LastEnqueuedOffset

  Obsoleted method overloads:

  • EventPosition.FromOffset
  • EventHubsModelFactory.EventData
  • BlobCheckpointStore.UpdateCheckpointAsync
  • EventProcessorClient.UpdateCheckpointAsync

Bugs Fixed

• Fixed behavior in the UpdateCheckpointAsync method of EventProcessorClient to properly await the call to the checkpoint store. Previously, this was not awaited, causing the OTel span to open and close almost immediately and skip reporting any errors observed.

Other Changes

• Enhanced retry logic to consider additional cases for web socket-based failures. In many cases, a WebSocketException is triggered which wraps a SocketException with the details for the specific network conditions. Retry decisions are now based on the internal exception, if present, to ensure retries are correctly applied.

• Added annotations to make the package compatible with trimming and native AOT compilation.

• Added Event Hub name to processor load balancing logs for additional context. (A community contribution, courtesy of tovyhnal)

• Updated the Microsoft.Azure.Amqp dependency to 2.6.9, which contains several bug fixes. (see: commits)

Other Changes

• Bump Azure.Messaging.EventHubs dependency to 5.11.6, which includes bumps to several transitive dependencies.

Other Changes

• Bump Azure.Core.Amqp dependency to 1.3.1, which includes a fix to serialization of binary application properties.

Bugs Fixed

• Fixed an error that prevented relative URIs from being used with application properties in the EventData.Properties collection.

• Fixed an error with ETW logs which caused structured arguments for Id 105 (Event Processor position determined) to be out-of-order with the message format. This also caused the date to render incorrectly for some captures.

Other Changes

• The processor will now refresh the maximum message size each time a new AMQP link is opened; this is necessary for large message support, where the maximum message size for entities can be reconfigured and adjusted on the fly. Because the client had cached the value, it would not be aware of the change and would enforce the wrong size for batch creation.

• Adjusted the options used by the processor during startup validation to reduce the amount of data transferred and minimize impact on the outgoing events metric.

Bugs Fixed

• Fixed an error that caused connection strings using host names without a scheme to fail parsing and be considered invalid.

Other Changes

• Removed the restriction that endpoints used with the development emulator had to resolve to a localhost variant.

• Updated the Microsoft.Azure.Amqp dependency to 2.6.7, which contains several bug fixes, including for an internal NullReferenceException that would sometimes impact creating new links. (see: #258)

Bugs Fixed

• The EventProcessorClient will now create a unique span for each event emitted to the handler for processing. Previously a single span was created for all events in a batch. (#31922)

• Fixed a parameter ordering error in ETW 22 (EventBatchProcessingError) that caused structured data arguments to be ordered differently than the associated replacement token in the log message.

Other Changes

• Added option EnableCheckpointing to allow automatic checkpointing to be disabled for applications that only wish to read current data rather than tracking the state of processing.

Other Changes

• Updated the Microsoft.Azure.Amqp dependency to 2.6.5, which includes several bug fixes. A notable fix addresses an obscure race condition when a cancellation token is signaled while service operations are being invoked concurrently which caused those operations to hang.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Dave Trainer (GitHub)

Features Added

• Added a CheckpointPosition struct to use when updating a checkpoint. The specified position indicates that an event processor should begin reading from the next event. Added new UpdateCheckpointAsync overloads to EventProcessorClient and BlobCheckpointStore that accept the CheckpointPosition struct instead of individual values for offset and sequence number.

Breaking Changes

• The type of several existing values in the EventData.SystemProperties collection have been changed so that they are properly represented as .NET string types. Previously, the underlying AMQP types were unintentionally returned, forcing callers to call ToString() to read the value.

  This is a behavioral breaking change that will impacts only those callers who were explicitly casting system property values to AmqpAddress or AmqpMessageId before calling ToString(). The affected system properties are:

  • MessageId
  • CorelationId
  • To
  • ReplyTo

• The base implementations of both UpdateCheckpointAsync method overloads in PluggableCheckpointStoreEventProcessor<TPartition> and EventProcessor<TPartition> now choose sequence number over offset when writing a checkpoint and both values are provided. Previously, writing a checkpoint prioritized offset over sequence number. There is no difference in behavior for normal usage scenarios.

Bugs Fixed

• Fixed a race condition that could lead to a synchronization primitive being double-released if IsRunning was called concurrently while starting or stopping the processor.

• Fixed a bug in which cancellation honored by the processor was interpreted as an error surfaced by developer code and a warning was inappropriately emitted to the error handler.

• Fixed an issue with event processor validation where an exception for quota exceeded may inappropriately be surfaced when starting the processor.

• In the rare case that an event processor's load balancing and health monitoring task cannot recover from an error, it will now properly surrender ownership when processing stops.

Other Changes

• A new sample that demonstrates using the EventProcessorClient with an ASP.NET hosted service is now available. (A community contribution, courtesy of davetrainer)

• Updated the Microsoft.Azure.Amqp dependency to 2.6.4, which enables support for TLS 1.3.

• Removed the custom sizes for the AMQP sending and receiving buffers, allowing the optimized defaults of the host platform to be used. This offers non-trivial performance increase on Linux-based platforms and a minor improvement on macOS. Windows performance remains unchanged as the default and custom buffer sizes are equivalent.

• Improved efficiency of partition management during load balancing, reducing the number of operations performed and deferring waiting for lost partitions until the processor is stopped or the partition is reclaimed. Allocations were also non-trivially reduced.

• Improved the approach used by the processor to manage the background tasks for partition processing and load balancing. These tasks are now marked as long-running and have improved error recovery.

• Initialization of the load balancing task is now performed in the background and will no longer cause delays when starting the processor.

• Loosened validation for the fully qualified namespace name passed to the processor constructor. A URI is now also accepted as a valid format. This is intended to improve the experience when using the management library, CLI, Bicep, or ARM template to create the namespace, as they return only an endpoint for the namespace. Previously, callers were responsible for parsing the endpoint and extracting the host name for use with the processor.

• In the rare case that an event processor's load balancing and health monitoring task cannot recover from an error, the processor now signals the error handler with a wrapped exception that makes clear that processing will terminate. Previously, the source exception was surfaced to the error handler and the impact was not clear.

Breaking Changes

• Change ActivitySource name used to report message activity from Azure.Messaging.EventHubs.EventHubs to Azure.Messaging.EventHubs.Message and message Activity name from EventHubs.Message to Message.
• Updated tracing attributes names to conform to OpenTelemetry semantic conventions version 1.23.0.

Bugs Fixed

• Fixed a parameter type mismatch in ETW #7 (ReceiveComplete) which caused the duration argument of the operation to be interpreted as a Unicode string and fail to render properly in the formatted message.

Other Changes

• When an Event Hub is disabled, it will now be detected and result in a terminal EventHubsException with its reason set to FailureReason.ResourceNotFound.

Other Changes

• Several improvements to logging have been made to capture additional context and fix typos. Most notable among them is the inclusion of starting and ending sequence numbers when events are read from Event Hubs and dispatched for processing.

• The reference for the AMQP transport library, Microsoft.Azure.Amqp, has been bumped to 2.6.3. This fixes an issue with timeout duration calculations during link creation and includes several efficiency improvements.

Other Changes

• The reference for the AMQP transport library, Microsoft.Azure.Amqp, has been bumped to 2.6.2. This resolves a potential issue opening TLS connections on .NET 6+.

Bugs Fixed

• Removed the 30 second cap applied when opening AMQP links; this allows developers to fully control the timeout for service operations by tuning the TryTimeout as appropriate for the application.

Bugs Fixed

• Changed the approach that the event processor uses to validate permissions on startup to ensure that it does not interrupt other processors already running by temporarily asserting ownership of a partition.

Other Changes

• Enhanced the log emitted when an event processor begins reading from a partition to report whether the offset chosen was based on a checkpoint or default value.

Bugs Fixed

• Fix null reference exception when using the EventProcessorClient.

Features Added

• ActivitySource activities that are used when using the experimental OpenTelemetry support will include the az.schema_url tag indicating the OpenTelemetry schema version. They will also include the messaging attribute specified here.

Bugs Fixed

• Corrected log message issue causing formatting to fail when developer code for processing events leaks an exception. This obscured the warning that was intended to be emitted to the error handler.

Other Changes

• Calling ToString on an EventHubsException now includes details of any inner exception.

Bugs Fixed

• Corrected an indexing issue with the log event source, causing an exception to surface in some publishing scenarios.

Bugs Fixed

• Telemetry will now use a parent activity instead of links when the event processor is configured to use a CacheEventCount of 1.

• The reference for the AMQP transport library, Microsoft.Azure.Amqp, has been bumped to 2.5.12. This resolves a rare race condition encountered when creating an AMQP link that could cause the link to hang.

Other Changes

• Adjusted the frequency that a warning is logged when the processor owns more partitions than a basic heuristic believes is ideal. Warnings will no longer log on each load balancing cycle, only when the number of partitions owned changes.

• Added timing information to logs for AMQP publish and read operations.

Other Changes

• Added additional heuristics for the EventProcessorClient configuration to help discover issues that can impact processor performance and stability; these validations will produce warnings at processor start-up should potential concerns be found.

• Exception messages have been updated to include a link to the Event Hubs troubleshooting guide.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Daniel Marbach (GitHub)

Other Changes

• Miscellaneous performance improvements by reducing memory allocations. (A community contribution, courtesy of danielmarbach)

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Chad Vidovcich (GitHub)

Features Added

• The event processor error handler will now raise warning when an unhandled exception propagated from the event processing handler causing partition processing to fault and restart.

Bugs Fixed

• Fixed an issue with event processor startup validation where an invalid consumer group was not properly detected.

Other Changes

• EventProcessorClient and BlobCheckpointStore will now detect when an ownership blob has been deleted externally while the processor is running and gracefully recover.

• Samples now each have a table of contents to help discover and navigate to the topics discussed for a scenario. (A community contribution, courtesy of chadvidovcich)

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Daniel Marbach (GitHub)

Features Added

• The BlobCheckpointStore implementation used internally by the processor has been made public and now conforms to the CheckpointStore contract, allowing it to be used with custom processor implementations.

Other Changes

• Based on a new series of profiling and testing in real-world application scenarios, the default values for processor load balancing are being updated to provide better performance and stability. The default load balancing interval was changed from 10 seconds to 30 seconds. The default ownership expiration interval was changed from 30 seconds to 2 minutes. The default load balancing strategy has been changed from balanced to greedy.

• Added additional heuristics for the EventProcessorClient load balancing cycle to help discover issues that can impact processor performance and stability; these validations will produce warnings should potential concerns be found.

• EventProcessorClient will now log a verbose message indicating what event position was chosen to read from when initializing a partition.

• Removed allocations from Event Source logging by introducing WriteEvent overloads to handle cases that would otherwise result in boxing to object[] via params array. (A community contribution, courtesy of danielmarbach)

• Enhanced documentation to call attention to the need for the Azure Storage container used with the processor to exist, and highlight that it will not be implicitly created.

Bugs Fixed

• Dependencies have been updated to resolve an error when creating EventSource instances when used with Xamarin.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Andrey Shihov (GitHub)

Bugs Fixed

• Fixed an issue with refreshing authorization where redundant requests were made to acquire AAD tokens that were due to expire. Refreshes will now coordinate to ensure a single AAD token acquisition.

• Fixed an issue with authorization refresh where attempts may have been made to authorize a faulted link. Links that fail to open are no longer be considered valid for authorization.

Other Changes

• Documentation has been enhanced to provide additional context for client library types, notably detailing non-obvious validations applied to parameters and options members.

Bugs Fixed

• Fixed an issue where partition processing would ignore cancellation when the processor was shutting down or partition ownership changed and continue dispatching events to the handler until the entire batch was complete. Cancellation will now be properly respected.

Other Changes

• Added the ability to adjust the connection idle timeout using the EventHubConnectionOptions available within the options for each client type.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Daniel Marbach (GitHub)

Changes

Features Added

• When stopping, the EventProcessorClient will now attempt to force-close the connection to the Event Hubs service to abort in-process read operations blocked on their timeout. This should significantly help reduce the amount of time the processor takes to stop in many scenarios. (Based on a community prototype contribution, courtesy of danielmarbach)

• When the EventProcessorClient detects a partition being stolen outside of a load balancing cycle, it will immediately surrender ownership rather than waiting for a load balancing cycle to confirm the ownership change. This will help reduce event duplication from overlapping ownership of processors.

• The ConnectionOptions available when creating a processor now support registering a callback delegate for participating in the validation of SSL certificates when connections are established. This delegate may override the built-in validation and allow or deny certificates based on application-specific logic.

• The ConnectionOptions available when creating a processor now support setting a custom size for the send and receive buffers of the transport.

Key Bugs Fixed

• The EventProcessorClient will now properly respect another another consumer stealing ownership of a partition when the service forcibly terminates the active link in the background. Previously, the client did not observe the error directly and attempted to recover the faulted link which reasserted ownership and caused the partition to "bounce" between owners until a load balancing cycle completed.

• The EventProcessorClient will now be less aggressive when considering whether or not to steal a partition, doing so only when it will correct an imbalance and preferring the status quo when the overall distribution would not change. This will help reduce event duplication due to partitions moving between owners.

Changes

Features Added

• The processor will now perform validation of core configuration and permissions at startup, in order to attempt to detect unrecoverable problems more deterministically. Validation is non-blocking and will not delay claiming of partitions. One important note is that validation should be considered point-in-time and best effort; it is not meant to replace monitoring of error handler activity.

• Partition initialization has been moved to a background operation. This will allow partitions to be more efficiently managed and speed up ownership claims, especially when using the LoadBalancingStrategy.Greedy configuration or when the processor is recovering from some error conditions.

Key Bugs Fixed

• Dependencies have been updated to resolve security warnings for CVE-2021-26701. (The Event Hubs client library does not make use of the vulnerable components, directly or indirectly)

• The processor will no longer inappropriately determine that it should attempt to steal partitions from itself or when the load is balanced but there is an uneven ownership distribution. Previously, stealing was attempted but no candidates were found, leading to log spam but no interruption in processing.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Daniel Marbach (GitHub)

Changes

Features Added

• The EventProcessorClient now supports shared key and shared access signature authentication using the AzureNamedKeyCredential and AzureSasCredential types in addition to the connection string. Use of the credential allows the shared key or SAS to be updated without the need to create a new processor.

• Multiple enhancements were made to the AMQP transport paths for reading events to reduce memory allocations and increase performance. (A community contribution, courtesy of danielmarbach)

Key Bugs Fixed

• The AMQP library used for transport has been updated, fixing several issues including a potential unobserved ObjectDisposedException that could cause the host process to crash. (see: release notes)

Changes

Key Bugs Fixed

• Fixed an issue where long-lived credentials (more than 49 days) were overflowing refresh timer limits and being rejected.

• Added detection and recovery for a race condition that occurred when the Event Hubs service closed a connection or link after the client had validated its state and was performing an operation; this will now be properly retried with a fresh connection/link.

• Extended retry scenarios to include generic I/O exceptions, as they are typically transient network failures.

• Extended retry scenarios to include authorization failures, as the Event Hubs services do not differentiate between authentication and authorization, callers cannot reason about whether an Unauthorized return from an operation indicates that the call will never succeed or may trigger a credential renewal that may allow success.

Changes

Features Added

• Additional options for tuning load balancing have been added to the EventProcessorClientOptions.

• It is now possible to specify a custom endpoint to use for establishing the connection to the Event Hubs service in the EventHubConnectionOptions for the processor.

• Interactions with Blob Storage have been tuned for better performance and more efficient resource use. This will also improve start-up time, especially when using the Greedy load balancing strategy.

• Errors occurring in the Event Hubs service or active transport are now preserved in full and propagated as an inner exception; this will provide deeper context for diagnosing and troubleshooting exceptions.

• Documentation used for auto-completion via Intellisense and other tools has been enhanced in many areas, addressing gaps and commonly asked questions.

Key Bugs Fixed

• Upgraded the Microsoft.Azure.Amqp library to resolve crashes occurring in .NET 5.

• The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.

Acknowledgments

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Daniel Marbach (GitHub)

Changes

Key Bugs Fixed

• The approach used for creation of checkpoints has been updated to interact with Azure Blob storage more efficiently. This will yield major performance improvements when soft delete was enabled and minor improvements otherwise.

• The EventProcessorClient will now perform an eager validation of connection strings upon creation. Previously, validation was deferred until a partition was claimed which made debugging difficult.

• Fixed an issue where failure to create an AMQP link would lead to an AMQP session not being explicitly closed, causing connections to the Event Hubs service to remain open until a garbage collection pass was performed.

Features Added

• Load balancing will now detect when it has reached a balanced state more accurately; this will allow it to operate more efficiently when LoadBalancingStrategy.Greedy is in use.

• The EventProcessorClient now supports a configurable strategy for load balancing, allowing control over whether it claims ownership of partitions in a balanced manner (default) or more aggressively. The strategy may be set in the EventProcessorClientOptions when creating the processor. More details about strategies can be found in the associated documentation.

• The EventProcessorClientOptions now support setting a PrefetchCount and CacheEventCount for performance tuning. More details about each can be found in the associated documentation.

• Connection strings for each of the clients now supports a SharedAccessSignature token, allowing a pre-generated SAS to be used for authorization.

• Load balancing now has better recognition for being in a recovery state and will aggressively reclaim partitions for which it is the recognized owner, regardless of whether the current instance made the ownership claim. Previously, those partitions were redistributed on a 1-by-1 basis as part of the standard cycle.

Changes

General availability of preview features

• The set of features from v5.1.0-preview.1 are now generally available. This includes the EventProcessor<TPartition> and PartitionReceiver types which focus on advanced application scenarios which require greater low-level control.

Bug fixes and foundation

• Minor enhancements to reduce allocations and improve efficiency

Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:

• Christopher Scott (GitHub)

Changes

General

• A migration guide is now available for those moving from the 4.x version of the Microsoft.Azure.EventHubs libraries to the 5.0.1 version under the Azure.Messaging.EventHubs namespace.

Organization and naming

• Namespaces have been reorganized to align types to their functional area, reducing the number of types in the root namespace and offering better context for where a type is used. Cross-functional types have been left in the root while specialized types were moved to the Producer, Consumer, or Processor namespaces.

Event processing

• Load balancing has been tuned for better performance and lower resource use. (A community contribution, courtesy of christothes)

• Reduction of reliance on Azure resources for Event Processor tests. (A community contribution, courtesy of christothes)

• Logging has been implemented for Event Processor operations interacting with storage. (A community contribution, courtesy of christothes)

• Logging has been implemented for general Event Processing operations, including background execution.

• A bug with resuming from a storage checkpoint was fixed, ensuring that processing resumes from the next available event rather than reprocessing the event from which the checkpoint was created.

• The protected On[EventName] members have been marked private to reduce the public surface and reduce confusion. They provided no benefit over providing a handler and the cognative cost was not justified.

Code | Docs

Support: Active

Event Grid Namespaces

Azure.Messaging.EventGrid.Namespaces