github.com/redpanda-data/benthos

Added

• New metric processor.

Added

• New redis_streams input and output.

Added

• New kinesis input and output.

Added

• The index field of the elasticsearch output can now be dynamically set using function interpolation.
• New hash processor.

Changed

• API: The metrics.Type interface has been changed in order to add labels.

Changed

• Significant restructuring of amqp inputs and outputs. These changes should be backwards compatible for existing pipelines, but changes the way in which queues, exchanges and bindings are declared using these types.

Added

• New durable fields for amqp input and output types.

Changed

• Improved statsd client with better cached aggregation.

Added

• New tls fields for amqp input and output types.

Added

• New type field for elasticsearch output.

Added

• New throttle processor.

Added

• New less_than and greater_than operators for metadata condition.

Added

• New metadata condition type.
• More metadata fields for kafka input.
• Field commit_period_ms for kafka and kafka_balanced inputs for specifying a commit period.

Added

• New retries field to s3 input, to cap the number of download attempts made on the same bucket item.
• Added metadata based mechanism to detect final message from a read_until input.
• Added field to split processor for specifying target batch sizes.

Added

• Metadata fields are now per message part within a batch.
• New metadata_json_object function interpolation to return a JSON object of metadata key/value pairs.

Changed

• The metadata function interpolation now allows part indexing and no longer returns a JSON object when no key is specified, this behaviour can now be done using the metadata_json_object function.

Added

• Fields for the http processor to enable parallel requests from message batches.

Changed

• Broker level output processors are now applied before the individual output processors.
• The dynamic input and output HTTP paths for CRUD operations are now /inputs/{input_id} and /outputs/{output_id} respectively.
• Removed deprecated amazon_s3, amazon_sqs and scalability_protocols input and output types.
• Removed deprecated json_fields field from the dedupe processor.

Added

• Add conditions to process_map processor.

Changed

• TLS config fields have been cleaned up for multiple types. This affects the kafka, kafka_balanced and http_client input and output types, as well as the http processor type.

Added

• New delete_all and delete_prefix operators for metadata processor.
• More metadata fields extracted from the AMQP input.
• HTTP clients now support function interpolation on the URL and header values, this includes the http_client input and output as well as the http processor.

Added

• New key field added to the dedupe processor, allowing you to deduplicate using function interpolation. This deprecates the json_paths array field.

Added

• New s3 and sqs input and output types, these replace the now deprecated amazon_s3 and amazon_sqs types respectively, which will eventually be removed.
• New nanomsg input and output types, these replace the now deprecated scalability_protocols types, which will eventually be removed.

Added

• Metadata fields are now collected from MQTT input.
• AMQP output writes all metadata as headers.
• AMQP output field key now supports function interpolation.

Added

• New metadata processor and configuration interpolation function.

Added

• New config interpolator function json_field for extracting parts of a JSON message into a config value.

Changed

• Log level config field no longer stutters, logger.log_level is now logger.level.

Added

• Ability to create batches via conditions on message payloads in the batch processor.
• New --examples flag for generating specific examples from Benthos.

Added

• New text processor.

Changed

• Processor process_map replaced field strict_premapping with premap_optional.

Added

• New process_field processor.
• New process_map processor.

Changed

• Removed mapping fields from the http processor, this behaviour has been put into the new process_map processor instead.

Changed

• Renamed content condition type to text in order to clarify its purpose.

Added

• Latency metrics for caches.
• TLS options for kafka and kafka_partitions inputs and outputs.

Changed

• Metrics for items configured within the resources section are now namespaced under their identifier.

Added

• New copy and move operators for the json processor.

Added

• Metrics for recording http request latencies.

Changed

• Improved and rearranged fields for http_client input and output.

Added

• More compression and decompression targets.
• New lines option for archive/unarchive processors.
• New encode and decode processors.
• New period_ms field for the batch processor.
• New clean operator for the json processor.

Added

• New http processor, where payloads can be sent to arbitrary HTTP endpoints and the result constructed into a new payload.
• New inproc inputs and outputs for linking streams together.

Added

• New streams endpoint /streams/{id}/stats for obtaining JSON metrics for a stream.

Changed

• Allow comma separated topics for kafka_balanced.

Added

• Support for PATCH verb on the streams mode /streams/{id} endpoint.

Changed

• Sweeping changes were made to the environment variable configuration file. This file is now auto generated along with its supporting document. This change will impact the docker image.

Added

• New filter_parts processor for filtering individual parts of a message batch.
• New field open_message for websocket input.

Changed

• No longer setting default input processor.

Added

• New root_path field for service wide http config.

Added

• New regexp_exact and regexp_partial content condition operators.