• use cgroup limit for maxproc if no cpu count is set

• update changelog and dependencies

• add note that -n 0 respects cgroup limits

update consumer.file docs

This is a critical patch release.

It fixes various problems with producer.Spooling that prevented it from working at all.
The issues were most likely introduced during the transition from 0.4.x to 0.5.x.

This version has been built with go 1.12.3

Fixed with 0.5.4

• producer.spooling is now functional again as messages were not written correctly since 0.5.0 (#248).
• producer.spooling now does not block upon shutdown (#248).
• metadata is now handled correctly when messages are sent to fallback (#247).
• producer.socket now sends messages directly to fallback if connect fails.

• producer.spooling is now functional again as messages were not written correctly since 0.5.0 (#248).
• producer.spooling now does not block upon shutdown (#248).
• metadata is now handled correctly when messages are sent to fallback (#247).
• producer.socket now sends messages directly to fallback if connect fails.

0.5.3

This is a patch / minor features release.

0.5.4

Compare

Choose a tag to compare

• producer.spooling is now functional again as messages were not written correctly since 0.5.0 (#248).
• producer.spooling now does not block upon shutdown (#248).
• metadata is now handled correctly when messages are sent to fallback (#247).
• producer.socket now sends messages directly to fallback if connect fails.

0.5.3

This is a patch / minor features release.

This is a critical patch release.

It fixes a GC crash caused by the message payload memory handler.
If your plugins use core. MessageDataPool.get(size) please replace it with make([]byte, size).

The buffer causing this was introduced with 0.5.0 but the bug seems to occur only when building with go 1.10.
We decided to remove the buffer as it's allocation speed improvements showed to be only minor anyways.

• Fixed a GC panic/crash caused by tgo.ByteBuffer.

0.5.2

This is a patch / minor features release.

0.5.3

Compare

Choose a tag to compare

This is a patch / minor features release.

All binaries have been compiled with go 1.10.1.

New with 0.5.2

• The version number is now generated via make and git. This will properly identify versions between releases.
• New producer.AwsCloudwatchLogs. Thanks to @luqasz
• The makefile has been cleaned up and go meta-linter support has been added

Fixed with 0.5.2

• consumer.Kafka now properly commits the consumer offsets to kafka. Thanks to @crewton
• producer.awsKinesis failed to produce records under certain conditions
• The consumer.Kafka folderPermissions property is now correctly applied
• formt.ExtractJSON trimValues property is now correctly applied
• The gollum binary inside the Dockerfile is built on the same baseimage as deployed
• Filter will now always filter out the MODIFIED message, not the original. This behavior is more "expected".

• The version number is now generated via make and git. This will properly identify versions between releases.
• New producer.AwsCloudwatchLogs. Thanks to @luqasz
• The makefile has been cleaned up and go meta-linter support has been added

v0.5.2

Compare

Choose a tag to compare

This is a patch / minor features release.

All vendor dependencies have been updated to the latest version and binaries have been compiled with go 1.9.3.

New with 0.5.1

• format.MetadataCopy has been updated to support free copying between metadata and payload
• producer.ElasticSearch alles setting the format of timeBasedIndex
• format.GrokToJSON has new options: RemoveEmptyValues, NamedCapturesOnly and SkipDefaultPatterns
• using dep for dependencies instead of glide

Fixed with 0.5.1

• fixed inversion of -lc always
• fixed a nil pointer panic with producer.elasticsearch when receiving messages with unassigned streams
• producer.ElasticSearch settings are now named according to config
• producer.ElasticSearch dayBasedIndex renamed to timeBasedIndex and it's now working as expected
• updated dependencies to latest version (brings support for kafka 1.0, fixes user agent parsing for format.processTSV)

• format.MetadataCopy has been updated to support free copying between metadata and payload
• producer.ElasticSearch alles setting the format of timeBasedIndex
• format.GrokToJSON has new options: RemoveEmptyValues, NamedCapturesOnly and SkipDefaultPatterns
• Using dep for dependencies instead of glide

v0.5.1

Compare

Choose a tag to compare

This is a major release with a lot of breaking changes.
The configuration format has changed, so you HAVE to change your configuration files.
A guide can be found in the detailed release notes.

All binaries for this release have been compiled with Go 1.9.2

New features

• Filters and Formatters have been merged into one list
• You can now use a filter or formatter more than once in the same plugin
• Consumers can now do filtering and formatting, too
• Messages can now store metadata. Formatters can affect the payload or a metadata field
• All plugins now have an automatic log scope
• Message payloads are now backed by a memory pool
• Messages now store the original message, i.e. a backup of the payload state after consumer processing
• Gollum now provides per-stream metrics
• Plugins are now able to implement health checks that can be queried via http
• There is a new pseudo plugin type "Aggregate" that can be used to share configuration between multiple plugins
• New base types for producers: Direct, Buffered, Batched
• Plugin configurations now support nested structures
• The configuration process has been simplified a lot by adding automatic error handling and struct tags
• Added a new formatter format.GrokToJSON
• Added a new formatter format.JSONToInflux10
• Added a new formatter format.Double
• Added a new formatter format.MetadataCopy
• Added a new formatter format.Trim
• Consumer.File now supports filesystem events
• Consumers can now define the number of go routines used for formatting/filtering
• All AWS plugins now support role switching
• All AWS plugins are now based on the same credentials code

Bugfixes

• The plugin lifecycle has been reimplemented to avoid gollum being stuck waiting for plugins to change state
• Any errors during the configuration phase will cause gollum to exit
• Integration test suite added
• Producer.HTTPRequest port handling fixed
• The test-config command will now produce more meaningful results
• Duplicating messages now properly duplicates the whole message and not just the struct
• Several race conditions have been fixed
• Producer.ElasticSearch is now based on a more up-to-date library
• Producer.AwsS3 is now behaving more like producer.File
• Gollum metrics can now bind to a specific address instead of just a port

Breaking changes

• The config format has changed to improve automatic processing
• A lot of plugins have been renamed to avoid confusion and to better reflect their behavior
• A lot of plugins parameters have been renamed
• The instances plugin parameter has been removed
• Most of gollum's metrics have been renamed
• Plugin base types have been renamed
• All message handling function signatures have changed to use pointers
• All formatters don't daisy chain anymore as they can now be listed in proper order
• Stream plugins have been renamed to Router plugins
• Routers are not allowed to modify message content anymore
• filter.All and format.Forward have been removed as they are not required anymore
• Producer formatter listss dedicated to format a key or similar constructs have been removed
• Logging framework switched to logrus
• The package gollum.shared has been removed in favor of trivago.tgo
• Fuses have been removed from all plugins
• The general message sequence number has been removed
• The term "drop" has been replaced by the term "fallback" to emphasise it's use
• The _DROPPED_ stream has been removed. Messages are discarded if no fallback is set
• Formatters can still the stream of a message but cannot trigger routing by themselves
• Compiling contrib plugins now requires a specific loader.go to be added
• The docker file on docker hub is now a lot smaller and only contains the gollum binary

• Filters and Formatters have been merged into one list
• You can now use a filter or formatter more than once in the same plugin
• Consumers can now do filtering and formatting, too
• Messages can now store metadata. Formatters can affect the payload or a metadata field
• All plugins now have an automatic log scope
• Message payloads are now backed by a memory pool
• Messages now store the original message, i.e. a backup of the payload state after consumer processing
• Gollum now provides per-stream metrics
• Plugins are now able to implement health checks that can be queried via http
• New base types for producers: Direct, Buffered, Batched
• Plugin configurations now support nested structures
• The configuration process has been simplified a lot by adding automatic error handling and struct tags
• All plugin configuration keys are now case insensitive
• Added a new formatter format.GrokToJSON
• Added a new formatter format.JSONToInflux10
• Added a new formatter format.Double
• Added a new formatter format.MetadataCopy
• Added a new formatter format.Trim
• Consumer.File now supports filesystem events
• Consumers can now define the number of go routines used for formatting/filtering
• All AWS plugins now support role switching
• All AWS plugins are now based on the same credentials code

v0.5.0

Compare

Choose a tag to compare

This is a pre-release of v0.5.0

All vendor dependencies have been updated to the latest version and binaries have been compiled with go 1.9-beta2.

The current release notes can be found under readthedocs.io

Changes since 0.5.0-rc3

• fix format.MetadataCopy to actually copy the payload and not reference it

v0.5.0-rc3.1 Pre-release

Pre-release

Compare

Choose a tag to compare

v0.5.0-rc3 Pre-release

Pre-release

Compare

Choose a tag to compare

v0.5.0-rc2 Pre-release

Pre-release

Compare

Choose a tag to compare

v0.5.0-rc1 Pre-release

Pre-release

Compare

Choose a tag to compare

• producer.Kafka will discard messages returned as "too large" to avoid spooling
• consumer.HTTP does not truncate messages with WithHeaders:false anymore (thanks @mhils)
• producer.Websocket now uses gorilla websockets (thanks @glaslos)
• Dockerfile is now working again
• It is now possible to (optionally) send nil messages with producer.kafka again
• Consumer.Kinesis will renew the iterator object when hitting a timeout
• Consumer.Kinesis now runs with an offset file set that does not exist
• Consumer.Kinesis offset file is now written less often (after each batch)
• Consumer.Kafka does now retry with an "oldest" offset after encountering an OutOfRange exception.
• Fixed a crash when using producer.ElasticSearch with date based indexes (thanks @relud)
• format.Base64Decode now uses data from previous formatters as intended
• format.JSON arr and obj will now auto create a key if necessary
• format.JSON now checks for valid state references upon startup
• format.JSON now properly encodes strings when using "enc"
• format.SplitToJSON may now keep JSON payload and is better at escaping string
• "gollum -tc" will exit with error code 1 upon error
• "gollum -tc" will now properly display errors during config checking

• Fixed file offset handling in consumer.Kinesis (thanks @relud)
• Fixed producer.File RotatePruneAfterHours setting
• Producer.File symlink switch is now atomic
• Fixed panic in producer.Redis when Formatter was not set
• Fixed producer.Spooling being stuck for a long time during shutdown
• Fixed native.KafkaProducer to map all topics to "default" if no topic mapping was set
• Fixed a concurrent map write during initialization in native.KafkaProducer
• Fixed consumer.Kafka OffsetFile setting stopping gollum when the offset file was not present
• consumer.Kafka will retry to connect to a not (yet) existing topic every PersistTimeoutMs
• Consumer.Kafka now tries to connect every ServerTimeoutSec if initial connect fails
• Consumer.Kafka MessageBufferCount default value increased to 8192
• Producer.Kafka and native.KafkaProducer now discard messages with 0-Byte content
• Producer.Kafka SendRetries set to 1 by default to circumvent a reconnect issue within sarama
• Fixed panic in producer.Kafka when shutting down
• Added manual heartbeat to check a valid broker connection with producer.Kafka
• Format.Base64Encode now returns the original message if decoding failed
• socket.producer TCP can be used without ACK
• Consumer.Syslogd message handling differences between RFC3164 and RFC5424 / RFC6587 fixed

• Fixed several race conditions reported by Go 1.6 and go build -race
• Fixed the scribe producer to drop unformatted message in case of error
• Fixed file.consumer rotation to work on regular files, too
• Fixed file.consumer rotation to reset the offset file after a SIGHUP
• Dockerfiles updated
• Producer.Kafka now sends messages directly to avoid sarama performance bottlenecks
• consumer.Kafka offset file is properly read on startup if DefaultOffset "oldest" or "newest" is
• Exisiting unix domain socket detection changed to use create instead of stat (better error handling)
• Kafka and Scribe specific metrics are now updated if there are no messages, too
• Scribe producer is now reacting better to server connection errors
• Filters and Formatters are now covered with unittests

• consumer.SysLogD now has more meaningful errormessages
• consumer.File now properly supports file rotation if the file to read is a symlink
• Scribe and Kafka metrics are now only updated upon successful send
• Fixed an out of bounds panic when producer.File was rotating logfiles without an extension
• Compression of files after rotation by produer.File now works (again)
• producer.Kafka now only reconnects if all topics report an error
• producer.Spool now properly respools long messages
• producer.Spool will not delete a file if a message in it could not be processed
• producer.Spool will try to automatically respool files after a restart
• producer.Spool will rotate non-empty files even if no new messages come in
• producer.Spool will recreate folders when removed during runtime
• producer.Spool will drop messages if rotation failes (not reroute)
• Messages that are spooled twice now retain their original stream
• Better handling of situations where Sarama (Kafka) writes become blocking
• Plugins now start up as "initializing" not as "dead" preventing dropped messages during startup

• InfluxDB JSON and line protocol fixed
• shared.WaitGroup.WaitFor with duration 0 falls back to shared.WaitGroup.Wait
• proper io.EOF handling for shared.BufferedReader and shared.ByteStream
• HTTP consumer now responds with 200 instead of 203
• HTTP consumer properly handles EOF
• Increased test coverage

• shared.RuntimeType renamed to TypeRegistry
• core.StreamTypes renamed to StreamRegistry
• ?ControlLoop callback parameters for command handling moved to callback members
• ?ControlLoop renamed to ?Loop, where ? can be a combination of Control (handling of control messages), Message (handling of messages) or Ticker (handling of regular callbacks)
• PluginControlStop is now splitted into PluginControlStopConsumer and PluginControlStopProducer to allow plugins that are producer and consumers.
• Producer.Enqueue now takes care of dropping messages and accepts a timeout overwrite value
• MessageBatch has been refactored to store messages instead of preformatted strings. This allows dropping messages from a batch.
• Message.Drop has been removed, Message.Route can be used instead
• The LoopBack consumer has been removed. Producers can now drop messages to any stream using DropToStream.
• Stream plugins are now allowed to only bind to one stream
• Renamed producer.HttpReq to producer.HTTPRequest
• Renamed format.StreamMod to format.StreamRoute
• For format.Envelope postfix and prefix configuration keys have been renamed to EnvelopePostifx and EnvelopePrefix
• Base64Encode and Base64Decode formatter parameters have been renamed to "Base64*"
• Removed the MessagesPerSecAvg metric
• Two functions were added to the MessageSource interface to allow blocked/active state query
• The low resolution timer has been removed