Serilog.Formatting.Elasticsearch

What's Changed

• Add support for API Keys by @qgbo in #545
• Upgraded dependencies

Full Changelog: v9.0.1...v9.0.2

What's Changed

• fixes: v9.0.0 fails template creation with ES v7 with ElasticsearchCl… by @nenadvicentic in #533

Full Changelog: v9.0.0...v9.0.1

What's Changed

• Fix indentation by @Leh2 in #128
• [WIP] Added more extensive error handling by @mivano in #127
• correct 'nog' to 'not' by @ColoradoOrion in #141
• Get rid of 'deprecated' warnings in Elastic 5.6.2 by @VanHuychemG in #142
• Added logLevelSwitch option by @piaste in #139
• Add support for dynamic Pipeline by @Kahbazi in #136
• Fix field index option for 6.1+ template to use boolean value. by @jnus in #148
• Updated to elasticsearch 6 libraries by @EamonHetherton in #153
• Extend LoggerConfiguration.Elasticsearch with additional attributes by @letargis in #150
• Explain use of AutoRegisterTemplateVersion in the README by @piaste in #157
• Render message by default , Could disable MessageTemplate by @klettier in #160
• Expose interface-typed options via appsettings by @reedz in #162
• Merge dev to master by @mivano in #168
• New NuGet API Key by @merbla in #172
• feature: add option to skip sending oversized data to Elasticsearch by @MinhNguyenDev in #169
• Changed FailureSink from FileSink(deprecated) to RollingFileSink by @josoder in #190
• fixing repository url by @MaximRouiller in #192
• for ES v6 use "index_patterns" instead of "template" field by @CaringDev in #207
• only build packaging-related commits by @MV10 in #195
• Using durable seq by @pierresetteskog in #205
• Ship formatters in separate nuget package by @vhatsura in #208
• Notes about formatters in readme by @vhatsura in #209
• fix title of changelog by @tomap in #211
• add .netstandard2.0 target by @vhatsura in #222
• Update serilog nuget package to 2.8.0 version by @vhatsura in #225
• Added FormatStackTraceAsArray option to write StackTrace as an array by @oscarmorasu in #230
• Format stacktrace as array by @oscarmorasu in #233
• Delete ElasticSearch.Net dependency for formatter package by @vhatsura in #234
• Update NuGet API Key by @merbla in #241
• Elasticsearch.Net 7.0.0 support, V7 templates by @kanadaj in #249
• Feature/7.x support by @Mpdreamz in #256
• Fixes two problems, needing to use indexers on a dynamic, and a logic… by @matthewDDennis in #266
• [Urgent] Fix KeyNotFoundException when items[i].errors is undefined by @piaste in #271
• Updated README.md by @SirSeven in #279
• Fix log events not being written to Sink when exception has no stacktrace by @marcin-brzozowski in #285
• Fixed two issues regarding BufferCleanPayload by @daniel-oster in #295
• Adding aliases when creating index. by @jorjika in #290
• Added tests for 0 replicas by @mivano in #303
• Minor comment fix in ElasticsearchSinkOptions.cs by @connorads in #308
• ExceptionAsObjectJsonFormatter add renderMessageTemplate param by @deathgore in #304
• added TemplateCustomSettings as Dictionary<string,string> by @tresoldigiorgio in #311
• Merge dev to master to create new 8.1 release by @mivano in #330
• templateCustomSettings loaded from settings.json file by @tresoldigiorgio in #315
• Upgrade Elasticsearch.Net to 7.8.1 by @uvw in #343
• Update README.md by @chucklu in #344
• Fixed small spelling issue in issue template. by @SeppPenner in #352
• add async methods by @HassanHashemi in #349
• Fix HasProperty when settings is a Dictionary by @Evolva in #358
• Issue nr: 14594 - removing old implementation where fields where dot-… by @rajmondburgaj in #351
• use snupkg by @mivano in #360
• Added support for controlling the op_type when indexing by @orjan in #356
• Make sure the sink (and app) do not crash when the ES is unreachable … by @mivano in #359
• Corrected docs for batchAction config by @orjan in #363
• Override type name if specified by @orjan in #365
• Create v9.0.0 by @mivano in #509

New Contributors

• @Leh2 made their first contribution in #128
• @ColoradoOrion made their first contribution in #141
• @VanHuychemG made their first contribution in #142
• @piaste made their first contribution in #139
• @Kahbazi made their first contribution in #136
• @jnus made their first contribution in #148
• @EamonHetherton made their first contribution in #153
• @letargis made their first contribution in #150
• @reedz made their first contribution in #162
• @MinhNguyenDev made their first contribution in #169
• @josoder made their first contribution in #190
• @MaximRouiller made their first contribution in #192
• @CaringDev made their first contribution in #207
• @MV10 made their first contribution in #195
• @pierresetteskog made their first contribution in #205
• @vhatsura made their first contribution in #208
• @tomap made their first contribution in #211
• @oscarmorasu made their first contribution in #230
• @kanadaj made their first contribution in #249
• @matthewDDennis made their first contribution in #266
• @SirSeven made their first contribution in #279
• @marcin-brzozowski made their first contribution in #285
• @daniel-oster made their first contribution in #295
• @jorjika made their first contribution in #290
• @connorads made their first contribution in #308
• @deathgore made their first contribution in #304
• @tresoldigiorgio made their first contribution in #311
• @uvw made their first contribution in #343
• @chucklu made their first contribution in #344
• @SeppPenner made their first contribution in #352
• @HassanHashemi made their first contribution in #349
• @Evolva made their first contribution in #358
• @rajmondburgaj made their first contribution in #351
• @orjan made their first contribution in #356

Full Changelog: 5.4.0...v9.0.0

• Remove AssemblyInfo.cs and move attributes to the *.csproj file. … by @nenadvicentic in #501
• Read Elasticsearch server version from a root page response by @nenadvicentic in #502
• Bump actions/setup-dotnet from 2 to 3.0.2 by https://github.com/dependabot in #478
• Bump actions/download-artifact from 1 to 3 by https://github.com/dependabot in #439
• Bump actions/upload-artifact from 2 to 3 by https://github.com/dependabot in #438
• fix file references in the Visual Studio Solution file. by @nenadvicentic in #461
• Bump actions/setup-dotnet from 3.0.2 to 3.0.3 by https://github.com/dependabot in #487
• Bump github/codeql-action from 1 to 2 by https://github.com/dependabot in #426
• Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) by @nenadvicentic in #462
• Remove support for Elasticsearch v2 and v5. by @nenadvicentic in #488
• fix: Example in README is incorrect #402 by @nenadvicentic in #496

Full Changelog: v9.0.0-beta12...v9.0.0-rc1

What's Changed

• Bump actions/setup-dotnet from 2 to 3.0.2 by @dependabot in #478
• Bump actions/download-artifact from 1 to 3 by @dependabot in #439
• Bump actions/upload-artifact from 2 to 3 by @dependabot in #438
• fix file references in the Visual Studio Solution file. by @nenadvicentic in #461
• Bump actions/setup-dotnet from 3.0.2 to 3.0.3 by @dependabot in #487
• Bump github/codeql-action from 1 to 2 by @dependabot in #426
• Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) by @nenadvicentic in #462
• Remove support for Elasticsearch v2 and v5. by @nenadvicentic in #488
• fix: Example in README is incorrect #402 by @nenadvicentic in #496

New Contributors

• @nenadvicentic made their first contribution in #461

Full Changelog: v9.0.0-beta7...v9.0.0-beta8

What's Changed

• Remove AssemblyInfo.cs and move attributes to the *.csproj file. … by @nenadvicentic in #501
• Read Elasticsearch server version from a root page response by @nenadvicentic in #502
• Bump actions/setup-dotnet from 2 to 3.0.2 by https://github.com/dependabot in #478
• Bump actions/download-artifact from 1 to 3 by https://github.com/dependabot in #439
• Bump actions/upload-artifact from 2 to 3 by https://github.com/dependabot in #438
• fix file references in the Visual Studio Solution file. by @nenadvicentic in #461
• Bump actions/setup-dotnet from 3.0.2 to 3.0.3 by https://github.com/dependabot in #487
• Bump github/codeql-action from 1 to 2 by https://github.com/dependabot in #426
• Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) by @nenadvicentic in #462
• Remove support for Elasticsearch v2 and v5. by @nenadvicentic in #488
• fix: Example in README is incorrect #402 by @nenadvicentic in
  #496

New Contributors

@nenadvicentic made their first contribution in #461

Full Changelog: v9.0.0-beta10...v9.0.0-beta11

What's Changed

• Remove AssemblyInfo.cs and move attributes to the *.csproj file. … by @nenadvicentic in #501
• Read Elasticsearch server version from a root page response by @nenadvicentic in #502
• Bump actions/setup-dotnet from 2 to 3.0.2 by https://github.com/dependabot in #478
• Bump actions/download-artifact from 1 to 3 by https://github.com/dependabot in #439
• Bump actions/upload-artifact from 2 to 3 by https://github.com/dependabot in #438
• fix file references in the Visual Studio Solution file. by @nenadvicentic in #461
• Bump actions/setup-dotnet from 3.0.2 to 3.0.3 by https://github.com/dependabot in #487
• Bump github/codeql-action from 1 to 2 by https://github.com/dependabot in #426
• Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) by @nenadvicentic in #462
• Remove support for Elasticsearch v2 and v5. by @nenadvicentic in #488
• fix: Example in README is incorrect #402 by @nenadvicentic in
  #496

New Contributors

@nenadvicentic made their first contribution in #461

Full Changelog: v9.0.0-beta10...v9.0.0-beta11

What's Changed

• Bump actions/setup-dotnet from 2 to 3.0.2 by @dependabot in #478
• Bump actions/download-artifact from 1 to 3 by @dependabot in #439
• Bump actions/upload-artifact from 2 to 3 by @dependabot in #438
• fix file references in the Visual Studio Solution file. by @nenadvicentic in #461
• Bump actions/setup-dotnet from 3.0.2 to 3.0.3 by @dependabot in #487
• Bump github/codeql-action from 1 to 2 by @dependabot in #426
• Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) by @nenadvicentic in #462
• Remove support for Elasticsearch v2 and v5. by @nenadvicentic in #488
• fix: Example in README is incorrect #402 by @nenadvicentic in #496

New Contributors

• @nenadvicentic made their first contribution in #461

Full Changelog: v9.0.0-beta7...v9.0.0-beta8

Fixed

• Make sure TypeName is set to _doc when setting the template version to ESv7. The regression was introduced in 8.4.0. #364

Added

• Do not crash when ES is unreachable and the option DetectElasticsearchVersion is set to true. #359
• Create snupkg instead of the old style symbol files. #360
• Support for explicitly setting Options.TypeName to null this will remove the deprecated _type from the bulk payload being sent to Elastic. Earlier an exception was thrown if the Options.TypeName was null. _If you're using AutoRegisterTemplateVersion.ESv7 we'll not force the type to _doc if it's set to null. This is a small step towards support for writing logs to Elastic v8. #345
• Support for setting the Elastic op_type e.g. index or create for bulk actions. This is a requirement for writing to data streams that's only supporting create. Data streams is a more slipped stream way to handle rolling indices, that previous required an ILM, template and a magic write alias. Now it's more integrated in Elasticsearch and Kibana. If you're running Elastic 7.9 you'll get rolling indices out of the box with this configuration:

  TypeName = null,
  IndexFormat = "logs-my-stream",
  BatchAction = ElasticOpType.Create,
  

  Note: that current templates doesn't support data streams. #355

Removed

• Disable dot-escaping for field names, because ELK already supports dots in field names. #351

8.2

• Allow the use of templateCustomSettings when reading from settings json (#315)
• Updated Elasticsearch.Net dependency #340

8.1

• Updated sample to use .NET 2.1 and ES 7.5
• Change default TypeName to '_doc' #298

8.0

• Adds Elasticsearch 7.0 support #256
• Adds DetectElasticsearchVersion to the sink that will detect the running cluster version. Something we now use to support sending Esv6 templates to Elasticsearch 7.x and Esv7 templates to Elasticsearch 6.x which should simplify upgrades.
• Adds an integration test project. Spins up a 6.x and 7.x elasticsearch single node cluster in succession and asserts the default and the mixed mode through DetectElasticsearchVersion works.
• Dropped support for net45 and netstandard 1.3

7.1

• DurableElasticsearchSink is rewritten to use the same base code as the sink for Serilog.Sinks.Seq. Nuget Serilog.Sinks.File is now used instead of deprecated Serilog.Sinks.RollingFile. Lots of new fintuning options for file storage is added in ElasticsearchSinkOptions. Updated Serilog.Sinks.Elasticsearch.Sample.Main with SetupLoggerWithPersistantStorage with all available options for durable mode.
• Changed datatype on singleEventSizePostingLimit from int to long? with default value null. to make it possible ro reuse code from Sinks.Seq .
• IndexDecider didnt worked well in buffer mode because of LogEvent was null. Added BufferIndexDecider.
• Added BufferCleanPayload and an example which makes it possible to cleanup your invalid logging document if rejected from elastic because of inconsistent datatype on a field. It'seasy to miss errors in the self log now its possible to se logrows which is bad for elasticsearch in the elastic log.
• Added BufferRetainedInvalidPayloadsLimitBytes A soft limit for the number of bytes to use for storing failed requests.
• Added BufferFileCountLimit The maximum number of log files that will be retained.
• Formatting has been moved to seperate package.

6.4

• Render message by default (#160).
• Expose interface-typed options via appsettings (#162)

6.2

• Extra overload added to support more settings via AppSettings reader. (#150)

6.1

• Updated to elasticsearch 6 libraries (#153)
• Fix field index option for 6.1+ template to use boolean value. (#148)

5.7

• Supporting ES 6 template defitions while still supporting old versions. (https://github.com/serilog/serilog-sinks-elasticsearch/pull/142) See for details https://www.elastic.co/blog/strings-are-dead-long-live-strings
• Pipeline decider added.
• Ability to use loglevelswitch (https://github.com/serilog/serilog-sinks-elasticsearch/pull/139)

5.5

• Errors from Elasticsearch can now be handled. Either by looking into the selflog, sending the failty events to another sink, handle the logevent yourself by using a callback or let the sink throw an exception.
• BOM fix for buffered option.
• The creation of the template might fail. You can now specify what kind of action should be taken if this happens.
• Added a sample application.
• Added a docker-compose file that allows you to start a local elasticsearch and kibana instance.
• ConnectionTimeout is now set to be 5 seconds instead of 1 minute.
• You can now set the queueSizeLimit, which limits the amount of events stored in the PeriodicBatching buffer. Does not impact the durable buffer.

5.4

• Added support for pipelines in Elasticsearch. Pipelines allows you to change the ingress data by running it through Processors (https://www.elastic.co/blog/new-way-to-ingest-part-1).

5.3

• JSON project file converted to CSProj, references updated. PR #109

5.2

• Next to the number of shards, you can also set the number of replicas. This will only apply to newly created indices.

5.1

• You can specify the number of shards when creating the template mapping. This will only apply to newly created indices.

5.0

• To make the sink work in line with the other sinks, there is a breaking change as described in PR (https://github.com/serilog/serilog-sinks-elasticsearch/pull/94). minimumLogEventLevel is renamed to restrictedToMinimumLevel. The behaviour is now also consistent when you set the minimum level.

4.x

• BREAKING CHANGE This sink now uses Serilog 2.0. This is a breaking change, please use a version >=3.x of the sink if you want to use Serilog 1.x.

3.0.130

• Added an optional ExceptionAsObjectJsonFormatter to support serializing exceptions as a single object (not as an array).

3.0.128

• SpecificVersion set to False in order not to be dependent on a version of Elasticsearch or Serilog.

3.0.125

• Dropped support for .NET 4 since the Elasticsearch.NET client also does not support this version of the framework anymore.

3.0.121

• protected virtual ElasticsearchResponse<T> EmitBatchChecked<T>(IEnumerable<LogEvent> events) function now uses a generic type. This allows you to map to either DynamicResponse or to BulkResponse if you want to use NEST.

3.0.112

• Added exponential backoff strategy when unable to send data to Elasticsearch when using the durable sink option.

3.0.98

• Field names cannot contain a dot in ES 2, so they will get replaced by a / instead. See https://github.com/elastic/elasticsearch/issues/14594

3.x

• BREAKING CHANGE This sink now uses the Elasticsearch.Net 2.x library to be compatible with Elasticsearch version 2. This is a breaking change, use a 2.x version of the sink to support Elasticsearch 1.x versions.

2.0.49

• Fixed typo: ModifyConnectionSetttings to ModifyConnectionSettings.

2.0.42

• Added an overload so the AppSettings reader can be used to configure the ES sink.

2.0.38

• Fixes an issue where the index decider was not properly used with pusing events thorugh the ElasticLogShipper.

2.0.37

• When auto register of the template is enabled, but the ES server is unavailable, the exception is logged to the selflog instead of bubbling up the exception.
• omit_terms is set to true in the template.

2.0.0

• Moved the Elasticsearch sink from its original location