See the CHANGELOG for more details.

:star2: Features

• [breaking]: The db in file source is moved to the loggie.yml. If upgrading from an earlier version to v1.5, be sure to check whether db has been configured for file source. If it is not configured, you can just ignore it, and the default value will remain compatible.
• Added rocketmq sink (#530)
• Added franzKafka source (#573)
• Added kata runtime (#554)
• typePodFields/typeNodeFields is supported in LogConfig/ClusterLogConfig (#450)
• sink codec support printEvents (#448)
• Added queue in LogConfig/ClusterLogConfig (#457)
• Changed olivere/elastic to the official elasticsearch go client (#581)
• Supported copytruncate in file source (#571)
• Added genfiles sub command (#471)
• Added queue in LogConfig/ClusterLogConfig queue (#457)
• Added sortBy field in elasticsearch source (#473)
• Added host VM mode with Kubernetes as the configuration center (#449) (#489)
• New addHostMeta interceptor (#474)
• Added persistence driver badger (#475) (#584)
• Ignore LogConfig with sidecar injection annotation (#478)
• Added toStr action in transformer interceptor (#482)
• You can mount the root directory of a node to the Loggie container without mounting additional Loggie volumes (#460)
• Get loggie version with api and sub command (#496) (#508)
• Added the worker and the clientId in Kafka source (#506) (#507)
• Upgrade kafka-go version (#506) (#567)
• Added resultStatus in dev sink which can be used to simulate failure, drop (#531)
• Pretty error when unmarshal yaml configuration failed (#539)
• Added default topic if render kafka topic failed (#550)
• Added ignoreUnknownTopicOrPartition in kafka sink (#560)
• Supported multiple topics in kafka source (#548)
• Added default index if render elasticsearch index failed (#551) (#553)
• The default maxOpenFds is set to 4096 (#559)
• Supported default sinkRef in kubernetes discovery (#555)
• Added ${_k8s.clusterlogconfig} in typePodFields (#569)
• Supported omit empty fields in Kubernetes discovery (#570)
• Optimizes maxbytes interceptors (#575)
• Moved readFromTail, cleanFiles, fdHoldTimeoutWhenInactive, fdHoldTimeoutWhenRemove from watcher to outer layer in file source (#579) (#585)
• Added cheanUnfinished in cleanFiles (#580)
• Added target in maxbyte interceptor (#588)
• Added partionKey in franzKafka (#562)
• Added highPrecision in rateLimit interceptor (#525)
• Supported build core components of Loggie (#598)
• Added addonMetaSchema in file source (#599)
• Added timestamp and bodyKey in source (#600)
• Added discoverNodesOnStart and discoverNodesInterval in elasticsearch sink (#612)

:bug: Bug Fixes

• Fixed panic when kubeEvent Series is nil (#459)
• Upgraded automaxprocs version to v1.5.1 (#488)
• Fixed set defaults failed in fieldsUnderKey (#513)
• Fixed parse condition failed when contain ERROR in transformer interceptor (#514) (#515)
• Fixed grpc batch out-of-order data streams (#517)
• Fixed large line may cause oom (#529)
• Fixed duplicated batchSize in queue (#533)
• Fixed sqlite locked panic (#524)
• Fixed command can't be used in multi-arch container (#541)
• Fixed logger listener may cause block (#561) (#552)
• Fixed sink concurrency deepCopy failed (#563)
• Drop events when partial error in elasticsearch sink (#572)
• Fix convert to string type when returning byte array type in transformer interceptor (#596)
• Invalid pipelines will not stop all the pipelines running (#602)
• Fixed configuration npe when query pipelines (#604)
• Optimized elasticsearch sink request buffer (#608)
• Add pod namespace as prefix of source name when pod is matched by ClusterLogConfig (#613)

See the CHANGELOG for more details.

:star2: Features

• [breaking]: The db in file source is moved to the loggie.yml. If upgrading from an earlier version to v1.5, be sure to check whether db has been configured for file source. If it is not configured, you can just ignore it, and the default value will remain compatible.
• Added rocketmq sink (#530)
• Added franzKafka source (#573)
• Added kata runtime (#554)
• typePodFields/typeNodeFields is supported in LogConfig/ClusterLogConfig (#450)
• sink codec support printEvents (#448)
• Added queue in LogConfig/ClusterLogConfig (#457)
• Changed olivere/elastic to the official elasticsearch go client (#581)
• Supported copytruncate in file source (#571)
• Added genfiles sub command (#471)
• Added queue in LogConfig/ClusterLogConfig queue (#457)
• Added sortBy field in elasticsearch source (#473)
• Added host VM mode with Kubernetes as the configuration center (#449) (#489)
• New addHostMeta interceptor (#474)
• Added persistence driver badger (#475) (#584)
• Ignore LogConfig with sidecar injection annotation (#478)
• Added toStr action in transformer interceptor (#482)
• You can mount the root directory of a node to the Loggie container without mounting additional Loggie volumes (#460)
• Get loggie version with api and sub command (#496) (#508)
• Added the worker and the clientId in Kafka source (#506) (#507)
• Upgrade kafka-go version (#506) (#567)
• Added resultStatus in dev sink which can be used to simulate failure, drop (#531)
• Pretty error when unmarshal yaml configuration failed (#539)
• Added default topic if render kafka topic failed (#550)
• Added ignoreUnknownTopicOrPartition in kafka sink (#560)
• Supported multiple topics in kafka source (#548)
• Added default index if render elasticsearch index failed (#551) (#553)
• The default maxOpenFds is set to 4096 (#559)
• Supported default sinkRef in kubernetes discovery (#555)
• Added ${_k8s.clusterlogconfig} in typePodFields (#569)
• Supported omit empty fields in Kubernetes discovery (#570)
• Optimizes maxbytes interceptors (#575)
• Moved readFromTail, cleanFiles, fdHoldTimeoutWhenInactive, fdHoldTimeoutWhenRemove from watcher to outer layer in file source (#579) (#585)
• Added cheanUnfinished in cleanFiles (#580)
• Added target in maxbyte interceptor (#588)
• Added partionKey in franzKafka (#562)
• Added highPrecision in rateLimit interceptor (#525)

:bug: Bug Fixes

• Fixed panic when kubeEvent Series is nil (#459)
• Upgraded automaxprocs version to v1.5.1 (#488)
• Fixed set defaults failed in fieldsUnderKey (#513)
• Fixed parse condition failed when contain ERROR in transformer interceptor (#514) (#515)
• Fixed grpc batch out-of-order data streams (#517)
• Fixed large line may cause oom (#529)
• Fixed duplicated batchSize in queue (#533)
• Fixed sqlite locked panic (#524)
• Fixed command can't be used in multi-arch container (#541)
• Fixed logger listener may cause block (#561) (#552)
• Fixed sink concurrency deepCopy failed (#563)
• Drop events when partial error in elasticsearch sink (#572)

See the CHANGELOG for more details.

See the CHANGELOG for more details.

:star2: Features

• Added Loggie dashboard feature for easier troubleshooting (#416)
• Enhanced log alerting function with more flexible log alert detection rules and added alertWebhook sink (#392)
• Added sink concurrency support for automatic adaptation based on downstream delay (#376)
• Added franzKafka sink for users who prefer the franz kafka library (#423)
• Added elasticsearch source (#345)
• Added zinc sink (#254)
• Added pulsar sink (#417)
• Added grok action to transformer interceptor (#418)
• Added split action to transformer interceptor (#411)
• Added jsonEncode action to transformer interceptor (#421)
• Added fieldsFromPath configuration to source for obtaining fields from file content (#401)
• Added fieldsRef parameter to filesource listener for obtaining key value from fields configuration and adding to metrics as label (#402)
• In transformer interceptor, added dropIfError support to drop event if action execution fails (#409)
• Added info listener which currently exposes loggie_info_stat metrics and displays version label (#410)
• Added support for customized kafka sink partition key
• Added sasl support to Kafka source (#415)
• Added https insecureSkipVerify support to loki sink (#422)
• Optimized file source for large files (#430)
• Changed default value of file source maxOpenFds to 1024 (#437)
• ContainerRuntime can now be set to none (#439)
• Upgraded to go 1.18 (#440)
• Optimize the configuration parameters to remove the redundancy generated by rendering

:bug: Bug Fixes

• Added source fields to filesource listener (#402)
• Fixed issue of transformer copy action not copying non-string body (#420)
• Added fetching of logs file from UpperDir when rootfs collection is enabled (#414)
• Fix pipeline restart npe (#454)
• Fix create dir soft link job (#453)

See the CHANGELOG for more details.

:star2: Features

• Added Loggie dashboard feature for easier troubleshooting (#416)
• Enhanced log alerting function with more flexible log alert detection rules and added alertWebhook sink (#392)
• Added sink concurrency support for automatic adaptation based on downstream delay (#376)
• Added franzKafka sink for users who prefer the franz kafka library (#423)
• Added elasticsearch source (#345)
• Added zinc sink (#254)
• Added pulsar sink (#417)
• Added grok action to transformer interceptor (#418)
• Added split action to transformer interceptor (#411)
• Added jsonEncode action to transformer interceptor (#421)
• Added fieldsFromPath configuration to source for obtaining fields from file content (#401)
• Added fieldsRef parameter to filesource listener for obtaining key value from fields configuration and adding to metrics as label (#402)
• In transformer interceptor, added dropIfError support to drop event if action execution fails (#409)
• Added info listener which currently exposes loggie_info_stat metrics and displays version label (#410)
• Added support for customized kafka sink partition key
• Added sasl support to Kafka source (#415)
• Added https insecureSkipVerify support to loki sink (#422)
• Optimized file source for large files (#430)
• Changed default value of file source maxOpenFds to 1024 (#437)
• ContainerRuntime can now be set to none (#439)
• Upgraded to go 1.18 (#440)

:bug: Bug Fixes

• Added source fields to filesource listener (#402)
• Fixed issue of transformer copy action not copying non-string body (#420)
• Added fetching of logs file from UpperDir when rootfs collection is enabled (#414)

See the CHANGELOG for more details.

See the CHANGELOG for more details.

:star2: Features

• Add transformer interceptor (#267)
• Add api /api/v1/helper for debugging (#291)
• Monitoring listener for normalize interceptor (#224)
• Add typeNodeFields in kubernetes discovery, change k8sFields to typePodFields, add pod.uid, pod.container.id, pod.container.image to typePodFields (#348)
• File source support any lineDelimiter (#261)
• Logconfig/Clusterlogconfig labelSelector support wildcard * (#269)
• Add override annotation in logConfig/clusterLogConfig (#269)
• Add go test in makefile and github action (#282)
• Support total events in dev source (#307)
• Make file sink default file permissions from 600 to 644 (#304)
• Change goccy/go-yaml to gopkg.in/yaml.v2 (#317)
• Add schema interceptor (#340)
• Support collect log files from pod pv (#341)
• Support dynamic container logs (#334)
• Put the encoding conversion module in the source (#344)
• Support [a.b] when fields key is a.b other than a nested fields #301 #297
• Loki sink will replace fields including string '. / -' to '_' (#347)

:bug: Bug Fixes

• Set beatsFormat timestamp to UTC in sink codec (#268)
• Fix pipeline stop block (#280)(#284) (#289) (#305)(#309) (#335)
• Fix addonMeta panic (#285)
• Move private fields from header to meta in grpc source (#286)
• Fix clusterlogconfig npe when deleting (#306)
• Fix prometheus export topic (#314)
• Fix watchTask stop panic; fix big line collect block (#315)
• Fix type assert panic when deleting pod recieve DeletedFinalStateUnknown objects (#323)
• Fix maxbytes order (#326)
• Check if pod is ready when handle logconfig events (#328)
• Stop pipeline when config is empty (#330)
• Change leaderElection Identity in kubeEvent source (#349)
• Fix ignoreOlder missing in file source config
• Flush clusterlogconfig configuration when sink/interceptor is being updated
• Fix the problem of missing stream and time when collecting stdout logs (#363)
• Fix kubeEvent source when lost leader connection (#361)
• Do not ignore pod events when container id is changed (#377)
• Validate multiline pattern (#381)
• LabelSelector in LogConfig can be updated now (#369)
• Fix interceptor order in pipelines (#387)

Release v1.3.0-rc.0

🌟 Features

• Add transformer interceptor (#267)
• Add api /api/v1/helper for debugging (#291)
• Monitoring listener for normalize interceptor (#224)
• Add typeNodeFields in kubernetes discovery, change k8sFields to typePodFields, add pod.uid, pod.container.id, pod.container.image to typePodFields (#348)
• File source support any lineDelimiter (#261)
• Logconfig/Clusterlogconfig labelSelector support wildcard * (#269)
• Add override annotation in logConfig/clusterLogConfig (#269)
• Add go test in makefile and github action (#282)
• Support total events in dev source (#307)
• Make file sink default file permissions from 600 to 644 (#304)
• Change goccy/go-yaml to gopkg.in/yaml.v2 (#317)
• Add schema interceptor (#340)
• Support collect log files from pod pv (#341)
• Support dynamic container logs (#334)
• Put the encoding conversion module in the source (#344)
• Support [a.b] when fields key is a.b other than a nested fields (#301) (#297)
• Loki sink will replace fields including string '. / -' to '_' (#347)

🐛 Bug Fixes

• Set beatsFormat timestamp to UTC in sink codec (#268)
• Fix pipeline stop block (#280) (#284) (#289) (#305) (#309) (#335)
• Fix addonMeta panic (#285)
• Move private fields from header to meta in grpc source (#286)
• Fix clusterlogconfig npe when deleting (#306)
• Fix prometheus export topic (#314)
• Fix watchTask stop panic; fix big line collect block (#315)
• Fix type assert panic when deleting pod recieve DeletedFinalStateUnknown objects (#323)
• Fix maxbytes order (#326)
• Check if pod is ready when handle logconfig events (#328)
• Stop pipeline when config is empty (#330)
• Change leaderElection Identity in kubeEvent source (#349)

:star2: Features

• Add transformer interceptor (#267)
• Add api /api/v1/helper for debugging (#291)
• Monitoring listener for normalize interceptor (#224)
• Add typeNodeFields in kubernetes discovery, change k8sFields to typePodFields, add pod.uid, pod.container.id, pod.container.image to typePodFields (#348)
• File source support any lineDelimiter (#261)
• Logconfig/Clusterlogconfig labelSelector support wildcard * (#269)
• Add override annotation in logConfig/clusterLogConfig (#269)
• Add go test in makefile and github action (#282)
• Support total events in dev source (#307)
• Make file sink default file permissions from 600 to 644 (#304)
• Change goccy/go-yaml to gopkg.in/yaml.v2 (#317)
• Add schema interceptor (#340)
• Support collect log files from pod pv (#341)
• Support dynamic container logs (#334)
• Put the encoding conversion module in the source (#344)
• Support [a.b] when fields key is a.b other than a nested fields #301 #297
• Loki sink will replace fields including string '. / -' to '_' (#347)

:bug: Bug Fixes

• Set beatsFormat timestamp to UTC in sink codec (#268)
• Fix pipeline stop block (#280)(#284) (#289) (#305)(#309) (#335)
• Fix addonMeta panic (#285)
• Move private fields from header to meta in grpc source (#286)
• Fix clusterlogconfig npe when deleting (#306)
• Fix prometheus export topic (#314)
• Fix watchTask stop panic; fix big line collect block (#315)
• Fix type assert panic when deleting pod recieve DeletedFinalStateUnknown objects (#323)
• Fix maxbytes order (#326)
• Check if pod is ready when handle logconfig events (#328)
• Stop pipeline when config is empty (#330)
• Change leaderElection Identity in kubeEvent source (#349)

Feat: support collect log files from pod pv (#341)

fix pipeline stop block (#280)

:star2: Features

• Support container root filesystem log collection, so Loggie could collect the log files in container even if the pod is not mounting any volumes. #209
• Add sls sink for aliyun. #172
• Support specifying more time intervals(like 1d, 1w, 1y) for the "ignoreOlder" config of file source. #153 (by @iamyeka)
• Add ignoreError in normalize interceptor. #205
• Add system listener, which can export CPU/Memory metrics of Loggie. #88
• Set log.jsonFormat default to false, so Loggie would no longer print json format logs in default. #186
• Add callerSkipCount and noColor logger flag. #185
• file sink support codec. #215 (by @lyp256)
• Support linux arm64. #229 (by @ziyu-zhao)
• addk8smeta support adding workload metadata. #221 (by @lyp256)
• Support extra labels in prometheusExporter source. #233
• Logconfig matchFields labels, annotations and env support wildcards(*). #235 (by @LeiZhang-Hunter)
• Add source codec for file source. #239
• Some specific fields pattern now supports ${_env.XX}, ${+date}, ${object}, ${_k8s.XX}. #248
• We support rename body of events in normalize interceptor now. #238
• Use local time format in pattern fields. #234
• Refactor reader process chain. #166
• Optimize merge strategy of default interceptors. #174 (by @iamyeka)
• Merge defaults configurations support recursion. #212
• Cache env for optimizing performance when using fieldsFromEnv. #178 (by @iamyeka)
• Update yaml lib to goccy/go-yaml. #242
• Oomit empty pipeline configurations. #244

:bug: Bug Fixes

• Fix json error with go 1.18. #164
• Fix list logconfigs by namespace when handling pod events. #173
• Fix watcher stop timeout in file source. #188
• Fix MultiFileWriter in file sink. #202 (by @lyp256)
• Fix pipeline stop may block. #207
• Fix unstable interceptor sort. #216
• Fix grpc source ack panic. #218
• Fix sub path in kubernetes discovery. #236 (by @yuzhichang)

:star2: Features

• Loki sink supported, so you can send messages to Loki now. #119

  eg:

  sink:
    type: loki
    url: "http://localhost:3100/loki/api/v1/push"
  

• Add spec.interceptors and spec.sink fields in ClusterLogConfig/LogConfig CRD #120

  eg:

  apiVersion: loggie.io/v1beta1
  kind: LogConfig
  metadata:
    name: tomcat
    namespace: default
  spec:
    selector:
      type: pod
      labelSelector:
        app: tomcat
    pipeline:
      sources: |
        - type: file
          name: common
          paths:
            - stdout
      sink: |
        type: dev
        printEvents: false
      interceptors: |
        - type: rateLimit
          qps: 90000
  

• A new codec type raw is added. #137

  eg:

      sink:
        type: dev
        codec:
          type: raw
  

• A new approach to external configurations, -config.from=env, allows Loggie to read configs from env. #132 (by @chaunceyjiang)

• Normalize interceptor now support fmt processor. This can reformat the fields by patterns. #141

  eg:

  interceptors:
  - type: normalize
    processors:
    - fmt:
        fields:
          d: new-${a.b}-${c}
  

• Adding enabled fields in source/interceptor. #140

• We would flush the pipelines.yml immediately when updating sink/interceptor CR. #139

• Upgraded client-go version to 0.22.2 #99 (by @fengshunli)

• sniff are disabled in elasticsearch sink by default. #138

• Updated kubeEvent source #144

  • leader selection support.

  • blacklist of namespaces support.

  • adding watchLatestEvents fields, allows Loggie watch the lastest events from K8s.

  eg:

        - type: kubeEvent
          name: eventer
          kubeconfig: ～/.kube/config
          watchLatestEvents: true
          blackListNamespaces: ["default"]
  

• Brace Expansion：{alt1,...} suppport, matches a sequence of characters if one of the comma-separated alternatives matches.

  eg:

  log files following:

  1. /tmp/loggie/service/order/access.log
  2. /tmp/loggie/service/order/access.log.2022-04-11
  3. /tmp/loggie/service/pay/access.log
  4. /tmp/loggie/service/pay/access.log.2022-04-11

  then the path field in filesource could be: /tmp/loggie/**/access.log{,.[2-9][0-9][0-9][0-9]-[01][0-9]-[0123][0-9]}

• Adding local in normalize interceptor, in case we want to parse timestamp with local time.

• We support a simple way to add file state when collecting log files, enable addonMeta, then Loggie will add more metadata of files in events.

  eg:

  {
    "body": "this is test",
    "state": {
      "pipeline": "local",
      "source": "demo",
      "filename": "/var/log/a.log",
      "timestamp": "2006-01-02T15:04:05.000Z",
      "offset": 1024,
      "bytes": 4096,
      "hostname": "node-1"
    }
  }
  

• Adding pod.ip和node.ip in K8s discovery.

• Customized documentId supported in elasticsearch sink.

:bug: Bug Fixes

• Fix read chan size error in filesource #102
• Update default value of maxAcceptedBytes in kafka source which may cause panic in component starting. #112

First Loggie Release! :sparkler:

:star2: Features

add components:

• sources:

  • file
  • kafka
  • grpc
  • unix
  • kubeEvent
  • prometheusExporter
  • dev

• interceptors:

  • normalize
  • rateLimit
  • addK8sMeta
  • logAlert
  • metric
  • retry
  • maxbytes

• sink:

  • elasticsearch
  • kafka
  • grpc
  • file
  • dev

• discovery:

  • Kubernetes discovery, introduced CRD: ClusterLogConfig/LogConfig/Interceptor/Sink

• monitor:

  • filesource
  • filewatcher
  • logAlert
  • queue
  • sink
  • reload

:bug: Bug Fixes