github.com/jaegertracing/jaeger

Backend Changes

⛔ Breaking Changes

List of service operations can be classified by span kinds (#1943, #1942, #1937, @guo0693)

• Endpoint changes:
  • Both Http & gRPC servers now take new optional parameter spanKind in addition to service. When spanKind is absent or empty, operations from all kinds of spans will be returned.
  • Instead of returning a list of string, both Http & gRPC servers return a list of operation struct. Please update your client code to process the new response. Example response:

    curl 'http://localhost:6686/api/operations?service=UserService&spanKind=server' | jq
    {
        "data": [{
            "name": "UserService::getExtendedUser",
            "spanKind": "server"
        },
        {
            "name": "UserService::getUserProfile",
            "spanKind": "server"
        }],
        "total": 2,
        "limit": 0,
        "offset": 0,
        "errors": null
    }
    

  • The legacy http endpoint stay untouched:

    /services/{%s}/operations
    

• Storage plugin changes:
  • Memory updated to support spanKind on write & read, no migration is required.
  • Badger & ElasticSearch to be implemented: For now spanKind will be set as empty string during read & write, only name will be valid operation name.
  • Cassandra updated to support spanKind on write & read (#1937, @guo0693): If you don't run the migration script, nothing will break, the system will use the old table operation_names and set empty spanKind in the response. Steps to get the updated functionality:
    1. You will need to run the command below on the host where you can use cqlsh to connect to Cassandra:

       KEYSPACE=jaeger_v1 CQL_CMD='cqlsh host 9042 -u test_user -p test_password --request-timeout=3000'
       bash ./v002tov003.sh
       

       The script will create new table operation_names_v2 and migrate data from the old table. spanKind column will be empty for those data. At the end, it will ask you whether you want to drop the old table or not.
    2. Restart ingester & query services so that they begin to use the new table

Trace and Span IDs are always padded to 32 or 16 hex characters with leading zeros (#1956, @yurishkuro)

Previously, Jaeger backend always rendered trace and span IDs as the shortest possible hex string, e.g. an ID with numeric value 255 would be rendered as a string ff. This change makes the IDs to always render as 16 or 32 characters long hex string, e.g. the same id=255 would render as 00000000000000ff. It mostly affects how UI displays the IDs, the URLs, and the JSON returned from jaeger-query service.

Motivation: Among randomly generated and uniformly distributed trace IDs, only 1/16th of them start with 0 followed by a significant digit, 1/256th start with two 0s, and so on in decreasing geometric progression. Therefore, trimming the leading 0s is a very modest optimization on the size of the data being transmitted or stored.

However, trimming 0s leads to ambiguities when the IDs are used as correlations with other monitoring systems, such as logging, that treat the IDs as opaque strings and cannot establish the equivalence between padded and unpadded IDs. It is also incompatible with W3C Trace Context and Zipkin B3 formats, both of which include all leading 0s, so an application instrumented with OpenTelemetry SDKs may be logging different trace ID strings than application instrumented with Jaeger SDKs (related issue #1657).

Overall, the change is backward compatible:

• links with non-padded IDs in the UI will still work
• data stored in Elasticsearch (where IDs are represented as strings) is still readable

However, some custom integration that rely on exact string matches of trace IDs may be broken.

Change default rollover conditions to 2 days (#1963, @pavolloffay)

Change default rollover conditions from 7 days to 2 days.

Given that by default Jaeger uses daily indices and some organizations do not keep data longer than 7 days the default of 7 days seems unreasonable - it might result in a too big index and running curator would immediately remove the old index.

New Features

• Support collector tags, similar to agent tags (#1854, @radekg)
• Support insecure TLS and only CA cert for Elasticsearch (#1918, @pavolloffay)
• Allow tracer config via env vars (#1919, @yurishkuro)
• Allow turning off tags/logs indexing in Cassandra (#1915, @joe-elliott)
• Blacklisting/Whitelisting tags for Cassandra indexing (#1904, @joe-elliott)

Bug fixes, Minor Improvements

• Support custom basepath in HotROD (#1894, @jan25)
• Deprecate tchannel reporter flags (#1978, @objectiser)
• Do not truncate tags in Elasticsearch (#1970, @pavolloffay)
• Export SaveSpan to enable multiplexing (#1968, @albertteoh)
• Make rollover init step idempotent (#1964, @pavolloffay)
• Update python urllib3 version required by curator (#1965, @pavolloffay)
• Allow changing max log level for gRPC storage plugins (#1962, @yyyogev)
• Fix the bug that operation_name table can not be init more than once (#1961, @guo0693)
• Improve migration script (#1946, @guo0693)
• Fix order of the returned results from badger backend. (#1939, @burmanm)
• Update python pathlib to pathlib2 (#1930, @objectiser)
• Use proxy env vars if they're configured (#1910, @zoidbergwill)

UI Changes

• UI pinned to version 1.6.0. The changelog is available here v1.6.0

Bug fixes, Minor Improvements

• Build platform binaries as part of CI (#1909, @yurishkuro)
• Upgrade and fix dependencies (#1907, @yurishkuro)

Backend Changes

⛔ Breaking Changes

• The default value for the Ingester's flag ingester.deadlockInterval has been changed to 0 (#1868, @jpkrohling)

  With the new default, the ingester won't panic if there are no messages for the last minute. To restore the previous behavior, set the flag's value to 1m.

• Mark --collector.grpc.tls.client.ca flag as deprecated for jaeger-collector. (#1840, @yurishkuro)

  The deprecated flag will still work until being removed, it's recommended to use --collector.grpc.tls.client-ca instead.

New Features

• Support TLS for Kafka (#1414, @MichaHoffmann)
• Add ack and compression parameters for Kafka #1359 (#1712, @chandresh-pancholi)
• Propagate the bearer token to the gRPC plugin server (#1822, @radekg)
• Add Truncate and ReadOnly options for badger (#1842, @burmanm)

Bug fixes, Minor Improvements

• Use correct context on ES search methods (#1850, @rubenvp8510)
• Handling of expected error codes coming from grpc storage plugins #1741 (#1814, @chandresh-pancholi)
• Fix ordering of indexScanKeys after TraceID parsing (#1809, @burmanm)
• Small memory optimizations in badger write-path (#1771, @burmanm)
• Set an empty value when a default env var value is missing (#1777, @jpkrohling)
• Decouple storage dependencies and bump Go to 1.13.x (#1886, @yurishkuro)
• Update gopkg.in/yaml.v2 dependency to v2.2.4 (#1865, @objectiser)
• Upgrade jaeger-client 2.19 and jaeger-lib 2.2 and prom client 1.x (#1810, @yurishkuro)
• Unpin grpc version and use serviceConfig to set the load balancer (#1786, @guanw)

UI Changes

• UI pinned to version 1.5.0. The changelog is available here v1.5.0