github.com/google/cloudprober

Metrics export enhancements

• Two shiny new surfacers: AWS CloudWatch and Datadog. (#141 & #583).
• You can now configure any surfacer to export metrics as gauge. You just need to add export_as_gauge to the surfacer config. (#604)
• Also, now you can add failure metric to any surfacer by adding add_failure_metric to the surfacer config. (#604)
• A common way to filter metrics. You now have fine-grained control of which metrics will be exported to which surfacer, e.g. you may want to send only critical probe metrics to an expensive monitoring system, while you may send all metrics to pub/sub or files. (#597)

Labeling

• Multiple substitutions: We now support multiple substitutions in labels, e.g. a label like @target.name@:@target.port@@target.relative_url@ will get appropriately substituted. (#582)

Targets Discovery

• RDS provider for file based targets. To allow sharing a large set of targets across multiple probes.
• Implement caching in RDS protocol. No need to re-process (on RDS server or client) targets that haven't changed. (#646)
  [Above two changes allowed cloudprober to work with 1M targets in a single targets file. (#634)]
• Add "node" and "pod" labels for K8s endpoints resources. (#612)

Bug Fixes

• External probe: Generate metrics even if requests timeout. (#653)
• External probe: Add additional labels to payload based metrics as well. (#654)
• HTTP Probe: Fix parallelism when running multiple requests per probe. (#647)

Complete list of commits since the last release: v0.11.2...v0.11.3
Bugs/PR in this release: Milestone v0.11.3.

Enhancements

More accurate "ping" latency measurements
Cloudprober now uses SO_TIMESTAMP socket option (on non-windows OS) to make latency measurements much more accurate. (#546)

Better isolation in HTTP probes for multiple targets
HTTP probes for multiple targets execute in independent goroutines now. This allows for better isolation between probes.

Add HTTP header validator for HTTP probe
You can now add a header validator to an HTTP probe (example).

Bug Fixes

#537 Datagram socket ping (unprivileged ping) over IPv4 is broken for MacOS.
#554 Error initializing cloudprober on GKE.
#584 Goroutine leaking bug in external probe.

Other improvements and fix.

Complete list of commits since the last release: v0.11.1...v0.11.2
Bugs/PR in this release: Milestone v0.11.2.

Enhancements

External probe

• Dynamic labels for the external probe metrics (#486). External probe programs can now add labels to payload-generated metrics by specifying labels in payload like this:

db_status{name=db01,region=us-west-2,status=running} 1.0
db_status{name=db02,region=us-east-1,status=running} 1.0

• Enhance payload parsing to support spaces in labels and string values (#491). Example:

version{service="service A",dc="xx"} "version 1.5.0"

Stackdriver surfacer

• Export unit for the "latency" metric (#349, #492). Not having unit for latency metric has been a constant source of confusion.
• Export "failure" metric along with the standard success, total and latency metrics. (#498).

Also added stackdriver surfacer documentation and an example.

Other enhancements

• Reliability and scalability improvement to GCE resources discovery: use per-scope caches (#479).
• OAuth: Check for ID token in OAuth token if access token is not set (#483).

Complete list of commits since the last release: v0.11.0...v0.11.1
Bugs/PR in this release: Milestone v0.11.1.

Targets Discovery Enhancements

• Support for discovering Kubernetes Ingress resources. Here is an example of how to discovery ingresses: #419 (comment).

• Make target labels available for external probe commands. For example, you can use this feature to access target's fqdn label to construct the Host header (Issue with an example: #451).

• More target label based customizations for HTTP probe type. HTTP probe now automatically uses "fdqn" label for the Host header and host part of the URL, and "relative_url" label for the relative URL part of the URL. Using these labels, now you can configure just one probe for multiple targets where each target has a different host header and relative URL to probe.

Bug Fixes

• Fix a bug the gRPC service to dynamically configure probes: #460.
• Fix a bug in cloudprober extensions which will not allow adding more than one probe or target extension: #450.
• Fix a bug in external probe's command parsing: #473.
• Improve CLOUDPROBER_PORT environment variable handling while running on Kubernetes: #476.

Other Enhancements

• HTTP Probe: Add support for large request bodies (#448).

Many thanks to @networkop, @evanSpendlove, @sunshinekitty, @JohnWillker, and @evgenii-petrov-arrival for reporting bugs, and suggesting and contributing fixes.

Complete list of commits since the last release: v0.10.9...v0.11.0
Bugs/PR in this release: Milestone v0.11.0.

File based targets discovery.

Targets can be provided through a file (file can be on GCS as well), in text or json format corresponding to the RDS Resource protobuf (#394).

Kubernetes targets discovery improvements

• Support for filterting endpoints by port (#370).
• Make discovered service's port available to probes that use ports (e.g. HTTP) (#420).
• Provide a way to specify per-target HTTP host header through a target label ("fqdn"). Having a service be accessible only with host header service.namespace.cluster.local is common in Kubernetes and Istio. With this option you can attach the "fqdn" label to the service and cloudprober HTTP probes will do the right thing (#438).
• Fix service discovery support for AWS ELB (#418).

Miscellaneous

• For keep_alive HTTP probes, keep track of and report number of connections done, through a metric called "connect_event" (#421).
• Handle external probe process termination more gracefully (#413).
• Provide a flag (--cloud_metadata=auto|ec2|gce|none) to control cloud-specific initialization to improve cloudprober start-up time (#417).

Bug Fixes

This releases fixes a couple of uncommon bugs:

• If kubernetes resources have same names across namespaces, it would cause cloudprober to have duplicate resources (#437), and duplicate resources can cause probe data to get mangled (#436). We track kubernetes resources by both, name and namespace, now (#440) and make sure that targets discovery client doesn't return duplicate resources (#439).

• Fix a bug where running configtest on GCE VM panics (#441).

Backward incompatible changes

We are changing how you access resource labels in cloudprober configs:

• Make GCE instance labels available to configs as label_<label-name> instead of labels_<label-name>. For example, in configs, you can use {{.label_app}} to refer to the instance's "app" label.
• To add additional labels (example) dynamically, based on targets, enclose them within @. For example, to add a label based on target's app label, use @target.label.app@.

These are not widely advertised or used features, so we expect the impact to be minimum.

See v0.10.7...v0.10.9 for the complete list of commits since the last release.

This release is now abandoned in favor of v0.10.9.

• Support for probing OAuth protected HTTP URLs. This was a popular feature request (#27). OAuth credentials can be specified in multiple ways (Google default credentials, JSON key config, a command that returns access token).

• Contrib package: We are introducing a way to host additional binaries and scripts for running with Cloudprober (as external probe for example). These additional binaries will be available in the cloudprober/cloudprober-contrib docker image.

  • First contrib package comes with a BigQuery prober (#348).

• Bug fixes:

  • HTTP probe: Include port in the host header. This is as per the standard; it mainly impacts the redirected URLs (#365).
  • Prometheus: Fix metric type header for histograms (#364).
  • Fix "services" discovery support in the recently added Kubernetes targets discovery (#345).

• Other features:

  • Support for accessing GCE customer metadata in cloudprober configs (#363).
  • Load balancing support in RDS client (#362). You can run two RDS servers and have RDS client round-robin across them.
  • Support for accessing files from GCS, simply prefix their path with gs://. Useful for storing TLS certs or OAuth keys on GCS (#344).

See v0.10.6...v0.10.7 for the complete list of commits since the last release.