Zone Label

The datacenter property was removed from the Server object (see changelog. For backwards compatibility, we statically compute the datacenter from the location to fill the topology.kubernetes.io/zone and failure-domain.beta.kubernetes.io/zone labels. In the next major release we will stop setting these values. To stop setting these values now, you can set the environment variable HCLOUD_INSTANCES_ZONE_LABEL_ENABLED=false. See here for more details.

Features

• helm: allow annotations on the service account (#1295) (f4bf0d3)
• load-balancer: support http timeout-idle annotation (#1301) (926a08a)
• load-balancer: improve error logs on invalid input (#1301) (926a08a)
• node: make the zone label configurable (#1311) (8c2952e)

Compare to previous version

Zone Label

The datacenter property was removed from the Server object (see changelog. For backwards compatibility, we statically compute the datacenter from the location to fill the topology.kubernetes.io/zone and failure-domain.beta.kubernetes.io/zone labels. In the next major release we will stop setting these values. To stop setting these values now, you can set the environment variable HCLOUD_INSTANCES_ZONE_LABEL_ENABLED=false. See here for more details.

Features

• helm: allow annotations on the service account (#1295) (f4bf0d3)
• load-balancer: support http timeout-idle annotation (#1301) (926a08a)
• load-balancer: improve error logs on invalid input (#1301) (926a08a)
• node: make the zone label configurable (#1311) (8c2952e)

v1.35.0 Latest

Latest

Compare

Choose a tag to compare

Features

• chart: expose podSecurityContext and securityContext values (#1258) (ebd8cd4)

Bug Fixes

• routes: reject IPv6 routes early (#1280) (0cba535)
• drop warning for untagged clouds (#1281) (574262f)

Compare to previous version

Features

• chart: expose podSecurityContext and securityContext values (#1258) (ebd8cd4)

Bug Fixes

• routes: reject IPv6 routes early (#1280) (0cba535)
• drop warning for untagged clouds (#1281) (574262f)

v1.34.0

Compare

Choose a tag to compare

Changed

Renamed HCLOUD_SERVER_CACHE_TTL to HCLOUD_SERVER_CACHE_MAX_AGE. The server cache lifetime is now configured via HCLOUD_SERVER_CACHE_MAX_AGE (default 10s). Individual controllers may override this default for specific lookups — for example, the routes controller uses a longer max age. Action required: if you set HCLOUD_SERVER_CACHE_TTL, switch to HCLOUD_SERVER_CACHE_MAX_AGE; the old variable is no longer recognized.

Removed

Removed the hcops/AllServersCache.* operation metrics. The legacy AllServersCache was replaced by the shared server cache, so the following cloud_controller_manager_operations_total series labeled op="hcops/AllServersCache.*" are no longer emitted:

• hcops/AllServersCache.ByID
• hcops/AllServersCache.ByName
• hcops/AllServersCache.ByPrivateIP
• hcops/AllServersCache.getCache
• hcops/AllServersCache.refreshCache

They are superseded by the new server cache metric cloud_controller_manager_server_cache_requests_total, a counter partitioned by subsystem, mode, and result:

• subsystem: instances_v2, routes (or none when unset)
• mode: all, one, off
• result: hit, miss

cloud_controller_manager_server_cache_requests_total{subsystem="instances_v2", mode="all", result="hit"}
cloud_controller_manager_server_cache_requests_total{subsystem="routes",       mode="all", result="miss"}

Update any dashboards or alerts referencing the old op series accordingly.

Features

• cache: replace TTL with max-age
• cache: use server cache in routes controller

Bug Fixes

• cache: when api returns not found do not return an expired entry (#1271)

Changed

Renamed HCLOUD_SERVER_CACHE_TTL to HCLOUD_SERVER_CACHE_MAX_AGE. The server cache lifetime is now configured via HCLOUD_SERVER_CACHE_MAX_AGE (default 10s). Individual controllers may override this default for specific lookups — for example, the routes controller uses a longer max age. Action required: if you set HCLOUD_SERVER_CACHE_TTL, switch to HCLOUD_SERVER_CACHE_MAX_AGE; the old variable is no longer recognized.

Removed

Removed the hcops/AllServersCache.* operation metrics. The legacy AllServersCache was replaced by the shared server cache, so the following cloud_controller_manager_operations_total series labeled op="hcops/AllServersCache.*" are no longer emitted:

• hcops/AllServersCache.ByID
• hcops/AllServersCache.ByName
• hcops/AllServersCache.ByPrivateIP
• hcops/AllServersCache.getCache
• hcops/AllServersCache.refreshCache

They are superseded by the new server cache metric cloud_controller_manager_server_cache_requests_total, a counter partitioned by subsystem, mode, and result:

• subsystem: instances_v2, routes (or none when unset)
• mode: all, one, off
• result: hit, miss

cloud_controller_manager_server_cache_requests_total{subsystem="instances_v2", mode="all", result="hit"}
cloud_controller_manager_server_cache_requests_total{subsystem="routes",       mode="all", result="miss"}

Update any dashboards or alerts referencing the old op series accordingly.

Features

• cache: replace TTL with max-age
• cache: use server cache in routes controller

Bug Fixes

• cache: when api returns not found do not return an expired entry (#1271)

v1.33.0

Compare

Choose a tag to compare

Cache Server Lookups in Node Controllers

This release introduces an experimental server cache to reduce Hetzner Cloud API calls. During an experimental phase, breaking changes on those features may occur within minor releases.

The node and node lifecycle controllers look up Servers by ID or name, generating significant API traffic during cluster scaling. A new cache sits between the controllers and the API to serve these lookups, reducing the number of requests.

It is enabled by default since we believe the implementation is safe in practice, but is experimental and may see breaking changes within minor releases. Configure it via environment variables:

• HCLOUD_SERVER_CACHE_MODE (all | one | off, default all):

  • all — fetch all Servers once and serve lookups from the snapshot until the TTL expires.
  • one — cache each Server individually with its own expiration.
  • off — disable caching; every lookup hits the API.

• HCLOUD_SERVER_CACHE_TTL (duration, default 10s): lifetime of cached entries (e.g. 30s, 2m); values above a minute are not recommended.

Features

• instances: cache server lookups to reduce API calls (#1252)

Cache Server Lookups in Node Controllers

This release introduces an experimental server cache to reduce Hetzner Cloud API calls. During an experimental phase, breaking changes on those features may occur within minor releases.

The node and node lifecycle controllers look up Servers by ID or name, generating significant API traffic during cluster scaling. A new cache sits between the controllers and the API to serve these lookups, reducing the number of requests.

It is enabled by default since we believe the implementation is safe in practice, but is experimental and may see breaking changes within minor releases. Configure it via environment variables:

• HCLOUD_SERVER_CACHE_MODE (all | one | off, default all):

  • all — fetch all Servers once and serve lookups from the snapshot until the TTL expires.
  • one — cache each Server individually with its own expiration.
  • off — disable caching; every lookup hits the API.

• HCLOUD_SERVER_CACHE_TTL (duration, default 10s): lifetime of cached entries (e.g. 30s, 2m); values above a minute are not recommended.

Features

• instances: cache server lookups to reduce API calls (#1252)

v1.32.0

Compare

Choose a tag to compare

Bug Fixes

• routes controller on node name drift (#1221)

Bug Fixes

• routes controller on node name drift (#1221)

v1.31.1

Compare

Choose a tag to compare

Features

• robot: allow Robot support without API credentials for IP-based LB targets (#1163)
• helm: allow customizing chart deployment strategy (#1190)
• support Kubernetes v1.36
• drop support for Kubernetes v1.32

Bug Fixes

• helm: remove permissions for persistentvolumes (#1203)
• route: error handling on locked networks (#1215)

Features

• robot: allow Robot support without API credentials for IP-based LB targets (#1163)
• helm: allow customizing chart deployment strategy (#1190)
• support Kubernetes v1.36
• drop support for Kubernetes v1.32

Bug Fixes

• helm: remove permissions for persistentvolumes (#1203)
• route: error handling on locked networks (#1215)

v1.31.0

Compare

Choose a tag to compare

Datacenter Deprecation

The server.datacenter field is deprecated and will be removed from the API response after July 2026: https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters

To avoid a breaking change in HCCM, we decided to statically map from the old location names to the existing datacenter names for the topology.kubernetes.io/zone label. For new locations we will return the location name without a -dcxx suffix.

Deployments of hcloud-cloud-controller-manager that are not updated when the field is removed from the API will panic with the following error:

"Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue=""invalid memory address or nil pointer dereference""

We have published an RFC which details how we want to continue utilizing the labels topology.kubernetes.io/zone and topology.kubernetes.io/region: #1146 (comment)

Bug Fixes

• statically map deprecated datacenter names (#1159)

Datacenter Deprecation

The server.datacenter field is deprecated and will be removed from the API response after July 2026: https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters

To avoid a breaking change in HCCM, we decided to statically map from the old location names to the existing datacenter names for the topology.kubernetes.io/zone label. For new locations we will return the location name without a -dcxx suffix.

Deployments of hcloud-cloud-controller-manager that are not updated when the field is removed from the API will panic with the following error:

"Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue=""invalid memory address or nil pointer dereference""

We have published an RFC which details how we want to continue utilizing the labels topology.kubernetes.io/zone and topology.kubernetes.io/region: https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/1146#issuecomment-3919929223

Bug Fixes

• statically map deprecated datacenter names (#1159)

v1.30.1

Compare

Choose a tag to compare

Features

• support Kubernetes v1.35
• drop support for Kubernetes v1.31

Bug Fixes

• load-balancer: wait for action of managed certificate (#1144)

Features

• support Kubernetes v1.35
• drop support for Kubernetes v1.31

Bug Fixes

• load-balancer: wait for action of managed certificate (#1144)

v1.30.0

Compare

Choose a tag to compare

Bug Fixes

• tcp service created for service with unsupported load balancer protocol (#1129)

Bug Fixes

• tcp service created for service with unsupported load balancer protocol (#1129)

v1.29.2

Compare

Choose a tag to compare

Bug Fixes

• respect proxy protocol default from config for load balancer status (#1131)

Bug Fixes

• respect proxy protocol default from config for load balancer status (#1131)

v1.29.1

Compare

Choose a tag to compare

Watch-Based Route Reconciliation

Previously, route reconciliation is performed at a fixed interval of 30s. This leads to unnecessary API requests, as a GET /v1/networks/{id} call is triggered every 30s, even when no changes have occurred.

Upstream, we have contributed an event-driven approach, similar to the mechanisms used by other controllers such as the Load Balancer controller. With this new approach, route reconciliation is triggered by node additions, node deletions, or changes to a node’s PodCIDRs or Addresses. Additionally, to ensure consistency, reconciliation still occurs periodically at a randomized interval between 12 and 24 hours.

Enabled by default

This feature is now enabled by default.

If you encounter any problems you can disable the feature by setting the following Helm value:

args.feature-gates=CloudControllerManagerWatchBasedRoutesReconciliation=false

Global Load Balancer Defaults

Configure cluster-wide defaults for Load Balancers via the extended HCLOUD_LOAD_BALANCERS_* env vars. These values automatically apply during Load Balancer creation and reconciliation whenever annotations are omitted. Learn more about it in the reference documentation

Features

• extend environment variables for default load balancer configuration (#1052)
• enable watch based route reconciliation by default (#1112)

Updated ClusterRole for HCCM

We have introduced a custom ClusterRole for the HCCM component, based on the upstream recommendation from sig-cloud-provider.

To ensure a smooth transition, we renamed the ClusterRoleBinding by adding the :restricted suffix. This change was necessary because the roleRef field in a ClusterRoleBinding is immutable, which would otherwise cause errors during a Helm upgrade.

As a result, users who deploy HCCM using the provided Kubernetes manifests must manually delete the old ClusterRoleBinding after applying the updated manifests. Users deploying via the Helm chart do not need to take any action.

kubectl delete clusterrolebindings.rbac.authorization.k8s.io system:hcloud-cloud-controller-manager

Features

• helm: use custom ClusterRole
• configurable metrics address (#1055)

Bug Fixes

• add client-side timeouts to hcloud and robot clients (#1058)

Updated ClusterRole for HCCM

We have introduced a custom ClusterRole for the HCCM component, based on the upstream recommendation from sig-cloud-provider.

Due to this change, users who deploy HCCM using the provided Kubernetes manifests must manually remove the old ClusterRoleBinding after applying the updated manifests. Users deploying via the Helm chart do not need to perform any migration steps.

kubectl delete clusterrolebindings.rbac.authorization.k8s.io system:hcloud-cloud-controller-manager

Updated ClusterRole for HCCM

We have introduced a custom ClusterRole for the HCCM component, based on the upstream recommendation from sig-cloud-provider.

Because of this change, a simple helm upgrade may fail due to immutable fields in the existing ClusterRoleBinding. To ensure a successful upgrade, you must delete the existing ClusterRoleBinding before upgrading. This will make HCCM temporarily unavailable until the new version is deployed. Please avoid making any changes to the clusters, especially node- or service-related modifications, during this time.

kubectl delete clusterrolebindings.rbac.authorization.k8s.io system:hcloud-cloud-controller-manager

Features

• helm: use custom ClusterRole
• configurable metrics address (#1055)

Bug Fixes

• add client-side timeouts to hcloud and robot clients (#1058)

Attach Load Balancer to a Subnet

If your CCM is configured for a Private Network, Load Balancers can now join one of its subnets. To place a Load Balancer in a specific subnet, use the new load-balancer.hetzner.cloud/private-subnet-ip-range annotation. Learn more about this feature here.

Watch-Based Route Reconciliation (Experimental)

Currently, route reconciliation is performed at a fixed interval of 30s. This leads to unnecessary API requests, as a GET /v1/networks/{id} call is triggered every 30s, even when no changes have occurred.

Upstream we have proposed an event-driven approach, similar to the mechanism used by other controllers such as the Load Balancer Controller. With this new approach, route reconciliation is triggered on node additions, node deletions, or when the PodCIDRs or Addresses of nodes change. Additionally, to ensure consistency, reconciliation will still occur periodically at a randomized interval between 12 and 24 hours.

We are close to merging a Kubernetes Enhancement Proposal (KEP). Furthermore, a pull request containing the implementation is already open in the Kubernetes repository.

Forked Upstream Libraries

In this release, we replaced the upstream controller-manager and cloud-provider libraries with our own forks. These forks are based on the upstream v0.34.1 release (aligned with Kubernetes v1.34.1) and include our patches on top.

Enabling the Feature

This feature is disabled by default and will not affect existing deployments unless explicitly enabled. We recommend testing it in a non-production environment before considering use in production.

As the KEP has not yet been reviewed for production readiness, the feature gate name may change in an upcoming release. Since this feature is marked as experimental, such changes will not be considered breaking.

To enable the feature, set the following Helm value:

args.feature-gates=CloudControllerManagerWatchBasedRoutesReconciliation=true

Features

• watch-based route reconciliation (#970)
• set dns config via helm chart values (#1027)
• support Kubernetes v1.34 and drop v1.30 (#1037)
• load-balancer: attach load balancer to specific subnetwork (#1031)

Bug Fixes

• feature gate cannot be enabled (#980)

This release introduces an experimental feature to address #395.

Watch-Based Route Reconciliation (Experimental)

Currently, route reconciliation is performed at a fixed interval of 30s. This leads to unnecessary API requests, as a GET /v1/networks/{id} call is triggered every 30s, even when no changes have occurred.

Upstream we have proposed an event-driven approach, similar to the mechanism used by other controllers such as the Load Balancer Controller. With this new approach, route reconciliation is triggered on node additions, node deletions, or when the PodCIDRs or Addresses of nodes change. Additionally, to ensure consistency, reconciliation will still occur periodically at a randomized interval between 12 and 24 hours.

We are close to merging a Kubernetes Enhancement Proposal (KEP). Furthermore, a work-in-progress pull request containing the implementation is already open in the Kubernetes repository.

Forked Upstream Libraries

In this release, we replaced the upstream controller-manager and cloud-provider libraries with our own forks. These forks are based on the upstream v0.33.2 release (aligned with Kubernetes v1.33.2) and include our patches on top.

Enabling the Feature

This feature is disabled by default and will not impact existing deployments unless explicitly enabled. We do not recommend running this feature in production environments at this stage. However, we welcome early testers who can try it in non-critical setups. Running with this feature active is enough for us to analyze its impact. No additional feedback is required.

To enable the feature, set the following Helm value:

args.feature-gates=CloudControllerManagerWatchBasedRoutesReconciliation=true

Bug Fixes

• feature gate cannot be enabled (#980)

This release introduces an experimental feature to address #395.

Watch-Based Route Reconciliation (Experimental)

Currently, route reconciliation is performed at a fixed interval of 30 seconds. This leads to unnecessary API requests, as a GET /v1/networks/{id} call is triggered every 30 seconds, even when no changes have occurred.

Upstream we have proposed an event-driven approach, similar to the mechanism used by other controllers such as the Load Balancer Controller. With this new approach, route reconciliation is triggered on node additions, node deletions, or when the PodCIDRs or Addresses of nodes change. Additionally, to ensure consistency, reconciliation will still occur periodically at a randomized interval between 12 and 24 hours.

We are close to merging a Kubernetes Enhancement Proposal (KEP). Furthermore, a work-in-progress pull request containing the implementation is already open in the Kubernetes repository.

Forked Upstream Libraries

In this release, we replaced the upstream controller-manager and cloud-provider libraries with our own forks. These forks are based on the upstream v0.33.2 release (aligned with Kubernetes v1.33.2) and include our patches on top.

Enabling the Feature

This feature is disabled by default and will not impact existing deployments unless explicitly enabled. We do not recommend running this feature in production environments at this stage. However, we welcome early testers who can try it in non-critical setups. Running with this feature active is enough for us to analyze its impact. No additional feedback is required.

To enable the feature, set the following Helm value:

args.feature-gates=CloudControllerManagerWatchBasedRoutesReconciliation=true

Features

• watch-based route reconciliation (#970)

Features

• add Kubernetes v1.33 and remove EOL version v1.29 (#934)
• helm: add value to disable ClusterRoleBinding for out-of-cluster install (#962)

Bug Fixes

• binaries are missing in the release (#930)

Bug Fixes

• binaries are missing in the release (#930)

Features

• helm: support custom priorityClassName (#908)
• helm: support imagePullSecrets (#907)

Improved Robot Support in hcloud-cloud-controller-manager

The hcloud-cloud-controller-manager now forwards InternalIPs by default on Robot nodes when the --node-ip flag is used. If the provided IP is not already registered as an ExternalIP and matches the expected address family, it will be forwarded automatically during initialization.

This allows the use of vSwitch IPs in private networks and Load Balancers.

🔗 Learn more in our updated Robot documentation 📘 Follow our how-to-guide to set up Load Balancers with vSwitch IPs.

Features

• robot: forward InternalIPs by default on Robot nodes (#865)
• load-balancer: enable use-private-ip annotation for Robot servers (#898)

Bug Fixes

• load-balancer: don't print entire node object when adding robot target (#904)

Features

• add support and tests for Kubernetes 1.32 (#846)

Bug Fixes

• metrics: do not emit kubernetes_build_info metric (#849)

This release includes an extension of our current metrics to also include the internals of k8s.io/cloud-provider with respect to the work queue depth and requests to the Kubernetes API.

Besides having all data available, this will also help us with debugging #661.

Features

• metrics: add metrics from cloud-provider library (#824)
• load-balancer: emit warning if unsupported port protocol is configured (#828)
• allow arbitrary length API tokens (#752)

Feature Highlights & Upgrade Notes

Load Balancer IPs set to Private IPs

If networking support is enabled, the load balancer IPs are now populated with the private IPs, unless the load-balancer.hetzner.cloud/disable-private-ingress annotation is set to true. Please make sure that you configured the annotation according to your needs, for example if you are using external-dns.

Provided-By Label

We introduced a the label instance.hetzner.cloud/provided-by, which will be automatically added to all new nodes. This label can have the values cloud or robot to distinguish between our products. We use this label in the csi-driver to ensure the daemonset is only running on cloud nodes. We recommend to add this label to your existing nodes with the appropriate value.

• kubectl label node $CLOUD_NODE_NAME instance.hetzner.cloud/provided-by=cloud
• kubectl label node $ROBOT_NODE_NAME instance.hetzner.cloud/provided-by=robot

Load Balancer IPMode Proxy

Kubernetes KEP-1860 added a new field to the Load Balancer Service Status that allows us to mark if the IP address we add should be considered as a Proxy (always send traffic here) and VIP (allow optimization by keeping the traffic in the cluster).

Previously Kubernetes considered all IPs as VIP, which caused issues when when the PROXY protocol was in use. We have previously recommended to use the annotation load-balancer.hetzner.cloud/hostname to workaround this problem.

We now set the new field to Proxy if the PROXY protocol is active so the issue should no longer appear. If you only added the load-balancer.hetzner.cloud/hostname annotation for this problem, you can remove it after upgrading.

Further information:

• https://github.com/kubernetes/enhancements/issues/1860
• https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/160#issuecomment-788638132

Features

• service: Specify private ip for loadbalancer (#724)
• add support & tests for Kubernetes 1.31 (#747)
• helm: allow setting extra pod volumes via chart values (#744)
• instance: add label to distinguish servers from Cloud and Robot (#764)
• emit event when robot server name and node name mismatch (#773)
• load-balancer: Set IPMode to "Proxy" if load balancer is configured to use proxy protocol (#727) (#783)
• routes: emit warning if cluster cidr is misconfigured (#793)
• load-balancer: ignore nodes that don't use known provider IDs (#780)
• drop tests for kubernetes v1.27 and v1.28

Bug Fixes

• populate ingress private ip when disable-private-ingress is false (#715)
• wrong version logged on startup (#729)
• invalid characters in label instance-type of robot servers (#770)
• no events are emitted as broadcaster has no sink configured (#774)

Kubernetes Support

This version was tested with Kubernetes 1.29 - 1.31. Furthermore, we dropped v1.27 and v1.28 support.