NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Go modules
Go modules, #652 by repository stars
Latest stable
v1.7.2
Last release 13 days ago
09 Nov 2026
Ships on a steady schedule
a new release about every 9 days
Nearly every release is documented
notes for 28 of 28 stable releases
Nothing withdrawn
no release was ever pulled
5 years old
1431 releases · first in 2021
Drag across the ruler to compare, or tap a release to open it.
0561679589aa957ef62ea84018d0eafdaa50d8bf Add stale issue/PR bot workflow
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Bump golang.org/x/net, golang.org/x/text and cert-manager to fix CVEs
Full Changelog: https://github.com/kuberay-project/kuberay/compare/v1.7.1...v1.7.2
Fixed a bug where the kubectl plugin did not re-check RayJob status after losing the port-forward connection, which could leave kubectl ray job submit
kubectl ray job submit hanging. https://github.com/ray-project/kuberay/pull/5147rayproject/ray-ml:2.46.0.0e19ea-* tags were removed from Docker Hub and the samples failed with ImagePullBackOff. https://github.com/ray-project/kuberay/pull/5194Full Changelog: https://github.com/ray-project/kuberay/compare/v1.7.0...v1.7.1
The ray.io/v1alpha1 API version is now marked deprecated (\#5122). Please migrate to ray.io/v1.
The Ray History Server is graduating to beta. Introduced as alpha in KubeRay v1.6, the History Server lets you view the Ray Dashboard and debug ephemeral clusters (such as those managed by RayJob) even after they have been terminated.
Since v1.6, the History Server has received major improvements:
historyserver:v1.7.0 and collector:v1.7.0)Try it here: History Server Quick Start Guide.
The operator can now automatically inject the History Server event collector sidecar into your RayCluster’s pods, no manual sidecar configuration needed. Enable the RayClusterHistoryServer feature gate on the operator and set spec.historyServerOptions on your RayCluster:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: raycluster-historyserver
spec:
historyServerOptions:
collectorOptions:
image: quay.io/kuberay/collector:v1.7.0
See ray-cluster.historyserver.yaml for a full example.
:warning: Warning: RayClusterHistoryServer is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterHistoryServer=true
The RayServiceIncrementalUpgrade feature gate is now beta and enabled by default. This feature allows RayService to perform zero-downtime upgrades by gradually shifting traffic to the new cluster using the Kubernetes Gateway API, instead of requiring 2x resources for a blue/green switch.
Since its Alpha release, the feature has received several improvements for more reliable and robust upgrades, including:
KubeRay v1.7 can automatically provision and rotate certificates for mutual TLS between RayCluster’s pods using cert-manager.
:warning: Warning: RayClusterMTLS is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterMTLS=true
Enable mTLS feature with following to try this feature:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: raycluster-mtls
spec:
rayVersion: '2.55.1'
tlsOptions:
enabled: true
See ray-cluster.mtls.yaml for a full example.
KubeRay v1.7 adds a new spec.networkPolicy field to RayCluster for network isolation. When set, the operator creates separate NetworkPolicies for head and worker pods, with configurable modes (DenyAll, DenyAllIngress, DenyAllEgress) and per-worker-group ingress/egress rules. Intra-cluster pod-to-pod traffic is always permitted.
Try it here: Network Policy Quick Start Guide.
:warning: Warning: RayClusterNetworkPolicy is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterNetworkPolicy=true
Below is an example of enabling network policy feature:
apiVersion: ray.io/v1
kind: RayCluster
spec:
networkPolicy:
mode: DenyAll
head:
ingressRules:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <prometheus-namespace>
ports:
- port: 8080
protocol: TCP
See ray-cluster.network-policy-deny-all.yaml for an example.
GCS fault tolerance no longer requires an external Redis. With the GCSFaultToleranceEmbeddedStorage feature gate enabled, you can persist GCS metadata in an embedded RocksDB store backed by a persistent volume on the head Pod:
spec:
gcsFaultToleranceOptions:
backend: rocksdb
storage:
size: 1Gi
:warning: Warning: GCSFaultToleranceEmbeddedStorage is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=GCSFaultToleranceEmbeddedStorage=true
See ray-cluster.embedded-gcs-ft.yaml for a full example.
headGroupSpec.ingressOptions with host, path, pathType, and tls fields.spec.timeZone field, and child RayJobs use deterministic names to prevent duplicate firings.priority field for scheduler integrations.SidecarSubmitterRestart prevents premature job termination during transient head Pod resource spikes in RayJob sidecar mode.defaultPodAnnotations and defaultPodLabels applied to all Ray Pods.kubectl ray logs support --gke-link to print a Cloud Logging link on GKE.metrics.serviceMonitor.selector has been renamed to metrics.serviceMonitor.additionalLabels, and empty labels are no longer emitted (#4979). Update your values file if you set this field.AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, ...) instead of AWS_S3ID/AWS_S3SECRET/AWS_S3TOKEN (#4665), the runtime-class flag has been replaced by --storage-backend (#5085), and the on-storage event layout has changed (#4670). Redeploy the collector and history server together on v1.7 and see the updated setup guide.ray.io/v1alpha1 API version is now marked deprecated (#5122). Please migrate to ray.io/v1.tasks/timeline generation with Ray Dashboard by @AndySung320 in https://github.com/ray-project/kuberay/pull/4546Spec.Suspend for incremental upgrade by @kevin85421 in https://github.com/ray-project/kuberay/pull/4881NewClusterWithIncrementalUpgrade RayService rollback load tests by @ryanaoleary in https://github.com/ray-project/kuberay/pull/4915/enter_cluster by @chiayi in https://github.com/ray-project/kuberay/pull/4907RAY_ROOT_DIR TO STORAGE_ROOT_DIR and examples to use root of the bucket path as root by @chiayi in https://github.com/ray-project/kuberay/pull/5121Full Changelog: https://github.com/ray-project/kuberay/compare/v1.6.2...v1.7.0
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
587c6ff977ffa525cb4f1590b9cd60db0656d7bd Bump go to 1.22.4 to fix ray-operator vulnerabilities
NewClusterWithIncrementalUpgrade RayService rollback load tests (#4915)ray.io/originated-from labels (#1830)rayStartParams in Kuberay (#1057)HeadGroupSpec and WorkerGroupSpec (#4106)role.yaml and multiple_namespaces_role.yaml to use the same template in _helpers.tpl to ensure consistency. (#2244)sumGPUs to include MIGs in count (#3933)[raycluster|rayjob|rayservice]_types_test.go unnecessary tests (#2935)kind from to v0.20.0 to v0.11.1 (#1313)Labels and label_selector API (#4136)set_up_historyserver.md (#4492)BatchSchedulerManager into reconciler option (#3935)ray-pod.tls.yaml (#3762)values.yaml (#3950)/api/v0/logs/file (#4411)TestRayClusterAutoscalerAddNewWorkerGroup (#3682)make deploy (#2059)BuildServiceForHeadPod function (#1095)Complete and a JobStatus SUCCEEDED (#1919)kind-in-Docker test to Buildkite CI (#1243)testifylint empty rule (#2908)testifylint error-nil rule (#2907)testifylint expected-actual rule (#2914)testifylint float-compare rule (#2910)testifylint require-error rule (#2909)kind load $RAY_IMAGE for e2e sample YAML tests (#1863)test_detached_actor until https://github.com/ray-project/ray/issues/41343 (#1694)testifylint compares rule (#2977)helm update command to chart validation step in release process (#1165)helm-chart/kuberay-operator/README.md.gotmpl with go template (#3763)Note truncated.
Nothing published for this version
Fix a bug where Redis clean up job would never execute when using RayService with Kueue
Fixed a bug where using spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavio
spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavior in some cases with Kueue when queueing or pre-empting RayService objects https://github.com/ray-project/kuberay/pull/4739KubeRay v1.6 introduces alpha support for the Ray History Server. This project enables users to collect and aggregate events from a Ray cluster, repla
KubeRay v1.6 introduces alpha support for the Ray History Server. This project enables users to collect and aggregate events from a Ray cluster, replaying them to restore historical snapshots of the cluster's state. By providing an alternative backend for the Ray Dashboard, the History Server allows users to view the Ray dashboard and debug ephemeral clusters (such as those managed via RayJob) even after they have been terminated.
Try the history server here: History Server Quick Start Guide.
:warning: Warning: This feature is in alpha status, meaning future KubeRay releases may include breaking updates. We’d love to hear your experience with it! Please drop your feedback in this tracking issue to help us shape its development.
Starting in KubeRay v1.6 and Ray v2.55, you can use Kubernetes RBAC to manage user access control to Ray clusters that have token authentication enabled. With this feature enabled, Ray will be configured to delegate token authentication to Kubernetes. This means you can use the same credentials used with Kubernetes to access Ray clusters and platform operators can use standard Kubernetes RBAC to control access to Ray clusters. See Configure Ray clusters to use Kubernetes RBAC authentication for a step-by-step guide.
You can now also reference Secrets containing static auth tokens for Ray cluster token authentication.
apiVersion: v1
kind: Secret
metadata:
name: ray-cluster-token
type: Opaque
stringData:
auth_token: "super-secret-example-token"
---
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: ray-cluster-with-auth
spec:
authOptions:
mode: token
secretName: ray-cluster-token
rayVersion: '2.53.0'
headGroupSpec:
rayStartParams: {}
KubeRay v1.6 introduces the RayCronJob Custom Resource Definition (CRD), enabling users to schedule RayJobs on a recurring schedule using standard cron expressions. This is useful for periodic batch processing, scheduled training runs, or recurring data pipelines.
Try the Ray CronJob API here: Ray CronJob Quick Start Guide.
:warning: Warning: RayCronJob is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayCronJob=true
Below is an example of the new custom resource:
apiVersion: ray.io/v1
kind: RayCronJob
metadata:
name: raycronjob-sample
spec:
schedule: "* * * * *"
jobTemplate:
entrypoint: python /home/ray/samples/sample_code.py
shutdownAfterJobFinishes: true
ttlSecondsAfterFinished: 600
runtimeEnvYAML: |
pip:
- requests==2.26.0
- pendulum==2.1.2
env_vars:
counter_name: "test_counter"
rayClusterSpec:
rayVersion: '2.52.0'
headGroupSpec:
...
...
See ray-cronjob.sample.yaml for a full example.
The RayJobDeletionPolicy feature gate is graduating to Beta and enabled by default. This feature enables a more advanced and flexible API for expressing deletion policies within the RayJob specification. This new design moves beyond the singular boolean field, spec.shutdownAfterJobFinishes, and allows users to define different cleanup strategies using configurable TTL values based on the Ray job's status.
Below is an example of how to use this new, flexible API structure:
apiVersion: ray.io/v1
kind: RayJob
metadata:
name: rayjob-deletion-rules
spec:
deletionStrategy:
deletionRules:
- policy: DeleteWorkers
condition:
jobStatus: FAILED
ttlSeconds: 100
- policy: DeleteCluster
condition:
jobStatus: FAILED
ttlSeconds: 600
See ray-job.deletion-rules.yaml for a comprehensive example.
RayMultihostIndexing feature gate is graduating to Beta and enabled by default. This feature provides ordered replica and host index labels that are useful for managing Ray clusters for multi-host TPU/GPU workloads that require atomic scheduling and scaling. These labels are only applied if numOfHosts > 1 in worker group configuration.spec.upgradeStrategy field to RayCluster. Supported values are Recreate and None. The Recreate strategy is useful for automatically recreating all Ray cluster Pods when the Ray cluster spec changes. This upgrade strategy is not recommended if Ray cluster state needs to be persisted.When using Sidecar submission mode with RayJob, the Head Pod will no longer be automatically recreated after initial provisioning. This is because the submission container runs along the Head container and recreating the Pod would result in restarting the job entirely. See #4141 for more details.
Thanks to all the contributors who made this release possible!
@400Ping, @AndySung320, @ChenYi015, @CheyuWu, @EthanGuoliang, @Future-Outlier, @JiangJiaWei1103, @KunWuLuan, @LilyLinh, @MiniSho, @Narwhal-fish, @Tomlord1122, @alimaazamat, @andrewsykim, @cchung100m, @chiayi, @divyamraj18, @dushulin, @enoodle, @fangyinc, @fscnick, @fweilun, @hango880623, @harryge00, @ikchifo, @justinyeh1995, @kash2104, @kevin85421, @lorriexingfang, @lw309637554, @machichima, @marosset, @my-vegetable-has-exploded, @nojnhuh, @rueian, @ryanaoleary, @ryankert01, @seanlaii, @spencer-p, @win5923, @yuhuan130
v1.6.0-rc.0 Pre-release Pre-release Compare # Choose a tag to compare
v1.6.0-rc.0 Pre-release
Pre-release
Compare
Fixed a bug where using spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavio
spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavior in some cases with Kueue when queueing or pre-empting RayService objects https://github.com/ray-project/kuberay/pull/4739This release adds support for Ray token authentication using RayCluster, RayJob and RayService.
This release adds support for Ray token authentication using RayCluster, RayJob and RayService.
You can enable Ray token authentication using the following API:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: ray-cluster-with-auth
spec:
rayVersion: '2.52.0'
authOptions:
mode: token
You must specify spec.rayVersion to 2.52.0 or newer. See full example at ray-cluster.auth.yaml.
Ray v2.49 introduced a label selector API. Correspondingly, KubeRay v1.5 now features a top-level API for defining Ray labels and resources. This new
Ray v2.49 introduced a label selector API. Correspondingly, KubeRay v1.5 now features a top-level API for defining Ray labels and resources. This new top-level API is the preferred method going forward, replacing the previous practice of setting labels and custom resources within rayStartParams.
The new API will be consumed by the Ray autoscaler, improving autoscaling decisions based on task and actor label selectors. Furthermore, labels configured through this API are mirrored directly into the Pods. This mirroring allows users to more seamlessly combine Ray label selectors with standard Kubernetes label selectors when managing and interacting with their Ray clusters.
You can use the new API in the following way:
apiVersion: ray.io/v1
kind: RayCluster
spec:
...
headGroupSpec:
rayStartParams: {}
resources:
Custom1: "1"
labels:
ray.io/zone: us-west-2a
ray.io/region: us-west-2
workerGroupSpec:
- replicas: 1
rayStartParams: {}
resources:
Custom1: "1"
labels:
ray.io/zone: us-west-2a
ray.io/region: us-west-2
The RayJob resource now supports a new value for spec.submissionMode called SidecarMode.
Sidecar mode directly addresses a key limitation in both K8sJobMode and HttpMode: the network connectivity requirement from an external Pod or the KubeRay operator for job submission. With Sidecar mode, job submission is orchestrated by injecting a sidecar container into the Head Pod. This solution eliminates the need for an external client to handle the submission process and reduces job submission failure due to network failures.
To use this feature, set spec.submissionMode to SidecarMode in your RayJob:
apiVersion: ray.io/v1
kind: RayJob
metadata:
name: my-rayjob
spec:
submissionMode: "SidecarMode"
...
KubeRay now supports a more advanced and flexible API for expressing deletion policies within the RayJob specification. This new design moves beyond the singular boolean field, spec.shutdownAfterJobFinishes, and allows users to define different cleanup strategies using configurable TTL values based on the Ray job's status.
This API unlocks new use cases that require specific resource retention after a job completes or fails. For example, users can now implement policies that:
By linking specific TTLs to Ray job statuses (e.g., success, failure) and strategies (e.g. DeleteWorkers, DeleteCluster, DeleteSelf), users gain fine-grained control over resource cleanup and cost management.
Below is an example of how to use this new, flexible API structure:
apiVersion: ray.io/v1
kind: RayJob
metadata:
name: rayjob-deletion-rules
spec:
deletionStrategy:
deletionRules:
- policy: DeleteWorkers
condition:
jobStatus: FAILED
ttlSeconds: 100
- policy: DeleteCluster
condition:
jobStatus: FAILED
ttlSeconds: 600
- policy: DeleteCluster
condition:
jobStatus: SUCCEEDED
ttlSeconds: 0
This feature is disabled by default and requires enabling the RayJobDeletionPolicy feature gate.
KubeRay v1.5 introduces the capability to enable zero-downtime incremental upgrades for RayServices. This new feature improves the upgrade process by leveraging the Gateway API and Ray autoscaling to incrementally migrate user traffic from the existing Ray cluster to the newly upgraded one.
This approach is more efficient and reliable compared to the former mechanism. The previous method required creating the upgraded Ray cluster at its full capacity and then shifting all traffic at once, which could lead to disruptions and unnecessary resource usage. By contrast, the incremental approach gradually scales up the new cluster and migrates traffic in smaller, controlled steps, resulting in improved stability and resource utilization during upgrade.
To enable this feature, set the following fields in RayService:
apiVersion: ray.io/v1
kind: RayService
metadata:
name: example-rayservice
spec:
upgradeStrategy:
type: "NewClusterWithIncrementalUpgrade"
clusterUpgradeOptions:
maxSurgePercent: 40
stepSizePercent: 5
intervalSeconds: 10
gatewayClassName: "cluster-gateway"
This feature is disabled by default and requires enabling the RayServiceIncrementalUpgrade feature gate.
Previous KubeRay versions supported multi-host worker groups via the numOfHosts API, but this capability lacked fundamental capabilities required for managing multi-host accelerators. Firstly, it lacked logical grouping of worker Pods belonging to the same multi-host unit (or slice). As a result, it was not possible to run operations like “replace all workers in this group”. In addition, there was no ordered indexing, which is often required for coordinating multi-host workers when using TPUs.
When using multi-host in KubeRay v1.5, KubeRay will automatically set the following labels for multi-host Ray workers:
labels:
ray.io/worker-group-replica-name: tpu-group-af03de
ray.io/worker-group-replica-index: 0
ray.io/replica-host-index: 1
Below is a description of each label and its purpose:
ray.io/worker-group-replica-name: this label provides a unique identifier for each replica (i.e. host group or slice) in a worker group. The label enables KubeRay to rediscover all other pods in the same group and apply group operators.ray.io/worker-group-replica-index: this label is an ordered replica index in the worker group. This label is particularly important for cases like multi-slice TPUs, where each slice must be aware of its slice index.ray.io/replica-host-index: this label is an ordered host index per replica (host group or slice).These changes collectively enable reliable, production-level scaling and management of multi-host GPU workers or TPU slices.
This feature is disabled by default and requires enabling the RayMultiHostIndexing feature gate.
For RayCluster objects created by a RayJob, KubeRay will no longer attempt to recreate the Head Pod if it fails or is deleted after its initial successful provisioning. To retry failed jobs, use spec.backoffLimit which will result in KubeRay provisioning a new RayCluster.
test-e2e-rayservice in Makefile (#4145, @seanlaii)Labels and label_selector API (#4136, @ryanaoleary)HeadGroupSpec and WorkerGroupSpec (#4106, @ryanaoleary)apiserversdk may return incomplete response body causing ERR_INCOMPLETE_CHUNKED_ENCODING (#4061, @CheyuWu)GetDefaultSubmitterTemplate (#4056, @owenowenisme)httpproxy_httpclient.go (#4010, @owenowenisme)dashboard-httpclient.go (#4009, @owenowenisme)FetchHeadServiceURL to util.go to reduce imported packages in dashboard_httpclient.go (#3983, @kevin85421)-s with test -s (#3970, @MiniSho)values.yaml (#3950, @SheldonTsen)sumGPUs to include MIGs in count (#3933, @kimminw00)BatchSchedulerManager into reconciler option (#3935, @owenowenisme)rayStartParams (#3586, @davidxia)true (#3805, @owenowenisme)ray-job.pytorch-distributed-training.yaml (#3796, @MortalHappiness)ray-job.interactive-mode.yaml (#3779, @CheyuWu)helm-chart/kuberay-operator/README.md.gotmpl with go template (#3763, @MortalHappiness)ray-pod.tls.yaml (#3762, @kevin85421)v1.5.0-rc.1 Pre-release Pre-release Compare # Choose a tag to compare
v1.5.0-rc.1 Pre-release
Pre-release
Compare
Nothing published for this version
One column per quarter.