Highlights

Ray History Server (beta)

The Ray History Server is graduating to beta. Introduced as alpha in KubeRay v1.6, the History Server lets you view the Ray Dashboard and debug ephemeral clusters (such as those managed by RayJob) even after they have been terminated.

Since v1.6, the History Server has received major improvements:

• Performance and scalability: lazy loading, an LRU byte-based session cache, a disk-first event storage pipeline, and event compression significantly reduce memory usage for both the collector and the server.
• Ray token authentication support for both the collector and live-cluster proxying.
• Better UI and API coverage: a cluster selection page, task logs, and Serve / placement group / Ray Data endpoints polled by default.
• History Server images are now published to quay.io/kuberay (historyserver:v1.7.0 and collector:v1.7.0)

Try it here: History Server Quick Start Guide.

Automatic History Server Sidecar Configuration (alpha)

The operator can now automatically inject the History Server event collector sidecar into your RayCluster’s pods, no manual sidecar configuration needed. Enable the RayClusterHistoryServer feature gate on the operator and set spec.historyServerOptions on your RayCluster:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-historyserver
spec:
  historyServerOptions:
    collectorOptions:
      image: quay.io/kuberay/collector:v1.7.0

See ray-cluster.historyserver.yaml for a full example.

⚠️ Warning: RayClusterHistoryServer is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterHistoryServer=true  

RayService Incremental Upgrade graduates to Beta

The RayServiceIncrementalUpgrade feature gate is now beta and enabled by default. This feature allows RayService to perform zero-downtime upgrades by gradually shifting traffic to the new cluster using the Kubernetes Gateway API, instead of requiring 2x resources for a blue/green switch.

Since its Alpha release, the feature has received several improvements for more reliable and robust upgrades, including:

• Rollback support: Users can now revert the RayService spec mid-upgrade, with safe traffic shifting between the pending and active clusters.
• Improved reliability: Improved capacity management, traffic migration, failure handling, and cluster cleanup during incremental upgrades.
• E2E test coverage: Expanded test scenarios to improve coverage.

mTLS support via cert-manager (alpha)

KubeRay v1.7 can automatically provision and rotate certificates for mutual TLS between RayCluster’s pods using cert-manager.

⚠️ Warning: RayClusterMTLS is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterMTLS=true  

Enable mTLS feature with following to try this feature:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-mtls
spec:
  rayVersion: '2.55.1'
  tlsOptions:
    enabled: true

See ray-cluster.mtls.yaml for a full example.

NetworkPolicy support for RayCluster (alpha)

KubeRay v1.7 adds a new spec.networkPolicy field to RayCluster for network isolation. When set, the operator creates separate NetworkPolicies for head and worker pods, with configurable modes (DenyAll, DenyAllIngress, DenyAllEgress) and per-worker-group ingress/egress rules. Intra-cluster pod-to-pod traffic is always permitted.

Try it here: Network Policy Quick Start Guide.

⚠️ Warning: RayClusterNetworkPolicy is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterNetworkPolicy=true  

Below is an example of enabling network policy feature:

apiVersion: ray.io/v1
kind: RayCluster
spec:
  networkPolicy:
    mode: DenyAll
   head:
      ingressRules:
      - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <prometheus-namespace>
        ports:
        - port: 8080
          protocol: TCP

See ray-cluster.network-policy-deny-all.yaml for an example.

GCS fault tolerance with embedded storage (alpha)

GCS fault tolerance no longer requires an external Redis. With the GCSFaultToleranceEmbeddedStorage feature gate enabled, you can persist GCS metadata in an embedded RocksDB store backed by a persistent volume on the head Pod:

spec:
  gcsFaultToleranceOptions:
    backend: rocksdb
    storage:
      size: 1Gi

⚠️ Warning: GCSFaultToleranceEmbeddedStorage is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=GCSFaultToleranceEmbeddedStorage=true  

See ray-cluster.embedded-gcs-ft.yaml for a full example.

Other Notable Features

• RayCluster now supports built-in Ingress configuration via headGroupSpec.ingressOptions with host, path, pathType, and tls fields.
• RayCronJob now supports the spec.timeZone field, and child RayJobs use deterministic names to prevent duplicate firings.
• The WorkerGroup API includes a new priority field for scheduler integrations.
• A new alpha feature gate SidecarSubmitterRestart prevents premature job termination during transient head Pod resource spikes in RayJob sidecar mode.
• The operator configuration supports defaultPodAnnotations and defaultPodLabels applied to all Ray Pods.
• Memory usage improvements in the operator: Pod informer cache selectors and more efficient pod resource calculation.
• kubectl ray logs support --gke-link to print a Cloud Logging link on GKE.
• Upgraded to Go 1.26; Grafana dashboards updated for Ray 2.56.

Breaking Changes

• Helm: in the kuberay-operator chart, metrics.serviceMonitor.selector has been renamed to metrics.serviceMonitor.additionalLabels, and empty labels are no longer emitted (#4979). Update your values file if you set this field.
• History Server (alpha to beta): configuration has changed since the v1.6 alpha. S3 credentials now use the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, ...) instead of AWS_S3ID/AWS_S3SECRET/AWS_S3TOKEN (#4665), the runtime-class flag has been replaced by --storage-backend (#5085), and the on-storage event layout has changed (#4670). Redeploy the collector and history server together on v1.7 and see the updated setup guide.

Deprecations

• The ray.io/v1alpha1 API version is now marked deprecated (#5122). Please migrate to ray.io/v1.

Changelog

• Disable RayMultiHostIndexing feature for TestReconcile_Multihost_Replicas by @Future-Outlier in #4583
• Disable the field alignment lint check by @jinbum-kim in #4560
• Mark RC releases as pre-release in GoReleaser by @Future-Outlier in #4590
• [RayCluster] Add example YAML for manually enabling Ray k8s auth by @andrewsykim in #4582
• [RayService] Rollback Support for Incremental Upgrades by @ryanaoleary in #4109
• [RayService] Promote Incremental Upgrade Feature to Beta by @ryanaoleary in #4599
• Revert "[RayService] Promote Incremental Upgrade Feature to Beta" by @Future-Outlier in #4602
• [Github Action] Skip krew-index update for pre-release tags by @Future-Outlier in #4587
• [Helm] update ray-cluster chart to apiVersion: v2 by @mboersma in #4593
• Update kind configs and docs to v1.29.0 by @mboersma in #4595
• [Helm] remove ingress template support for k8s < 1.19 by @mboersma in #4591
• [Helm] Update ray-cluster default resource values by @Future-Outlier in #4588
• [Chore] Use RayCluster name as ServiceAccount name for RBAC authentication by @JiangJiaWei1103 in #4611
• [History Server] Add guide to push image to Google Artifact Registry for History Server images by @chiayi in #4618
• [History Server] Fix API response format to match Ray Dashboard frontend schema by @Future-Outlier in #4615
• [Test][Release] Change upgrade test version to test upgrade from 1.5.1 to 1.6.0 by @Future-Outlier in #4623
• [History Server] Typo and minor fix for image guide readme by @chiayi in #4621
• [Fix] Align error msg with the head and serve svc behavior by @JiangJiaWei1103 in #4614
• [Refactor] [history server] Rename event collector to align with REP by @JiangJiaWei1103 in #4574
• [fix] Fix flaky RayJob e2e test by cleaning up resources after fail subtest by @seanlaii in #4603
• [Fix] [Flaky] Use head node ID to get event_JOBS logs by @JiangJiaWei1103 in #4626
• [Test] Add load tests and behavioral checks to incremental upgrade E2E by @JiangJiaWei1103 in #4541
• Remove FieldIndexer on PodUID that is not used in raycluster controller by @troy0820 in #4629
• [KubeRay DashBoard] Fix history server rayjob yaml and raycluster yaml cause error by @CheyuWu in #4640
• [History Server] Add Google Cloud storage history server deployment example by @chiayi in #4641
• Update the codegen script to be vendor-aware by @jinbum-kim in #4394
• [History Server] Add Google Cloud Storage RayCluster manifest example for History Server by @chiayi in #4642
• [History Server] Fix image & env var for Google Cloud Storage example yaml. by @chiayi in #4655
• [Helm] use service.ports list for apiserver ingress backend port by @mboersma in #4609
• [ray-operator] add example YAML for using Redis as a sidecar with persistence by @andrewsykim in #4657
• [historyserver] Fix S3 client to use default credential chain when static credentials are not provided by @ashutosh1807 in #4653
• [Fix] [Flaky] Insert firewall rules to the correct table by @JiangJiaWei1103 in #4662
• [ray-operator] remove TODO for setting service account token audiences by @andrewsykim in #4666
• [Refactor] Use IsJobDeploymentTerminal for EndTime update by @JiangJiaWei1103 in #4646
• [Refactor] [history server] Align tasks/timeline generation with Ray Dashboard by @AndySung320 in #4546
• [Fix] Fail RayService if StepSizePercent exceeds MaxSurgePercent by @JiangJiaWei1103 in #4663
• [historyserver] Migrate S3 credentials to standard AWS environment variables by @ashutosh1807 in #4665
• [History Server] Change RayClusterID on collector side to RayClusterNamespace by @chiayi in #4673
• [Fix] [RayService] Check serve label before iptables injection in upgrade test by @Future-Outlier in #4677
• Example high throughput LLM serving RayService by @spencer-p in #4684
• [Fix] Add timeout to Ray HTTP proxy client for k8s proxy mode by @JiangJiaWei1103 in #4680
• feat(historyserver): upgrade Aliyun OSS Go SDK from v1 to v2 by @ChenYi015 in #4687
• CARRY: refactor isopenshift check by @laurafitzgerald in #4365
• [kubectl plugin] kubectl ray logs --link=gke by @spencer-p in #4683
• chore(samples): remove unmaintained RayJob Modin example YAML by @EagleLo in #4700
• Fix file matching in Aliyun OSS GetContent by @utafrali in #4692
• [Docs] Clarify per-cluster serve service behavior by @JiangJiaWei1103 in #4699
• [RayService][e2e] fix flaky test in getting HTTPRoute by @fscnick in #4701
• [History Server] Fix /logical/actors response format to match Ray Dashboard API by @win5923 in #4563
• [RayService][e2e] fix flaky test TestAutoscalingRayService by @AndySung320 in #4702
• [golangci-lint] Enable deprecated API warnings linter by @mboersma in #4605
• [Config]: standardize quoting style in Kubernetes manifests (#4353) by @DejusDevspace in #4359
• [PodPool-VK] add main entry function for cache pod manager (#4250) by @hzyfox in #4584
• Quick fix for kubectl ray logs --link=gke by @spencer-p in #4725
• [Docs] Add History Server local development guide by @JiangJiaWei1103 in #4719
• fix delete volcano PodGroup unconditionally by @fscnick in #4669
• [History Server] refactor event collector to use path/filepath for local FS paths by @400Ping in #4514
• Avoid nil dereference by @dentiny in #4737
• Fix Issue #4686 (rayservice): update RayCluster when Suspend toggles by @lorriexingfang in #4739
• Include error in panic by @dentiny in #4747
• [Refactor][history server] Replace all hardcoded root temporary paths with path constants by @400Ping in #4548
• [ray-operator] Export additional client go metrics by @spencer-p in #4764
• fix: include metadata in RayJob submit command when using clusterSelector by @EagleLo in #4616
• RayService incremental upgrade reliability improvements by @ryanaoleary in #4601
• [Fix] [History Server] Normalize hex ID for data integrity by @JiangJiaWei1103 in #4775
• [Test] Fix race in DELETE_RAYJOB_CR_AFTER_JOB_FINISHES envtest by @marosset in #4790
• Avoid dashboard hard-code namespace to apiserver by @dentiny in #4794
• Upgrade to golang v1.26 from v1.25 by @hango880623 in #4597
• [RayJob] revert 4160 background goroutine get job info by @fscnick in #4766
• Go 1.26 and K8s 1.36 client-go updates by @marosset in #4703
• [RayJob][volcano] consistent naming on the return of calculatePodGroupParams by @fscnick in #4734
• Add Pod informer cache selector to reduce memory usage by @AndySung320 in #4761
• [RayJob] Add GCS health wait to K8sJobMode submitter by @seanlaii in #4798
• fix: HistoryServer unit tests failure caused by ID normalization in ConvertBase64ToHex by @kenchung285 in #4769
• Update community meeting info in README.md by @marosset in #4803
• [RayJob] Replace DefaultReadinessProbeFailureThreshold with RayDashboardGCSHealthCheckTimeoutSeconds by @seanlaii in #4800
• update doc string for K8sControllerRuntimeCacheSelectors by @AndySung320 in #4802
• [kubectl-plugin][Refactor] Standardize namespace resolution to respect kubeconfig context. by @kash2104 in #4367
• [Doc] Update text-summarizer sample to ray-ml 2.49.2 by @kevin85421 in #4811
• Fix dashboard error display by @dentiny in #4806
• [Autoscaler][Bug] inject RAY_enable_autoscaler_v2=false when user set autoscalerOptions.version=v1 by @justinyeh1995 in #4810
• [history server] Display history task logging by @dentiny in #4818
• [CI] Add History Server Build and Test to Github Actions by @kenchung285 in #4776
• Add RayService finalizer by @yaroslava-serdiuk in #4807
• [Docs] Remove comments in JSON by @JiangJiaWei1103 in #4826
• [apiserver][test] Trust E2E_API_SERVER_RAY_IMAGE verbatim (closes #4346) by @1fanwang in #4770
• [CI] Enable dependabot for github actions by @dentiny in #4848
• Feat/add autoscaler start cmd by @fscnick in #4835
• [ray-operator] Allow disabling log_monitor via rayStartParams by @raulchen in #4857
• [1/N] [History Server Beta] [Feat] Lazy loading by @JiangJiaWei1103 in #4795
• Bump the all-actions group across 1 directory with 16 updates by @dependabot[bot] in #4854
• chore(deps): bump idna from 3.10 to 3.15 in /clients/python-client by @dependabot[bot] in #4861
• chore(deps): bump next from 15.4.10 to 15.5.18 in /dashboard by @dependabot[bot] in #4862
• chore(deps): bump ip-address from 10.0.1 to 10.2.0 in /dashboard by @dependabot[bot] in #4870
• [History Server] Add compression library for history server by @chiayi in #4828
• [ray-operator] support overriding ulimit open files with env var by @andrewsykim in #4860
• [RayService][Kueue] Support top-level Spec.Suspend for zero-downtime upgrade by @kevi

Note truncated.