NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #3333 most downloaded on PyPI
SkyPilot: Manage all your AI compute.
Latest stable
0.14.2
Last release 25 days ago
28 Oct 2026
Release timing varies
gaps range from 8 days to 2 months
Nearly every release is documented
notes for 41 of 44 stable releases
2 versions withdrawn
withdrawn after publishing
4 years old
65 releases · first in 2022
Drag across the ruler to compare, or tap a release to open it.
uv pip install " skypilot>=0.14.2 "
Get it now with:
uv pip install "skypilot>=0.14.2"Full Changelog: v0.14.1...v0.14.2
Get it now with:
uv pip install "skypilot>=0.14.2"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.14.1...v0.14.2
uv pip install " skypilot>=0.14.1 "
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Get it now with:
uv pip install "skypilot>=0.14.1"Full Changelog: v0.14.0rc1...v0.14.1
Get it now with:
uv pip install "skypilot>=0.14.1"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.14.0rc1...v0.14.1
Deprecations & Breaking Changes :
SkyPilot v0.14.0 introduces dynamic job groups, letting a running job group launch more managed jobs that are tracked and cancelled with it, and makes Slurm a multi-user backend: jobs are submitted as the authenticated user, with per-user QoS/account settings and stop/start for container clusters. This release also adds AWS Trainium/Inferentia (Neuron) on Kubernetes, read-only visibility for private workspaces, a conda-free default image, and a set of opt-in API server scalability features for large deployments.
Get it now with:
uv pip install "skypilot[all]==0.14.0"Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.14.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-valuesDeprecations & Breaking Changes:
provision.install_conda now defaults to false, and conda has been removed from the default Kubernetes images, which instead ship system Python plus a default user venv (~/sky-user-env). Set provision.install_conda: true to opt back in; the opt-in install now uses a current Miniforge with conda-forge as the only channel.inter_connection: false on the group to allow cross-cluster placement.sky debug-dump is admin-only (#10277), download paths are authorized against the user's own roots (#10683), principals with an unrecognized role are denied (#10501), and non-admins can no longer grant the admin role via service accounts (#9932, #10482).disk_size instead of the ephemeral_storage field (#9273).A task in a job group can now launch further managed jobs with sky jobs launch or sky.jobs.launch, and they attach to the group automatically as dynamic tasks (#10698, #10701, #10702, #10725, #10734, #10736, #10732). Dynamic tasks are listed under the group in sky jobs queue and the dashboard, numbered after the group's own tasks, addressed like any other task (sky jobs logs 42 3, sky jobs cancel 42 --task 3), and cancelled together with the group. A common use is an eval watcher that launches one evaluation job per checkpoint while the trainer keeps training:
---
name: train-and-eval
execution: parallel
primary_tasks: [trainer, eval-watcher]
---
name: trainer
resources:
accelerators: H100:8
run: python train.py --checkpoint-dir /checkpoints
---
name: eval-watcher
resources:
cpus: 2
run: |
for ckpt in $(python watch_checkpoints.py /checkpoints); do
sky jobs launch -y -d -n "eval-$ckpt" eval.yaml --env CKPT=$ckpt
done$ sky jobs queue
ID TASK NAME ... STATUS
42 - train-and-eval ... RUNNING
↳ 0 trainer [P] ... RUNNING
↳ 1 eval-watcher [P] ... RUNNING
↳ 2 eval-step-1000 ... SUCCEEDED
↳ 3 eval-step-2000 ... RUNNINGJob groups also gain an inter_connection: true | false knob (#10297): networking is a real guarantee when enabled, and false skips the networking machinery entirely so tasks can be placed across clusters or clouds.
Slurm clusters shared through one API server can now run each user's work under their own Unix identity (#10294, #10477, #10685, #10686). With slurm.submit_as_user, provisioning, srun, rsync, cancellation, and SSH all run as the mapped user, so Slurm accounting, fairshare, and file permissions apply per person. Service accounts default to their creator's identity. A new slurm.quota block maps to sbatch --qos / --account and resolves per workspace (#10680, #10631).
slurm:
submit_as_user: true
username_map:
jane.doe@example.com: jdoe
inference-prod: inference-svc
quota:
queue: normal # -> #SBATCH --qos=normal
account: pre-training # -> #SBATCH --account=pre-trainingSlurm container (Pyxis) clusters also support sky stop / sky start, snapshotting each rank's filesystem to shared storage and restoring it into a new allocation (#10544), plus autostop (#10645) and host path volume mounts (#10438).
SkyPilot now schedules AWS Neuron accelerators (Trainium and Inferentia) on Kubernetes (#10169). It recognizes the aws.amazon.com/neuron resource, detects accelerator type and count from node labels (including Karpenter), adds the Neuron taint toleration, and lists Neuron devices in sky show-gpus.
sky launch --gpus Trainium2:16 --infra k8sOther new hardware in this release: GCP RTX PRO 6000 Blackwell (G4) (#9634), InfiniBand on Nebius B200/B300 (#10369), OCI GB200/GB300 NCCL profiles for network_tier: best (#10204), and SR-IOV RDMA delivery on Kubernetes via kubernetes.rdma (#10554).
A private workspace can now let non-members see it and its clusters and managed jobs (including logs) without being able to modify anything (#10228). Members and admins keep full access. The same change hardens per-resource write isolation so mutating an existing cluster is always gated by the cluster's own workspace.
workspaces:
private-ws:
private: true
read_access: all # default: allowed_users
allowed_users:
- alice@example.comSet workspace_config.read_access: all to make this the org-wide default.
v0.14.0 adds several opt-in building blocks for running the API server at scale on PostgreSQL:
SKYPILOT_DB_POOL_HOSTPORT / SKYPILOT_DB_POOL_CONNECTION_URI, keeping advisory locks and migrations on a direct connection (#10301, #10463), plus a configurable state-DB pool size (#10791).SKYPILOT_LEADER_ELECTION_BACKEND=lease) (#10347, #10372).SO_REUSEPORT to balance connections across API server workers (SKYPILOT_SERVER_REUSE_PORT) (#10016).WAITING instead of pinning a worker, including jobs-controller launches (#10210, #10291, #10013), and the client tells you why a parked request is waiting (#10534).Managed jobs now explain themselves. sky jobs queue -v shows the latest launch-progress event for a waiting job, such as a Slurm pending reason or a Kubernetes image pull (#10692), and SkyPilot reports where a job's start-up time went, broken down by stage (queueing, provisioning, setup) (#10638). Event logs also record who requested a cluster stop/termination or job cancellation (#10621, #10619, #10661).
$ sky jobs queue -v
ID NAME STATUS ... DETAILS
56266 train STARTING ... Launching (pending: Resources; partition: dev)SKYPILOT_API_SERVER_LOCAL_PORT / sky api start --port and SKY_RUNTIME_DIR (#10213); SKYPILOT_DISABLE_LOCAL_API_SERVER blocks local server autostart (#10093).last_used_at writes (#10391).allowed_users before their first login (#10079); surface the resolved active_workspace to admin policies (#10241)./api/get for requests interrupted by a restart (#10257), and reclaim orphaned request logs (#10675, #10657).storage.existingClaim to attach a pre-provisioned PVC (#10085, #10086), apiService.host to set the bind interface (#10050), configurable clients-dir subPath (#10101)./api/get latency by request name (#10349), client-side DB instrumentation (#10525), SQLite size and local disk metrics (#10363, #10674), SSH round-trip timing (#10656), and event-loop stall attribution (#10523)./metrics runs on its own event loop, stays responsive when a collector hangs, and caches the multiprocess merge (#10503, #10377, #10573, #10575)./federate scoping for a shared Prometheus (#10572, #10603).kubernetes.rdma (#10554), with a pod_config opt-out from OCI RoCE host networking (#10546).provision_timeout (#10252), keep the spinner updating while queued (#10305), and pools now work under Kueue quota (#10718).set_pod_resource_limits after an OOM (#10551).sky local up --num-nodes for multi-node local clusters (#10138); package-manager-agnostic bootstrap and graceful Docker CLI skip on non-Debian images (#10077, #10382).kubernetes.auto_mounts entries take scope: personal | workspace | global (#10411), and the volume listing is filtered to the caller's workspaces (#10531).inter_connection knob (see Highlights).sky jobs queue (#10613) and smaller payloads for field-filtered queries (#10176).--num-jobs without a pool (#10552); fixed partial state writes on Job Group sync failure and parallelized group teardown (#10415).least_load replica decrement and rotation (#10445).sky debug-dump: fast-fail unreachable kube contexts, parallel per-cluster collection, and an overall deadline (#10041, #10161, #9852, #10068).~/.bashrc and missing Git (#10245, #10230).Slurm
username_map, and quota.queue / quota.account (see Highlights).AllocTRES, CPU/memory allocation and usage in node info, and batched inventory queries over one SSH connection (#10285, #10293, #10425); best-effort multi-cluster aggregation (#10284, #10289).SLURM_* env leaking into user scripts (#10309), and containerized runs broken by an exported which function (#10716).AWS
GCP
Azure
Nebius
OCI / RunPod
network_tier: best (#10204).Dashboard
CLI
sky api login works when SKYPILOT_API_SERVER_ENDPOINT is set and follows HTTP to HTTPS redirects (#10404, #10624).sky jobs logs / sky jobs cancel address job-group tasks by index (#10734); fixed sky api logs traceback on 4xx (#10341).typing_extensions cap and dropped pydantic<2.12 (#10649).concurrency_group (#10150), no auto-retry for exclusive steps (#10385), and cancellation mirrored from GitHub (#10760); nightly tagging and Slack notifications (#10192, #10719).Thank you to all contributors who made this release possible!
@DanielZhangQD, @aylei, @zpoint, @kevinmingtarja, @SeungjinYang, @lloyd-brown, @rohansonecha, @cg505, @Michaelvll, @romilbhardwaj, @concretevitamin, and many others.
Note: this is a draft. Use GitHub's "Generate release notes" on the tagged release to finalize the exact contributor @Handles and the "New Contributors" section.
For a complete list of changes, see the commit history.
SkyPilot v0.14.0 introduces dynamic job groups, letting a running job group launch more managed jobs that are tracked and cancelled with it, and makes Slurm a multi-user backend: jobs are submitted as the authenticated user, with per-user QoS/account settings and stop/start for container clusters. This release also adds AWS Trainium/Inferentia (Neuron) on Kubernetes, read-only visibility for private workspaces, and a conda-free default image.
Get it now with:
uv pip install "skypilot[all]==0.14.0"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.14.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
Deprecations & Breaking Changes:
provision.install_conda now defaults to false, and conda has been removed from the default Kubernetes images, which instead ship system Python plus a default user venv (~/sky-user-env). Set provision.install_conda: true to opt back in; the opt-in install now uses a current Miniforge with conda-forge as the only channel.inter_connection: false on the group to allow cross-cluster placement.sky debug-dump is admin-only (#10277), download paths are authorized against the user's own roots (#10683), principals with an unrecognized role are denied (#10501), and non-admins can no longer grant the admin role via service accounts (#9932, #10482).rbac.restrict_config_to_admins now defaults to true (#10042): users with the user role can no longer read the API server config. Set it to false to restore the previous behavior.disk_size instead of the ephemeral_storage field (#9273).A task in a job group can now launch further managed jobs with sky jobs launch or sky.jobs.launch, and they attach to the group automatically as dynamic tasks (#10698, #10701, #10702, #10725, #10734, #10736, #10732). Dynamic tasks are listed under the group in sky jobs queue and the dashboard, numbered after the group's own tasks, addressed like any other task (sky jobs logs 122 3, sky jobs cancel 122 --task 3), and cancelled together with the group. A common use is an eval watcher that launches one evaluation job per checkpoint while the trainer keeps training:
---
name: train-and-eval
execution: parallel
primary_tasks: [trainer, eval-watcher]
---
name: trainer
resources:
accelerators: H100:8
run: python train.py --checkpoint-dir /checkpoints
---
name: eval-watcher
resources:
cpus: 2
run: |
for ckpt in $(python watch_checkpoints.py /checkpoints); do
sky jobs launch -y -d -n "eval-$ckpt" eval.yaml --env CKPT=$ckpt
done
$ sky jobs queue
ID TASK NAME ... STATUS
122 - train-and-eval ... RUNNING
↳ 0 trainer [P] ... RUNNING
↳ 1 eval-watcher [P] ... RUNNING
↳ 2 eval-step-1 ... SUCCEEDED
↳ 3 eval-step-2 ... RUNNING
↳ 4 eval-step-3 ... RUNNING
Job groups also gain an inter_connection: true | false knob (#10297): networking is a real guarantee when enabled, and false skips the networking machinery entirely so tasks can be placed across clusters or clouds.
Slurm clusters shared through one API server can now run each user's work under their own Unix identity (#10294, #10477, #10685, #10686). With slurm.submit_as_user, provisioning, srun, rsync, cancellation, and SSH all run as the mapped user, so Slurm accounting, fairshare, and file permissions apply per person. Service accounts default to their creator's identity. A new slurm.quota block maps to sbatch --qos / --account and resolves per workspace (#10680, #10631).
slurm:
submit_as_user: true
username_map:
jane.doe@example.com: jdoe
inference-prod: inference-svc
quota:
queue: normal # -> #SBATCH --qos=normal
account: pre-training # -> #SBATCH --account=pre-training
Slurm container (Pyxis) clusters also support sky stop / sky start, snapshotting each rank's filesystem to shared storage and restoring it into a new allocation (#10544), plus autostop (#10645) and host path volume mounts (#10438).
SkyPilot now schedules AWS Neuron accelerators (Trainium and Inferentia) on Kubernetes (#10169). It recognizes the aws.amazon.com/neuron resource, detects accelerator type and count from node labels (including Karpenter), adds the Neuron taint toleration, and lists Neuron devices in sky show-gpus.
sky launch --gpus Trainium2:16 --infra k8s
Other new hardware in this release: GCP RTX PRO 6000 Blackwell (G4) (#9634), InfiniBand on Nebius B200/B300 (#10369), OCI GB200/GB300 NCCL profiles for network_tier: best (#10204), and SR-IOV RDMA delivery on Kubernetes via kubernetes.rdma (#10554).
A private workspace can now let non-members see it and its clusters and managed jobs (including logs) without being able to modify anything (#10228). Members and admins keep full access. The same change hardens per-resource write isolation so mutating an existing cluster is always gated by the cluster's own workspace.
workspaces:
private-ws:
private: true
read_access: all # default: allowed_users
allowed_users:
- alice@example.com
Set workspace_config.read_access: all to make this the org-wide default.
Managed jobs now explain themselves. sky jobs queue -v shows the latest launch-progress event for a waiting job, such as a Slurm pending reason or a Kubernetes image pull (#10692), and, with API server metrics enabled, SkyPilot exports where a job's start-up time went as metrics broken down by stage (controller queue, provisioning, node start-up, setup) (#10638). Event logs also record who requested a cluster stop/termination or job cancellation (#10621, #10619, #10661).
$ sky jobs queue -v
ID NAME STATUS ... DETAILS
56266 train STARTING ... Launching (pending: Resources; partition: dev)
SKYPILOT_API_SERVER_LOCAL_PORT / sky api start --port and SKY_RUNTIME_DIR (#10213); SKYPILOT_DISABLE_LOCAL_API_SERVER blocks local server autostart (#10093).last_used_at writes (#10391).allowed_users before their first login (#10079); surface the resolved active_workspace to admin policies (#10241)./api/get for requests interrupted by a restart (#10257), and reclaim orphaned request logs (#10675, #10657).SKYPILOT_MEMORY_AWARE_WORKER_SIZING) (#10550); fixed retention-daemon cadence (#10538) and hourly requests GC (#10555).storage.existingClaim to attach a pre-provisioned PVC (#10085, #10086), apiService.host to set the bind interface (#10050), configurable clients-dir subPath (#10101).SO_REUSEPORT for spreading connections across API server workers (#10016)./api/get latency by request name (#10349), client-side DB instrumentation (#10525), SQLite size and local disk metrics (#10363, #10674), SSH round-trip timing (#10656), and event-loop stall attribution (#10523)./metrics runs on its own event loop, stays responsive when a collector hangs, and caches the multiprocess merge (#10503, #10377, #10573, #10575)./federate scoping for a shared Prometheus (#10572, #10603).kubernetes.rdma (#10554), with a pod_config opt-out from OCI RoCE host networking (#10546).provision_timeout and show when a launch is waiting for queue admission (#10252), and pools now work under Kueue quota (#10718).set_pod_resource_limits after an OOM (#10551).sky local up --num-nodes for multi-node local clusters (#10138); package-manager-agnostic bootstrap and graceful Docker CLI skip on non-Debian images (#10077, #10382).kubernetes.auto_mounts entries take scope: personal | workspace | global (#10411), and the volume listing is filtered to the caller's workspaces (#10531).inter_connection knob (see Highlights).sky jobs queue (#10613) and smaller payloads for field-filtered queries (#10176).--num-jobs without a pool (#10552); fixed partial state writes on Job Group sync failure and parallelized group teardown (#10415).least_load replica decrement and rotation (#10445).sky debug-dump: fast-fail unreachable kube contexts, parallel per-cluster collection, and bounded controller steps so a stuck controller cannot hang the dump (#10041, #10161, #10068).~/.bashrc and missing Git (#10245, #10230).Slurm
username_map, and quota.queue / quota.account (see Highlights).AllocTRES, CPU/memory allocation and usage in the Slurm node-info API, and batched inventory queries over one SSH connection (#10285, #10293, #10425); best-effort multi-cluster aggregation (#10284, #10289).SLURM_* env leaking into user scripts (#10309), and containerized runs broken by an exported which function (#10716).AWS
GCP
Azure
Nebius
OCI / RunPod
network_tier: best (#10204).Dashboard
CLI
sky api login works when SKYPILOT_API_SERVER_ENDPOINT is set and follows HTTP to HTTPS redirects (#10404, #10624).sky jobs logs / sky jobs cancel address job-group tasks by index (#10734); fixed sky api logs traceback on 4xx (#10341).typing_extensions cap and dropped pydantic<2.12 (#10649).concurrency_group (#10150), no auto-retry for exclusive steps (#10385), and cancellation mirrored from GitHub (#10760); nightly tagging and Slack notifications (#10192, #10719).Thank you to all contributors who made this release possible!
@DanielZhangQD, @aylei, @zpoint, @kevinmingtarja, @ishankaul1, @SeungjinYang, @lloyd-brown, @rohansonecha, @cg505, @stephenkevn, @Michaelvll, @MadhavJivrajani, @trevanth24, @hopechong, @ryxli, @Anai-Guo, @liuxu623, @romilbhardwaj, @ciprianf, @hentt30, @ahluwalij, @alex000kim, @complynx, @concretevitamin, @davanstrien, @daw3rd, @edmsimplicio, @himanshu1573, @intagliated, @JGSweets, @Jobarion, @Khas-Erdene-Tsogtsaikhan, @klenovda, @kyuds, @nicosuter, @pauldonzier, @pavneeta, @samuelv8, @spiffaz, @typesafeainate, @yash170106065
For a complete list of changes, see the commit history.
`bash uv pip install "skypilot>=0.14.0rc1" `
Get it now with:
uv pip install "skypilot>=0.14.0rc1"
replace_skypilot_config when the body raises by @kevinmingtarja in https://github.com/skypilot-org/skypilot/pull/10236resolved_config by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/10270sky api logs traceback on 4xx from /api/stream by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/10341sky api login when SKYPILOT_API_SERVER_ENDPOINT is set by @zpoint in https://github.com/skypilot-org/skypilot/pull/10404--num-jobs without a pool by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/10552proc by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/10593which bash function by @daw3rd in https://github.com/skypilot-org/skypilot/pull/10716Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.13.1rc1...v0.14.0rc1
uv pip install " skypilot>=0.13.2rc1 "
Get it now with:
uv pip install "skypilot>=0.13.2rc1"replace_skypilot_config when the body raises by @kevinmingtarja in #10236resolved_config by @DanielZhangQD in #10270sky api logs traceback on 4xx from /api/stream by @DanielZhangQD in #10341Note truncated.
`bash uv pip install "skypilot>=0.13.1rc1" `
Get it now with:
uv pip install "skypilot>=0.13.1rc1"
disk_size instead of ephemeral_storage field by @kyuds in https://github.com/skypilot-org/skypilot/pull/9273Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.13.0rc1...v0.13.1rc1
Deprecations & Breaking Changes:
SkyPilot v0.13.0 introduces Sky Batch for large-scale batch inference and data processing across worker pools, a native Hugging Face storage backend (hf://), a generalized lifecycle-hooks framework for running custom logic around cluster events, and the sky debug-dump command for one-shot troubleshooting. This release also adds per-instance cost caps, NVIDIA Blackwell / CUDA 13 images, and GKE Autopilot support.
Get it now with:
uv pip install "skypilot[all]==0.13.0"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.13.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
Deprecations & Breaking Changes:
api_access renamed to api_server_access, now defaulting to True (#9169). Tasks now automatically receive API server credentials. The legacy api_access key is still accepted for backward compatibility.sky.jobs.queue(version=1) remains deprecated and continues to emit a warning; use sky.jobs.queue(version=2), which returns richer job metadata as dictionaries (#9118).all_includes_in_cluster from the SkyPilot config schema (#9671).**sky batch** brings large-scale batch inference and data processing to SkyPilot with a simple Python API (#8850, #9438, #9456). Given a dataset and a processing function, Sky Batch splits the data into batches, distributes them across a pool of GPU/CPU workers, and collects the results, with no manual sharding, provisioning, or failure handling. Workers are reused across jobs, so expensive setup (installing packages, downloading weights, loading models onto GPUs) happens only once.
import sky
# Point at your input data in cloud storage
ds = sky.batch.Dataset(source='s3://my-bucket/prompts/')
# Map a processing function over the dataset across a worker pool
ds.map(my_inference_fn, pool='my-gpu-pool', output='s3://my-bucket/results/')
Inputs and outputs can be read from and written to Amazon S3 (s3://) or Google Cloud Storage (gs://).
SkyPilot now supports Hugging Face as a first-class storage backend via the hf:// scheme (#9698, #9763). It wraps read-write Hugging Face Buckets (Xet-backed, S3-like object storage) and, through the same handle, read-only mounting of model, dataset, and space repos. Bucket lifecycle operations go through huggingface_hub, and MOUNT/MOUNT_CACHED modes use the FUSE-based hf-mount, consistent with SkyPilot's existing mount infrastructure.
file_mounts:
# Read-write Hugging Face Bucket
/data:
source: hf://buckets/my-org/my-bucket
# Read-only model / dataset / space repo
/model:
source: hf://meta-llama/Llama-3.1-8B
mode: MOUNT
URL formats: hf://buckets/<ns>/<bucket> (bucket, read-write), hf://<ns>/<model>[@<rev>], hf://datasets/<ns>/<ds>, and hf://spaces/<ns>/<sp> (repos, read-only). Authentication reads HF_TOKEN from the environment or the Hugging Face CLI token file.
The autostop hook introduced in v0.12 has grown into a generalized lifecycle-hooks framework via resources.hooks (#9064). Each hook declares a script and the events it fires on: autostop, preemption, and down. Use it to checkpoint state, sync experiment trackers, or send notifications around cluster lifecycle transitions.
resources:
hooks:
- run: |
wandb sync
curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
events: [autostop, down]
timeout: 300
The legacy autostop.hook YAML continues to work with a deprecation warning.
sky debug-dump: One-Shot TroubleshootingThe new **sky debug-dump** command collects diagnostic information (logs, state, and configuration) for clusters, requests, and managed jobs into a single zip for troubleshooting and bug reports (#8868, #9752, #9753). Dumps are stored under ~/.sky/debug_dumps/ and automatically cross-link related resources (e.g. a cluster and its associated requests) so you capture complete context in one command.
sky debug-dump --cluster my-cluster
You can now set a per-instance hourly budget cap with resources.max_hourly_cost (#9132, #9170). Only instances at or below the limit are considered during optimization. When use_spot is set, the cap applies against spot prices; otherwise against on-demand prices.
resources:
accelerators: A100
max_hourly_cost: 10.0
SkyPilot's default GPU image now ships NVIDIA driver 580 (open) + CUDA 13 for Blackwell support (#9639), with a legacy CUDA 12 image retained for pre-Turing GPUs and an arm64 build fix (#9789). RunPod adds RTX PRO Blackwell GPUs to its catalog (#9187).
SkyPilot now detects GKE Autopilot clusters and trusts Node Auto-Provisioning (NAP) instead of walking existing node pools, so GPU requests provision correctly on Autopilot instead of being incorrectly rejected (#9565).
MIN_COMPATIBLE_API_VERSION bump for 0.12 clients (#9121)._next/ assets instead of index.html (#9722); don't cache error responses on /dashboard/_next paths (#9498).async_creator (#9483), put_async for the queue backend (#9346).enabled_clouds and sky check (#9284), cached permission checks (#9125).devMode flag for in-cluster iteration (#9609), persist the whole sky_logs dir (#9454), optional server log mirror (#9754), configurable probes (#9477), disable session affinity on ingress (#9347), option to disable serving the server log (#9140).autopilot.enabled detection and NAP trust (#9565).context_configs (#9640).**ephemeral-storage requests/limits** on pods (#9053), allowed_nodes config for node-level filtering (#9151), pod spread constraints (#9296).pod_config to override runtimeClassName for GPU pods (#9241); probe Ray ports when hostNetwork: true (#9644).network_tier: best for OCI OKE RoCE (#9622); allow excluding the in-cluster context from allowed_contexts: all (#9637).volumeMounts patch-merge key from name to mountPath (#9358), write permissions for all volume mounts at startup (#9221), owner-identity check for scoped kubeconfig names (#9725, #9190). Updated default/base k8s images (#9618, #9590).k8s-hostpath volume type for node-local cache (#9165).access_mode from an existing PVC for use_existing volumes (#9359); check the existing backend when creating volumes (#9104); add volume mount check (#9343).Ti units for volume sizes ≥ 1024Gi (#9369); don't rewrite region to in-cluster when in-cluster auth is unavailable (#9743).hf://) for buckets and repos (#9698, #9763).MOUNT (#9225).S3CompatibleStore (#9080); fixed git "dubious ownership" during workdir sync (#9352).__pycache__ from the built wheel (#9413).file_mounts destinations again (#9537); fixed cluster re-provisioning and atomic SSH-key writes (#9476); prioritize user SSH keys over cluster keys in Host * config (#9099)./dev/fuse fd after passing it to the client (#9463); fixed a flock leak in the uv_spawn child process (#9431); fixed browser opening on WSL for sky api login (#9178).SKY_RUNTIME_DIR and removed import-time filesystem side effects (#9605); let packages expose client SDKs under the sky namespace (#9784).**--tail for sky jobs logs** (#9449), preload_content for sky.jobs.tail_logs (#9662), and a wait helper in the programmatic SDK (#9138).sky jobs launch (#9761, #9135, #9191); optionally merge cluster launch-progress events into job events (#9804).ServiceStatusRunner extension point (#9801), new user-configurable options (#9116), and improved max-services error messages (#8897).AWS
aws login (2026 AWS CLI command) (#9526) and subnet selection (#9127).subnet_names span multiple AZs (#9791); added AZ-fetch timeouts to prevent sky check hangs (#9244, #9240).GCP
provisioningModel=RESERVATION_BOUND for DENSE reservations (#9779); added missing required instance types to the catalog (#9472); pinned/bumped the gcloud SDK (#9587, #9535); sped up sky check (#9095).Nebius
open_ports replacing UFW (#9563); use managed disks to create instances (#9432); parallelize multi-node VM creation (#9570); custom image_id (#9133); fixed autodown (#9573).Slurm
cpu_partition config for CPU-only tasks (#9281) and gpu_partition_map for GRES without a GPU type (#9237).--time and fall back to DefaultTime when MaxTime is UNLIMITED (#9633); preserve SLURM_CONF* env vars when launching srun (#9425).sbatch (#9174), numeric partition names treated as integers (#9168), and --mem when memory is not consumable (#9171).RunPod / Vast / PrimeIntellect
create_template (#9550).vastai-sdk v6+ (#9314); made account-level SSH key registration best-effort (#9631).query_instances TypeError on retry (#9259) and instance update after _get_instance_info (#9327).sky ssh up (#9402).Dashboard
allowed_users (#9696); show counts above the users and service-account tables (#9655)./status when the controller is up (#9645).CLI
**sky api login --no-browser** for headless environments, continuing to poll when a browser can't be opened (#9677, #9676).**sky debug-dump** troubleshooting command (#8868).cost-report (#9504).kubernetes < 36 due to an RBAC regression, while accepting dict[K, V] openapi_types for kubernetes >= 36 (#9685, #9682).azure-cli < 2.87.0 (azure-mgmt-storage 25.0.0 breaks Azure storage) (#9774).pyopenssl upper limit (#8070); pinned click < 8.3.0 in relevant runtime installs (#9459, #9763).MOUNT_CACHED config documentation (#9130); api_server_access documentation (#9115); refreshed managed jobs, quickstart, and agent-skills docs (#9564, #9250).Thank you to all contributors who made this release possible!
@aylei, @Michaelvll, @DanielZhangQD, @zpoint, @kevinmingtarja, @lloyd-brown, @concretevitamin, @cg505, @SeungjinYang, @rohansonecha, @kevinzwang, @kyuds, @romilbhardwaj, @alex000kim, @andylizf, @cblmemo, @hentt30, @williamsnell, @Jobarion, and many others.
Note: this is a draft. Use GitHub's "Generate release notes" on the tagged release to finalize the exact contributor @handles and the "New Contributors" section.
For a complete list of changes, see the commit history.
Deprecations & Breaking Changes:
SkyPilot v0.13.0rc1 introduces Sky Batch for large-scale batch inference and data processing across worker pools, a native Hugging Face storage backend (hf://), a generalized lifecycle-hooks framework for running custom logic around cluster events, and the **sky debug-dump** command for one-shot troubleshooting. This release also adds per-instance cost caps, NVIDIA Blackwell / CUDA 13 images, and GKE Autopilot support.
Get it now with:
uv pip install "skypilot[all]==0.13.0rc1"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.13.0rc1
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
Deprecations & Breaking Changes:
**api_access renamed to api_server_access, now defaulting to True** (#9169). Tasks now automatically receive API server credentials. The legacy api_access key is still accepted for backward compatibility.**sky.jobs.queue(version=1) remains deprecated** and continues to emit a warning; use sky.jobs.queue(version=2), which returns richer job metadata as dictionaries (#9118).all_includes_in_cluster from the SkyPilot config schema (#9671).**sky batch** brings large-scale batch inference and data processing to SkyPilot with a simple Python API (#8850, #9438, #9456). Given a dataset and a processing function, Sky Batch splits the data into batches, distributes them across a pool of GPU/CPU workers, and collects the results, with no manual sharding, provisioning, or failure handling. Workers are reused across jobs, so expensive setup (installing packages, downloading weights, loading models onto GPUs) happens only once.
import sky
# Point at your input data in cloud storage
ds = sky.batch.Dataset(source='s3://my-bucket/prompts/')
# Map a processing function over the dataset across a worker pool
ds.map(my_inference_fn, pool='my-gpu-pool', output='s3://my-bucket/results/')
Inputs and outputs can be read from and written to Amazon S3 (s3://) or Google Cloud Storage (gs://).
SkyPilot now supports Hugging Face as a first-class storage backend via the hf:// scheme (#9698, #9763). It wraps read-write Hugging Face Buckets (Xet-backed, S3-like object storage) and, through the same handle, read-only mounting of model, dataset, and space repos. Bucket lifecycle operations go through huggingface_hub, and MOUNT/MOUNT_CACHED modes use the FUSE-based hf-mount, consistent with SkyPilot's existing mount infrastructure.
file_mounts:
# Read-write Hugging Face Bucket
/data:
source: hf://buckets/my-org/my-bucket
# Read-only model / dataset / space repo
/model:
source: hf://meta-llama/Llama-3.1-8B
mode: MOUNT
URL formats: hf://buckets/<ns>/<bucket> (bucket, read-write), hf://<ns>/<model>[@<rev>], hf://datasets/<ns>/<ds>, and hf://spaces/<ns>/<sp> (repos, read-only). Authentication reads HF_TOKEN from the environment or the Hugging Face CLI token file.
The autostop hook introduced in v0.12 has grown into a generalized lifecycle-hooks framework via resources.hooks (#9064). Each hook declares a script and the events it fires on: autostop, preemption, and down. Use it to checkpoint state, sync experiment trackers, or send notifications around cluster lifecycle transitions.
resources:
hooks:
- run: |
wandb sync
curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
events: [autostop, down]
timeout: 300
The legacy autostop.hook YAML continues to work with a deprecation warning.
sky debug-dump: One-Shot TroubleshootingThe new **sky debug-dump** command collects diagnostic information (logs, state, and configuration) for clusters, requests, and managed jobs into a single zip for troubleshooting and bug reports (#8868, #9752, #9753). Dumps are stored under ~/.sky/debug_dumps/ and automatically cross-link related resources (e.g. a cluster and its associated requests) so you capture complete context in one command.
sky debug-dump --cluster my-cluster
You can now set a per-instance hourly budget cap with resources.max_hourly_cost (#9132, #9170). Only instances at or below the limit are considered during optimization. When use_spot is set, the cap applies against spot prices; otherwise against on-demand prices.
resources:
accelerators: A100
max_hourly_cost: 10.0
SkyPilot's default GPU image now ships NVIDIA driver 580 (open) + CUDA 13 for Blackwell support (#9639), with a legacy CUDA 12 image retained for pre-Turing GPUs and an arm64 build fix (#9789). RunPod adds RTX PRO Blackwell GPUs to its catalog (#9187).
SkyPilot now detects GKE Autopilot clusters and trusts Node Auto-Provisioning (NAP) instead of walking existing node pools, so GPU requests provision correctly on Autopilot instead of being incorrectly rejected (#9565).
MIN_COMPATIBLE_API_VERSION bump for 0.12 clients (#9121)._next/ assets instead of index.html (#9722); don't cache error responses on /dashboard/_next paths (#9498).async_creator (#9483), put_async for the queue backend (#9346).enabled_clouds and sky check (#9284), cached permission checks (#9125).devMode flag for in-cluster iteration (#9609), persist the whole sky_logs dir (#9454), optional server log mirror (#9754), configurable probes (#9477), disable session affinity on ingress (#9347), option to disable serving the server log (#9140).autopilot.enabled detection and NAP trust (#9565).context_configs (#9640).**ephemeral-storage requests/limits** on pods (#9053), allowed_nodes config for node-level filtering (#9151), pod spread constraints (#9296).pod_config to override runtimeClassName for GPU pods (#9241); probe Ray ports when hostNetwork: true (#9644).network_tier: best for OCI OKE RoCE (#9622); allow excluding the in-cluster context from allowed_contexts: all (#9637).volumeMounts patch-merge key from name to mountPath (#9358), write permissions for all volume mounts at startup (#9221), owner-identity check for scoped kubeconfig names (#9725, #9190). Updated default/base k8s images (#9618, #9590).k8s-hostpath volume type for node-local cache (#9165).access_mode from an existing PVC for use_existing volumes (#9359); check the existing backend when creating volumes (#9104); add volume mount check (#9343).Ti units for volume sizes ≥ 1024Gi (#9369); don't rewrite region to in-cluster when in-cluster auth is unavailable (#9743).hf://) for buckets and repos (#9698, #9763).MOUNT (#9225).S3CompatibleStore (#9080); fixed git "dubious ownership" during workdir sync (#9352).__pycache__ from the built wheel (#9413).file_mounts destinations again (#9537); fixed cluster re-provisioning and atomic SSH-key writes (#9476); prioritize user SSH keys over cluster keys in Host * config (#9099)./dev/fuse fd after passing it to the client (#9463); fixed a flock leak in the uv_spawn child process (#9431); fixed browser opening on WSL for sky api login (#9178).SKY_RUNTIME_DIR and removed import-time filesystem side effects (#9605); let packages expose client SDKs under the sky namespace (#9784).**--tail for sky jobs logs** (#9449), preload_content for sky.jobs.tail_logs (#9662), and a wait helper in the programmatic SDK (#9138).sky jobs launch (#9761, #9135, #9191); optionally merge cluster launch-progress events into job events (#9804).ServiceStatusRunner extension point (#9801), new user-configurable options (#9116), and improved max-services error messages (#8897).AWS
aws login (2026 AWS CLI command) (#9526) and subnet selection (#9127).subnet_names span multiple AZs (#9791); added AZ-fetch timeouts to prevent sky check hangs (#9244, #9240).GCP
provisioningModel=RESERVATION_BOUND for DENSE reservations (#9779); added missing required instance types to the catalog (#9472); pinned/bumped the gcloud SDK (#9587, #9535); sped up sky check (#9095).Nebius
open_ports replacing UFW (#9563); use managed disks to create instances (#9432); parallelize multi-node VM creation (#9570); custom image_id (#9133); fixed autodown (#9573).Slurm
cpu_partition config for CPU-only tasks (#9281) and gpu_partition_map for GRES without a GPU type (#9237).--time and fall back to DefaultTime when MaxTime is UNLIMITED (#9633); preserve SLURM_CONF* env vars when launching srun (#9425).sbatch (#9174), numeric partition names treated as integers (#9168), and --mem when memory is not consumable (#9171).RunPod / Vast / PrimeIntellect
create_template (#9550).vastai-sdk v6+ (#9314); made account-level SSH key registration best-effort (#9631).query_instances TypeError on retry (#9259) and instance update after _get_instance_info (#9327).sky ssh up (#9402).Dashboard
allowed_users (#9696); show counts above the users and service-account tables (#9655)./status when the controller is up (#9645).CLI
**sky api login --no-browser** for headless environments, continuing to poll when a browser can't be opened (#9677, #9676).**sky debug-dump** troubleshooting command (#8868).cost-report (#9504).kubernetes < 36 due to an RBAC regression, while accepting dict[K, V] openapi_types for kubernetes >= 36 (#9685, #9682).azure-cli < 2.87.0 (azure-mgmt-storage 25.0.0 breaks Azure storage) (#9774).pyopenssl upper limit (#8070); pinned click < 8.3.0 in relevant runtime installs (#9459, #9763).MOUNT_CACHED config documentation (#9130); api_server_access documentation (#9115); refreshed managed jobs, quickstart, and agent-skills docs (#9564, #9250).Thank you to all contributors who made this release possible!
@aylei, @Michaelvll, @DanielZhangQD, @zpoint, @kevinmingtarja, @lloyd-brown, @concretevitamin, @cg505, @SeungjinYang, @rohansonecha, @kevinzwang, @kyuds, @romilbhardwaj, @alex000kim, @andylizf, @cblmemo, @hentt30, @williamsnell, @Jobarion, and many others.
Note: this is a draft. Use GitHub's "Generate release notes" on the tagged release to finalize the exact contributor @handles and the "New Contributors" section.
For a complete list of changes, see the commit history.
`bash uv pip install "skypilot>=0.12.3.post1" `
Get it now with:
uv pip install "skypilot>=0.12.3.post1"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.12.3rc1...v0.12.3.post1
`bash uv pip install "skypilot>=0.12.3" `
Get it now with:
uv pip install "skypilot>=0.12.3"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.12.3rc1...v0.12.3
Nothing published for this version
`bash uv pip install "skypilot>=0.12.2.post1" `
Get it now with:
uv pip install "skypilot>=0.12.2.post1"
What's changed?
`bash uv pip install "skypilot>=0.12.2" `
Get it now with:
uv pip install "skypilot>=0.12.2"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.12.1...v0.12.2
Nothing published for this version
`bash uv pip install "skypilot>=0.12.1" `
Get it now with:
uv pip install "skypilot>=0.12.1"
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.12.0...v0.12.1
Nothing published for this version
Fix Zip Slip vulnerability in /upload endpoint (#8723).
SkyPilot v0.12.0 brings major new capabilities: Slurm integration for running SkyPilot on existing Slurm clusters, Job Groups for heterogeneous parallel workloads like RL training, an Agent Skill that teaches AI coding agents to use SkyPilot, Recipes for sharing reusable YAML templates across teams, and significant Pool enhancements including autoscaling. This release also drops Python 3.7/3.8, with Python 3.9+ now required.
Get it now with:
uv pip install "skypilot[all]>=0.12.0"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.12.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
Deprecations & Breaking Changes:
sky.jobs.queue(version=1) is deprecated and will be removed in v0.13. Use sky.jobs.queue(version=2) instead. The new version returns richer job metadata as dictionaries (#9118).SkyPilot now supports connecting your Slurm clusters, bringing its unified interface to one of the most widely used job schedulers in high-performance computing (#5491, #8198, #8219, #8268, #8291, #8470, #8604, #8729, and 25+ additional PRs). Users can launch SkyPilot clusters and managed jobs on Slurm clusters with the same CLI and YAML they use for cloud and Kubernetes, enabling seamless workload portability across all AI infra.
<p align="center"> <img width="80%" alt="SkyPilot dashboard showing Slurm GPU availability" src="https://github.com/user-attachments/assets/2790a276-1fd7-4edb-8643-e04200236dae" /> </p>
Key capabilities include:
sky show-gpus for Slurm partitionssbatch_options in task YAML# Launch on a Slurm cluster
resources:
accelerators: H100:8
infra: slurm
# View GPU availability across Slurm clusters
sky show-gpus --infra slurm
# Launch a training job on Slurm
sky launch --infra slurm/my-cluster train.yaml
SkyPilot now ships an official Agent Skill that teaches AI coding agents—Claude Code, Codex, and others—how to use SkyPilot (#8823, #9017, #9037). With the skill installed, your agent can launch clusters, run managed jobs, serve models, compare GPU pricing, and manage cloud resources—all through natural language.
Install the skill (docs) by telling your agent:
Fetch and follow the install guide https://github.com/skypilot-org/skypilot/blob/HEAD/agent/INSTALL.md
In our Scaling Autoresearch blog post, we gave Claude Code the SkyPilot agent skill and access to a 16-GPU Kubernetes cluster. Over 8 hours, the agent autonomously submitted ~910 experiments in parallel, achieving a 9x speedup over sequential search—and even discovered hardware-specific optimizations on its own.
<p align="center"> <img width="60%" alt="Scaling Autoresearch with SkyPilot Agent Skill" src="https://blog.skypilot.co/scaling-autoresearch/assets/banner.png" /> </p>
Example interactions your agent can now handle:
| Capability | Example Prompt |
|---|---|
| Launch dev clusters | "Launch a cluster with 4 A100 GPUs. Auto-stop after 30 min idle." |
| Fine-tune models | "Fine-tune Llama 3.1 8B on my dataset at s3://my-data. Use spot instances." |
| Distributed training | "Run PyTorch DDP training across 4 nodes with 8 H100s each." |
| Serve models | "Deploy Llama 3.1 70B with vLLM. Autoscale 1-3 replicas based on QPS." |
| Compare pricing | "What's the cheapest 8x H200 across AWS, GCP, Lambda, and CoreWeave?" |
| Multi-cloud failover | "Submit jobs that try our Slurm cluster first and fall back to AWS." |
SkyPilot Job Groups let you define multiple tasks with different resource requirements that run together as a single managed job (#8456, #8664, #8686, #8688, #8713, #8940). This is ideal for reinforcement learning workflows where training, inference, and data servers need different hardware—H100s for policy training, cheaper GPUs for rollout inference, and high-memory CPUs for replay buffers.
SkyPilot provisions all resources together, configures networking automatically with built-in service discovery ({task_name}-{node_index}.{job_group_name}), and manages the full lifecycle as one unit.
# Multi-document YAML for a Job Group
---
name: rl-training
execution: parallel
primary_tasks: [ppo-trainer]
---
name: data-server
resources:
cpus: 4+
run: |
python data_server.py --port 8000
---
name: ppo-trainer
resources:
accelerators: H100:1
run: |
python ppo_trainer.py --data-server data-server-0.rl-training:8000
<p align="center"> <img width="60%" alt="Job Groups architecture" src="https://blog.skypilot.co/job-groups/assets/banner.png" /> </p>
Recipes allow teams to store and share SkyPilot YAMLs in a centralized, team-accessible registry (#8755, #8825, #8851, #8873, #8876, #8882). Launch standardized workloads directly from the CLI or dashboard without local YAML files, reducing DevOps overhead and ensuring consistent configurations across the team.
Recipes support clusters, managed jobs, pools, and SkyServe, with built-in validation that blocks local file dependencies to ensure portability.
# Launch a recipe directly
sky launch recipes:dev-cluster
# Launch with custom overrides
sky launch recipes:gpu-cluster --cpus 16 --gpus H100:4 --env DATA_PATH=s3://my-data
<p align="center"> <img width="80%" alt="Recipes dashboard" src="https://blog.skypilot.co/skypilot-recipes/images/create-recipe.png" /> </p>
SkyPilot Pools receive major upgrades with autoscaling, multiple jobs per worker, heterogeneous pools, and memory-aware scheduling (#8483, #8192, #8315, #8279, #8509, #7891).
# autoscaling-pool.yaml
pool:
min_workers: 0
max_workers: 10
resources:
accelerators: H100
setup: |
echo "Setup complete!"
The SkyPilot dashboard receives significant performance improvements and new observability features (#8523, #8718, #8651, #8534, #8539).
Deploy-mode API servers now auto-enable consolidation mode (#9090), which runs managed job controllers on the API server itself instead of launching separate controller VMs. This eliminates controller overhead costs and simplifies managed job operations for team deployments.
Parallel uploads are now the default for MOUNT_CACHED file mounts, delivering a 7x speedup — flush time dropped from 151s to 21s for a ~14.6 GB test workload (#8455). Advanced tuning options are available via data.mount_cached config (#8810, #8831, #8994).
<p align="center"> <img width="50%" alt="mount-cached-speedup" src="https://github.com/user-attachments/assets/03dab45c-3694-44ba-948e-166ea8a6450b" /> </p>
SkyPilot now automatically configures Elastic Fabric Adapter (EFA) on EKS, delivering ~78.8 GB/s inter-node bandwidth (vs ~4.1 GB/s without EFA) — critical for distributed training performance (#8557, #8771):
resources:
network_tier: best
<p align="center"> <img width="50%" alt="efa-speedup" src="https://github.com/user-attachments/assets/e28b3d47-5378-41b6-a2eb-b8faf496f21a" /> </p>
A new autostop hook mechanism allows running custom scripts before a cluster is automatically stopped — for example, to save checkpoints, sync W&B runs, or send notifications (#8412):
resources:
autostop:
idle_minutes: 10
hook: |
wandb sync
curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
hook_timeout: 300
The SkyPilot dashboard now automatically detects W&B links and other external URLs generated by your AI workloads — no more digging through job logs (#8405).
<p align="center"> <img width="60%" alt="External Links" src="https://github.com/user-attachments/assets/26c38623-60e1-4994-aab6-24d1f29249b6" /> </p>
Users can now specify exit codes that trigger automatic job recovery in managed jobs, useful for transient failures with known error codes (#8324):
resources:
job_recovery:
recover_on_exit_codes: [29]
SkyPilot now auto-detects WSL and seamlessly configures VSCode Remote-SSH for Windows users (#8669).
/upload endpoint (#8723).sky api login (#8590).sky api stop hang on zombie processes in containers (#8839)./users endpoint (#8973).AUTOSTOPPING status to UP for old clients (#9073).fullnameOverride support (#8528), RWX persistent storage with RollingUpdate (#8537), unified ingress resource (#8532), disabling basic auth middleware (#8694), CoreWeave credentials (#8200), DigitalOcean credentials (#7931), SSH node pool config (#8249), Slurm credentials (#8729), AWS config mounting (#8827).remote_identity override in task config (#8659).sky show-gpus and infra dashboard (#8222).skypilot-workspace) for filtering (#8893) and user annotations on k8s pods (#9065).allowed_contexts is set (#8821).sky launch (#8589) and revamped volume background refresh (#8524).MOUNT_CACHED with configurable sequential fallback (#8455).MOUNT_CACHED configurations for workload-specific tuning (#8810, #8831, #8994).vastdata://) (#9041, #9072).--graceful flag for cluster storage operations (#8753).provision.install_conda config to disable conda installation on provisioned nodes (#8662).SKYPILOT_USER environment variable available in jobs (#8747)..sky directory location—put .sky somewhere other than $HOME (#8153).rclone flush script (#8930).sky cancel reliability (#8203), uv >=0.10.5 stripping execute permissions on XFS (#8904), WebSocket SSH proxy timeout under concurrent connections (#9001), openssh version and SetEnv compatibility (#9075).sky down failure (#8460).WORKDIR set to site-packages causing setup failures (#8378).--no-owner --no-group for both uploads and downloads (#8556).RuntimeError: dictionary changed size during iteration in ContextualEnviron (#8962).internal_external_ips and internal_services to ManagedJobRecord for richer job metadata (#8735).AWS
LocalDisk feature: Access NVMe instance storage on supported instance types (#8762, #8807, #8661).GCP
Azure
use_internal_ips, vpc_name, ssh_proxy_command (#8986).Nebius
subnet_id (#8814) and disk tiers (#8905).Vast
RunPod
New Clouds
Other
lstrip('ssh-') bug (#8417), autoscaler config polluting SSH node pools (#9012).Admin Control
UserRequest (#8741).Dashboard
CLI
sky gpus command group with list and label subcommands (#8691).--secret-file option for passing secrets from files (#8646).sky show-gpus (#8576), fix pandas >=3.0.0 compatibility (#8643).ssh even if Ray cluster is in bad state (#8649).sky serve status fails (#8406).--all flag for sky jobs pool status (#8392).volume from sky -h (#8228).sky api info (#9076).sky serve (#8141).Thank you to all contributors who made this release possible!
@aflah02, @alex000kim, @andylizf, @atoniolo76, @aylei, @bilelomrani1, @Bokki-Ryu, @cblmemo, @cg505, @concretevitamin, @dan-blanchard, @DanielZhangQD, @haimmarko-lgtm, @hentt30, @huksley, @ibrahimnd2000, @Jayachander123, @Jobarion, @kevinmingtarja, @kevinzwang, @kyuds, @liuwb, @lloyd-brown, @lucamanolache, @m-braganca, @Michaelvll, @nakinnubis, @ngi, @oelachqar, @oliviert, @otutukingsley, @panf2333, @Philmod, @php-workx, @qicz, @rohansonecha, @romilbhardwaj, @SalikovAlex, @seahyinghang8, @SeungjinYang, @smwaqas89, @williamsnell, @wurambo, @zpoint
New Contributors:
Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!
For a complete list of changes, see the commit history.
Nothing published for this version
Nothing published for this version
Zip Slip vulnerability fix — patched path traversal in /upload endpoint
SkyPilot v0.11.2 delivers Slurm support in Beta, JobGroups for heterogeneous parallel workloads, and significantly enhanced Pools with autoscaling, multi-job scheduling and heterogeneous GPU support. This release also brings Autostop Hooks, 7x MOUNT_CACHED mode uploads speed up, automatic EFA on EKS, and numerous admin, security, and performance improvements.
Get it now with:
uv pip install "skypilot>=0.11.2"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.11.2
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
Breaking Change: Python 3.9+ required — Python 3.7 and 3.8 are no longer supported (#8489). Please upgrade before installing this release.
SkyPilot now supports Slurm as a new infrastructure backend, enabling users to orchestrate workloads on HPC clusters alongside cloud VMs and Kubernetes — all through the same unified interface (docs, #5491, #8138).
This release brings comprehensive Slurm capabilities:
--image-id for reproducible, GPU-accelerated workloads (#8604, #8609)ssh <cluster> drops you inside the Slurm job allocation, so nvidia-smi correctly reflects only your allocated GPUs (#8268)See the Slurm documentation for setup instructions.
<p align="center"> <img width="3024" height="792" alt="image" src="https://github.com/user-attachments/assets/0510257b-c158-4ef1-8432-e7a619a679bc" /> </p>
JobGroups enable running multiple jobs with different resource requirements together as a managed group (blog, #8456). Define multi-task pipelines in a single multi-document YAML file:
# Header: job group metadata
name: rl-training
execution: parallel
primary_tasks: [trainer]
termination_delay: 30s
---
name: trainer
resources:
accelerators: A100:8
run: python train.py
---
name: reward-server
resources:
accelerators: A100:1
run: python reward_server.py
Key capabilities:
sky jobs logs --task-name <name> for viewing specific task logs<p align="center"> <img width="80%" alt="JobGroups banner" src="https://blog.skypilot.co/job-groups/assets/banner.png" /> </p>
Example applications included: RL post-training (RLHF) pipeline and parallel train-eval pipeline.
SkyPilot Pools receive significant upgrades in this release:
min_workers, max_workers, and target queue length; a QueueLengthAutoscaler handles the rest while protecting running jobs from cancellation (#8483)any_of resource configurations and the scheduler dynamically resolves to available hardware (#8315):resources:
any_of:
- accelerators: T4:1
- accelerators: A100:1
--num-jobs argument (#7891)SkyPilot dashboard now automatically detects your W&B links generated by your AI workloads. No need to dig into the job logs to figure out where your training panels are. (#8405)
<p align="center"> <img width="60%" alt="image 1" src="https://github.com/user-attachments/assets/26c38623-60e1-4994-aab6-24d1f29249b6" /> </p>
An autostop hook mechanism allow running custom scripts before a cluster is automatically stopped — for example, to save checkpoints, sync W&B runs, or send Slack notifications (#8412):
resources:
autostop:
idle_minutes:10
hook:|
wandb sync
curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
hook_timeout:300
sky logs --autostop command to view hook execution logssky exec is rejected on AUTOSTOPPING clusters; sky launch waits for autostop to complete before restartingSkyPilot now automatically configures Elastic Fabric Adapter (EFA) on EKS with a single flag (#8557):
resources:
network_tier: best
This automates what was previously a complex manual setup, delivering ~78.8 GB/s inter-node bandwidth (vs ~4.1 GB/s without EFA), critical for distributed training performance. EFA interfaces are allocated proportionally to the requested GPU count.
<p align="center"> <img width="50%" alt="efa-speedup" src="https://github.com/user-attachments/assets/e28b3d47-5378-41b6-a2eb-b8faf496f21a" /> </p>
Parallel uploads are now the default for MOUNT_CACHED file mounts, delivering a 7x speedup — flush time dropped from 151s to 21s for a ~14.6 GB test workload (#8455). A new data.mount_cached.sequential_upload config option allows reverting to sequential uploads if needed.
<p align="center"> <img width="50%" alt="mount-cached-speedup" src="https://github.com/user-attachments/assets/03dab45c-3694-44ba-948e-166ea8a6450b" /> </p>
Users can now specify exit codes that trigger automatic job recovery in managed jobs (#8324):
resources:
job_recovery:
recover_on_exit_codes:[29]
When a job exits with a specified code, SkyPilot automatically recovers it — useful for transient failures with known error codes.
Automatically detect that SkyPilot is running in WSL, and seamless set up VSCode Remote-SSH for Windows users (#8669)
ray-node container (#8353, #8444)kubernetes.set_pod_resource_limits — set pod CPU/memory limits relative to requests for pod resource limit enforcement (#8644)NOT_READY status — actionable error messages for PVC issues; background refresh daemon; --refresh flag (#8524)remote_identity override (#8659), not-ready node exclusion (#8172), GKE autoscaler compatibility (#8326)/upload endpoint (#8723)sky api login — works around Chrome Private Network Access restrictions (#8590)SERVICE_ACCOUNT (#8386)fullnameOverride (#8528)p5e.48xlarge H200 and Melbourne region support (#8465, #8055)create_instance_kwargs (#8536), Vast.ai SSH fix (#8614)--secret-file CLI flag — avoid shell history exposure (#8646)provision.install_conda config — skip Miniconda for faster launches (#8662).sky location (#8153), restore autostop on start (#8022), multiple skylets per host (#8156)greenlet import fix (#8653), Resources.copy() fix (#8648), pandas compatibility (#8643), Docker MOTD fix (#8632)lstrip('ssh-') bug fix (#8417), package distribution fix (#8508)Thank you to all contributors who made this release possible!
@aflah02, @alex000kim, @andylizf, @aylei, @Bokki-Ryu, @brianstrauch, @cblmemo, @cg505, @concretevitamin, @cwhitak3r, @dan-blanchard, @DanielZhangQD, @Elden123, @funkypenguin, @hentt30, @isagi-y22, @JiangJiaWei1103, @kevinmingtarja, @koaning, @kyuds, @laimis9133, @liuwb, @lloyd-brown, @lucamanolache, @m-braganca, @Maknee, @Michaelvll, @mk0walsk, @mmcclean-aws, @mt5225, @nakinnubis, @oelachqar, @otutukingsley, @Philmod, @php-workx, @qicz, @rohansonecha, @romilbhardwaj, @sachdva, @SalikovAlex, @seahyinghang8, @SeungjinYang, @YashIIT0909, @yurekami, @atoniolo76, @zpoint
New Contributors:
Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!
For a complete list of changes, see the commit history.
Nothing published for this version
Nothing published for this version
Nothing published for this version
This patch release is a minor bump from v0.11.1 to to get you the latest fixes:
This patch release is a minor bump from v0.11.1 to to get you the latest fixes:
Install the release candidate:
# Select needed clouds.
uv pip install 'skypilot[kubernetes,aws,gcp]==0.11.2rc1'
Upgrade your remote API server:
NAMESPACE=skypilot # TODO: change to your installed namespace
RELEASE_NAME=skypilot # TODO: change to your installed release name
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--version 0.11.2-rc.1 \
--reset-then-reuse-values \
--set apiService.image=null # reset to default image
Full Changelog: https://github.com/skypilot-org/skypilot/compare/v0.11.1...v0.11.2rc1
This patch release is a minor bump from v0.11.0 to to get you the latest fixes:
This patch release is a minor bump from v0.11.0 to to get you the latest fixes:
sqlite3.OperationalError: database is locked
See the full v0.11 release notes for everything new in SkyPilot v0.11!
Deprecation: rbac.rules → rbac.namespaceRules; ingress.nodePortEnabled → ingress-ngnix.controller.service.type=NodePort (see details) helm chart
SkyPilot v0.11.0 delivers major new features: Pools and Managed Jobs Consolidation Mode; significant improvement enterprise-readiness at large scale: supports hundreds of AI engineers with a single API server instance, avoid OOM, >10x performance improvement on many requests, additional observability, and more; and UX improvements: Templates, Python SDK, CI/CD, Git Support, and more.
Get it now with:
uv pip install "skypilot>=0.11.0"
Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.11.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
SkyPilot supports spawning a pool that launches a set of workers across many clouds and Kubernetes clusters (docs, #6260, #6426, #6459, #6552, #6591, #6665, #6675, #7332, #7963, #8008, #7876, #8047, #7930,#8039, #7846, #7855, #7919, #7920). Jobs can be scheduled on this pool and distributed to workers as they become available.
Key benefits include:
Learn more in our blog post.
<p align="center"> <img width="80%" alt="batch_inference_architecture" src="https://github.com/user-attachments/assets/964a7c82-5c25-4848-a40f-30bb3ab65c45" /> </p>
Consolidation Mode is general available (#7122, #7127, #7396, #7459, #7498, #7560, #7601, #7619, #7717, #7720, #7847, #8082, #8021, #8106). This enables:
# config.yaml
jobs:
controller:
consolidation_mode: true # Currently defaults to False.
<p align="center"> <img width="80%" alt="Frame 5" src="https://github.com/user-attachments/assets/62873c90-7634-4e58-85e5-f3cf13b01d31" /> </p>
We have significantly optimized the Managed Jobs controller, allowing it to handle 2000+ parallel jobs on a single 8-CPU controller—an 18x improvement in job capacity with the same controller size (#7051, #7371, #7379, #7408, #7432, #7473, #7487, #7488, #7494, #7519, #7585, #7595, #7945, #7966, #7979,#8036, #8095).
<p align="center"> <img width="80%" alt="image 1" src="https://github.com/user-attachments/assets/50a0127c-e3d6-43dd-a898-830c58fd9610" /> </p>
<p align="center"> <img width="80%" alt="image_1" src="https://github.com/user-attachments/assets/56d4b8b9-c394-4b37-90f2-99e7586bb851" /> </p>
<p align="center"> <img width="80%" alt="image_2" src="https://github.com/user-attachments/assets/243d63a5-98eb-44df-a5db-ef6a187c9e92" /> </p>
<p align="center"> <img width="80%" alt="image_3" src="https://github.com/user-attachments/assets/81fc122c-ea77-49b9-ad62-09e0da58e809" /> </p>
<p align="center"> <img width="80%" alt="image_4" src="https://github.com/user-attachments/assets/f8b5c1ce-9ea9-4c36-8d3b-274987295961" /> </p>
UX, Robustness, Performance Improvement
Volume Support for Existing PVC and Ephemeral Volumes (#7915, #7971,#8179)
Exising PVC: Reference pre-existing Kubernetes PersistentVolumeClaims as a SkyPilot volume (#7915).
# volume.yaml
name: existing-pvc-name
type: k8s-pvc
infra: k8s/context1
use_existing: true
config:
namespace: namespace
Ephemeral Volumes: automatically create volumes when a cluster is launched and deleted when the cluster is torn down, making them ideal for temporary storage across multiple nodes, such as caches and intermediate results.
# task.sky.yaml
file_mounts:
/mnt/cache:
size: 100Gi
CoreWeave Integration Announcement
CoreWeave now officially support SkyPilot (#6386, #6519, #6895, #7756, #7838). This integration provides: Infiniband Support, Object Storage, Autoscaling.
See the announcement on Coreweave blog. <p align="center"> <img width="80%" alt="image_5" src="https://github.com/user-attachments/assets/f6679b23-4aeb-44be-95e4-79640a17d9a8" /> </p>
AMD GPU Support Announcement
SkyPilot now fully supports AMD GPUs on Kubernetes clusters (#6378, #6944). This includes: GPU Detection and Scheduling, Dashboard Metrics, ROCm Support.
See the announcement on AMD Rocm blog. <p align="center"> <img width="80%" alt="image_6" src="https://github.com/user-attachments/assets/b4f6c078-9d65-4a72-86b5-8a36e9e51e1a" /> </p>
More Clouds Support
SkyPilot Templates
SkyPilot now ships predefined YAML templates for launching clusters with popular frameworks and patterns. Templates are automatically available on all new SkyPilot clusters. (#7935, #7965)
You can now launch a multi-node Ray cluster by adding a single line to your YAML’s run block:
run: |
# One-line setup for a distributed Ray cluster
~/.sky/templates/ray/start_cluster
# Submit your job
python train.py
Programmability: SkyPilot Python SDK
SkyPilot Python SDK is significantly improved with:
<p align="center"> <img width="80%" alt="image_7" src="https://github.com/user-attachments/assets/2fda4419-ddd8-437d-a576-8b3114ca5a7e" /> </p>
logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
for line in logs:
if line is not None:
if 'needle in the haystack' in line:
print("found it!")
break
logs.close()
resource_config = user_request.task.get_resource_config()
resource_config['use_spot'] = True
user_request.task.set_resources(resource_config)
Integrating with CI/CD
With the improved SDK, we can integrate SkyPilot with GitHub Actions and other orchestrator to spin off your AI workloads automatically. (#7932) <p align="center"> <img width="80%" alt="image_8" src="https://github.com/user-attachments/assets/5371b954-466a-4613-a726-ce8f856fb860" /> </p>
Native Git Support
You can now use your private git repositories directly as your SkyPilot workdir (#6294, #6257, #6268). SkyPilot handles the cloning and syncing automatically. You can also use --git-url and --git-ref options with sky serve up (#8012).
# task.sky.yaml
workdir:
url: <https://github.com/my-org/my-repo.git>
ref: 1234ab # commit hash or branch name
You can also find the commit hash of your workdir in the Dashboard: <p align="center"> <img width="80%" alt="image_1 1" src="https://github.com/user-attachments/assets/c9632f54-0693-464c-9956-f775668559ed" /> </p>
Autostop/Autodown Based on SSH Sessions
In addition to running jobs on clusters, you can now configure autostop/autodown to wait for active SSH sessions (#6361, #6485).
# task.sky.yaml
resources:
autostop:
wait_for: jobs_and_ssh
We released high-performance distributed training examples for large models with checkpointing support (#6525, #6551, #6242, #6273, #6443).
<p align="center"> <img width="80%" alt="image_2 1" src="https://github.com/user-attachments/assets/3ace5534-d289-476a-b4b4-c6b73c6925f4" /> </p>
rbac.rules → rbac.namespaceRules; ingress.nodePortEnabled → ingress-ngnix.controller.service.type=NodePort (see details) helm chartpost_provision_runcmd (#7943).allowed_contexts configuration (#7878).--retry-until-up: Fixed the -retry-until-up flag to properly retry failed launches across all cloud zones (#8079).sky logs --provision during cluster launches (#7888).sky show-gpus, fix the default Neuron-based AMI (#7896, #7958).AWS_CONFIG_FILE environment variable for custom credential paths (#8050).SSH Node Pools
pod_config and custom_metadata specifications for multiple SSH node pools through context_configs (#7660, #7913).provision_timeout field for controlling SSH node provisioning wait times (#7660).SecretStr to prevent accidental exposure in logs (#8040).Task.get_resources_config() returns resources as a dictionary matching YAML format, and Task.set_resources() now accepts dictionaries—making programmatic resource manipulation as intuitive as editing YAML (#7857).SKYPILOT_SETUP_NUM_GPUS_PER_NODE environment variable available during setup phase for configuring software based on GPU count (#7092).sky serve logs (#8002)sky status --kubernetes for the team SkyPilot API server (#7989), removed sky local up --ips in favor of sky ssh up (#8065)Thank you to all contributors who made this release possible!
@adocherty, @alex000kim, @aylei, @brianstrauch, @cblmemo, @cg505, @coopslarhette, @cwhitak3r, @DanielZhangQD, @hyoxt121, @kevinmingtarja, @kyuds, @lloyd-brown, @massaindustries, @Michaelvll, @rohansonecha, @romilbhardwaj, @SalikovAlex, @seongsukwon-moreh, @SeungjinYang, @zpoint
New Contributors:
Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!
For a complete list of changes, see the commit history.
Nothing published for this version
This is a preview of the upcoming 0.11.0 release!
This is a preview of the upcoming 0.11.0 release!
Install the release candidate:
# Select needed clouds.
uv pip install 'skypilot[kubernetes,aws,gcp]==0.11.0rc1'
Upgrade your remote API server:
NAMESPACE=skypilot # TODO: change to your installed namespace
RELEASE_NAME=skypilot # TODO: change to your installed release name
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--version 0.11.0-rc.1 \
--reset-then-reuse-values \
--set apiService.image=null # reset to default image
If you try out 0.11.0rc1, please let us know in Slack!
One column per quarter.