github.com/NVIDIA/gpu-operator

New Features

Support for NVIDIA Data Center GPU Driver version 470.57.02 .

Added support for NVSwitch systems such as HGX A100. The driver container detects the presence of NVSwitches in the system and automatically deploys the Fabric Manager for setting up the NVSwitch fabric.

The driver container now builds and loads the nvidia-peermem kernel module when GPUDirect RDMA is enabled and Mellanox devices are present in the system. This allows the GPU Operator to complement the NVIDIA Network Operator to enable GPUDirect RDMA in the Kubernetes cluster. Refer to the RDMA documentation on getting started.

Note

This feature is available only when used with R470 drivers on Ubuntu 20.04 LTS.

Added support for upgrades of the GPU Operator components. A new k8s-driver-manager component handles upgrades of the NVIDIA drivers on nodes in the cluster.

NVIDIA DCGM is now deployed as a component of the GPU Operator. The standalone DCGM container allows multiple clients such as DCGM-Exporter and NVSM to be deployed and connect to the existing DCGM container.

Added a nodeStatusExporter component that exports operator and node metrics in a Prometheus format. The component provides information on the status of the operator (e.g. reconciliation status, number of GPU enabled nodes).

Improvements

Reduced the size of the ClusterPolicy CRD by removing duplicates and redundant fields.

The GPU Operator now supports detection of the virtual PCIe topology of the system and makes the topology available to vGPU drivers via a configuration file. The driver container starts the nvidia-topologyd daemon in vGPU configurations.

Added support for specifying the RuntimeClass variable via Helm.

Added nvidia-container-toolkit images to support CentOS 7 and CentOS 8.

nvidia-container-toolkit now supports configuring containerd correctly for RKE2.

Added new debug options (logging, verbosity levels) for nvidia-container-toolkit

Fixed issues

The driver container now loads ipmi_devintf by default. This allows tools such as ipmitool that rely on ipmi char devices to be created and available.

Known Limitations

GPUDirect RDMA is only supported with R470 drivers on Ubuntu 20.04 LTS and is not supported on other distributions (e.g. CoreOS, CentOS etc.)

The operator supports building and loading of nvidia-peermem only in conjunction with the Network Operator. Use with pre-installed MOFED drivers on the host is not supported. This capability will be added in a future release.

Support for DGX A100 with GPU Operator 1.8 will be available in an upcoming patch release.

This version of GPU Operator does not work well on RedHat OpenShift when a cluster-wide proxy is configured and causes constant restarts of driver container. This will be fixed in an upcoming patch release v1.8.2 .

Fixed issues

NFD version bumped to v0.8.2 to support correct kernel version labeling on Anthos nodes. See NFD issue for more details.

New Features

Support for NVIDIA Data Center GPU Driver version 460.73.01 .

Added support for automatic configuration of MIG geometry on NVIDIA Ampere products (e.g. A100) using the k8s-mig-manager .

GPU Operator can now be deployed on systems with pre-installed NVIDIA drivers and the NVIDIA Container Toolkit.

DCGM-Exporter now supports telemetry for MIG devices on supported Ampere products (e.g. A100).

Added support for a new nvidia RuntimeClass with containerd .

The Operator now supports PodSecurityPolicies when enabled in the cluster.

Improvements

Changed the label selector used by the DaemonSets of the different states of the GPU Operator. Instead of having a global label nvidia.com/gpu.present=true , each DaemonSet now has its own label, nvidia.com/gpu.deploy.<state>=true . This new behavior allows a finer grain of control over the components deployed on each of the GPU nodes.

Migrated to using the latest operator-sdk for building the GPU Operator.

The operator components are deployed with node-critical PriorityClass to minimize the possibility of eviction.

Added a spec for the initContainer image, to allow flexibility to change the base images as required.

Added the ability to configure the MIG strategy to be applied by the Operator.

The driver container now auto-detects OpenShift/RHEL versions to better handle node/cluster upgrades.

Validations of the container-toolkit and device-plugin installations are done on all GPU nodes in the cluster.

Added an option to skip plugin validation workload pod during the Operator deployment.

Fixed issues

The gpu-operator-resources namespace is now created by the Operator so that they can be used by both Helm and OpenShift installations.

Known Limitations

DCGM does not support profiling metrics on RTX 6000 and RTX 8000. Support will be added in a future release of DCGM Exporter.

After uninstall of GPU Operator, NVIDIA driver modules might still be loaded. Either reboot the node or forcefully remove them using sudo rmmod nvidia nvidia_modeset nvidia_uvm command before re-installing GPU Operator.

When MIG strategy of mixed is configured, device-plugin-validation may stay in Pending state due to incorrect GPU resource request type. User would need to modify the pod spec to apply correct resource type to match the MIG devices configured in the cluster.