github.com/kubeflow/mpi-operator

Changes since v0.8.0

• Bug fixes:
  • Fix missing DeepCopy when mutating the launcher Job, which prevents accidental mutation of the shared informer cache. (#819, @GonzaloLuminary)
  • Prevent terminating launcher pods from being recreated. (#810, @GonzaloLuminary)
  • Fix the Running condition being re-emitted when the pod and Job informers are out-of-sync. (#787, @GonzaloSaez)
  • Honor stderrthreshold when logtostderr is enabled by opting into the fixed klog behavior. (#794, @pierluigilenoci)

Acknowledgments

Thank you to all the contributors (in no particular order): @terrytangyuan @GonzaloLuminary @GonzaloSaez @pierluigilenoci @kannon92 @tenzen-y

Changes since v0.7.0

• Features:
  • Support Mutable Scheduling Directives for suspended Jobs
    on Launcher Job, which allows us to propagate the scheduling directives from the MPIJob Launcher template to the Launcher batch/v1 Job only when in the suspended state. (#772, @GonzaloSaez)
• Clean ups:
  • Upgrade Kubernetes dependencies to v1.35. (#769, @tenzen-y)

Acknowledgments

Thank you to all the contributors (in no particular order): @mimowo @terrytangyuan @GonzaloSaez @tenzen-y

Changes since v0.6.0

• Features:
  • Support custom cluster domain in MPI hostfile generation. (#704, #707, #738, @tenzen-y)
  • Enable Service publishNotReadyAddresses when runLauncherAsWorker to improve DNS discovery for workers. (#703, @tenzen-y)
  • Expose job controller workqueue rate-limiting configuration via operator flags to improve scalability tuning. (#674, @rotemelad)
• Bug fixes:
  • Fix crash in PodGroup when runLauncherAsWorker=true. (#669, @GonzaloSaez)
  • Fix missing ReplicaIndexLabel when runLauncherAsWorker=true so the launcher pod gets the expected pod index label (helps Kueue/TAS rank discovery). (#690, @GonzaloSaez)
• Clean ups:
  • Upgrade Kubernetes dependencies to v1.34. (#742, @tenzen-y)
  • Fix kustomize v5 warnings in manifests. (#700, @vikas-saxena02)
  • Upgrade Debian version to trixie and MPI versions are upgraded in the following: (#685, @tenzen-y)
    • OpenMPI: v4.1.4 -> v5.0.7
    • MPICH: v3.4.1 -> v4.2.1

Acknowledgments

Thank you to all the contributors (in no particular order): @rotemelad @mimowo @terrytangyuan @GonzaloSaez @vikas-saxena02 @tenzen-y

Changes since v0.5.0

• Features:
  • Support ManagedBy feature (.spec.runPolicy.managedBy) inspired by batch/v1 Job.
    • This allows us to dispatch MPIJobs to the multiple clusters powered by Kueue's MultiKueue. (#650, @mszadkow)
• Clean ups:
  • Upgrade k8s libraries to v1.31 (#664, @ArangoGutierrez)
  • Upgrade debian version to bookworm and MPI versions are upgraded in the following: (#661, @tenzen-y)
    • OpenMPI: v4.1.0 -> v4.1.4
    • MPICH: 3.4.1 -> 4.0.2

Acknowledgments

Thank you to all the contributors (in no particular order): @mszadkow @mimowo @alculquicondor @terrytangyuan @ArangoGutierrez @tenzen-y

Full Changelog: v0.5.0...v0.6.0

Changes since v0.4.0

• Features:
  • Add support for MPICH (#562, @sheevy)
  • Field runLauncherAsWorker allows to add the launcher pod into the hostfile as a worker (#612, @kuizhiqing)
  • Add PodGroup minResources calculation for volcano integration (#566, @lowang-bh)
• Bug fixes:
  • Fix panic when using PodGroups and PriorityClasses (#561, @tenzen-y)
  • Fix installation of mpijob Python module (#579, @vsoch)
  • Fix hostfile when jobs in different namespaces have the same name (#622, @kuizhiqing)
• Clean ups:
  • Upgrade k8s libraries to v1.29 (#633, @tenzen-y)
  • Fail the mpi-operator binary if access to API is denied (#619, @emsixteeen)

Acknowledgments

Thank you to all the contributors (in no particular order): @sheevy @alculquicondor @terrytangyuan @tenzen-y @kuizhiqing @lowang-bh @vsoch @emsixteeen @wang-mask @benash @yeahdongcn @xhejtman @pheianox @lianghao208

Changes since 0.3.0

• Breaking changes
  • Removed v1 operator. If you want to use MPIJob v1, you can use the training-operator.
• Support for suspending semantics. Third party controllers can leverage the suspend field to implement queuing and preemption for an MPIJob.
• Support for the coscheduling plugins of the scheduler-plugins.
• The operator supports multi-architecture (amd64, aarch64, and ppc64le).
• Bug fixes
  • Fix support for elastic Horovod.

Acknowledgements

Special thanks to @tenzen-y for multiple contributions.
Thank you to all the contributors (in no particular order): @mimowo @adilhusain-s @davidLif @ArangoGutierrez @shaowei-su @ggaaooppeenngg @pugangxa @HeGaoYuan @Dimss @alculquicondor @terrytangyuan

Release v0.3.0

• Scalability improvements
  • Worker start up no longer issues requests to kube-apiserver.
  • Dropped kubectl-delivery init container, reducing stress on kube-apiserver.
• Support for Intel MPI.
• Support for runPolicy (ttlSecondsAfterFinish, activeDeadlineSeconds, backoffLimit)
  by using a k8s Job for the launcher.
• Samples for plain MPI applications.
• Production readiness improvements:
  • Increased coverage throughout unit, integration and E2E tests.
  • More robust API validation.
  • Revisited v2beta1 MPIJob API.
  • Using fully-qualified label names, in consistency with other kubeflow operators.