PyPI · #4317 most downloaded on PyPI

deepspeed

0.19.71.1M downloads/modeepspeedai/DeepSpeed ↗New releases in your daily digest, changelog included.

DeepSpeed library

Last release 17 days ago
16 Sep 2026
Ships on a steady schedule
a new release about every 3 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
6 years old
132 releases · first in 2020

Release timeline

132 releases since 2020

2020202120222023202420252026

20202026

One column per quarter.

Releases

0.11.19 Oct 2023
Fix bug in bfloat16 optimizer related to checkpointing by @okoge-kaz in https://github.com/microsoft/DeepSpeed/pull/4434
What's Changed

Fix bug in bfloat16 optimizer related to checkpointing by @okoge-kaz in https://github.com/microsoft/DeepSpeed/pull/4434

Move tensors to device if mp is not enabled by @deepcharm in https://github.com/microsoft/DeepSpeed/pull/4461

Fix torch import causing release build failure by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4468

add lm_head and embed_out tensor parallel by @Yejing-Lai in https://github.com/microsoft/DeepSpeed/pull/3962

Fix release workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4483

New Contributors

@okoge-kaz made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4434

@deepcharm made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4461

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.11.0...v0.11.1
Open source →
0.11.06 Oct 2023
DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs [English] [中文] [日本語]
New features

DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs [English] [中文] [日本語]

Announcing the DeepSpeed4Science Initiative: Enabling large-scale scientific discovery through sophisticated AI system technologies [DeepSpeed4Science website] [Tutorials] [Blog] [中文] [日本語]

What's Changed

added a model check for use_triton in deepspeed by @stephen-youn in https://github.com/microsoft/DeepSpeed/pull/4266

Update release and bump patch versioning flow by @loadams in https://github.com/microsoft/DeepSpeed/pull/4286

README update by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4303

Update README.md by @NinoRisteski in https://github.com/microsoft/DeepSpeed/pull/4316

Handle empty parameter groups by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4277

Clean up modeling code by @loadams in https://github.com/microsoft/DeepSpeed/pull/4320

Fix Zero3 contiguous grads, reduce scatter false accuracy issue by @nelyahu in https://github.com/microsoft/DeepSpeed/pull/4321

Add release version checking by @loadams in https://github.com/microsoft/DeepSpeed/pull/4328

clear redundant timers by @starkhu in https://github.com/microsoft/DeepSpeed/pull/4308

DS-Chat BLOOM: Fix Attention mask by @lekurile in https://github.com/microsoft/DeepSpeed/pull/4338

Fix a bug in the implementation of dequantization for inference by @sakogan in https://github.com/microsoft/DeepSpeed/pull/3433

Suppress noise by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4310

Fix skipped inference tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4336

Fix autotune to support Triton 2.1 by @stephen-youn in https://github.com/microsoft/DeepSpeed/pull/4340

Pass base_dir to model files can be loaded for auto-tp/meta-tensor. by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/4348

Support InternLM by @wangruohui in https://github.com/microsoft/DeepSpeed/pull/4137

DeepSpeed4Science by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4357

fix deepspeed4science links by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4358

Add the policy to run llama model from the official repo by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/4313

Check inference input_id tokens length by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4349

add deepspeed4science blog link by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4364

Update conda env to have max pydantic version by @loadams in https://github.com/microsoft/DeepSpeed/pull/4362

Enable workflow dispatch on Torch 1.10 CI tests by @loadams in https://github.com/microsoft/DeepSpeed/pull/4361

deepspeed4science chinese blog by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4366

deepspeed4science japanese blog by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4369

Openfold fix by @cctry in https://github.com/microsoft/DeepSpeed/pull/4368

[BUG] add the missing method to MPS accelerator by @cli99 in https://github.com/microsoft/DeepSpeed/pull/4363

Fix multinode runner to properly append to PDSH_SSH_ARGS_APPEND by @loadams in https://github.com/microsoft/DeepSpeed/pull/4373

Fix min torch version by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4375

Fix llama meta tensor loading in AutoTP and kernel injected inference by @zeyugao in https://github.com/microsoft/DeepSpeed/pull/3608

adds triton flash attention2 kernel by @stephen-youn in https://github.com/microsoft/DeepSpeed/pull/4337

Allow multiple inference engines in single script by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4384

Save/restore step in param groups with zero 1 or 2 by @tohtana in https://github.com/microsoft/DeepSpeed/pull/4396

Fix incorrect assignment of self.quantized_nontrainable_weights by @VeryLazyBoy in https://github.com/microsoft/DeepSpeed/pull/4399

update deepspeed4science blog by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4408

Add torch no grad condition by @ajindal1 in https://github.com/microsoft/DeepSpeed/pull/4391

Update nv-transformers workflow to use cu11.6 by @loadams in https://github.com/microsoft/DeepSpeed/pull/4412

Add condition when dimension is greater than 2 by @ajindal1 in https://github.com/microsoft/DeepSpeed/pull/4390

[CPU] Add CPU AutoTP UT. by @Yejing-Lai in https://github.com/microsoft/DeepSpeed/pull/4263

fix cpu loading model partition OOM by @Yejing-Lai in https://github.com/microsoft/DeepSpeed/pull/4353

Update cpu_inference checkout action by @loadams in https://github.com/microsoft/DeepSpeed/pull/4424

Zero infinity xpu support by @Liangliang-Ma in https://github.com/microsoft/DeepSpeed/pull/4130

[CCLBackend] Using parallel memcpy for inference_all_reduce by @delock in https://github.com/microsoft/DeepSpeed/pull/4404

Change default set_to_none=true in zero_grad methods by @Jackmin801 in https://github.com/microsoft/DeepSpeed/pull/4438

Small docstring fix by @Jackmin801 in https://github.com/microsoft/DeepSpeed/pull/4431

fix: check-license by @Jackmin801 in https://github.com/microsoft/DeepSpeed/pull/4432

Fixup check release version script by @loadams in https://github.com/microsoft/DeepSpeed/pull/4413

Enable ad-hoc running of cpu_inference by @loadams in https://github.com/microsoft/DeepSpeed/pull/4444

Fix wrong documentation of ignore_unused_parameters by @UniverseFly in https://github.com/microsoft/DeepSpeed/pull/4418

DeepSpeed-VisualChat Blog by @xiaoxiawu-microsoft in https://github.com/microsoft/DeepSpeed/pull/4446

Fix a bug in DeepSpeedMLP by @sakogan in https://github.com/microsoft/DeepSpeed/pull/4389

documenting load_from_fp32_weights config parameter by @clumsy in https://github.com/microsoft/DeepSpeed/pull/4449

Add Japanese translation of DS-VisualChat blog by @tohtana in https://github.com/microsoft/DeepSpeed/pull/4454

fix blog format by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4456

Update README-Japanese.md by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4457

DeepSpeed-VisualChat Chinese blog by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4458

CI fix for torch 2.1 release by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4452

fix lm head overriden issue, move it from checkpoint in-loop loading … by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/4206

feat: add Lion by @enneamer in https://github.com/microsoft/DeepSpeed/pull/4331

pipe engine eval_batch: add option to disable loss broadcast by @nelyahu in https://github.com/microsoft/DeepSpeed/pull/4326

Add release flow by @loadams in https://github.com/microsoft/DeepSpeed/pull/4467

New Contributors

@nelyahu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4321

@starkhu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4308

@sakogan made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3433

@cctry made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4368

@zeyugao made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3608

@VeryLazyBoy made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4399

@ajindal1 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4391

@Liangliang-Ma made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4130

@Jackmin801 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4438

@UniverseFly made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4418

@enneamer made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4331

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.10.3...v0.11.0

0.10.311 Sept 2023

ZeRO-Inference: 20X faster inference through weight quantization and KV cache offloading

New Features

ZeRO-Inference: 20X faster inference through weight quantization and KV cache offloading

What's Changed

Add Mixed Precision ZeRO++ tutorial by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/4241
DeepSpeed-Chat Llama2/stability release by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/4240
Update README.md by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/4244
Pin Triton version to >=2.0.0 and <2.1.0 by @lekurile in https://github.com/microsoft/DeepSpeed/pull/4251
Allow modification of zero partitioned parameters by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4192
Checks for user injection policy by @satpalsr in https://github.com/microsoft/DeepSpeed/pull/3052
Add check that opening issues on CI failure requires schedule by @loadams in https://github.com/microsoft/DeepSpeed/pull/4242
Code Refactoring by @tosemml in https://github.com/microsoft/DeepSpeed/pull/4262
tolerating missing optimizer states for MoE [2nd attempt] by @clumsy in https://github.com/microsoft/DeepSpeed/pull/4120
Fix nv-inference/un-pin transformers by @loadams in https://github.com/microsoft/DeepSpeed/pull/4269
check for zero (empty) param groups in llama + hf/accelerate. by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/4270
use non_reentrant_checkpoint fix requires_grad of input must be true for activation checkpoint layer in pipeline train. by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/4224
The PostBackwardFunction class should be more clearly named to distinguish it from the PreBackwardFunction class. by @Crispig in https://github.com/microsoft/DeepSpeed/pull/2548
fix iteration timing used in autotuning when gradient_accumulation_steps > 1 by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2888
Update README.md by @NinoRisteski in https://github.com/microsoft/DeepSpeed/pull/4284
update deepspeed to run with the most recent triton 2.1.0 by @stephen-youn in https://github.com/microsoft/DeepSpeed/pull/4278
Keep hpz secondary tensor in forward pass by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/4288
Support iterators with incompletely defined len functions by @codedecde in https://github.com/microsoft/DeepSpeed/pull/2445
AMD Kernel Compatibility Fixes by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3180
ZeRO-Inference refresh by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4197
fix user args parsing of string with spaces on runner by @YudiZh in https://github.com/microsoft/DeepSpeed/pull/4265
Update index.md by @NinoRisteski in https://github.com/microsoft/DeepSpeed/pull/4297

New Contributors

@tosemml made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4262
@Crispig made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2548
@NinoRisteski made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4284
@codedecde made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2445
@YudiZh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4265

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.10.2...v0.10.3

Open source →

0.10.231 Aug 2023

MP ZeRO++ by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3954

What's Changed

MP ZeRO++ by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3954
do allgather only in shared optimizer states groups by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/4167
Permit empty environment variables as unset in setup.py by @loadams in https://github.com/microsoft/DeepSpeed/pull/4185
enable autoTP for mpt in huggingface model hub without trust_remote_c… by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/4062
Fix nv-nightly workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4163
Fix the path in tutorial by @kytimmylai in https://github.com/microsoft/DeepSpeed/pull/4193
Add unit test to check HF low_cpu_mem_usage_flag by @loadams in https://github.com/microsoft/DeepSpeed/pull/4184
Fix ZeRO parameter initialization for tensors with requires_grad=True by @XuehaiPan in https://github.com/microsoft/DeepSpeed/pull/4138
DeepSpeed Ulysses tutorial by @minjiaz in https://github.com/microsoft/DeepSpeed/pull/4200
Load z3 checkpoints for inference by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4171
DeepSpeed Ulysses release by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4198
Deepspeed-Ulysses blog by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4201
Ds ulysses news by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4202
DS-Ulysses formating by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4204
Update Ulyssess by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4205
Update README.md by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/4211
Add Japanese blog of DS-Ulysses by @tohtana in https://github.com/microsoft/DeepSpeed/pull/4209
DeepSpeed Ulysses Chinese blog translation by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/4210
add ulysses blog index by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4215
Add MuP optimizers by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2043
Simplify Gradient Attribute Names by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/4214
add meta onDevice support for LLAMA2 by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/4147
Fixes timer error referenced in #4212 by @bjoernpl in https://github.com/microsoft/DeepSpeed/pull/4213
Fix pipline dataloader when batch elements contain tuple by @ghosthamlet in https://github.com/microsoft/DeepSpeed/pull/565
feat(activation_checkpointing): add non_reentrant_checkpoint to support inputs require no grad by @hughpu in https://github.com/microsoft/DeepSpeed/pull/4118
add npu support dtypes by @CurryRice233 in https://github.com/microsoft/DeepSpeed/pull/4223
Fix fused qkv sizing for bloom by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/4161
added port argument for ssh by @Hiromasa-H in https://github.com/microsoft/DeepSpeed/pull/4117
Empty tensor size check by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/4186
fix: linker issues in conda environments #3929 by @maximegmd in https://github.com/microsoft/DeepSpeed/pull/4235
Enable AMD MI200 and H100 to run on branches for testing by @loadams in https://github.com/microsoft/DeepSpeed/pull/4238
fix MegatronLayerPolicy to be compatible with the newest ParallelTransformerLayer by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/4236
Enable hpz when running with torch.no_grad by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/4232

New Contributors

@kytimmylai made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4193
@bjoernpl made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4213
@Hiromasa-H made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4117
@maximegmd made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4235

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.10.1...v0.10.2

Open source →

0.10.118 Aug 2023

[docs] add zero++ paper link by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3974

What's Changed

[docs] add zero++ paper link by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3974
Avoid race condition with port selection in unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3975
Remove duplicated inference unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3951
Switch to torch.linalg.norm by @loadams in https://github.com/microsoft/DeepSpeed/pull/3984
Simplify chain comparisons, remove redundant parentheses by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3912
[CPU] Support HBM flatmode and fakenuma mode by @delock in https://github.com/microsoft/DeepSpeed/pull/3918
Fix checkpoint conversion when model layers share weights by @awaelchli in https://github.com/microsoft/DeepSpeed/pull/3825
fixing flops profiler formatting, units and precision by @clumsy in https://github.com/microsoft/DeepSpeed/pull/3927
Specify language=python in pre-commit hook by @wangruohui in https://github.com/microsoft/DeepSpeed/pull/3994
[CPU] Skip CPU support unimplemented error by @Yejing-Lai in https://github.com/microsoft/DeepSpeed/pull/3633
ZeRO Gradient Accumulation Dtype. by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2847
[CPU] Use allreduce_low_latency for AutoTP and implement low latency allreduce for CPU backend (single node) by @delock in https://github.com/microsoft/DeepSpeed/pull/3919
Re-enable skipped unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3939
Make AMD/ROCm apex install to /blob to save test/compile time. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3997
Option to exclude frozen weights for checkpoint save by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3953
Allow user to select name of .deepspeed_env by @loadams in https://github.com/microsoft/DeepSpeed/pull/4006
Silence backend warning by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4009
Fix user arg parsing in single node deployment by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4007
Specify triton 2.0.0 requirement by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4008
Re-enable elastic training for torch 2+ by @loadams in https://github.com/microsoft/DeepSpeed/pull/4010
add /dev/shm size to ds_report by @jeffra in https://github.com/microsoft/DeepSpeed/pull/4015
Make Ascend NPU available by @hipudding in https://github.com/microsoft/DeepSpeed/pull/3831
RNNprofiler: fix gates size retrieval logic in _rnn_flops by @pinstripe-potoroo in https://github.com/microsoft/DeepSpeed/pull/3921
fix typo in SECURITY.md by @jstan327 in https://github.com/microsoft/DeepSpeed/pull/4019
add llama2 autoTP support in replace_module by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/4022
[zero_to_fp32] 3x less cpu memory requirements by @stas00 in https://github.com/microsoft/DeepSpeed/pull/4025
[CPU] FusedAdam and CPU training support by @delock in https://github.com/microsoft/DeepSpeed/pull/3991
remove duplicate check for pp and zero stage by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/4033
Pass missing positional arguments in DeepSpeedHybridEngine.generate() by @XuehaiPan in https://github.com/microsoft/DeepSpeed/pull/4026
Remove print of weight parameter in RMS norm by @puneeshkhanna in https://github.com/microsoft/DeepSpeed/pull/4031
Monitored Loss Calculations by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/4030
fix(pipe): make pipe module load_state_dir non-strict-mode work by @hughpu in https://github.com/microsoft/DeepSpeed/pull/4020
polishing timers and log_dist by @clumsy in https://github.com/microsoft/DeepSpeed/pull/3996
Engine side fix for loading llama checkpoint fine-tuned with zero3 by @minjiaz in https://github.com/microsoft/DeepSpeed/pull/3981
fix: Remove duplicate word the by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/4051
[Bug Fix] Fix comm logging for inference by @delock in https://github.com/microsoft/DeepSpeed/pull/4043
fix opt-350m shard loading issue in AutoTP by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3600
enable autoTP for MPT by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3861
autoTP for fused qkv weight by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/3844
[CPU] Faster reduce kernel for SHM allreduce by @delock in https://github.com/microsoft/DeepSpeed/pull/4049
Multiple zero stage 3 related fixes by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3886
Fix deadlock when SHM based allreduce spin too fast by @delock in https://github.com/microsoft/DeepSpeed/pull/4048
[MiCS] [Bugfix] set self.save_non_zero_checkpoint=True only for first partition group by @zarzen in https://github.com/microsoft/DeepSpeed/pull/3787
add reproducible compilation environment by @fecet in https://github.com/microsoft/DeepSpeed/pull/3943
fix: remove unnessary # punct in the second sed command by @hughpu in https://github.com/microsoft/DeepSpeed/pull/4061
Refactor autoTP inference for HE by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/4040
Fix transformers unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4079
Fix Stable Diffusion Injection by @lekurile in https://github.com/microsoft/DeepSpeed/pull/4078
Spread layers more uniformly when using partition_uniform by @marcobellagente93 in https://github.com/microsoft/DeepSpeed/pull/4053
fix typo: change polciies to policies by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/4090
update ut/doc for glm/codegen by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/4057
zero_to_fp32 script adds support for tag argument by @EeyoreLee in https://github.com/microsoft/DeepSpeed/pull/4089
add type checker ignore by @EeyoreLee in https://github.com/microsoft/DeepSpeed/pull/4102
Fix generate config validation error on inference unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4107
use correct ckpt path when base_dir not available by @polisettyvarma in https://github.com/microsoft/DeepSpeed/pull/4101
Disable z3 tracing profiler by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4106
Pass correct node size for ZeRO++ by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/4085
add deepspeed chat arxiv report by @conglongli in https://github.com/microsoft/DeepSpeed/pull/4110
enable pipeline checkpoint loading mode by @leiwen83 in https://github.com/microsoft/DeepSpeed/pull/3629
Fix Issue 4083 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/4084
Add full list of DS_BUILD_* by @loadams in https://github.com/microsoft/DeepSpeed/pull/4119
Update nightly workflows to open an issue if CI fails by @loadams in https://github.com/microsoft/DeepSpeed/pull/3952
Update torch1.9 tests to 1.10 to match latest accelerate. by @loadams in https://github.com/microsoft/DeepSpeed/pull/4126
Handle PermissionError in os.chmod Call - Update engine.py by @M-Chris in https://github.com/microsoft/DeepSpeed/pull/4139
Generalize frozen weights unit test by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4140
Respect memory pinning config by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/4131
Remove incorrect async-io library checking code. by @loadams in https://github.com/microsoft/DeepSpeed/pull/4150
Return nn.parameter type for weights and biases by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/4146
Fixes #4151 by @saforem2 in https://github.com/microsoft/DeepSpeed/pull/4152
Handling for SIGTERM as well by @loadams in https://github.com/microsoft/DeepSpeed/pull/4160
Fix CI Badges by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4162
Add DS-Chat CI workflow by @lekurile in https://github.com/microsoft/DeepSpeed/pull/4127
[CPU][Bugfix] Make uid and addr_port part of SHM name in CCL backend by @delock in https://github.com/microsoft/DeepSpeed/pull/4115
Add DSE branch input to nv-ds-chat by @lekurile in https://github.com/microsoft/DeepSpeed/pull/4173
Pin transformers by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/4174

New Contributors

@awaelchli made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3825
@wangruohui made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3994
@jstan327 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4019
@XuehaiPan made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4026
@puneeshkhanna made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4031
@hughpu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4020
@fecet made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3943
@marcobellagente93 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4053
@polisettyvarma made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4101
@leiwen83 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3629
@M-Chris made their first contribution in https://github.com/microsoft/DeepSpeed/pull/4139

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.10.0...v0.10.1

Open source →

0.10.017 Jul 2023Deprecated

[Docs] chrome://tracing is deprecated by @keyboardAnt in https://github.com/microsoft/DeepSpeed/pull/3805

New features

ZeRO++: A leap in speed for LLM and chat model training with 4X less communication[English] [中文] [日本語]
H100 support and testing w. FP8 using NVIDIA's TransformerEngine

What's Changed

Documentation for DeepSpeed Accelerator Abstraction Interface by @delock in https://github.com/microsoft/DeepSpeed/pull/3184
FP8 unittest for H100 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3731
Fix apex install bugs by @loadams in https://github.com/microsoft/DeepSpeed/pull/3741
Fix Autotuner get_gas_from_user_config by @straywarrior in https://github.com/microsoft/DeepSpeed/pull/3664
Include cublas error details when getting cublas handle fails by @jli in https://github.com/microsoft/DeepSpeed/pull/3695
fix hybrid engine mlp module by @tensor-tang in https://github.com/microsoft/DeepSpeed/pull/3736
Fix output transpose dimension bugs by @loadams in https://github.com/microsoft/DeepSpeed/pull/3747
remove UtilsBuilder load, use torch (un)flatten ops by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/3728
add Chinese Zhihu social account by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3755
Account for expert parameters when calculating the total number of pa… by @alito in https://github.com/microsoft/DeepSpeed/pull/3720
fix ccl_backend and residual_add problems by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/3642
Fix url in getting-started guide (docs) by @acforvs in https://github.com/microsoft/DeepSpeed/pull/3768
Update deepspeed-chat/japanese/README.md by @eltociear in https://github.com/microsoft/DeepSpeed/pull/3765
Add H100 workflow and status badge. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3754
Add an api in deepspeed engine for adjusting micro batch size during training by @kisseternity in https://github.com/microsoft/DeepSpeed/pull/3773
Prevent hangs in CI during parallel run compilation by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2844
Revert "Prevent hangs in CI during parallel run compilation" by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3817
[Docs] chrome://tracing is deprecated by @keyboardAnt in https://github.com/microsoft/DeepSpeed/pull/3805
Support model declaration in zero.Init context by @tohtana in https://github.com/microsoft/DeepSpeed/pull/3592
Update zeropp.md by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/3821
Reduce Unit Test Times (Part 1) by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3829
Re-enable GPT-J unit tests and refactor inference tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3618
Fix racing condition in GatheredParameters by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3819
zero/mics.py: use on_accelerator instead of cuda only by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3806
Disable AMD test flows in YML by @loadams in https://github.com/microsoft/DeepSpeed/pull/3847
Reduce Unit Test Time (Part 2) by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3838
[profiling]add show_straggler argument to log_summary() by @delock in https://github.com/microsoft/DeepSpeed/pull/3579
checking process_group before merging bucket ranges (#3521) by @clumsy in https://github.com/microsoft/DeepSpeed/pull/3577
scripts/check-torchcuda.py: add checking for tensor.is_cuda by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3843
Zero3 Fix allreduce optimization for extra large tensor by @hablb in https://github.com/microsoft/DeepSpeed/pull/3832
[zero] revert PR #3166, it disabled grad clip for bf16 by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3790
Fix transpose convolution FLOPS profiler (retrieval of out_channels) by @pinstripe-potoroo in https://github.com/microsoft/DeepSpeed/pull/3834
Fix LoRA Fuse/Unfuse in Hybrid Engine by @sxjscience in https://github.com/microsoft/DeepSpeed/pull/3563
Update pytorch-lightning version in CI by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3882
[Docs] MMEngine has integrated deepspeed. by @HAOCHENYE in https://github.com/microsoft/DeepSpeed/pull/3879
Add FALCON Auto-TP Support by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/3640
Update apex installation to resolve apex's pyproject.toml issues. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3745
Extend HE-Lora test with Z3 support + Fix/add guard in HE for Z3 by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/3883
Separate ZeRO3 InflightParamRegistry for train and eval by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3884
Add GPTNeoX AutoTP support by @Yejing-Lai in https://github.com/microsoft/DeepSpeed/pull/3778
Fix Meta Tensor checkpoint load for BLOOM models by @lekurile in https://github.com/microsoft/DeepSpeed/pull/3885
fix error :Dictionary expression not allowed in type annotation Pylance by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3708
Fix rnn flop profiler to compute flops instead of macs by @pinstripe-potoroo in https://github.com/microsoft/DeepSpeed/pull/3833
Update workflows for merge queue by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3892
Avoid deprecation warnings in CHECK_CUDA by @Flamefire in https://github.com/microsoft/DeepSpeed/pull/3854
Silence comm.py warning by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3893
Fix a typo of global variable in comm.py by @hipudding in https://github.com/microsoft/DeepSpeed/pull/3852
[ROCm] Enable TestCUDABackward::test_backward unit tests by @rraminen in https://github.com/microsoft/DeepSpeed/pull/3849
[profiling][mics]Fix some issues for log_summary(). by @ys950902 in https://github.com/microsoft/DeepSpeed/pull/3899
fix "undefined symbol: curandCreateGenerator" for quantizer op by @jinzhen-lin in https://github.com/microsoft/DeepSpeed/pull/3846
fix memory leak with zero-3 by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3903
fix some typo docs/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3917
fix: change ==NONE to is under deepspeed/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3923
Del comment deepspeed.zero.Init() can be used as a decorator by @hipudding in https://github.com/microsoft/DeepSpeed/pull/3894
Remove the param.ds_tensor from print by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3928
Reduce Unit Test Times (Part 3) by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3850
Update zero_to_fp32.py - to support deepspeed_stage_1 by @PicoCreator in https://github.com/microsoft/DeepSpeed/pull/3936
[docs] add xTrimoPGLM by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3940
Update Nvidia docker base image by @KaiChen1008 in https://github.com/microsoft/DeepSpeed/pull/3930
Fix inference tutorial docs for checkpoints by @loadams in https://github.com/microsoft/DeepSpeed/pull/3955
fix Megatron-DeepSpeed links by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3956
skip bcast when enable pp but pp_group_size=1 by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/3915
Use device_name instead of device index to support other device by @hipudding in https://github.com/microsoft/DeepSpeed/pull/3933
Create accelerator for apple silicon GPU Acceleration by @NripeshN in https://github.com/microsoft/DeepSpeed/pull/3907
fix(cpu_accelerator): :bug: Convert LOCAL_SIZE to integer by @javsalgar in https://github.com/microsoft/DeepSpeed/pull/3971

New Contributors

@straywarrior made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3664
@alito made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3720
@acforvs made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3768
@keyboardAnt made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3805
@pinstripe-potoroo made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3834
@HAOCHENYE made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3879
@Yejing-Lai made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3778
@Flamefire made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3854
@hipudding made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3852
@PicoCreator made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3936
@KaiChen1008 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3930
@NripeshN made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3907
@javsalgar made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3971

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.4...v0.10.0

Open source →

0.9.522 Jun 2023

Documentation for DeepSpeed Accelerator Abstraction Interface by @delock in https://github.com/microsoft/DeepSpeed/pull/3184

What's Changed

Documentation for DeepSpeed Accelerator Abstraction Interface by @delock in https://github.com/microsoft/DeepSpeed/pull/3184
FP8 unittest for H100 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3731
Fix apex install bugs by @loadams in https://github.com/microsoft/DeepSpeed/pull/3741
Fix Autotuner get_gas_from_user_config by @straywarrior in https://github.com/microsoft/DeepSpeed/pull/3664
Include cublas error details when getting cublas handle fails by @jli in https://github.com/microsoft/DeepSpeed/pull/3695
fix hybrid engine mlp module by @tensor-tang in https://github.com/microsoft/DeepSpeed/pull/3736
Fix output transpose dimension bugs by @loadams in https://github.com/microsoft/DeepSpeed/pull/3747
remove UtilsBuilder load, use torch (un)flatten ops by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/3728
add Chinese Zhihu social account by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3755
Account for expert parameters when calculating the total number of pa… by @alito in https://github.com/microsoft/DeepSpeed/pull/3720
fix ccl_backend and residual_add problems by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/3642
Fix url in getting-started guide (docs) by @acforvs in https://github.com/microsoft/DeepSpeed/pull/3768
Update deepspeed-chat/japanese/README.md by @eltociear in https://github.com/microsoft/DeepSpeed/pull/3765
Add H100 workflow and status badge. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3754
Zero++ tutorial PR by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3783
[Fix] _conv_flops_compute when padding is a str and stride=1 by @zhiruiluo in https://github.com/microsoft/DeepSpeed/pull/3169
fix interpolate flops compute by @cli99 in https://github.com/microsoft/DeepSpeed/pull/3782
use Flops Profiler to test model.generate() by @CaffreyR in https://github.com/microsoft/DeepSpeed/pull/2515
[zero] revert PR #3611 by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3786

New Contributors

@straywarrior made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3664
@alito made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3720
@acforvs made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3768
@zhiruiluo made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3169
@CaffreyR made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2515

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.4...v0.9.5

Open source →

0.9.49 Jun 2023

[MiCS] [Fix] saving and loading model checkpoint logic for MiCS sharding by @zarzen in https://github.com/microsoft/DeepSpeed/pull/3440

What's Changed

[MiCS] [Fix] saving and loading model checkpoint logic for MiCS sharding by @zarzen in https://github.com/microsoft/DeepSpeed/pull/3440
fix some typo by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3675
Use logger in accelerator by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3682
Update README to add ICS'23 paper on Tensor Parallel MoEs by @siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/3687
non-JIT build fix on ROCm by @rraminen in https://github.com/microsoft/DeepSpeed/pull/3638
Fix local rank mismatch error when training on nodes with different number of GPUs by @byungsoo-oh in https://github.com/microsoft/DeepSpeed/pull/3409
Correct world_size/backend for mpi by @abhilash1910 in https://github.com/microsoft/DeepSpeed/pull/3694
Fix incorrectly formatted f string in hostfile checking by @loadams in https://github.com/microsoft/DeepSpeed/pull/3698
fix typo name of hybrid engine func by @tensor-tang in https://github.com/microsoft/DeepSpeed/pull/3689
Revert "fix typo name (#3689)" by @loadams in https://github.com/microsoft/DeepSpeed/pull/3702
Fix gpt-j inference issue by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/3639
change partititon_name to partition_name by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3700
Fix unit test typo in tests/unit/ops/transformer/inference by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3697
Small tweak on cuda version mismatch documentation by @jli in https://github.com/microsoft/DeepSpeed/pull/3706
DeepSpeed overview in Japanese by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3709
zero3 performance optimizations by @hablb in https://github.com/microsoft/DeepSpeed/pull/3622
Fix typo in name of hybrid engine function by @loadams in https://github.com/microsoft/DeepSpeed/pull/3704
Increase tensor creator coverage by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3684
[Bugfix][CPU] Remove C++ version in CPU OpBuilder by @delock in https://github.com/microsoft/DeepSpeed/pull/3643
Single Node is using unreferenced pdsh kill cmd while terminating by @abhilash1910 in https://github.com/microsoft/DeepSpeed/pull/3730
Update Dockerfile with newer cuda and torch. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3716

New Contributors

@byungsoo-oh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3409
@abhilash1910 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3694
@tensor-tang made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3689
@jli made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3706

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.3...v0.9.4

Open source →

0.9.32 Jun 2023

Enable auto TP policy for llama model by @jianan-gu in https://github.com/microsoft/DeepSpeed/pull/3170

What's Changed

Enable auto TP policy for llama model by @jianan-gu in https://github.com/microsoft/DeepSpeed/pull/3170
Allow users to use mis-matched CUDA versions by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3436
Hybrid Engine Refactor and Llama Inference Support by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3425
add sharded checkpoint loading for AutoTP path to reduce the peak mem… by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3102
launcher/multinode_runner.py: mapping env variables by @YizhouZ in https://github.com/microsoft/DeepSpeed/pull/3372
Update automatic-tensor-parallelism.md by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3198
Build: Update license in setup by @PabloEmidio in https://github.com/microsoft/DeepSpeed/pull/3484
Doc corrections by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3435
Fix spelling errors in comments and documents by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3486
Fix spelling error in function GetMaxTokenLength() by @luliyucoordinate in https://github.com/microsoft/DeepSpeed/pull/3482
Fix a type error on bf16+Pipeline Parallelism by @ys950902 in https://github.com/microsoft/DeepSpeed/pull/3441
Fix spelling errors in DeepSpeed codebase by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3494
fix spelling error with docs/index.md by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3443
delete the line to keep user_zero_stages by @MrZhengXin in https://github.com/microsoft/DeepSpeed/pull/3473
Update Inference Engine checkpoint loading + meta tensor assertions by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2940
fix regression in shard checkpoint loading in AutoTP Path caused by qkv_copy() is deleted and add UT case for shard checkpoint loading in AutoTP by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3457
Add snip_momentum structured pruning which supports higher sparse ratio by @ftian1 in https://github.com/microsoft/DeepSpeed/pull/3300
Update README.md by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3504
Hybrid Engine Fix Llama by @lekurile in https://github.com/microsoft/DeepSpeed/pull/3505
fix spelling error with deepspeed/runtime/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3509
Skip autoTP if tp_size is 1 by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/3449
Changing monitor loss to aggregate loss over gradient accumulation steps by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3428
change actions/checkout@v2 to v3 by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3526
fix typo with docs/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3523
Doc updates by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3520
Fix bug in Hybrid Engine by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3497
Fix wrong passing of offload_optimizer_config to DeepSpeedZeRoOffload by @mmhab in https://github.com/microsoft/DeepSpeed/pull/3420
Fix broadcast error on multi-node training with ZeroStage3 and TensorParallel=2 by @YizhouZ in https://github.com/microsoft/DeepSpeed/pull/2999
share inflight registry between PartitionedParameterCoordinators by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3462
Syncing FusedAdam with new Apex features by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3434
fix typo in comments with deepspeed/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3537
[ROCm] Hip headers fix by @rraminen in https://github.com/microsoft/DeepSpeed/pull/3532
[CPU] Support Intel CPU inference by @delock in https://github.com/microsoft/DeepSpeed/pull/3041
Clone tensors to avoid torch.save bloat by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3348
Fix attribute error when loading FusedAdamBuilder() by @rraminen in https://github.com/microsoft/DeepSpeed/pull/3527
fix typo by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/3559
Fixing bf16 test by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3551
Fix Hybrid Engine for BLOOM by @lekurile in https://github.com/microsoft/DeepSpeed/pull/3580
Fix op_builder against PyTorch nightly by @malfet in https://github.com/microsoft/DeepSpeed/pull/3596
data efficiency bug fix, avoid invalid range step size by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3609
DS init should not broadcast or move zero.Init models by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3611
Expose Consecutive Hysteresis to Users by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/3553
Align InferenceEngine to store ms in _model_times by @HolyFalafel in https://github.com/microsoft/DeepSpeed/pull/3501
AISC launcher fixes by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3637
stage3.py: do not scale if gradient_predivide_factor is 1.0 by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3630
Add Ascend NPU accelerator support by @CurryRice233 in https://github.com/microsoft/DeepSpeed/pull/3595
Skip tests on docs-only changes by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3651
Update megatron.md by @wjessup in https://github.com/microsoft/DeepSpeed/pull/3641
Typo Correction by @MicahZoltu in https://github.com/microsoft/DeepSpeed/pull/3621
deepspeed/comm/comm.py: fix typo of warning message by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3636
Fix RuntimeError when using ZeRO Stage3 with mpu: #3564 by @eggiter in https://github.com/microsoft/DeepSpeed/pull/3565
Allow dict datatype for checkpoints (inference) by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3007
fix typo with deepspeed/ by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3547
flops_profiler: add option recompute_fwd_factor for the case of activation c… by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3362
fix typo deepspeed/runtime by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3663
Refactor check_enabled root validator in DeepSpeedMonitorConfig by @bgr8 in https://github.com/microsoft/DeepSpeed/pull/3616

New Contributors

@jianan-gu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3170
@YizhouZ made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3372
@PabloEmidio made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3484
@luliyucoordinate made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3482
@ys950902 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3441
@MrZhengXin made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3473
@ftian1 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3300
@mmhab made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3420
@malfet made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3596
@HolyFalafel made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3501
@CurryRice233 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3595
@wjessup made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3641
@MicahZoltu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3621
@eggiter made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3565
@bgr8 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3616

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.2...v0.9.3

Open source →

0.9.23 May 2023

MiCS implementation by @zarzen in https://github.com/microsoft/DeepSpeed/pull/2964

What's Changed

MiCS implementation by @zarzen in https://github.com/microsoft/DeepSpeed/pull/2964
Fix formatting by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3343
[ROCm] Hipify cooperative_groups headers by @rraminen in https://github.com/microsoft/DeepSpeed/pull/3323
Diffusers 0.15.0 bug fix by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/3345
Print default values for DeepSpeed --help by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3347
add bf16 cuda kernel support by @dc3671 in https://github.com/microsoft/DeepSpeed/pull/3092
README.md: Update MosaicML docs link by @kobindra in https://github.com/microsoft/DeepSpeed/pull/3344
hybrid_engine: check tuple size when fusing lora params by @adammoody in https://github.com/microsoft/DeepSpeed/pull/3311
fix mpich launcher issue in multi-node by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3078
Update DS-Chat issue template by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3368
add deepspeed chat blog links, add tags by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3369
Fix redundant shared_params in zero_to_fp32.py by @ShijieZZZZ in https://github.com/microsoft/DeepSpeed/pull/3149
fixing default communication_data_type for bfloat16_enabled and docs by @clumsy in https://github.com/microsoft/DeepSpeed/pull/3370
Auto TP Tutorial with T5 Example by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2962
stage_1_and_2.py: do gradient scale only for fp16 by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/3166
Fix memory leak in zero2 contiguous gradients by @hablb in https://github.com/microsoft/DeepSpeed/pull/3306
remove megatron-lm, no longer pip installable by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3389
Fix pipeline module evaluation when contiguous activation checkpoin… by @hablb in https://github.com/microsoft/DeepSpeed/pull/3005
doc updates by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3415
Save tensors in context of memory_efficient_linear by @tohtana in https://github.com/microsoft/DeepSpeed/pull/3413
Add HE support for the rest of model containers by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/3191
Update PyTorch Lightning/DeepSpeed examples links by @loadams in https://github.com/microsoft/DeepSpeed/pull/3424
Fix PipelineEngine.eval_batch result by @nrailgun in https://github.com/microsoft/DeepSpeed/pull/3316
OPT Activation Function Hotfix by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3400
Add ZeRO 1 support to PP for BF16. by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3399
[zero_to_fp32] fix shared param recovery by @stas00 in https://github.com/microsoft/DeepSpeed/pull/3407
Adagrad support in ZeRO by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/3401
Update 2020-09-09-sparse-attention.md by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3432

New Contributors

@dc3671 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3092
@kobindra made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3344
@hablb made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3306
@nrailgun made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3316

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.1...v0.9.2

Open source →

0.9.121 Apr 2023

Update DS-Chat docs for v0.9.0 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3216

What's Changed

Update DS-Chat docs for v0.9.0 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3216
Update DeepSpeed-Chat docs with latest changes to scripts by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3219
Nested zero.Init() and dynamically defined model class by @tohtana in https://github.com/microsoft/DeepSpeed/pull/2989
Update torch version check in building sparse_attn by @loadams in https://github.com/microsoft/DeepSpeed/pull/3152
Fix for Stable Diffusion by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3218
[update] reference in cifar-10 by @dtunai in https://github.com/microsoft/DeepSpeed/pull/3212
[fp16/doc] correct initial_scale_power default value by @stas00 in https://github.com/microsoft/DeepSpeed/pull/3275
update link to PL docs by @Borda in https://github.com/microsoft/DeepSpeed/pull/3237
fix typo in autotuner.py by @eltociear in https://github.com/microsoft/DeepSpeed/pull/3269
improving int4 asymmetric quantization accuracy by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/3190
Update install.sh by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3270
Fix cupy install version detection by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3276
[ROCm] temporary workaround till __double2half support enabled in HIP by @bmedishe in https://github.com/microsoft/DeepSpeed/pull/3236
Fix pydantic and autodoc_pydantic version to <2.0.0 until support is added. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3290
Add contribution images to readme by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3282
remove torch.cuda.is_available() check when compiling ops by @jinzhen-lin in https://github.com/microsoft/DeepSpeed/pull/3085
Update MI200 workflow to install apex with changes from pip by @loadams in https://github.com/microsoft/DeepSpeed/pull/3294
Add pre-compiling ops test by @loadams in https://github.com/microsoft/DeepSpeed/pull/3277
Update README.md by @digger-yu in https://github.com/microsoft/DeepSpeed/pull/3315
Update Dockerfile to use python 3.6 specifically by @bobowwb in https://github.com/microsoft/DeepSpeed/pull/3298
zero3 checkpoint frozen params by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3205
Fix for dist not being initialized when constructing main config by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3324
Fix missing scale attributes for GPTJ by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3256
Explicitly check for OPT activation function by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3278

New Contributors

@dtunai made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3212
@Borda made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3237
@digger-yu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3270
@bmedishe made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3236
@jinzhen-lin made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3085
@bobowwb made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3298

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.9.0...v0.9.1

Open source →

0.9.013 Apr 2023Deprecated

♻️ replace deprecated functions for communication by @mayank31398 in https://github.com/microsoft/DeepSpeed/pull/2995

New features

🚀 DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales 🚀

What's Changed

[docs] add MCR-DL paper to readme/docs by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/3066
Several fixes to unblock CI by @loadams in https://github.com/microsoft/DeepSpeed/pull/3047
Assert mp_size is factor of model dimensions by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2891
[CI] follow-up fixes by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3072
fix return prev key and value , added strides to from_blob by @mzusman in https://github.com/microsoft/DeepSpeed/pull/2828
Remove bf16 from inference config dtye enum by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/3010
Softmax Scheduling Cleanup by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/3046
Fix nebula in save_16bit_model issue by @FreyaRao in https://github.com/microsoft/DeepSpeed/pull/3023
Allow lists by @satpalsr in https://github.com/microsoft/DeepSpeed/pull/3042
Goodbye Torch 1.8 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3082
Empty ZeRO3 partition cache by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3060
pre-commit check for torch.cuda in code by @delock in https://github.com/microsoft/DeepSpeed/pull/2981
Move cuda check into utils by @loadams in https://github.com/microsoft/DeepSpeed/pull/3074
update yapf version and style settings by @jeffra in https://github.com/microsoft/DeepSpeed/pull/3098
Fix comms benchmark import issues and support MPI/slurm launching by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2932
Disable Stage 1&2 CPUAdam pathways by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3097
♻️ replace deprecated functions for communication by @mayank31398 in https://github.com/microsoft/DeepSpeed/pull/2995
Make fp32 default communication data type by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2970
Update DeepSpeed copyright license to Apache 2.0 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3111
Add Full Apache License by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3119
VL MoE Blog by @yaozhewei in https://github.com/microsoft/DeepSpeed/pull/3120
Update SD triton version in requirements-sd.txt by @lekurile in https://github.com/microsoft/DeepSpeed/pull/3135
Fix launch issue by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3137
Fix CI badges by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3138
Optimize Softmax Kernel by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/3112
Use generic O_DIRECT by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3115
Enable autoTP for bloom by @sywangyi in https://github.com/microsoft/DeepSpeed/pull/3035
[cleanup] remove pass calls where they aren't needed by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2826
[ci] nv-transformers-v100 - use the same torch version as transformers CI by @stas00 in https://github.com/microsoft/DeepSpeed/pull/3096
Fixes code and tests skipping/asserting incorrectly on torch 2+. by @loadams in https://github.com/microsoft/DeepSpeed/pull/3136
fix example symlink about DeepSpeed+AzureML by @EeyoreLee in https://github.com/microsoft/DeepSpeed/pull/3127
Remove Extra Bracket by @VHellendoorn in https://github.com/microsoft/DeepSpeed/pull/3101
Recover shared parameters by @ShijieZZZZ in https://github.com/microsoft/DeepSpeed/pull/3033
Fix for Diffusers 0.14.0 by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/3142
Fix copyright check, add copyright replace script by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3141
Update curriculum-learning.md by @goodship1 in https://github.com/microsoft/DeepSpeed/pull/3031
Remove benchmark code by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/3157
fixing a bug in CPU Adam and Adagrad by @xiexbing in https://github.com/microsoft/DeepSpeed/pull/3109
op_builder: conditionally compute relative path for hip compiled files by @adammoody in https://github.com/microsoft/DeepSpeed/pull/3095
zero.Init() should pin params in GPU memory as requested by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2953
deepspeed/runtime/utils.py: reset_peak_memory_stats when empty cache by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2803
Add DeepSpeed-Chat Blogpost by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/3185
[docs] add run command for 13b by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/3187
add news item. by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/3188
DeepSpeed Chat by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/3186
Fix references to figures by @tohtana in https://github.com/microsoft/DeepSpeed/pull/3189
Fix typo by @zhouzaida in https://github.com/microsoft/DeepSpeed/pull/3183
Fix typo by @dawei-wang in https://github.com/microsoft/DeepSpeed/pull/3164
Chatgpt chinese blog by @yaozhewei in https://github.com/microsoft/DeepSpeed/pull/3193
Add Japanese version of ChatGPT-like pipeline blog by @tohtana in https://github.com/microsoft/DeepSpeed/pull/3194
fix hero figure by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3199
feat: Add support for NamedTuple when sharding parameters [#3029] by @AlexanderVanEck in https://github.com/microsoft/DeepSpeed/pull/3037
fix license badge by @conglongli in https://github.com/microsoft/DeepSpeed/pull/3200
Update AMD workflows by @loadams in https://github.com/microsoft/DeepSpeed/pull/3179
[CPU support] Optionally bind each rank to different cores on host by @delock in https://github.com/microsoft/DeepSpeed/pull/2881

New Contributors

@mzusman made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2828
@FreyaRao made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3023
@sywangyi made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3035
@EeyoreLee made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3127
@VHellendoorn made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3101
@goodship1 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3031
@zhouzaida made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3183
@dawei-wang made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3164
@AlexanderVanEck made their first contribution in https://github.com/microsoft/DeepSpeed/pull/3037

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.8.3...v0.9.0

Open source →

0.8.27 Mar 2023Deprecated

Remove deprecated torch._six imports by @yasyf in https://github.com/microsoft/DeepSpeed/pull/2863

What's Changed

add auto-generated PR workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2822
Fix typo in auto-sync workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2850
Fix example command for building wheel with dev version specified. by @loadams in https://github.com/microsoft/DeepSpeed/pull/2815
Create tensor parallelism blog/tutorial by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2766
Data efficiency library update by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2866
Make z3 respect comm dtype by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2807
Automatic Tensor Parallelism Blog Links by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2877
Check device count before running dist tests by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/2799
AutoTP tutorial web formatting and news by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2883
Remove deprecated torch._six imports by @yasyf in https://github.com/microsoft/DeepSpeed/pull/2863
Reduce I/O size by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2814
add missing license info to top of all source code by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2889
Enable tensor fragments for zero 2 & 3 by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2727
better eval sampler for val or test dataset by @mayank31398 in https://github.com/microsoft/DeepSpeed/pull/2907
using container when loading inference checkpoints by @HeyangQin in https://github.com/microsoft/DeepSpeed/pull/2875
Fix CPUAdam for when vendor_id_raw is not provided by @FarzanT in https://github.com/microsoft/DeepSpeed/pull/2836
Fix Bloom logits mismatch by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2851
Fixes AttributeError in #2853 by @saforem2 in https://github.com/microsoft/DeepSpeed/pull/2854
Add MPICH Multinode Runner by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/2839
TP unsupported models and assertions by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2810
AutoTP Assert Kernel Injection Support by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2939
Check for local CUDA graphs when enable_cuda_graph=True by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2941
Improve overflow handling by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2944
[RFC] add device abstraction to allow other device than CUDA be used by @delock in https://github.com/microsoft/DeepSpeed/pull/2221
deepspeed.init_distributed() support for TCP protocols by @noabauma in https://github.com/microsoft/DeepSpeed/pull/2905

New Contributors

@HeyangQin made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2799
@yasyf made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2863
@mayank31398 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2907
@FarzanT made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2836
@saforem2 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2854
@noabauma made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2905

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.8.1...v0.8.2

Open source →

0.8.117 Feb 2023

CUDA optional deepspeed ops by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2507

What's Changed

CUDA optional deepspeed ops by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2507
Remove CI trigger for push to master by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2712
[install] only add deepspeed pkg at install by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2714
Fix nightly tests for new lm-eval release by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2713
BF16 optimizer for BF16+ZeRO Stage 1 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2706
Fix typo in diffusers transformer block by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2718
Inference Refactor (replace_with_policy, model_implementations) by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2554
Change zero_grad() argument to match pytorch by @loadams in https://github.com/microsoft/DeepSpeed/pull/2741
Automatic tensor parallelism v2 by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2670
Fixing Optimizer Sanity Check by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2742
[GatheredParameters] fix memory leak by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2665
Abstract accelerator (step 3) by @delock in https://github.com/microsoft/DeepSpeed/pull/2677
Fix autotuning so that it records Floating Point Operations per second, not microsecond by @dashstander in https://github.com/microsoft/DeepSpeed/pull/2711
fix a misspelled attribute by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2750
[zero] remove misleading dtype log by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2732
Fix softmax backward by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2709
Skip test_bias_gelu unit test if torch < 1.12 by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2754
Conditionally Make Op Building More Verbose by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2759
Bing/formatting correction by @xiexbing in https://github.com/microsoft/DeepSpeed/pull/2764
Add links to new azureML examples by @cassieesvelt in https://github.com/microsoft/DeepSpeed/pull/2756
Fix hardcoded instances to fp16 in optimizer creation log messages to the correct dtype. by @loadams in https://github.com/microsoft/DeepSpeed/pull/2743
Refactor/Pydantify monitoring config by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2640
Pin minimum packaging requirement by @carmocca in https://github.com/microsoft/DeepSpeed/pull/2771
Fix for diffusers v0.12.0 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2753
some fix in flops_profiler by @lucasleesw in https://github.com/microsoft/DeepSpeed/pull/2068
fix upsample flops compute by skipping unused kargs by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2773
Fix broken kernel inject bug by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2776
Fix Checkpoint-loading with Meta-tensor by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2781
Add hjson support for user configs by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2783
Reset KV-cache at the beginning of text-generation by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2669
Container param cleanup + remove qkv_merging by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2780
Common location to install libaio-dev by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2779
Fixing broken link to azureml-examples recipes by @rtanase in https://github.com/microsoft/DeepSpeed/pull/2795
remove outdated comment by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2786
Enable page-locked tensors without CUDA by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2775
Add container load checkpoint error reporting + refactor by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2792
Add user defined launcher args for PDSH launcher by @loadams in https://github.com/microsoft/DeepSpeed/pull/2804
Fix Slurm launcher user args by @loadams in https://github.com/microsoft/DeepSpeed/pull/2806
Handle hanged tests in CI by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2808
Fix inference CI device error by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2824
Fix permissions issue with pip upgrade by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2823
Fix cpu-only CI hangs by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2825
Fix Pipeline Parallel resize unit test by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2833
Fix auto TP for duplicate modules with different gems by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2784
Refactor DS inference API. No longer need replace_method. by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2831
Port Reza's INT8-quantization fix to container architecture by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2725
Fix gpt-Neox rotary embedding implementation by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2782
Fix for CI failure on system upgrade by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2849

New Contributors

@loadams made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2741
@xiexbing made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2764
@carmocca made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2771
@lucasleesw made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2068
@rtanase made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2795

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.8.0...v0.8.1

Open source →

0.8.017 Jan 2023

DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality

New features

DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality
DeepSpeed Data Efficiency Library by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2585

What's Changed

fix blog link by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2600
Migrate ops tests to new inference_ops marker by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2599
Move layer norm to new schedule by @lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2590
[deepspeed/autotuner] Bug fix for binary search for batch size by @rahilbathwal5 in https://github.com/microsoft/DeepSpeed/pull/2162
Fix for older versions of pydantic by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2611
Use rocm/pytorch:latest for ROCm Dockerfile by @jithunnair-amd in https://github.com/microsoft/DeepSpeed/pull/2613
skip torch.zeros and tensor.copy_ when model parallel is not used by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2479
call empty_cache to really free up GPU memory as described in comment by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2620
Remove GatheredParameters context from replace_with_policy by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2591
fixes #2498 by @clumsy in https://github.com/microsoft/DeepSpeed/pull/2603
Update AVX512 Detection by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2621
Add Megatron CI workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2614
[inference] check for unsupported model generate args by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2627
[launcher] parse hostfile via regex and added error checks by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2626
Unit tests setup own venv by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2628
Fix #2409: add enable_each_rank_log to deepspeed/launcher/runner.py by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/2571
Fix typo in autotuner.py by @eltociear in https://github.com/microsoft/DeepSpeed/pull/2639
[zero-3] Handle forward parameter return correctly in nested cases by @samyam in https://github.com/microsoft/DeepSpeed/pull/2642
[inference] ds-attention refactor w.r.t. ops by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2623
Fix issue w. bloom int8 when changing tp size by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2645
fix assertion error in zero stage 3 by @GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2647
tweaks to ds-attn, distilbert policy, and mup by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2649
[doc] fix min_loss_scale default by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2660
[launcher] fail gracefully if hostname -i doesn't work as expected by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2631
Fix Opt injection by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2541
Abstract accelerator (step 2) by @delock in https://github.com/microsoft/DeepSpeed/pull/2560
Remove unnecessary device synchronization for stage 2 by @li-yi-dong in https://github.com/microsoft/DeepSpeed/pull/2500
[Bug Fixed] torch.cuda.is_available -> torch.cuda.is_available() by @wkcn in https://github.com/microsoft/DeepSpeed/pull/2661
[fp16] lower initial_scale_power to 16 by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2663
fix Tensor contiguous bug in model_compression by @xiaoxiawu-microsoft in https://github.com/microsoft/DeepSpeed/pull/2671
[inference] ds-mlp refactor w.r.t. ops by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2668
real_accelerator validation check for both accelerator and deepspeed accelerator path by @delock in https://github.com/microsoft/DeepSpeed/pull/2685
fix typo and remove duplicated code in ZeRO stage 1 and 2 by @wkcn in https://github.com/microsoft/DeepSpeed/pull/2655
Add mlflow logging for aml by @cassieesvelt in https://github.com/microsoft/DeepSpeed/pull/2495
Fix import error of op_builder by @tohtana in https://github.com/microsoft/DeepSpeed/pull/2687
Pass training flag to forward call from module config by @lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2604
Extend quantization utils features by @lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2683
[GatheredParameters] add support for any iterable by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2664
Fix for latest diffusers by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2699
exclude benchmarks during install by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2698
Correct loss scale in ZeRO step by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2695
[ZeRO] non-MoE stage 1 requires CG disabled by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2703
remove print side effect from importing deepspeed by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2704
ZeRO3 handling frozen weights by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2653

New Contributors

@eltociear made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2639
@li-yi-dong made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2500
@wkcn made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2661
@xiaoxiawu-microsoft made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2671
@cassieesvelt made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2495
@tohtana made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2687

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.7...v0.8.0

Open source →

0.7.61 Dec 2022

DeepSpeed inference config. (#2459) by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2472

What's Changed

DeepSpeed inference config. (#2459) by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2472
Update docs to autogenerate pydantic config model docs by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2509
Add max_tokens alias to max_out_tokens arg to maintain backwards compatibility by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2508
Deepspeed quantization library v0.1 by @lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2450
Fix backward compatibility for InferenceConfig by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2516
Add missing Inference sub-configs by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2518
Add note about nvcc/hipcc requirement by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2519
Update codeowners by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2525
Dequantization Utils Library by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2521
Fixes for torch 1.14 due to new torch.numel return type by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2522
Ensure MOE is initialized for SD by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2534
Make DS-Inference config readable from JSON by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2537
Add MII tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2533
Remove mutable default parameter in init_inference() by @aphedges in https://github.com/microsoft/DeepSpeed/pull/2540
Change Where DS/Triton is Used in Stable Diffusion by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2536
Pass down the new DS inference config to replace_transformer_layer. by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2539
Adding Gradient Accumulation Data Type Config by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2512
Report progress at gradient accumulation boundary by @ShijieZZZZ in https://github.com/microsoft/DeepSpeed/pull/2553
encoded ds config into command line argument when launching child processes in autotuning by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2524
Add missing MoE fields to inference config for backward compatibility by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2556
Abstract accelerator (step 1) by @delock in https://github.com/microsoft/DeepSpeed/pull/2504
Fix invalid check of recorded parameter orders in zero stage3. by @inkcherry in https://github.com/microsoft/DeepSpeed/pull/2550

New Contributors

@ShijieZZZZ made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2553
@delock made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2504
@inkcherry made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2550

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.5...v0.7.6

Open source →

0.7.514 Nov 2022

Fix Bug #2319 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2438

What's Changed

Fix Bug #2319 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2438
update pytorch pool operator function signiture by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2443
Fix build issues on Windows by @eltonzheng in https://github.com/microsoft/DeepSpeed/pull/2428
rollback ds config changes by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2395
Use CUDA events for inference model profiling by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2371
Fixing a config mismatch in unit test. by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2447
Reduction Kernel Utility by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2436
deepspeed/launcher/launch.py: add option enable_each_rank_log by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2409
Fixes for various CI problems by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2457
Cache Allocation and Softmax Fixes by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2433
Fix checkpoint loading at inference-engine by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2429
Create a new folder structure to isolate model-specific code in DS by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2464
don't gather partitioned activations for mp size 1 by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2454
Updating autotune json default in docs. by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2476
Added MLFLOW environment variables for logging metrics within trainig… by @savitamittal1 in https://github.com/microsoft/DeepSpeed/pull/2477
fix accelerate link in README by @kyoto7250 in https://github.com/microsoft/DeepSpeed/pull/2481
Fix Stable-Diffusion: Add correct memory-allocation at DeepSpeed-Attention by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2474
Fix CI issues related to cupy install by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2483
Add scale_attn_by_inverse_layer_idx feature by @hyunwoongko in https://github.com/microsoft/DeepSpeed/pull/2486
Stable Diffusion Enhancements by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2491
stage_1_and_2.py: no allreduce needed when mp size is 1 by @guoyejun in https://github.com/microsoft/DeepSpeed/pull/2494
Make bf16_optimizer work for non pipeline parallelism by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2470
Fix nightly CI tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2493
Make data contiguous before the inplace reshape-copy_ function. by @lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2489
Fix typos: deepseed -> deepspeed by @jinyouzhi in https://github.com/microsoft/DeepSpeed/pull/2499

New Contributors

@guoyejun made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2409
@savitamittal1 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2477
@kyoto7250 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2481
@lokoppakmsft made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2489
@jinyouzhi made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2499

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.4...v0.7.5

Open source →

0.7.421 Oct 2022

MOE residual matmult unit test by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2323

What's Changed

MOE residual matmult unit test by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2323
MOE matmult with memaccess by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2336
Refactor residual add kernels by @arashb in https://github.com/microsoft/DeepSpeed/pull/2333
mem access for quantize kernel by @GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2331
increase min pre-commit versions by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2346
Extend scratch buffer for long prompts by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2212
[docs] fix zero docs by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2350
Staging profile inference v1 (#2348) by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2349
Kernel Data Conversion Utility by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2327
Add Onebit Optimizers in init by @l4d2boomer in https://github.com/microsoft/DeepSpeed/pull/2340
docs(mixture-of-experts-inference): fix typo in tuto by @jqueguiner in https://github.com/microsoft/DeepSpeed/pull/2345
Use blob storage for datasets in unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2342
Refactor gptj_residual_add kernels for better readability by @arashb in https://github.com/microsoft/DeepSpeed/pull/2358
Updated issue templates by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2363
fix cuda invalid config error in dequant kernel by @GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2362
Add missing pytest fixture scope by @arashb in https://github.com/microsoft/DeepSpeed/pull/2353
Extend residual_add kernel tests to cover pre_attn_norm by @arashb in https://github.com/microsoft/DeepSpeed/pull/2354
Refactor fused_bias_residual kernels for better readability by @arashb in https://github.com/microsoft/DeepSpeed/pull/2356
Capture error message during sweep tests by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2351
Fix an exception when auto-casting dicts to fp16 by @mjksmith in https://github.com/microsoft/DeepSpeed/pull/2370
Refactor remaining distributed tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2216
Fix the MLP output tensor's shape by @arashb in https://github.com/microsoft/DeepSpeed/pull/2380
add 11.8 to cuda_minor_mismatch_ok to allow building with current CUDA by @Thomas-MMJ in https://github.com/microsoft/DeepSpeed/pull/2390
Pin Transformers test version by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2402
Change type to tuple in replace_wo_policy isinstance check by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2387
Checkpoint backwards-compatbility workaround by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2384
Add Predicated Global Load to Memory Access Utils by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2373
MII blog post by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2418
Fix figure reference by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2419
Add SLURM Multinode Runner by @dashstander in https://github.com/microsoft/DeepSpeed/pull/2404
Fix issue with corrupted output on long generation for GPT by @andrewchernyh in https://github.com/microsoft/DeepSpeed/pull/2359
Fix GPT Neo-X multi-gpu inference by @andrewchernyh in https://github.com/microsoft/DeepSpeed/pull/2401
CI fixes related to triton by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2422
[docs] update mii blog title by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2423
add SD injection policy by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2381
Fix checkpoint loading when it is a dictionary by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2425
Make error regex more generic in collect_results.py by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2415
fixes #2389 by @clumsy in https://github.com/microsoft/DeepSpeed/pull/2411
Fix for inference gpt-j test by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2430
Fixing bug 2361 by @jomayeri in https://github.com/microsoft/DeepSpeed/pull/2410
Universal checkpoint for zero stage 1 by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2284
only add deps if extra is explicitly called by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2432
Add TestInjectionPolicy inference unittest class for testing custom injection policies by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2426
[memory estimators] new config args sync by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2431
parallelize writing of layer checkpoint files across data parallel instances by @adammoody in https://github.com/microsoft/DeepSpeed/pull/1419
Fix broken link to DeepSpeed Megatron fork by @lekurile in https://github.com/microsoft/DeepSpeed/pull/2440

New Contributors

@l4d2boomer made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2340
@jqueguiner made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2345
@mjksmith made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2370
@Thomas-MMJ made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2390
@lekurile made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2387
@dashstander made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2404
@andrewchernyh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2359
@clumsy made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2411
@jomayeri made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2410

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.3...v0.7.4

Open source →

0.7.319 Sept 2022

Add blob storage to CI runners by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2260

What's Changed

Add blob storage to CI runners by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2260
Update replace_module.py, test-gptj.py related fix by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2269
Fix OrderedDict import for python3.6 by @Dipet in https://github.com/microsoft/DeepSpeed/pull/2267
Ds inference/fix mp2 by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2270
Trajepl: nebula load fix by @trajepl in https://github.com/microsoft/DeepSpeed/pull/2182
Prevent torch ext folder mkdir at tmp by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2274
Ds-inference Int8 support through ZeroQuant technology by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2217
add a new unit test for cuda ops by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2278
Addition to code owners file by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2279
Memory Access Utility by @cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2276
Fp32 accuracy bug fix by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2285
Refactor universal checkpointing and tensor fragments by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2253
[ds-inference] fix progress bar by @stas00 in https://github.com/microsoft/DeepSpeed/pull/2286
Offload all gradients to nvme by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2282
fused bias relu unittest by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2297
Fix for pytest picking up wrong deepspeed by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2299
Fix for Zero3 when MP>1 by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2289
Unit test for bias add kernel by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2298
Update relu.cu with mem_access_utils by @molly-smith in https://github.com/microsoft/DeepSpeed/pull/2306
Add tensor parallel inference unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2232
Fix the residual add mp scaling for GPTNeoX by @arashb in https://github.com/microsoft/DeepSpeed/pull/2310
Add unit tests for residual_add kernel by @arashb in https://github.com/microsoft/DeepSpeed/pull/2307
add inference eval scripts by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2303
Upgrade P40 tests to torch 1.8 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2316
ZeRO-Inference blog by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2271
ZeRO-Inference blog - wrap up by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2321
ZeRO-Inference blog - Update README by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2322
Refactor relu bias add with mem_access utils by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2317
add quant unit test by @GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2315
only override forward if using cuda-graph by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2291
Add more options to inference benchmark by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2325

New Contributors

@molly-smith made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2269

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.2...v0.7.3

Open source →

0.7.123 Aug 2022

Fix for distributed tests on pytorch>=1.12 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2141

What's Changed

Fix for distributed tests on pytorch>=1.12 by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2141
delay torch import for inference compatability check by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2167
Fix wrong unit of latency in flops-profiler (#2090) by @zionwu in https://github.com/microsoft/DeepSpeed/pull/2095
[docs] adoption updates by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2173
Update for AMD CI workflow by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2172
[docs] update offload docs to include stage 1 by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2178
Fixing model partitioning without injection by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2179
Match compute and reduce dtype by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2145
Enable fused_lamb_cuda_kernel on ROCm by @rraminen in https://github.com/microsoft/DeepSpeed/pull/2148
Update README to latest Composer version by @hanlint in https://github.com/microsoft/DeepSpeed/pull/2177
[deepspeed/autotuner] Missing hjson import by @rahilbathwal5 in https://github.com/microsoft/DeepSpeed/pull/2175
[docs] add more models to adoption by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2189
[CI] fix lightning tests by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2190
Fix typos on README.md by @gasparitiago in https://github.com/microsoft/DeepSpeed/pull/2192
Fix the layer-past for GPT based models by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2196
Add gradient_average flag support for sparse grads by @Dipet in https://github.com/microsoft/DeepSpeed/pull/2188
Adding the compression tutorial on GPT distillation and quantization by @minjiaz in https://github.com/microsoft/DeepSpeed/pull/2197
Log user config exactly by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2201
Fix the tensor-slicing copy for qkv parameters by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2198
Refactor Distributed Tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2180
fix table syntax by @kamalkraj in https://github.com/microsoft/DeepSpeed/pull/2204
Correctly detect offload configuration by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2208
add cuda 11.7 by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2211
use torch 1.9 in accelerate tests by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2215
[zero-3] print warning once and support torch parameter by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/2127
Add support of OPT models by @arashb in https://github.com/microsoft/DeepSpeed/pull/2205
fix typos in readme. by @zhjohnchan in https://github.com/microsoft/DeepSpeed/pull/2218
Fix regression w. dist_init_required by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2225
add doc for new bert example by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2224
Remove the random-generator from context during inference by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2228
allow saving ckpt w/o ckpt json + bloom copy fix by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2237
Correctly detect zero_offload by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2213
[docs] update community videos by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2249
Refactor dist tests: Checkpointing by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2202
Make OPT policy backward compatible with pre-OPT transformers versions by @arashb in https://github.com/microsoft/DeepSpeed/pull/2254
fix ds-inference without policy by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2247

New Contributors

@zionwu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2095
@hanlint made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2177
@rahilbathwal5 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2175
@gasparitiago made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2192
@arashb made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2205
@zhjohnchan made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2218

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.7.0...v0.7.1

Open source →

0.7.01 Aug 2022

DeepSpeed Compression: https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-

New features

DeepSpeed Compression: https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/

What's Changed

Adding DeepSpeed Compression Composer by @yaozhewei in https://github.com/microsoft/DeepSpeed/pull/2105
Remove hardcoded ROCm install path by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2093
Fix softmax dim of Residual MoE implementation in moe/layer.py by @hero007feng in https://github.com/microsoft/DeepSpeed/pull/2110
reduce ds-inference log verbosity by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2111
DeepSpeed Compression announcement by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2114
Checkpoint reshaping by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1953
Fix init_process_group by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2121
DS Benchmarks QoL Improvements by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2120
[ROCm] Wrong command broke ROCm build. by @jpvillam-amd in https://github.com/microsoft/DeepSpeed/pull/2118
DeepSpeed Communication Profiling and Logging by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2012
Add flake8 to pre-commit checks by @aphedges in https://github.com/microsoft/DeepSpeed/pull/2051
Fix conflict between Tutel and top-2 gate in MoE layer by @yetiansh in https://github.com/microsoft/DeepSpeed/pull/2053
adding HF Accelerate+DS tests workflow by @pacman100 in https://github.com/microsoft/DeepSpeed/pull/2134
[inference tests] turn off time check for now by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2142
Allow turning off loss scaling wrt GAS + update tput calculator by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2140
Refactor ZeRO configs to use Pydantic by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2004
Add purely-local sliding window sparse attention config by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/1962
Trajepl/nebula ckpt engine by @trajepl in https://github.com/microsoft/DeepSpeed/pull/2085
Graceful exit on failures for multi-node runs by @jerrymannil in https://github.com/microsoft/DeepSpeed/pull/2008
fix: fix BF16_Optimizer compatibility issue by @shjwudp in https://github.com/microsoft/DeepSpeed/pull/2152
Fix random token-generation issue + MP-checkpoint loading/saving by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2132
Added retain_graph as a kwarg to the main engine backward function by @ncilfone in https://github.com/microsoft/DeepSpeed/pull/1149
Elastic Training support in DeepSpeed by @aj-prime in https://github.com/microsoft/DeepSpeed/pull/2156
prevent cuda 10 builds of inference kernels on ampere by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2157
[zero-3] shutdown zero.Init from within ds.init by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2150
enable fp16 input autocasting by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2158
Release swap buffers for persisted params by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2089
Tensor parallelism for Mixture of Experts by @siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2074

New Contributors

@hero007feng made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2110
@jpvillam-amd made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2118
@yetiansh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2053
@pacman100 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2134
@jimwu6 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2144
@trajepl made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2085
@ncilfone made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1149

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.6.7...v0.7.0

Open source →

0.6.618 Jul 2022

[docs] add 530b paper by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1979

What's Changed

[docs] add 530b paper by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1979
small fix for the HF Bert models by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/1984
Add unit test for various model families and inference tasks by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1981
Fix for lightning tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1988
fix typo when getting kernel dim in conv calculation by @cli99 in https://github.com/microsoft/DeepSpeed/pull/1989
Add torch-latest and torch-nightly CI workflows by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1990
[bug] Add user-defined launcher args for MPI launcher by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1933
Propagate max errorcode to deepspeed when using PDSH launcher by @jerrymannil in https://github.com/microsoft/DeepSpeed/pull/1994
[docs] add new build badges to landing page by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1998
DeepSpeed Comm. Backend v1 by @awan-10 in https://github.com/microsoft/DeepSpeed/pull/1985
Relax DeepSpeed MoE ZeRO-1 Assertion by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2007
update CODEOWNERS by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2017
[CI] force upgrade HF dependencies & output py env by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2015
[inference] test suite for ds-kernels (bert, roberta, gpt2, gpt-neo, gpt-j) by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1992
DeepSpeed examples refresh by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2021
Fix transformer API for training-evaluation pipeline by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2018
DataLoader Length Fix by @Sanger2000 in https://github.com/microsoft/DeepSpeed/pull/1718
DeepSpeed Monitor Module (Master) by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2013
Use partition numel by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2011
fix import errors by @KMFODA in https://github.com/microsoft/DeepSpeed/pull/2026
Fix inference unit test import error catching by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2024
Retain available params until last use by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2016
Split parameter offload from z3 by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/2009
Fix flops profiler print statements by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2038
Add compression papers by @conglongli in https://github.com/microsoft/DeepSpeed/pull/2042
Fix the half-precision version of CPU-Adam by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2032
Fix for AMD unit tests by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2047
Wrong partition_id while copying fp32_params -> fp16 params in Z2 for MoE by @siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2058
Fix missing import in replace_module.py by @aphedges in https://github.com/microsoft/DeepSpeed/pull/2050
Comms Benchmarks by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2040
add ds inference paper by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2072
Comments for better understanding of zero stage1_2 by @kisseternity in https://github.com/microsoft/DeepSpeed/pull/2027
[docs] fix broken read-the-docs build by @jeffra in https://github.com/microsoft/DeepSpeed/pull/2075
Fix building package without a GPU by @aphedges in https://github.com/microsoft/DeepSpeed/pull/2049
Fix partition id in the fp32->fp16 param copying step for z2+cpu-offload by @siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2059
Codeowner addendum and fix to small model debugging script by @samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2076
remove require grad in params count by @cli99 in https://github.com/microsoft/DeepSpeed/pull/2065
Add missing newline for ZeroOneAdam parameter table by @manuelciosici in https://github.com/microsoft/DeepSpeed/pull/2088
fixed "None type has no len()" by @xiazeyu in https://github.com/microsoft/DeepSpeed/pull/2091
Improving memory utilization of Z2+MoE by @siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2079

New Contributors

@jerrymannil made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1994
@Sanger2000 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1718
@KMFODA made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2026
@siddharth9820 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2058
@samadejacobs made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2076
@xiazeyu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2091

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.6.5...v0.6.6

Open source →

0.6.525 May 2022

GatheredParameters - accept a tuple of params by @stas00 in https://github.com/microsoft/DeepSpeed/pull/1941

What's Changed

GatheredParameters - accept a tuple of params by @stas00 in https://github.com/microsoft/DeepSpeed/pull/1941
Update partition_parameters.py by @manuelciosici in https://github.com/microsoft/DeepSpeed/pull/1943
fix step in adam by @szhengac in https://github.com/microsoft/DeepSpeed/pull/1823
[pipe] prevent deadlock with multiple evals sequence by @stas00 in https://github.com/microsoft/DeepSpeed/pull/1944
Fairseq support by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1915
DeepSpeed needs to start cleaning up by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1947
trivial fix by @kisseternity in https://github.com/microsoft/DeepSpeed/pull/1954
Enabling CUDA-graph for the bert-type models by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/1952
Add loss scale guard to avoid inf loop by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/1958
[launcher] add option to bypass ssh check by @liamcli in https://github.com/microsoft/DeepSpeed/pull/1957
Bump nokogiri from 1.13.4 to 1.13.6 in /docs by @dependabot in https://github.com/microsoft/DeepSpeed/pull/1965
Fix typo in timer.py by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/1964
[docs] fix dependabot version issue by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1966
Don't add curand on rocm by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1968
Add Unidirectional Sparse Attention Type to BigBird and BSLongformer by @Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/1959
Fix: Sparse tensors not updating by @Dipet in https://github.com/microsoft/DeepSpeed/pull/1914
Fixing several bugs in the inference-api and the kernels by @RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/1951

New Contributors

@Quentin-Anthony made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1958

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.6.4...v0.6.5

Open source →

0.6.327 Apr 2022

Fix setup.py crash when torch is not installed. by @PaperclipBadger in https://github.com/microsoft/DeepSpeed/pull/1866

What's Changed

Fix setup.py crash when torch is not installed. by @PaperclipBadger in https://github.com/microsoft/DeepSpeed/pull/1866
Add support for AWS SageMaker. by @matherit in https://github.com/microsoft/DeepSpeed/pull/1868
Fix broken links by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1873
[docs] add amd blog to website by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1874
[docs] add moe paper by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1875
Supporting multiple modules injection with a single policy when they … by @samyam in https://github.com/microsoft/DeepSpeed/pull/1869
[docs] fix dead links by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1877
add now required -lcurand to solve undefined symbol: curandCreateGenerator by @stas00 in https://github.com/microsoft/DeepSpeed/pull/1879
Bug fix for flops profilers output by @VisionTheta in https://github.com/microsoft/DeepSpeed/pull/1885
Bump nokogiri from 1.13.3 to 1.13.4 in /docs by @dependabot in https://github.com/microsoft/DeepSpeed/pull/1889
[docs] fix commonmarker security issue by @jeffra in https://github.com/microsoft/DeepSpeed/pull/1892
bf16+pipeline parallelism by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1801
fix file ordering by @szhengac in https://github.com/microsoft/DeepSpeed/pull/1822
Use f-strings where possible by @manuelciosici in https://github.com/microsoft/DeepSpeed/pull/1900
[partition_parameters.py] better diagnostics by @stas00 in https://github.com/microsoft/DeepSpeed/pull/1887
comm backend: cast bool when not supported by torch2cupy by @conglongli in https://github.com/microsoft/DeepSpeed/pull/1894
Use cuda events to improve timing for multi-stream execution by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1881
Fix multiple zero 3 tracing errors by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1901
Improve ds_report output for HIP/ROCm by @mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1906
Fix launcher for reading env vars by @szhengac in https://github.com/microsoft/DeepSpeed/pull/1907
Fix OOM and type mismatch by @tjruwase in https://github.com/microsoft/DeepSpeed/pull/1884

New Contributors

@PaperclipBadger made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1866
@matherit made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1868
@VisionTheta made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1885
@szhengac made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1822

Misc

v0.6.2 was skipped due to a build/deploy issue with that release

Full Changelog: https://github.com/microsoft/DeepSpeed/compare/v0.6.1...v0.6.3

Open source →

deepspeed

Release timeline

Releases

What's Changed

New Contributors

New features

What's Changed

New Contributors

New Features

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

New features

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

New features

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

New features

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

New features

What's Changed

New Contributors

What's Changed

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

Misc

DeepSpeed v0.6.0

Release notes

New features

Special thanks to our contributors in this release

DeepSpeed v0.4.0

v0.3.16 Release notes

v0.3.15 Release notes

v0.3.13 Release notes

v0.3.10 Release notes

Special thanks to our contributors in this release