PyPI · #5538 most downloaded on PyPI

nemo-toolkit

3.0.0489K downloads/moNVIDIA-NeMo/NeMo ↗New releases in your daily digest, changelog included.

NeMo - a toolkit for Conversational AI

Last release 1 months ago
07 Aug 2026
Release timing varies
gaps range from 2 weeks to 4 months
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
7 years old
109 releases · first in 2019

Release timeline

109 releases since 2019

20192020202120222023202420252026

Releases

3.0.07 Aug 2026Security
Speech Data Explorer gained S3 reading, comparison mode, security fixes, and tutorial updates (#15500, #15137).
NVIDIA NeMo Speech 3.0 Release Notes

NeMo Speech 3.0 is the first major release after the repo split and rename to NVIDIA-NeMo/Speech. The repo now focuses on ASR, TTS, audio processing, speaker tasks, and SpeechLM. Non-speech Framework, LLM, VLM, diffusion, export/deploy, evaluator components can now be found under separate repos under NVIDIA-NeMo organization.

While this release brings new major features, the central focus is addressing technical debt: removed 800k deprecated LOC, migration to uv package manager for cleaner installs, stronger model test coverage, reduced the number of dependencies, revamped documentation, lighter containers, and AGENTS.md + agentic skills.

Top-Level Changes and Highlights

Breaking Changes and Migration Notes

LLM/VLM/diffusion/NLP/export/evaluator collections have been moved out to separate repos.

Unsupported ASR/TTS models and old tutorials were removed. Use active examples under examples/ and docs under docs/source/.

uv sync reproduces the supported NeMo Speech stack and may replace Python/PyTorch/CUDA inside .venv. To keep your own stack, install PyTorch first, then use uv pip or pip.

SpeechLM2 compiled acceleration is optional. Use the supported Docker build for TE/FlashAttention/Mamba/grouped-GEMM/DeepEP performance paths.

SpeechLM2 and NeMo Automodel Backend

SpeechLM2 now supports training SpeechLM models with NeMo Automodel through SALMAutomodel, targeting both dense and MoE LLM backbones such as Nemotron 3.

SALMAutomodel adds an Automodel-backed SALM path with native LoRA, deferred configure_model() initialization, shard-aware distributed loading, and Automodel-owned mesh creation (#15447).

AutomodelParallelStrategy supports FSDP2, HSDP, TP, CP, and EP, letting SpeechLM training combine sharded data parallelism, tensor/model partitioning, long-context sequence sharding, and MoE expert routing (#15447, #15648, #15679, #15773).

Optimized backend support now covers Transformer Engine, FlashAttention, Mamba/state-space kernels, grouped GEMM for MoE experts, and DeepEP for expert-parallel all-to-all communication (#15737, #15758, #15679).

THD packed-sequence training reduces padding waste in variable-length speech batches and is the supported path for CP long-audio training (#15679).

Long-audio support was expanded with encoder chunking, activation-checkpointing controls, CP-safe data handling, and fixes around dtype/checkpoint restoration (#15648, #15679, #15686, #15716).

Export and serving improved with vLLM integration, HF/vLLM-ready checkpoint export, backbone-native chat templates, offline HF export, and buffered SALM inference (#15364, #15520, #15623, #15736).

Data format support now includes ShareGPT conversations, indexed ShareGPT JSONL, and WebDataset-backed ShareGPT conversations (#15316, #15410).

Nemotron VoiceChat Training Modules

NeMo Speech 3.0 adds the train/eval modules behind Nemotron VoiceChat.

Added Nemotron VoiceChat speech-decoder modules for the response-generation side of the duplex stack (#15066).

Added DuplexSTT training and inference for generating agent text from user speech plus text context (#15092).

Added Duplex/EAR-TTS speech-generation pieces used by VoiceChat, including MagpieTTS decoder integration and semantic-codec training support (#15277, #15524).

Added the joint NemotronVoiceChat STT + TTS class for validation, offline speech-to-speech inference, and export workflows (#15456).

Added formatter/reproducibility work for the VoiceChat speech decoder, with faster training and half-precision inference support (#15583).

Expanded Voice Agent examples for Nemotron Nano v2/v3, MagpieTTS, tool calling, audio logging, and README guidance (#14325, #14704, #15243, #15269, #15318, #15547, #15642).

Hardened Voice Agent behavior around default Parakeet EOU STT, text aggregation, end-of-bot handling, RTVI messages, empty tools, and logging (#14951, #15068, #15069, #15634).

Note: NemotronVoiceChat class is inference/eval only. Train DuplexSTTModel and DuplexEARTTS/speech-decoder modules separately.

ASR

Unified prompt-model support now covers multilingual ASR and streaming inference (#15666).

Streaming ASR gained Unified RNNT inference, batched streaming beam search, simulated chunked transducer decoding, streaming speech translation, and Canary streaming policies (#15522, #15753, #15517, #15132, #14765).

Transducer decoding is faster and more complete: 2.4x speedup with confidence, buffered/cache-aware confidence, fewer redundant decoding passes, lower streaming memory, and WER fixes (#15315, #15765, #15301, #15148).

Phrase boosting expanded across transducers, cache-aware RNN-T, CTC/RNN-T/TDT GPU phrase boosting, and cache-aware customization (#15125, #15344, #14277, #14757, #14800).

New and updated model families include prompt Parakeet Hybrid RNNT/CTC, ASR EOU models, Streaming Sortformer, Canary2 with NFA, and ASR Transformer/Conformer-Transformer encoders (#14561, #14740, #14627, #14121, #15661, #15703).

Diarization/VAD removed the Pyannote dependency and gained more flexible input handling (#15632, #15184).

TTS

MagpieTTS received a major refresh: decoder model, refactor, longform inference, unified longform/standard paths, and updated model internals (#15277, #15504, #15210, #15241, #15477, #15375, #15031).

Added CFG distillation, local transformer CFG distillation, and MagpieTTS MoE support (#15568, #15748, #15370).

Expanded language/text support: Hindi, Japanese, Arabic char tokenizer, Japanese-English katakana, hi-IN/ko-KR/pt-BR IPA, and Japanese G2P accent support (#15320, #15248, #15614, #15567, #15170).

Improved EasyMagpie zero-shot disabling and speaker-encoder behavior (#15639, #15503, #15564).

Codec and evaluation work added semantic-codec training, codec conversion/bandwidth extension, HF audio-codec loading, Frechet Codec Distance, parallel scoring, comparison reports, and deterministic multi-GPU eval (#15524, #15191, #15172, #15223, #15417, #15621, #15427).

Inference paths now support reference-free inference, dataset selection, clearer inference config, and fp16 Whisper loading (#15213, #15212, #15254, #15680).

Audio

Added and fixed Conformer U-Net speech-enhancement models (#14442, #14626).

Added a data-prediction objective for flow-matching speech enhancement (#14749).

Added streaming mode to SpectrogramToAudio (#14524).

Fixed BNR 2.0 inference alignment with padded input signals (#15388).

Reduced duplication and applied small fixes in the audio collection (#15587).

DataLoader and Speech Data Tools

Lhotse support expanded to Parquet/Arrow embedded audio, AIS batch loading, AIS-hosted input configs, non-tarred S3 audio, temperature re-weighting, randomized tarred shard slicing, and in-manifest channel selection (#15303, #15102, #15538, #14891, #15732, #15200, #14558, #14586).

Audio-codec data loading gained Lhotse training format support plus follow-up fixes (#15622, #15742).

Online data augmentation now covers clipping, lowpass, lossy codec augmentation, and saving augmented audio from Lhotse samplers/dataloaders (#14809, #14808).

Speech Data Explorer gained S3 reading, comparison mode, security fixes, and tutorial updates (#15500, #15137).

Docs

Documentation was rewritten around speech scope, uv installs, ASR/TTS/SpeechLM2 guides, and cleaner navigation.

Repo Rename and 3.0 Release

Project identity is now NVIDIA NeMo Speech in NVIDIA-NeMo/Speech (#15783, #15788).

README/docs now describe the speech-focused repo, current status, and nightly docs (#15127, #15217, #15602, #15766).

Technical Debt Reduction

Removed LLM, VLM, diffusion, NLP, multimodal, ExportDeploy, evaluator, Automodel, and language-modeling paths (#14095, #14192, #14617, #14677, #14790, #14934, #14964, #15033, #15044, #15357, #15378).

Cleaned unsupported ASR/TTS modules, old tutorials, old buffered CTC scripts, and deprecated SpeechLM1 scripts (#14660, #15061, #15368, #15507, #15655).

Removed unsafe pickle paths and hardened target/export handling (#14540, #15065, #15232, #15266, #15288, #15629).

Installation, Packaging, and Containers

Switched to a uv-first source install: uv sync --extra all --extra cu13 (#15769).

Added clear bring-your-own Python/PyTorch/CUDA guidance via uv pip or pip, addressing common install feedback (#15769).

Routed optional SpeechLM2 compiled deps through uv-compatible sources and container builds: Transformer Engine, FlashAttention, Mamba, grouped GEMM, DeepEP (#15737, #15758).

Refreshed Docker around official PyTorch bases, lighter staged builds, CUDA 12/13 support, and CVE dependency updates (#15638, #15747, #15756, #15761).

Updated CI/publish flows to install from uv.lock and simplify release builds (#15659, #15668, #15685, #15697).

Tests and Release Hardening

Added functional init/train-step/inference tests for every supported released model (#15433).

Added golden-value MoE dispatch tests for grouped GEMM/expert routing (#15698).

Split L0 tests into ASR GPU/CPU and SpeechLM2 buckets (#15654).

Hardened streaming ASR, Sortformer, MagpieTTS, codec, and TTS L2/e2e tests (#14416, #14417, #14435, #14823, #15272, #15508, #15584, #15607).

Refactored release CI and moved to AWS ephemeral runners (#15620, #15668, #15718).

Detailed PR Breakdown

Release Identity and Docs

#15788 releases r3.0.0; #15783 bumps the next release to 3.0.

#15127 renames docs/title to NVIDIA NeMo Speech; #15217 and #15602 update repo status/release messaging.

#15363 and #15460 restructure docs; #15542 refactors ASR docs; #15647 refactors diarization docs; #15745 improves docs visuals.

#15769 rewrites install docs for uv, bring-your-own stacks, CUDA extras, and pip fallback.

#15773 documents SpeechLM parallelism; #15738 adds prompt RNNT docs; #15546/#15302 add MagpieTTS docs; #15526 adds streaming ASR inference docs.

#15744 adds community featured models; #15766 updates the release README.

Technical-Debt Cleanup

#15378 removes deprecated collections; #15363 starts docs-side transformation.

#14095, #14677, #14934, #14617, and #14192 remove NeMo1, multimodal/vision, NLP, and language-modeling code.

#15357, #14790, #15033, #15044, and #14964 remove deprecated LLM/VLM/diffusion tutorials, export-deploy, old Automodel, and evaluator code.

#15507 cleans old ASR modules; #15368 removes unsupported TTS models/docs; #14660 removes old TTS tutorials; #15061 removes old buffered CTC; #15655 removes SpeechLM1 scripts.

#15147 updates TRT references; #15146 removes old checkpoint save support; #15211 removes torchaudio usage.

#15687 applies black formatting; #15656 makes formatting CI check-only with pre-commit hooks.

Install, Packaging, Docker, Dependencies

#15769 adds uv-first install plus pip/uv-pip fallback for existing PyTorch/CUDA stacks.

#15734 removes stale uv-base deps; #15758 moves DeepEP/TE git refs to uv sources; #15697 installs CI deps from uv.lock.

#15638 switches Docker to official PyTorch base; #15747 adds CUDA 12 builds; #15761 moves code copy to the final Docker stage.

#15737 builds Automodel compiled deps in CI image; #15756 updates CVE-sensitive deps and Dockerfile.

#15659 unblocks TestPyPI publish; #15668 refactors release workflows; #15685 removes obsolete build workflows.

#15314 sets torch >= 2.6.0; #15353 documents TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD; #15506/#15018 fix CUDA Python/numba-cuda handling.

#15630 removes diskcache; #15502 removes protobuf; #15183 removes CUDA bindings pins; #15438 relaxes kaldialign.

Security and Safety

#15288 fixes command injection and insecure permissions.

#14540 hardens target resolution.

#15065 guards trust_remote_code.

#15266, #15232, and #15629 remove or block unsafe pickle paths.

#15636 adds SECURITY.md.

#15500 adds SDE S3/comparison updates plus a security fix.

Tests, CI, Release Reliability

#15433 adds init/train-step/inference functional tests for released models.

#15698 adds golden MoE dispatch tests.

#15654 splits L0 ASR/SpeechLM2 test buckets; #15651 raises GPU speech test timeouts.

#14823, #14953, and #14184 harden streaming/cache-aware transducer and CUDA-graph tests.

#14416, #14417, and #14435 complete Streaming Sortformer fixes/tests.

#15508, #15584, #15272, and #15607 harden TTS/MagpieTTS/codec tests.

#15620, #15718, #15484, #15519, and #15501 improve CI runners, branch rules, triggering, and e2e layout.

#15613 adds local dev/PR babysitting support; #15612 adds distributed-log debug support.

SpeechLM2, SALMAutomodel, Scaling, vLLM

#15447 adds SALM + NeMo Automodel for Nemotron Nano V3, including native LoRA and Automodel-managed distributed setup.

#15648 expands SALMAutomodel with long-context support, encoder chunking, activation-checkpointing controls, and stability fixes.

#15679 adds THD packed sequences, context parallelism, CP-safe long-audio training, and related packed-input handling.

#15773 documents FSDP2, HSDP, TP, PP, CP, and EP strategy usage for SpeechLM training.

#15737 and #15758 wire compiled Automodel deps, DeepEP, and Transformer Engine into reproducible uv/container builds.

#15520 adds vLLM support for NeMo SpeechLM, including a registered SpeechLM model/config path.

#15623 exports vLLM-ready SpeechLM checkpoints with backbone-native chat_template support.

#15716 adds encoder input chunking for SALM vLLM inference.

#15736 adds offline HF export for SpeechLM2 checkpoints.

#15686 preserves perception checkpoint dtype; #15570 fixes Qwen3 SALM LoRA initialization.

#15316 adds TDT decoder input, ShareGPT format support, and related SALM improvements.

#15410 adds indexed ShareGPT JSONL and WebDataset formats.

#15364 adds buffered SALM inference; #15478 updates SpeechLM2 docs and dataloader resampling; #15281 adds multimodal offset-key support.

Nemotron VoiceChat, Duplex Models, Voice Agent

#15066 implements the Nemotron VoiceChat speech decoder for the response-generation side of duplex speech-to-speech.

#15092 implements DuplexSTT for speech/text-conditioned agent text generation.

#15456 adds the joint Nemotron VoiceChat STT + TTS class for validation, offline inference, and export.

#15583 improves VoiceChat decoder reproducibility, training speed, formatting, and fp16 inference.

#15277 adds the MagpieTTS decoder model used by the speech-generation stack.

#15524 adds semantic-codec training support.

#14325 adds NeMo Voice Agent as the app/integration layer.

#14704 adds Nemotron Nano v2 support; #15318 adds Nemotron Nano v3 and MagpieTTS improvements.

#15069 sets Parakeet EOU as default STT for the voice agent.

#14951 fixes text aggregation, end-of-bot handling, and logging; #15068 fixes missing RTVI bot messages.

#15243 and #15269 improve tool-calling examples, tool-calling behavior, and logging UX.

#15634 fixes empty tools; #15642 updates Voice Agent docs.

ASR, Streaming, Prompt Models, Transducers

#15666 adds unified prompt-model support for multilingual ASR and streaming; #14561 adds Parakeet hybrid prompt support; #15738 adds docs.

#15522 adds Unified RNNT streaming inference.

#15753 adds batched streaming beam search; #15315 speeds up transducer decoding 2.4x with confidence; #15765 adds buffered/cache-aware confidence.

#15517 adds simulated chunked transducer decoding; #15301 removes double decoding; #15148 fixes streaming transducer memory/WER.

#15125, #15344, #14277, #14800, and #14757 expand phrase boosting and cache-aware customization.

#15132 adds streaming speech translation; #14765 adds Canary Wait-K/AlignAtt; #14766 adds streaming timestamps.

#15268 adds Canary timestamp-model restore; #15291 fixes tensor-input timestamps.

#14740, #15462, #15497, and #15493 add EOU models/metrics and fixes.

#14627, #14388, #14416, #14417, and #14435 cover Streaming Sortformer train/inference/docs/tests.

#14905, #14932, and #15025 update Multi-Talker Parakeet streaming docs/notebooks; #14121 adds Canary2 with NFA; #15317 improves short-audio Canary.

#15632 removes Pyannote from diarization/VAD; #15184 improves diarization inputs; #15573 aligns cpWER with meeteval.

#15605 removes tail-margin algorithms; #15591 removes PnC flag; #15576 fixes ASR inference pipeline bugs.

#15661 adds ASR Transformer encoder; #15703 adds Conformer I/O-styled Transformer encoder.

TTS, MagpieTTS, Codecs, Text

#15277 adds MagpieTTS decoder; #15504 refactors MagpieTTS; #15031 updates the model.

#15210, #15241, #15477, and #15375 add/unify longform MagpieTTS inference.

#15568 and #15748 add CFG distillation; #15370 adds MagpieTTS MoE.

#15320, #15248, #15567, #15614, and #15170 add Hindi/Japanese/IPA/Arabic/Katakana/G2P support.

#15639, #15503, and #15564 adjust EasyMagpie zero-shot disabling.

#15191, #15172, #15622, and #15742 improve codec conversion, bandwidth extension, HF codec loading, and Lhotse codec data.

#15524 adds semantic-codec training; #15223 adds Frechet Codec Distance; #15417 adds parallel scoring; #15621 adds TTS comparison reports.

#15348 and #15427 improve multi-validation and deterministic multi-GPU eval.

#15213, #15212, #15254, and #15680 improve reference-free inference, dataset selection, inference config, and fp16 Whisper loading.

Audio

#14442 adds a Conformer U-Net model for speech enhancement.

#14626 fixes the Conformer U-Net implementation.

#14749 adds a data-prediction objective for flow-matching speech-enhancement models.

#14524 adds streaming mode to SpectrogramToAudio.

#15388 fixes BNR 2.0 inference alignment with padded input signals.

#15587 reduces duplication in the audio collection and applies small fixes.

DataLoader and Speech Data Tools

#15303 adds Parquet/Arrow embedded-audio support via Lhotse.

#15102 and #15538 add AIS batch loading for ASR audio processing and LhotseSpeechToTextBpeDataset.

#14891 supports input_cfg.yaml directly from AIS buckets.

#15732 fixes non-tarred S3 audio in LazyNeMoIterator.

#15200 adds on-the-fly temperature re-weighting for Lhotse datasets.

#14558 adds randomized shard slicing for tarred data.

#14586 adds in-manifest channel selection for multichannel recordings through Lhotse.

#15316 and #15410 add ShareGPT, indexed ShareGPT JSONL, and WebDataset data formats for SpeechLM2.

#14294 adds ShareGPT data and a test loader.

#15622 adds audio-codec Lhotse training format support; #15742 updates and fixes audio-codec Lhotse loading.

#14809 adds clipping, lowpass, and lossy-codec online augmentations; #14808 adds augmented-audio saving.

#15500 updates Speech Data Explorer with S3 read support, comparison mode, and security fixes; #15137 updates the SDE tutorial.

Smaller Fixes

#15437 adds .nemo torchaudio-preprocessor checkpoint migration.

#15461 auto-detects use_bucketing and validates batch size.

#15325 fixes TDT fusion scores; #15322 moves transducer fusion models to the base class.

#15245 speeds .nemo tar extraction/config processing.

#15409 handles an exp-manager timer race.

#15238 uses PurePosixPath.

#15444 replaces assert with ValueError in EncDecMultiTaskModel.

#15688 fixes validation audio logging; #15699 adds UTMOSv2Calculator.verbose.

2.7.026 Feb 2026Deprecated

Add deprecation notice to modules by @chtruong814 :: PR: #15050

Highlights

Speech
- Adds Per-Stream Phrase Boosting in ASR Decoding (Transducers) #15125
- Adds support for streaming speech translation #15132
- Released new model nemotron-speech-streaming-en-0.6b that performs English Streaming ASR
- Released new TTS model magpie_tts_multilingual_357m for multilingual Text-to-Speech

Starting with the next release, NeMo 2.8.0, the following collections will be removed: avlm, diffusion, llm, multimodal, multimodal-autoregressive, nlp, speechlm, vision, vlm, and this repo will focus solely on speech tasks: ASR, TTS, speaker diarization, and speech enhancement.

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

Enable CUDA graphs in streaming tests by @artbataev :: PR: #14953
Update ctc-segmentation by @chtruong814 :: PR: #14991
check asr models by @nithinraok :: PR: #14989
Unified inference of streaming ASR by @naymaraq :: PR: #14817
Update numba to numba-cuda and update cuda python bindings usage by @chtruong814 :: PR: #15018
Fixing lines for multispeaker pipeline by @tango4j :: PR: #15030
Inference optimization for cache-aware pipelines by @naymaraq :: PR: #15035
fix loading of hyb ctc rnnt bpe models when using from pretrained by @nithinraok :: PR: #15042
removed old buffered CTC script by @naymaraq :: PR: #15061
remove nlp related notebooks by @nithinraok :: PR: #15070
Update MagpieTTS model with latest changes by @blisc :: PR: #15031
ASR inference: expose RNN-T decoding params for context biasing by @artbataev :: PR: #15091
update notebook by @nithinraok :: PR: #15093
Fix: Obsolete Attribute [SDE] by @Jorjeous :: PR: #15105
Upgrade NeMo ASR tutorials from Mozilla/CommonVoice to Google/FLEURS by @KunalDhawan :: PR: #15103
Add support for AIS batch loading for ASR audio processing by @gaikwadabhishek :: PR: #15102
Multi-Talker Parakeet Streaming - NeMo Documents and Tutorial Notebooks PR 03 by @tango4j :: PR: #15025
[Fix] Fix the notebook errors on multispeaker data simulation and end to end diarization training by @tango4j :: PR: #15149
Streaming transducer inference: fix memory usage, improve WER by @artbataev :: PR: #15148
Execute with subprocess list by @nithinraok :: PR: #15165
Chunking fix by @nune-tadevosyan :: PR: #15163
ASR Decoding: allow fallback to CUDA graphs without while loops by @artbataev :: PR: #15173
remove nlp/modules by @dimapihtar :: PR: #14934
Asr numpy 2 fix by @nithinraok :: PR: #15166
Adding flexible input sources for Diarization Mixin by @tango4j :: PR: #15184
Add support for streaming speech translation by @naymaraq :: PR: #15132
Confidence fix get_correct_marks by @nune-tadevosyan :: PR: #15128
Chunking edge cases by @nune-tadevosyan :: PR: #15182
update subprocess cmd by @nithinraok :: PR: #15218
Changes required for enabling prompt based models in Nemo Inference by @arushidNV :: PR: #15036
Fixing the missing sample_rate argument in mixin calling in Sortformer model file by @tango4j :: PR: #15228
Fix audio tensor loading canary2 by @nithinraok :: PR: #15265
Fix word confidence return by @nithinraok :: PR: #15249
feat(asr): add optional auxiliary timestamp model restoration for Canary by @chaosido :: PR: #15268
Performance: Optimize .nemo tar extraction & model config processing by @paulirish :: PR: #15245
fix speech commands notebook by @nithinraok :: PR: #15290
fix timestamps processing with audio tensor input by @nithinraok :: PR: #15291
Update conv_asr.py preventing unnecessary calculations by @tamilselvan0x0 :: PR: #15239
Bump to pytorch 25.11 by @chtruong814 :: PR: #15247
Add FeatureBuffer support to Cache-Aware streaming pipeline by @arushidNV :: PR: #15188
Per-Stream Phrase Boosting in ASR Decoding (Transducers) by @artbataev :: PR: #15125
Sort audio by duration in ASR streaming inference script by @artbataev :: PR: #15297
ASR transcribe: fix forced decoder reinstantiation with timestamps=True by @artbataev :: PR: #15298
Removes use of torchaudio and moves transforms inside of NeMo by @blisc :: PR: #15211
Add sacrebleu to ASR requirements by @pzelasko :: PR: #15016
SpeechLM2 : Add support for offset key in Multimodal conversation by @AudranBert :: PR: #15281
Add cross-attention to output hypotheses by @mgaido91 :: PR: #15229
Add warm-ups for RTFX calculation in streaming ASR pipelines by @naymaraq :: PR: #15313
Speedup buffered transducer inference: remove double decoding by @artbataev :: PR: #15301
improve canary performance on short audio by @nithinraok :: PR: #15317
Transducer Decoding: Move fusion models to the base class by @artbataev :: PR: #15322
Add typing to speech_to_text_finetune.py by @Garvys :: PR: #15326
Bugfix: correct fusion scores for TDT by @artbataev :: PR: #15325
Fix ASR streaming script: correctly add biasing requests to model by @artbataev :: PR: #15334
Fix ASR context biasing in streaming TDT decoding by @artbataev :: PR: #15327

</details>

TTS

<details><summary>Changelog</summary>

Remove HeteronymClassificationModel by @blisc :: PR: #14980
remove nlp.parts collection by @dimapihtar :: PR: #14617
Update MagpieTTS model with latest changes by @blisc :: PR: #15031
remove nlp/modules by @dimapihtar :: PR: #14934
[TTS] MagpieTTS Inference Refactoring by @subhankar-ghosh :: PR: #15178
[DRAFT][TTS] Magpietts Simple API and loading audiocodec from Huggingface by @subhankar-ghosh :: PR: #15172
[TTS][MagpieTTS] Change French tokenizer to use 'french_chartokenizer' by @subhankar-ghosh :: PR: #15205
Add Japanese g2p katakana accent support by @quapham :: PR: #15170
[TTS][MagpieTTS] Longform TTS using MagpieTTS by @subhankar-ghosh :: PR: #15210
[voice agent] Fixing the missing arguments calling in NemoSTTService by @SangwonSUH :: PR: #15233
[TTS] MagpieTTS inference: Add command line option to select a subset of datasets to run inference on by @rfejgin :: PR: #15212
[TTS] Allow inference without reference audio by @rfejgin :: PR: #15213
[TTS] Refactor Magpie to support codec conversion and bandwidth extension by @rlangman :: PR: #15191
[TTS] MagpieTTS: Implement Frechet Codec Distance metric + some minor inference bugfixes by @rfejgin :: PR: #15223
Update MagpieTTS' Inference Parameter Configuration by @blisc :: PR: #15254
[TTS][MagpieTTS] Add longform capability to do_tts method by @subhankar-ghosh :: PR: #15241
[TTS] Add tests of the MagpieTTS inference CLI by @rfejgin :: PR: #15272
[MagpieTTS][TTS] Support local transformer in longform magpietts by @subhankar-ghosh :: PR: #15296
Removes use of torchaudio and moves transforms inside of NeMo by @blisc :: PR: #15211
[MagpieTTS][Docs] Add magpietts docs by @subhankar-ghosh :: PR: #15302
Add Hindi (hi-IN) support for TTS by @quapham :: PR: #15248
build: Explicitly set torch >= 2.6.0 and remove weights_only=False by @chtruong814 :: PR: #15314
[MagpieTTS] Fix incorrect sort order comment in pareto_rank function by @matteolippi :: PR: #15333

</details>

NLP / NMT

<details><summary>Changelog</summary>

remove nlp.parts collection by @dimapihtar :: PR: #14617
chore: remove ExportDeploy by @pablo-garay :: PR: #15033
remove nlp related notebooks by @nithinraok :: PR: #15070
Add deprecation notice to modules by @chtruong814 :: PR: #15050
[OMNIML-3034] ModelOpt rename from TRT ModelOpt to ModelOpt by @yueshen2016 :: PR: #15147
remove nlp/modules by @dimapihtar :: PR: #14934
Add support for streaming speech translation by @naymaraq :: PR: #15132
Remove hardcoded DEBUG logging level in gpt_oss.py by @yurekami :: PR: #15236
Docs: replace removed preprocess_data_for_megatron.py with Megatron-L… by @Saibabu7770 :: PR: #15222
remove nlp documentation by @dimapihtar :: PR: #15304
fix speech translation vllm dockerfile by @naymaraq :: PR: #15310

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Add import guards for mcore lightning module by @chtruong814 :: PR: #14970
chore: update Lightning requirements version by @liquor233 :: PR: #15004

</details>

NeMo Tools

<details><summary>Changelog</summary>

Fix: Obsolete Attribute [SDE] by @Jorjeous :: PR: #15105
Updated tutorial on SDE, due to changes in colab and libraries by @Jorjeous :: PR: #15137

</details>

Export

<details><summary>Changelog</summary>

chore: remove ExportDeploy by @pablo-garay :: PR: #15033
[OMNIML-3034] ModelOpt rename from TRT ModelOpt to ModelOpt by @yueshen2016 :: PR: #15147
fix: Raise exception in nemo.export instead of allowing pickle.loads by @chtruong814 :: PR: #15266

</details>

Bugfixes

<details><summary>Changelog</summary>

Fix PEFT resume with resume_from_path by @maanug-nv :: PR: #14966
Update deprecated env var by @maanug-nv :: PR: #14975
Revert lhotse patch after updating to lhotse 1.32.2 by @chtruong814 :: PR: #15329

</details>

Uncategorized:

<details><summary>Changelog</summary>

Version bump to 2.7.0rc0.dev0 by @github-actions[bot] :: PR: #14956
Update changelog for v2.5.1 by @github-actions[bot] :: PR: #14967
Bump MCore, TE, Pytorch, and modelopt for 25.11 by @chtruong814 :: PR: #14946
Remove code related to nemo-evaluator (aka nemo-eval) by @athitten :: PR: #14964
Update changelog for r2.5.0 by @github-actions[bot] :: PR: #14990
Add clear resharding message error message by @mikolajblaz :: PR: #14962
Fix Evo2 checkpoint backward compatibility by @farhadrgh :: PR: #14914
Pass timeout when running speech functional tests by @chtruong814 :: PR: #15012
[Voice Agent] Fix text aggregation, eob handling, logging by @stevehuang52 :: PR: #14951
Fix speechlm inference configuration by @stevehuang52 :: PR: #14931
Enable EP in PTQ by @jenchen13 :: PR: #15015
revert ckpt scripts removal from #14617 by @dimapihtar :: PR: #15048
fix: fix update-buildcache workflow after ED remove by @pablo-garay :: PR: #15051
Update changelog for v2.5.3 by @github-actions[bot] :: PR: #15055
[voice agent] Fix RTVI missing bot message by @stevehuang52 :: PR: #15068
[voice agent] make parakeet-eou model default stt by @stevehuang52 :: PR: #15069
chore: Remove Automodel module by @thomasdhc :: PR: #15044
add support for parallel ckpt removal by @dimapihtar :: PR: #15073
Fix VLM mcore engine by @meatybobby :: PR: #15076
Revert "Fix vlm engine changes in mcore (#15076)" by @pablo-garay :: PR: #15090
fix: fix lines with malformed anchor tags by @pablo-garay :: PR: #15095
Update Gemma3VL model training scripts by @genquan9 :: PR: #15041
fix MR layer b2b filter to be comptatible with baseline FFTConv by @moradza :: PR: #15100
guard trust_remote_code by @dimapihtar :: PR: #15065
Fix get_new_ctm_lines_from_alignments function in scripts/speaker_tasks/create_alignment_manifest.py by @KunalDhawan :: PR: #15118
Change title to 'NVIDIA NeMo Speech Collection' by @snowmanwwg :: PR: #15127
remove pinning cuda bindings by @nithinraok :: PR: #15183
Update URL to ModelOpt Speculative by @AAnoosheh :: PR: #15075
remove ckpt_save_pre_mcore_014 support by @dimapihtar :: PR: #15146
Removing pip install instruction for NeMo voice agent environment setting by @tango4j :: PR: #15101
replace deprecated type comment with type annotation. by @XuesongYang :: PR: #15175
Set L2_NeMo_2_llama3_pretraining_recipe to be optional by @chtruong814 :: PR: #15192
Update README with latest news on nano-v3 by @snowmanwwg :: PR: #15197
[speechlm] replace pcikle.loads with json.loads by @stevehuang52 :: PR: #15232
[Fix] Fix safety issue for fsdp_dtensor by @BoxiangW :: PR: #15227
[voice agent] Add examples for tool calling by @stevehuang52 :: PR: #15243
Update README to reflect current Repo Status by @nithinraok :: PR: #15217
remove checks for hydra installation by @nithinraok :: PR: #15267
[voice agent] Improve tool calling and logging ux by @stevehuang52 :: PR: #15269
Implement Nemotron-VoiceChat Speech Decoder by @Edresson :: PR: #15066
Update CONTRIBUTING.md by @chtruong814 :: PR: #15260
Update changelog for r2.6.0 by @github-actions[bot] :: PR: #15282
set dynamo=False to support latest version of pytorch by @nithinraok :: PR: #15284
Fix progress_printer using wrong variable in on_test_batch_end by @yurekami :: PR: #15237
[voice agent] Add audio logging to NeMo Voice Agent by @tango4j :: PR: #15279
bump transformers version by @nithinraok :: PR: #15271
Use PurePosixPath for cross-platform path handling by @yurekami :: PR: #15238
Fix website link for clearml by @orena1 :: PR: #14128
Clipping, lowpass and lossy codec online augmentations for Lhotse dataloader/sampler by @racoiaws :: PR: #14809
ci: Enable label to force run CI tests by @chtruong814 :: PR: #15242
Remove Codeowners for now by @blisc :: PR: #15307
unset weights_only=False by @dimapihtar :: PR: #15312
update readme by @nithinraok :: PR: #15341
add safe globals to documentation by @dimapihtar :: PR: #15342
Security: Fix command injection and insecure permissions by @AkCodes23 :: PR: #15288
Freeze tags in in r2.7.0 by @github-actions[bot] :: PR: #15351
Update Imports in Audio Notebook (15345) into r2.7.0 by @blisc :: PR: #15352
cp: Clarify when to use TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD (15353) into r2.7.0 by @chtruong814 :: PR: #15359
cp: Fix macro accuracy when changing labels (15379) into r2.7.0 by @chtruong814 :: PR: #15386
cp: [voice agent] fix dependency for nemo26.02 (15380) into r2.7.0 by @chtruong814 :: PR: #15383
cp: Remove deprecated LLM, VLM, and diffusion tutorials (15357) into r2.7.0 by @chtruong814 :: PR: #15392
update yaml to include new location of losses that were removed in #15211 (r.2.7.0 fix) by @blisc :: PR: #15391
cp: fixes nemo tutorial for loading non registered classes (15398) into r2.7.0 by @chtruong814 :: PR: #15399
cp: default weights to false (15397) into r2.7.0 by @chtruong814 :: PR: #15401
cp: Fixing could not find ctc_segmentation. in CTC tutorial (15403) into r2.7.0 by @chtruong814 :: PR: #15404
cp: Adapt to use env variable for adapter mixin model loading (15406) into r2.7.0 by @chtruong814 :: PR: #15407
Fix BNR 2.0 inference alignment error with input signal padding by @ManasiRemane :: PR: #15390
chore: Remove pre-release suffix for 2.7.0 by @chtruong814 :: PR: #15415
cp: Update release workflow to include generated changelog (#15429) by @chtruong814 :: PR: #15430

</details>

Open source →

2.6.03 Dec 2025Deprecated

Non-Speech NeMo 2.0 collections are deprecated and will be removed in a later release. Their functionality is available in the Megatron Bridge repo at…

Highlights

Speech
- Add Timestamps to streaming ASR PR
- Add Streaming decoding policies (Wait-K and AlignAtt) for Canary model PR
- Add NeMo Voice Agent PR
- Hybrid RNNT-CTC Prompted Parakeet Model support PR
- [New] MT-Parakeet Streaming Models release
Removed the Automodel module. Automodel is available in the repo https://github.com/NVIDIA-NeMo/Automodel.
Removed the Deploy module. Export & Deploy is available in the repo https://github.com/NVIDIA-NeMo/Export-Deploy.
Non-Speech NeMo 2.0 collections are deprecated and will be removed in a later release. Their functionality is available in the Megatron Bridge repo at https://github.com/NVIDIA-NeMo/Megatron-Bridge.

Known Issues

NeMo voice agent pipecat connecting issues

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

fixing kernel restarting when transcribing by @weiqingw4ng :: PR: #14665
Downgrade "datasets" library version in ASR tutorial to ensure compatibility with HF Datasets used by @KunalDhawan :: PR: #14679
Fixing Sortformer training tutorial notebook by @tango4j :: PR: #14680
Fix for "EncDecRNNTBPEModel transcribe() failed with TypeError" by @andrusenkoau :: PR: #14698
Force activations and weights cast to FP32 Jasper Encoder Squeeze-Excite (merge to main) by @erastorgueva-nv :: PR: #14743
Use lhotse dataloader for ASR models to support in-manifest channel selection for multichannel recordings by @racoiaws :: PR: #14586
add transducer timestamps without alignments, timestamps to streaming by @lilithgrigoryan :: PR: #14766
Adding bf16 Sortformer train and inference by @tango4j :: PR: #14627
Replace texterrors with kaldialign library by @andrusenkoau :: PR: #14775
fix: Use shutil.copy fallback to handle file metadata permission errors by @vipnydav :: PR: #14639
Add Customization Capabilities to Cache-Aware Models by @artbataev :: PR: #14757
Documentation for gpu-based phrase boosting by @andrusenkoau :: PR: #14800
Streaming decoding policies (Wait-K and AlignAtt) for Canary model by @andrusenkoau :: PR: #14765
Add tests for streaming buffered and cache-aware transducer models by @artbataev :: PR: #14823
Merge updates of Multi-Talker Parakeet Model, Modules, Dataloader and Utils PR 01 by @weiqingw4ng :: PR: #14905
Merge updates of Multi-Talker Parakeet - Unit tests and CI tests PR 02 by @weiqingw4ng :: PR: #14932
Add Parakeet Hybrid RNNT CTC BPE Model with Prompt support by @ealbasiri :: PR: #14561
fix notebooks by @nithinraok :: PR: #15079
cherry pick #15070 by @nithinraok :: PR: #15082

</details>

TTS

<details><summary>Changelog</summary>

Remove outdated TTS Tutorials by @blisc :: PR: #14660
Add KokoroTTS support for voice agent framework by @tango4j :: PR: #14910
remove language_modeling by @dimapihtar :: PR: #14192

</details>

NLP / NMT

<details><summary>Changelog</summary>

Add gpt-oss by @cuichenx :: PR: #14457
Fix sequence packing loss calculation by @rayandasoriya :: PR: #14437
[Perf script] Llama and GPT3 perf script use mlp cast fusion by @guyueh1 :: PR: #14575
Delete tutorials/llm/llama/biomedical-qa directory by @cuichenx :: PR: #14653
Add gpt-oss lora exporter by @cuichenx :: PR: #14589
Replace MegatronTokenizer with MegatronLegacyTokenizer by @chtruong814 :: PR: #14721
Update ModelCommPGs API from megatron-core by @yaoyu-33 :: PR: #14578
feat: Compatibility modification of megatron-fsdp by @shjwudp :: PR: #14593
imported get_moe_layer_wise_logging_tracker from megatron core moe_utils by @prathamk-tw :: PR: #14694
Fix gpt-oss yarn_original_max_position_embeddings value by @cuichenx :: PR: #14706
Update docs per guidance by @pablo-garay :: PR: #14841
Fixing three mcore links by @aschilling-nv :: PR: #14839
Documentation for gpu-based phrase boosting by @andrusenkoau :: PR: #14800
Update gpt-oss configs by @cuichenx :: PR: #14674
remove language_modeling by @dimapihtar :: PR: #14192
cp: remove ExportDeploy into r2.6.0 by @pablo-garay :: PR: #15053
cherry pick #15070 by @nithinraok :: PR: #15082

</details>

Export

<details><summary>Changelog</summary>

fix: fix missing rope scaling in exporting llama embedding model by @ZhiyuLi-Nvidia :: PR: #14523
Add gpt-oss lora exporter by @cuichenx :: PR: #14589
Skip trt-llm and vllm install in install test by @chtruong814 :: PR: #14663
Fix deepseek export dtype by @cuichenx :: PR: #14307
Remove export-deploy, automodel, and eval tutorials by @chtruong814 :: PR: #14790
cp: remove ExportDeploy into r2.6.0 by @pablo-garay :: PR: #15053

</details>

Uncategorized:

<details><summary>Changelog</summary>

Version bump to 2.6.0rc0.dev0 by @github-actions[bot] :: PR: #14512
[Audio]: added conformer U-Net model for SE by @nasretdinovr :: PR: #14442
hyena/evo2: Make sure to convert to real after fp32 conversion by @antonvnv :: PR: #14515
Force-set restore path for student in KD mode by @AAnoosheh :: PR: #14532
Skip PTQ if PTQ model path exists by @jenchen13 :: PR: #14536
Support QwenVL for inference API by @meatybobby :: PR: #14534
Hyena: Allow to use unfused RMSNorm + TELinear to restore accuracy and some speed by @antonvnv :: PR: #14542
[Audio]: added streaming mode to SpectrogramToAudio by @nasretdinovr :: PR: #14524
Update evo2 defaults so converted checkpoints have the right parameters by @jstjohn :: PR: #14514
deprecate t0 scripts by @dimapihtar :: PR: #14585
cfg typo correction by @malay-nagda :: PR: #14588
[Perf script] Add use_te_activation_func and activation_func_fp8_input_store flags by @guyueh1 :: PR: #14522
Modify logging message to signal that RestoreConfig will be used by @balvisio :: PR: #14469
Bump TE and Mcore by @chtruong814 :: PR: #14568
Avoid host-device sync in PTL logging by @WanZzzzzz :: PR: #14489
Integrate implicit filter kernel with Hyena layer by @farhadrgh :: PR: #14621
Fix kv_channels configuration for Gemma2 27b by @ananthsub :: PR: #14590
[Flux] small fixes by @CarlosGomes98 :: PR: #14333
[Flux] Add MXFP8 Support by @alpha0422 :: PR: #14473
Use hugginface_hub for downloading the FLUX checkpoint by @suiyoubi :: PR: #14638
Fine-tune embedding models (E5-Large-V2 and LLaMA-3.2-1B) on the allnli triplet dataset with NeMo Framework by @girihemant19 :: PR: #14584
remove service launch scripts by @dimapihtar :: PR: #14647
Warn instead of error when chat template doesn't contain generation keyword by @jenchen13 :: PR: #14641
Fix function calling notebook by @cuichenx :: PR: #14643
[Audio]: fixed bug in conformer unet by @nasretdinovr :: PR: #14626
Fix code checkout during test by @chtruong814 :: PR: #14658
Fix Flux seed as optional Arg by @suiyoubi :: PR: #14652
Remove PEFT scheme condition from recipe by @JRD971000 :: PR: #14661
Add NeMo Voice Agent by @stevehuang52 :: PR: #14325
Update get_tensor_shapes function whose signature was refactored by @AAnoosheh :: PR: #14594
Delete nemo1 notebooks by @cuichenx :: PR: #14677
Bump latest Mcore 020abf01 by @chtruong814 :: PR: #14676
[Flux] correct vae_downscale_factor by @CarlosGomes98 :: PR: #14425
Bump modelopt to 0.35.0 and remove safe_import("modelopt") in llm collection by @kevalmorabia97 :: PR: #14656
Canary tutorial fix by @nune-tadevosyan :: PR: #14699
Add option for LoRA with Transformer Engine op fuser by @timmoon10 :: PR: #14411
add load-in-4bit param by @dimapihtar :: PR: #14636
Support NVFP4 recipe by @WanZzzzzz :: PR: #14625
Fix broken link in Reasoning-SFT.ipynb by @cuichenx :: PR: #14716
Remove artificial block to vortex fp8 TP by @jstjohn :: PR: #14684
Drop speech_llm example suite by @yaoyu-33 :: PR: #14683
remove env var by @malay-nagda :: PR: #14739
detach arg option for run scripts by @malay-nagda :: PR: #14722
Randomized shard slicing for tarred data by @pzelasko :: PR: #14558
Data prediction objective for flow matching speech enhancement models by @racoiaws :: PR: #14749
Fix Some Failures by @alpha0422 :: PR: #14763
Support additional Slurm parameters (#14701) by @bdubauski :: PR: #14742
[Flux] Remove Redundant Host & Device Sync by @alpha0422 :: PR: #14711
[Flux] Full Iteration CUDA Graph by @alpha0422 :: PR: #14744
Update prune-distill notebooks to Qwen3 + simplify + mmlu eval by @kevalmorabia97 :: PR: #14785
ci: Automodel deprecation warning by @thomasdhc :: PR: #14787
Bug in MXFP8 recipe by @adityavavreNVDA :: PR: #14793
feat: Disable blank Issues by @pablo-garay :: PR: #14788
ci: Add community label bot by @chtruong814 :: PR: #14796
Add mistral small3 24B config and recipe by @eagle705 :: PR: #14784
Update changelog for r2.3.0 by @github-actions[bot] :: PR: #14812
QWEN2.5-VL 7B FP8 Recipe by @tomlifu :: PR: #14801
Feat: Disk space management: for nemo install test by @pablo-garay :: PR: #14822
Evo2 address rare over-masking in 1m context dataset by @jstjohn :: PR: #14821
Update cherry-pick workflow to use version 0.63.0 by @pablo-garay :: PR: #14832
Removing automodel items by @aschilling-nv :: PR: #14840
Update changelog for v2.4.1 by @github-actions[bot] :: PR: #14828
Fix lm_eval installation in pruning tutorial for 25.09 container by @kevalmorabia97 :: PR: #14865
Add nemotron-nano-v2 support to voice agent by @stevehuang52 :: PR: #14704
Update changelog for 2.5.0 by @chtruong814 :: PR: #14890
[Qwen3] Fix the flop cal for Qwen3 by @gdengk :: PR: #14897
[lhotse][aistore] added support input_cfg.yaml directly from aistore bucket by @XuesongYang :: PR: #14891
Harden _is_target_allowed by adding runtime class validation on top of prefix checks to prevent unsafe target resolution by @KunalDhawan :: PR: #14540
Enable simplified DistOpt checkpoint formats by @mikolajblaz :: PR: #14428
Fix the load checkpointing issue -- onelogger callback gets called multiple time in some case. by @liquor233 :: PR: #14945
Revert "new changelog-build" by @pablo-garay :: PR: #14949
feat: new changelog-build by @pablo-garay :: PR: #14950
Update llama4 utils kwargs by @yaoyu-33 :: PR: #14924
Update README.md by @snowmanwwg :: PR: #14917
Update all outdated NeMo Curator links by @sarahyurick :: PR: #14760
Freeze tags in in r2.6.0 by @github-actions[bot] :: PR: #14957
cp: Bump MCore, TE, Pytorch, and modelopt for 25.11 (14946) into r2.6.0 by @chtruong814 :: PR: #14976
cp: Update ctc-segmentation (14991) into r2.6.0 by @chtruong814 :: PR: #14998
cherry-pick of #14962 by @dimapihtar :: PR: #15000
cp: Pass timeout when running speech functional tests (15012) into r2.6.0 by @chtruong814 :: PR: #15013
cp: check asr models (14989) into r2.6.0 by @chtruong814 :: PR: #15002
cp: Enable EP in PTQ (15015) into r2.6.0 by @chtruong814 :: PR: #15026
cp: Update numba to numba-cuda and update cuda python bindings usage (15018) into r2.6.0 by @chtruong814 :: PR: #15024
cp: Add import guards for mcore lightning module (14970) into r2.6.0 by @chtruong814 :: PR: #14981
cp: fix loading of hyb ctc rnnt bpe models when using from pretrained (15042) into r2.6.0 by @chtruong814 :: PR: #15045
cp: fix: fix update-buildcache workflow after ED remove (15051) into r2.6.0 by @chtruong814 :: PR: #15052
cp: chore: update Lightning requirements version (15004) into r2.6.0 by @chtruong814 :: PR: #15049
cp: update notebook (15093) into r2.6.0 by @chtruong814 :: PR: #15094
cp: Fix: Obsolete Attribute [SDE] (15105) into r2.6.0 by @chtruong814 :: PR: #15106
cp: Upgrade NeMo ASR tutorials from Mozilla/CommonVoice to Google/FLEURS (15103) into r2.6.0 by @chtruong814 :: PR: #15107
cp: chore: Remove Automodel module (15044) into r2.6.0 by @chtruong814 :: PR: #15084
cp: Add deprecation notice to modules (15050) into r2.6.0 by @chtruong814 :: PR: #15110

</details>

Open source →

2.5.07 Oct 2025Deprecated

Deprecate Confidence Ensemble models

Highlights

Collections:
- LLM
  - Nano v2 12B and 9B
- Speech
  - New SpeechLM2 collection
  - Streaming Softformer model
  - Deprecate Confidence Ensemble models
  - parakeet-tdt-0.6b-v3 and canary-1b-v2 models
  - Added chunk inference support with .transcribe() for canary based models
  - Enable prediction of timestamps with streaming ASR
  - Improve ASR models’ invariance to padding/batch size
  - Qwen prompt format support, SALM generation fixes
  - High-level SALM model.generate API closely resembling HF models
  - SALM model initialization with time/memory optimization
  - SpeechLM2: fixed excessive padding, support on-the-fly resampling for SALM
Automodel and Export-Deploy functionality are available in their individual repositories respectively and deprecated in NeMo2

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

Modernize logger interface by @emmanuel-ferdman :: PR: #13783
Higher-level API for SALM.generate by @pzelasko :: PR: #14034
add/refactor docs for asr lm customization by @lilithgrigoryan :: PR: #14088
Improve NEST GPU Utilization 1/N by @MahmoudAshraf97 :: PR: #14086
Improve ASR models' invariance to padding/batch size by @pzelasko :: PR: #13827
Clean up transducer decoding initialization by @artbataev :: PR: #14112
Improve NEST GPU Utilization 2/N by @MahmoudAshraf97 :: PR: #14089
GPU-accelerated Phrase-Boosting (GPU-PB) for AED decoding by @andrusenkoau :: PR: #14108
Fix decoding with ngpu-lm when training (#13994) by @hoangtran9122 :: PR: #13995
fix eval_beamsearch_ngram_ctc script by @lilithgrigoryan :: PR: #14238
fix wrong typing for ctc-ws context graph by @andrusenkoau :: PR: #14262
fix frame vad by @stevehuang52 :: PR: #14337
Improve NEST GPU Utilization 3/N by @MahmoudAshraf97 :: PR: #14234
remove confidence ensemble models by @lilithgrigoryan :: PR: #14343
Fix ASR decoding issues with CUDA graphs in training by @artbataev :: PR: #14184
Streaming Sortformer release PR01: uploading bugfixes, refactored variables and yaml file name changes by @tango4j :: PR: #14416
Streaming Sortformer release PR02: unit tests for streaming models and modules by @tango4j :: PR: #14417
GPU-accelerated Phrase-Boosting (GPU-PB) for CTC, RNN-T, and TDT decoding by @andrusenkoau :: PR: #14277
Fix subsampling chunking test by @monica-sekoyan :: PR: #14452
Canary2 with NFA by @monica-sekoyan :: PR: #14121
Initial Chunking by @nune-tadevosyan :: PR: #14321
Chunking fix by @nune-tadevosyan :: PR: #14482
Tutorial and doc update by @nune-tadevosyan :: PR: #14484
Streaming Sortformer release PR03: NeMo documentations and tutorial notebook by @tango4j :: PR: #14388
Add wget_from_nemo by @nune-tadevosyan :: PR: #14623
Downgrade "datasets" library version in ASR tutorial to ensure compatibility with HF Datasets used by @KunalDhawan :: PR: #14685
Canary tutorial fix by @nune-tadevosyan :: PR: #14708
Force activations and weights cast to FP32 Jasper Encoder Squeeze-Excite by @erastorgueva-nv :: PR: #14715

</details>

TTS

<details><summary>Changelog</summary>

Improve ASR models' invariance to padding/batch size by @pzelasko :: PR: #13827
remove nlp modules by @dimapihtar :: PR: #14127
Temporarily Remove Encoder PP Support by @yaoyu-33 :: PR: #14167
Remove T5-TTS by @blisc :: PR: #14252

</details>

NLP / NMT

<details><summary>Changelog</summary>

add extra params for MegatronDataSampler by @dimapihtar :: PR: #13956
Modernize logger interface by @emmanuel-ferdman :: PR: #13783
remove dialogue collection by @dimapihtar :: PR: #14087
remove QA collection by @dimapihtar :: PR: #14092
remove text nlp collection by @dimapihtar :: PR: #14110
remove nlp modules by @dimapihtar :: PR: #14127
remove rag collection by @dimapihtar :: PR: #14157
remove nmt collection by @dimapihtar :: PR: #14191
Fix importerror in transformer_lm_model after nlp module removals by @chtruong814 :: PR: #14199
fix QA comments NVBug by @huvunvidia :: PR: #14196
Temporarily Remove Encoder PP Support by @yaoyu-33 :: PR: #14167
remove mixins collections by @dimapihtar :: PR: #14281
feat: print expert groups on megatron init by @clumsy :: PR: #13874
[speechlm2] [lhotse] sharegpt data and testloader by @huckiyang :: PR: #14294
Add notebook for LoRA on GPT-OSS-20B by @shashank3959 :: PR: #14439
Sketch dist-ckpt content versioning by @mikolajblaz :: PR: #13839
Change to enable full iteration CUDA graph for LLMs by @vasunvidia :: PR: #14077

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Check lightning and core imports in install test by @chtruong814 :: PR: #14403

</details>

Export

<details><summary>Changelog</summary>

ci: Set L2_NeMo_2_Export_Deploy_Query_In_Framework to be optional by @chtruong814 :: PR: #13946
Remove old export doc by @oyilmaz-nvidia :: PR: #14292
Llama4 Export: Remove outdated MLP weight transform by @suiyoubi :: PR: #14297
Update mllama hf import/export for transformers 4.53 by @meatybobby :: PR: #14327

</details>

Bugfixes

<details><summary>Changelog</summary>

Bugfix for Hyena to the get_t function which comes up when doing longer context inference by @jstjohn :: PR: #14256
fix skipped cuHyena kernel while training by @farhadrgh :: PR: #14365
Remove flaky Evo2 dataset performance test by @jstjohn :: PR: #14371
Use module prefix in restore_modelopt_state by @jenchen13 :: PR: #14384

</details>

Uncategorized:

<details><summary>Changelog</summary>

Version bump to 2.5.0rc0.dev0 by @github-actions[bot] :: PR: #13944
[Llama4] Enable tp comm overlap for llama4 by @gdengk :: PR: #13940
Fix for Squad Dataset Download by @rhmukundan :: PR: #13893
add nmh HF conversion by @JRD971000 :: PR: #13941
Speechlm2 SALM improvements by @pzelasko :: PR: #13829
fix dataset issue by @dimapihtar :: PR: #13953
Editing MMLU to pull from the correct repo by @ruchaa-apte :: PR: #13991
move classes to module to use target feature (#14023) by @nithinraok :: PR: #14031
Add Nemotron-H prompt format, fix cut-to-conversation custom attr propagation by @pzelasko :: PR: #13963
Bump release_library template to v0.40.0 by @chtruong814 :: PR: #14046
[automodel] add support for layer-freezing by @akoumpa :: PR: #14000
[Qwen3] Recipe config bug fix by @gdengk :: PR: #14084
Add TE import guard in qwen2vl vision module by @chtruong814 :: PR: #14091
Update bitsandbytes dependency to v0.46.0 by @pramodk :: PR: #14050
Update FSDP2 docstring by @BoxiangW :: PR: #14105
Interface to enable fsdp-double-buffer without enabling NCCL-UB by @youngeunkwon0405 :: PR: #14076
SpeechLM2 SALM: load ckpt faster, with less GPU memory by @pzelasko :: PR: #14113
Add object_storage_cache_path to PreTrainingDataModule by @shunjiad :: PR: #14103
Update changelog for r2.3.0 by @github-actions[bot] :: PR: #14160
Fix FLUX test with correct env var by @suiyoubi :: PR: #14149
add mmap_bin_files param by @dimapihtar :: PR: #14122
Add option to suppress import checks in Dockerfile.speech by @artbataev :: PR: #14185
Safely import optional python packages by @roclark :: PR: #13936
Set flux test as optional by @chtruong814 :: PR: #14190
Revert "Safely import optional python packages (#13936)" by @chtruong814 :: PR: #14197
Fix "Safely import optional python packages (#13936)" by @chtruong814 :: PR: #14198
Add fix for evo2 generate/inference by @jwilber :: PR: #14027
Fixing file path suffix by @gautham-kollu :: PR: #14179
Update AVLM finetune example for vanilla fine-tuning by @huvunvidia :: PR: #14232
[finetune] Add dataset_kwargs to prepare packed sequence data by @jiajunly :: PR: #14169
Allow exception in hf ckpt load attempt before fallback to standard l… by @trvachov :: PR: #14214
Load master weights from checkpoint by @kunlunl :: PR: #14072
Add deploy lora adapter portion by @ruchaa-apte :: PR: #14255
fix speechlm lhotse loading nemo_tarred by @stevehuang52 :: PR: #14314
Update changelog for r2.4.0 by @github-actions[bot] :: PR: #14334
Flaky test timing out: @pytest.mark.pleasefixme by @pablo-garay :: PR: #14351
Support dump perf recipe diff from base recipe by @guyueh1 :: PR: #14206
Bugfix degenerate bases evo2 dataset by @jstjohn :: PR: #14359
Hyena support for flash decode API by @jstjohn :: PR: #14315
Fix Gemma2/3 & Llava (Next) & Llama4 conversion issue with latest transformers by @suiyoubi :: PR: #14367
fix: reduce the excessive test time of test_msdd_diar_inference by @tango4j :: PR: #14366
SpeechLM2: S2S->S2T data reader, excessive padding fixes by @pzelasko :: PR: #14124
chore: Release 2.5.0rc0 by @ko3n1g :: PR: #14389
Add pyxis flag for container writable. by @sudostock :: PR: #14395
[MoE] Partial Cudagraph support for MoE by @gdengk :: PR: #14362
Revert "[MoE] Partial Cudagraph support for MoE (#14362)" by @chtruong814 :: PR: #14402
Update AVLM recipes for NeMo-CI runs by @huvunvidia :: PR: #14397
Remove nemo1 multimodal and vision by @yaoyu-33 :: PR: #14095
Fix LazyNeMoIterator supervision for multi-channel cuts by @anteju :: PR: #14409
Bump Mcore to 7f7439f by @chtruong814 :: PR: #14373
Use cuhyena rearrange when available. by @moradza :: PR: #14383
Fix model training/eval state after PTL validation loop by @paul-gibbons :: PR: #14152
Add deprecation notice to eval code by @athitten :: PR: #14316
Streaming Sortformer release PR04: Adding functional tests for streaming sortformer by @tango4j :: PR: #14435
QWEN2.5-VL 7B Performance Recipe by @tomlifu :: PR: #14401
Discount FLOPs in dot-product att by @erhoo82 :: PR: #14424
Bump to pytorch 25.06 and newer TE commit by @chtruong814 :: PR: #14423
Enable precision aware optimizer for dsv3 by @guyueh1 :: PR: #14444
Make VBoost activation conditional by @bdubauski :: PR: #14458
cuHyena FFTConv support for Hyena Long Implicit (LI) Layer by @farhadrgh :: PR: #14396
Alit/nano v2 by @JRD971000 :: PR: #14464
Fix reuse_grad_buf_for_mxfp8_param_ag for mxfp8 by @guyueh1 :: PR: #14445
Fix loss mask for chat datasets by @cuichenx :: PR: #14369
Rename to subquadratic_ops by @farhadrgh :: PR: #14486
Allows using other signals (than SIGTERM) with PreemptionPlugin by @zachmoshe :: PR: #14248
Qwen2.5-VL 32B Performance Recipe by @tomlifu :: PR: #14485
Alit/nanov2 12b by @JRD971000 :: PR: #14483
Freeze tags in in r2.5.0 by @github-actions[bot] :: PR: #14513
deprecate t0 by @dimapihtar :: PR: #14599
Cherry pick Use hugginface_hub for downloading the FLUX checkpoint (14638) into r2.5.0 by @chtruong814 :: PR: #14640
Cherry pick Fix function calling notebook (14643) into r2.5.0 by @chtruong814 :: PR: #14650
Cherry pick remove service launch scripts (14647) into r2.5.0 by @chtruong814 :: PR: #14648
Cherry pick Delete tutorials/llm/llama/biomedical-qa directory (14653) into r2.5.0 by @chtruong814 :: PR: #14654
Cherry pick Remove PEFT scheme condition from recipe (14661) into r2.5.0 by @chtruong814 :: PR: #14662
Cherry pick fixing kernel restarting when transcribing (14665) into r2.5.0 by @chtruong814 :: PR: #14672
Delete nemo 1 notebooks by @cuichenx :: PR: #14675
Cherry pick Fixing Sortformer training tutorial notebook (14680) into r2.5.0 by @chtruong814 :: PR: #14681
Cherry-pick Update get_tensor_shapes function whose signature was refactored (14594) into r2.5.0 by @chtruong814 :: PR: #14678
Cherry pick Skip trt-llm and vllm install in install test (14663) into r2.5.0 by @chtruong814 :: PR: #14697
Cherry pick Fix for \EncDecRNNTBPEModel transcribe() failed with TypeError\ (14698) into r2.5.0 by @chtruong814 :: PR: #14709
Cherry pick Fix broken link in Reasoning-SFT.ipynb (14716) into r2.5.0 by @chtruong814 :: PR: #14717
cherry-pick add load-in-4bit param (14636) into r2.5.0 by @dimapihtar :: PR: #14719
Cherry pick Fix deepseek export dtype (14307) into r2.5.0 by @chtruong814 :: PR: #14682
Cherry pick remove env var (14739) into r2.5.0 by @chtruong814 :: PR: #14746
Cherry-pick 'Bump modelopt to 0.35.0 and remove safe_import("modelopt") in llm collection (#14656)' into 'r2.5.0' by @chtruong814 :: PR: #14771
Cherry pick Update prune-distill notebooks to Qwen3 + simplify + mmlu eval (14785) into r2.5.0 by @chtruong814 :: PR: #14789
Cherry pick Remove export-deploy, automodel, and eval tutorials (14790) into r2.5.0 by @chtruong814 :: PR: #14792
Cherry pick ci: Automodel deprecation warning (14787) into r2.5.0 by @chtruong814 :: PR: #14791

</details>

Open source →

2.4.025 Jul 2025Security

Cherry pick Moving export security fixes over here (14254) into r2.4.0 by @chtruong814 :: PR: #14261

Highlights

Collections:
- Speech
  - Batched beam search for transducers (RNN-T and TDT)
    - RNNT/TDT buffered/streaming inference + batched decoding support in cache-aware
    - add support for CTC batched beam search with GPU-LM
    - Key fixes
      - Punctuation Marks in Timestamps
      - Fix timestamps when cuda graphs enabled
      - Fix masking of <pad> tokens in AED inference
      - TDT streaming inference fix
- LLM
  - Qwen 3 235B-A22B Perf Optimized
  - DeepSeek V3 Perf Optimized
  - Gemma3 support from Google
  - Embedding and Reranker models
- MM
  - Llama 4
  - AVLM
Training performance (speed)
- NVL sharp + IB sharp for DP/FSDP-communications on H100 and B200
- MXFP8 with TP communication overlap
- MXFP8 with reduced memory allocation
- FP8 sub-channel recipe (128x128 for weight and 1x128 for activation)
- cudnn fused attention for MLA (both Hopper and Blackwell)
- Advanced custom asymmetric pipelining (for MTP, loss func, and embd)
- BF16 optimizer for model memory saving
- CUDA graph fix for fine-tuning benchmarks
- CUDA graph support for LLAMA4

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

ci: Fix ASR container by @ko3n1g :: PR: #13288
Set L2_Segmentation_Tool_Parallel_ctc_segmentation test to be optional by @chtruong814 :: PR: #13296
Revert "WebDataset URL refactoring" by @ko3n1g :: PR: #13421
Update flagged docs links by @erastorgueva-nv :: PR: #13391
[Docs] Fix incorrectly formatted reference tags by @erastorgueva-nv :: PR: #13445
Update CP by @pablo-garay :: PR: #13532
Tdt buffered inference fix by @hainan-xv :: PR: #13500
Fix transcribe when nbest hypotheses are returned by @lilithgrigoryan :: PR: #13540
Set ASR test to be optional by @chtruong814 :: PR: #13633
Enabling chunked inference for AED models in asr_evaluator by @melllinia :: PR: #13674
Ko3n1g/chore/asr only by @ko3n1g :: PR: #13704
decompressing joblib file before checking it by @Ssofja :: PR: #13732
Revert "decompressing joblib file before checking it (#13732)" by @chtruong814 :: PR: #13791
Punctuation Marks in Timestamps by @monica-sekoyan :: PR: #13353
AIStore with Webdataset by @monica-sekoyan :: PR: #13604
Update to add default for dataclass variables by @nithinraok :: PR: #13814
This PR addresses to known security issues by @Ssofja :: PR: #13804
remove model_stride var by @nithinraok :: PR: #13867
add CTC batched beam search by @lilithgrigoryan :: PR: #13337
Clean up streaming ASR script and tests by @artbataev :: PR: #13894
add NGPU-LM fusion during CTC greedy by @lilithgrigoryan :: PR: #13917

</details>

TTS

<details><summary>Changelog</summary>

Revert "WebDataset URL refactoring" by @ko3n1g :: PR: #13421
Update flagged docs links by @erastorgueva-nv :: PR: #13391
[Docs] Fix incorrectly formatted reference tags by @erastorgueva-nv :: PR: #13445
Update CP by @pablo-garay :: PR: #13532
fix: vpp stage refactoring to match mcore by @ZhiyuLi-Nvidia :: PR: #13673
AIStore with Webdataset by @monica-sekoyan :: PR: #13604

</details>

NLP / NMT

<details><summary>Changelog</summary>

Migrate Hyena to Megatron inference_context. by @cspades :: PR: #13436
Update CP by @pablo-garay :: PR: #13532
fix broken links by @dimapihtar :: PR: #13544
Add nlp import checks by @thomasdhc :: PR: #13563
PTQ model support, quant_cfg, and documentation updates by @janekl :: PR: #13519
feat - GPTSFTChatDataset alignment with OpenAI Messages, compatibility with packed sequences by @soluwalana :: PR: #13367
fix: vpp stage refactoring to match mcore by @ZhiyuLi-Nvidia :: PR: #13673
Fix resume with MegatronPretrainingBatchSampler by @ashors1 :: PR: #13565
Punctuation Marks in Timestamps by @monica-sekoyan :: PR: #13353
Revert Adding more doc-strings to megatron_parallel.py #12767 by @ko3n1g :: PR: #13824
reasoning model evaluation mmlu gpqa by @ruchaa-apte :: PR: #13880
Remove unused DynamicRetrievalServer and Bert dataset loader classes by @dimapihtar :: PR: #14209
Huvu/avlm qafix cherrypick from by @huvunvidia :: PR: #14253

</details>

Export

<details><summary>Changelog</summary>

Improve Nemo2Exporter for Models Using Custom Modelling Files on HF by @suiyoubi :: PR: #13400
Adding more export tests by @oyilmaz-nvidia :: PR: #13410
Add Warning to Export when output_path exists by @suiyoubi :: PR: #13465
Move libsox-fmt-all from Dockerfile.ci.export_deploy to Dockerfile.ci by @chtruong814 :: PR: #13452
ci: Remove trt-llm breakpoint by @ko3n1g :: PR: #13499
Add Qwen2VL export_ckpt by @AtsunoriFujita :: PR: #13398
Add MLlama export_ckpt by @AtsunoriFujita :: PR: #13346
Update vLLMExporter to use vLLM V1 by @janekl :: PR: #13498
Add vLLM Mixtral and TRT-LLM qnemo export tests (plus a couple of bugfixes) by @janekl :: PR: #13697
Fix Qwen3 export + misc by @cuichenx :: PR: #13679
Extra int cast for successful tracing during ONNX export by @janekl :: PR: #13782
FP8 lora export by @cuichenx :: PR: #13748
Add PEFT export check by @cuichenx :: PR: #13835
Update llm api import_ckpt/export_ckpt docstring by @meatybobby :: PR: #13714
Use modelopt export and disable dataset calibration for weight only PTQ by @jenchen13 :: PR: #13756

</details>

Bugfixes

<details><summary>Changelog</summary>

[automodel] move liger kernel patching by @akoumpa :: PR: #13579

</details>

Uncategorized

<details><summary>Changelog</summary>

build: various bumps by @ko3n1g :: PR: #13285
ci: Fixes to selective triggering by @ko3n1g :: PR: #13287
ci: Set timeout by @ko3n1g :: PR: #13294
Set L2_NeMo_2_T5_Pretraining test as optional by @chtruong814 :: PR: #13282
Add test environment approval step for CI by @chtruong814 :: PR: #13297
update num nodes in deepseek v3 finetune recipe by @cuichenx :: PR: #13314
ci: Increase cache pool by @ko3n1g :: PR: #13306
Rename adam_with_cosine_annealing as adam since cosin LR is not setup by @ShriyaRishab :: PR: #13315
ci: Update test queue bot to not assume a workflow is launched from a PR by @chtruong814 :: PR: #13318
Fix TE pytorch attention doc link by @thomasdhc :: PR: #13327
ci: Add all recent buildcaches to update-buildcache job by @ko3n1g :: PR: #13289
Fix neva notebook by @yaoyu-33 :: PR: #13334
Fix transformer offline for CI/CD llama4 tests by @yaoyu-33 :: PR: #13339
[automodel] convert lm head to full tensor before passing to lce by @yuanzhedong :: PR: #13319
ci: No dups in queue by @ko3n1g :: PR: #13352
ci(hotfix): VLM CPU unit tests by @ko3n1g :: PR: #13348
vLLM==0.8.5 update by @janekl :: PR: #13350
ci: Allow bypassing approval by @ko3n1g :: PR: #13365
Avoid the need to specify optional attributes for lhotse/nemo reader functions by @pzelasko :: PR: #13307
ci: Fix selective-triggering for non-PR events by @ko3n1g :: PR: #13374
ci: Revert no-concurrency-group-on-main by @ko3n1g :: PR: #13375
ci: Improve no-fail-fast mechanism by @ko3n1g :: PR: #13370
2d buckets estimation fix by @monica-sekoyan :: PR: #13377
ci: Fix scheduled runs by @ko3n1g :: PR: #13378
Ko3n1g/ci/fix nightly runs by @ko3n1g :: PR: #13382
[automodel] fix none issue in dataset for qwen model by @yuanzhedong :: PR: #13311
update table by @akoumpa :: PR: #13397
Improve test coverage for audio modules by @anteju :: PR: #13333
Disable failing maxine loss test by @anteju :: PR: #13361
Ko3n1g/ci/no notification on cancel by @ko3n1g :: PR: #13403
document fp8_recipe by @akoumpa :: PR: #13405
Weekly bump main by @ko3n1g :: PR: #13408
Handle boolean args for performance scripts and log received config by @guyueh1 :: PR: #13291
[automodel] add FirstRankPerNode by @akoumpa :: PR: #13373
tests: Disable flaky audio test by @ko3n1g :: PR: #13429
ci: Disable flaky audio test by @ko3n1g :: PR: #13435
Fix loss compute and reduction by @xrennvidia :: PR: #13295
ci: Skip link check on github links by @chtruong814 :: PR: #13425
Add NCCL cfg interface to perf scripts by @erhoo82 :: PR: #13407
ci: Success only if Run CICD label attached by @ko3n1g :: PR: #13430
ci: Add tests to selective triggering by @ko3n1g :: PR: #13404
ci: Remove jq by @ko3n1g :: PR: #13440
ci: Fix deps tree for tests by @ko3n1g :: PR: #13443
Ko3n1g/ci/fix dependency tree by @ko3n1g :: PR: #13448
Adding additional unit tests for the deploy module by @pthombre :: PR: #13411
[Audio] fix a flaky test (and also make some tests run faster) by @racoiaws :: PR: #13439
[automodel] ignore tail padding in TPS calculation by @akoumpa :: PR: #13329
Ko3n1g/ci/selective triggering 3 by @ko3n1g :: PR: #13460
ci: Disable broken neva tests by @ko3n1g :: PR: #13461
fix speechlm data module by @stevehuang52 :: PR: #13362
ci: Enter queue only with passing linting by @ko3n1g :: PR: #13462
Adding tests for Schroedinger Bridge model by @nasretdinovr :: PR: #13401
add more detailed description by @dimapihtar :: PR: #13464
[Audio] tests for score-based and flow matching enhancement models by @racoiaws :: PR: #13406
Use expandable cuda memory segmentation by @erhoo82 :: PR: #13418
Fix llava tokenizer caused nan issue by @yaoyu-33 :: PR: #13466
Remove cuda method from ModelPT by @erastorgueva-nv :: PR: #13394
Fix BNR 2 unit test + input, case where input length was not specified by @nitin9252 :: PR: #13467
ci: Do not run any tests if no match is found by @ko3n1g :: PR: #13479
Ko3n1g/ci/selective triggering 4 by @ko3n1g :: PR: #13489
Fix typo in the performance script by @youngeunkwon0405 :: PR: #13487
ci: No runs on main by @ko3n1g :: PR: #13490
ci: Upload on schedule by @ko3n1g :: PR: #13491
ci: Run selective triggering on dockerfiles and dependencies by @ko3n1g :: PR: #13493
[automodel] fallback FP8 + LCE -> FP8 + CE by @akoumpa :: PR: #13349
Update changelog for r2.3.0 by @github-actions[bot] :: PR: #13501
Update 2.3.0 changelog by @chtruong814 :: PR: #13504
Enabling flash decode for float16 precision only by @pthombre :: PR: #13471
Fix changelog formatting by @chtruong814 :: PR: #13505
Updating the long context performance number for B200 by @youngeunkwon0405 :: PR: #13468
ci: Add more files to filter by @ko3n1g :: PR: #13517
Improve error message when HF checkpoint cannot be loaded by @ashors1 :: PR: #13513
Add Resume_path to llama_nemotron models by @suiyoubi :: PR: #13515
Add Llama4 GHA by @suiyoubi :: PR: #13442
add memory profile interface to perf scripts by @erhoo82 :: PR: #13413
Add fp8_param argument back to mixed precision plugin for backward compatibility by @guyueh1 :: PR: #13522
[automodel] add find_unused_parameters=True for DDP by @akoumpa :: PR: #13366
ci: Update success message by @ko3n1g :: PR: #13541
[Audio] TransformerUNet: predictive model support added by @nasretdinovr :: PR: #13470
Test Hyena mixer CP equivalency by @farhadrgh :: PR: #13330
use null tokenizer by @malay-nagda :: PR: #13480
ci: Remove optional marker by @ko3n1g :: PR: #13469
Update extra_requires and requirements by @thomasdhc :: PR: #13359
Fix default config for LlamaNemotron Ultra by @suiyoubi :: PR: #13542
[audio] Improve test coverage for audio losses by @anteju :: PR: #13309
deepseek finetuning callback error change by @SDcodehub :: PR: #13483
ci(fix): Add __init__ to selective-triggering by @ko3n1g :: PR: #13577
nsys profile filename ranks info by @malay-nagda :: PR: #13576
chore: Update setup.py by @ko3n1g :: PR: #13566
Fix Llama importer by @suiyoubi :: PR: #13583
[automodel] fix --mbs/gbs dtype and chat-template by @akoumpa :: PR: #13602
Reconfigure 'limit_<train|val>_batches' by @maanug-nv :: PR: #13523
ci: Optional speech tests by @ko3n1g :: PR: #13606
[Automodel] Fix CP device_mesh issue, use PTL distsampler by @BoxiangW :: PR: #13473
[automodel] fix log message by @akoumpa :: PR: #13612
Tests for evaluation with NVIDIA Evals Factory by @chtruong814 :: PR: #13627
Fix ptl import in notebooks by @maanug-nv :: PR: #13608
[automodel] dist.abort -> dist.destroy_process_group by @akoumpa :: PR: #13578
Skip eval unit test by @chtruong814 :: PR: #13635
Fix image_processor config in Energon path by @AtsunoriFujita :: PR: #13618
Add Gemma3 VL model by @xiangxu-google :: PR: #13536
Set L2_NeMo_2_EVAL as optional by @chtruong814 :: PR: #13644
Update install to use pip install by @thomasdhc :: PR: #13605
Multi node settings for evaluation nemo-run script by @athitten :: PR: #13568
[Llama4] Fix the missing args in the recipe by @gdengk :: PR: #13649
Bump nvidia-modelopt to 0.29.0 by @AAnoosheh :: PR: #13599
Update README.md for 25.04 release by @snowmanwwg :: PR: #13654
[automodel] consolidate sft peft scripts by @akoumpa :: PR: #13634
Qwen3 by @cuichenx :: PR: #13554
Set env variables for eval tests by @marta-sd :: PR: #13658
build: multimodal-only by @ko3n1g :: PR: #13665
[Audio] TransformerUNet: predictive model tests added by @nasretdinovr :: PR: #13648
[automodel] consolidate vllm scripts by @akoumpa :: PR: #13670
build: Pin transformers by @ko3n1g :: PR: #13675
ci: Enable codecov checks by @ko3n1g :: PR: #13497
ci: Add init-file-checker by @ko3n1g :: PR: #13684
Add use_sharp and use user buffer registration args in perf scripts by @youngeunkwon0405 :: PR: #13521
Remove is-optional marker for L2_NeMo_2_EVAL by @marta-sd :: PR: #13669
gpu type and #devices CLI args by @malay-nagda :: PR: #13620
perf scripts updates by @malay-nagda :: PR: #13456
Use audio codec without discriminators in SpeechLM2 tests by @pzelasko :: PR: #13711
Update changelog for r2.3.1 by @github-actions[bot] :: PR: #13719
Recipe default value fix for Llama4 by @suiyoubi :: PR: #13696
build: Lift numba by @ko3n1g :: PR: #13735
New key override for timestamps by @melllinia :: PR: #13743
Fixed Mllama Energon config by @AtsunoriFujita :: PR: #13574
Update convert_to_tarred_audio_dataset.py by @ssh-meister :: PR: #13755
Enable dropout recompute in LoRA by @michal2409 :: PR: #13745
Address VDR feedback for NeMo FW evaluations by @athitten :: PR: #13701
remove blocks unused to increase coverage by @romanbrickie :: PR: #13511
Fix Flux Recipe for FSDP/DDP by @suiyoubi :: PR: #13715
Try soften protobuf version requirement by @pablo-garay :: PR: #13747
Flux FP8 recipe by @Victor49152 :: PR: #13584
Gemma3 Fix and Tests by @suiyoubi :: PR: #13661
Disable local gradient checker in performance scripts by @erhoo82 :: PR: #13768
[Audio] Tests: training for mask, pred and SB models by @nasretdinovr :: PR: #13736
Refactor MSC integration in exp manager by @shunjiad :: PR: #13626
[fix] vpp error in Gemma3 by @ZhiyuLi-Nvidia :: PR: #13784
ci: Ensure approval queue fetches all CICD workflows using pagnation by @chtruong814 :: PR: #13798
ci: make_request in approval test queue appends next url for status checks only by @chtruong814 :: PR: #13802
Remove guard for masking tests and improve coverage by @anteju :: PR: #13787
fix: After mcore bump by @ko3n1g :: PR: #13781
Fix Gemma3VL training bugs by @sharanmayank :: PR: #13766
[NeMo 2.0] Remove the restriction of load_model_state_dict for cfsdp by @shjwudp :: PR: #13512
Add option to construct Llama model with Transformer Engine op fuser by @timmoon10 :: PR: #13776
[Evaluation] Add support for simple-evals and tasks that require logprobs by @marta-sd :: PR: #13647
remove stale section by @akoumpa :: PR: #13759
fix moe_router_pre_softmax for Mixtral by @akoumpa :: PR: #13678
fix: improve sequence length handling to fix nan in loss when turning on cudagraph by @katec846 :: PR: #13779
Gemma3 Energon Dataset by @suiyoubi :: PR: #13813
Rectify BLEU evaluation by @ankitapasad :: PR: #13762
ci: Moved workflows by @ko3n1g :: PR: #13828
ci: Moved templates by @ko3n1g :: PR: #13830
[Build] Bump bitsandbytes dependency to 0.45.5 (ubuntu 22.04 compatibility) by @pramodk :: PR: #13789
update for PYTORCH_CUDA_ALLOC_CONF env var by @malay-nagda :: PR: #13837
[Llama4] Enable VLM Dec cudagraph by @gdengk :: PR: #13767
Support MSC URL in LLM checkpointing by @shunjiad :: PR: #13805
additional metrics by @dimapihtar :: PR: #13754
Expand modelopt version range by @chtruong814 :: PR: #13850
Alit/nmh4b by @JRD971000 :: PR: #13481
[Tutorial] Train your own reasoning model in 48 hours on a single GPU by @Maghoumi :: PR: #13853
Enabled C2C-PCie bridge through NCCL by @sanandaraj5597 :: PR: #13621
Added safe loading of models by @nithinraok :: PR: #13607
Add NemotronH Performance Script by @guyueh1 :: PR: #13528
Hyena SE/MR B2B Kernel integration by @farhadrgh :: PR: #13518
chore: Destroy buildcache by @ko3n1g :: PR: #13869
tests: Fix Qwen test by @ko3n1g :: PR: #13888
fix: improve error handling in is_multistorageclient_url by @shunjiad :: PR: #13885
feat(eval): adds benchmark adapters that allow specisal reasoning models by @agronskiy :: PR: #13709
perf scripts 25.07 refactor by @malay-nagda :: PR: #13875
Fix E5 and LlamaEmbedding Conversion by @suiyoubi :: PR: #13890
Bug fix for NCCL vars by @sanandaraj5597 :: PR: #13908
Reranker Model Support by @suiyoubi :: PR: #13876
numa cmd in bash by @malay-nagda :: PR: #13914
Fix BERT issue with PP by @suiyoubi :: PR: #13916
[Llama4] Fix Vp_stage to enable VP for VLM llama4 by @gdengk :: PR: #13873
Enable NVTX profiling in MCore by @minitu :: PR: #13820
[Qwen3-MoE] Add Qwen3 MoE perf recipe for 30b and 235b by @gdengk :: PR: #13895
lazy import bnbconfig by @akoumpa :: PR: #13919
Set TRANSFORMERS_OFFLINE=1 and HF_HUB_OFFLINE=1 in CI tests by @chtruong814 :: PR: #13932
[peft] align adapter output shape with wrapped module output shape by @guyueh1 :: PR: #13922
[automodel] move only lora adapters to cpu by @akoumpa :: PR: #13931
Fix vp_stage not found when fsdp by @gautham-kollu :: PR: #13817
Fix single optional import if ModelOpt not installed by @AAnoosheh :: PR: #13923
Revert "Set TRANSFORMERS_OFFLINE=1 and HF_HUB_OFFLINE=1 in CI tests by @chtruong814 :: PR: #13938
Enable LoRA for TELinear layers by @cuichenx :: PR: #13929
Freeze tags in in r2.4.0 by @github-actions[bot] :: PR: #13945
Cherry pick Use jiwer less than 4.0.0 (13997) into r2.4.0 by @ko3n1g :: PR: #13998
Cherry pick Remove container license reference (14010) into r2.4.0 by @ko3n1g :: PR: #14017
move classes to module to use target feature by @nithinraok :: PR: #14023
Cherry pick bf16 grads for bf16 jobs (14016) into r2.4.0 by @ko3n1g :: PR: #14020
Cherry pick Remove nemo1 stable diffusion test (14018) into r2.4.0 by @ko3n1g :: PR: #14019
Version bump to 2.4.0rc1.dev0 by @github-actions[bot] :: PR: #14047
Cherry pick Fix Loading Custom Quantization Config (13934) into r2.4.0 by @ko3n1g :: PR: #13950
Cherry pick [automodel] fix sft notebook (14002) into r2.4.0 by @ko3n1g :: PR: #14003
Cherry pick Use average reduction in FSDP grad reduce-scatter when grad dtype is … (13981) into r2.4.0 by @ko3n1g :: PR: #14004
Cherry pick GPU memory logging update (13982) into r2.4.0 by @ko3n1g :: PR: #14021
Cherry pick Remove kaldiio (14006) into r2.4.0 by @ko3n1g :: PR: #14032
Cherry pick Set L2_NeMo_2_Flux_Import_Test to be optional (14056) into r2.4.0 by @ko3n1g :: PR: #14058
Cherry pick Bump protobuf to 5.29.5 (14045) into r2.4.0 by @ko3n1g :: PR: #14060
Cherry pick Detect hardware before enabling DeepEP (14022) into r2.4.0 by @ko3n1g :: PR: #14068
Version bump to 2.4.0rc2.dev0 by @github-actions[bot] :: PR: #14115
Cherry pick Fix SFT Dataset Bug (13918) into r2.4.0 by @ko3n1g :: PR: #14074
Cherry pick Align adapter shape with base linear output shape (14009) into r2.4.0 by @ko3n1g :: PR: #14083
Cherry pick [MoE] Update the fp8 precision interface for llama4 and qwen3 (14094) into r2.4.0 by @ko3n1g :: PR: #14104
Cherry pick [Llama4] Tokenizer naming update (14114) into r2.4.0 by @ko3n1g :: PR: #14123
Cherry pick Bump to pytorch 25.05 container along with TE update (13899) into r2.4.0 by @ko3n1g :: PR: #14145
Cherry pick Perf scripts updates (14005) into r2.4.0 by @ko3n1g :: PR: #14129
Cherry pick Remove unstructured (14070) into r2.4.0 by @ko3n1g :: PR: #14147
Version bump to 2.4.0rc3.dev0 by @github-actions[bot] :: PR: #14165
Cherry pick Add checkpoint info for NIM Embedding Expor Tutorial (14177) into r2.4.0 by @ko3n1g :: PR: #14178
Cherry pick Fix dsv3 script (14007) into r2.4.0 by @ko3n1g :: PR: #14182
Cherry pick 405b perf script updates (14176) into r2.4.0 by @chtruong814 :: PR: #14195
Cherry pick Fix nemotronh flops calculator (14161) into r2.4.0 by @chtruong814 :: PR: #14202
Cherry pick Add option to disable gloo process groups (#14156) into r2.4.0 by @chtruong814 :: PR: #14220
Cherry pick Remove g2p_en (14204) into r2.4.0 by @chtruong814 :: PR: #14212
Cherry pick diffusion mock data null args (14173) into r2.4.0 by @chtruong814 :: PR: #14217
Cherry pick perf-scripts: Change b200 config to EP8 (14207) into r2.4.0 by @chtruong814 :: PR: #14223
Cherry pick Change RerankerSpecter Dataset question key (14200) into r2.4.0 by @chtruong814 :: PR: #14224
Cherry pick Fix the forward when final_loss_mask is not present (14201) into r2.4.0 by @chtruong814 :: PR: #14225
Cherry pick Fix Llama Nemotron Nano Importer (14222) into r2.4.0 by @chtruong814 :: PR: #14226
Cherry pick [automodel] fix loss_mask pad token (14150) into r2.4.0 by @chtruong814 :: PR: #14227
[Performance script] FSDP-UBR related recipe update (#14208) by @youngeunkwon0405 :: PR: #14233
Fix for MCore dist ckpt loading #14229 by @stevehuang52 :: PR: #14239
cherry-pick fix eval beam search ctc script by @lilithgrigoryan :: PR: #14242
Cherry pick Moving export security fixes over here (14254) into r2.4.0 by @chtruong814 :: PR: #14261
Cherry pick Confidence fix for tutorial (14250) into r2.4.0 by @chtruong814 :: PR: #14266
Cherry pick added new models to documentation (14264) into r2.4.0 by @chtruong814 :: PR: #14278
Cherry-pick FIx Flux & Flux_Controlnet initialization issue (#14263) into r2.4.0 by @chtruong814 :: PR: #14273
Cherry pick update ffmpeg install (14237) into r2.4.0 by @chtruong814 :: PR: #14279

</details>

Open source →

2.3.125 May 2025Security

Updated vLLMExporter to use vLLM V1 to address a security vulnerability.

Highlights

Collections
- LLM
  - Llama 4: Fixed an accuracy issue caused by MoE probability normalization. Improved pre-train and fine-tune performance.
Export & Deploy
- Updated vLLMExporter to use vLLM V1 to address a security vulnerability.
AutoModel
- Improved chat-template handling.
Fault Tolerance
- Local checkpointing: Fixed support for auto-inserted metric names for resuming from local checkpoints.

Detailed Changelogs:

</details>

Export

<details><summary>Changelog</summary>

Cherry-pick Update vLLMExporter to use vLLM V1 (#13498) into r2.3.0 by @chtruong814 :: PR: #13631

</details>

Uncategorized:

<details><summary>Changelog</summary>

Bump to 2.3.1 by @chtruong814 :: PR: #13507
Cherry pick Use explicitly cached canary-1b-flash in CI tests (13237) into r2.3.0 by @ko3n1g :: PR: #13508
Cherry pick [automodel] bump liger-kernel to 0.5.8 + fallback (13260) into r2.3.0 by @ko3n1g :: PR: #13308
Cherry-pick Add recipe and ci scripts for qwen2vl to r2.3.0 by @romanbrickie :: PR: #13336
Cherry pick Fix skipme handling (13244) into r2.3.0 by @ko3n1g :: PR: #13376
Cherry pick Allow fp8 param gather when using FSDP (13267) into r2.3.0 by @ko3n1g :: PR: #13383
Cherry pick Handle boolean args for performance scripts and log received config (13291) into r2.3.0 by @ko3n1g :: PR: #13416
Cherry pick new perf configs (13110) into r2.3.0 by @ko3n1g :: PR: #13431
Cherry pick Adding additional unit tests for the deploy module (13411) into r2.3.0 by @ko3n1g :: PR: #13449
Cherry pick Adding more export tests (13410) into r2.3.0 by @ko3n1g :: PR: #13450
Cherry pick [automodel] add FirstRankPerNode (13373) into r2.3.0 by @ko3n1g :: PR: #13559
Cherry pick [automodel] deprecate global_batch_size dataset argument (13137) into r2.3.0 by @ko3n1g :: PR: #13560
Cherry-pick [automodel] fallback FP8 + LCE -> FP8 + CE (#13349) into r2.3.0 by @chtruong814 :: PR: #13561
Cherry pick [automodel] add find_unused_parameters=True for DDP (13366) into r2.3.0 by @ko3n1g :: PR: #13601
Cherry pick Add CI test for local checkpointing (#13012) into r2.3.0 by @ananthsub :: PR: #13472
Cherry pick [automodel] fix --mbs/gbs dtype and chat-template (13598) into r2.3.0 by @akoumpa :: PR: #13613
Cherry-pick Update t5.py (#13082) to r2.3.0 and bump mcore to f98b1a0 by @chtruong814 :: PR: #13642
[Automodel] Fix CP device_mesh issue, use PTL distsampler (#13473) by @akoumpa :: PR: #13636
[Llama4] Fix the recipe bug - cherrypick #13649 by @gdengk :: PR: #13650
build: Pin transformers (#13675) by @ko3n1g :: PR: #13692

</details>

Open source →

2.3.08 May 2025

ONNX and TensorRT Export for NIM Embedding Container

Highlights

Export & Deploy
- NeMo 2.0 export path for NIM
- ONNX and TensorRT Export for NIM Embedding Container
- In-framework deployment for HF Models
- TRT-LLM deployment for HF Models in NeMo Framework
Evaluation
- Integrate nvidia-lm-eval to NeMo FW for evaluations with OpenAI API compatible in-framework deployment
AutoModel
- VLM AutoModelForImageForTextToText
- FP8 for AutoModel
- Support CP with FSDP2
- Support TP with FSDP2
- Performance Optimization
  - add support for cut cross entropy & liger kernel
  - Gradient Checkpointing
Fault Tolerance
- Integrate NVRx v0.3 Local checkpointing
Collections
- LLM
  - Llama4
  - Llama Nemotron Ultra
  - Llama Nemotron Super
  - Llama Nemotron Nano
  - Nemotron-h/5
  - DeepSeek V3 Pretraining
  - Evo2
  - Qwen 2.5
  - LoRA for Qwen3-32B and Qwen3-30B-A3B
- MultiModal
  - FLUX
  - Gemma 3
  - Qwen2-VL
- ASR
  - NeMo Run support for ASR training
  - N-Gram LM on GPU for AED
  - N-Gram LM on GPU + Transducer greedy decoding (RNN-T, TDT)
  - Timestamps support for AED timestamp supported models
  - Migrate SpeechLM to NeMo 2.0
  - Canary-1.1
  - Replace ClassificationModels class with LabelModels
Performance
- Functional MXFP8 support for (G)B200
- Current scaling recipe with TP communication overlap and FP8 param gathers
- Custom FSDP support that fully utilizes GB200 NVL72

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

Added model config params for Canary-1B-Flash, Canary-180M-Flash models by @KunalDhawan :: PR: #12588
Canary tutorial by @ankitapasad :: PR: #12613
Canary tutorial fix timestamp by @ankitapasad :: PR: #12677
revert config by @nithinraok :: PR: #12689
canary longform inference script with timestamps option by @krishnacpuvvada :: PR: #12653
Fix default timestamps value for Hybrid ASR models by @artbataev :: PR: #12681
Fix k2 installation with PyTorch 2.6.0 by @artbataev :: PR: #12686
Improve time and RTFx report for ASR by @artbataev :: PR: #12680
Modify train args by @ankitapasad :: PR: #12700
Fix asr doc warnings by @nithinraok :: PR: #12720
Rename FastNGramLM -> NGramGPULanguageModel by @artbataev :: PR: #12755
transcribe fix for new hypotheses by @nune-tadevosyan :: PR: #12801
Fix timestamps when cuda graphs enabled by @monica-sekoyan :: PR: #12808
update streaming conformer by @stevehuang52 :: PR: #12846
AED Decoding with N-Gram LM by @artbataev :: PR: #12730
update notebook by @nithinraok :: PR: #13088
bugfix ASR_Context_Biasing.ipynb by @lilithgrigoryan :: PR: #13109
Change branch for installation from main to r2.3.0 by @ankitapasad :: PR: #13266

</details>

TTS

<details><summary>Changelog</summary>

Add Magpie-TTS and Updates NeMo Audio Codecs by @blisc :: PR: #12606
fix bug from prior commit (#13264) by @blisc :: PR: #13328

</details>

NLP / NMT

<details><summary>Changelog</summary>

Remove old peft docs by @cuichenx :: PR: #12675
Add code coverage for llm gpt models conversion tests by @suiyoubi :: PR: #12665
Make BERT TransformerBlockWithPostLNSupport accept more inputs from Mcore by @suiyoubi :: PR: #12685
remove gifs from documentation by @dimapihtar :: PR: #12732
Rename FastNGramLM -> NGramGPULanguageModel by @artbataev :: PR: #12755
fix NeMo documentation by @dimapihtar :: PR: #12754
GPT Model/Data/Recipe Unit Test by @suiyoubi :: PR: #12757
ci: Exclude nlp, mm, vision collections by @ko3n1g :: PR: #12816
Add vocab size as attr to GPT and T5 Configs, use file name based logger in llm.gpt.data by @hemildesai :: PR: #12862
Fix transformer layer api with megatron cbc89b3 by @yaoyu-33 :: PR: #12885

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Rename FastNGramLM -> NGramGPULanguageModel by @artbataev :: PR: #12755

</details>

Export

<details><summary>Changelog</summary>

GHA Conversion Test and Importer/Exporter Refactor by @suiyoubi :: PR: #12597
Fix Llama Embedding Model Exporting keys by @suiyoubi :: PR: #12691
build: Add trtllm by @ko3n1g :: PR: #12672
Fix trt-llm install by @chtruong814 :: PR: #12827
Update LLaVA's next HF exporter to load ViT checkpoint from YAML by @eagle705 :: PR: #12841
Support huggingface export to tensorrtllm by @pthombre :: PR: #12889
Adds a built stage for the trt-llm wheel to reduce the overall test image size by @chtruong814 :: PR: #12883

</details>

Uncategorized:

<details><summary>Changelog</summary>

Update changelog-build.yml by @ko3n1g :: PR: #12584
Update changelog for r2.2.0 by @github-actions[bot] :: PR: #12585
Add comments for requirements by @thomasdhc :: PR: #12603
[automodel] FSDP2Strategy: move to device if using a single-device by @akoumpa :: PR: #12593
build: Remove numba pin by @ko3n1g :: PR: #12604
docs: Update installation guides by @ko3n1g :: PR: #12596
Change Llama Scaling Factor type to Float by @suiyoubi :: PR: #12616
ci: Test multiple python versions by @ko3n1g :: PR: #12619
ci: Disable reformat by @ko3n1g :: PR: #12620
Updating ModelOpt to 0.25.0 by @janekl :: PR: #12633
[automodel] add additional hf_dataset tests by @akoumpa :: PR: #12646
[automodel] add jit_transform tests by @akoumpa :: PR: #12645
[automodel] init eos_token_id inside data module by @yuanzhedong :: PR: #12610
[automodel] grad ckpt by @akoumpa :: PR: #12644
bugfix(llm/LLaMa) - dropout_position can never be equal to extended string by @soluwalana :: PR: #12649
Fix inference pipeline quality issue by @Victor49152 :: PR: #12639
[automodel] switch to direct=True to propage return codes in nemorun by @akoumpa :: PR: #12651
add Auto Conf support for bert, t5, qwen, starcoder models by @dimapihtar :: PR: #12601
ci: Upload coverage by @ko3n1g :: PR: #12668
ci: Re-enable changed-files action by @ko3n1g :: PR: #12683
build: Pin sox by @ko3n1g :: PR: #12701
add neva quantization by @linnanwang :: PR: #12698
Clip coverage by @abhinavg4 :: PR: #12696
GHA CI test: Remove unnecessary directive by @pablo-garay :: PR: #12714
minor perf fixes by @malay-nagda :: PR: #12656
Add DeepSeek V2 Lite into llm init.py by @suiyoubi :: PR: #12664
Add Llama-Nemotron Nano and 70B models by @suiyoubi :: PR: #12712
Save batch norm running stats in PEFT checkpoints by @cuichenx :: PR: #12666
Fix document Readme under nemo to add more information by @yaoyu-33 :: PR: #12699
Fix ub_overlap_ag by @cuichenx :: PR: #12721
Toggle fast tokenizer if error occurs by @cuichenx :: PR: #12722
Update README.md for blackwell and AutoModel by @snowmanwwg :: PR: #12612
Raise error on import_ckpt with overwrite=False plus README for checkpoint_converters by @janekl :: PR: #12693
[automodel] fix validation_step by @soluwalana :: PR: #12659
[automodel] vlm tests by @akoumpa :: PR: #12716
Auto Configurator code coverage by @dimapihtar :: PR: #12694
[automodel] fix automodle benchmark script by @yuanzhedong :: PR: #12605
Remove unnecessary directives by @pablo-garay :: PR: #12743
Add recipe tests for coverage by @cuichenx :: PR: #12737
Add Qwen2.5 in NeMo2 by @suiyoubi :: PR: #12731
add fallback_module to safe_import_from by @akoumpa :: PR: #12726
Update quantization scripts & relax modelopt requirement specifier by @janekl :: PR: #12709
Import guard fasttext by @thomasdhc :: PR: #12758
[automodel] chunked cross entropy by @akoumpa :: PR: #12752
Add fsdp automodel test by @BoxiangW :: PR: #12718
[automodel] if peft move only adapters to cpu by @akoumpa :: PR: #12735
[automodel] update hf mockdataset by @akoumpa :: PR: #12643
[automodel] remove unused cell in multinode notebook by @yuanzhedong :: PR: #12624
Yash/llava next coverage by @yashaswikarnati :: PR: #12745
Tidy code: remove unneeded statements/lines by @pablo-garay :: PR: #12771
Pass tensor instead of raw number in _mock_loss_function in PTQ by @janekl :: PR: #12769
ci: Run on nightly schedule by @ko3n1g :: PR: #12775
Add logs for checkpoint saving start and finalization by @lepan-google :: PR: #12697
Alit/test coverage by @JRD971000 :: PR: #12762
Fix loss mask with packed sequence by @ashors1 :: PR: #12642
Add pruning recipe by @kevalmorabia97 :: PR: #12602
Update qwen2-v1 to use NeMo quick_gelu by @thomasdhc :: PR: #12787
[doc] Fixes for audio doc warnings by @anteju :: PR: #12736
ci: Measure multiprocessing by @ko3n1g :: PR: #12778
ci: Fix flaky LLM tests by @ko3n1g :: PR: #12807
Add BERT/Qwen2.5 Unit test and Refactor all GHA Conversion Tests by @suiyoubi :: PR: #12785
Fix TransformerBlock cuda_graphs compatibility with MCore by @buptzyb :: PR: #12779
ci: Remove --branch by @ko3n1g :: PR: #12809
ci: Move scripts fully down to files by @ko3n1g :: PR: #12802
add init.py to make this a package by @akoumpa :: PR: #12814
Update changelog for r2.2.1 by @github-actions[bot] :: PR: #12818
add finetune support for Auto Configurator by @dimapihtar :: PR: #12770
[automodel] add cpu:gloo to backend by @akoumpa :: PR: #12832
add missing call to _apply_liger_kernel_to_instance by @akoumpa :: PR: #12806
Prune docker images in GHA older than 8hrs by @chtruong814 :: PR: #12838
[audio] Adding tests for predictive models by @anteju :: PR: #12823
Update resiliency example notebook readme and add links to the brev launchable by @ShriyaRishab :: PR: #12843
[automodel] qlora peft by @yzhang123 :: PR: #12817
ci: Increase prune time by @ko3n1g :: PR: #12860
Update base container in Dockerfile.speech by @artbataev :: PR: #12859
Fix qwen2.5 1.5b configuration inheritance bug by @Aprilistic :: PR: #12852
Update modelopt upperbound to 0.27 by @thomasdhc :: PR: #12788
Non-blocking checkpoint cleanup failure by @jstjohn :: PR: #12804
Improve evo2 dataset test and testability by @jstjohn :: PR: #12857
Expand test converage neva / mllama by @yaoyu-33 :: PR: #12715
Weekly bump by @ko3n1g :: PR: #12891
ci: Optional_L2_NeMo_2_SSM_Finetuning by @ko3n1g :: PR: #12893
docs: Update guide to PEP508 by @ko3n1g :: PR: #12890
Replace lm-eval with nvidia-lm-eval by @chtruong814 :: PR: #12888
Handle CUDA_DEVICE_MAX_CONNECTIONS before job launch by @guyueh1 :: PR: #12833
add nemotron5 by @JRD971000 :: PR: #12660
Bump vllm 0.8.2 by @Laplasjan107 :: PR: #12753
DeepseekV3 SFT finetuning perf config by @gdengk :: PR: #12829
add apply_chat_template method to TokenizerSpec + AutoTokenizer by @akoumpa :: PR: #12878
add accelerate to dependencies by @akoumpa :: PR: #12871
[automodel] Add FSDPv2-compatible context parallelism support. by @cspades :: PR: #12821
[fault tolerance] Add local checkpointing support by @ananthsub :: PR: #12839
ci: Bump release-freeze by @ko3n1g :: PR: #12914
ci: Use PAT for code-freeze by @ko3n1g :: PR: #12915
ci: Use correct environment by @ko3n1g :: PR: #12917
Freeze tags in in r2.3.0 by @github-actions[bot] :: PR: #12919
chore: Bump version to 2.3.0.rc2 by @chtruong814 :: PR: #12920
Version bump to 2.3.0rc3.dev0 by @github-actions[bot] :: PR: #12921
Cherry pick [automodel] Add linear ce loss support (12825) into r2.3.0 by @ko3n1g :: PR: #12922
Cherry pick DeepSeek V3 Multi Token Prediction (12550) into r2.3.0 by @ko3n1g :: PR: #12928
Cherry pick Set L2_NeMo_2_EVAL test to be optional (12949) into r2.3.0 by @ko3n1g :: PR: #12951
Cherry pick GB200 LLM performance scripts tuning (12791) into r2.3.0 by @ko3n1g :: PR: #12923
Cherry pick Allow configuration of PP communication backend to UCC in nemo2 (11755) into r2.3.0 by @ko3n1g :: PR: #12946
Cherry pick guard bitsandbytes based on cuda availability (12937) into r2.3.0 by @ko3n1g :: PR: #12958
Cherry pick Hugging Face model deployment support (12628) into r2.3.0 by @ko3n1g :: PR: #12962
Cherry pick fix macro-acc for pair-audio eval (12908) into r2.3.0 by @ko3n1g :: PR: #12963
Cherry pick Add energon dataset support for Qwen2VL (12831) into r2.3.0 by @ko3n1g :: PR: #12966
Cherry pick Make TETransformerLayerAutocast Support Cuda Graph (12075) into r2.3.0 by @ko3n1g :: PR: #12967
Cherry pick Use nvidia-lm-eval for evaluation (12902) into r2.3.0 by @ko3n1g :: PR: #12971
Cherry pick [NeMo 2.0] Interface for using MXFP8 and FP8 current scaling recipes (12503) into r2.3.0 by @ko3n1g :: PR: #12974
Cherry pick Fix trtllm and lightning conflict (12943) into r2.3.0 by @ko3n1g :: PR: #12981
Cherry pick Update v3 finetuning recipe (12950) and Specify PP first/last in strategy (12992) into r2.3.0 by @ko3n1g :: PR: #12984
Cherry pick Resolve an issue in custom megatron FSDP config setting (12948) into r2.3.0 by @ko3n1g :: PR: #12987
Cherry pick Remove getattr_proxy to avoid problematic edge cases (12176) into r2.3.0 by @ko3n1g :: PR: #12990
Cherry pick Enable async requests for in-fw deployment with OAI compatible server (12980) into r2.3.0 by @ko3n1g :: PR: #12994
Cherry pick initialize model with metadata (12496) into r2.3.0 by @ko3n1g :: PR: #12997
Cherry pick Bugfix for logits support for hf deployment (12965) into r2.3.0 by @ko3n1g :: PR: #13001
Cherry pick Update nvidia-resiliency-ext to be >= 0.3.0 (12925) into r2.3.0 by @ko3n1g :: PR: #13000
Cherry-pick Fix params_dtype for distillation and GPT HF Exporter head_dim for pruning to r2.3.0 by @kevalmorabia97 :: PR: #13002
Install nvidia-pytriton on arm (#13011) by @thomasdhc :: PR: #13013
Version bump to 2.3.0rc4.dev0 by @github-actions[bot] :: PR: #13041
Cherry pick Alit/nemotron h (12942) into r2.3.0 by @ko3n1g :: PR: #13007
Cherry pick [Automodel] Add TP/SP support with default llama-like sharding plan (12796) into r2.3.0 by @ko3n1g :: PR: #13017
Cherry pick Add initial docs broken link check (12977) into r2.3.0 by @ko3n1g :: PR: #13045
Cherry pick Fix MoE Init to not use Bias in test_strategy_lib.py (13009) into r2.3.0 by @ko3n1g :: PR: #13014
Cherry pick cleaner tflops log name (13005) into r2.3.0 by @ko3n1g :: PR: #13024
Cherry pick Improve t5 test coverage (12803) into r2.3.0 by @ko3n1g :: PR: #13025
Cherry pick put the warning on the right place (12909) into r2.3.0 by @ko3n1g :: PR: #13035
Cherry pick Temporary disable CUDA graphs in DDP mode for transducer decoding (12907) into r2.3.0 by @ko3n1g :: PR: #13036
Cherry pick [automodel] peft fix vlm (13010) into r2.3.0 by @ko3n1g :: PR: #13037
Cherry pick Only run the docs link check on the container (13068) into r2.3.0 by @ko3n1g :: PR: #13070
Cherry pick Add fp8 recipe option to perf script (13032) into r2.3.0 by @ko3n1g :: PR: #13055
Cherry pick Unified ptq export (12786) into r2.3.0 by @ko3n1g :: PR: #13062
Cherry pick Fix VP list index out of range from Custom FSDP (13021) into r2.3.0 by @ko3n1g :: PR: #13077
Cherry pick Add logging to cancel out PTL's warning about dataloader not being resumable (13072) into r2.3.0 by @ko3n1g :: PR: #13100
Cherry pick Fix long sequence generation after new arg introduced in mcore engine (13049) into r2.3.0 by @ko3n1g :: PR: #13104
Cherry pick Support Mamba models quantization (12631) into r2.3.0 by @ko3n1g :: PR: #13105
Cherry pick Add track_io to user buffer configs (13071) into r2.3.0 by @ko3n1g :: PR: #13111
ci: Onboard 8-GPU runner (#13115) by @ko3n1g :: PR: #13121
Cherry pick Add fine-tuning dataset function for FineWeb-Edu and update automodel… (13027) into r2.3.0 by @ko3n1g :: PR: #13118
Cherry pick Re-add sox to asr requirements (13092) into r2.3.0 by @ko3n1g :: PR: #13120
Cherry pick Update Mllama cross attn signature to match update MCore (13048) into r2.3.0 by @ko3n1g :: PR: #13122
Cherry pick Fix Exporter for baichuan and chatglm (13095) into r2.3.0 by @ko3n1g :: PR: #13126
ci: Faster builds (#13142) by @ko3n1g :: PR: #13144
Version bump to 2.3.0rc5.dev0 by @github-actions[bot] :: PR: #13146
ci: Fix mcore install in test container (#13152) by @ko3n1g :: PR: #13159
ci: Fix race-condition of container setup (#13162) by @ko3n1g :: PR: #13163
Cherry pick Guard decord and triton import (12861) into r2.3.0 by @ko3n1g :: PR: #13132
Cherry pick Bump TE version and apply patch (13087) into r2.3.0 by @ko3n1g :: PR: #13139
Cherry pick Update Llama-Minitron pruning-distillation notebooks from NeMo1 to NeMo2 + NeMoRun (12968) into r2.3.0 by @ko3n1g :: PR: #13141
Cherry pick Export and Deploy Tests (13076) into r2.3.0 by @ko3n1g :: PR: #13150
Cherry pick ub fp8 h100 fixes (13131) into r2.3.0 by @ko3n1g :: PR: #13153
Cherry pick Fix Transducer Decoding with CUDA Graphs in DDP with Mixed Precision (12938) into r2.3.0 by @ko3n1g :: PR: #13154
Cherry pick build: Pin modelopt (13029) into r2.3.0 by @chtruong814 :: PR: #13170
Cherry pick add fixes for nemotron-h (13073) into r2.3.0 by @JRD971000 :: PR: #13165
Add dsv3 pretrain script, support flops calculation (previous #12947) by @guyueh1 :: PR: #13186
ci: Allow running CI on weekly bump branch by @ko3n1g :: PR: #13233
Cherry pick Add Llama Nemotron Super/Ultra models (13044) into r2.3.0 by @ko3n1g :: PR: #13212
Cherry pick Add Blockwise FP8 to PTQ & EP to modelopt resume (12670) into r2.3.0 by @ko3n1g :: PR: #13239
Cherry pick [OAI Serving] Validate greedy generation args (redo) (13216) into r2.3.0 by @ko3n1g :: PR: #13242
Cherry pick drop sample_alpha in speechlm (13208) into r2.3.0 by @ko3n1g :: PR: #13246
Cherry pick [Eval bugfix] Move global eval-related imports inside the evaluate function (13166) into r2.3.0 by @ko3n1g :: PR: #13249
Cherry pick [Eval bugfix] Change default val of parallel_requests in eval script (13247) into r2.3.0 by @ko3n1g :: PR: #13253
Cherry pick Add tutorial for evaluation with Evals Factory (13259) into r2.3.0 by @ko3n1g :: PR: #13271
Cherry pick Fix default token durations (13168) into r2.3.0 by @ko3n1g :: PR: #13261
Cherry pick [Evaluation] Add support for nvidia-lm-eval==25.04 (13230) into r2.3.0 by @ko3n1g :: PR: #13274
Cherry pick [bug fix] set inference max seq len in inference context (13245) into r2.3.0 by @ko3n1g :: PR: #13276
Cherry pick More export and deploy unit tests (13178) into r2.3.0 by @ko3n1g :: PR: #13283
Cherry pick Reopen 13040 (13199) into r2.3.0 by @ko3n1g :: PR: #13303
Cherry pick Fix nemo1's neva notebook (13218) into r2.3.0 by @ko3n1g :: PR: #13312
Cherry pick build: various bumps (13285) into r2.3.0 by @ko3n1g :: PR: #13313
Cherry-pick ci: Increase cache pool into r2.3.0 by @chtruong814 :: PR: #13317
Cherry pick update num nodes in deepseek v3 finetune recipe (13314) into r2.3.0 by @ko3n1g :: PR: #13316
Cherry pick Fix neva notebook (13334) into r2.3.0 by @ko3n1g :: PR: #13335
Cherry-pick Add Llama4 Scout and Maverick Support (#12898) by @ko3n1g :: PR: #13331
Cherry pick Fix handling Llama Embedding dimensions param and prompt type in the ONNX export tutorial (13262) into r2.3.0 by @ko3n1g :: PR: #13326
Cherry-pick Fix transformer offline for CI/CD llama4 tests (#13339) to r2.3.0 by @chtruong814 :: PR: #13340
Fix llama4 test names by @chtruong814 :: PR: #13358
Cherry pick vLLM==0.8.5 update (13350) into r2.3.0 by @ko3n1g :: PR: #13354
Cherry-pick a test and doc fix to r2.3.0 by @chtruong814 :: PR: #13338
Cherry pick Add llama4 training recipe (12952) into r2.3.0 by @ko3n1g :: PR: #13386

</details>

Open source →

2.2.012 Mar 2025Deprecated

Remove deprecated tests/infer_data_path.py by @janekl :: PR: #11997

Highlights

Training
- Blackwell and Grace Blackwell support
- Pipeline parallel support for distillation
- Improved NeMo Framework installation
Export & Deploy
- vLLM export for NeMo 2.0
Evaluations
- Integrate lm-eval-harness
Collections
- LLM
  - DAPT Example and best practices in nemo 2.0
  - [NeMo 2.0] Enable Tool Learning and add a tutorial
  - Support GPT Embedding Model (Llama 3.2 1B/3B)
  - Qwen2.5, Phi4 (via AutoModel)
  - SFT for Llama 3.3 model (via AutoModel)
  - Support BERT Embedding Model with NeMo 2.0
  - DeepSeek SFT & PEFT Support
- MultiModal
  - Clip
  - SP for NeVA
  - CP for NeVA
  - Intern-VIT
Automodel
- Preview release.
- PEFT and SFT support for LLMs available via Hugging Face’s AutoModelForCausalLM.
- Support for Hugging Face-native checkpoints (full model and adapter only).
- Support for distributed training via DDP and FSDP2.
ASR/TTS
- Lhotse: TPS-free 2D bucket estimation and filtering
- Update model outputs to make all asr outputs to be in consistent format
- Sortformer Release Model

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

removed the line which caused a problem in nfa_tutorial by @Ssofja :: PR: #11710
TPS-free 2D bucket estimation and filtering by @pzelasko :: PR: #11738
Update transcribe_utils.py by @stevehuang52 :: PR: #11984
Sortformer Diarizer 4spk v1 model PR Part 4: Sortformer Documents and Notebook Tutorials by @tango4j :: PR: #11707
fix the issue during batched inference of Sortformer diarizer by @tango4j :: PR: #12047
changed asr models outputs to be consistent by @Ssofja :: PR: #11818
chore: Update notebooks by @ko3n1g :: PR: #12161
add ctc segmentation by @ko3n1g :: PR: #12312
clean up VAD tutorial by @stevehuang52 :: PR: #12410
copy from main by @nithinraok :: PR: #12423
ci: Disable ASR tests for now (#12443) by @ko3n1g :: PR: #12466
ASR_CTC_Language_Finetuning.ipynb bugfix by @lilithgrigoryan :: PR: #12538

</details>

TTS

<details><summary>Changelog</summary>

Add New Transformer Backbone for TTS Models by @blisc :: PR: #11911
changed asr models outputs to be consistent by @Ssofja :: PR: #11818
chore: Update notebooks by @ko3n1g :: PR: #12161

</details>

NLP / NMT

<details><summary>Changelog</summary>

Use explicit imports from megatronllm_deployable.py by @janekl :: PR: #11705
Bug fix minor bug in TRT-LLM deployment by @oyilmaz-nvidia :: PR: #11714
gpt moe perf scripts by @malay-nagda :: PR: #11760
Bump mcore by @ko3n1g :: PR: #11740
Enable packed seqs for validation by @jiemingz :: PR: #11748
Revert Mcore update since it caused regression by @pablo-garay :: PR: #11791
Fix Gemma2 Attention Init Args by @suiyoubi :: PR: #11792
Add null tokenizer by @erhoo82 :: PR: #11789
Fix DistCP inference issue by @suiyoubi :: PR: #11801
Add BERT Embedding Models E5 Recipe by @suiyoubi :: PR: #11787
Add rope scaling configs for NeMo 1 by @BoxiangW :: PR: #11807
Fix calculating num_available_samples by @huvunvidia :: PR: #11830
fix sentencepiece tokenizer special tokens by @akoumpa :: PR: #11811
add chat sft dataset to support agent tool calling by @chenrui17 :: PR: #11759
Revert "Revert Mcore update since it caused regression (#11791)" by @ko3n1g :: PR: #11799
fix checkpoint load issue by @dimapihtar :: PR: #11859
Fix nemo 1 packed sequence TE version error by @cuichenx :: PR: #11874
enable loading older TE checkpoints by @dimapihtar :: PR: #11930
ci: Use single runner machines for unit tests by @ko3n1g :: PR: #11937
llm performance scripts by @malay-nagda :: PR: #11736
[MoE] add expert tensor parallelism support for NeMo2.0 MoE by @gdengk :: PR: #11880
add exception when loading ckpt saved by TE < 1.13 by @dimapihtar :: PR: #11988
remove renormalize_blend_weights flag by @dimapihtar :: PR: #11975
Llama3.2 1B Embedding Model Support by @suiyoubi :: PR: #11909
Weekly bump by @ko3n1g :: PR: #11896
Debug Apex distributed optimizer to handle Transformer Engine 2.0 by @timmoon10 :: PR: #12004
throw MegatronOptimizerModule warning only with mcore models by @akoumpa :: PR: #12085
fix nmt dataclass issue by @dimapihtar :: PR: #12081
Propogate dp last changes from mcore by @ryantwolf :: PR: #12012
Add error message when downloading failed. by @yuanzhedong :: PR: #12139
interface for asymmetric pipeline schedule by @erhoo82 :: PR: #12039
chore: Update notebooks by @ko3n1g :: PR: #12161
Cherrypick #12382, #12415 and #12424 by @cuichenx :: PR: #12425
ASR_CTC_Language_Finetuning.ipynb bugfix by @lilithgrigoryan :: PR: #12538

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

surface attn_implementation option by @akoumpa :: PR: #11873
attn_implementation eager fallback by @akoumpa :: PR: #12060

</details>

NeMo Tools

<details><summary>Changelog</summary>

build: Add sox to SDE by @ko3n1g :: PR: #11882
add ctc segmentation by @ko3n1g :: PR: #12312

</details>

Export

<details><summary>Changelog</summary>

Bug fix minor bug in TRT-LLM deployment by @oyilmaz-nvidia :: PR: #11714
In-framework deployment NeMo 2.0 nemo_export.py test by @janekl :: PR: #11749
Fix starcoder2 missing bias in nemo2 config for TRTLLM by @meatybobby :: PR: #11809
Autodetect dtype on exporting to TensorRT-LLM by @janekl :: PR: #11907
PTQ & TRT-LLM updates related to upcoming PyTorch 25.01 bump by @janekl :: PR: #11941
Run Flake8 for nemo.export module by @janekl :: PR: #11728
Skip initialization in hf export by @cuichenx :: PR: #12136
update export io call by @akoumpa :: PR: #12144
add default kwargs for trtllm model runner by @pablo-garay :: PR: #12248
cherry-pick: fix[export]: reshard model correctly handles extra_state when it's a tensor (#12132) by @terrykong :: PR: #12335

</details>

Bugfixes

<details><summary>Changelog</summary>

added required instalation for sox to process mp3 file by @Ssofja :: PR: #11709
removed the line which caused a problem in nfa_tutorial by @Ssofja :: PR: #11710
Bug fix minor bug in TRT-LLM deployment by @oyilmaz-nvidia :: PR: #11714

</details>

Uncategorized:

<details><summary>Changelog</summary>

Allow using vocab size from config by @shanmugamr1992 :: PR: #11718
Fix baseline recipes by @erhoo82 :: PR: #11725
Update changelog for r2.1.0 by @github-actions[bot] :: PR: #11745
ci: Fix changelog generator by @ko3n1g :: PR: #11744
Fix 'http_port' parameter name in DeployPyTriton usages and update .qnemo compress=True path by @janekl :: PR: #11747
Conversion NeMo and HF checkpoint script for T5 by @huvunvidia :: PR: #11739
Add BERT Embedding Models by @suiyoubi :: PR: #11737
Add server ready check before starting evaluation by @athitten :: PR: #11731
only install bitsandbytes on x86 by @akoumpa :: PR: #11781
[Bugfix] Skip processing if extra_state loads as None by @janekl :: PR: #11778
chore(beep boop 🤖): Bump MCORE_TAG=4dc8977... (2025-01-07) by @ko3n1g :: PR: #11768
make progress printer compatible with PTL v2.5.0 by @ashors1 :: PR: #11779
Fix Mistral Conversion Issue by @suiyoubi :: PR: #11786
build: Fix build-arg by @ko3n1g :: PR: #11815
Lora ckpt in HF format for NeMo AutoModel by @oyilmaz-nvidia :: PR: #11712
8x22b seq len by @malay-nagda :: PR: #11788
Bugfix for output_generation_logits in tensorrtllm by @athitten :: PR: #11820
handle mistralai/Mistral-7B-Instruct-v0.3 tokenizer correctly by @akoumpa :: PR: #11839
remove tensorstore pin in requirements*.txt by @pstjohn :: PR: #11777
Do not load context for model transform in llm inference by @hemildesai :: PR: #11751
update nemo2sftpeft tutorial container verison by @HuiyingLi :: PR: #11832
Latest News updated for Cosmos by @lbliii :: PR: #11806
Removes tensorstore 0.1.45 pin from requirements_deploy.txt by @pstjohn :: PR: #11858
ci: Prune dangling images by @ko3n1g :: PR: #11885
Disable tests that download datasets from web by @akoumpa :: PR: #11878
Add context_logits for eval accuracy calculation in case of multi token prediction tasks by @athitten :: PR: #11753
add dataset_root to SpecterDataModule by @suiyoubi :: PR: #11837
Support both Path and str for APIs by @maanug-nv :: PR: #11865
Run nsys callback on GBS not on MBS by @akoumpa :: PR: #11861
ci: Set bump-branch to weekly by @ko3n1g :: PR: #11889
chore: Update mcore-tag-bump-bot.yml by @ko3n1g :: PR: #11891
ci: Bump Mcore in weekly PR by @ko3n1g :: PR: #11897
check restore_config first by @akoumpa :: PR: #11890
LinearAdapter: propagate args to _init_adapter by @akoumpa :: PR: #11902
NeMo 2.0 fp8 conversion by @Laplasjan107 :: PR: #11845
nemo ux expert tensor parallel by @akoumpa :: PR: #11903
Add CP support to Neva in NeMo2 by @yaoyu-33 :: PR: #11850
build: Move dependencies by @ko3n1g :: PR: #11790
Add Flux and Flux Controlnet Support to Diffusion folder by @Victor49152 :: PR: #11794
ci: Adjust bump mcore workflow by @ko3n1g :: PR: #11918
ci: Small fix to bump workflow by @ko3n1g :: PR: #11919
Revert #11890 and add a test that would have caught the error by @cuichenx :: PR: #11914
ci: Adjust input argument by @ko3n1g :: PR: #11921
Create test_phi3.py by @mayani-nv :: PR: #11843
Enable NeMo importer and loading dist CKPT for training by @Victor49152 :: PR: #11927
build: Pin triton by @ko3n1g :: PR: #11938
Add sharding for speechlm and vlm by @BoxiangW :: PR: #11876
Update torch load for load from disk by @thomasdhc :: PR: #11963
Add options to add mp_policy and parallel_fn for NeMo automodel fsdp2 by @BoxiangW :: PR: #11956
ci: Add coverage reports by @ko3n1g :: PR: #11912
Add batching support for evaluation by @athitten :: PR: #11934
add use_fast option by @akoumpa :: PR: #11976
improve error and debug messages in model connector by @cuichenx :: PR: #11979
[checkpoint][docs] Fix typos in dist checkpointing docs by @ananthsub :: PR: #11983
callbacks and bf16 grad by @malay-nagda :: PR: #11985
remove --disable-ckpt from tests by @akoumpa :: PR: #11996
nemo automodel sft squad data prep fix by @akoumpa :: PR: #11994
Introduce evaluation API by @Glorf :: PR: #11895
Remove deprecated tests/infer_data_path.py by @janekl :: PR: #11997
Checkpoint saving for automodels via ModelCheckpoint by @akoumpa :: PR: #11998
Mask vocab padding token ids from CE loss by @maanug-nv :: PR: #11999
Add the NeMo2 memory profiling plugin by @gdengk :: PR: #12009
chore(ci): Disable VMs cron job on forks by @mikemckiernan :: PR: #12020
Adding speechlm AutoModel test by @oyilmaz-nvidia :: PR: #11990
minor fix and simplify by @akoumpa :: PR: #12007
ci: Build wheel workflow by @ko3n1g :: PR: #12021
ci: Release workflow by @ko3n1g :: PR: #12022
Version bump to 2.2.0rc1 by @github-actions[bot] :: PR: #12023
ci: Run unit tests on main by @ko3n1g :: PR: #11986
[Audio] Fix extra step in Euler sampler for flow matching inference by @racoiaws :: PR: #11989
Set zarr range to >=2.18.2 and <3.0.0 by @chtruong814 :: PR: #12005
ci: Run linting per domain by @ko3n1g :: PR: #12027
Replace reference of requirements_infer.txt with requirements_deploy.txt by @chtruong814 :: PR: #12029
ci: Always run linting by @ko3n1g :: PR: #12035
ci: Retry on timeout by @ko3n1g :: PR: #11974
[MoE] fix run err in mixtral22B recipe and update its perf config by @gdengk :: PR: #12036
Version bump to 2.2.0rc2.dev0 by @github-actions[bot] :: PR: #12040
ci: Update weekly brain by @ko3n1g :: PR: #12043
ci: Update workflow by @ko3n1g :: PR: #12044
nemo-automodel: fsdp2 support for peft by @akoumpa :: PR: #12008
fix llama-3.1 hf model_id by @AtsunoriFujita :: PR: #11774
Clip Model in Nemo2 by @abhinavg4 :: PR: #11980
Adding TFLOPs callback for Multimodal models and NeVA calculator by @parthmannan :: PR: #11969
ci: Allow skipping docs by @ko3n1g :: PR: #12048
avoid missmatch error when loading older TE checkpoints by @dimapihtar :: PR: #12028
Add padding in mllama vision encoder to align with HF by @meatybobby :: PR: #11808
chore: Add warning for rebase by @ko3n1g :: PR: #12061
ci: Lint Python files only by @ko3n1g :: PR: #12064
Recipe changes for performance by @guyueh1 :: PR: #11763
Pipeline-parallel support for Knowledge Distillation (NeMo 2) by @AAnoosheh :: PR: #11766
add cp_comm_type param to Mistral config by @dimapihtar :: PR: #12049
Conformer-based spectrogram estimator by @anteju :: PR: #12002
Adding nemo CI by @abhinavg4 :: PR: #12052
Update optimization features readme from nemo1 to nemo2 by @yaoyu-33 :: PR: #12071
Add Llama Embedding Tutorial by @suiyoubi :: PR: #12042
Fix Linting by @suiyoubi :: PR: #12079
Fix hf_dataset bug by @BoxiangW :: PR: #12072
set TOKENIZERS_PARALLELISM=True by @akoumpa :: PR: #12083
minor fix in model's summary identation during logging by @akoumpa :: PR: #12084
Refactor VLM modules / Add InternVit submodule support by @yaoyu-33 :: PR: #11851
Fix SBERT with sequence_len_offset by @suiyoubi :: PR: #12057
ci: codecov by @ko3n1g :: PR: #12030
build: Improve installer by @ko3n1g :: PR: #12016
ci: Modular unit tests by @ko3n1g :: PR: #12104
ci: Update bump workflow by @ko3n1g :: PR: #12106
etp docs by @akoumpa :: PR: #12111
build: Better caching by @ko3n1g :: PR: #12109
ci: Fix flaky test by @ko3n1g :: PR: #12113
Ensure nemo.collections.vlm does not strictly require transformer engine by @chtruong814 :: PR: #12108
build: Optimize by @ko3n1g :: PR: #12112
refactor peft module matching; introduce exclude_modules by @akoumpa :: PR: #12066
Update mcore commit (02.06.25) by @pablo-garay :: PR: #12114
ci: Bump Mcore inplace by @ko3n1g :: PR: #12115
ci: Bump bot by @ko3n1g :: PR: #12117
Add neva pretrain script by @yaoyu-33 :: PR: #12033
DAPT playbooks - with NeMo 2.0 by @jvamaraju :: PR: #12067
Malay/bw scripts by @malay-nagda :: PR: #11961
[MoE] Add type annotation for mixtral configs by @gdengk :: PR: #12126
ci: Disable checks by @ko3n1g :: PR: #12129
Add performance-optimized example for llama2 70b LoRA by @vysarge :: PR: #12055
Add Automodel support for Deepseek v3 model by @BoxiangW :: PR: #12099
Bug fix with generation of expert_tensor_parallel_rank by @guyueh1 :: PR: #12125
Rename neva datamodule by @yaoyu-33 :: PR: #12121
Update vLLM to 0.7.2 by @Laplasjan107 :: PR: #12078
Prevent downloading dataset every time in ci test by @cuichenx :: PR: #12095
AudioToAudioModel: fix model->dataloader sample_rate parameter injection by @racoiaws :: PR: #12092
Minor Bug Fixes - LLaMa Embedding by @soluwalana :: PR: #12146
build: Force re-install VCS dependencies by @ko3n1g :: PR: #12155
Cherry pick build: Force re-install VCS dependencies (12155) into r2.2.0 by @ko3n1g :: PR: #12191
Cherry pick Add function calling SFT NeMo2.0 tutorial (11868) into r2.2.0 by @ko3n1g :: PR: #12180
Cherry pick Update TTS code to remove calls to deprecated functions (12153) into r2.2.0 by @ko3n1g :: PR: #12201
Cherry pick Fix multi-GPU in-framework deployment (12090) into r2.2.0 by @ko3n1g :: PR: #12172
Cherry pick disable moe logging to avoid deepseek hang (12168) into r2.2.0 by @ko3n1g :: PR: #12192
Cherry pick build: Pin down transformers (12229) into r2.2.0 by @ko3n1g :: PR: #12230
Cherry pick Fix loading extra states from torch tensor (12185) into r2.2.0 by @ko3n1g :: PR: #12226
Cherry pick nemo-automodel checkpoint-io refactor (12070) into r2.2.0 by @ko3n1g :: PR: #12234
ci: Flaky tests release by @ko3n1g :: PR: #12293
Cherry pick Set L2_Speech_Batch_Size_OOMptimizer_Canary to be optional (12299) into r2.2.0 by @ko3n1g :: PR: #12300
build: Editable nemo install (#12304) by @ko3n1g :: PR: #12308
ci: Fix test workflow by @ko3n1g :: PR: #12311
Cherry pick build: Exclude tensorstore 0.1.72 (12317) into r2.2.0 by @ko3n1g :: PR: #12318
Cherry pick Fix the local path in Sortformer diarizer training tutorial (12135) into r2.2.0 by @ko3n1g :: PR: #12316
Cherry pick Add eval requirement to setup.py (12152) into r2.2.0 by @ko3n1g :: PR: #12277
Cherry pick Add modelopt to requirements_nlp.txt (12261) into r2.2.0 by @ko3n1g :: PR: #12278
cherry pick 12209 by @akoumpa :: PR: #12240
Cherry pick Energon ckpt multimodal (12245) into r2.2.0 by @ko3n1g :: PR: #12307
Cherry pick [nemo1] Fix Mamba/Bert loading from checkpoint after TE extra states were introduced (12275) into r2.2.0 by @ko3n1g :: PR: #12314
Cherry pick fix masked loss calculation (12255) into r2.2.0 by @ko3n1g :: PR: #12286
chore: Cherry pick deepseek by @ko3n1g :: PR: #12324
build: Bump PyT to 25.01 (#11973) by @ko3n1g :: PR: #12323
Cherry pick build: Bump mcore (12320) into r2.2.0 by @ko3n1g :: PR: #12328
Cherry pick [automodel] re-enable FSDP2 tests (12325) into r2.2.0 by @ko3n1g :: PR: #12331
Cherry pick [automodel] fix loss reporting (12303) into r2.2.0 by @ko3n1g :: PR: #12334
build: Bump Mcore by @ko3n1g :: PR: #12340
Cherry-pick Asr fixes 2.2 (#12227) by @ko3n1g :: PR: #12345
Cherry-pick Bug fixes (#12315) by @chtruong814 :: PR: #12346
Cherry pick [automodel] remove fix_progress_bar from fsdp2 strategy (12339) into r2.2.0 by @ko3n1g :: PR: #12347
Cherry pick Fix NeMo1 Bert Embedding Dataset args (12341) into r2.2.0 by @ko3n1g :: PR: #12349
Cherry pick Fix NeMo1 sequence_len_offset in Bert fwd (12350) into r2.2.0 by @ko3n1g :: PR: #12359
Cherry pick Add nemo-run recipe for evaluation (12301) into r2.2.0 by @ko3n1g :: PR: #12352
Cherry pick Add DeepSeek-R1 Distillation NeMo 2.0 tutorial (12187) into r2.2.0 by @ko3n1g :: PR: #12355
chore: Update package_info.py by @ko3n1g :: PR: #12362
Version bump to 2.2.0rc4.dev0 by @github-actions[bot] :: PR: #12363
Bump mcore to latest commit on release branch by @chtruong814 :: PR: #12360
Cherry pick [automodel] add lr scheduler (12351) into r2.2.0 by @ko3n1g :: PR: #12361
Cherry pick [automodel] add distributed data sampler (12326) into r2.2.0 by @ko3n1g :: PR: #12373
Cherry pick [NeVA] Fix for CP+THD (12366) into r2.2.0 by @ko3n1g :: PR: #12375
Cherry pick Ignore attribute error when serializing mcore specs (12353) into r2.2.0 by @ko3n1g :: PR: #12383
Cherry pick Avoid init_ddp for inference (12011) into r2.2.0 by @ko3n1g :: PR: #12385
Cherry pick [docs] fix notebook render (12374) into r2.2.0 by @ko3n1g :: PR: #12394
Cherry pick Neva finetune scripts and PP fix (12387) into r2.2.0 by @ko3n1g :: PR: #12397
Cherry pick [automodel] update runner tags for notebooks (12428) into r2.2.0 by @ko3n1g :: PR: #12431
Cherry pick [automodel] update examples (12411) into r2.2.0 by @ko3n1g :: PR: #12432
Cherry pick Evaluation docs (12348) into r2.2.0 by @ko3n1g :: PR: #12460
Cherry pick Update prompt format (12452) into r2.2.0 by @ko3n1g :: PR: #12455
Cherry pick Fixing a wrong Sortformer Tutorial Notebook path. (12479) into r2.2.0 by @ko3n1g :: PR: #12480
Cherry pick added a needed checks and changes for bugfix (12400) into r2.2.0 by @Ssofja :: PR: #12447
Cherry pick [automodel] fix loss/tps reporting across ranks (12389) into r2.2.0 by @ko3n1g :: PR: #12413
Cherry pick enable fsdp flag for FSDP2Strategy (12392) into r2.2.0 by @ko3n1g :: PR: #12429
Cherry pick Fix lita notebook issue (12474) into r2.2.0 by @ko3n1g :: PR: #12476
Cherrypick multinode tut changes by @BoxiangW :: PR: #12501
Cherry pick Changed the argument types passed to metrics calculation functions (12500) into r2.2.0 by @ko3n1g :: PR: #12502
Cherry pick added needed fixes (12495) into r2.2.0 by @ko3n1g :: PR: #12509
Cherry pick update transformers version requirements (12475) into r2.2.0 by @ko3n1g :: PR: #12507
Cherry pick [checkpoint] Log timings for checkpoint IO save and load (11972) into r2.2.0 by @ko3n1g :: PR: #12520
Cherry pick few checkings needed because of the change of asr models output (12499) into r2.2.0 by @ko3n1g :: PR: #12513
Oyilmaz nvidia/chore/cherry pick 12242 by @oyilmaz-nvidia :: PR: #12523
Cherry pick Remove _attn_implementationinLlamaBidirectionalModel constructor (12364) into r2.2.0 by @ko3n1g :: PR: #12525
Cherry pick Configure FSDP to keep module params (12074) into r2.2.0 by @ko3n1g :: PR: #12524
Cherry pick [automodel] docs (11942) into r2.2.0 by @ko3n1g :: PR: #12530
Cherry pick [automodel] update examples' comments (12518) and [automodel] Move PEFT to configure_model (#12491) into r2.2.0 by @ko3n1g :: PR: #12529
Cherry pick update readme to include latest pytorch version (12539) into r2.2.0 by @ko3n1g :: PR: #12577
Publish r2.2.0 by @chtruong814 :: PR: #12583

</details>

Open source →

2.1.03 Jan 2025Deprecated

Added deprecation notice by @Ssofja :: PR: #11133

Highlights

Training
- Fault Tolerance
  - Straggler Detection
  - Auto Relaunch
LLM & MM
- MM models
  - Llava-next
  - Llama 3.2
- Sequence Model Parallel for NeVa
- Enable Energon
- SigLIP (NeMo 1.0 only)
- LLM 2.0 migration
  - Starcoder2
  - Gemma 2
  - T5
  - Baichuan
  - BERT
  - Mamba
  - ChatGLM
- DoRA support
Export
- Nemo 2.0 base model export path for NIM
- PTQ in Nemo 2.0
ASR
- Timestamps with TDT decoder
- Timestamps option with .transcribe()

Detailed Changelogs:

ASR

<details><summary>Changelog</summary>

[Fix] Fixed sampler override and audio_key in prepare_audio_data by @anteju :: PR: #10980
Akoumparouli/mixtral recipe fix r2.0.0 by @akoumpa :: PR: #10994
TDT compute timestamps option and Extra Whitespace handling for SPE by @monica-sekoyan :: PR: #10875
ci: Switch to CPU only runner by @ko3n1g :: PR: #11035
Fix timestamps tests by @monica-sekoyan :: PR: #11053
ci: Pin release freeze by @ko3n1g :: PR: #11143
Fix RNN-T loss memory usage by @artbataev :: PR: #11144
Added deprecation notice by @Ssofja :: PR: #11133
Fixes for Canary adapters tutorial by @pzelasko :: PR: #11184
add ipython import guard by @nithinraok :: PR: #11191
Self Supervised Pre-Training tutorial Fix by @monica-sekoyan :: PR: #11206
update the return type by @nithinraok :: PR: #11210
Timestamps to transcribe by @nithinraok :: PR: #10950
[Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045
Beam search algorithm implementation for TDT models by @lilithgrigoryan :: PR: #10903
Update import 'pytorch_lightning' -> 'lightning.pytorch' by @maanug-nv :: PR: #11252
Remove pytorch-lightning by @maanug-nv :: PR: #11306
update hypothesis when passed through cfg by @nithinraok :: PR: #11366
Revert "update hypothesis when passed through cfg" by @pablo-garay :: PR: #11373
Fix transcribe speech by @nithinraok :: PR: #11379
Lhotse support for transcribe_speech_parallel by @nune-tadevosyan :: PR: #11249
Sortformer Diarizer 4spk v1 model PR Part 1: models, modules and dataloaders by @tango4j :: PR: #11282
Removing unnecessary lines by @nune-tadevosyan :: PR: #11408
Support for initializing lhotse shar dataloader via field: list[path] mapping by @pzelasko :: PR: #11460
New extended prompt format for Canary, short utterances inference fix, and training micro-optimizations by @pzelasko :: PR: #11058
Fixing Multi_Task_Adapters.ipynb by replacing canary2 with canary_custom by @weiqingw4ng :: PR: #11636

</details>

TTS

<details><summary>Changelog</summary>

[Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045
Add T5TTS by @blisc :: PR: #11193
Update import 'pytorch_lightning' -> 'lightning.pytorch' by @maanug-nv :: PR: #11252
Remove pytorch-lightning by @maanug-nv :: PR: #11306
Add nvidia/low-frame-rate-speech-codec-22khz model on docs by @Edresson :: PR: #11457

</details>

NLP / NMT

<details><summary>Changelog</summary>

Move collectiob.nlp imports inline for t5 by @marcromeyn :: PR: #10877
Use a context-manager when opening files by @akoumpa :: PR: #10895
Packed sequence bug fixes by @cuichenx :: PR: #10898
ckpt convert bug fixes by @dimapihtar :: PR: #10878
remove deprecated ci tests by @dimapihtar :: PR: #10922
Update T5 tokenizer (adding additional tokens to tokenizer config) by @huvunvidia :: PR: #10972
Add support and recipes for HF models via AutoModelForCausalLM by @akoumpa :: PR: #10962
gpt3 175b cli by @malay-nagda :: PR: #10985
Fix for crash with LoRA + tp_overlap_comm=false + sequence_parallel=true by @vysarge :: PR: #10920
Update BaseMegatronSampler for compatibility with PTL's _BatchProgress by @ashors1 :: PR: #11016
add deprecation note by @dimapihtar :: PR: #11024
Update ModelOpt Width Pruning example defaults by @kevalmorabia97 :: PR: #10902
switch to NeMo 2.0 recipes by @dimapihtar :: PR: #10948
NeMo 1.0: upcycle dense to moe by @akoumpa :: PR: #11002
Gemma2 in Nemo2 with Recipes by @suiyoubi :: PR: #11037
Add Packed Seq option to GPT based models by @suiyoubi :: PR: #11100
Fix MCoreGPTModel import in llm.gpt.model.base by @hemildesai :: PR: #11109
TP+MoE peft fix by @akoumpa :: PR: #11114
GPT recipes to use full te spec by @JimmyZhang12 :: PR: #11119
Virtual pipeline parallel support for LoRA in NLPAdapterModelMixin by @vysarge :: PR: #11128
update nemo args for mcore flash decode arg change by @HuiyingLi :: PR: #11138
Call ckpt_to_weights_subdir from MegatronCheckpointIO by @ashors1 :: PR: #10897
[Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045
fix(export): GPT models w/ bias=False convert properly by @terrykong :: PR: #11255
Use MegatronDataSampler in HfDatasetDataModule by @akoumpa :: PR: #11274
Add T5TTS by @blisc :: PR: #11193
ci: Exclude CPU machines from scan by @ko3n1g :: PR: #11300
Revert "fix(export): GPT models w/ bias=False convert properly" by @terrykong :: PR: #11301
remove redundant docs by @sharathts :: PR: #11302
Update import 'pytorch_lightning' -> 'lightning.pytorch' by @maanug-nv :: PR: #11252
Add attention_bias argument in transformer block and transformer layer modules, addressing change in MCore by @yaoyu-33 :: PR: #11289
Remove pytorch-lightning by @maanug-nv :: PR: #11306
Update T5 attention-mask shapes to be compatible with all attention-backend in new TE versions by @huvunvidia :: PR: #11059
Add support for restoring from 2.0 checkpoint in 1.0 by @hemildesai :: PR: #11347
Fix Gemma2 Attention Args by @suiyoubi :: PR: #11365
mlm conversion & tiktokenizer support by @dimapihtar :: PR: #11349
[Nemo1] Generate sharded optimizer state dicts only if needed for saving by @ananthsub :: PR: #11451
add hindi tn/itn coverage by @mgrafu :: PR: #11382
chore(beep boop 🤖): Bump MCORE_TAG=67a50f2... (2024-11-28) by @ko3n1g :: PR: #11427
Handle exception when importing RetroGPTChunkDatasets by @guyueh1 :: PR: #11415
Update restore from config for gpt type continual training in NeMo1 by @yaoyu-33 :: PR: #11471
ci: Re-enable L2_Megatron_LM_To_NeMo_Conversion by @ko3n1g :: PR: #11484
Apply packed sequence params change for fused rope compatibility by @ananthsub :: PR: #11506
Huvu/tiktoken tokenizer update by @huvunvidia :: PR: #11494

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Adding support for LightningDataModule inside Fabric-API by @marcromeyn :: PR: #10879
Add registry to register all needed classes with artifacts in nemo.lightning.io by @hemildesai :: PR: #10861
Update import 'pytorch_lightning' -> 'lightning.pytorch' by @maanug-nv :: PR: #11252
Remove pytorch-lightning by @maanug-nv :: PR: #11306
add hindi tn/itn coverage by @mgrafu :: PR: #11382

</details>

Export

<details><summary>Changelog</summary>

Update engine build step for TRT-LLM 0.13.0 by @janekl :: PR: #10880
Nemo 2.0 ckpt support in TRT-LLM export by @oyilmaz-nvidia :: PR: #10891
Fix TRTLLM parallel_embedding by @meatybobby :: PR: #10975
Export & deploy updates (part I) by @janekl :: PR: #10941
Add doc-strings to import & export + improve logging by @marcromeyn :: PR: #11078
NeMo-UX: fix nemo-ux export path by @akoumpa :: PR: #11081
Fix TRTLLM nemo2 activation parsing by @meatybobby :: PR: #11062
Support exporting Nemotron-340B for TensorRT-LLM by @jinyangyuan-nvidia :: PR: #11015
vLLM Hugging Face exporter by @oyilmaz-nvidia :: PR: #11124
Fix export of configuration parameters to Weights and Biases by @soluwalana :: PR: #10995
Change activation parsing in TRTLLM by @meatybobby :: PR: #11173
Remove builder_opt param from trtllm-build for TensorRT-LLM >= 0.14.0 by @janekl :: PR: #11259
fix(export): GPT models w/ bias=False convert properly by @terrykong :: PR: #11255
fix(export): update API for disabling device reassignment in TRTLLM for Aligner by @terrykong :: PR: #10863
Add openai-gelu in gated activation for TRTLLM export by @meatybobby :: PR: #11293
Revert "fix(export): GPT models w/ bias=False convert properly" by @terrykong :: PR: #11301
Adding alinger export by @shanmugamr1992 :: PR: #11269
Export & deploy updates (part II) by @janekl :: PR: #11344
Introducing TensorRT lazy export and caching option with trt_compile() by @borisfom :: PR: #11266
fix: export converts properly if no model_prefix by @terrykong :: PR: #11477

</details>

Bugfixes

<details><summary>Changelog</summary>

Change default ckpt name by @maanug-nv :: PR: #11277
Fix patching of NeMo tokenizers for correct Lambada evaluation by @janekl :: PR: #11326

</details>

Uncategorized:

<details><summary>Changelog</summary>

ci: Use Slack group by @ko3n1g :: PR: #10866
Bump Dockerfile.ci (2024-10-14) by @ko3n1g :: PR: #10871
Fix peft resume by @cuichenx :: PR: #10887
call post_init after altering config values by @akoumpa :: PR: #10885
Late import prettytable by @maanug-nv :: PR: #10912
Bump Dockerfile.ci (2024-10-17) by @ko3n1g :: PR: #10919
Warning for missing FP8 checkpoint support for vLLM deployment by @janekl :: PR: #10906
Fix artifact saving by @hemildesai :: PR: #10914
Lora improvement by @cuichenx :: PR: #10918
Huvu/t5 nemo2.0 peft by @huvunvidia :: PR: #10916
perf recipes and Mcore DistOpt params by @malay-nagda :: PR: #10883
ci: Fix cherry pick team by @ko3n1g :: PR: #10945
Fix requirements for MacOS by @artbataev :: PR: #10930
Fix nemo 2.0 recipes by @BoxiangW :: PR: #10915
Akoumparouli/nemo ux fix dir or string artifact by @akoumpa :: PR: #10936
Fix typo in docstring by @ashors1 :: PR: #10955
[Nemo CICD] Remove deprecated tests by @pablo-garay :: PR: #10960
Restore NeMo 2.0 T5 pretraining CICD test by @huvunvidia :: PR: #10952
Convert perf plugin env vars to strings by @hemildesai :: PR: #10947
disable dynamo for ddp checker by @akoumpa :: PR: #10961
Bump Dockerfile.ci (2024-10-21) by @ko3n1g :: PR: #10965
respect warnings' filters by @akoumpa :: PR: #10953
Alit/mamba recipe by @JRD971000 :: PR: #10935
Long context performance doc hot fix by @youngeunkwon0405 :: PR: #10946
Performance mode by @malay-nagda :: PR: #10926
Bump Dockerfile.ci (2024-10-22) by @ko3n1g :: PR: #10979
Add more recipes by @cuichenx :: PR: #10957
ci: Update tests by @ko3n1g :: PR: #10987
Bump Dockerfile.ci (2024-10-23) by @ko3n1g :: PR: #11001
llm.generate fixes by @HuiyingLi :: PR: #10983
use dict in check by @akoumpa :: PR: #11012
LoRA support for HF::AutoModelForCausalLM by @akoumpa :: PR: #10982
Change default for always_save_context to True by @athitten :: PR: #11014
Fix pip install by @marcromeyn :: PR: #11026
Change dist ckpt defaults by @ShriyaPalsamudram :: PR: #10913
Fix _strategy_lib tests by @maanug-nv :: PR: #11033
Basic online dynamic FP8 quantization with vLLM by @janekl :: PR: #10904
Expose packed seq in finetuning recipes by @cuichenx :: PR: #11006
PEFT Inference by @cuichenx :: PR: #11030
added Lhotse online augmentation tutorial for SE by @nasretdinovr :: PR: #10944
Bump Dockerfile.ci (2024-10-27) by @ko3n1g :: PR: #11051
ci: Send team alerts on specific keywords by @ko3n1g :: PR: #10986
Qwen2 Recipe by @suiyoubi :: PR: #10974
Bump Dockerfile.ci (2024-10-28) by @ko3n1g :: PR: #11054
Generalizing Inference pipeline in NeMo 2.0 to support encoder-decoder models by @huvunvidia :: PR: #10924
[Bug fix] In energon MultiModalSampleConfig use default_factory in dataclass by @guyueh1 :: PR: #11041
fix: Resolve mutable default issue in MultiModalSampleConfig dataclass by @michal2409 :: PR: #11061
SC1/SC2 Recipe by @suiyoubi :: PR: #10971
Wrap batch_sampler with _IndexBatchSamplerWrapper by @farhadrgh :: PR: #10934
Performance fine-tuning recipes for llama3 8b + 70b by @vysarge :: PR: #11046
Set TE spec name for NeMo to HF checkpoint converters by @kevalmorabia97 :: PR: #11036
ci: Re-add secrets detector by @ko3n1g :: PR: #11038
Adding nemo-run recipes for NeMo 2.0 T5 by @huvunvidia :: PR: #10964
Minor fixes for NeMo 2.0 PTQ by @Laplasjan107 :: PR: #11079
Add copyright check by @pablo-garay :: PR: #11048
Fix finalize model grad for PEFT by @cuichenx :: PR: #11065
ci: Less verbose infra alerts by @ko3n1g :: PR: #11080
Add copyright notice by @pablo-garay :: PR: #11085
ci: Fix cron schedule by @ko3n1g :: PR: #11076
ci: Use code-freeze via Nemo-FW-Templates by @ko3n1g :: PR: #11073
Akoumparouli/hf lit module peft ckpt bugfix by @akoumpa :: PR: #11022
PEFT perf and TE spec fixes by @JimmyZhang12 :: PR: #11070
Bump Dockerfile.ci (2024-10-30) by @ko3n1g :: PR: #11092
NeMorun for NeMo 2.0 T5 finetuning by @huvunvidia :: PR: #11040
fix model_checkpoint.py by @ethanhe42 :: PR: #11057
Update PTQ tests and ModelOpt version by @janekl :: PR: #11095
Fix datasets in CLI by @marcromeyn :: PR: #11097
Fix yaml serialization in io mixin by @hemildesai :: PR: #11106
disable overlap_param_gather_with_optimizer_step by @JimmyZhang12 :: PR: #11102
nemo1 to nemo2 checkpoint convert by @HuiyingLi :: PR: #10937
fix expert regex filter by @akoumpa :: PR: #11103
Remove stale checkpoint deletion on checkpoint saving failure by @akoumpa :: PR: #11116
NeMo-UX: Mistral/mixtral peft ci test by @akoumpa :: PR: #11094
Make nemo.collections.llm PreTrainingDataModule num samples configurable by @hemildesai :: PR: #11088
Fix packed seq path by @cuichenx :: PR: #11121
Allow arguments passed to dataset class + Gemma recipe fix by @cuichenx :: PR: #11125
Nemotron Recipe by @suiyoubi :: PR: #11118
NeMo-UX: HF PeFT fix by @akoumpa :: PR: #11096
Remove deprecated tests by @pablo-garay :: PR: #11134
Recipe Fix for NeMo CI by @suiyoubi :: PR: #11127
Fix freeze_model call in peft by @cuichenx :: PR: #11146
Bump Dockerfile.ci (2024-11-05) by @ko3n1g :: PR: #11159
NeMo-UX: Add sgd optim by @akoumpa :: PR: #11157
Update copyright check by @pablo-garay :: PR: #11168
add lora recipt for 405b by @JRD971000 :: PR: #10991
dit training diagrams by @zpx01 :: PR: #10873
ci: Switch to FW templates for build by @ko3n1g :: PR: #11077
Bump Dockerfile.ci (2024-11-06) by @ko3n1g :: PR: #11174
feat: Run PyLint by @ko3n1g :: PR: #11147
Add Alpaca Finetune Datamodule by @suiyoubi :: PR: #11185
Updated Diffusion Collection README by @zpx01 :: PR: #11179
Add support for Cosmos Tokenizers by @jojennin :: PR: #11194
Run formatting only if files changed. Echo message if pylint fails. by @artbataev :: PR: #11188
Bump Dockerfile.ci (2024-11-07) by @ko3n1g :: PR: #11196
Fix rotary_percentage parsing in nemo2 config by @meatybobby :: PR: #11197
ci: Update cherry pick workflow by @ko3n1g :: PR: #11202
ci: Build, test, publish a wheel by @ko3n1g :: PR: #11183
Bump Dockerfile.ci (2024-11-08) by @ko3n1g :: PR: #11222
update default pipeline_parallelism_type by @akoumpa :: PR: #11213
check actual value of vocab_file by @akoumpa :: PR: #11228
Fix VP Initialization Issue with Latest MCore by @suiyoubi :: PR: #11209
ci: Run Pylint strictly on new files, softly on history by @ko3n1g :: PR: #11212
ci: Add release workflow by @ko3n1g :: PR: #11180
Fix llm.generate by @hemildesai :: PR: #11217
Bump Dockerfile.ci (2024-11-11) by @ko3n1g :: PR: #11247
Bump Dockerfile.ci (2024-11-12) by @ko3n1g :: PR: #11254
Handling tokenizer in PTQ for Nemo 2.0 by @janekl :: PR: #11237
Fix finetuning datamodule resume by @cuichenx :: PR: #11187
ci: Move bump mcore to templates by @ko3n1g :: PR: #11229
ci: Fix secrets detector by @ko3n1g :: PR: #11205
chore(beep boop 🤖): Bump MCORE_TAG=aded519... (2024-11-12) by @ko3n1g :: PR: #11260
ci: Run secrets detector on pull_request_target by @ko3n1g :: PR: #11263
Advanced Diffusion Training Features by @zpx01 :: PR: #11246
Update pruning and distillation tutorial notebooks by @gvenkatakris :: PR: #11091
update nemo1->2 conversion according to changes in main by @HuiyingLi :: PR: #11253
Add llama 3.1 recipes by @cuichenx :: PR: #11273
Fix Finetune Recipe by @suiyoubi :: PR: #11267
Configure no restart validation loop in nl.Trainer by @hemildesai :: PR: #11029
Handle _io_unflatten_object when _thread_local.output_dir is not available by @hemildesai :: PR: #11199
Remove opencc upperbound by @thomasdhc :: PR: #10909
Fix head_size in NeMo to HF checkpoint converters for width pruned model support by @eagle705 :: PR: #11230
Fixes per comments by @gvenkatakris :: PR: #11280
Create phi3mini.py by @mayani-nv :: PR: #11281
ci: Fix release workflow by @ko3n1g :: PR: #11286
fix perf plugin CUDA_DEVICE_MAX_CONNECTIONS setting by @JimmyZhang12 :: PR: #11299
PTQ via NeMo-Run CLI by @janekl :: PR: #10984
PTQ memory optimization by @Laplasjan107 :: PR: #11257
Update README.md for collection page by @yaoyu-33 :: PR: #11223
Adding multimodal examples by @shanmugamr1992 :: PR: #11279
Add HF untrusted code toggle by @akoumpa :: PR: #11313
P2p chunk size setting in nemo 2.0 by @erhoo82 :: PR: #11312
Nemo2 batcheval by @HuiyingLi :: PR: #11158
DoRA by @cuichenx :: PR: #11104
Profiling - support Chakra & Kineto trace dumping by @lilyw97 :: PR: #11115
NeMo 2.0 SFT PEFT notebooks by @HuiyingLi :: PR: #10874
Update symlink option for save_last in ModelCheckpoint by @paul-gibbons :: PR: #11319
ci: Pass-through of workflow_event by @ko3n1g :: PR: #11322
Add StragglerDetection and auto-relaunch to NeMo2.0 by @ShriyaPalsamudram :: PR: #11328
Huvu/t5 nemo2.0 nemoci by @huvunvidia :: PR: #11291
TE acceleration using callbacks by @oyilmaz-nvidia :: PR: #11261
Leave target_module as default in PEFT Recipes by @cuichenx :: PR: #11334
More robust tar file loading from AIStore by @pzelasko :: PR: #11323
Fix CLIP transformer layer api by @yaoyu-33 :: PR: #11337
pass trust_remote_code to AutoTokenizer by @akoumpa :: PR: #11343
Fix linear layer replacement by @oyilmaz-nvidia :: PR: #11356
fix typo by @JRD971000 :: PR: #11351
Add torchrun local executor to recipes by @marcromeyn :: PR: #11342
Add PP support in NeVA along with few bug fixes by @yaoyu-33 :: PR: #11170
nemo2 peft merge by @HuiyingLi :: PR: #11017
Add dora recipes by @cuichenx :: PR: #11330
add fix to recipe by @JRD971000 :: PR: #11368
Add missing test to CICD needed list by @pablo-garay :: PR: #11376
update SquadDataModule to use run.config by @huvunvidia :: PR: #11358
Add llama 3.2 1b and 3b by @cuichenx :: PR: #11335
calculate metrics for nemo2 sftpeft notebook by @HuiyingLi :: PR: #11381
Enable packed dataset for validation; add a2a_experimental argument by @michal2409 :: PR: #11378
Fix DDP unused param error when TE is enabled in NeMo Lite by @oyilmaz-nvidia :: PR: #11364
Update llama32 vision (mllama) use attention bias by @yaoyu-33 :: PR: #11316
Fix environment variables in torchrun executor by @hemildesai :: PR: #11363
Add sample generate to PTQ for NeMo 2.0 by @Laplasjan107 :: PR: #11339
Fix selective restore by explicitly verifying keys by @hemildesai :: PR: #11377
Minor fix by @gvenkatakris :: PR: #11353
Add a fix for single-GPU nsys. by @tfogal :: PR: #11354
capitalize HF as HF instead of Hf by @akoumpa :: PR: #11384
ci: Add HF cache by @ko3n1g :: PR: #11398
Remove logic to skip checkpoint save if checkpoint exists by @ashors1 :: PR: #11362
Rewire tokenizer exception handling in model resume by @cuichenx :: PR: #11375
Adding LLava-Next model class by @yashaswikarnati :: PR: #11399
Fix vllm test issue when run_accuracy is enabled by @oyilmaz-nvidia :: PR: #11413
data modules for llava_next by @yashaswikarnati :: PR: #11400
Fix strategies saving unsharded optimizer states by @ananthsub :: PR: #11392
Adjust CLI support for PTQ by @janekl :: PR: #11421
Nemo run recipe's and example scripts for Llava Next by @yashaswikarnati :: PR: #11405
Huvu/t5 nemo2.0 nemoci 3b11b by @huvunvidia :: PR: #11388
ci: Allow dry-run of release by @ko3n1g :: PR: #11418
fix dtype when init HF model from config by @akoumpa :: PR: #11420
Handle import errors in virtual environment when running vLLM tests by @janekl :: PR: #11435
Fix loss mask when answer_only_loss=True by @ashors1 :: PR: #11444
[audio] Keep input directory structure when saving processed files by @anteju :: PR: #11403
Add different recipe examples to NeMo 2.0 by @BoxiangW :: PR: #11317
[Scripts] Remove fixed seed for adding noise by @anteju :: PR: #11401
Add option to provide prior NeMo 2 ckpt path to convert_nemo1_to_nemo… by @hemildesai :: PR: #11452
PTQ CLI and param updates by @janekl :: PR: #11459
Add tests for resiliency feature integration by @maanug-nv :: PR: #11406
ci: Disable HexHighEntropyString plugin by @ko3n1g :: PR: #11470
Fix broken links by @shashank3959 :: PR: #11294
Nemo 2.0 canonical lora by @cuichenx :: PR: #11416
ci: Run secrets detector on merge-commit by @ko3n1g :: PR: #11479
Formatting (minor) by @pablo-garay :: PR: #11485
Fix bug related to naming by @pablo-garay :: PR: #11487
Add BERT Model To NeMo2.0 by @suiyoubi :: PR: #11333
Update Nemo Distributed Checkpoint User Guide by @FortunaZhang :: PR: #11489
fix: regular torch optims (e.g., sgd) no longer error with closure spec by @terrykong :: PR: #11189
Add recipe configs validating by @BoxiangW :: PR: #10954
Fix finetuning PP by @cuichenx :: PR: #11474
[docs] Documentation for audio collection by @anteju :: PR: #11426
config hierarchy by @malay-nagda :: PR: #11145
Force param sync when using distributed optimizer and overlap_param_gather by @hemildesai :: PR: #11486
chore(beep boop 🤖): Bump MCORE_TAG=bd677bf... (2024-12-06) by @ko3n1g :: PR: #11492
Remove default mutable arguments from AbstractEmbModel constructor by @ananthsub :: PR: #11348
minor fix for nemo2 sftpeft readme by @HuiyingLi :: PR: #11502
Update Llama3 Fine-Tuning Notebook by @roclark :: PR: #11522
Fix CI issue on validation config by @BoxiangW :: PR: #11521
Freeze tags in in r2.1.0 by @github-actions[bot] :: PR: #11556
Cherrypick all + R2.1.0 fix cicd by @pablo-garay :: PR: #11622
Cherry pick Add fix docstring for speech commands (11638) into r2.1.0 by @ko3n1g :: PR: #11639
Cherrypick #11628 to r2.1.0 by @nasretdinovr :: PR: #11630
Update package_info.py by @ko3n1g :: PR: #11646
Cherry pick Add fix docstring for VAD (11659) into r2.1.0 by @ko3n1g :: PR: #11660
Fix tokenizer trust_remote_code by @cuichenx :: PR: #11657
Cherrypick 11568 by @cuichenx :: PR: #11656
Cherry pick Downgrading the 'datasets' package from 3.0.0 to 2.21.0 for Multilang_ASR.ipynb and ASR_CTC_Language_Finetuning.ipynb (11675) into r2.1.0 by @ko3n1g :: PR: #11677
r2.1.0 cherrypick by @pablo-garay :: PR: #11680
Cherry pick Rename multimodal data module - EnergonMultiModalDataModule (11654) into r2.1.0 by @ko3n1g :: PR: #11685
chore: Bump to r2.1.0rc2 by @ko3n1g :: PR: #11693
r2.1.0 ptl fix by @pablo-garay :: PR: #11694

</details>

Open source →

2.0.014 Nov 2024Deprecated

Added deprecation notice by @Ssofja :: PR: #11133

Highlights

Large language models & Multi modal

Training
- Long context recipe
- PyTorch Native FSDP 1
Models
- Llama 3
- Mixtral
- Nemotron
NeMo 1.0
- SDXL (text-2-image)
- Model Opt
  - Depth Pruning (docs)
  - Logit based Knowledge Distillation (docs)

Export

TensorRT-LLM v0.12 integration
LoRA support for vLLM
FP8 checkpoint

ASR

Parakeet large (ASR with PnC model)
Added Uzbek offline and Gregorian streaming models
Optimization feature for efficient bucketing to improve bs consumption on GPUs

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

add parakeet-tdt_ctc-110m model by @nithinraok :: PR: #10461
fix asr finetune by @stevehuang52 :: PR: #10508
replace unbiased with correction by @nithinraok :: PR: #10555
Update Multi_Task_Adapters.ipynb by @pzelasko :: PR: #10600
Fix asr warnings by @nithinraok :: PR: #10469
Fix typo in ASR RNNT BPE model by @pzelasko :: PR: #10742
TestEncDecMultiTaskModel for canary parallel by @karpnv :: PR: #10740
fix chunked infer by @stevehuang52 :: PR: #10581
training code for hybrid-autoregressive inference model by @hainan-xv :: PR: #10841
remove stacking operation from batched functions by @lilithgrigoryan :: PR: #10524
Add lhotse fixes for rnnt model training and WER hanging issue with f… by @nithinraok :: PR: #10821
Fix ASR tests by @artbataev :: PR: #10794
[Fix] Fixed sampler override and audio_key in prepare_audio_data by @anteju :: PR: #10980
[WIP] Add docs for NEST SSL by @stevehuang52 :: PR: #10804
Akoumparouli/mixtral recipe fix r2.0.0 by @akoumpa :: PR: #10994
TDT compute timestamps option and Extra Whitespace handling for SPE by @monica-sekoyan :: PR: #10875
ci: Switch to CPU only runner by @ko3n1g :: PR: #11035
Fix timestamps tests by @monica-sekoyan :: PR: #11053
ci: Pin release freeze by @ko3n1g :: PR: #11143
Fix RNN-T loss memory usage by @artbataev :: PR: #11144
Added deprecation notice by @Ssofja :: PR: #11133
Fixes for Canary adapters tutorial by @pzelasko :: PR: #11184
add ipython import guard by @nithinraok :: PR: #11191
Self Supervised Pre-Training tutorial Fix by @monica-sekoyan :: PR: #11206
update the return type by @nithinraok :: PR: #11210
Timestamps to transcribe by @nithinraok :: PR: #10950
[Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045
Beam search algorithm implementation for TDT models by @lilithgrigoryan :: PR: #10903

</details>

TTS

<details><summary>Changelog</summary>

Fix asr warnings by @nithinraok :: PR: #10469
Make nemo text processing optional in TTS by @blisc :: PR: #10584
[Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045

</details>

NLP / NMT

<details><summary>Changelog</summary>

Bump Dockerfile.ci (2024-09-09) by @ko3n1g :: PR: #10423
MCORE interface for TP-only FP8 AMAX reduction by @erhoo82 :: PR: #10437
Remove Apex dependency if not using MixedFusedLayerNorm by @cuichenx :: PR: #10468
Add missing import guards for causal_conv1d and mamba_ssm dependencies by @janekl :: PR: #10429
Update doc for fp8 trt-llm export by @Laplasjan107 :: PR: #10444
set mock in GPTDatasetConfig by @akoumpa :: PR: #10435
Remove running validating after finetuning by @huvunvidia :: PR: #10560
Extending modelopt spec for TEDotProductAttention by @janekl :: PR: #10523
Fix mb_calculator import in lora tutorial by @BoxiangW :: PR: #10624
.nemo conversion bug fix by @dimapihtar :: PR: #10598
Updating modelopt spec for Mixtral by @janekl :: PR: #10660
Require setuptools>=70 and update deprecated api by @thomasdhc :: PR: #10659
Akoumparouli/fix get tokenizer list by @akoumpa :: PR: #10596
[McoreDistOptim] fix the naming to match apex.dist by @gdengk :: PR: #10707
[fix] Ensures disabling exp_manager with exp_manager=null does not error by @terrykong :: PR: #10651
[feat] Update get_model_parallel_src_rank to support tp-pp-dp ordering by @terrykong :: PR: #10652
feat: Migrate GPTSession refit path in Nemo export to ModelRunner for Aligner by @terrykong :: PR: #10654
[MCoreDistOptim] Add assertions for McoreDistOptim and fix fp8 arg specs by @gdengk :: PR: #10748
Fix for crashes with tensorboard_logger=false and VP + LoRA by @vysarge :: PR: #10792
Adding init_model_parallel to FabricMegatronStrategy by @marcromeyn :: PR: #10733
Moving steps to MegatronParallel to improve UX for Fabric by @marcromeyn :: PR: #10732
Adding setup_megatron_optimizer to FabricMegatronStrategy by @marcromeyn :: PR: #10833
Make FabricMegatronMixedPrecision match MegatronMixedPrecision by @marcromeyn :: PR: #10835
Fix VPP bug in MegatronStep by @marcromeyn :: PR: #10847
Expose drop_last in MegatronDataSampler by @farhadrgh :: PR: #10837
Move collectiob.nlp imports inline for t5 by @marcromeyn :: PR: #10877
Use a context-manager when opening files by @akoumpa :: PR: #10895
Packed sequence bug fixes by @cuichenx :: PR: #10898
ckpt convert bug fixes by @dimapihtar :: PR: #10878
remove deprecated ci tests by @dimapihtar :: PR: #10922
Adithyare/oai chat completion by @arendu :: PR: #10785
Update T5 tokenizer (adding additional tokens to tokenizer config) by @huvunvidia :: PR: #10972
Add support and recipes for HF models via AutoModelForCausalLM by @akoumpa :: PR: #10962
- gpt3 175b cli by @malay-nagda :: PR: #10985
- Fix for crash with LoRA + tp_overlap_comm=false + sequence_parallel=true by @vysarge :: PR: #10920
- Update BaseMegatronSampler for compatibility with PTL'''s _BatchProgress by @ashors1 :: PR: #11016
- add deprecation note by @dimapihtar :: PR: #11024
- Update ModelOpt Width Pruning example defaults by @kevalmorabia97 :: PR: #10902
- switch to NeMo 2.0 recipes by @dimapihtar :: PR: #10948
- NeMo 1.0: upcycle dense to moe by @akoumpa :: PR: #11002
- Update mcore parallelism initialization in nemo2 by @yaoyu-33 :: PR: #10643
- Gemma2 in Nemo2 with Recipes by @suiyoubi :: PR: #11037
- Add Packed Seq option to GPT based models by @suiyoubi :: PR: #11100
- Fix MCoreGPTModel import in llm.gpt.model.base by @hemildesai :: PR: #11109
- TP+MoE peft fix by @akoumpa :: PR: #11114
- GPT recipes to use full te spec by @JimmyZhang12 :: PR: #11119
- Virtual pipeline parallel support for LoRA in NLPAdapterModelMixin by @vysarge :: PR: #11128
- update nemo args for mcore flash decode arg change by @HuiyingLi :: PR: #11138
- Call ckpt_to_weights_subdir from MegatronCheckpointIO by @ashors1 :: PR: #10897
- fix typo by @dimapihtar :: PR: #11234
- [Doc fixes] update file names, installation instructions, bad links by @erastorgueva-nv :: PR: #11045
- fix(export): GPT models w/ bias=False convert properly by @terrykong :: PR: #11255
</details>

Open source →

2.0.0rc115 Aug 2024pre-releaseDeprecated

deprecate NeMo NLP tutorial by @dimapihtar :: PR: #9864

Highlights

Large language models

PEFT: QLoRA support, LoRA/QLora for Mixture-of-Experts (MoE) dense layer
State Space Models & Hybrid Architecture support (Mamba2 and NV-Mamba2-hybrid)
Support Nemotron, Minitron, Gemma2, Qwen, RAG
Custom Tokenizer training in NeMo
Update the Auto-Configurator for EP, CP and FSDP

Multimodal

NeVA: Add SOTA LLM backbone support (Mixtral/LLaMA3) and suite of model parallelism support (PP/EP)
Support Language Instructed Temporal-Localization Assistant (LITA) on top of video NeVA

ASR

SpeechLM and SALM
Adapters for Canary Customization
Pytorch allocator in PyTorch 2.2 improves training speed up to 30% for all ASR models
Cuda Graphs for Transducer Inference
Replaced webdataset with Lhotse - gives up to 2x speedup
Transcription Improvements - Speedup and QoL Changes
ASR Prompt Formatter for multimodal Canary

Export & Deploy

In framework PyTriton deployment with backends: - PyTorch - vLLM - TRT-LLM update to 0.10
TRT-LLM C++ runtime

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Support dataloader as input to audio for transcription by @titu1994 :: PR: #9201
Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
Fix Online_Offline_Microphone_VAD_Demo.ipynb by @stevehuang52 :: PR: #9251
Remove .nemo instead of renaming by @mikolajblaz :: PR: #9281
Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer. by @galv :: PR: #9347
Revert "Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer." by @titu1994 :: PR: #9351
Prompt formatter API and canary transcribe tensor input support by @pzelasko :: PR: #9206
Fix prompt formatter's defaults=None case in multi-task model by @pzelasko :: PR: #9366
move AED chunked infer script by @stevehuang52 :: PR: #9367
Use model-cast-to-bfloat16 rather than AMP-to-bfloat16 for inference. by @galv :: PR: #9198
ci: Fix `L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_C… by @ko3n1g :: PR: #9399
Fix logging message for ASR by @titu1994 :: PR: #9469
Add support to change Multi task model prompt by @titu1994 :: PR: #9542
Enable encoder adapters for Canary and MultiTaskAED models by @titu1994 :: PR: #9409
Audio model collection by @anteju :: PR: #9263
TitaNet Batch Verify Speaker by @monica-sekoyan :: PR: #9337
Fix the arguments of forward_for_export function in msdd_models by @tango4j :: PR: #9624
chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
refactor: notebook branch release by @ko3n1g :: PR: #9711
Canary Adapters tutorial (#9670) by @nithinraok :: PR: #9777
typos and branch name update to r2.0.0rc1 by @nithinraok :: PR: #9846
Fix RNNT alignments test by @artbataev :: PR: #9770
By default trust remote code from HF Datasets by @nithinraok :: PR: #9886
Temporarily disable cuda graph based RNN-T greedy inference for r2.0.0rc1 by @galv :: PR: #9904
Enable CUDA graphs by default, but require CUDA 12.6 for full graphs by @artbataev :: PR: #9919
update branch name for script by @nithinraok :: PR: #9936
updte branch by @nithinraok :: PR: #9942 </details>

TTS

<details><summary>Changelog</summary>

Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
Add mel codec checkpoints by @anteju :: PR: #9228
GPU unit tests: Mark flaky tests to be fixed by @pablo-garay :: PR: #9559
chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
refactor: notebook branch release by @ko3n1g :: PR: #9711

</details>

LLM/Multimodal

<details><summary>Changelog</summary>

Update nemo.export module for quantized models by @janekl :: PR: #9218
Add save option to the TRT-LLM export test script by @oyilmaz-nvidia :: PR: #9221
Checkpoint resuming compatible for 2403 container by @suiyoubi :: PR: #9199
Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
use get with fallback when reading checkpoint_callback_params by @akoumpa :: PR: #9223
Revert rope fusion defaults by @cuichenx :: PR: #9237
fix import by @akoumpa :: PR: #9240
Add TRT-LLM params like max_num_tokens and opt_num_tokens by @oyilmaz-nvidia :: PR: #9210
sum-reduce grad_norm in DP+CP domain by @erhoo82 :: PR: #9262
Alit/bert convert fix by @JRD971000 :: PR: #9285
conv1d stable version by @JRD971000 :: PR: #9330
Fix trainer builder when exp_manager is not in config by @yaoyu-33 :: PR: #9293
Fix Peft Weights Loading in NeVA by @yaoyu-33 :: PR: #9341
Skip sequence_parallel allreduce when using Mcore DistOpt by @akoumpa :: PR: #9344
Fix FSDP gradient calculation with orig params by @janEbert :: PR: #9335
TRT-LLM Export Code Cleanup by @oyilmaz-nvidia :: PR: #9270
support null/None truncation field by @arendu :: PR: #9355
NeVa token fusion by @paul-gibbons :: PR: #9245
bugfix if using mcore distOpt with sft by @akoumpa :: PR: #9356
Re-org export code by @oyilmaz-nvidia :: PR: #9353
QLoRA by @cuichenx :: PR: #9340
PeFT fix for distOpt by @akoumpa :: PR: #9392
[NeMo-UX] Integrating mcore's DistributedDataParallel into MegatronStrategy by @marcromeyn :: PR: #9387
cherry pick of #9266 by @dimapihtar :: PR: #9411
Enable specifying alpha for PTQ INT8 SmoothQuant method by @janekl :: PR: #9423
add support for new mcore ds features by @dimapihtar :: PR: #9388
LoRA for MoE Layer by @cuichenx :: PR: #9396
Mistral-7B: apply user's precision to output checkpoint by @akoumpa :: PR: #9222
Add option to merge distributed optimizer buckets by @timmoon10 :: PR: #9414
TRT-LLM 0.10 Update by @oyilmaz-nvidia :: PR: #9402
In-framework deployment by @oyilmaz-nvidia :: PR: #9438
Bugfix missing variables and argument changes to MegatronPretrainingRandomSampler by @jstjohn :: PR: #9458
Hyena Operator by @guyjacob :: PR: #9264
Refactor Quantizer for reusing in QAT by @kevalmorabia97 :: PR: #9276
move load state dict after initialize parallel state in nlp_model by @ryxli :: PR: #9382
Enable user to optionally upgrade Megatron by @jstjohn :: PR: #9478
Fix unwrap model by @cuichenx :: PR: #9480
fix operator precedence by @akoumpa :: PR: #9403
[NeMo-UX] Adding context- & expert-parallelism to MegatronStrategy by @marcromeyn :: PR: #9525
update mcoreddp call by @akoumpa :: PR: #9345
mcore distOpt restore fix by @akoumpa :: PR: #9421
vLLM Export Support by @apanteleev :: PR: #9381
PL: Delete precision if using plugin. TODO switch to MegatronTrainerB… by @akoumpa :: PR: #9535
extend get_gpt_layer_modelopt_spec to support MoE by @akoumpa :: PR: #9532
fix mock data generation for legacy dataset by @dimapihtar :: PR: #9530
add reset learning rate functionality by @dimapihtar :: PR: #9372
Use closed-formula to round by multiple by @akoumpa :: PR: #9307
GPU unit tests: Mark flaky tests to be fixed by @pablo-garay :: PR: #9559
Consolidate gpt continue training script into pretraining script by @yaoyu-33 :: PR: #9413
Enable encoder adapters for Canary and MultiTaskAED models by @titu1994 :: PR: #9409
PTQ refinements by @janekl :: PR: #9574
Add ModelOpt QAT example for Llama2 SFT model by @kevalmorabia97 :: PR: #9326
Multimodal projection layer adapter fix for PP>1 by @paul-gibbons :: PR: #9445
Add offline quantization script for QLoRA deployment by @cuichenx :: PR: #9455
Make QLoRA more model-agnostic by @cuichenx :: PR: #9488
Set n_gpu to None in nemo export by @oyilmaz-nvidia :: PR: #9593
[NeMo-UX] Fix Megatron-optimizer by @marcromeyn :: PR: #9599
Chat template support for megatron_gpt_eval.py by @akoumpa :: PR: #9354
[NeMo-UX] Add PEFT by @cuichenx :: PR: #9490
Alit/mamba tmp by @JRD971000 :: PR: #9612
Enable MCore checkpointing optimizations by @mikolajblaz :: PR: #9505
Change mixtral moe key name for trt-llm by @oyilmaz-nvidia :: PR: #9620
fix ckpt load bug by @dimapihtar :: PR: #9621
Alit/mamba by @JRD971000 :: PR: #9575
Unwrap ckpt_io for model opt (async save) by @mikolajblaz :: PR: #9622
MCore T5 support for NeMo - Training by @huvunvidia :: PR: #9432
[Nemo-UX] Expose transformer_layer_spec inside GPTConfig by @marcromeyn :: PR: #9592
Update NeMo Clip to Use MCore Modules by @yaoyu-33 :: PR: #9594
Mistral + Mixtral Support for NeVa by @paul-gibbons :: PR: #9459
Adding support for mcore generate by @shanmugamr1992 :: PR: #9566
Improve error messaging during trt-llm export by @oyilmaz-nvidia :: PR: #9638
[Cherrypick] support lora when kv_channel != hidden_size / num_heads by @cuichenx :: PR: #9644
Parametrize FPS group by @mikolajblaz :: PR: #9648
Cherry-pick megatron export fix from main by @borisfom :: PR: #9643
add documentation for reset_lr feature by @dimapihta
chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
Cherry pick: LITA Integration by @Slyne :: PR: #9684
SDXL improvements (and support for Draft+) by @rohitrango :: PR: #9654
Gemma 2 by @cuichenx :: PR: #9672
Allows non-strict load with distributed checkpoints by @mikolajblaz :: PR: #9613
refactor: notebook branch release by @ko3n1g :: PR: #9711
[NeMo-UX] Make TE and Apex dependencies optional by @ashors1 :: PR: #9550
Alit/r2.0.0 by @JRD971000 :: PR: #9718
Manually cherry-pick from PR 9679 (PR to main - Support SFT/Eval/PEFT for mcore T5) by @huvunvidia :: PR: #9737
In framework export by @oyilmaz-nvidia :: PR: #9658
T5 changes based on mcore changes by @pablo-garay :: PR: #9829
[NeMo-UX] Use single instance of loss reductions in GPTModel by @hemildesai :: PR: #9801
deprecate NeMo NLP tutorial by @dimapihtar :: PR: #9864
Disable nvFuser setup with PyTorch 23.11 and later by @athitten :: PR: #9837
make torch_dist ckpt strategy as default by @dimapihtar :: PR: #9852
add rampup bs documentation by @dimapihtar :: PR: #9884
copy of #9576 by @dimapihtar :: PR: #9986
Support Nvidia Torch and Arch versions by @thomasdhc :: PR: #9897
Bug fix for pooler causing dist checkpointing exception by @shanmugamr1992 :: PR: #10008

</details>

Export

<details><summary>Changelog</summary>

Update nemo.export module for quantized models by @janekl :: PR: #9218
Add save option to the TRT-LLM export test script by @oyilmaz-nvidia :: PR: #9221
Add TRT-LLM params like max_num_tokens and opt_num_tokens by @oyilmaz-nvidia :: PR: #9210
TRT-LLM Export Code Cleanup by @oyilmaz-nvidia :: PR: #9270
Re-org export code by @oyilmaz-nvidia :: PR: #9353
Use TensorRT-LLM native parameter names in nemo.export module by @janekl :: PR: #9424
TRT-LLM 0.10 Update by @oyilmaz-nvidia :: PR: #9402
vLLM Export Support by @apanteleev :: PR: #9381
Add page context fmha option in TensorRTLLM export by @meatybobby :: PR: #9526
Test C++ runtime on demand in nemo_export.py to avoid possible OOMs by @janekl :: PR: #9544
Fix nemo export test by @oyilmaz-nvidia :: PR: #9547
Add tps and pps params to the export script by @oyilmaz-nvidia :: PR: #9558
Add Multimodal Exporter by @meatybobby :: PR: #9256
Set n_gpu to None in nemo export by @oyilmaz-nvidia :: PR: #9593
Inflight nemo model export support by @JimmyZhang12 :: PR: #9527
vLLM Export Improvements by @apanteleev :: PR: #9596
Akoumparouli/nemo ux mixtral export by @akoumpa :: PR: #9603
Change mixtral moe key name for trt-llm by @oyilmaz-nvidia :: PR: #9620
Fix the arguments of forward_for_export function in msdd_models by @tango4j :: PR: #9624
Improve error messaging during trt-llm export by @oyilmaz-nvidia :: PR: #9638
Cherry-pick megatron export fix from main by @borisfom :: PR: #9643
In framework export by @oyilmaz-nvidia :: PR: #9658
Add missing imports for torch dist ckpt in export by @oyilmaz-nvidia :: PR: #9826~

</details>

Bugfixes

<details><summary>Changelog</summary>

use get with fallback when reading checkpoint_callback_params by @akoumpa :: PR: #9223
fix import by @akoumpa :: PR: #9240
Remove .nemo instead of renaming by @mikolajblaz :: PR: #9281
call set_expert_model_parallel_world_size instead of set_cpu_expert_m… by @akoumpa :: PR: #9275
Fix typos in Mixtral NeMo->HF and Starcoder2 NeMo->HF conversion scripts by @evellasques :: PR: #9325
Skip sequence_parallel allreduce when using Mcore DistOpt by @akoumpa :: PR: #9344
Add OpenAI format response to r2.0.0rc1 by @athitten :: PR: #9796
[NeMo UX] Support generating datasets using different train/valid/test distributions by @ashors1 :: PR: #9771
Add missing imports for torch dist ckpt in export by @oyilmaz-nvidia :: PR: #9826

</details>

General Improvements

<details><summary>Changelog</summary>

[Nemo CICD] run_cicd_for_release_branches_also by @pablo-garay :: PR: #9213
rename paths2audiofiles to audio by @github-actions[bot] :: PR: #9220
Fix ASR_Context_Biasing.ipynb contains FileNotFoundError by @github-actions[bot] :: PR: #9234
ci: Remove duplicated job by @ko3n1g :: PR: #9258
Fix document links by @yaoyu-33 :: PR: #9260
Pin transformers by @github-actions[bot] :: PR: #9273
Fix loading github raw images on notebook by @github-actions[bot] :: PR: #9283
Accept None as an argument to decoder_lengths in GreedyBatchedCTCInfer::forward by @github-actions[bot] :: PR: #9278
Refactor Sequence Packing Script by @cuichenx :: PR: #9271
[Nemo-UX] Move code to collections + fix some small bugs by @marcromeyn :: PR: #9277
Fix typo in HF tutorial by @github-actions[bot] :: PR: #9304
Expand documentation for data parallelism and distributed optimizer by @timmoon10 :: PR: #9227
Install alerting by @ko3n1g :: PR: #9311
typos by @github-actions[bot] :: PR: #9315
FP8 feature documentation by @ksivaman :: PR: #9265
[Nemo CICD] Comment out flaky tests by @pablo-garay :: PR: #9333
Fixed typos in README.rst by @gdevakumar :: PR: #9322
Update README.rst to clarify installation via Conda by @SimonCW :: PR: #9323
[Nemo CICD] update flaky test by @pablo-garay :: PR: #9339
fix lora and ptuning and isort/black by @github-actions[bot] :: PR: #9295
Fix P-tuning for Llama based models by @github-actions[bot] :: PR: #9300
add large model stable training fix and contrastive loss update for variable seq by @github-actions[bot] :: PR: #9348
Guard cuda memory allocator update by @github-actions[bot] :: PR: #9313
[Nemo CICD] Remove unnecessary commented out code by @pablo-garay :: PR: #9364
Update Gemma conversion script by @yaoyu-33 :: PR: #9365
Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer. (#9347) by @github-actions[bot] :: PR: #9371
Re-enable cuda graphs in training modes. by @github-actions[bot] :: PR: #9343
fix typo infer_seq_lenght -> infer_seq_length by @akoumpa :: PR: #9370
Make a backward compatibility for old MSDD configs in label models by @github-actions[bot] :: PR: #9378
Dgalvez/fix greedy batch strategy name r2.0.0rc0 by @github-actions[bot] :: PR: #9253
Update README.rst by @jgerh :: PR: #9393
Force diarizer to use CUDA if cuda is available and if device=None. by @github-actions[bot] :: PR: #9390
ci: Properly catch failed tests by introduction of workflow templates by @ko3n1g :: PR: #9324
Fix T5 G2P Input and Output Types by @github-actions[bot] :: PR: #9269
Huvu/rag pipeline citest by @huvunvidia :: PR: #9384
Fix circular import for MM dataprep notebook by @github-actions[bot] :: PR: #9292
add check if num layers is divisible by pp size by @github-actions[bot] :: PR: #9298
[Nemo CICD] timeouts fix by @pablo-garay :: PR: #9407
[NeMo-UX] Removing un-used ModelConfig class by @marcromeyn :: PR: #9389
Add tutorial for Llama-3-8B lora training and deployment by @shashank3959 :: PR: #9359
[NeMo-UX] Removing default_path from ModelConnector by @marcromeyn :: PR: #9401
Fix README by @ericharper :: PR: #9415
[SD] Fix SD CUDA Graph Failure by @alpha0422 :: PR: #9319
[NeMo-UX] Adding file-lock to Connector by @marcromeyn :: PR: #9400
Add Dev Container Bug Report by @pablo-garay :: PR: #9430
Akoumparouli/profiling docs by @akoumpa :: PR: #9420
ci: Enrich notifications by @ko3n1g :: PR: #9412
Fix failing RIR unit test with lhotse 1.24+ by @pzelasko :: PR: #9444
[NeMo-UX] Adding support for mcore distributed optimizer by @marcromeyn :: PR: #9435
Use ModelOpt build_tensorrt_llm for building engines for qnemo checkpoints by @janekl :: PR: #9452
ci(notifications): Fix extraction of last 2K chars by @ko3n1g :: PR: #9450
Update readme with mlperf news by @ericharper :: PR: #9457
[NeMo-UX] Add nsys callback by @ashors1 :: PR: #9461
[NeMo UX] Introducing optimizer module by @marcromeyn :: PR: #9454
Fix minor import bug in deploy module by @oyilmaz-nvidia :: PR: #9463
ci(notifications): Fetch all jobs by @ko3n1g :: PR: #9465
Update build_dataset.py by @stevehuang52 :: PR: #9467
bionemo: bn2/add pipelineparallel dtype by @skothenhill-nv :: PR: #9475
[NeMo-UX] Integrate experiment manager features with NeMo-UX APIs by @ashors1 :: PR: #9460
Add python_requires by @galv :: PR: #9431
[NeMo-UX] Fixing imports of NeMoLogging, AutoResume & ModelCheckpoint by @marcromeyn :: PR: #9476
Modelopt Refactor for SDXL Quantization by @suiyoubi :: PR: #9279
[NeMo-UX] Fixing defaults in llm.train & Mistral7BModel by @marcromeyn :: PR: #9486
In framework deploy using deploy script by @oyilmaz-nvidia :: PR: #9468
[NeMo-UX] Integrate tokenizer import into model.import_ckpt by @marcromeyn :: PR: #9485
append to file by @malay-nagda :: PR: #9483
[NeMo-UX] Fix bug in import_ckpt by @marcromeyn :: PR: #9492
Add nemotron news by @ericharper :: PR: #9510
Add CICD test for Stable Diffusion by @michal2409 :: PR: #9464
Akoumparouli/nemo ux mixtral by @akoumpa :: PR: #9446
[NeMo-UX] Llama and Gemma by @cuichenx :: PR: #9528
[NeMo-UX] minor logging bug fixes by @ashors1 :: PR: #9529
Update neva conversion script from and to HF by @yaoyu-33 :: PR: #9296
[Nemo-UX] IO fixes by @marcromeyn :: PR: #9512
Fix lhotse tests for v1.24.2 by @pzelasko :: PR: #9546
[Nemo CICD] Make GPU Unit Tests non-optional by @pablo-garay :: PR: #9551
Add Python AIStore SDK to container and bump min Lhotse version by @pzelasko :: PR: #9537
[NeMo-UX] Fix tokenizer IO by @marcromeyn :: PR: #9555
[NeMo UX] Move mistral_7b.py to mistral.py by @akoumpa :: PR: #9545
ci: Do not attempt to send slack on fork by @ko3n1g :: PR: #9556
Fix SDXL incorrect name in Docs by @suiyoubi :: PR: #9534
Bump PTL version by @athitten :: PR: #9557
[Resiliency] Straggler detection by @jbieniusiewi :: PR: #9473
[NeMo-UX] Switch to torch_dist as default distributed checkpointing backend by @ashors1 :: PR: #9541
[NeMo-UX] Checkpointing bug fixes by @ashors1 :: PR: #9562
Expose MCore path_to_cache option by @maanug-nv :: PR: #9570
[NeMo-UX] Fix Trainer serialization by @marcromeyn :: PR: #9571
Update click version requirement by @thomasdhc :: PR: #9580
[Fault tolerance] Heartbeat detection by @maanug-nv :: PR: #9352
[Nemo-UX] Add fabric-API for manual forward-pass by @marcromeyn :: PR: #9577
[Nemo-UX] Add SDK-factories to llm-collection by @marcromeyn :: PR: #9589
[NeMo-UX] Some improvements to NeMoLogger by @marcromeyn :: PR: #9591
Set no_sync_func & grad_sync_fucn by @akoumpa :: PR: #9601
[NeMo-UX] Fix nemo logger when trainer has no loggers by @ashors1 :: PR: #9607
Fix the dictionary format returned by the scheduler method by @sararb :: PR: #9609
[NeMo-UX] Dataloading enhancements and bug fixes by @ashors1 :: PR: #9595
Fix serialization of AutoResume by @sararb :: PR: #9616
Jsonl support by @adityavavre :: PR: #9611
Akoumparouli/mistral import instruct chat template fix by @akoumpa :: PR: #9567
Remove .cuda calls, use device isntead by @akoumpa :: PR: #9602
fix converter defautl args by @akoumpa :: PR: #9565
fix: remove non_blocking from PTL's .cuda call by @akoumpa :: PR: #9618
NeVA Minor Fixes by @yaoyu-33 :: PR: #9608
[NeMo-UX] fix pretrianing data sizes and weights by @cuichenx :: PR: #9627
[NeMo-UX] async checkpointing support by @ashors1 :: PR: #9466
Change default parallel_save to False by @mikolajblaz :: PR: #9632
Add REST API to deploy module by @athitten :: PR: #9539
ci: Timeout per step, not job by @ko3n1g :: PR: #9635
[NeMo-UX] Fix when optimizers are setup for PEFT by @marcromeyn :: PR: #9619
[NeMo-UX] Fix pipeline parallel bug by @ashors1 :: PR: #9637
Fixing import error fior llama-index (RAG pipeline) by @pablo-garay :: PR: #9662
llama CI fix by @rohitrango :: PR: #9663
[NeMo-UX] Make 'load_directly_on_device' configurable by @ashors1 :: PR: #9657
[Nemo-UX] Including all trainable-params in a PEFT-checkpoint by @marcromeyn :: PR: #9650
[NeMo-UX] Fix imports so local configuration of runs works again by @marcromeyn :: PR: #9690
Set TE flag in legacy -> mcore conversion script by @terrykong :: PR: #9722
Update starthere docs text by @erastorgueva-nv :: PR: #9724
TorchAudio installation workaround for incorrect PYTORCH_VERSION variable by @artbataev :: PR: #9736
[NeMo-UX] Match nemo 1's default behavior for drop_last and pad_samples_to_global_batch_size by @ashors1 :: PR: #9707
add a bit more for timeout (#9702) by @pablo-garay :: PR: #9754
Fix missing parallelisms by @maanug-nv :: PR: #9725
update branch by @nithinraok :: PR: #9764
Fix data preprocessing script by @cuichenx :: PR: #9759
vLLM 0.5.1 update by @apanteleev :: PR: #9779
upper bound hf-hub by @akoumpa :: PR: #9805
Fix few issues and docs for neva and clip in r2.0.0rc1 by @yaoyu-33 :: PR: #9681
add dummy vision and text transformer config (assumed mcore to be false) by @rohitrango :: PR: #9699
fix lita bugs by @Slyne :: PR: #9810
[NeMo-UX] Log val_loss by @ashors1 :: PR: #9814
[NeMo-UX] Fix some dataloading bugs by @ashors1 :: PR: #9807
[NeMo-UX] Adding recipes by @marcromeyn :: PR: #9720
[NeMo-UX] Set async_save from strategy rather than ModelCheckpoint by @ashors1 :: PR: #9800
Fix hf hub for 0.24+ by @titu1994 :: PR: #9806
[NeMo-UX] Fix a minor bug with async checkpointing by @ashors1 :: PR: #9856
[NeMo-UX] make progress bar easier to parse by @ashors1 :: PR: #9877
Docs: add "Nemo Fundamentals" page by @erastorgueva-nv :: PR: #9835
Create init.py by @stevehuang52 :: PR: #9892
[NeMo-UX] Fixes to make PreemptionCallback work by @hemildesai :: PR: #9830
Fix Docker build. Make Dockerfile consistent with CI by @artbataev :: PR: #9784
Multimodal data prep notebook fix by @cuichenx :: PR: #9910
[NeMo-UX] Add distributed checkpointing unit tests by @ashors1 :: PR: #9794
r2.0.0rc1 fix for dist checkpoint loading by @yaoyu-33 :: PR: #9854
[NeMo-UX] Rename sdk references to NeMo Run by @hemildesai :: PR: #9872
[NeMo-UX] Fix some serialization bugs by @ashors1 :: PR: #9868
add mixtral neva tutorial (moe + token fusion + siglip) by @paul-gibbons :: PR: #9926
[NeMo-UX] Add more NeMo Logger tests by @ashors1 :: PR: #9795
Akoumparouli/mixtral fixes for r2.0.0rc1 by @akoumpa :: PR: #9911
R2.0.0rc1 clip fix by @Slyne :: PR: #9871
[NeMo-UX] Add missing docstrings and update some defaults by @ashors1 :: PR: #9895
Add REST service requirements.txt by @oyilmaz-nvidia :: PR: #9923
add bert latest fix by @JRD971000 :: PR: #9921
remove empy reconfigure_limit_batches by @akoumpa :: PR: #9934
fix mem by @terrykong :: PR: #9964
Run a sample query for a quantized model conditionally by @janekl :: PR: #9965
Add pydantic-settings by @oyilmaz-nvidia :: PR: #9961
Resiliency features update by @jbieniusiewi :: PR: #9714
[NeMo-UX] Wrap task config save in a try/except by @ashors1 :: PR: #9956
[NeMo-UX] Update default PTL logging save_dir by @ashors1 :: PR: #9954
Fix lita tutorial by @Slyne :: PR: #9980
Add deploy and REST API support to NeMo 2.0 by @athitten :: PR: #9834
ci: Allow changelog manual (#10156) by @ko3n1g :: PR: #10157
docs: Add changelog by @ko3n1g :: PR: #10155
add manifest file by @ko3n1g :: PR: #10161

</details>

Open source →

2.0.0rc06 Jun 2024pre-releaseDeprecated

Cleanup deprecated files and temporary changes by @cuichenx :: PR: #9088

Highlights

LLM and MM

Models

Megatron Core RETRO
- Pre-training
- Zero-shot Evaluation
Pretraining, conversion, evaluation, SFT, and PEFT for:
- Mixtral 8X22B
- Llama 3
- SpaceGemma
Embedding Models Fine Tuning
- Mistral
- BERT
BERT models
- Context Parallel
- Distributed checkpoint
Video capabilities with NeVa

Performance

Distributed Checkpointing
- Torch native backend
- Parallel read/write
- Async write
Multimodal LLM (LLAVA/NeVA)
- Pipeline Parallelism support
- Sequence packing support

Export

Integration of Export & Deploy Modules into NeMo Framework container
- Upgrade to TRT-LLM 0.9

Speech (ASR & TTS)

Models

AED Multi Task Models (Canary) - Multi-Task Multi-Lingual Speech Recognition / Speech Translation model
Multimodal Domain - Speech LLM supporting SALM Model
Parakeet-tdt_ctc-1.1b Model - RTFx of > 1500 (can transcribe 1500 seconds of audio in 1 second)
Audio Codec 16kHz Small - NeMo Neural Audio Codec for discretizing speech for use in LLMs
- mel_codec_22khz_medium
- mel_codec_44khz_medium

Perf Improvements

Transcribe() upgrade - Enables one line transcribe with files, tensors, data loaders
Frame looping algorithm for RNNT faster decoding - Improves Real Time Factor (RTF) by 2-3x
Cuda Graphs + Label-Looping algorithm for RNN-T and TDT Decoding - Transducer Greedy decoding at over 1500x RTFx, on par with CTC Non-Autoregressive models
Semi Sorted Batching support - External User contribution that speeds up training by 15-30%.

Customization

Context biasing for CTC word stamping - Improve accuracy for custom vocabulary and pronunciation
- Longform Inference
- Longform inference support for AED models
Transcription of multi-channel audio for AED models

Misc

Upgraded webdataset - Speech and LLM / Multimodal unified container

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Enable using hybrid asr models in CTC Segmentation tool by @erastorgueva-nv :: PR: #8828
TDT confidence fix by @GNroy :: PR: #8982
Fix union type annotations for autodoc+mock-import rendering by @pzelasko :: PR: #8956
NeMo dev doc restructure by @yaoyu-33 :: PR: #8896
Improved random seed configuration for Lhotse dataloaders with docs by @pzelasko :: PR: #9001
Fix #8948, allow preprocessor to be stream captured to a cuda graph when doing per_feature normalization by @galv :: PR: #8964
[ASR] Support for transcription of multi-channel audio for AED models by @anteju :: PR: #9007
Add ASR latest news by @titu1994 :: PR: #9073
Fix docs errors and most warnings by @erastorgueva-nv :: PR: #9006
PyTorch CUDA allocator optimization for dynamic batch shape dataloading in ASR by @pzelasko :: PR: #9061
RNN-T and TDT inference: use CUDA graphs by default by @artbataev :: PR: #8972
Fix #8891 by supported GPU-side batched CTC Greedy Decoding by @galv :: PR: #9100
Update branch for notebooks and ci in release by @ericharper :: PR: #9189
Enable CUDA graphs by default only for transcription by @artbataev :: PR: #9196
rename paths2audiofiles to audio by @nithinraok :: PR: #9209
Fix ASR_Context_Biasing.ipynb contains FileNotFoundError by @andrusenkoau :: PR: #9233
Cherrypick: Support dataloader as input to audio for transcription (#9201) by @titu1994 :: PR: #9235
Update Online_Offline_Microphone_VAD_Demo.ipynb by @stevehuang52 :: PR: #9252
Dgalvez/fix greedy batch strategy name r2.0.0rc0 by @galv :: PR: #9243
Accept None as an argument to decoder_lengths in GreedyBatchedCTCInfer::forward by @galv :: PR: #9246
Fix loading github raw images on notebook by @nithinraok :: PR: #9282
typos by @nithinraok :: PR: #9314
Re-enable cuda graphs in training modes. by @galv :: PR: #9338
add large model stable training fix and contrastive loss update for variable seq by @nithinraok :: PR: #9259
Fix conv1d package in r2.0.0rc0 by @pablo-garay :: PR: #9369
Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer. (#9347) by @titu1994 :: PR: #9350
Make a backward compatibility for old MSDD configs in label models by @tango4j :: PR: #9377
Force diarizer to use CUDA if cuda is available and if device=None. by @tango4j :: PR: #9380

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Add tutorial for training audio codecs by @rlangman :: PR: #8723
Update radtts.py by @blisc :: PR: #9097
[Nemo CICD] RADTTS test optional by @pablo-garay :: PR: #9112
Remove Radtts CI test by @blisc :: PR: #9144
Fix T5 G2P Input and Output Types by @blisc :: PR: #9224

</details>

LLM and MM

<details><summary>Changelog</summary>

Rachitg/dpa by @rachitgarg91 :: PR: #8911
Remove precision args in trainer due to PTL update by @yaoyu-33 :: PR: #8908
Huvu/mcore retro by @huvunvidia :: PR: #8861
fsdp tp > 1 bug fix by @dimapihtar :: PR: #8947
Fix memory leak at loss func by @minitu :: PR: #8868
change the condition for get qkv tensor from linear_qkv output in mcoremixin by @HuiyingLi :: PR: #8965
Add safety checks for 'data' key in MegatronGPTModel cfg by @HuiyingLi :: PR: #8991
[NeMo-UX] Adding MegatronParallel by @cuichenx :: PR: #8987
Skip top_p computations when set to 1.0 by @odelalleau :: PR: #8905
Gemma bug by @cuichenx :: PR: #8962
[NeMo-UX] Adding megatron strategy by @marcromeyn :: PR: #8995
Quantized checkpoint support in export and deploy modules by @janekl :: PR: #8859
add geglu to mlp swap by @JRD971000 :: PR: #8999
add timeout for new_group by @acphile :: PR: #8998
Zero-shot evaluation pipeline for mcore RETRO by @huvunvidia :: PR: #8941
Added fusion for squared relu by @sanandaraj5597 :: PR: #8963
Developer Documents for mcore RETRO by @huvunvidia :: PR: #9026
[NeMo-UX] Adding GPTModel & MockDataModule by @marcromeyn :: PR: #9011
Adding unit test for mcore RETRO model by @huvunvidia :: PR: #9022
docs and simplification of cmd args by @arendu :: PR: #8979
[NeMo-UX] Add checkpoint-io to MegatronStrategy by @marcromeyn :: PR: #9057
Enable Sequence Packing and Pipeline Parallel in NeVA by @yaoyu-33 :: PR: #8957
Mingyuanm/add back fp8 support to sd by @Victor49152 :: PR: #9070
unfused lora by @arendu :: PR: #9004
Handle case where num_query_groups is set to null for LoRA config setup by @vysarge :: PR: #9075
Alit/griffin by @JRD971000 :: PR: #9021
Implement DistributedCheckpointIO by @mikolajblaz :: PR: #9016
Video Neva Pretraining + Inference Implementation by @paul-gibbons :: PR: #9095
HF to .nemo for Mixtral-8x22B-instruct by @akoumpa :: PR: #9060
mcore ds updates by @dimapihtar :: PR: #8951
Alit/griffin perf by @JRD971000 :: PR: #9107
Add assert for max_steps to be positive in MegatronGPTSFTModel by @athitten :: PR: #9110
Extend sequence length padding for GPT SFT to account for context parallel by @vysarge :: PR: #8869
Update gpt dataset config parameter for mock by @thomasdhc :: PR: #9118
Add Mcore DistributedDataParallel and distributed optimizer into Nemo by @gdengk :: PR: #9034
Revert "Add assert for max_steps to be positive in MegatronGPTSFTMode… by @pablo-garay :: PR: #9128
scripts to convert HF lora to nemo by @arendu :: PR: #9102
Prevent duplicated checkpoints by @mikolajblaz :: PR: #9015
add TN/ITN link in speech tools list by @erastorgueva-nv :: PR: #9142
Cleanup deprecated files and temporary changes by @cuichenx :: PR: #9088
Use DP+CP groups as the FSDP sharding domain by @erhoo82 :: PR: #9145
CUDA memory profile by @erhoo82 :: PR: #9096
Fix missing func for T5 model by @gdengk :: PR: #9141
Add knob for load_directly_on_device by @mikolajblaz :: PR: #9125
Revert rope fusion defaults by @cuichenx :: PR: #9238
Update nemo.export module for quantized models by @janekl :: PR: #9250
Fix circular import for MM dataprep notebook by @cuichenx :: PR: #9287
neva media_type + text generation default fix by @paul-gibbons :: PR: #9257
fix lora and ptuning and isort/black by @oyilmaz-nvidia :: PR: #9290
add check if num layers is divisible by pp size by @dimapihtar :: PR: #9208
Fix P-tuning for Llama based models by @apanteleev :: PR: #9297
add deprecation warnings by @pablo-garay :: PR: #9266
move pooler under post_process by @dimapihtar :: PR: #9328
add deprecation note for nmt by @dimapihtar :: PR: #9342
Fix incorrect checkpoint removal logic (#9192) by @mikolajblaz :: PR: #9204
fix fp16 precision issue by @dimapihtar :: PR: #9376
Fix module.training for Neva in FusedAttn backward which causes nan by @yaoyu-33 :: PR: #8877

</details>

Export

<details><summary>Changelog</summary>

Updates for TRT-LLM 0.9 by @oyilmaz-nvidia :: PR: #8873
Mingyuanm/sdxl export by @Victor49152 :: PR: #8926
Avoid unpacking NeMo checkpoints before exporting to TRT-LLM by @apanteleev :: PR: #8866
Update gemma for trt-llm 0.9 by @oyilmaz-nvidia :: PR: #8974
TRT-LLM export P-tuning related fixes by @apanteleev :: PR: #8863

</details>

General Improvements

<details><summary>Changelog</summary>

Update package info by @ericharper :: PR: #8793
[Nemo CICD] Update mcore 4.13.24 by @pablo-garay :: PR: #8917
Akoumparouli/low mem mixtral ckpt converter by @akoumpa :: PR: #8895
Adding RETRO tests to Action Tests (cicd-main.yml) by @huvunvidia :: PR: #8942
Akoumparouli/fix sd train 2 by @akoumpa :: PR: #8883
Update te install for jenkins by @ericharper :: PR: #8954
[Nemo CICD] Add last job depending on others for blocking check by @pablo-garay :: PR: #8959
Minor quantization pipeline updates by @janekl :: PR: #8924
Fix External CLIP Converter by @yaoyu-33 :: PR: #8960
PP support in LoRA merge script by @cuichenx :: PR: #8934
Update PR template by @ericharper :: PR: #8978
Update Latest News by @shashank3959 :: PR: #8837
Fix incorrect link to latest news in README by @shashank3959 :: PR: #8985
Update dependency install for LLM and MM by @ericharper :: PR: #8990
Temporarily remove mcore dep by @ericharper :: PR: #9010
[Nemo CICD] further specialize runners for more parallelism by @pablo-garay :: PR: #9036
Update mm dataprep notebook based on feedback by @cuichenx :: PR: #9029
Fix import in lora merge script by @cuichenx :: PR: #9032
[Nemo CICD] Run when labeled:Run CICD by @pablo-garay :: PR: #9044
[Nemo CICD] Add tag/label for 1-gpu runner by @pablo-garay :: PR: #9046
[Nemo CICD] checkout v4 by @pablo-garay :: PR: #9048
[Nemo CICD] Remove temp test change by @pablo-garay :: PR: #9049
remove in-place addition for dreambooth train with text encoder by @Victor49152 :: PR: #8825
Mingyuanm/sdxl quantization notebook by @Victor49152 :: PR: #9042
[Nemo CICD] Trigger on comment issued by @pablo-garay :: PR: #9062
zarr ckpt to torch_dist ckpt converter by @dimapihtar :: PR: #8842
Restore PTQ tests for Llama2 (reopened) by @janekl :: PR: #9064
add clip H config by @JRD971000 :: PR: #9082
[NeMo-UX] Add mixed-precision plugin by @marcromeyn :: PR: #9065
Comment baichuan test and update pr template by @ericharper :: PR: #9085
Add safe extraction of nemo tar files by @athitten :: PR: #8976
Improved shard_id parsing in LazyNemoTarredIterator, enables AIS dataloading by @pzelasko :: PR: #9077
[NeMo-UX] Add mistral-7b model by @marcromeyn :: PR: #9066
Llama3 Conversion Script Update by @suiyoubi :: PR: #9089
dehardcode test string by @JimmyZhang12 :: PR: #8865
[Nemo CICD] Try trigger cicd run on comment by @pablo-garay :: PR: #9111
Lhotse dataloading: RIR augmentation and nemo/tarred input support for RIR and noise aug by @pzelasko :: PR: #9109
mixtral evaluation PR by @Slyne :: PR: #8989
[Nemo CICD] Revert: run GHA cicd on comment by @pablo-garay :: PR: #9119
[Nemo CICD] Comment out flaky test: running too long by @pablo-garay :: PR: #9123
[Nemo CICD] Add timeout to unit tests by @pablo-garay :: PR: #9132
[Nemo CICD] Indicate optional test in name (prefix) by @pablo-garay :: PR: #9139
video neva null image+video folder path fix by @paul-gibbons :: PR: #9116
[NeMo-UX] Add data module by @cuichenx :: PR: #9133
NeMo Inference Requirements by @oyilmaz-nvidia :: PR: #9093
Remove debug print by @maanug-nv :: PR: #9074
Remove legacy CI by @pablo-garay :: PR: #9149
Update support for push_to_hf_hub() by @titu1994 :: PR: #9159
[Nemo CICD] comment out flaky PTQ tests by @pablo-garay :: PR: #9160
Update branch by @ericharper :: PR: #9211
dist adam transpose fix by @dimapihtar :: PR: #9239
[Nemo CICD] Increase time limit for Speech_Checkpoints_tests (#9186) by @pablo-garay :: PR: #9247
Pin transformers by @ericharper :: PR: #9261
Fix typo in HF tutorial by @titu1994 :: PR: #9302

</details>

Open source →

1.23.028 Feb 2024Deprecated

[NLP] Remove replace_sampler_ddp (deprecated in Trainer) by @janekl :: PR: #7981

Highlights

Models

Nvidia Starcoder 2 - 15B

Announcement - https://developer.nvidia.com/blog/unlock-your-llm-coding-potential-with-starcoder2/
AI Foundation Model Inference - https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/starcoder2-15b
https://huggingface.co/bigcode/starcoder2-15b

NeMo Canary

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-02-canary/

https://huggingface.co/nvidia/canary-1b

NeMo LLM

Falcon
Code Llama
StarCoder
GPT perf improvements
Context parallelism
Mistral
Mixtral (without expert parallelism)
Mcore GPT Dataset integration

NeMo MM

CLIP
Stable Diffusion (supporting LoRA)
Imagen
ControlNet (for SD)
Instruct pix2pix (for SD)
LLAVA
NeVA
DreamFusion++
NSFW filtering

NeMo ASR

Lhotse Dataloading support #7880
Canary: Multi task multi lingual ASR #8242
LongForm Audio for Diarization #7737
Faster algorithm for RNN-T Greedy #7926
Cache-Aware streaming notebook #8296

NeMo TTS

NeMo Vision

Known Issues

ASR

RNNT WER calculation when fused batch size > 1 during validation / test step()

Previously, the RNNT metric was stateful while the CTC one was not (r1.22.0, r1.23.0)

Therefore this calculation in the RNNT joint for fused operation worked properly. However with the unification of metrics in r1.23.0, a bug was introduced where only the last sub-batch of metrics calculates the scores and does not accumulate. This is patched via https://github.com/NVIDIA/NeMo/pull/8587 and will be fixed in the next release.

Workaround: Explicitly disable fused batch size during inference using the following command

from omegaconf import open_dict
model = ...
decoding_cfg = model.cfg.decoding
with open_dict(decoding_cfg):
  decoding_cfg.fused_batch_size = -1
model.change_decoding_strategy(decoding_cfg)

Note: This bug does not affect scores calculated via model.transcribe() (since it does not calculate metrics during inference, just text), or using the transcribe_speech.py or speech_to_text_eval.py in examples/asr.

Two failing unit tests due to a change in expected results, caused by lhotse version update.

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:24.01.speech

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Update link to yaml file in ASR_with_Transducers.ipynb by @Faith-Nchifor :: PR: #8014
Use convert_hf_dataset_to_nemo by @karpnv :: PR: #8017
Update asr_language_modeling.rst: Add a missing word by @martin0258 :: PR: #8007
spelling mistake by @orena1 :: PR: #7903
update asr eval by @stevehuang52 :: PR: #8045
fix noise aug by @stevehuang52 :: PR: #8057
Various fixes for typos and urls by @titu1994 :: PR: #8066
[Fix] Increase length check tolerance to prevent test failing by @anteju :: PR: #8067
Add text metrics to asr eval by @stevehuang52 :: PR: #8087
fix device setting to allow using accelerator cpu by @orena1 :: PR: #8084
.ctm in data simulator annotator compliant with RT-09 specification by @popcornell :: PR: #8004
Fix AST eval by @stevehuang52 :: PR: #8112
fix: numba.*_num_threads resets torch num_threads #8141 by @itzsimpl :: PR: #8145
Update dependencies by @titu1994 :: PR: #8156
NeMo + Lhotse integration by @pzelasko :: PR: #7880
Speedup RNN-T greedy decoding by @artbataev :: PR: #7926
[docker] Install k2 before NeMo for faster image rebuilding by @pzelasko :: PR: #8204
[docs] Add --force_codec to tarred dataset creation examples by @pzelasko :: PR: #8227
Temporarily use the previous RNN-T decoding algorithm as default by @artbataev :: PR: #8226
Make TDT inference not require duration params by @hainan-xv :: PR: #8207
Cache Aware Streaming tutorial notebook by @erastorgueva-nv :: PR: #8296
fix path location and branch by @nithinraok :: PR: #8304
Attention encoder-decoder models for multiple speech-to-text tasks … by @titu1994 :: PR: #8324
Remove asr webapp by @titu1994 :: PR: #8347
remove target at model level in aed model config [ASR] by @krishnacpuvvada :: PR: #8351
Add change_vocabulary and save_tokenizers() support to Multitask ASR models by @titu1994 :: PR: #8357
Change default beam size by @titu1994 :: PR: #8371
adding jenkins test for speech_to_text_aed model by @krishnacpuvvada :: PR: #8368
Add Finetuning tutorial with HF Datasets by @nithinraok :: PR: #8356
wer fix by @tbartley94 :: PR: #8404
add ensemble decoding fix by @nithinraok :: PR: #8427
Update k2 by @artbataev :: PR: #8492

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Scale sampler steps by number of devices by @rlangman :: PR: #7947
Add All Multimodal Source Code Part 2: Text to image, x to nerf by @yaoyu-33 :: PR: #7970
[TTS] Add period discriminator and feature matching loss to codec recipe by @rlangman :: PR: #7884
Added VectorQuantizer base class by @anteju :: PR: #8011

</details>

LLMS

<details><summary>Changelog</summary>

Add interface to set NCCL options of each process group by @erhoo82 :: PR: #7923
Support O2 training of PEFT and SFT by @cuichenx :: PR: #7971
[NLP] Access scaler only in FP16 case by @janekl :: PR: #7916
[NLP] Minor improvements in Llama conversion script by @janekl :: PR: #7978
[NLP] Use helpers from utils_funcs.py in Llama conversion by @janekl :: PR: #7979
[NLP] Remove replace_sampler_ddp (deprecated in Trainer) by @janekl :: PR: #7981
Reworked MegatronPretrainingRandomBatchSampler to correctly handle epochs > 1 by @trias702 :: PR: #7920
Remove deprecated arguments from TE's TransformerLayer by @jbaczek :: PR: #7917
Add All Multimodal Source Code by @yaoyu-33 :: PR: #7791
First draft of mcore bert model in NeMo by @shanmugamr1992 :: PR: #7814
Support Falcon Variants (7B/40B/180B) in Mcore NeMo by @xuanzic :: PR: #7666
FSDP + Tensor Parallelism by @erhoo82 :: PR: #7897
Packed Sequence by @cuichenx :: PR: #7945
Adding method back that was removed accidentally by @ericharper :: PR: #8038
[NLP] ArtifactItem with init=True to make it debuggable by @janekl :: PR: #7980
SFT patch: (1) enable sequence parallelism and (2) enable profile by @erhoo82 :: PR: #7963
migration to PTL 2.0 for spellmapper model by @bene-ges :: PR: #7924
Change the megatron config lr scheduler default and fix to change partitions script by @shan18 :: PR: #8094
(1) Add SHARP interface to M-CORE, (2) use send/recv to send train loss to the first rank instead of b-cast by @erhoo82 :: PR: #7793
Reconfigure limit_val_batches only for int by @athitten :: PR: #8099
Fixing wrapper and moving it to base class by @shanmugamr1992 :: PR: #8055
fix gated_linear_unit bug by @Agoniii :: PR: #8042
Fix Adapter for MCore models by @cuichenx :: PR: #8124
add war fix for sync issues by @gshennvm :: PR: #8130
Improve PEFT UX by @cuichenx :: PR: #8131
Enhance flexibility by passing callbacks as method argument by @michal2409 :: PR: #8015
context parallelism by @xrennvidia :: PR: #7739
Make pipelined TP comm overlap available with mcore by @erhoo82 :: PR: #8005
remove deprecated scripts by @arendu :: PR: #8138
adding OnlineSampleMapping by @arendu :: PR: #8137
Add distopt support for FP8 params and BF16 optimizer state by @timmoon10 :: PR: #7909
Revert adding OnlineSampleMapping by @pablo-garay :: PR: #8164
Token count and sequence length logging for MegatronGPTSFTModel by @vysarge :: PR: #8136
Use latest apex internal API by @jbaczek :: PR: #8129
tune specific params in the base model by @arendu :: PR: #7745
Virtual pipeline parallel support for MegatronGPTSFTModel by @vysarge :: PR: #7964
removed deprecated peft model by @arendu :: PR: #8183
remove more deprecated files by @arendu :: PR: #8169
Pre-generate cu_seqlens argmin and max_seqlen to remove host-to-device sync by @erhoo82 :: PR: #8108
Add the interface to use SHARP to FSDP strategy by @erhoo82 :: PR: #8202
Multimodal required NLP base model changes by @yaoyu-33 :: PR: #8188
[NLP] Improve and unify loading state_dict for community models by @janekl :: PR: #7977
Rename Finetuning Scripts by @cuichenx :: PR: #8201
Final multimodal PR with our recent developments on MM side by @yaoyu-33 :: PR: #8127
Add include_text parameter to SFT dataloaders by @Kipok :: PR: #8198
Add random_seed argument to generate by @Kipok :: PR: #8162
Added support for neptune logger by @harishankar-gopalan :: PR: #8210
Pre-compute max_seqlen and cu_seqlens_argmin in all model-parallel cases by @erhoo82 :: PR: #8222
Use PackedSeqParams in accordance with changes in Megatron-LM by @cuichenx :: PR: #8205
Fix to peft & virtual pipeline parallel unsupported check by @vysarge :: PR: #8216
Fixed the tp overlap switch by @sanandaraj5597 :: PR: #8195
add knobs for rope/swiglu fusion by @lhb8125 :: PR: #8184
Added sample cpu_offloading switch to YAML by @sanandaraj5597 :: PR: #8148
Syncing random seed between ranks in generate by @Kipok :: PR: #8230
add first_val_step to mcore scheduler by @JimmyZhang12 :: PR: #8150
Correct padding for SFT input data to account for sequence parallel + TE's fp8 op dimension requirements by @vysarge :: PR: #8240
Mistral 7b conversion script by @akoumpa :: PR: #8052
switch to mcore dataset [with FIM support] by @dimapihtar :: PR: #8149
Mixtral to NeMo conversion script. by @akoumpa :: PR: #8155
fixes to accomendate mcore changes by @HuiyingLi :: PR: #8261
Allow MegatronPretrainingRandomSampler to do multi-epoch training by @trias702 :: PR: #8239
Add dist ckpt support for regular optimizers by @mikolajblaz :: PR: #7749
add deallocate pipeline output optimization by @JimmyZhang12 :: PR: #8279
Fix memory leak caused by context parallelism hanging references by omegaconf by @JimmyZhang12 :: PR: #8299
distributed fused adam + rampup bs support by @dimapihtar :: PR: #8302
Update PEFT Doc by @cuichenx :: PR: #8262
Converter script fixes for mixtral/mistral by @akoumpa :: PR: #8272
Keep max_seqlen and cu_seqlens_argmin for later micro-batches when PP>1 by @erhoo82 :: PR: #8334
Enable megatron core loggers for GPT pretraining by @ashbhandare :: PR: #8354
mcore ds fix by @dimapihtar :: PR: #8283
release updates by @dimapihtar :: PR: #8378
Mcore customization doc by @HuiyingLi :: PR: #8298
updated link to pubmed by @nithinraok :: PR: #8402
mcore customization doc minor fix by @HuiyingLi :: PR: #8421
Fixing mcore bert for TP, PP and SP by @shanmugamr1992 :: PR: #8336
Add settings to suppress bf16 compile errors in CI on V100 by @athitten :: PR: #8481
MoE parameter passing by @akoumpa :: PR: #8255
Add fp8 support for SD/Update notebook paths by @Victor49152 :: PR: #8489

</details>

NeMo Tools

<details><summary>Changelog</summary>

SDE bugfix log by @Jorjeous :: PR: #8430

</details>

General Improvements

<details><summary>Changelog</summary>

Add news section to README by @ericharper :: PR: #7984
Fixing conversion script to work for code llama by @shanmugamr1992 :: PR: #7997
Fix crash when converting to mcore a model using rotary embeddings by @odelalleau :: PR: #7998
Added a procedure for Windows users, README by @Jorjeous :: PR: #7942
Update manifest.py to speedup loading tarred datasets by @stevehuang52 :: PR: #7900
[Fix] Fixed name of a test by @anteju :: PR: #7986
Fix lora merge script by @cuichenx :: PR: #8113
Support transcoding audio formats when saving tarred datasets (FLAC, OPUS) by @pzelasko :: PR: #8102
README edit to change Apple Silicon install instructions (to fix a break introduced by pytorch 2) by @stephenmcconnachie :: PR: #8122
Fixes NVIDIA/apex installation to not erroneously install the pkg by @terrykong :: PR: #8126
Graphviz fix by @GNroy :: PR: #7843
Update README.rst by @fayejf :: PR: #8154
Fix TP>1 issue for conversion script by @cuichenx :: PR: #8144
Support torch jit script by @artbataev :: PR: #8027
NeMo Multimodal Docs and Tests Initial PR by @yaoyu-33 :: PR: #8028
Remove left-over prints in NeMo+Lhotse code by @pzelasko :: PR: #8180
Upgrade to DLFW PyTorch 23.12 by @ericharper :: PR: #8163
Add Lhotse support for key in NeMo manifests by @pzelasko :: PR: #8197
Fix CPU Initialization and TP>1 for LoRA Merge Script by @cuichenx :: PR: #8199
Add support in Neural Typecheck to disable semantic checks by @titu1994 :: PR: #8212
Pin lhotse=1.19.2 in r1.23.0 by @pzelasko :: PR: #8303
Multimodal r1.23.0 bug fix by @yaoyu-33 :: PR: #8315
MCore dataset compatibility for tokenizers by @vysarge :: PR: #8390
Update NFA video download link by @erastorgueva-nv :: PR: #8406
Update MM Dataprep Tutorial by @cuichenx :: PR: #8410
Fix dreambooth data sampler issue by @yaoyu-33 :: PR: #8400
Fix a bug in CTM line processing function for multi-speaker data simulations by @tango4j :: PR: #8416
Akoumparouli/mistral bugfix by @akoumpa :: PR: #8353
pin to 0.5.0 by @ericharper :: PR: #8465
Update NeMo Multimodal Requirements by @yaoyu-33 :: PR: #8515
Fix link in multimodal dataprep tutorial by @cuichenx :: PR: #8517

</details>

Open source →

1.22.010 Jan 2024

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet/

Highlights

Models

NeMo Parakeet

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet/

https://huggingface.co/nvidia/parakeet-rnnt-1.1b
https://huggingface.co/nvidia/parakeet-ctc-1.1b
https://huggingface.co/nvidia/parakeet-rnnt-0.6b
https://huggingface.co/nvidia/parakeet-ctc-0.6b

NeMo Parakeet-TDT

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet-tdt/

https://huggingface.co/nvidia/parakeet-tdt-1.1b

ASR

stt_en_fastconformer_transducer_large_ls #7641
stt_en_fastconformer_ctc_larg_ls #7641
stt_en_fastconformer_hybrid_large_streaming_multi
stt_nl_fastconformer_hybrid_large_pc
stt_fa_fastconformer_hybrid_large

NeMo ASR

Multi-lookahead cache-aware streaming Conformer #6711
Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) by @burchim #7330
Speech ehancement tutorial #6492
Support punctuation error rate #7538

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.10

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Fix missing pip package 'einops' by @RobinDong :: PR: #7397
Fix failure of installing pyaudio in Online_Offline_Speech_Commands_Demo.ipynb by @RobinDong :: PR: #7396
[ASR] Confidence measure -> method renames by @GNroy :: PR: #7434
RNN-T confidence and alignment bugfix by @GNroy :: PR: #7381
Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) by @burchim :: PR: #7330
[TTS] Read audio as int32 to avoid flac read errors by @rlangman :: PR: #7477
Fix typos in confidence tutorial notebooks by @Kipok :: PR: #7581
Safeguard nemo_text_processing installation on ARM by @blisc :: PR: #7485
add fc large ls models by @nithinraok :: PR: #7641
[ASR] RNN-T greedy decoding max_frames fix for alignment and confidence by @GNroy :: PR: #7635
Create per.py by @ssh-meister :: PR: #7538
Update docs: readme, getting started, ASR intro by @erastorgueva-nv :: PR: #7679
[ASR] Multichannel mask estimator with flex number of channels by @anteju :: PR: #7317
Fix code block typo in docs by @erastorgueva-nv :: PR: #7717
Replace gpus with devices by @athitten :: PR: #7743
docs: fix typos by @shuoer86 :: PR: #7758
Snake act by @nithinraok :: PR: #7736
fix(clustering_diarizer.py): fix typo by @jqueguiner :: PR: #7772
Add some docs and update scripts for ASR by @titu1994 :: PR: #7790
remove TN from ctc_segm tut by @ekmb :: PR: #7807
Add support for finetuning with huggingface datasets by @stevehuang52 :: PR: #7834
Adding long-form audio speaker diarization (clustering) class and functions by @tango4j :: PR: #7737
Fix k2 installation: update for latest PyTorch, move script to dir by @artbataev :: PR: #7887
[ASR] GSS-based mask estimator by @anteju :: PR: #7849
add Dutch P&C FC model info by @zhehuaichen :: PR: #7892
Add checks for unit tests that are looking for data from CI machine by @ericharper :: PR: #7943
update branch name by @nithinraok :: PR: #7990
fix librosa display issue by @nithinraok :: PR: #7991
Fixes Notebooks for ASR by @titu1994 :: PR: #7994
cherry pick bug 4405781 by @karpnv :: PR: #8044
fix noise augmentation by @stevehuang52 :: PR: #8056
Fix various issues with broken links and bugs by @titu1994 :: PR: #8064
run with non-dev option by @nithinraok :: PR: #8077
update broken links by @nithinraok :: PR: #8079
langid bug fix by @karpnv :: PR: #8134

</details>

TTS

<details><summary>Changelog</summary>

Add steps for document of getting dataset 'SF Bilingual Speech' by @RobinDong :: PR: #7378
Fix checking of cuda/cpu device for inputs of Decoder by @RobinDong :: PR: #7444
Fix failure of ljspeech's get_data.py by @RobinDong :: PR: #7430
[TTS] Fix audio codec type checks by @rlangman :: PR: #7373
[TTS] Add dataset to path of logged artifacts by @rlangman :: PR: #7462
Fix adding positional embeddings in-place in FFTransformerDecoder by @The0nix :: PR: #7440
Add dataset 'AISHELL-3' from OpenSLR for training mandarin TTS by @RobinDong :: PR: #7409
[TTS] Fix FastPitch data prep tutorial by @rlangman :: PR: #7524
add italian tokenization by @GiacomoLeoneMaria :: PR: #7486
Remap speakers to continuous range of speaker_id for dataset AISHELL3 by @RobinDong :: PR: #7536
add ItalianPhonemesTokenizer by @GiacomoLeoneMaria :: PR: #7587
[TTS] Add STFT and SI-SDR loss to audio codec recipe by @rlangman :: PR: #7468
Fix typo in audio codec config, encoder target by @anteju :: PR: #7697
Group-residual vector quantizer by @anteju :: PR: #7643
French g2p with pronunciation dictionary by @mgrafu :: PR: #7601
add pleasefixme marker for potential failed nightly tests. by @XuesongYang :: PR: #7678
Add new text segmentation library for better TTS quality by @RobinDong :: PR: #7645
ConditionalInput: cat along the feature dim, not the batch dim by @anferico :: PR: #7785
Add selection criteria for reference audios in the submodule by @anferico :: PR: #7788
[Codec] Update codec checkpoint config by @anteju :: PR: #7835
[Codec] Finite scalar quantizer by @anteju :: PR: #7886
Tar codec by @nithinraok :: PR: #7867

</details>

LLM

<details><summary>Changelog</summary>

Allow disabling sanity checking when num_sanity_val_steps=0 by @athitten :: PR: #7413
Add comprehensive error messages by @PeganovAnton :: PR: #7261
layer selection for ia3 by @arendu :: PR: #7417
Add rope dynamic linear scaling by @hsiehjackson :: PR: #7437
Fix sft dataset truncation by @hsiehjackson :: PR: #7464
fix bug when loading dist ckpt in peft by @lhb8125 :: PR: #7452
Fix sft chat dataset truncation by @hsiehjackson :: PR: #7478
SFT model parallel fix for dist ckpt by @aklife97 :: PR: #7511
remove auto generated examples by @arendu :: PR: #7510
Add the argument to by @odelalleau :: PR: #7264
PEFT GPT & T5 Refactor by @meatybobby :: PR: #7308
fix a typo by @BestJuly :: PR: #7496
StarCoder SFT test + bump PyT NGC image to 23.09 by @janekl :: PR: #7540
fix llama2 70b lora tuning bug by @cuichenx :: PR: #7622
generalized chat sft prompt by @yidong72 :: PR: #7655
Set base frequency from config by @shan18 :: PR: #7734
Megatron LLM documentation updates by @ssh-meister :: PR: #7400
Remove incorrect extra argument of load_from_checkpoint_dir() by @RobinDong :: PR: #7500
Add nemo to mcore GPT conversion script by @cuichenx :: PR: #7730
set context for text memmap to fork by @arendu :: PR: #7784
Support flash decoding by @hsiehjackson :: PR: #7744
update text server to support compute logprobs by @Zhilin123 :: PR: #7733
Revert PEFT eval fix by @ericharper :: PR: #7693
Fix tn duplex by @ekmb :: PR: #7808
Multimodal merge by @yaoyu-33 :: PR: #7728
Fix flash decoding precision by @hsiehjackson :: PR: #7852
Removing duplicate Megatron-LM installation by @Davood-M :: PR: #7864
adding special_tokens from tokenizer config for transformer-lm model by @clumsy :: PR: #7613
Add Adapter and IA3 support for MCore models by @cuichenx :: PR: #7750
Add back import guard by @cuichenx :: PR: #7882
Change FP8 Defaults by @cuichenx :: PR: #7894
Added knob for ub_tp_comm_overlap for the MCORE pass by @sanandaraj5597 :: PR: #7902
Upgrade NeMo to latest mcore and TE by @dimapihtar :: PR: #7862
Pad sequences to multiples of 16 for GPTSFTDataset by @vysarge :: PR: #7904
upgrade to latest mcore and TE by @dimapihtar :: PR: #7908
added missing torch import by @Davood-M :: PR: #7913
Fix CPU initialization of GPT models by @cuichenx :: PR: #7889
Fix pinned triton version by @hsiehjackson :: PR: #7925
fix tp_overlap config var name by @xrennvidia :: PR: #7928
only enable query key scaling during fp16 by @gshennvm :: PR: #7946
Fix for gpt3 eval hang with PP (a dtype issue) by @yaoyu-33 :: PR: #7927
Pass in rotary_base to mcore and from HF by @Kipok :: PR: #7933
Use NLPDDPStrategyNotebook in Multitask_Prompt_and_PTuning.ipynb by @athitten :: PR: #8061

</details>

General Improvements

<details><summary>Changelog</summary>

Add fix for max time to quit trainer gracefully, without running validation by @SeanNaren :: PR: #7731
SDE Tutorial minor fix by @Jorjeous :: PR: #7598
Temporary pin Lightning-Utilities version due to broken NamedTuple by @artbataev :: PR: #8022
Karpnv/issue 7320 by @karpnv :: PR: #7418
Speech Simulator, update README.md: output_path --> output_manifest_filepath by @popcornell :: PR: #7442
Fix None dataloader issue in PTL2.0 by @KunalDhawan :: PR: #7455
HF StarCoder to NeMo conversion script by @janekl :: PR: #7421
[doc] fix broken link by @stas00 :: PR: #7481
dllogger - log on rank 0 only by @stas00 :: PR: #7513
Add two youtube introductory videos to README and Docs. by @XuesongYang :: PR: #7570
defaults changed by @arendu :: PR: #7600
Bound transformers version in requirements by @athitten :: PR: #7620
Fix import error no module name model_utils by @menon92 :: PR: #7629
Fix in the confidence ensemble test by @Kipok :: PR: #7682
move core install to /workspace by @aklife97 :: PR: #7706
distributed checkpoint average script by @yidong72 :: PR: #7721
fix hybrid eval by @karpnv :: PR: #7757
fix(diarization-README): typo by @jqueguiner :: PR: #7771
Configure MCore logger by @mikolajblaz :: PR: #7781
Nemo to HF converter for LLaMA model by @uppalutkarsh :: PR: #7770
[Fix] Save best NeMo model only when necessary by @anteju :: PR: #7836
add guard if its a distributed checkpoint by @gshennvm :: PR: #7845
Update transformers cache on Jenkins by @ericharper :: PR: #7854
Update README.rst for container update by @fayejf :: PR: #7844
Fix mcore conversion bug by @cuichenx :: PR: #7846
add comment on script and fix target check by @gshennvm :: PR: #7881
fix issues with convert_nemo_llama_to_hf.py by @Zhilin123 :: PR: #7922
Instructions for running ci on pr template by @ericharper :: PR: #7944
Distributed checkpoint averaging supports bf16 type by @yidong72 :: PR: #7888
Fix tokenizer argparse in scripts by @titu1994 :: PR: #8012
Check dependencies in installation script by @artbataev :: PR: #8019
[SE Tutorial] USe GPU for inference, when available by @anteju :: PR: #8048
update reqs by @ericharper :: PR: #8072
Remove typo by @ericharper :: PR: #8146

</details>

Open source →

1.21.025 Oct 2023Deprecated

deprecation warning by @arendu :: PR: #7193

Highlights

Models

NeMo ASR

Multi-lookahead cache-aware streaming
Speech enahncement tutorial #6492
Online code switching dataset #6579

NeMo TTS

AudioCodec: Training recipe for EnCodec #6852

NeMo Framework

GPT from Mcore #7093
GPT distributed checkpointing #7116
Hidden transformations #6332
LLama-2 #7299

NeMo Core

Update to PTL 2.0 #6433

NeMo Tools

Forced aligner tutorial #7210

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.08

ASR

<details><summary>Changelog</summary>

Fix require_grad typos by @kit1980 :: PR: #6930
rnnt_greedy_decoding.py: typos? auto-repressively -> auto-regressively by @vadimkantorov :: PR: #6989
Adding tutorial for confidence ensembles by @Kipok :: PR: #6932
Add support for Numba FP16 RNNT Loss by @titu1994 :: PR: #6991
fix install_beamsearch_decoders by @karpnv :: PR: #7011
rnnt and char utils by @karpnv :: PR: #6971
ASR Confidence update and tutorial by @GNroy :: PR: #6810
st standalone model by @AlexGrinch :: PR: #6969
Fix typo in ASR-TTS tutorial by @artbataev :: PR: #7049
Update Frame-VAD doc and fix onnx export by @stevehuang52 :: PR: #7076
Fast Conformer global token fix by @sam1373 :: PR: #7085
Added script to extract ASR CTC and RNNT models from ASR hybrid models by @trias702 :: PR: #7092
Fix absolute path in path join call by @kingjan1999 :: PR: #7099
NeMo ASR Demo by @lleaver :: PR: #7110
Fix plot function in vad_utils.py by @stevehuang52 :: PR: #7113
Fixed small bug with NoisePerturbationWithNormalization by @trias702 :: PR: #7118
Merge release r1.20.0 to main by @ericharper :: PR: #7167
minor fix for conformer subsampling docstring. by @XuesongYang :: PR: #7195
[ASR] Fix GPU memory leak in transcribe_speech.py by @rlangman :: PR: #7249
Adding Multilingual, Code-Switched, and Hybrid ASR models by @KunalDhawan :: PR: #7250
fix partial transcribe by @stevehuang52 :: PR: #7284
Conv1d subsampling by @burchim :: PR: #7294
add bf16 inference support and fix seq_len stft issue by @nithinraok :: PR: #7338
Add finetuning scripts by @nithinraok :: PR: #7263
Move parameter: trainer -> exp_manager (for PTL 2.0) by @artbataev :: PR: #7339
Fix typos by @omahs :: PR: #7361
Fix wrong calling of librosa.get_duration() in notebook by @RobinDong :: PR: #7376
RNN-T confidence and alignment bugfix (#7381) by @GNroy :: PR: #7459
update branch by @nithinraok :: PR: #7488
Replace strategy = None with strategy = auto for notebooks by @athitten :: PR: #7521
Fix PTL2.0 related ASR bugs in r1.21.0: Val metrics logging, None dataloader issue by @KunalDhawan :: PR: #7531
gpus -> devices by @nithinraok :: PR: #7542
[BugFix] Add missing quotes for auto strategy in tutorial notebooks by @athitten :: PR: #7541
Append output of val_step to self.validation_step_outputs in EncMaskDecAudioToAudioModel by @athitten :: PR: #7543
fix validation_step_outputs initialization for multi-dataloader by @KunalDhawan :: PR: #7546
Append val/test output to instance variable in EncDecSpeakerLabelModel by @athitten :: PR: #7562
update strategy by @nithinraok :: PR: #7577
Typo fixes by @Kipok :: PR: #7591
Fix metrics for SE tutorial by @anteju :: PR: #7604
fix ssl models ptl monitor val through logging by @nithinraok :: PR: #7608
Fix py3.11 dataclasses issue by @titu1994 :: PR: #7582
bugfix: trainer.gpus, trainer.strategy, trainer.accelerator by @XuesongYang :: PR: #7621
Safeguard nemo_text_processing installation on ARM (#7485) by @blisc :: PR: #7619
[ASR] Fix type error in jasper by @rlangman :: PR: #7636
Fix vad & speech command tutorial - onnx by @fayejf :: PR: #7671
Replace strategy='dp'/None with 'auto' by @athitten :: PR: #7681
Fix multi rank finetune for ASR by @titu1994 :: PR: #7684
fix ptl_bugs in slu_models.py by @jzi040941 :: PR: #7689
Add NLPDDPStrategyNotebook and change trainer gpus to devices by @athitten :: PR: #7741
Updated installation of ctc-decoders by @vsl9 :: PR: #7746
Fix bug wrt change decoding strategy for bpe models by @titu1994 :: PR: #7762

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Add cosine distance option to TTS aligner by @rlangman :: PR: #6806
[TTS] Add tutorial for TTS data prep scripts by @rlangman :: PR: #6922
update TTS readme by @XuesongYang :: PR: #7088
[TTS] Create EnCodec training recipe by @rlangman :: PR: #6852
[TTS][ZH] add Chinese TTS recipes based on IPA symbol sets. by @XuesongYang :: PR: #6893
[TTS] Add output audio format to preprocessing by @rlangman :: PR: #6889
[TTS] Remove nested TTS configs by @rlangman :: PR: #7154
[TTS] Fix TTS recipes with PTL 2.0 by @rlangman :: PR: #7188
[TTS] Add license to ported EnCodec code by @rlangman :: PR: #7197
[Fix] Discriminator update in AudioCodecModel by @anteju :: PR: #7209
Adapter ipa Tutorial and config update by @styagi130 :: PR: #7260
[TTS] Audio codec fixes by @rlangman :: PR: #7266
[TTS] minor fix typos and input_types by @XuesongYang :: PR: #7272
specify explicitly to set pretrained model paths by @styagi130 :: PR: #7305
[TTS] Update AudioCodec API by @anteju :: PR: #7310
[TTS] Add additional config to preprocess_text and compute_feature_stats by @rlangman :: PR: #7321
[TTS] Change audio codec token type to TokenIndex by @rlangman :: PR: #7356
fixed trainer.strategy=auto from None. by @XuesongYang :: PR: #7369
[TTS] Added a callback for logging initial data by @anteju :: PR: #7384
[TTS] bugfix: trainer.accelerator=auto from None. by @XuesongYang :: PR: #7492
bugfix: specify trainer.strategy=auto when devices=1 by @XuesongYang :: PR: #7509
Fix dimensionality in get_dist function by @redoctopus :: PR: #7506
Fix TTS FastPitch tutorial by @hsiehjackson :: PR: #7494
[TTS] remove curly braces from in jupyer notebook cell. by @XuesongYang :: PR: #7554
[TTS] fixed trainer's accelerator and strategy. by @XuesongYang :: PR: #7569
Change hifigan finetune strategy to ddp_find_unused_parameters_true by @hsiehjackson :: PR: #7579
Fix validation in G2PModel and ThutmoseTaggerModel by @athitten :: PR: #7597
[TTS] Fix FastPitch data prep tutorial by @rlangman :: PR: #7602
[TTS] Add dataset to path of logged artifacts by @rlangman :: PR: #7651

</details>

NLP / NMT

<details><summary>Changelog</summary>

Minor MPT-7B fixes and creation script update by @trias702 :: PR: #6982
remove hard coded input and output fields by @arendu :: PR: #7008
RoPE length extrapolation with interpolation by @MaximumEntropy :: PR: #7005
add async + distopt to sft by @MaximumEntropy :: PR: #7018
ptuning inference table bug fix by @arendu :: PR: #7015
Fix missing import for GPT SFT by @MaximumEntropy :: PR: #7026
Add end_strings to SamplingParams by @markelsanz14 :: PR: #6986
Fix race condition for downloading cache when executing with multi-node by @findkim :: PR: #7016
added back the retro documents. by @yidong72 :: PR: #7033
remove pos emb from state dict for old models by @ekmb :: PR: #7068
memmap worker arg by @arendu :: PR: #7062
Disable distopt contiguous param buffer by default by @timmoon10 :: PR: #7095
[Fix] load_state_dict in nlp_model.py by @stevehuang52 :: PR: #7086
Fix tokenizer file caching where torch.distributed may not be initialized yet by @findkim :: PR: #7061
freeze base mode on init during peft by @arendu :: PR: #7152
Include the scripts for preprocessing OAST and unit tests for chat sft datasets by @yidong72 :: PR: #7112
T5 metrics fix by @jubick1337 :: PR: #7037
megatron gpt training fix by @anmolgupt :: PR: #7199
Fix T5 using FA by @hsiehjackson :: PR: #7196
fix-causal-fa-infer by @hsiehjackson :: PR: #7200
Fix gpt trainer test by @hsiehjackson :: PR: #6915
Load ub_cfg from hydra config by @jbaczek :: PR: #7003
Fixes for lightning 2.0 upgrade by @athitten :: PR: #7176
Fix which was off by one batch by @odelalleau :: PR: #7212
Start using ModelParallelConfig from Megatron Core by @ericharper :: PR: #6885
deprecation warning by @arendu :: PR: #7193
Fix attention mask inference by @hsiehjackson :: PR: #7213
Use GPTModel from mcore by @ericharper :: PR: #7093
Add bf16-mixed and 16-mixed in module.py by @athitten :: PR: #7227
Refactor LLM pretraining examples by @maanug-nv :: PR: #7159
Add only trainable parameters to optimizer group in PEFT by @guyueh1 :: PR: #7230
Dummy class for ModelParallelConfig by @ericharper :: PR: #7254
[TN][Docs] update language coverage matrix and refs by @mgrafu :: PR: #7247
tied weights for adapters by @arendu :: PR: #6928
Fix skip generation by @hsiehjackson :: PR: #7270
Hidden transforms model parallel config + CI with Perceiver by @michalivne :: PR: #7241
Fix restore sequence parallel by @hsiehjackson :: PR: #7273
fix ptuning and lora model_parallel_config by @blahBlahhhJ :: PR: #7287
Fix adapters and ptuning for amp O2 by @guyueh1 :: PR: #7285
remove additional line in peft state dict by @blahBlahhhJ :: PR: #7293
loss mask aware final layer applicaiton by @arendu :: PR: #7275
Adding server option to peft eval by @Davood-M :: PR: #7292
migrated class CSVFieldsMemmapDataset from BioNeMo by @dorotat-nv :: PR: #7314
remove old prompt table for storing cached ptunig representations by @arendu :: PR: #7295
Bugfix and optimization in by @odelalleau :: PR: #7267
Set a default value when getting by @yaox12 :: PR: #7115
Distributed checkpointing with mcore GPT by @ericharper :: PR: #7116
Fix activation checkpoint by @hsiehjackson :: PR: #7334
Replace prefetch with val iterator check in megatron models by @athitten :: PR: #7318
Fixing indentation bug in indexed_dataset memory deallocation by @michalivne :: PR: #7352
NeMo MCore llama2 support + MCore PEFT adapters by @blahBlahhhJ :: PR: #7299
Hiddens modules documentation by @michalivne :: PR: #7303
Support for flash attention 2.0 by @MaximumEntropy :: PR: #7063
multiple fields can form a context by @arendu :: PR: #7147
adding bias_dropout_add_fusion option for BERT by @clumsy :: PR: #7332
enable selective unfreeze by @arendu :: PR: #7326
Upgrade pytorch container to 23.08 by @ericharper :: PR: #7353
enable fp32 optimizer for output_layer in mcore by @lhb8125 :: PR: #7355
Revert comment by @ericharper :: PR: #7368
fix pipeline parallel inference by @blahBlahhhJ :: PR: #7367
fix for peft tied weights by @arendu :: PR: #7372
add O2 option in gpt eval by @blahBlahhhJ :: PR: #7358
Move model precision copy by @maanug-nv :: PR: #7336
Fix PEFT checkpoint loading by @blahBlahhhJ :: PR: #7388
Use distributed optimizer support for multiple dtypes by @timmoon10 :: PR: #7359
[PATCH] PEFT import mcore by @blahBlahhhJ :: PR: #7393
Use cfg attribute in bert by @maanug-nv :: PR: #7394
Add support for bias conversion in Swiglu models by @titu1994 :: PR: #7386
Update save_to and restore_from for dist checkpointing by @ericharper :: PR: #7343
fix forward for with mcore=false by @JimmyZhang12 :: PR: #7403
Fix logging to remove 's/it' from progress bar in Megatron models and add train_step_timing by @athitten :: PR: #7374
Set Activation Checkpointing Defaults by @aklife97 :: PR: #7404
Make loss mask default to false by @ericharper :: PR: #7407
Add dummy userbuffer config files by @erhoo82 :: PR: #7408
Add missing ubconf files by @aklife97 :: PR: #7412
Update ptl training ckpt conversion script to work with dist ckpt by @ericharper :: PR: #7416
Add strategy as ddp_find_unused_parameters_true for glue_benchmark.py by @athitten :: PR: #7454
fix bug when loading dist ckpt in peft by @lhb8125 :: PR: #7479
Fix CustomProgressBar for resume by @athitten :: PR: #7427
Append val output to self.validation_step_outputs in GLUEModel by @athitten :: PR: #7530
Cherry pick Fix sft dataset truncation (#7464) to r1.21.0 by @ericharper :: PR: #7550
Avoid duplicated dist checkpoint save by @mikolajblaz :: PR: #7555
layernorm1p fix by @dimapihtar :: PR: #7523
r1.21: SFT model parallel fix for dist ckpt by @aklife97 :: PR: #7520
PEFT needs mp config propagated for dist ckpt by @ericharper :: PR: #7589
Fix ptuning crash for llama 2 ckpt by @yuanzhedong :: PR: #7594
PEFT eval fix by @cuichenx :: PR: #7626
Propagate mp config for continue training by @ericharper :: PR: #7637
Add ddp_find_unused_parameters=True and change accelerator to auto by @athitten :: PR: #7623
Add find_unused_parameters_true for text_classiftn and punctuation_capitalization by @athitten :: PR: #7649
conversion issue fix by @dimapihtar :: PR: #7648
Fix a nlp nb onnx by @fayejf :: PR: #7703
Add activations_checkpoint related args for model cfg in lora.ipynb by @athitten :: PR: #7752
Change accelerator to 'auto' in nlp_checkpoint_port.py by @athitten :: PR: #7747
Add reconfigure microbatch calculator before inference and update GBS, MBS for inference by @athitten :: PR: #7763
Create PrecisionPlugin for megatron_ckpt_to_nemo.py trainer by @athitten :: PR: #7767

</details>

NeMo Tools

<details><summary>Changelog</summary>

Update doc, new tutorial on SDE by @Jorjeous :: PR: #7405
Fix branch version for SDE by @titu1994 :: PR: #7528

</details>

Export

<details><summary>Changelog</summary>

Added bool types to neural_types export by @tbartley94 :: PR: #7032

</details>

General Improvements

<details><summary>Changelog</summary>

Add migration guide for lightning 2.0 upgrade by @athitten :: PR: #7360
add support for max_total_length=4096 for 43b by @Zhilin123 :: PR: #6763
Change Jenkins timeout by @ericharper :: PR: #6997
Update SDP docs page with a new documentation link by @Kipok :: PR: #7029
Fixed tutorial's name by @vsl9 :: PR: #7047
Revert Fix import guard checks by @titu1994 :: PR: #7125
Fix import guard checks by @titu1994 :: PR: #7126
fix evaluator.py for various exceptions by ast by @stevehuang52 :: PR: #7150
NFA bugfix: remove any empty segments by @erastorgueva-nv :: PR: #7155
NFA subtitle file config - specify colors and vertical alignment by @erastorgueva-nv :: PR: #7160
add paths to labeler. by @XuesongYang :: PR: #7087
[Bugfix] Fix a bug in filtering checkpoints by @yaox12 :: PR: #6851
Update README.rst by @fayejf :: PR: #7175
Make NFA subtitles stay until end of video by @erastorgueva-nv :: PR: #7189
Uncomment removal of exp_dir in JenkinsFile by @athitten :: PR: #7198
NFA: replace ellipses in text with 3 periods by @erastorgueva-nv :: PR: #7208
NFA tutorial notebook by @erastorgueva-nv :: PR: #7210
NFA docs: update READMEs and links, add docs page by @erastorgueva-nv :: PR: #7219
Make image centering in NFA README actually work by @erastorgueva-nv :: PR: #7220
Add mcore installation to Dockerfile by @ericharper :: PR: #7237
Checkpoint averaging for model parallel by @Kipok :: PR: #7252
Upgrade hydra and omegaconf by @athitten :: PR: #7243
Update numba support in docker by @titu1994 :: PR: #7271
remove deprecated scripts from ci by @arendu :: PR: #7239
Logging model checkpoints as artifacts in MlFlow by @AlirezaMorsali :: PR: #7258
Adithyare/peft metric calculation by @arendu :: PR: #7304
Resume checkpoint priority by @maanug-nv :: PR: #7335
lora merge fix for O2 names by @arendu :: PR: #7325
Llama load buffers in checkpoint by @blahBlahhhJ :: PR: #7357
pin numba=0.57.1 to fix reinstall.sh error by @XuesongYang :: PR: #7366
Update to core 23.08 branch ToT by @aklife97 :: PR: #7371
Upper bounding ptl by @ericharper :: PR: #7370
minor fix for llama ckpt conversion script by @blahBlahhhJ :: PR: #7387
Update Core Commit by @aklife97 :: PR: #7402
Fix resume from checkpoint in exp_manager by @athitten :: PR: #7424
add sleep by @gshennvm :: PR: #7498
Fix exp manager check for sleep by @titu1994 :: PR: #7503
unpin setuptools by @fayejf :: PR: #7534
Update FFMPEG version to fix issue with torchaudio by @titu1994 :: PR: #7551
fix typos in nfa and speech enhancement tutorials by @erastorgueva-nv :: PR: #7580
best ckpt fix by @dimapihtar :: PR: #7564
add build os key by @nithinraok :: PR: #7596
Fix issues with Dockerfile by @titu1994 :: PR: #7650
Change confidence parameters in the test by @Kipok :: PR: #7680
bugfix: pin nemo-text-process to fix Chinese normalizer error. by @XuesongYang :: PR: #7627
Remove PUBLICATIONS.md, point to github.io NeMo page instead by @erastorgueva-nv :: PR: #7694
Pin mcore to 0.3 by @ericharper :: PR: #7751
fix hybrid eval by @karpnv :: PR: #7759
Update Apex install command in Dockerfile by @ericharper :: PR: #7794

</details>

Open source →

1.20.04 Aug 2023Deprecated

[TTS] corrected misleading deprecation warnings. by @XuesongYang :: PR: #6702

Highlights

Models

STT En Fast Conformer CTC XXLarge - 1.2 B param Fast Conformer CTC
STT En Fast Conformer Transducer XXLarge - 1.2 B param Fast Conformer Transducer
STT En Fast Conformer Transducer XLarge - XLarge Fast Conformer English
STT En Fast Conformer CTC XLarge - XLarge Fast Conformer CTC
STT En Fast Conformer Transducer XLarge - XLarge Fast Conformer Transducer
STT En Fast Conformer CTC Large - Large Fast Conformer CTC
STT En Fast Conformer Transducer Large - Large Fast Conformer Transducer
STT It Fast Conformer Hybrid Large P&C - Large P&C Italian Fast Conformer
STT Ua Fast Conformer Hybrid Large P&C - Large Ukranian Fast Conformer

NeMo ASR

Graph-RNN-T #6168
WildCard-RNN-T #6168
Confidence Ensembles for ASR
Token-and-Duration Transducer (TDT) #6536
Spellchecking ASR #6179
Numba FP16 RNNT Loss #6991

NeMo TTS

TTS Adapter Customization
TTS Dataloader Framework

NeMo Framework

LoRA for T5 and mT5 #6612
Flash Attention integration #6666
Mosaic 7B compatibility
Models with LongContext (32K) #6666, #6687, #6773

NeMo Tools

Speech Data Explorer: Utterance level ASR model comparsion #6669
Speech Data Processor: Spanish P&C
NeMo Forced Aligner: Large sequence alignment + memory reduction #6695

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.06

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

[ASR] Adding ssl config for fast-conformer by @krishnacpuvvada :: PR: #6672
Fix for interctc test random failure by @Kipok :: PR: #6644
sharded manifests docs by @bmwshop :: PR: #6751
[TTS] Implement new vocoder dataset by @rlangman :: PR: #6670
TDT model pull request by @hainan-xv :: PR: #6536
Spec aug fix by @tbartley94 :: PR: #6775
Support large inputs to Conformer and Fast Conformer by @bmwshop :: PR: #6556
sharded manifests updated docs by @bmwshop :: PR: #6833
added fc-xl, xxl and titanet-s models by @nithinraok :: PR: #6832
Multi-lookahead cache-aware streaming models by @VahidooX :: PR: #6711
Update transcribe_utils.py by @stevehuang52 :: PR: #6865
Fix k2 build topo helper by @artbataev :: PR: #6887
Fix transcribe_utils.py for hybrid models in partial transcribe mode by @stevehuang52 :: PR: #6899
Add hybrid model support to transcribe_speech_parallel.py by @stevehuang52 :: PR: #6906
Update Frame-VAD doc by @stevehuang52 :: PR: #6902
Make sure asr_model.change_attention_model is run if either cfg.model_path or cfg.pretrained_name is specified by @erastorgueva-nv :: PR: #6908
Update fvad doc by @stevehuang52 :: PR: #6920
Online Code Switching Dataset for ASR by @trias702 :: PR: #6579
Fix AN4 dataset links by @artbataev :: PR: #6926
Fix confidence ensembles RNNT logprobs selection logic for exclude_blank scenario by @KunalDhawan :: PR: #6937
Adding cache-aware streaming ASR checkpoints. by @VahidooX :: PR: #6940
Remove from metrics by @titu1994 :: PR: #6979
Hybrid conformer export by @borisfom :: PR: #6983
Cache handling without input tensors mutation by @borisfom :: PR: #6980
Fixing an issue with confidence ensembles by @Kipok :: PR: #6987
Add ASR with TTS Tutorial. Fix enhancer usage. by @artbataev :: PR: #6955
fix install_beamsearch_decoders.sh by @karpnv :: PR: #7019
Add support for Numba FP16 RNNT Loss (#6991) by @titu1994 :: PR: #7038
Fix typo and branch in tutorial by @artbataev :: PR: #7048
Refined export_config by @borisfom :: PR: #7053
Fix documentation for Numba by @titu1994 :: PR: #7065
Adding docs and models for multiple lookahead cache-aware ASR by @VahidooX :: PR: #7067
Add updated fc ctc and rnnt xxl models by @nithinraok :: PR: #7128
Update notebook branch by @ericharper :: PR: #7135
Fixed main and merging this to r1.20 by @tango4j :: PR: #7127
Fix default context size by @nithinraok :: PR: #7141
Fix incorrect embedding grads with distopt BF16 grad reductions by @timmoon10 :: PR: #6958

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Add callback for saving audio during FastPitch training by @rlangman :: PR: #6665
[TTS] Add script for text preprocessing by @rlangman :: PR: #6541
[TTS] Fix adapter duration issue by @hsiehjackson :: PR: #6697
[TTS] Filter out silent audio files during preprocessing by @rlangman :: PR: #6716
[TTS] fix inconsistent type hints for IpaG2p by @XuesongYang :: PR: #6733
[TTS] relax hardcoded prefix for phonemes and tones and infer phoneme set through dict by @XuesongYang :: PR: #6735
[TTS] corrected misleading deprecation warnings. by @XuesongYang :: PR: #6702
Fix TTS adapter tutorial by @hsiehjackson :: PR: #6741
[TTS][zh] refine hardcoded lowercase for ASCII letters. by @XuesongYang :: PR: #6781
[TTS] Append pretrained FastPitch & SpectrogamEnhancer pair to available models by @racoiaws :: PR: #7012

</details>

NLP / NMT

<details><summary>Changelog</summary>

minor fix for missing chat attr by @arendu :: PR: #6671
eval fix by @arendu :: PR: #6685
VP Fixes for converter + Config management by @titu1994 :: PR: #6698
lora notebook by @arendu :: PR: #6765
peft eval directly from ckpt by @arendu :: PR: #6785
GPT inference long context by @ekmb :: PR: #6687
Fix validation with drop_last=False by @mikolajblaz :: PR: #6704
fix spellmapper tutorial, change branch to main by @bene-ges :: PR: #6803
text_generation_utils memory reduction if no logprob needed by @yzhang123 :: PR: #6773
Add optional index mapping dir in mmap text datasets by @gheinrich :: PR: #6683
Add inference kv cache support for transformer TE path by @yen-shi :: PR: #6627
add reference to our paper by @bene-ges :: PR: #6821
added changes to ramp up bs by @dimapihtar :: PR: #6799
t5 lora tuning by @arendu :: PR: #6612
Added rouge monitoring support for T5 by @jubick1337 :: PR: #6737
GPT extrapolatable position embedding (xpos/sandwich/alibi/kerple) and Flash Attention by @hsiehjackson :: PR: #6666
Import Enum for chatbot component by @ericharper :: PR: #6877
typo fix from #6666 by @arendu :: PR: #6882
removed unnecessary print by @dimapihtar :: PR: #6884
Fix destructor for delayed mmap dataset case by @mikolajblaz :: PR: #6703
Make Gradio library optional by @yidong72 :: PR: #6904
Fix fast-glu activation in change partitions by @hsiehjackson :: PR: #6909
Documentation for ONNX export of Megatron Models by @asfiyab-nvidia :: PR: #6914
FixTextMemMapDataset index file creation in multi-node setup by @gheinrich :: PR: #6768
Fix flash-attention by @hsiehjackson :: PR: #6901
ptuning oom fix by @arendu :: PR: #6916
add rampup bs assertion by @dimapihtar :: PR: #6927
Enable methods in bert-like models by @sararb :: PR: #6898
support value attribution condition by @yidong72 :: PR: #6934
Add missing save restore connector to eval scripts by @titu1994 :: PR: #6935
Merge release r1.19.0 into main by @ericharper :: PR: #6948
Stop at the stop token by @yidong72 :: PR: #6957
fixes for spellmapper by @bene-ges :: PR: #6994
Fix tabular data text generation by @yidong72 :: PR: #7022
fix pos id - hf update by @ekmb :: PR: #7075
fix syntax error introduced in PR-7079 by @bene-ges :: PR: #7102

</details>

NeMo Tools

<details><summary>Changelog</summary>

SDE unt lvl comparison by @Jorjeous :: PR: #6669
hot fix SDE by @Jorjeous :: PR: #6897

</details>

Bugfixes

<details><summary>Changelog</summary>

small Bugfix by @fayejf :: PR: #7079
Fix caching bug in causal convolutions for cache-aware ASR models by @VahidooX :: PR: #7034
Fix masking bug for TTS Aligner by @redoctopus :: PR: #6677
[bugfix] avoid the random shuffle of phoneme and tone tokens. by @XuesongYang :: PR: #6855
fix ptuning residuals bug by @arendu :: PR: #6866
TE bug fix by @dimapihtar :: PR: #7027
Update distopt API for coalesced NCCL calls by @timmoon10 :: PR: #6886

</details>

General Improvements

<details><summary>Changelog</summary>

update batch size recommendation to min 32 for 43b by @Zhilin123 :: PR: #6675
Make Note usage consistent in adapter_mixins.py by @BrianMcBrayer :: PR: #6678
Update all invalid tree references to blobs for NeMo samples by @BrianMcBrayer :: PR: #6679
Update README.rst about container by @fayejf :: PR: #6686
karpnv/issues6690 by @karpnv :: PR: #6705
Limit codeql scope by @titu1994 :: PR: #6710
Not pinning Gradio version by @yidong72 :: PR: #6680
preprocess squad in sft format by @arendu :: PR: #6727
Fix Codeql config by @titu1994 :: PR: #6731
Fix fastpitch test nightly by @hsiehjackson :: PR: #6730
Lora/PEFT training script CI test by @arendu :: PR: #6664
fixed decor to show messages only when the wrapped object is called. by @XuesongYang :: PR: #6793
lora pp2 by @arendu :: PR: #6818
Upperbound Numpy to < 1.24 by @titu1994 :: PR: #6829
Fix typo in documentation by @Dounx :: PR: #6838
NFA updates by @erastorgueva-nv :: PR: #6695
Update container for import action by @ericharper :: PR: #6883
removed some tests by @arendu :: PR: #6900
Update container info in README.rst by @fayejf :: PR: #6913
Removed optional optimize_for_inference by @borisfom :: PR: #6933
Update core commit for CI by @aklife97 :: PR: #6939
lora inference ci by @arendu :: PR: #6931
Upgrade base pytorch container to 23.06 by @ericharper :: PR: #6938
Fix requirements for pydantic + inflect by @titu1994 :: PR: #6956
Remove pyyaml by @titu1994 :: PR: #7052
Fix links in Segmentation tutorial by @ekmb :: PR: #7117
Update evaluator.py by @stevehuang52 :: PR: #7151

</details>

Open source →

1.19.015 Jun 2023

Sharded Manifests for Tarred Datasets #6395

Highlights

NeMo ASR

Sharded Manifests for Tarred Datasets #6395
Frame-VAD model + datasets support #6441
Noise Norm Perturbation #6445
Code Switched Dataset with IID Sampling #6448

NeMo TTS

Speaker adaptation for FastPitch #6416, #6417

NeMo Megatron

Batch size rampup #6424
Unify dataset and model classes for all PEFT #6391
LoRA for GPT #6391
Convert interleaved pipeline model to non-interleaved #6498
Dialog Dataset for SFT #6654
Dynamic length batches for GPT SFT #6510
Merge LoRA weights into base model #6597

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.04

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Sharded manifests for tarred datasets by @bmwshop :: PR: #6395
Update script for ngram rnnt and hat beam search decoding by @andrusenkoau :: PR: #6370
Add disclaimer about dataset for ASR by @titu1994 :: PR: #6496
New noise_norm perturbation based on Riva work by @trias702 :: PR: #6445
Add Frame-VAD model and datasets by @stevehuang52 :: PR: #6441
removing unnecessary avoid_bfloat16_autocast_context by @bmwshop :: PR: #6481
FC models in menu by @bmwshop :: PR: #6473
Separate punctuation by whitespace by @karpnv :: PR: #6574
Cherry pick commits in #6601 to main by @fayejf :: PR: #6611
Offline and streaming inference support for hybrid model by @fayejf :: PR: #6570
Disable interctc tests by @Kipok :: PR: #6638
ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. by @artbataev :: PR: #6620
Confidence ensembles implementation by @Kipok :: PR: #6614
Confidence ensembles: fix issues and add tuning functionality by @Kipok :: PR: #6657
Add support for RNNT/hybrid models to partial transcribe by @stevehuang52 :: PR: #6609
eval_beamsearch_ngram.py with hybrid ctc by @karpnv :: PR: #6656

</details>

TTS

<details><summary>Changelog</summary>

[TTS] FastPitch adapter fine-tune and conditional layer normalization by @hsiehjackson :: PR: #6416
[TTS] whitelist broken path fix. by @XuesongYang :: PR: #6412
[TTS] FastPitch speaker encoder by @hsiehjackson :: PR: #6417
Update NeMo_TTS_Primer.ipynb by @pythinker :: PR: #6436
[TTS] Create functions for TTS preprocessing without dataloader by @rlangman :: PR: #6317
[TTS] Fix FastPitch energy code by @rlangman :: PR: #6511
[TTS] Add script for computing feature stats by @rlangman :: PR: #6508
[TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters by @hsiehjackson :: PR: #6431
[TTS] Create initial TTS dataset feature processors by @rlangman :: PR: #6507
[TTS] Add script for mapping speaker names to indices by @rlangman :: PR: #6509
[TTS] Implement new TextToSpeech dataset by @rlangman :: PR: #6575

</details>

NLP / NMT

<details><summary>Changelog</summary>

Add patches for Virtual Parallel conversion by @titu1994 :: PR: #6589
Update wfst_text_normalization.rst by @jimregan :: PR: #6374
add rampup batch size support for Megatron GPT by @dimapihtar :: PR: #6424
Add interleaved pp support by @titu1994 :: PR: #6498
Support dynamic length batches with GPT SFT by @aklife97 :: PR: #6510
Framework for PEFT via mixins by @arendu :: PR: #6391
Add GPT eval mode fix for interleaved to main (#6449) by @aklife97 :: PR: #6610
sft model can use this script for eval by @arendu :: PR: #6637
Patch memory used for NeMo Megatron models by @titu1994 :: PR: #6615
merge lora weights into base model by @arendu :: PR: #6597
Dialogue dataset by @yidong72 :: PR: #6654
check for first or last stage by @ericharper :: PR: #6708
A few small typo fixes by @Kipok :: PR: #6599
Lddl bert by @wdykas :: PR: #6761
Debug Transformer Engine FP8 support with Megatron-core infrastructure by @timmoon10 :: PR: #6740
Tensor-parallel communication overlap with userbuffer backend by @erhoo82 :: PR: #6780
Add ub communicator initialization to validation step by @erhoo82 :: PR: #6807
Add trainer.validate example for GPT by @ericharper :: PR: #6794
Add API docs for NeMo Megatron by @ericharper :: PR: #6850
Apply garbage collection interval to validation steps by @erhoo82 :: PR: #6870

</details>

Bugfixes

<details><summary>Changelog</summary>

[BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator by @tango4j :: PR: #6499
small bugfix for asr_evaluator by @fayejf :: PR: #6636
fix bucketing bug issue for picking new bucket by @nithinraok :: PR: #6663
[TTS] Fix TTS audio preprocessing bugs by @rlangman :: PR: #6628
Fix a bug, use _ceil_to_nearest instead as _round_to_nearest is not d… by @BestJuly :: PR: #6681
Bug fix to restore act ckpt by @markelsanz14 :: PR: #6753
Bug fix to reset sequence parallelism by @markelsanz14 :: PR: #6756
Bug fix for reset_sequence_parallel_args by @markelsanz14 :: PR: #6802
Fix adapter tutorial r1.19.0 by @hsiehjackson :: PR: #6776
Fix error appearing when using tar datasets by @Jorjeous :: PR: #6502
Fix normalization of impulse response in ImpulsePerturbation by @anteju :: PR: #6505
Fix typos by @titu1994 :: PR: #6523
Fix notebook bad json by @titu1994 :: PR: #6561
[ASR] Fix for old models in change_attention_model by @sam1373 :: PR: #6608
Fix k2 installation in Docker with CUDA 12 by @artbataev :: PR: #6707
Tutorial fixes by @titu1994 :: PR: #6717
Vp fixes by @titu1994 :: PR: #6738
[TTS] Fix aligner nan loss in fp32 by @hsiehjackson :: PR: #6435
fix conversion and eval by @arendu :: PR: #6648
Fix checkpointed forward and add test for full activation checkpointing by @aklife97 :: PR: #6744
add call to p2p overlap by @aklife97 :: PR: #6779
Fix get_parameters when using main params optimizer by @ericharper :: PR: #6764
Fix GPTDataset Assert by @MaximumEntropy :: PR: #6798
fix notebook error by @yidong72 :: PR: #6840
final fix of notebook by @yidong72 :: PR: #6842

</details>

General Improvements

<details><summary>Changelog</summary>

Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format by @KunalDhawan :: PR: #6448
Fix an invalid link in get_data.py of ljspeech by @pythinker :: PR: #6456
Update manifest.py to use os.path for get_full_path by @stevehuang52 :: PR: #6598
Cherry pick commits in #6528 to main by @timmoon10 :: PR: #6613
Move black parameters to pyproject.toml by @artbataev :: PR: #6647
handle artifacts when path is an extracted dir by @arendu :: PR: #6658
remove upgrading setuptools in reinstall.sh by @XuesongYang :: PR: #6659
Upgrade to PyTorch 23.04 Container by @ericharper :: PR: #6660
Fix fastpitch test nightly by @hsiehjackson :: PR: #6742
Fix Links for tutorials by @titu1994 :: PR: #6777
Update core version in Jenkinsfile by @aklife97 :: PR: #6817
Update mcore requirement to 0.2.0 by @ericharper :: PR: #6875

</details>

Open source →

1.18.012 May 2023

GPT-2B-001, trained on 1.1T tokens with 4K sequence length.

Highlights

Models

NeMo ASR

Hybrid Autoregressive Transducer (HAT) #6260
Apple MPS Support for ASR Inference #6289
InterCTC Support for Hybrid ASR Models #6215
RNNT N-Gram Fusion with mAES algo #6118
ASR + Apple M2 CPU/GPU MPS #6289

NeMo TTS

TTS directory structure refactor
User-set symbol vocabulary #6172

NeMo Megatron

Model parallelism from Megatron Core #6393
Continued training for P-tuning #6273
SFT for GPT-3 #6210
Tensor and pipeline model parallel conversion #6218
Megatron NMT Export to Riva

NeMo Core

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

minor cleanup by @messiaen :: PR: #6311
docs on the use of heterogeneous test / val manifests by @bmwshop :: PR: #6352
[WIP] add buffered chunked streaming for nemo force aligner by @Slyne :: PR: #6185
Word boosting for Flashlight decoder by @trias702 :: PR: #6367
Add installation and ASR inference instructions for Mac by @artbataev :: PR: #6377
specaug speedup by @1-800-BAD-CODE :: PR: #6347
updated lr for FC configs by @bmwshop :: PR: #6379
Make possible to control tqdm progress bar in ASR models by @SN4KEBYTE :: PR: #6375
[ASR] Conformer global tokens in local attention by @sam1373 :: PR: #6253
fixed torch warning on using a list of numpy arrays by @MKNachesa :: PR: #6382
Fix FastConformer config: correct bucketing strategy by @artbataev :: PR: #6413
fixing the ability to use temp sampling with concat datasets by @bmwshop :: PR: #6423
add conformer configs for hat model by @andrusenkoau :: PR: #6372
[ASR] Add optimization util for linear sum assignment algorithm by @tango4j :: PR: #6349
Added/updated new Conformer configs by @VahidooX :: PR: #6426
Fix typos by @titu1994 :: PR: #6494
Fix typos (#6523) by @titu1994 :: PR: #6539
added back the fast emit section to the configs. by @VahidooX :: PR: #6540
Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY by @KunalDhawan :: PR: #6549
Add scores for FastConformer models by @titu1994 :: PR: #6557
Patch transcribe and support offline transcribe for hybrid model by @fayejf :: PR: #6550
More streaming conformer export fixes by @messiaen :: PR: #6567
Documentation for ASR-TTS models by @artbataev :: PR: #6594
Patch transcribe_util for steaming mode and add wer calculation back to inference scripts by @fayejf :: PR: #6601
Add HAT image to docs by @andrusenkoau :: PR: #6619
Patch decoding for PC models by @titu1994 :: PR: #6630
Fix wer.py where 'errors' variable was not set by @stevehuang52 :: PR: #6633
Fix for old models in change_attention_model by @VahidooX :: PR: #6635

</details>

TTS

<details><summary>Changelog</summary>

VITS HiFiTTS doc by @treacker :: PR: #6288
fix broken links r1.18.0 by @ekmb :: PR: #6501
[TTS] fixed broken path. by @XuesongYang :: PR: #6514

</details>

NLP / NMT

<details><summary>Changelog</summary>

[Core] return_config=True now extracts just config, not full tarfile by @titu1994 :: PR: #6346
restore path for p-tuning by @arendu :: PR: #6273
taskname and early stopping for adapters by @arendu :: PR: #6366
Adapter tuning accepts expanded language model dir by @arendu :: PR: #6376
Update gpt_training.rst by @blisc :: PR: #6378
Megatron GPT model finetuning by @MaximumEntropy :: PR: #6210
[NeMo Megatron] Cleanup configs to infer the models TP PP config automatically by @titu1994 :: PR: #6368
Fix prompt template unescaping by @MaximumEntropy :: PR: #6399
Add support for Megatron GPT Untied Embd TP PP Change by @titu1994 :: PR: #6388
Move Parallelism usage from Apex -> Megatron Core by @aklife97 :: PR: #6393
Add ability to enable/disable act ckpt and seq parallelism in GPT by @markelsanz14 :: PR: #6327
Refactor PP conversion + add support for TP only conversion by @titu1994 :: PR: #6419
fix CPU overheads of GPT synthetic dataset by @xrennvidia :: PR: #6427
check if grad is none before calling all_reduce by @arendu :: PR: #6428
Fix replace_bos_with_pad not found by @aklife97 :: PR: #6443
Support Swiglu in TP PP Conversion by @titu1994 :: PR: #6437
BERT pre-training mp fork to spawn by @aklife97 :: PR: #6442
Meagtron encoder decoder fix for empty validation outputs by @michalivne :: PR: #6459
Reduce workers on NMT CI by @aklife97 :: PR: #6472
Switch to NVIDIA Megatron repo by @aklife97 :: PR: #6465
Megatron KERPLE positional embeddings by @michalivne :: PR: #6478
Support in external sample mapping for Megatron datasets by @michalivne :: PR: #6462
Fix custom by @aklife97 :: PR: #6512
GPT fp16 inference fix by @MaximumEntropy :: PR: #6543
Fix for T5 FT model by @aklife97 :: PR: #6529
Pass instead of scaler object to core by @aklife97 :: PR: #6545
Change Megatron Enc Dec model to use persistent_workers by @aklife97 :: PR: #6548
Turn autocast off when precision is fp32 by @aklife97 :: PR: #6554
Fix batch size reconf for T5 FT for multi-validation by @aklife97 :: PR: #6582
Make tensor split contiguous for qkv and kv in attention by @aklife97 :: PR: #6580
Patches from main to r1.18.0 for Virtual Parallel by @titu1994 :: PR: #6592
Create dummy iters to satisy iter type len checks in core + update core commit by @aklife97 :: PR: #6600
Restore GPT support for interleaved pipeline parallelism by @timmoon10 :: PR: #6528
Add megatron_core to requirements by @ericharper :: PR: #6639

</details>

Export

<details><summary>Changelog</summary>

</details>

Bugfixes

<details><summary>Changelog</summary>

Fix the GPT SFT datasets loss mask bug by @yidong72 :: PR: #6409
[BugFix] Fix multi-processing bug in data simulator by @tango4j :: PR: #6310
Fix cache aware hybrid bugs by @VahidooX :: PR: #6466
[BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… by @tango4j :: PR: #6500
Fixing bug in unsort_tensor by @borisfom :: PR: #6320
Bugfix for BF16 grad reductions with distopt by @timmoon10 :: PR: #6340
Limit urllib3 version to patch issue with RTD by @aklife97 :: PR: #6568

</details>

General improvements

<details><summary>Changelog</summary>

Pin the version to hopefully fix rtd build by @SeanNaren :: PR: #6334
enabling diverse datasets in val / test by @bmwshop :: PR: #6306
extract inference weights by @arendu :: PR: #6353
Add opengraph support for NeMo docs by @titu1994 :: PR: #6380
Adding basic preemption code by @athitten :: PR: #6161
Add documentation for preemption support by @athitten :: PR: #6403
Update hyperparameter recommendation based on experiments by @Zhilin123 :: PR: #6405
exceptions with empty test / val ds config sections by @bmwshop :: PR: #6421
Upgrade pt 23.03 by @ericharper :: PR: #6430
Update README to add core installation by @aklife97 :: PR: #6488
Not doing CastToFloat by default by @borisfom :: PR: #6524
Update manifest.py for speedup by @stevehuang52 :: PR: #6565
Update SDP docs by @erastorgueva-nv :: PR: #6485
Update core commit hash in readme by @aklife97 :: PR: #6622
Remove from jenkins by @ericharper :: PR: #6641
Remove dup by @ericharper :: PR: #6643

</details>

Open source →

1.17.04 Apr 2023Deprecated

[TTS] deprecate AudioToCharWithPriorAndPitchDataset. by @XuesongYang :: PR: #5959

Highlights

NeMo ASR

Online Clustering Diarizer
High Level Diarization API
PyCTC Decode Beam Search Support
RNNT Beam Search Alignment Extraction
InterCTC Loss
AIStore Documentation
ASR & AWS Multi-node Integration
Convolution Invariant SDR losses

NeMo TTS

NeMo Megatron

SqaredReLU, SwiGLU, No-Dropout
Rotary Position Embedding
Untie word embeddings and output projection

NeMo Core

Dynamic freezing of modules during training
NeMo Multi-Run Documentation
ClearML Logging
Early Stopping
Experiment Manager Docs Update

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.02

Detailed Changelogs

ASR

<details><summary>Changelog</summary>

Support Alignment Extraction for all RNNT Beam decoding methods by @titu1994 :: PR: #5925
Use module-based k2 import guard by @artbataev :: PR: #6006
Default RNNT loss to int64 targets by @titu1994 :: PR: #6011
Added documentation section for ASR datasets from AIStore by @anteju :: PR: #6008
Change perturb rng for reproducing results easily by @fayejf :: PR: #6042
InterCTC loss and stochastic depth implementation by @Kipok :: PR: #6013
Add pyctcdecode to high level beam search API by @titu1994 :: PR: #6026
Convert esperanto into a notebook by @SeanNaren :: PR: #6070
[ASR] Added a script for evaluating metrics for audio-to-audio by @anteju :: PR: #5971
[ASR] Convolution-invariant SDR loss + unit tests by @anteju :: PR: #5992
Adjust stochastic depth dropout probability calculation by @anteju :: PR: #6120
Add file class based inference API for diarization by @SeanNaren :: PR: #5945
Ngram by @karpnv :: PR: #6063
remove duplicate definition of manifest read and write func. by @XuesongYang :: PR: #6088
Streaming conformer CTC export by @messiaen :: PR: #5837
[TTS] Make mel spectrogram norm configurable by @rlangman :: PR: #6155
Ngram lm fusion for RNNT maes decoding by @andrusenkoau :: PR: #6118
ASR Beam search documentation by @titu1994 :: PR: #6244

</details>

TTS

<details><summary>Changelog</summary>

[TTS][ZH] added new NGC model cards with polyphone disambiguation. by @XuesongYang :: PR: #5940
[TTS] deprecate AudioToCharWithPriorAndPitchDataset. by @XuesongYang :: PR: #5959
[TTS][G2P] deprecate add_symbols by @XuesongYang :: PR: #5961
Added list_available_models by @treacker :: PR: #5967
Update Fastpitch energy bug by @blisc :: PR: #5969
removed WHATEVER(1) ˌhwʌˈtɛvɚ from scripts/tts_dataset_files/ipa_cmudict-0.7b_nv22.10.txt by @MikyasDesta :: PR: #5869
ONNX export for RadTTS by @borisfom :: PR: #5880
Add some info about FastPitch SSL model by @redoctopus :: PR: #5994
Vits doc by @treacker :: PR: #5989
Ragged batching changes for RadTTS, some refactoring by @borisfom :: PR: #6020
Working enabled ragged batching with ONNX by @borisfom :: PR: #6030
[TTS/TN/G2P] Remove Text Processing from NeMo, move G2P to TTS by @ekmb :: PR: #5982
[TTS] Add Spanish IPA dictionaries and heteronyms by @rlangman :: PR: #6037
[TTS] Separate TTS tokenization and g2p util to fix circular import by @rlangman :: PR: #6080
[TTS][refactor] Part 7 - move module from model file. by @XuesongYang :: PR: #6098
[TTS][refactor] Part 1 - nemo.collections.tts.data by @XuesongYang :: PR: #6099
[TTS][refactor] Part 2 - nemo.colletions.tts.parts by @XuesongYang :: PR: #6105
[TTS][refactor] Part 6 - remove nemo.collections.tts.torch.README.md and tts_dataset.yaml by @XuesongYang :: PR: #6103
[TTS][refactor] Part 3 - nemo.collections.tts.g2p.models by @XuesongYang :: PR: #6113
[TTS] update German NGC models trained on Thorsten Datasets by @XuesongYang :: PR: #6125
[TTS] remove old waveglow model that relies on torch_stft. by @XuesongYang :: PR: #6128
[TTS] Move Spanish polyphones from heteronym to dictionary by @rlangman :: PR: #6123
[TTS][refactor] Part 8 - added model inference tests to safeguard changes. by @XuesongYang :: PR: #6129
remove duplicate definition of manifest read and write func. by @XuesongYang :: PR: #6088
[TTS][refactor] update tutorial import paths. by @XuesongYang :: PR: #6176
[TTS] Add univnet scheduler by @ArtyomZemlyak :: PR: #6157
[TTS] Make mel spectrogram norm configurable by @rlangman :: PR: #6155

</details>

NLP / NMT

<details><summary>Changelog</summary>

add new lannguages to doc by @yzhang123 :: PR: #5939
Distributed Adam optimizer overlaps param all-gather with forward compute by @timmoon10 :: PR: #5684
Refactor the retrieval services for microservice architecture by @yidong72 :: PR: #5910
make validation accuracy reporting optional for adapters/ptuning by @arendu :: PR: #5843
Add BERT support for overlapping forward compute with distopt communication by @timmoon10 :: PR: #6024
[TTS/TN/G2P] Remove Text Processing from NeMo, move G2P to TTS by @ekmb :: PR: #5982
adding early stop callback to ptuning by @arendu :: PR: #6028
Pr doc tn by @yzhang123 :: PR: #6041
Adds several configurable flags for Megatron GPT models by @MaximumEntropy :: PR: #5991
P-tuning refactor Part 1/N by @arendu :: PR: #6054
Fast glu activations by @MaximumEntropy :: PR: #6058
P-tuning refactor Part 2/N by @arendu :: PR: #6056
P-tuning refactor Part 3/N by @arendu :: PR: #6106
Explicitly check for united embeddings when logging params by @MaximumEntropy :: PR: #6085
Add flag to get attention from fusion by @ericharper :: PR: #6049
Improving text memmap generated index files error messages by @michalivne :: PR: #6093
Megatron Encoder-Decoder Sampler Function by @michalivne :: PR: #6095
Sentence piece legacy false compatibility by @arendu :: PR: #6154
convert Megatron LM ckpt to NeMo PP support. by @yidong72 :: PR: #6159
Avoid multiple warnings for loss mask by @mikolajblaz :: PR: #6062
Propagate LayerNorm1P to TE by @mikolajblaz :: PR: #6061
Filter p-tuning by example length by @arendu :: PR: #6182
Add sequence parallel support to Rope positional embedding by @yidong72 :: PR: #6178
Use a separate communicator for DP AMAX reduction by @erhoo82 :: PR: #6022
Add persistent workers to GPT by @ericharper :: PR: #6205
Micro batch loader for bert model by @shanmugamr1992 :: PR: #6046
GPT P tuning Eval changes (#5952) by @aklife97 :: PR: #6272
add template for taskname=taskname by @Zhilin123 :: PR: #6283
added RPE + fixed RMSNorm by @Davood-M :: PR: #6304
simplified notebook for p-tuning by @arendu :: PR: #6326
Added num decoder blocks in megatron export by @Davood-M :: PR: #6331

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

[TTS/TN/G2P] Remove Text Processing from NeMo, move G2P to TTS by @ekmb :: PR: #5982

</details>

Export

<details><summary>Changelog</summary>

ONNX export for RadTTS by @borisfom :: PR: #5880
Working enabled ragged batching with ONNX by @borisfom :: PR: #6030
Update docs for ExpManager and Exportable frameworks by @titu1994 :: PR: #6165
Streaming conformer CTC export by @messiaen :: PR: #5837
MixedFusedRMSNorm Export Fix by @Davood-M :: PR: #6296
Added num decoder blocks in megatron export by @Davood-M :: PR: #6331

</details>

Bugfixes

<details><summary>Changelog</summary>

Fix bug where GPT always enabled distopt overlapped param sync by @timmoon10 :: PR: #5995
CS bugfix by @bmwshop :: PR: #6122
RNNT patch by @titu1994 :: PR: #6231
Notebook fixes by @titu1994 :: PR: #6212
Small fixes for flashlight decoder by @trias702 :: PR: #6071
Various fixes in docs and RNNT by @titu1994 :: PR: #6156
Fix k2 and torchaudio installation (Docker, macOS) by @artbataev :: PR: #6094
update and deprecate warning for Mic notebook by @fayejf :: PR: #6307
small bugfix and add asr evaluator to doc by @fayejf :: PR: #6229
Bug fixing for bucketing dataset by @VahidooX :: PR: #6191
Fix character beam decoding algorithm with vocab index map by @titu1994 :: PR: #6140
fix typo in asr evaluator readme by @fayejf :: PR: #6053
Fix typos by @titu1994 :: PR: #6241
[ASR]:fixed augmentor arguments for transcribe functionality of Hybrid CTC-RNNT model by @KunalDhawan :: PR: #6290
Fix hybrid transcribe by @ArtyomZemlyak :: PR: #6003
Fix buckeing seeding by @VahidooX :: PR: #6254
Fix for CTC decoder setup by @vsl9 :: PR: #6303
Fix RNNT Joint narrow() by @titu1994 :: PR: #6336
Fix bugs with interctc mixin by @Kipok :: PR: #6228
Update IPA dict path in tutorial by @redoctopus :: PR: #6208
[TTS] fix broken tutorial for Tacotron2 by @XuesongYang :: PR: #6199
[TTS] fix bugs for chinese and german tutorials. by @XuesongYang :: PR: #6216
Fix radtts sort r17 by @borisfom :: PR: #6344
Quick Fix for RadTTS test by @blisc :: PR: #6034
Disabling radtts tests untin we have real model by @borisfom :: PR: #6036
fix val loss computation in megatron by @anmolgupt :: PR: #5871
Fix incomplete batches by @mikolajblaz :: PR: #6083
Avoid unnecessarily accessing data loader with pipeline parallelism by @timmoon10 :: PR: #6164
bugfix: file handlers are not closed. by @XuesongYang :: PR: #5956
Fix Silence Sampling Algorithm for ASR Multi-speaker Data Simulator by @stevehuang52 :: PR: #5897
Fix Windows bug with save_restore_connector by @trias702 :: PR: #5919
fix broken link by @ericharper :: PR: #5968
Fix torchaudio installation by @artbataev :: PR: #5850
Fix reinstall.sh dependencies by @titu1994 :: PR: #6027
Adding changes to fix the mv error by @tango4j :: PR: #6087
Fix README by @flx42 :: PR: #6137
Fix typos in voiceapp notebook by @titu1994 :: PR: #6262
[BugFix] Fix diarization result path errors in tutorial notebook for r1.17.0 by @tango4j :: PR: #6234
[BugFix] Fix the wrong branch name in speaker diarization inference notebook by @tango4j :: PR: #6301

</details>

General Improvements

<details><summary>Changelog</summary>

Dynamic freezing in Nemo by @trias702 :: PR: #5879
Move settings to . Remove deprecated by @artbataev :: PR: #5947
update container info in readme by @fayejf :: PR: #5981
Update PUBLICATIONS.md by @titu1994 :: PR: #5963
[G2P] backward compatibility for english tokenizer and bugfix by @github-actions[bot] :: PR: #5984
replace symbols by @github-actions[bot] :: PR: #5990
correct bash style according to SC2236. by @XuesongYang :: PR: #6025
Update align.py by @github-actions[bot] :: PR: #6045
Add Customization Dataset Preparation Tool by @Zhilin123 :: PR: #6029
Updated data simulator config part in Speaker_Diarization_Training.ipynb by @tango4j :: PR: #6072
Add citation by @ericharper :: PR: #6077
[TTS] Spectrogram Enhancer: correct dim for length when loading data by @github-actions[bot] :: PR: #6074
Add ClearML Logging by @ArtyomZemlyak :: PR: #6014
update readme with new badges by @XuesongYang :: PR: #6110
[CI] Set readthedocs python version to 3.8 by @SeanNaren :: PR: #6079
Update dataset preparation tool to fix bug relating to non jsonl input file by @Zhilin123 :: PR: #6147
update finetune configs by @nithinraok :: PR: #6152
Added ckpt to nemo for T5/T0 models by @Davood-M :: PR: #6141
Save model parallel .nemo in ExpManager by @arendu :: PR: #6115
Upgrade setuptools by @fayejf :: PR: #6163
Update container version in main readme by @fayejf :: PR: #6171
metric update by @arendu :: PR: #6169
Upgrade base container to PyTorch 23.02 by @ericharper :: PR: #6162
Link to nm launcher by @ericharper :: PR: #6226
Make AIS CLI installation optional by @anteju :: PR: #6314
remove pinned numba version in Dockerfile by @fayejf :: PR: #6341
Cherry-pick recent distopt commits by @timmoon10 :: PR: #6343
Update readme by @ericharper :: PR: #6363

</details>

Open source →

1.16.08 Mar 2023

Multi-channel dereverberation algorithm

Highlights

NeMo ASR

ASR Evaluator
Multi-channel dereverberation algorithm
Hybrid ASR-TTS Models
Flashlight Decoder Beam Search
FastConformer Encoder with 8x subsampling

NeMo TTS

SSL Voice Conversion
Spectrogram Enhancer
VITS

NeMo Megatron

Per microbatch dataloader for GPT and BERT
Adapters compatible with Faster Transformer

NeMo Core

Nested model support

NeMo Tools

NeMo Forced Aligner

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.01

ASR

<details><summary>Changelog</summary>

Fix for incorrect computation of batched alignment in transducers by @Kipok :: PR: #5692
Set the stream position to 0 for pydub by @jonghwanhyeon :: PR: #5752
[Fix] ConformerEncoder forward when length is None by @anteju :: PR: #5761
ASR evaluator by @fayejf :: PR: #5728
[ASR][Test] Enable test for cache audio with a single worker by @anteju :: PR: #5763
Flashlight Decoder for Nemo by @trias702 :: PR: #5790
Fix data simulator by @stevehuang52 :: PR: #5813
[ASR] Mask-based dereverb algorithm by @anteju :: PR: #5693
Concat dataset and aistore support for label models by @Kipok :: PR: #5826
Adding new features and speed up for multi-speaker data simulator by @tango4j :: PR: #5846
Add Esperanto ASR example by @andrusenkoau :: PR: #5772
Fix memory allocation of NeMo Multi-speaker Data Simulator by @stevehuang52 :: PR: #5864
[ASR] Separate Audio-to-Text (BPE, Char) dataset construction by @artbataev :: PR: #5774
Reduce memory usage in getMultiScaleCosAffinityMatrix function by @gabitza-tech :: PR: #5876
Hybrid ASR-TTS models by @artbataev :: PR: #5659
Set providers for onnxruntime inference session by @athitten :: PR: #5903
[ASR] Configurable metrics for audio-to-audio + removed experimental decorators by @anteju :: PR: #5827
Correct doc for RNNT transcribe() function by @titu1994 :: PR: #5904
Update isort to the latest version by @artbataev :: PR: #5895
FilterbankFeaturesTA to match FilterbankFeatures by @msis :: PR: #5913
Fix hybridasr bug by @VahidooX :: PR: #5950
replace symbols by @nithinraok :: PR: #5974
fast conformer configs and doc by @bmwshop :: PR: #5970
Update TitaNet-L and MSDD models by @nithinraok :: PR: #6023
Fix enhancer usage by @artbataev :: PR: #6059
update librosa args by @nithinraok :: PR: #6086
Fix enhancer usage in ASR-TTS examples by @artbataev :: PR: #6116
Fix k2 and torchaudio installation (Docker, macOS). Cherry-pick (#6094) by @artbataev :: PR: #6124

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Update Spanish TTS model to 1.15 by @rlangman :: PR: #5742
[TTS][DE] refine grapheme-based tokenizer and fastpitch training recipe on thorsten's neutral datasets. by @XuesongYang :: PR: #5753
No-script TS export, prepared for ONNX export by @borisfom :: PR: #5653
Fixing masking in RadTTS bottleneck layer by @borisfom :: PR: #5771
Port Riva's mel cepstral distortion w/ dynamic time warping notebook by @redoctopus :: PR: #5778
Update radtts' infer path by @blisc :: PR: #5788
[TTS][DE] Augment tokenization/G2P to preserve capitalization of words and mix phonemes with word-level graphemes for an input text. by @XuesongYang :: PR: #5805
[TTS] porting VITS implementation by @treacker :: PR: #5600
[TTS][DE] updated IPA dictionary and heteronyms by @XuesongYang :: PR: #5860
[TTS] GAN-based spectrogram enhancer by @racoiaws :: PR: #5565
TTS inference with Heteronym classification model, hc model inference refactoring by @ekmb :: PR: #5768
Remove MCD_DTW tarball by @redoctopus :: PR: #5889
Hybrid ASR-TTS models by @artbataev :: PR: #5659
Moved eval notebook data to aws by @redoctopus :: PR: #5911
[G2P] fixed typos and broken import library. by @XuesongYang :: PR: #5978
[G2P] backward compatibility for english tokenizer and bugfix by @XuesongYang :: PR: #5980
fix links, add missing file by @ekmb :: PR: #6044
[TTS] Spectrogram Enhancer: correct dim for length when loading data by @racoiaws :: PR: #6048
[TTS] bugfix for fastpitch German tutorial by @XuesongYang :: PR: #6051
[TTS] bugfix Chinese Fastpitch tutorial by @XuesongYang :: PR: #6055
Fix enhancer usage by @artbataev :: PR: #6059
[TTS] Spectrogram Enhancer: support arbitrary input length by @racoiaws :: PR: #6060
Fix enhancer usage in ASR-TTS examples by @artbataev :: PR: #6116
[TTS] Spectrogram Enhancer: add option to zero out the initial tensor by @racoiaws :: PR: #6136
[TTS][DE] Augment tokenization/G2P to preserve capitalization of words and mix phonemes with word-level graphemes for an input text. by @XuesongYang :: PR: #5805

</details>

NLP / NMT

<details><summary>Changelog</summary>

Fix P-Tuning Truncation by @vadam5 :: PR: #5663
Adithyare/prompt learning seed by @arendu :: PR: #5749
Add extra data args to support proper finetuning of HF converted T5 checkpoints by @MaximumEntropy :: PR: #5719
Don't add output directory twice when creating shared sentencepiece tokenizer by @pks :: PR: #5737
add constraint info on batch size for tar dataset by @yzhang123 :: PR: #5812
remove transformer version upper bound by @Zhilin123 :: PR: #5831
Adithyare/adapter new placement by @arendu :: PR: #5791
Add SSL import functionality for Audio Lexical PNC Models by @trias702 :: PR: #5834
validation batch sizing and drop_last controls by @arendu :: PR: #5830
Remove ending newlines when encoding strings w/ sentencepiece tokenizer by @pks :: PR: #5739
Fix segmenting for pcla inference by @jubick1337 :: PR: #5849
RETRO model finetuning by @yidong72 :: PR: #5800
Optimizing distributed Adam when running with one work queue by @timmoon10 :: PR: #5560
Add option to disable distributed parameters in distributed Adam optimizer by @timmoon10 :: PR: #5685
set max_steps for lr decay through config by @anmolgupt :: PR: #5780
Fix Prompt text space issue by @aklife97 :: PR: #5983
Add batch_size to prompt_learning generate by @aklife97 :: PR: #6091

</details>

NeMo Tools

<details><summary>Changelog</summary>

[Tools] NeMo Forced Aligner by @erastorgueva-nv :: PR: #5571
[Tools] Fix ctc segmentation: exclude audacity files by @ekmb :: PR: #6009

</details>

Export

<details><summary>Changelog</summary>

No-script TS export, prepared for ONNX export by @borisfom :: PR: #5653
Set providers for onnxruntime inference session by @athitten :: PR: #5903
Add segmentation export to Audacity label file by @Ca-ressemble-a-du-fake :: PR: #5857

</details>

General Improvements

<details><summary>Changelog</summary>

Pin lightning version less than 1.9.0 by @SeanNaren :: PR: #5822
Davidm/cherrypick r1.16.0 by @Davood-M :: PR: #6082
Update files for lightning 1.9.0 by @SeanNaren :: PR: #5823
Tn doc 16 by @yzhang123 :: PR: #5954
Ensure EMA checkpoints are also deleted when normal checkpoints are by @SeanNaren :: PR: #5724
[Fix] ConformerEncoder forward when length is None by @anteju :: PR: #5761
Fix EMA topk checkpoint deletion by @SeanNaren :: PR: #5758
[BugFix] decoder timestamp count has a mismatch when <unk> is decoded by @tango4j :: PR: #5825
Update 00_NeMo_Primer.ipynb by @schaltung :: PR: #5740
Sanitize params before DLLogger log_hyperparams by @milesial :: PR: #5736
NeMo Forced Aligner by @erastorgueva-nv :: PR: #5571
Add EMA Docs, fix common collection documentation by @SeanNaren :: PR: #5757
Add container info to main page by @fayejf :: PR: #5816
CommonVoice support for script by @SeanNaren :: PR: #5797
Support nested NeMo models by @artbataev :: PR: #5671
fix max len generation t5 by @ekmb :: PR: #5852
NFA samples fix by @erastorgueva-nv :: PR: #5856
fix(readme): fix typo by @jqueguiner :: PR: #5883
Block large files from being merged into NeMo main by @SeanNaren :: PR: #5898
Pin isort version by @artbataev :: PR: #5914
fixed missing long_description_content_type by @XuesongYang :: PR: #5909
Update container to 23.01 by @ericharper :: PR: #5917
remove conda pynini install by @ekmb :: PR: #5921
Update align.py by @Slyne :: PR: #6043
Fixing data simulator argument and bash scripting error by @tango4j :: PR: #6112
Update apex commit by @ericharper :: PR: #6148

</details>

Open source →

1.15.01 Feb 2023

Greedy timestamp decoding with inference script

Highlights

NeMo ASR

HybridTransducer-CTC ASR
Greedy timestamp decoding with inference script
MHA adapters
Conformer local attention (longformer)
High level beam search API
Multiblank transducer
Multi-channel audio processing model
AIstore for ASR datasets

NeMo Megatron

ALiBi position embeddings support for T5

NeMo TTS

Chinese TTS pipeline with polyphone disambiguation

NeMo Core

Optimizer based EMA
MLFlow logger support

Models

stt_eo_conformer_ctc_large (HF, NGC) Esperanto ASR model.
stt_eo_conformer_transducer_large (HF, NGC) Esperanto ASR model.

Detailed Changelogs

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.12

ASR

<details><summary>Changelog</summary>

optimized loop and bugfix by @Jorjeous :: PR: #5573
Update torchmetrics by @nithinraok :: PR: #5566
Add an option to defer data setup from init to setup by @anteju :: PR: #5569
AIStore for ASR datasets by @anteju :: PR: #5462
Add support for MHA adapters to ASR by @titu1994 :: PR: #5396
Update documentation and tutorials for Adapters by @titu1994 :: PR: #5610
Conformer local attention by @sam1373 :: PR: #5525
Add core classes and functions for online clustering diarizer part 1 by @tango4j :: PR: #5526
[Add] ASR+VAD Inference Pipeline by @stevehuang52 :: PR: #5575
[ASR] Audio processing base, multi-channel enhancement models by @anteju :: PR: #5356
Expose ClusteringDiarizer device by @SeanNaren :: PR: #5681
Add Beam Search support to ASR transcribe() by @titu1994 :: PR: #5443
Multiblank Transducer by @hainan-xv :: PR: #5527
pin torchmetrics version by @nithinraok :: PR: #5720
Update torchaudio dependency version for tutorials by @titu1994 :: PR: #5781
update torchmetrics to latest version by @nithinraok :: PR: #5801
Fix transducer and question answering tutorial bugs bugs by @Zhilin123 :: PR: #5809
[BugFix] Updated CTC decoders installation in tutorial by @vsl9 :: PR: #5833
update torchmetrics args confusionmatrix by @nithinraok :: PR: #5853
indentation fix by @nithinraok :: PR: #5861
Fix wrong label mapping in batch_inference for label_model by @fayejf :: PR: #5767

</details>

TTS

<details><summary>Changelog</summary>

Add support for MHA adapters to ASR by @titu1994 :: PR: #5396
[TTS] fix ranges of char set for accented letters. by @XuesongYang :: PR: #5607
[TTS] add type hints and change varialbe names for tokenizers and g2p by @XuesongYang :: PR: #5602
Fixed RadTTS unit test by @borisfom :: PR: #5572
[TTS][ZH] Disambiguate polyphones with augmented dict and Jieba segmenter for Chinese FastPitch by @yuekaizhang :: PR: #5541
Add duration padding support for RADTTS inference by @kevjshih :: PR: #5650
[TTS] add tts dict cust notebook by @ekmb :: PR: #5662
[TN/TTS docs] TN customization, g2p docs moved to tts by @ekmb :: PR: #5683
typo and link fixed by @ekmb :: PR: #5741
link fixed by @ekmb :: PR: #5745
Update Tacotron2 NGC checkpoint load to latest version by @redoctopus :: PR: #5760
Docs g2p update by @ekmb :: PR: #5769
[TTS][ZH] bugfix import jieba errors. by @XuesongYang :: PR: #5776

</details>

NLP / NMT

<details><summary>Changelog</summary>

Text generation improvement (UI client, data parallel support) by @yidong72 :: PR: #5437
O2 style amp for gpt3 ptuning by @JimmyZhang12 :: PR: #5246
Add support for MHA adapters to ASR by @titu1994 :: PR: #5396
Bert interleaved by @shanmugamr1992 :: PR: #5556
Port stateless timer to exp manager by @MaximumEntropy :: PR: #5584
Add interface for making amax reduction optional for FP8 by @ksivaman :: PR: #5447
Propagate attention_dropout flag for GPT-3 by @mikolajblaz :: PR: #5669
Enc-Dec model size reporting fixes by @MaximumEntropy :: PR: #5623
Add prompt learning tests by @arendu :: PR: #5649
Fix missing torchelastic fixes for PTL 1.8 by @MaximumEntropy :: PR: #5691
ALiBi Positional Embeddings by @michalivne :: PR: #5467
Megatron export triton update by @Davood-M :: PR: #5766
Fix transducer and question answering tutorial bugs bugs by @Zhilin123 :: PR: #5809
Update description for question answering tutorial by @Zhilin123 :: PR: #5814
TPMLP for T5-based models by @Davood-M :: PR: #5840
Megatron positional encoding alibi fix by @michalivne :: PR: #5808

</details>

Export

<details><summary>Changelog</summary>

Add keep_initializers_as_inputs to _export method by @pks :: PR: #5731
Megatron export triton update by @Davood-M :: PR: #5766

</details>

General Improvements

<details><summary>Changelog</summary>

Update to pytorch 22.12 container by @ericharper :: PR: #5694
optimized loop and bugfix by @Jorjeous :: PR: #5573
Expose ClusteringDiarizer device by @SeanNaren :: PR: #5681
remove useless files. by @XuesongYang :: PR: #5580
[Fix] setup_multiple validation/test data by @anteju :: PR: #5585
Move to optimizer based EMA implementation by @SeanNaren :: PR: #5169
[Temp workaround] Disable test with cache_audio to unblock CI by @anteju :: PR: #5615
[EMA] Change success message to reduce confusion by @SeanNaren :: PR: #5621
Temporarily disable prompt learning CI tests by @ericharper :: PR: #5633
[Dockerfile] Remove AIS archive from docker image by @anteju :: PR: #5629
[workflow] add exclude labels option to ignore cherry-picks in releas… by @XuesongYang :: PR: #5645
Add DLLogger support to exp_manager by @milesial :: PR: #5658
Fix EMA restart by allowing device to be set by the class init by @SeanNaren :: PR: #5668
Remove SDP (moved to separate repo) - merge to main by @erastorgueva-nv :: PR: #5630
temp disable speaker recognision CI test by @fayejf :: PR: #5696
Don't print exp_manager warning when max_steps == -1 by @milesial :: PR: #5725
Add tabular data generation documents to the index file by @yidong72 :: PR: #5733
fix token id bug by @yidong72 :: PR: #5777
Update numpy requirements from 1.21 to 1.22 by @Zhilin123 :: PR: #5785
Fix setuptools to usable version by @titu1994 :: PR: #5798
add apt-get upgrade -y in dockerfile by @fayejf :: PR: #5817
Update NeMo Multi-Run docs by @titu1994 :: PR: #5844
add ambernet to readme by @fayejf :: PR: #5872
update apex install instructions for 1.15 by @ericharper :: PR: #5901

</details>

Open source →

1.14.024 Dec 2022

Hybrid CTC + Transducer loss ASR #5364

Highlights

NeMo ASR

Hybrid CTC + Transducer loss ASR #5364
Sampled Softmax RNNT (Enables large vocab RNNT, for speech translation and multilingual ASR) #5216
ASR Adapters hyper parameter search scripts #5159
RNNT {ONNX, TorchScript} x GPU export infer #5248
Exportable MelSpectrogram (TorchScript) #5512
Audio To Audio Dataset Processor #5196
Multi Channel Audio Transcription #5479
Silence Augmentation #5476

NeMo Megatron

Support for the Mixture of Experts for T5
Fix PTL model size output for GPT-3 and BERT
BERT with Tensor Parallelism & Pipeline Parallel Support

NeMo Core

Hydra Multirun core support + NeMo HP optim in YAML #5159

NeMo Models

TTS Zh Fastpitch HifiGan SFSpeech

Detailed Changelogs

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.11

ASR

<details><summary>Changelog</summary>

[Tools][ASR] Tool for generating data using simulated RIRs by @anteju :: PR: #5158
Modernize RNNT ONNX export and add TS export by @titu1994 :: PR: #5248
Add Gradio App to ASR Docs by @titu1994 :: PR: #5270
Add support for Sampled Softmax for RNNT Joint by @titu1994 :: PR: #5216
Speed up HF data processing script for ASR by @titu1994 :: PR: #5330
bugfix in volume loss for CTC models by @bmwshop :: PR: #5348
Add cpWER for evaluation of ASR with diarization by @tango4j :: PR: #5279
Fix for getting tokenizer in character-based ASR models when using tarred dataset by @jonghwanhyeon :: PR: #5442
Refactor/unify ASR offline and buffered inference by @fayejf :: PR: #5440
Standalone diarization+ASR evaluation script by @tango4j :: PR: #5439
[ASR] Transcribe for multi-channel signals by @anteju :: PR: #5479
Add Silence Augmentation by @fayejf :: PR: #5476
add exportable mel spec by @1-800-BAD-CODE :: PR: #5512
add RNN-T loss implemented by PyTorch and test code by @hainan-xv :: PR: #5312
[ASR] AudioToAudio datasets and related test by @anteju :: PR: #5196
Add StreamingFeatureBufferer class for real-life streaming decoding by @tango4j :: PR: #5534
Pool stats with padding by @1-800-BAD-CODE :: PR: #5403
Adding Hybrid RNNT-CTC model by @VahidooX :: PR: #5364
Fix ASR Buffered inference scripts by @titu1994 :: PR: #5552
Add wer details - insertion, deletion, substitution rate by @fayejf :: PR: #5557
Add support for Time Stamp calculation using transcribe_speech.py by @titu1994 :: PR: #5568
[STT] Add Esperanto (Eo) ASR Conformer-CTC and Conformer-Transducer models by @andrusenkoau :: PR: #5639

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Fastpitch energy condition and refactoring by @subhankar-ghosh :: PR: #5218
[TTS] HiFi-TTS Download Script by @oleksiivolk :: PR: #5241
[TTS] Add Mandarin/English Bilingual Recipe for Training Fastpitch Models by @yuekaizhang :: PR: #5208
[TTS] fixed type of filepath and rename openslr. by @XuesongYang :: PR: #5276
[TTS] replace obsolete torch_tts unit test marker with run_only_on('CPU') by @XuesongYang :: PR: #5307
[TTS] bugfix IPAG2P and refactor to remove duplicate process. by @XuesongYang :: PR: #5304
Update path to get_data.py in TTS tutorial by @redoctopus :: PR: #5311
[TTS] Replace IPA lambda arguments with locale string by @rlangman :: PR: #5298
[TTS] expand to support flexible dictionary entry formats in IPAG2P. by @XuesongYang :: PR: #5318
[TTS] update organization of model checkpoints and their pointers. by @XuesongYang :: PR: #5327
[TTS] bugfix for the script of generating mels from fastpitch. by @XuesongYang :: PR: #5344
[TTS] Add Spanish model documentation by @rlangman :: PR: #5390
[TTS] Add Spanish FastPitch training configs by @rlangman :: PR: #5383
[TTS] replace pitch normalization params with ??? by @XuesongYang :: PR: #5392
[TTS] Create script for processing TTS training audio by @rlangman :: PR: #5262
[TTS] remove useless logic for set_tokenizer. by @XuesongYang :: PR: #5430
[TTS] Fixing RADTTS training - removing view buffer and fixing accuracy issue by @borisfom :: PR: #5358
JOC Optimization in FastPitch by @subhankar-ghosh :: PR: #5450
[TTS] Support speaker level pitch normalization by @rlangman :: PR: #5455
TTS tutorial update: use speaker 9017 instead of 6097 by @redoctopus :: PR: #5532
[TTS] Remove unused TTS eval function by @redoctopus :: PR: #5605
[TTS][ZH] add fastpitch and hifigan model NGC urls and update NeMo docs. by @XuesongYang :: PR: #5596
[TTS][DOC] add notes about automatic conversion to target sampling ra… by @XuesongYang :: PR: #5624
[TTS][ZH] bugfix for the tutorial and add NGC CLI installation guide. by @XuesongYang :: PR: #5643
[TTS][ZH] bugfix for ngc cli installation. by @XuesongYang :: PR: #5652
[TTS][ZH] fix broken link for the script. by @XuesongYang :: PR: #5666

</details>

NLP / NMT

<details><summary>Changelog</summary>

Option to pad the last validation input sequence if its smaller than the encoder sequence length for MegatronGPT by @anmolgupt :: PR: #5243
Fixes bugs with loss averaging with for Megatron GPT by @shanmugamr1992 :: PR: #5329
Fixing bug in Megatron BERT when loss mask is all zeros by @shanmugamr1992 :: PR: #5424
support to disable sequence length + 1 input tokens for each sample in MegatronGPT by @anmolgupt :: PR: #5363
[TN] raise NotImplementedError for unsupported languages and other minor fixes by @XuesongYang :: PR: #5414
Bug fix/gpt by @shanmugamr1992 :: PR: #5493
prompt tuning fix for unscale grad errors by @arendu :: PR: #5523
Bert sequence parallel support by @shanmugamr1992 :: PR: #5494
NLP docs fixes by @vsl9 :: PR: #5528
Switch order of args in optimizer_step override by @ericharper :: PR: #5549
Upgrade to 22.11 by @ericharper :: PR: #5550
Merge r1.13.0 main by @ericharper :: PR: #5570
some tokenizers do not have additional_special_tokens_ids attribute by @arendu :: PR: #5642
Remove cell output from tutorial by @ericharper :: PR: #5689

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

[ITN] fix year date graph, cardinals extension for hundreds by @ekmb :: PR: #5435
[TN] raise NotImplementedError for unsupported languages and other minor fixes by @XuesongYang :: PR: #5414

</details>

Export

<details><summary>Changelog</summary>

Fixed the onnx bug in conformer for non-streaming models. by @VahidooX :: PR: #5242
Modernize RNNT ONNX export and add TS export by @titu1994 :: PR: #5248
Fixes for Conformer-xl export by @borisfom :: PR: #5309
Remove onnx graphsurgery from Dockerfile by @titu1994 :: PR: #5320
add exportable mel spec by @1-800-BAD-CODE :: PR: #5512

</details>

General Improvements

<details><summary>Changelog</summary>

bugfix in volume loss for CTC models by @bmwshop :: PR: #5348
Fix setting up of learning rate scheduler by @PeganovAnton :: PR: #5444
Better patch hydra by @titu1994 :: PR: #5591
[TTS][ZH] bugfix for the tutorial and add NGC CLI installation guide. by @XuesongYang :: PR: #5643
Add fully torch.jit.script-able speaker clustering module by @tango4j :: PR: #5191
Update perturb.py by @stevehuang52 :: PR: #5231
remove CV requirements. by @XuesongYang :: PR: #5233
checks for accepted adapter type at module level by @arendu :: PR: #5194
fix hypotheses return by @nithinraok :: PR: #5253
Support for inserting additional subsampling in conformer encoder by @shan18 :: PR: #5224
update tutorials to use meeting config as default and VAD by @nithinraok :: PR: #5237
Specifying audio signal dropout separately for the Conformer Encoder by @shan18 :: PR: #5263
created by @bmwshop :: PR: #5268
Fix failing speaker counting for short audio samples by @tango4j :: PR: #5267
O2bert + apex pipeline functions by @shanmugamr1992 :: PR: #5221
Upperbound PTL by @titu1994 :: PR: #5302
Update Interface(s) phonetic entry by @blisc :: PR: #5212
add label inference support to EncDecSpeakerLabel class by @nithinraok :: PR: #5278
Add italian model checkpoints by @Kipok :: PR: #5315
Text Memmap Parsing Improvements by @michalivne :: PR: #5265
Update librosa signature in HF processing script by @titu1994 :: PR: #5321
Force wav file format for audio_filepath by @titu1994 :: PR: #5323
Updates to T0 Dataset and Model by @MaximumEntropy :: PR: #5201
[DOC] add sphinx-copybutton requirement to copy button on code snippets. by @XuesongYang :: PR: #5326
Add support for Hydra multirun to NeMo by @titu1994 :: PR: #5159
typo fix by @arendu :: PR: #5328
add precommit hood to automatic sort entries in requirements. by @XuesongYang :: PR: #5333
Add speaker clustering arguments to forward function by @tango4j :: PR: #5306
Fixing de-autocast by @borisfom :: PR: #5319
[Bugfix] Added rm -f / wget- nc command to avoid bash error in multispeaker sim notebook by @tango4j :: PR: #5292
[DOC] added ipython dependency to support IPython.sphinxext extension by @XuesongYang :: PR: #5345
Bug fix (removing old compute consumed samples) by @shanmugamr1992 :: PR: #5355
removed uninstall nemo_cv and nemo_simple_gan and relax numba version… by @XuesongYang :: PR: #5332
Enable mlflow logger by @whrichd :: PR: #4893
Fix Python type hints according to Python Docs by @artbataev :: PR: #5370
Distributed optimizer support for BERT by @timmoon10 :: PR: #5305
SpeakerClustering: fix tensor dimennsions in forward() by @virajkarandikar :: PR: #5387
add squad by @arendu :: PR: #5407
added python and c++ alignment code by @yzhang123 :: PR: #5346
Add MoE support for T5 model (w/o expert parallel) by @aklife97 :: PR: #5409
Fix for concat map dataset by @1-800-BAD-CODE :: PR: #5133
Support for finetuning and finetuning inference with .ckpt files & batch size refactoring by @MaximumEntropy :: PR: #5339
update doc in terms of get_label for lang id model by @fayejf :: PR: #5366
Debug support for interleaved pipeline parallelism with the distributed Adam optimizer by @timmoon10 :: PR: #5236
Create codeql.yml by @titu1994 :: PR: #5445
Update codeql.yml by @titu1994 :: PR: #5449
Fix support for legacy sentencepiece models by @Numeri :: PR: #5406
Update docs with Comparison tool info, and slightly change .sh for ea… by @Jorjeous :: PR: #5182
Add float32 type casting for get_samples function by @tango4j :: PR: #5399
Add missing import in transcribe_utils.py by @jonghwanhyeon :: PR: #5487
Add auto-labeler by @SeanNaren :: PR: #5498
Add more glob patterns for labeler by @SeanNaren :: PR: #5504
Fix issues with PL 1.8 by @SeanNaren :: PR: #5353
[BugFix] Removing <unk> tokens from decoding timestamp by @tango4j :: PR: #5481
Upperbound the torchmetrics version by @SeanNaren :: PR: #5537
Data parallel collect results by @michalivne :: PR: #5547
Fix log-rank-0-only logic by @mikolajblaz :: PR: #5555
Fixed Docker build by @borisfom :: PR: #5562
Patch hydra launch by @titu1994 :: PR: #5589
Fix race condition bug with hydra multirun by @titu1994 :: PR: #5594
Update Dockerfile to use numba==0.53.1 by @stevehuang52 :: PR: #5614
Fixed a missing import for gather_objects by @michalivne :: PR: #5622

</details>

Open source →

1.13.07 Dec 2022Deprecated

[TTS] deprecate TextToWaveform base class. by @XuesongYang :: PR: #5205

Highlights

NeMo ASR

Spoken Language Understanding (SLU) models based on Conformer encoder and transformer decoder
Support for codeswitched manifests during training
Support for Language ID during inference for ML models
Support of cache-aware streaming for offline models
Word confidence estimation for CTC & RNNT greedy decoding

NeMo Megatron

Interleaved Pipeline schedule
Transformer Engine for GPT
HF T5v1.1 -> NeMo-Megatron conversion and finetuning/p-tuning
IA3 and Adapter Tuning (Tensor + Pipeline Parallel)
Pipeline Parallel Support for T5 Prompt Learning
MegatronNMT export

NeMo TTS

TTS introductory tutorial
Phonemizer/espeak removal (Spanish/German)
Char-only support for Spanish/German models
Documentation Refactor

NeMo Core

Upgrade to NGC PyTorch 22.09 container
Add pre-commit hooks
Exponential moving average (EMA) of weights during training

NeMo Models

ASR Conformer Croatian: stt_hr_conformer_ctc_large and stt_hr_conformer_transducer_large
ASR Conformer Belarusian: stt_be_conformer_ctc_large and stt_be_conformer_transducer_large
ASR Squeezeformer Librispeech: 6 checkpoints (XS, S, SM, M, ML, L)
SLURP Intent Classification / Slot Filling: slu_conformer_transformer_large_slurp
LanguageID AmberNet: langid_ambernet

Detailed Changelogs

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.09

Known Issues

<details><summary>Issues</summary>

pytest for RadTTSModel_export_to_torchscript are failing intermittently due to random input values. Fixed in main.

</details>

ASR

<details><summary>Changelog</summary>

Add docs tutorial on kinyarwanda asr by @bene-ges :: PR: #4953
Asr codeswitch by @bmwshop :: PR: #4821
Add test for nested ASR model by @titu1994 :: PR: #5002
Greedy decoding confidence for CTC and RNNT by @GNroy :: PR: #4931
[ASR][Tools] RIR corpus generator by @anteju :: PR: #4927
Add Squeezeformer CTC model checkpoints on Librispeech by @titu1994 :: PR: #5121
adding loss normalization options to rnnt joint by @bmwshop :: PR: #4829
Asr concat dataloader by @bmwshop :: PR: #5108
Added ASR model comparison to SDE by @Jorjeous :: PR: #5043
Add scripts for converting Spoken Wikipedia to asr dataset by @bene-ges :: PR: #5138
ASR confidence bug fix for older Python versions by @GNroy :: PR: #5180
Update ASR Scores and Results by @titu1994 :: PR: #5254
[STT] Add Ru ASR Conformer-CTC and Conformer-Transducer by @ssh-meister :: PR: #5340

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Adding speaker embedding conditioning in fastpitch by @subhankar-ghosh :: PR: #4986
[TTS] Remove PhonemizerTokenizer by @rlangman :: PR: #4990
[TTS] FastPitch speaker interpolation by @subhankar-ghosh :: PR: #4997
RADTTS model changes to accommodate export with batch size > 1 by @borisfom :: PR: #4947
[TTS] remove phonemizer.py by @XuesongYang :: PR: #5090
[TTS] Add NeMo TTS Primer Tutorial by @rlangman :: PR: #4933
[TTS] Add SpanishCharsTokenizer by @rlangman :: PR: #5135
Fixes for docs/typos + remove max_utts parameter from tarred datasets as it causes hang in training by @Kipok :: PR: #5118
refactor TTS documentation organization and add new contents. by @XuesongYang :: PR: #5137
[TTS][DOC] update models trained on HifiTTS dataset. by @XuesongYang :: PR: #5173
[TTS] Fix TTS Primer image markup by @rlangman :: PR: #5192
[TTS] deprecate TextToWaveform base class. by @XuesongYang :: PR: #5205
[TTS] remove the avoidance of circular imports by @XuesongYang :: PR: #5214
[TTS] remove LinVocoder and apply Vocoder as parent class. by @XuesongYang :: PR: #5206
[TTS] unify requirements_tts.txt and requirements_torch_tts.txt by @XuesongYang :: PR: #5232
Minor typo fixes in TTS tutorial by @redoctopus :: PR: #5266
Radtts 1.13 by @borisfom :: PR: #5451
Radtts 1.13 plus by @borisfom :: PR: #5457

</details>

NLP / NMT

<details><summary>Changelog</summary>

IA3 support for GPT and T5 by @arendu :: PR: #4909
Fix and refactor consumed samples save/restore for Megatron models. by @MaximumEntropy :: PR: #5077
Remove unsupported arguments from MegatronNMT by @MaximumEntropy :: PR: #5065
Update megatron interface to dialogue by @Zhilin123 :: PR: #4936
gpt ia3 CI tests by @arendu :: PR: #5140
Fix NMT Eval Sampler by @aklife97 :: PR: #5154
Add interleaved pipeline schedule to GPT by @ericharper :: PR: #5025
fix for bug in bignlp by @arendu :: PR: #5172
Fixes some args that were not removed properly for multilingual Megatron NMT by @MaximumEntropy :: PR: #5142
Fix absolute path in GPT Adapter CI tests by @arendu :: PR: #5184
Add ability to configure drop last batch for validation datasets with MegatronGPT by @shanmugamr1992 :: PR: #5067
Megatron Export Update by @Davood-M :: PR: #5343
Fix GPT generation when using sentencepiece tokenizer by @MaximumEntropy :: PR: #5413
Disable sync_batch_comm in validation_step for GPT by @ericharper :: PR: #5397
Set sync_batch_comm=False in prompt learning and inference by @MaximumEntropy :: PR: #5448
Fix a bug with positional vs key-word based argument passing in the transformer layer by @MaximumEntropy :: PR: #5475

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

[Chinese text normalization] speed up graph building by @pengzhendong :: PR: #5128

</details>

NeMo Tools

<details><summary>Changelog</summary>

Added ASR model comparison to SDE by @Jorjeous :: PR: #5043

</details>

Export

<details><summary>Changelog</summary>

Fix export bug by @VahidooX :: PR: #5009
RADTTS model changes to accommodate export with batch size > 1 by @borisfom :: PR: #4947
Support TorchScript export for Squeezeformer by @titu1994 :: PR: #5164
Expose keep_initializers_as_inputs to Exportable class by @pks :: PR: #5052
Fix the self-attention export bug for cache-aware streaming Conformer by @VahidooX :: PR: #5114
replace ColumnParallelLinear with nn.Linear in export_utils by @arendu :: PR: #5217
Megatron Export Update by @Davood-M :: PR: #5343
Fix Conformer Export in 1.13.0 (cherry-pick from main) by @artbataev :: PR: #5446
export_utils bugfix by @Davood-M :: PR: #5480
Export fixes for Riva by @borisfom :: PR: #5496

</details>

General Improvements and Bugfixes

<details><summary>Changelog</summary>

don't use bfloat16 when in jit by @bmwshop :: PR: #5051
Set sync_batch_comm=False in prompt learning and inference by @MaximumEntropy :: PR: #5448
Fix a bug with positional vs key-word based argument passing in the transformer layer by @MaximumEntropy :: PR: #5475
Pin Transformers version to fix CI by @SeanNaren :: PR: #4955
Fix changelog builder (#4962) by @titu1994 :: PR: #4963
Checkpoint averaging class fix by @michalivne :: PR: #4946
Add ability to give seperate datasets for test, train and validation by @shanmugamr1992 :: PR: #4798
Add simple pre-commit file by @SeanNaren :: PR: #4983
Import pycuda.autoprimaryctx or pycuda.autoinit to init pycuda execut… by @liji-nv :: PR: #4951
Improvements to AMI script by @SeanNaren :: PR: #4974
clean warnings from tests and CI runs, and prepare for upgrade to PTL 1.8 by @nithinraok :: PR: #4830
Update libraries by @titu1994 :: PR: #5010
add close inactive issues and PRs github action. by @XuesongYang :: PR: #5015
Fix filename extraction in vad_utils.py by @GKPr0 :: PR: #4999
Add black to pre-commit by @SeanNaren :: PR: #5027
[CI] Enable previous build abort when new commit pushed by @SeanNaren :: PR: #5041
Tutorials and Docs for Multi-scale Diarization Decoder by @tango4j :: PR: #4930
Refactor output directory for MSDD Inference Notebook by @SeanNaren :: PR: #5044
text_memmap dataset index range testing fix by @michalivne :: PR: #5034
fix undefined constant in code example by @bene-ges :: PR: #5046
Text generation refactor and RETRO text generation implementation by @yidong72 :: PR: #4985
Lids by @bmwshop :: PR: #4820
Add datasets folder, add diarization datasets voxconverse/aishell by @SeanNaren :: PR: #5042
Fix the bugs in cache-aware streaming Conformer by @VahidooX :: PR: #5032
Bug fix - Limit val batches set to 1.0 by @shanmugamr1992 :: PR: #5023
[bug_fix] kv_channels is used when available by @arendu :: PR: #5066
Add spe_split_by_unicode_script arg by @piraka9011 :: PR: #5072
Transformer Engine Integration by @ericharper :: PR: #5104
Text memmap dataset index memory efficiency by @michalivne :: PR: #5056
Add NGC links for Aligner and FastPitch by @redoctopus :: PR: #5235
Fix link to inference notebook by @redoctopus :: PR: #5247
Fix links to speaker identification notebook by @SeanNaren :: PR: #5260
Fix bug into Dialogue tutorial by @Zhilin123 :: PR: #5277
PCLA tutorial typo fix by @jubick1337 :: PR: #5288
Fix dialogue tutorial bug by @Zhilin123 :: PR: #5297
small bugfix for r1.13.0 by @fayejf :: PR: #5310
Add italian model checkpoints by @Kipok :: PR: #5316
Pcla tutorial fixes by @jubick1337 :: PR: #5313
Fix issue with HF Model upload tutorial by @titu1994 :: PR: #5359
P&C LA tutorial fixes by @jubick1337 :: PR: #5354
Add SDP documentation by @erastorgueva-nv :: PR: #5274
[Bugfix] Added rm -f / wget- nc command in multispeaker sim notebook to r1.13.0 by @tango4j :: PR: #5375
Rename Speech Dataset Processor to Speech Data Processor by @erastorgueva-nv :: PR: #5378
fix for num worker 0 causing issues in losses after 1 epoch by @arendu :: PR: #5379
Fixed bug in notebook by @vadam5 :: PR: #5382
Force MHA QKV onto fp32 by @titu1994 :: PR: #5391
Fix for prompt table restore error by @vadam5 :: PR: #5393
Fix activation checkpoint args for T5 by @MaximumEntropy :: PR: #5410
Temporary hard code fix in PTL for CUDA Error by @yaoyu-33 :: PR: #5421
disable pc test by @ekmb :: PR: #5426
Revert Temporary hard code fix in PTL for CUDA Error by @yaoyu-33 :: PR: #5431
Revert workaround for T5 that sets number of workers to 0 & sync_batch_comm=False by @MaximumEntropy :: PR: #5420
Add num layers check for full activation checkpointing by @MaximumEntropy :: PR: #5470
Cherry Pick T5 finetuning changes into 1.13 by @MaximumEntropy :: PR: #5478
T5 Eval bugfix by @Davood-M :: PR: #5521
added set_start_method + function param bugfix by @Davood-M :: PR: #5539
Remove notebook by @ericharper :: PR: #5548
Remove broadcast from T5 prompt learning inference by @MaximumEntropy :: PR: #5558
Fix all gather while writing to a file during T5 finetuning by @MaximumEntropy :: PR: #5561

</details>

Open source →

1.12.010 Oct 2022

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.08

ASR

<details><summary>Changelog</summary>

Add support for RNNT Char/Word Timestamp Calculation by @titu1994 :: PR: #4665
add conditional logic to rnnt_wer to handle when arrays have no elements by @mgoldey :: PR: #4776
fix handling of the final word for rnnt word timestamps by @mgoldey :: PR: #4779
amend rnnt word timestamps by @mgoldey :: PR: #4782
fix type error in rnnt_wer.py, rnnt_wer_bpe.py, wer_bpe.py by @hainan-xv :: PR: #4822
add kab language asr models by @nithinraok :: PR: #4819
[Tutorial][ASR][Fix] Data paths in ASR with NeMo tutorial by @anteju :: PR: #4845
[ASR] Fix for multi-channel signals in AudioSegment by @anteju :: PR: #4824
[ASR] Generate multichannel noise by @anteju :: PR: #4870
Fix asr model order by @nithinraok :: PR: #4959
Fix ASR issues by @titu1994 :: PR: #4984
Fix diarization ASR inference link in notebook by @SeanNaren :: PR: #5016
Code switching by @KunalDhawan :: PR: #4784
Release SOTA Lang ID model by @fayejf :: PR: #5080
Stateless decoder for RNN-T by @hainan-xv :: PR: #4710

</details>

TTS

<details><summary>Changelog</summary>

[TTS] use consistent spline interpolation for fastpitch and hifigan. by @XuesongYang :: PR: #4679
TTS tokenizers moved to collections.common.tokenizers by @AlexGrinch :: PR: #4690
[TTS] Fix text normalizer bugs in TTS data loader by @rlangman :: PR: #4781
ARP to IPA mapping, g2p_encode for IPATokenizer by @ekmb :: PR: #4850
IPA G2P bugfixes by @redoctopus :: PR: #4869
[TTS] add missing WikiHomograph data entries to CMUdict, updates to match new ipa set by @ekmb :: PR: #4886
[TTS] fix wrong g2p path. by @XuesongYang :: PR: #4902
[TTS] FastPitch training: speed up align_prior_matrix calculation by @racoiaws :: PR: #4718
[TTS] fix broken tutorial for MixerTTS. by @XuesongYang :: PR: #4949
[TTS] bugfix 'EnglishPhonemesTokenizer' object has no attribute 'encode_from_g2p' by @XuesongYang :: PR: #4992
[TTS] added missing German phoneme tokenizer by @XuesongYang :: PR: #5070
[TTS] fixed wrong val loss for epoch 0 and inconsistent metrics names by @XuesongYang :: PR: #5087

</details>

NLP / NMT

<details><summary>Changelog</summary>

Fix bug intent slot classification tokenizer to dialogue by @Zhilin123 :: PR: #4694
Intent slot model onnx export test by @Zhilin123 :: PR: #4731
Fix megatron p tuning notebook by @nithinraok :: PR: #4741
Add support for Apex distributed Adam optimizer with GPT-3 by @timmoon10 :: PR: #4487
Fixes NLPModel's load from checkpoint due to PTL private function changes by @MaximumEntropy :: PR: #4755
Adapter tuning for Megatron GPT models by @arendu :: PR: #4717
Megatron Encoder Decoder models with RPE and PP > 2 by @MaximumEntropy :: PR: #4663
add kab language asr models by @nithinraok :: PR: #4819
add chinese to language doc and fix bug by @yzhang123 :: PR: #4834
Spoken Language Identification by @fayejf :: PR: #4846
Fix decoding bug for megatron enc-dec models with O2 by @MaximumEntropy :: PR: #4989
Updating Megatron LM conversion according to PTL 1.7 by @Davood-M :: PR: #5038
Adding RETRO model Faiss sharding index and KNN sharding index by @yidong72 :: PR: #4713
MLP Prompt Learning Encoder by @vadam5 :: PR: #4849
Update the prompt learning to handle large lanague model by @yidong72 :: PR: #4906

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

[TTS] Fix text normalizer bugs in TTS data loader by @rlangman :: PR: #4781
[Chinese text normalization]Chinese TN part in text_normalization by @mzxcpp :: PR: #4826
Fix zh tn by @yzhang123 :: PR: #5035
Bug fixes for parallel mp3 to wav conversion, PC notebook, update Readme for TN requirements by @ekmb :: PR: #5047
Added P&C lexical audio model by @jubick1337 :: PR: #4802

</details>

Export

<details><summary>Changelog</summary>

Intent slot model onnx export test by @Zhilin123 :: PR: #4731

</details>

General Improvements

<details><summary>Changelog</summary>

Fix logger reference by @SeanNaren :: PR: #4786
Fix error with class method reference in msdd by @SeanNaren :: PR: #4865
Add sync for logging calls to ensure aggregation across devices by @SeanNaren :: PR: #4876
Fix saving the last checkpoint when using val check interval by @SeanNaren :: PR: #4905
Add support for skipping validation on resume + extend saving last ckpt test by @SeanNaren :: PR: #4922
Move trainer calls for ssl models to training and validation steps only by @sam1373 :: PR: #4685
Change Num Partitions size expansion fix by @aklife97 :: PR: #4719
upgrade to PTL 1.7 by @nithinraok :: PR: #4672
Fixing outputs of infer() and use of NeMo length regulator helper by @borisfom :: PR: #4724
bug fix: enable async grad reduction when DP > 1 by @erhoo82 :: PR: #4740
Add LayerNorm1P, weight decay for LN and unscaled initialization by @mikolajblaz :: PR: #4743
Data Simulator by @chooper1 :: PR: #4686
jenkins data simulator fix by @nithinraok :: PR: #4751
Mutiscale Diarization Decoder (MSDD) model and module files by @tango4j :: PR: #4650
Fix logging in gradient clipping with PTL 1.7.2 by @MaximumEntropy :: PR: #4769
Fix checkpoint restoring by @nithinraok :: PR: #4777
avoid data clipping after convolution with rir samples by @nithinraok :: PR: #4806
Fixed in_features dim if bidirectional is True by @farisalasmary :: PR: #4588
Fix float/integer type error in WER.update() by @fujimotos :: PR: #4816
[Speech Data Explorer] An option to explicitly specify the base dir by @anteju :: PR: #4678
adding instancenorm as an option for conv normalization by @bmwshop :: PR: #4827
Fix small spelling mistakes by @SeanNaren :: PR: #4839
[Tutorials] Fix matplotlib version and directory name in Multispeaker_Simulator by @anteju :: PR: #4804
Update diarization folder structure by @tango4j :: PR: #4823
Missing types in clustering by @SeanNaren :: PR: #4858
add new models by @Jorjeous :: PR: #4852
Fix decoding for T5 models with RPE by @MaximumEntropy :: PR: #4847
Update Speaker Diarization notebooks with unknown oracle_num_speakers by @fayejf :: PR: #4861
Fix mha bug by @yzhang123 :: PR: #4859
Updates to adapter training by @arendu :: PR: #4842
Changes to MSDD code after review, fix test log call by @SeanNaren :: PR: #4881
Fixed output of BERT to be [batch x seq x hidden] by @michalivne :: PR: #4887
Add AMI dataset script by @SeanNaren :: PR: #4864
Update label_models.py by @stevehuang52 :: PR: #4891
Update tutorials.rst for question answering by @Zhilin123 :: PR: #4895
removed unused imports for all domains. by @XuesongYang :: PR: #4901
Fix ptl_load_state not providing cls by @MaximumEntropy :: PR: #4914
Remove unused cv collection by @okuchaiev :: PR: #4907
Add mixed-representation config to PhonemizerTokenizer by @rlangman :: PR: #4904
Fix implicit bug in _AudioLabelDataset by @stevehuang52 :: PR: #4923
Fix and refactor label models by @fayejf :: PR: #4913
Sparrowhawk deployment fix by @ekmb :: PR: #4928
Upgrade to NGC PyTorch 22.08 Container by @ericharper :: PR: #4929
Fixes for Cherry Picked PRs by @titu1994 :: PR: #4962
Fix cherry pick workflow by @ericharper :: PR: #4964
check for active conda environment by @nithinraok :: PR: #4970
fix label models restoring issue from weighted cross entropy by @nithinraok :: PR: #4968
Add simple pre-commit file (#4983) by @SeanNaren :: PR: #4995
Fix bug in Squeezeformer Conv block by @titu1994 :: PR: #5011
Fix bugs by @Zhilin123 :: PR: #5036
Add black to pre-commit (#5027) by @SeanNaren :: PR: #5045
Fix bug in question answering tutorial by @Zhilin123 :: PR: #5049
Missing fixes from r1.11.0 to T5 finetuning eval by @MaximumEntropy :: PR: #5054
P&C docs by @jubick1337 :: PR: #5068
probabilites -> probabilities by @nithinraok :: PR: #5078
Notebook bug fixes by @vadam5 :: PR: #5084
update strategy in notebook from ddp_fork to dp by @Zhilin123 :: PR: #5088
Fix Unhashable type list for Numba Cuda spec augment kernel by @titu1994 :: PR: #5093
Remove numba import by @titu1994 :: PR: #5095
T5 prompt learning fixes missing from r.11.0 merge by @MaximumEntropy :: PR: #5075
T5 Decoding with PP > 2 fix by @MaximumEntropy :: PR: #5091
Multiprocessing fix by @jubick1337 :: PR: #5106
[Bug fix] PC lexical + audio by @ekmb :: PR: #5109
bugfix: pybtex.database.InvalidNameString: Too many commas in author … by @XuesongYang :: PR: #5112

</details>

Open source →

1.11.08 Sept 2022Deprecated

Deprecated old scripts for ljspeech. by @XuesongYang :: PR: #4780

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.07

ASR

<details><summary>Changelog</summary>

Add ASR CTC Decoding module by @titu1994 :: PR: #4342
Fixing bugs in calling method ctc_decoder_predictions_tensor. by @VahidooX :: PR: #4414
Fixed WER initialization in ASR_with_Nemo notebook by @anteju :: PR: #4523
Update signature of Hypothesis alignments by @titu1994 :: PR: #4511
Add support for ASR Adapter Auxiliary Losses by @titu1994 :: PR: #4480
Catalan ASR NGC Resource by @stevehuang52 :: PR: #4576
Add kw asr models, add itn ru checkpoint (tagger-based) by @bene-ges :: PR: #4595
Add DALI char dataset support to SSL model by @piraka9011 :: PR: #4592
Customize arguments for trimming the leading/trailing silence by @XuesongYang :: PR: #4582
Update Offline ASR with CTC Decoding by @titu1994 :: PR: #4608
Add Squeezeformer to ASR by @titu1994 :: PR: #4416
Fix ASR notebooks by @titu1994 :: PR: #4738
Add pretrained ASR models for Croatian by @anteju :: PR: #4682
Dataloader, collector, loss and metric for multiscale diarization decoder by @tango4j :: PR: #4187
Multilingual VAD model by @fayejf :: PR: #4734
Adding support for models trained with full context for cache-aware streaming. by @VahidooX :: PR: #4687
Fp16 support for Conformer by @bmwshop :: PR: #4571
Tiny VAD refactoring for postprocessing by @fayejf :: PR: #4625
Add silence handling for speaker diarization pipeline by @nithinraok :: PR: #4512
Add Bucketing support to TarredAudioToClassificationLabelDataset by @entn-at :: PR: #4465

</details>

TTS

<details><summary>Changelog</summary>

Wrong order of returned tuple for general_collate_fn. by @XuesongYang :: PR: #4388
Pitch, voiced_mask, prob_voiced have the same values which is not expected. by @XuesongYang :: PR: #4392
Add static method decorator. by @XuesongYang :: PR: #4443
Fix typo in HiFi-GAN config's max steps by @XuesongYang :: PR: #4450
Relaxed support for both CPUs and GPUs by @XuesongYang :: PR: #4461
Multi-speaker fastpitch model training recipe on HUI-Audio-Corpus-German by @XuesongYang :: PR: #4413
Created the finetuning Hifigan 44100Hz recipe on HUI-Audio-Corpus-German by @XuesongYang :: PR: #4478
Fix dataset parameter typo on tacotron2 example yaml by @saarus72 :: PR: #4471
Update cmudict by @jasro23 :: PR: #4510
Customize arguments for trimming the leading/trailing silence by @XuesongYang :: PR: #4582
Fix off-by-1 bug in Beta Binomial Prior by @rlangman :: PR: #4616
G2P Aligner by @redoctopus :: PR: #4604
RADTTS ADLR-NEMO porting by @MikyasDesta :: PR: #4538
Fixed wrong pronunciations for r1.11. by @XuesongYang :: PR: #4677
Incremented the version number to 22.08 in tutorials. by @XuesongYang :: PR: #4684
Bugfix for missing configs. by @XuesongYang :: PR: #4725
Fix pynini install in TTS tutorials by @redoctopus :: PR: #4729
Updated config with a German IPA phoneme tokenizer by @XuesongYang :: PR: #4756
Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints by @XuesongYang :: PR: #4763
Add single male speaker German FastPitch and HiFiGAN NGC checkpoints by @XuesongYang :: PR: #4770
Deprecated old scripts for ljspeech. by @XuesongYang :: PR: #4780
Fix MixerTTS data loading index error by @redoctopus :: PR: #4811
G2P docs by @ekmb :: PR: #4841
NMESC speaker counting algorithm update by @tango4j :: PR: #4500

</details>

NLP / NMT

<details><summary>Changelog</summary>

Add O2 support for RETRO model by @yidong72 :: PR: #4411
Add MTEncDec Finetune support by @aklife97 :: PR: #4540
Fix metric setup for finetuning without a test set by @MaximumEntropy :: PR: #4585
T0 model and dataset by @MaximumEntropy :: PR: #4598
Add prompt learning for T5 by @HeyyyyyyG :: PR: #4391
Add MuTransfer Capablity to RETRO model pretraining by @yidong72 :: PR: #4643
Label Smoothing in VocabParallelCrossEntropy by @MaximumEntropy :: PR: #4602
Megatron BART BOS / EOS bug fix by @michalivne :: PR: #4495
GPT Prompt Learning Improvements by @vadam5 :: PR: #4496
Megatron perceiver with tensor parallelism only by @MaximumEntropy :: PR: #4318
Refactor for punctuation model by @jubick1337 :: PR: #4367
Update megatron prompt learning interface to dialogue by @Zhilin123 :: PR: #4545
Removed NLPDDPPlugin Import check by @vadam5 :: PR: #4555
Option to disregard document boundaries for t5, bart, ul2 by @MaximumEntropy :: PR: #4481
Add Tokenization and Normalization pre-proecssing script for NMT by @aklife97 :: PR: #4557
Integrating support for GPT/T5/BART for Question Answering by @ameyasm1154 :: PR: #4532
NeMo Megatron: Add sequence parallelism and selective activation checkpointing (rebased) by @ericharper :: PR: #4380
Update megatron t5 interface to dialogue by @Zhilin123 :: PR: #4626
Additional sentencepiece args - Byte fallback, split digits, split_on_whitespace by @MaximumEntropy :: PR: #4525
Maximum sample-based training for Megatron NMT and Text Memmap based Seq2seq Pre-training by @MaximumEntropy :: PR: #4396
NeMo Megatron Doc updates1 by @okuchaiev :: PR: #4633
Asymmetric Encoder and Decoder Configuration for Megatron Models by @MaximumEntropy :: PR: #4568
Add sentencepiece legacy arg to megatron tokenizer configs by @MaximumEntropy :: PR: #4659
Megatron encode function with RPE fix by @MaximumEntropy :: PR: #4692
Updates to NeMo Megatron OSS docs by @okuchaiev :: PR: #4709
Changes to make Megatron NMT exportable by @Davood-M :: PR: #4499
fix bug relating to ddp strategy in joint intent slot classification … by @Zhilin123 :: PR: #4762
Fix qa notebook typos and branch by @ericharper :: PR: #4788
Colab py37 compatibility megatron by @Zhilin123 :: PR: #4791
added/fixed export for Megatron models by @Davood-M :: PR: #4712
Fix providing glue in seq2seq eval by @MaximumEntropy :: PR: #4843
Fix Megatron NMT consumed samples and ckpt_to_nemo split rank by @MaximumEntropy :: PR: #4884
Fixing Megatron BERT output dimensions to [batch x sec x hidden] by @michalivne :: PR: #4894
Prompt Learning Inference Improvements by @vadam5 :: PR: #4566
MegaMolBART Compatibility by @michalivne :: PR: #4603

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Add ITN pt by @guidefloripa :: PR: #4516
add kw asr models, add itn ru checkpoint (tagger-based) by @bene-ges :: PR: #4595
Fix ITN pt by @guidefloripa :: PR: #4623
Bug fix hundred in Audio-based, added method so split text in sentences by @ekmb :: PR: #4610
Fix itn pt time by @guidefloripa :: PR: #4630
Pin lightning version to be < 1.7.0 by @MaximumEntropy :: PR: #4660
G2P for OOV and heteronyms by @ekmb :: PR: #4624
Publish pretrained itn t5 model for English by @bene-ges :: PR: #4748
Added MLM Scoring by @yzhang123 :: PR: #4476

</details>

Export

<details><summary>Changelog</summary>

update fastpitch to add export controls by @blisc :: PR: #4509
Fix Fastpitch Export by @blisc :: PR: #4676
Changes to make Megatron NMT exportable by @Davood-M :: PR: #4499
Added/fixed export for Megatron models by @Davood-M :: PR: #4712

</details>

Bugfixes

<details><summary>Changelog</summary>

Wrong order of returned tuple for general_collate_fn. by @XuesongYang :: PR: #4388
Pitch, voiced_mask, prob_voiced have the same values which is not expected. by @XuesongYang :: PR: #4392
Fix tarred dataset len when num shards is not divisible by workers by @itzsimpl :: PR: #4553
Fix multiple dev/test datasets after restoring from checkpoint by @PeganovAnton :: PR: #4636
Fix/need different cache dirs for different datasets by @PeganovAnton :: PR: #4640
Improve mAES algorithm with patches by @titu1994 :: PR: #4662

</details>

General Improvements

<details><summary>Changelog</summary>

Option to disable mp in VAD via num_workers=1 by @gkucsko :: PR: #4317
Remove redundant bias expand by @xrennvidia :: PR: #4382
Add option for specifying wandb save_dir from config by @shan18 :: PR: #4379
Quick wav2vec fix. In-place operation adding convolutional positions … by @bonham79 :: PR: #4383
Fixing import error in some cases by @borisfom :: PR: #4401
Update with new conformer checkpoints. by @VahidooX :: PR: #4417
Wav2vec fix by @bonham79 :: PR: #4467
Relative Audio Paths by @stevehuang52 :: PR: #4470
Allow Noam lr scheduler to run for more than max_steps by @alancucki :: PR: #4472
Support for Different LRs with Param Groups by @stevehuang52 :: PR: #4508
Fix runtime check by @borisfom :: PR: #4501
Update finetune label models by @nithinraok :: PR: #4504
Weighted bucketing by @tbartley94 :: PR: #4530
Relative Audio Path by @stevehuang52 :: PR: #4520
Fix duplex inference with grammars by @ekmb :: PR: #4517
Add nsys profiling by @ericharper :: PR: #4539
Remove the variable that is not used in the context. by @XuesongYang :: PR: #4547
Adding multispeaker fastpitch and hifigan en model links to available… by @subhankar-ghosh :: PR: #4550
Add length ratio filtering script by @MaximumEntropy :: PR: #4551
Relative audio path in speech data explorer by @anteju :: PR: #4570
Dividing generative question-answering CI tests by @ameyasm1154 :: PR: #4600
Updating the default parameters in the example adapters config file by @shan18 :: PR: #4607
Improve normalize_batch ValueError message by @piraka9011 :: PR: #4614
Support listing Hugging Face model info by @titu1994 :: PR: #4619
Update diarization data loader to train meeting data by @tango4j :: PR: #4567
Fix HF check for model card info by @titu1994 :: PR: #4628
Add Github Action for auto webpage build by @titu1994 :: PR: #4645
Empty commit by @titu1994 :: PR: #4646
Force git config for doc build by @titu1994 :: PR: #4647
Correct branch name for github page source by @titu1994 :: PR: #4648
Adding lang id to shard by @bmwshop :: PR: #4649
Fix special tokens in vocab to arguments of constructor by @gwarmstrong :: PR: #4631
Fix apex for r1.11 by @michalivne :: PR: #4666
Update readme by @nithinraok :: PR: #4667
Removed trailing spaces in CI test by @vadam5 :: PR: #4671
Pynini dependency fix by @ekmb :: PR: #4674
Fix for incorrect batch size issue while decoding by @rilango :: PR: #4675
Fix to fetch config file by @nithinraok :: PR: #4699
Fix notebook for buffered inference by @titu1994 :: PR: #4703
Prompt Learning Notebook Bug Fix by @vadam5 :: PR: #4689
Add psutils to mock imports by @ericharper :: PR: #4728
Update Aligner model and tutorial to add NGC checkpoint loading by @redoctopus :: PR: #4714
Updated docs and doc paths by @vadam5 :: PR: #4754
Update r1.11 to new heteronyms list by @redoctopus :: PR: #4745
Update CMUdict with more recent 0.7b entries by @redoctopus :: PR: #4768
Add pynini to Docker container by @artbataev :: PR: #4733
Fix tutorial formatting by @redoctopus :: PR: #4778
Fix initializing weights from ptl ckpt with exclude by @sam1373 :: PR: #4807
T5 prompt learning fixes by @MaximumEntropy :: PR: #4771
Updated inference code and squad scripts by @vadam5 :: PR: #4835
Fix uppercasing mismatch for IPA heteronyms by @redoctopus :: PR: #4860
Set the number of workers to 0 for validation and test sets in all enc-dec models by @MaximumEntropy :: PR: #4790
Fix mha by @yzhang123 :: PR: #4866
ipa bug fix by @ekmb :: PR: #4871
Added utf8 encoding by @vadam5 :: PR: #4892
Fix question answering docs r1p11 by @Zhilin123 :: PR: #4897

</details>

Open source →

1.10.01 Jul 2022

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.05

Known Issues

<details><summary>Issues</summary>

Tutorial: Fastpitch_Training_GermanTTS.ipynb is experimental and still being tested.

</details>

ASR

<details><summary>Changelog</summary>

Multilang asr tutorial by @bmwshop :: PR: #3931
Add ASR with Adapters Tutorial by @titu1994 :: PR: #4149
Add support for Decoder + Joint Adapters for ASR by @titu1994 :: PR: #4189
updating PretrainedModelInfo and benchmark sheet for ASR models by @krishnacpuvvada :: PR: #4259
Remove verbose flag from Dali Index Creator by @titu1994 :: PR: #4309
updating PretrainedModelInfo for ASR SSL models by @krishnacpuvvada :: PR: #4292
Adding docs for ASR SSL by @krishnacpuvvada :: PR: #4303
Add ASR Scores to Docs by @titu1994 :: PR: #4412
[ASR] Replace all paths with /content/ by @titu1994 :: PR: #4427
added conformer mandarin model. by @VahidooX :: PR: #4201
Runtime audio segment sampling for SSL by @krishnacpuvvada :: PR: #4126

</details>

TTS

<details><summary>Changelog</summary>

[TTS] Add volume passthrough to fp for riva by @blisc :: PR: #4167
Update TTS Configs from LAMB to AdamW by @redoctopus :: PR: #4233
Add benchmark=false to all TTS configs by @redoctopus :: PR: #4263
[TTS] add staticmethod decoration for BetaBinomialInterpolator by @XuesongYang :: PR: #4319
[TTS] capture exception of non-supported windows. by @XuesongYang :: PR: #4320
[TTS] enforced pin_memory = True by @XuesongYang :: PR: #4341
[TTS] Training Fastpitch on German text and phonemes and finetuning HiFi-GAN on predicted mels by @aroraakshit :: PR: #4266
IPA support for TTS by @redoctopus :: PR: #4310
Bits of RADTTS support by @borisfom :: PR: #4343

</details>

NLP / NMT

<details><summary>Changelog</summary>

Megatron NMT Restore from T5/BART and finetune by @MaximumEntropy :: PR: #3977
Binarized memmap dataloader for Megatron NMT, Inference and checkpoint -> nemo by @MaximumEntropy :: PR: #4137
Use unique names for temporary directories in punctuation and capitalization tests by @PeganovAnton :: PR: #4298
Removes debug logging statements in Megatron NMT by @MaximumEntropy :: PR: #4312
Raise error if trainer object is None for MegatronBaseModel by @MaximumEntropy :: PR: #4356
Punctuation and capitalization tests race condition by @PeganovAnton :: PR: #4399
unify intent slot dataset util functions in tutorials by @Zhilin123 :: PR: #4445
Fix for TP=2,PP=2 decoding with megatron encoder-decoder models by @MaximumEntropy :: PR: #4484
Add RETRO model for pretraining by @yidong72 :: PR: #4121
Add async grad allreduce and chunk optimization by @xrennvidia :: PR: #4084
Implements the UL2 Dataset and config by @MaximumEntropy :: PR: #4184
Add RETRO indexed dataset and inference by @yidong72 :: PR: #4220
Finetune T5 on the prefix-lm objective by @MaximumEntropy :: PR: #4328
Fuse bias with geglu in ParallelMLP by @xrennvidia :: PR: #4213
Support larger datasets for question answering by @Zhilin123 :: PR: #4205
Refactor bias act fusion by @MaximumEntropy :: PR: #4376
Prompt Learning Pipeline Parallel by @vadam5 :: PR: #4291
Text memmap dataset by @michalivne :: PR: #4068
Fuse grad division into async grad allreduce by @xrennvidia :: PR: #4327

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

[TN] WFST to normalize punctuation by @ekmb :: PR: #4108
[TN/TTS] Add graph to tag IPA words/sentences in square brackets and leave them unchanged by @ekmb :: PR: #4323
Tn tutorial by @yzhang123 :: PR: #4090
[TN] WFST to normalize punctuation by @ekmb :: PR: #4108
Tn add rules by @yzhang123 :: PR: #4302
[TN/TTS] Add graph to tag IPA words/sentences in square brackets and leave them unchanged by @ekmb :: PR: #4323
Tn install by @yzhang123 :: PR: #4055
Fix electronic bug, new time ITN rule by @ekmb :: PR: #4355
[TN] Bug fix: expand serial coverage of unknown symbol, remove constraints from word graph by @ekmb :: PR: #4463
Configure T5 finetuning metrics by @MaximumEntropy :: PR: #4122

</details>

Export

<details><summary>Changelog</summary>

Added support for subnet export by @borisfom :: PR: #4299

</details>

Core

<details><summary>Changelog</summary>

Add Module-level Adapters, Save-Restore and tests by @titu1994 :: PR: #4114
Add NeMo Adapters tutorial to Core by @titu1994 :: PR: #4311
NeMo Model to HF Hub Upload Tutorial by @titu1994 :: PR: #4322

</details>

General Improvements and Fixes

<details><summary>Changelog</summary>

Update container to 22.05 by @ericharper :: PR: #4329
Fix PTL step calculation by @titu1994 :: PR: #4307
[NLP] P&C Fix multi node cache issue, add pynini guard by @ekmb :: PR: #4410
NeMo Megatron GPT Unit Tests by @ericharper :: PR: #4099
Add the PP2 GPT eval CI test by @yidong72 :: PR: #4168
BigNLP perf regression fix by @MaximumEntropy :: PR: #4267
Fixes for Megatron Base Model Artifacts by @MaximumEntropy :: PR: #4248
Fix a wrong description in offline_diarization_with_asr.yaml by @tango4j :: PR: #4141
bugfix for import error in Offline_ASR_with_VAD_for_CTC_models by @fayejf :: PR: #4424
[Fix] ASR RNNT Tutorial by @stevehuang52 :: PR: #4352
[TTS] Fix Hifigan finetune tutorial by @subhankar-ghosh :: PR: #4182
[Bugfix][TTS] wrong order of returned tuple for general_collate_fn. by @XuesongYang :: PR: #4432
[bugfix][TTS] pitch, voiced_mask, prob_voiced have the same values. by @XuesongYang :: PR: #4435
[TTS] [bugfix] German FastPitch HiFi-GAN tutorial and lr by @aroraakshit :: PR: #4459
[TTS] [bugfix] update indentation by @aroraakshit :: PR: #4468
Fix some 's' cases for IPA G2P by @redoctopus :: PR: #4460
Fix ASR Typos in tutorials by @titu1994 :: PR: #4384
Use unique names for temporary directories in punctuation and capitalization tests by @PeganovAnton :: PR: #4298
Punctuation and capitalization tests race condition by @PeganovAnton :: PR: #4399
Dialogue tasks unit test by @Zhilin123 :: PR: #4112
fix error by @yzhang123 :: PR: #4120
fix typo by @stevehuang52 :: PR: #4134
Fix cmudict typo: phoneme YI1 -> IY1 in NVME by @redoctopus :: PR: #4139
transcribe: scan directories recursively by @virajkarandikar :: PR: #4159
Add 44KHz yaml file for Fastpitch training by @subhankar-ghosh :: PR: #4161
[bugfix] consistent highfreq to both fastpitch and hifigan in their 44100 configs. by @XuesongYang :: PR: #4177
Upperbound OmegaConf by @titu1994 :: PR: #4191
Prompt tokenization bugfix by @vadam5 :: PR: #4197
Updated to Prompt Learning Model to Use Distributed Sampler by @vadam5 :: PR: #4208
Freesound fixes by @virajkarandikar :: PR: #4155
Patch Hydra by @titu1994 :: PR: #4202
Prompt Learning Model Saving Changes by @vadam5 :: PR: #4212
Speakertasks manifest by @yzhang123 :: PR: #4185
SSL Multi-loss Update by @sam1373 :: PR: #4186
Support load_adapters with just adapter_name by @titu1994 :: PR: #4255
Add special tokens to existing (trained) SentencePiece models by @aklife97 :: PR: #4203
Fixing the speed slow-down for speech models. by @VahidooX :: PR: #4260
Fix and add functions in speaker utils by @tango4j :: PR: #4138
pt container 1.10->1.11.0 by @ekmb :: PR: #4273
ssl fixes by @sam1373 :: PR: #4268
Save Virtual Prompt Weights Only by @vadam5 :: PR: #4237
add 'relative positional embedding (RPE)' feature - re-creating after… by @khcs :: PR: #4256
Docs CSS: Update h4 tag style for the right side bar by @nickolyamba :: PR: #4284
Fix Docs CSS: align docs left and increase width for large screens by @nickolyamba :: PR: #4154
remove redundant condition for fastpitch. by @XuesongYang :: PR: #4281
[Add] automaticly resolving relative audio path by @stevehuang52 :: PR: #4277
forcing conv subsampling to 32 bit by @bmwshop :: PR: #4293
Add library name and version when downloading from the Hugging Face Hub by @osanseviero :: PR: #4304
clear access registry when adding if not empty by @sam1373 :: PR: #4306
[collections] bugfix for capturing NotImplementedError of non-supported sup data types. by @XuesongYang :: PR: #4297
Adjust lr for AdamW from LAMB default by @redoctopus :: PR: #4308
Fix bugs in indexed dataset exam script by @yidong72 :: PR: #4325
Torchaudio installation fix by @GNroy :: PR: #4330
Speedup the speech commands dataset processing script by @shan18 :: PR: #4347
fix wrong requirement by @yzhang123 :: PR: #4349
Refactored path to manifest by @treacker :: PR: #4251
Fix the post LN bug by @yidong72 :: PR: #4350
[Fix] Hanging for Fully Randomized Bucketing by @stevehuang52 :: PR: #4348
Auto-switch the input dimensions in the conformer encoder adapter to correct value by @shan18 :: PR: #4354
Set headscale false by @MaximumEntropy :: PR: #4364
Add wandb as dependency by @titu1994 :: PR: #4365
Fix trainer.global_steps in WandB logging by @titu1994 :: PR: #4366
Finetuning changes for BART by @MaximumEntropy :: PR: #4003
Make position embedding expansion specific to a batch to avoid checkpoint size mismatches by @MaximumEntropy :: PR: #4357
Correct support for dataclasses in default module dim by @titu1994 :: PR: #4372
Fix no attribute 'pad_id' bug when pre-processing by @yidong72 :: PR: #4377
Question answering bug fix by @Zhilin123 :: PR: #4381
Docs for NeMo Adapters by @titu1994 :: PR: #4369
Update NeMo docs by @titu1994 :: PR: #4397
Fixing import error in some cases by @borisfom :: PR: #4402
Fix tutorial typos and docs by @titu1994 :: PR: #4415
Add reconfigure on validation epoch start by @MaximumEntropy :: PR: #4393
Re-apply fixes from r1.9.0 by @redoctopus :: PR: #4425
Fix hanging issue by multiprocessing in SD tutorial and add ETA for VAD processing by @fayejf :: PR: #4405
Fix notebook text by @yidong72 :: PR: #4438
Update dialogue tutorial version by @Zhilin123 :: PR: #4437
Docs: Add table overflow handling by @nickolyamba :: PR: #4441
Docs: Decrease Font Size on Tables by @nickolyamba :: PR: #4444
Notebook bug fix: add subfolder by @ekmb :: PR: #4442
Fix typo in HiFi-GAN config's max steps by @redoctopus :: PR: #4446
Updated notebook to fix batch configuration and precision bugs by @vadam5 :: PR: #4447
fix branch in link by @ekmb :: PR: #4454
t5-rpe-fix targeting r1.10.0; raise exception for PP>2. by @khcs :: PR: #4469
Add kwargs to exact string match by @MaximumEntropy :: PR: #4479

</details>

Open source →

1.9.03 Jun 2022Deprecated

Deprecation by @blisc :: PR: #4082

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.04

ASR

<details><summary>Changelog</summary>

Fix changed function name in offline vad asr notebeook by @fayejf :: PR: #4007
NeMo Adapters Support + ASR Adapters by @titu1994 :: PR: #3942
Update ASR configs with num_workers and pin_memory by @titu1994 :: PR: #4270
Verbose k2 install, skip if failed by @GNroy :: PR: #4289
Torch conversion for VAD-Diarization pipeline by @tango4j :: PR: #3930
Multiprocess improvements by @nithinraok :: PR: #4127

</details>

TTS

<details><summary>Changelog</summary>

Tn tts e by @ekmb :: PR: #3988
Remove AudioToCharWithPriorAndPitchDataset dependency from fastpitch by @subhankar-ghosh :: PR: #4008
Deprecation by @blisc :: PR: #4082
FastPitch FT notebook - Improving Speech Quality clarifications by @redoctopus :: PR: #3954

</details>

NLP / NMT

<details><summary>Changelog</summary>

Option to remove bias terms from Megatron transformers by @MaximumEntropy :: PR: #3973
Add NMT method to translate with TN/ITN pre/post-processing by @MaximumEntropy :: PR: #4009
Fix Punctuation and Capitalization model batching. An issue with shuffling. by @PeganovAnton :: PR: #4050
Fix GPT model parallel eval by @yidong72 :: PR: #4054
Updating with main by @jpilaul :: PR: #4073
Cherry-pick fix for megatron ckpt conversion script when using BCP by @ericharper :: PR: #4089
Check implicit grad acc in GLUE dataset building by @MaximumEntropy :: PR: #4123
Fix/punctuation avoid overwritting tmp files by @PeganovAnton :: PR: #4144
Fix/punctuation/trainer required for setting test data by @PeganovAnton :: PR: #4199
Raise error if bicleaner is not installed in NMT Data preprocesing notebook by @MaximumEntropy :: PR: #4264
Fix epoch end for NeMo NMT by @MaximumEntropy :: PR: #4265
Update YAML with trainer.benchmark=False for NLP by @MaximumEntropy :: PR: #4261
Add NMT method to translate with TN/ITN pre/post-processing by @MaximumEntropy :: PR: #4009
Continuous prompt refactor by @vadam5 :: PR: #3877
T5 finetuning for generic small text-to-text datasets by @MaximumEntropy :: PR: #4032

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Tn special text support by @yzhang123 :: PR: #3969
Tn update numbers by @yzhang123 :: PR: #3992
Tn tts e by @ekmb :: PR: #3988
Itn vi by @yzhang123 :: PR: #4029
Refactor tn data folder, and update of measure by @yzhang123 :: PR: #4028
Remove conda dependency for tn by @yzhang123 :: PR: #4057
Tn electronic by @yzhang123 :: PR: #4053
ThutmoseTaggerModel, a new model for inverse text normalization by @bene-ges :: PR: #4011
Tutorial on ITN with Thutmose tagger and small fixes by @bene-ges :: PR: #4117
Cleaned up TN/ ITN doc by @yzhang123 :: PR: #4119
Update default for SH by @ekmb :: PR: #4135
Update ContextNet version by @titu1994 :: PR: #4207

</details>

NeMo Tools

<details><summary>Changelog</summary>

Added exception handling for audio player in SDE by @vsl9 :: PR: #4077

</details>

NeMo Core

<details><summary>Changelog</summary>

Support pre-extracted nemo checkpoint for restoration by @titu1994 :: PR: #4061
Fix type checking to be compatible with named tuples by @artbataev :: PR: #3986
Update num worker calculation due to PTL flag changes by @redoctopus :: PR: #4056
Refresh NeMo documentation to Sphinx Book Theme by @titu1994 :: PR: #3996
Generalize adapter merge strategy for future adapters by @titu1994 :: PR: #4091

</details>

General Improvements

<details><summary>Changelog</summary>

Fix Punctuation and Capitalization model batching. An issue with shuffling. by @PeganovAnton :: PR: #4050
Fix restoring from checkpoint for case when is provided by @PeganovAnton :: PR: #4136
Fix/punctuation avoid overwritting tmp files by @PeganovAnton :: PR: #4144
Fix/punctuation/trainer required for setting test data by @PeganovAnton :: PR: #4199
Ability to set log_prediction to false by @bmwshop :: PR: #3929
Glu activation variants by @MaximumEntropy :: PR: #3951
Ranking merge by @yzhang123 :: PR: #3906
Fix path in doc by @nithinraok :: PR: #3979
Adding fisher audio conversion script from old NeMo branch by @jbalam-nv :: PR: #3991
improvements to geet_commonvoice_data script by @bmwshop :: PR: #3999
Bugfix and variable name change for clustering code by @tango4j :: PR: #4023
Exp manager log rank 0 only arguments by @MaximumEntropy :: PR: #4026
Force import test on PR by @titu1994 :: PR: #4037
Drop support for kaldi-io by @titu1994 :: PR: #4042
Cherry pick HF integration and bug fixes from 1.8.1 by @ericharper :: PR: #4052
Make saving prompt encoder embeddings non-configurable by @vadam5 :: PR: #4071
Replace sampled tokens with EOD after EOD has been sampled once by @vadam5 :: PR: #4070
Added answer only loss for prompt learning by @vadam5 :: PR: #4069
added stacking suport to conformer. by @VahidooX :: PR: #4045
Update LJSpeech whitelist file path by @redoctopus :: PR: #4078
Added check for microbatch calculator by @vadam5 :: PR: #4043
Prompt Learning Docs by @vadam5 :: PR: #4046
Fix link to prompt tuning page by @SeanNaren :: PR: #4081
Add docs for by @titu1994 :: PR: #4079
Dialogue task by @Zhilin123 :: PR: #3884
RMSNorm, Normformer and fixes from merging 1.8.0 into main by @MaximumEntropy :: PR: #4048
Correct link to PTL by @titu1994 :: PR: #4088
Added encoder and decoder modules for RETRO model by @yidong72 :: PR: #4038
Upgrade container to NGC PyTorch 22.04 by @ericharper :: PR: #4085
Tarred fix label models by @nithinraok :: PR: #4092
Fix link to tutorial in dialogue docs by @Zhilin123 :: PR: #4093
Prompt learning Notebook by @vadam5 :: PR: #4031
Add more papers by @yzhang123 :: PR: #4097
Ignore speakers with few utterances by @nithinraok :: PR: #3722
Access mixin by @sam1373 :: PR: #4098
Add CharParser for Cyrillic letters by @karpov-nick :: PR: #4101
Restored tests previously disabled for 22.03 base by @borisfom :: PR: #4109
Add augmentation to label models by @nithinraok :: PR: #4113
Fix register artifacts by @ramanathan831 :: PR: #4116
Fix typo by @yzhang123 :: PR: #4140
bug_fix_diarization_manifest_creation by @yzhang123 :: PR: #4125
Tacotron2 retrain by @treacker :: PR: #4103
WaveGlow input type fixes by @redoctopus :: PR: #4151
Notebooks' link, typo and import fix by @fayejf :: PR: #4158
Thutmose tagger bug fixes by @bene-ges :: PR: #4162
Update speaker docs by @nithinraok :: PR: #4164
Set plugin to None when no apex by @ekmb :: PR: #4171
Fix doc by @yzhang123 :: PR: #4152
Small import name fix by @fayejf :: PR: #4180
Rename folder VAD -> vad by @fayejf :: PR: #4163
Fix the server key value problem in the notebook by @yidong72 :: PR: #4196
Pin omegaconf for r1.9.0 by @ericharper :: PR: #4195
Fix cherrypicks by @titu1994 :: PR: #4204
Fix bugs for dialogue tutorial by @Zhilin123 :: PR: #4211
Tacotron2 1.9.0 bugfixes by @redoctopus :: PR: #4209
Add docs for Thutmose Tagger by @bene-ges :: PR: #4173
Dialogue tutorial fix by @Zhilin123 :: PR: #4221
Fix syntax error in ipynb-file by @bene-ges :: PR: #4228
Fix JSON serialization problem by @yidong72 :: PR: #4235
Prompt Learning Typo Fixes by @vadam5 :: PR: #4238
Fixing bug 3642622 by @pasandi20 :: PR: #4250
Fix broken link in the tutorial by @bene-ges :: PR: #4257
Prompt learning notebook bugfix by @vadam5 :: PR: #4262
Fix missing validation dataset, whitelist certain keywords for datasets by @titu1994 :: PR: #4269
Set Save on train end to false by @vadam5 :: PR: #4274
Updated config to fix CI test OOM error by @vadam5 :: PR: #4279
Changed total virtual prompt tokens by @vadam5 :: PR: #4295

</details>

Open source →

1.8.020 Apr 2022Deprecated

Bump TTS deprecation version to 1.9 by @blisc :: PR: #3955

Known Issues

<details><summary>Issues</summary>

Megatron BERT export does not currently work in the NVIDIA NGC PyTorch 22.03 container. The issue will be fixed in the NGC PyTorch 22.04 container.
pytest for Vietnamese inverse text normalization are failing. Fixed in main

</details>

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:22.03

ASR

<details><summary>Changelog</summary>

ASR SSL Update by @sam1373 :: PR: #3714
Polylang asr by @bmwshop :: PR: #3721
Test grad accumulation for RNNT loss by @titu1994 :: PR: #3731
Add readme files describing model execution flow for ASR tasks by @titu1994 :: PR: #3812
add fr asr ckpt to doc by @yzhang123 :: PR: #3809
Fix asr tests in 22.02 by @titu1994 :: PR: #3823
Add new pretrained Spanish ASR models by @erastorgueva-nv :: PR: #3830
Documentation updates for ASR by @titu1994 :: PR: #3846
Offline VAD+ASR tutorial by @fayejf :: PR: #3828
Added Hindi and Marathi Models in Nemo pretrained ASR_CTC_BPE models … by @meghmak13 :: PR: #3856
Add a missing line to ASR_with_NeMo.ipynb by @lifefeel :: PR: #3908
Multilang asr models by @bmwshop :: PR: #3907
added stt_en_conformer_transducer_large_ls to NGC by @VahidooX :: PR: #3920
Fix DALI test on 22.03 by @titu1994 :: PR: #3911
Adding RNN encoder for LSTM-Transducer and LSTM-CTC models by @VahidooX :: PR: #3886
Fix issue with Segfault in ASR models by @titu1994 :: PR: #3956
Added Mandarin pretrained Conformer-Transducer-Large model trained on AISHELL2. by @VahidooX :: PR: #3970

</details>

TTS

<details><summary>Changelog</summary>

Bump TTS deprecation version to 1.9 by @blisc :: PR: #3955
Add pinned pynini and scipy installs to TTS training tutorial by @redoctopus :: PR: #3967
Compatability override to load_state_dict for old TTS checkpoints by @redoctopus :: PR: #3978

</details>

NLP / NMT

<details><summary>Changelog</summary>

Use worker processes for data preprocessing by @crcrpar :: PR: #3665
Set find_unused_parameters to False in GPT example script by @ericharper :: PR: #3837
GPT multinode eval by @ericharper :: PR: #3821
Fix MegatronPretrainingRandomSampler by taking into account by @crcrpar :: PR: #3826
Add slot filling into DST Generative model by @Zhilin123 :: PR: #3695
Disable nvfuser for gpt by @ericharper :: PR: #3845
Multi-Label Joint Intent Slot Classification by @chenrichard10 :: PR: #3742
fix bug in intent/slot model reloading by @carolmanderson :: PR: #3874
Make test_gpt_eval unit test less strict by @yidong72 :: PR: #3898
Comment gpt resume ci test by @MaximumEntropy :: PR: #3901
Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) by @MaximumEntropy :: PR: #3861
Megatron support by @ramanathan831 :: PR: #3893
Populate the GPT/BERT with uploaded models by @yidong72 :: PR: #3885
Megatron BART by @michalivne :: PR: #3666
Additional Japanese processor for NMT that uses MeCab segmentation. Fix for BLEU in one-many NMT by @MaximumEntropy :: PR: #3889
NMT GRPC sever URL fix by @MaximumEntropy :: PR: #3918
Megatron legacy conversion support by @ramanathan831 :: PR: #3919
Update max_epochs on megatron configs by @ericharper :: PR: #3958
Fix NMT variable passing bug by @aklife97 :: PR: #3985
Fix nemo megatron restore with artifacts by @ericharper :: PR: #3997
Fix megatron notebook by @ramanathan831 :: PR: #4004
Megatron work-arounds by @borisfom :: PR: #3998
Add T5 model P-tuning support by @yidong72 :: PR: #3768
Make index mappings dir configurable by @ericharper :: PR: #3868
T5 pipeline parallel by @MaximumEntropy :: PR: #3750

</details>

Text Normalization / Inverse Text Normalization

<details><summary>Changelog</summary>

Tn es by @bonham79 :: PR: #3632
Fix single GPU training issue + change deprecated Lightning args by @aklife97 :: PR: #4010

</details>

Export

<details><summary>Changelog</summary>

Conformer WARs for TRT8.2 by @borisfom :: PR: #3787
bert_module: fix inputs of export model by @virajkarandikar :: PR: #3815
Exports 22.03 war by @borisfom :: PR: #3957

</details>

Bugfixes

<details><summary>Changelog</summary>

patch librosa deprecation and fix by @fayejf :: PR: #3818

</details>

General Improvements

<details><summary>Changelog</summary>

Pynini pip by @yzhang123 :: PR: #3726
upgrade PTL trainer flags by @nithinraok :: PR: #3589
Updated Speech Data Explorer by @vsl9 :: PR: #3710
Fix spelling error in num_workers parameter to actually set number of dataset workers specified in yaml configs by @themikem :: PR: #3800
Support for Camembert Huggingface bert-like models by @itzsimpl :: PR: #3799
Update to 22.02 by @ericharper :: PR: #3771
Fixing the defaults of conformer models in the config files by @VahidooX :: PR: #3836
Fix T5 Encoder Mask while decoding by @MaximumEntropy :: PR: #3838
fix: multilingual transcribe does not require lang id param by @bmwshop :: PR: #3833
Misc improvements by @titu1994 :: PR: #3843
Change container by @MaximumEntropy :: PR: #3844
Making gender assignment random for cardinals, fractions, and decimal… by @bonham79 :: PR: #3759
Jenkinsfile test changes by @chenrichard10 :: PR: #3879
Adding a RegEx tokenizers by @michalivne :: PR: #3839
enable bias+dropout+add fusion with nvfuser at inference by @erhoo82 :: PR: #3869
Add text_generation_util to support TopK, TopP sampling + Tabular Data Generation. by @yidong72 :: PR: #3834
Ptl requirements bound by @MaximumEntropy :: PR: #3903
doc links update by @ekmb :: PR: #3891
add citations by @yzhang123 :: PR: #3902
Update NeMo CI to 22.03 by @MaximumEntropy :: PR: #3900
Add domain groups to changelog builder by @titu1994 :: PR: #3904
add input threshhold by @yzhang123 :: PR: #3913
improvements to commonvoice data script by @bmwshop :: PR: #3892
fixes to the cleanup flag by @bmwshop :: PR: #3921
Upgrade to PTL 1.6.0 by @ericharper :: PR: #3890
JSON output from diarization now includes sentences. Optimized senten… by @demsarjure :: PR: #3897
Stateless timer fix for PTL 1.6 by @MaximumEntropy :: PR: #3925
fix save_best missing chpt bug, update for setup_tokenizer() changes by @ekmb :: PR: #3932
Fix tarred sentence dataset length by @MaximumEntropy :: PR: #3941
remove old doc by @ekmb :: PR: #3946
Fix issues with librosa deprecations by @titu1994 :: PR: #3950
Fix notebook bugs for branch r1.8.0 by @yidong72 :: PR: #3948
Fix global batch fit loop by @ericharper :: PR: #3936
Refactor restorefrom by @ramanathan831 :: PR: #3927
Fix variable name and move models to CPU in Change partition by @aklife97 :: PR: #3972
Fix notebook error by @yidong72 :: PR: #3975
Notebook Bug Fixes for r1.8.0 by @vadam5 :: PR: #3989
Fix compat override for TalkNet Aligner by @redoctopus :: PR: #3993
docs fixes by @ekmb :: PR: #3987
Fixes val_check_interval, skip loading train data during eval by @MaximumEntropy :: PR: #3968
LogProb calculation performance fix by @yidong72 :: PR: #3984
Fix P-Tune T5 model by @yidong72 :: PR: #4001
Fix the broadcast shape mismatch by @yidong72 :: PR: #4017
Add known issues to notebook by @ericharper :: PR: #4024

</details>

Open source →

nemo-toolkit

Release timeline

Releases

NVIDIA NeMo Speech 3.0 Release Notes

Top-Level Changes and Highlights

Breaking Changes and Migration Notes

SpeechLM2 and NeMo Automodel Backend

Nemotron VoiceChat Training Modules

ASR

TTS

Audio

DataLoader and Speech Data Tools

Docs

Repo Rename and 3.0 Release

Technical Debt Reduction

Installation, Packaging, and Containers

Tests and Release Hardening

Detailed PR Breakdown

Release Identity and Docs

Technical-Debt Cleanup

Install, Packaging, Docker, Dependencies

Security and Safety

Tests, CI, Release Reliability

SpeechLM2, SALMAutomodel, Scaling, vLLM

Nemotron VoiceChat, Duplex Models, Voice Agent

ASR, Streaming, Prompt Models, Transducers

TTS, MagpieTTS, Codecs, Text

Audio

DataLoader and Speech Data Tools

Smaller Fixes

Highlights

Uncategorized:

ASR

Uncategorized:

ASR

Uncategorized:

Highlights

Detailed Changelogs:

ASR

TTS

NLP / NMT

Text Normalization / Inverse Text Normalization

NeMo Tools

Export

Bugfixes

Uncategorized:

Highlights

Detailed Changelogs:

Uncategorized:

Highlights

Detailed Changelogs:

ASR

TTS

NLP / NMT

Export

Uncategorized:

Highlights

Known Issues

Detailed Changelogs:

ASR

TTS

NLP / NMT

Export

Uncategorized:

Highlights

Detailed Changelogs:

Text Normalization / Inverse Text Normalization

Uncategorized:

Detailed Changelogs:

Text Normalization / Inverse Text Normalization

Uncategorized:

Highlights

Detailed Changelogs:

ASR

TTS

NLP / NMT

Uncategorized:

Highlights

Detailed Changelogs:

ASR