faster-whisper

What's Changed

• feat: allow passing specific revision to download by @felixmosh in https://github.com/SYSTRAN/faster-whisper/pull/1292
• Support distil-large-v3.5 by @MahmoudAshraf97 in https://github.com/SYSTRAN/faster-whisper/pull/1311
• Feature: Allow loading of private HF models by @r15hil in https://github.com/SYSTRAN/faster-whisper/pull/1309
• bugfix: Get correct chunk index when restoring timestamps by @MahmoudAshraf97 in https://github.com/SYSTRAN/faster-whisper/pull/1336
• Remove Silence in Batched transcription by @MahmoudAshraf97 in https://github.com/SYSTRAN/faster-whisper/pull/1297

New Contributors

• @djpg made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/1267
• @felixmosh made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/1292
• @r15hil made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/1309

Full Changelog: https://github.com/SYSTRAN/faster-whisper/compare/v1.1.1...v1.2.0

What's Changed

• Brings back original VAD parameters naming by @Purfview in https://github.com/SYSTRAN/faster-whisper/pull/1181
• Make batched suppress_tokens behaviour same as in sequential by @Purfview in https://github.com/SYSTRAN/faster-whisper/pull/1194
• Fixes OOM Errors - too high RAM usage by VAD by @Purfview in https://github.com/SYSTRAN/faster-whisper/pull/1198
• Add duration of audio and VAD removed duration to BatchedInferencePipeline by @greenw0lf in https://github.com/SYSTRAN/faster-whisper/pull/1186
• Fix neg_threshold by @Purfview in https://github.com/SYSTRAN/faster-whisper/pull/1191

New Contributors

• @greenw0lf made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/1186

Full Changelog: https://github.com/SYSTRAN/faster-whisper/compare/v1.1.0...v1.1.1

New Features

• New batched inference that is 4x faster and accurate, Refer to README on usage instructions.
• Support for the new large-v3-turbo model.
• VAD filter is now 3x faster on CPU.
• Feature Extraction is now 3x faster.
• Added log_progress to WhisperModel.transcribe to print transcription progress.
• Added multilingual option to transcription to allow transcribing multilingual audio. Note that Large models already have codeswitching capabilities, so this is mostly beneficial to medium model or smaller.
• WhisperModel.detect_language now has the option to use VAD filter and improved language detection using language_detection_segments and language_detection_threshold.

Bug Fixes

• Use correct features padding for encoder input when chunk_length <30s
• Use correct seek value in output

Other Changes

• replace NamedTuple with dataclass in Word, Segment, TranscriptionOptions, TranscriptionInfo, and VadOptions, this allows conversion to json without nesting. Note that _asdict() method is still available in Word and Segment classes for backward compatibility but will be removed in the next release, you can use dataclasses.asdict() instead.
• Added new tests for development
• Updated benchmarks in the Readme
• use jiwer instead of evaluate in benchmarks
• Filter out non_speech_tokens in suppressed tokens by @jordimas in https://github.com/SYSTRAN/faster-whisper/pull/898

New Contributors

• @Jiltseb made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/856
• @heimoshuiyu made their first contribution in https://github.com/SYSTRAN/faster-whisper/pull/1092

Full Changelog: https://github.com/SYSTRAN/faster-whisper/compare/v1.0.3...v1.1.0

Upgrade Silero-Vad model to latest V5 version (https://github.com/SYSTRAN/faster-whisper/pull/884)

Silero-vad V5 release: https://github.com/snakers4/silero-vad/releases/tag/v5.0

• window_size_samples parameter is fixed at 512.
• Change to use the state variable instead of the existing h and c variables.
• Slightly changed internal logic, now some context (part of previous chunk) is passed along with the current chunk.
• Change the dimensions of the state variable from 64 to 128.
• Replace ONNX file with V5 version

Other changes

• Improve language detection when using clip_timestamps (#867)
• Docker file improvements (#848)
• Fix #839 incorrect clip_timestamps being used in model (#842)

• Add support for distil-large-v3 (https://github.com/SYSTRAN/faster-whisper/pull/755) The latest Distil-Whisper model, distil-large-v3, is intrinsically designed to work with the OpenAI sequential algorithm.

• Benchmarks (https://github.com/SYSTRAN/faster-whisper/pull/773) Introduces functionality to measure benchmarking for memory, Word Error Rate (WER), and speed in Faster-whisper.

• Support initializing more whisper model args (https://github.com/SYSTRAN/faster-whisper/pull/807)

• Small bug fix:

  • code breaks if audio is empty (https://github.com/SYSTRAN/faster-whisper/pull/768)
  • Foolproof: Disable VAD if clip_timestamps is in use (https://github.com/SYSTRAN/faster-whisper/pull/769)
  • make faster_whisper.assets as a valid python package to distribute (https://github.com/SYSTRAN/faster-whisper/pull/774)
  • Loosen tokenizers version constraint (https://github.com/SYSTRAN/faster-whisper/pull/804)
  • CUDA version and updated installation instructions (https://github.com/SYSTRAN/faster-whisper/pull/785)

• New feature from original openai Whisper project:

  • Feature/add hotwords (https://github.com/SYSTRAN/faster-whisper/pull/731)
  • Improve language detection (https://github.com/SYSTRAN/faster-whisper/pull/732)

• Bug fixes and performance improvements:
  • Update logic to get segment from features before encoding (https://github.com/SYSTRAN/faster-whisper/pull/705)
  • Fix window end heuristic for hallucination_silence_threshold (https://github.com/SYSTRAN/faster-whisper/pull/706)

• Support distil-whisper model (https://github.com/SYSTRAN/faster-whisper/pull/557) Robust knowledge distillation of the Whisper model via large-scale pseudo-labelling.
  For more detail: https://github.com/huggingface/distil-whisper

• Upgrade ctranslate2 version to 4.0 to support CUDA 12 (https://github.com/SYSTRAN/faster-whisper/pull/694)

• Upgrade PyAV version to 11.* to support Python3.12.x (https://github.com/SYSTRAN/faster-whisper/pull/679)

• Small bug fixes

  • Illogical "Avoid computing higher temperatures on no_speech" (https://github.com/SYSTRAN/faster-whisper/pull/652)
  • broken prompt_reset_on_temperature (https://github.com/SYSTRAN/faster-whisper/pull/604)
  • Word timing tweaks (https://github.com/SYSTRAN/faster-whisper/pull/616)

• New improvements from original OpenAI Whisper project

  • Skip silence around hallucinations (https://github.com/SYSTRAN/faster-whisper/pull/646)
  • Prevent infinite loop for out-of-bound timestamps in clip_timestamps (https://github.com/SYSTRAN/faster-whisper/pull/697)

Fix the broken tag v0.10.0

• Support "large-v3" model with
  • The ability to load feature_size/num_mels and other from preprocessor_config.json
  • A new language token for Cantonese (yue)
• Update CTranslate2 requirement to include the latest version 3.22.0
• Update tokenizers requirement to include the latest version 0.15
• Change the hub to fetch models from Systran organization

• Add function faster_whisper.available_models() to list the available model sizes
• Add model property supported_languages to list the languages accepted by the model
• Improve error message for invalid task and language parameters
• Update tokenizers requirement to include the latest version 0.14

Expose new transcription options

Some generation parameters that were available in the CTranslate2 API but not exposed in faster-whisper:

• repetition_penalty to penalize the score of previously generated tokens (set > 1 to penalize)
• no_repeat_ngram_size to prevent repetitions of ngrams with this size

Some values that were previously hardcoded in the transcription method:

• prompt_reset_on_temperature to configure after which temperature fallback step the prompt with the previous text should be reset (default value is 0.5)

Other changes

• Fix a possible memory leak when decoding audio with PyAV by forcing the garbage collector to run
• Add property duration_after_vad in the returned TranscriptionInfo object
• Add "large" alias for the "large-v2" model
• Log a warning when the model is English-only but the language parameter is set to something else

• Fix a bug related to no_speech_threshold: when the threshold was met for a segment, the next 30-second window reused the same encoder output and was also considered as non speech
• Improve selection of the final result when all temperature fallbacks failed by returning the result with the best log probability

Improve word-level timestamps heuristics

Some recent improvements from openai-whisper are ported to faster-whisper:

• Squash long words at window and sentence boundaries (https://github.com/openai/whisper/commit/255887f219e6b632bc1a6aac1caf28eecfca1bac)
• Improve timestamp heuristics (https://github.com/openai/whisper/commit/f572f2161ba831bae131364c3bffdead7af6d210)

Support download of user converted models from the Hugging Face Hub

The WhisperModel constructor now accepts any repository ID as argument, for example:

model = WhisperModel("username/whisper-large-v2-ct2")

The utility function download_model has been updated similarly.

Other changes

• Accept an iterable of token IDs for the argument initial_prompt (useful to include timestamp tokens in the prompt)
• Avoid computing higher temperatures when no_speech_threshold is met (same as https://github.com/openai/whisper/commit/e334ff141d5444fbf6904edaaf408e5b0b416fe8)
• Fix truncated output when using a prefix without disabling timestamps
• Update the minimum required CTranslate2 version to 3.17.0 to include the latest fixes

Extend TranscriptionInfo with additional properties

• all_language_probs: the probability of each language (only set when language=None)
• vad_options: the VAD options that were used for this transcription

Improve robustness on temporary connection issues to the Hugging Face Hub

When the model is loaded from its name like WhisperModel("large-v2"), a request is made to the Hugging Face Hub to check if some files should be downloaded.

It can happen that this request raises an exception: the Hugging Face Hub is down, the internet is temporarily disconnected, etc. These types of exception are now catched and the library will try to directly load the model from the local cache if it exists.

Other changes

• Enable the onnxruntime dependency for Python 3.11 as the latest version now provides binary wheels for Python 3.11
• Fix occasional IndexError on empty segments when using word_timestamps=True
• Export __version__ at the module level
• Include missing requirement files in the released source distribution

Fix download_root to correctly set the cache directory where the models are downloaded.

Improved logging

Some information are now logged under INFO and DEBUG levels. The logging level can be configured like this:

import logging

logging.basicConfig()
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)

More control over model downloads

New arguments were added to the WhisperModel constructor to better control how the models are downloaded:

• download_root to specify where the model should be downloaded.
• local_files_only to avoid downloading the model and directly return the path to the cached model, it it exists.

Other changes

• Improve the default VAD behavior to prevent some words from being assigned to the incorrect speech chunk in the original audio
• Fix incorrect application of option condition_on_previous_text=False (note that the bug still exists in openai/whisper v20230314)
• Fix segment timestamps that are sometimes inconsistent with the words timestamps after VAD
• Extend the Segment structure with additional properties to match openai/whisper
• Rename AudioInfo to TranscriptionInfo and add a new property options to summarize the transcription options that were used

Fix some IndexError exceptions:

• when VAD is enabled and a predicted timestamp is after the last speech chunk
• when word timestamps are enabled and the model predicts a tokens sequence that is decoded to invalid Unicode characters

Integration of Silero VAD

The Silero VAD model is integrated to ignore parts of the audio without speech:

model.transcribe(..., vad_filter=True)

The default behavior is conservative and only removes silence longer than 2 seconds. See the README to find how to customize the VAD parameters.

Note: the Silero model is executed with onnxruntime which is currently not released for Python 3.11. The dependency is excluded for this Python version and so the VAD features cannot be used.

Speaker diarization using stereo channels

The function decode_audio has a new argument split_stereo to split stereo audio into seperate left and right channels:

left, right = decode_audio(audio_file, split_stereo=True)

# model.transcribe(left)
# model.transcribe(right)

Other changes

• Add Segment attributes avg_log_prob and no_speech_prob (same definition as openai/whisper)
• Ignore audio frames raising an av.error.InvalidDataError exception during decoding
• Fix option prefix to be passed only to the first 30-second window
• Extend suppress_tokens with some special tokens that should always be suppressed (unless suppress_tokens is None)
• Raise a more helpful error message when the selected model size is invalid
• Disable the progress bar when the model to download is already in the cache

• Converted models are now available on the Hugging Face Hub and are automatically downloaded when creating a WhisperModel instance. The conversion step is no longer required for the original Whisper models.

# Automatically download https://huggingface.co/guillaumekln/faster-whisper-large-v2
model = WhisperModel("large-v2")

• Run the encoder only once for each 30-second window. Before this change the same window could be encoded multiple times, for example in the temperature fallback or when word-level timestamps is enabled.

Initial publication of the library on PyPI: https://pypi.org/project/faster-whisper/