• Features:
  • Enabled data-aware compression methods (AWQ, GPTQ, Scale Estimation, and LoRA Correction) for all bit widths < 4, including INT2 and INT3 (#4131).
  • (OpenVINO) Added INT2 and INT3 symmetric compression types (#4123).
  • (OpenVINO) Added the GroupedMatMul operation support for data-free weight compression (#4130).
  • (ONNX) Added FP8 support for the ONNX backend (#4072).
• Fixes:
  • Fixed the aggregator formula for the HAWQ sensitivity metric used in mixed-precision weight compression (#4106).
  • Fixed the compression range for asymmetric compression when all weight values are positive or negative (#4074).
  • Fixed weight serialization to support safetensors >= 0.8.0 by converting NumPy scalars to arrays before saving (#4094).
• Tutorials:
  • Post-Training Optimization of OmniVoice Model
  • Post-Training Optimization of HunyuanOCR Model
  • Post-Training Optimization of Unlimited-OCR Model
  • Post-Training Optimization of Bernini-R
• Deprecations/Removals:
  • Removed unused code, including the Statistics, NNCFDataLoader, QuantizersCounter, and QuantizationStatistics classes, and the PyTorch patch_torch_operators, register_module, and PTInitializingDataLoader (#4093).
  • (PyTorch) Removed the unused CompressionParameter class (#4108).
• Requirements:
  • Updated onnx to 1.22.0 (#4101).
  • Updated networkx to <= 3.6.1 (#4059).

• Features:
  • Add support for ignoring specific weight nodes by constant node name within IgnoredScope for nncf.compress_weights and nncf.prune (#4063) (#4064).
• Fixes:
  • (PyTorch) Fixed TorchFX model extractor to prevent capture of nodes that do not affect target outputs (#3982).
  • Fixed QuantizationConstraints.from_config_dict to avoid unexpected behavior (#4046).
  • (OpenVINO) Fixed Scale Estimation to properly handle zero initial_steps and scale_steps in calculate_quantization_params (#4039).
• Improvements:
  • (OpenVINO) Added transpose_a attribute support for Scale Estimation algorithm in ONNX and OpenVINO backends (#3839).
  • (PyTorch) Removed no-op operations (to, dropout, etc.) from the inference graph in PTQ and Weight Compression algorithms (#4057).
  • (PyTorch) Improved load_from_config reliability by saving module path in serialized config (#4050) (#4054).
• Tutorials:
  • Post-Training Optimization of Qwen3-VL Embedding and Reranker Models
  • Post-Training Optimization of MiniCPM-V 4.6 Model
  • Post-Training Optimization of MinerU 2.5 Model
  • Post-Training Optimization of GLM-OCR Model
  • Post-Training Optimization of FLUX.2 Klein Model
  • Post-Training Optimization of MedASR Model
  • Post-Training Optimization of ERNIE-Image-Turbo Model
  • Post-Training Optimization of VoxCPM2 Model
  • Post-Training Optimization of HY-MT1.5 Models
  • Post-Training Optimization of YOLO26 Models
• Deprecations/Removals:
  • (PyTorch) Removed example_input parameter from load_from_config, which has been unused since 3.0.0 (#4051).
  • (PyTorch) Removed OVQuantizer from NNCF; use it from the executorch package instead (#3842).
  • Removed deprecated CompressWeightsMode.INT8 mode. Use CompressWeightsMode.INT8_SYM or CompressWeightsMode.INT8_ASYM instead (#4008).
  • Removed unused ProgressBar class (#4044).
• Requirements:
  • Updated onnx to 1.21.0 (#4006).
  • Updated torchao to 0.17.0 (#4013).
  • Updated onnxscript to 0.6.2 (#4015).

• General:
  • Migrated NNCFGraph from nx.DiGraph to nx.MultiDiGraph to support models with parallel/multi-edges, enabling correct quantization of models with complex graph structures such as YOLO26 and models like a = conv(x); return a * a (#3843).
• Features:
  • (OpenVINO) Added NVFP4 (f4e2m1) compression data type in Weight Compression. NVFP4 uses a constant group size of 16 with scales compressed to f8e4m3 using a second-degree scale (#3967).
  • (OpenVINO) Added backup_mode parameter for FP compression formats (MXFP4, MXFP8, FP4, FP8), allowing first/last layers to be compressed with a backup FP format instead of INT8 (#3886).
  • (OpenVINO) RoPe ignored pattern is updated to handle operations without a preceding transpose like in the Phi-3.5-MoE-instruct model (#3989).
  • (PyTorch) Added TopKMetatype support for TorchFX backend, enabling correct graph building for models with TopK operations such as YOLO26 (#3944).
  • (PyTorch) Migrated to use torchao instead of deprecated torch.ao (#3854).
• Fixes:
  • (OpenVINO) Fixed scale estimation for Adaptive Codebook weight compression (#3888).
  • (ONNX) Fixed nncf.errors.ValidationError: There is no tensor with the name error (#3988).
• Improvements:
  • (PyTorch) Added lazy import for nncf.torch module to reduce startup import time (#3862).
• Tutorials:
  • Post-Training Optimization of Gemma 4 Model
  • Post-Training Optimization of Code-specialized LLMs
  • Post-Training Optimization of Vision-Language Models (VLMs)
  • Post-Training Optimization of MiniCPM-o 4.5 Multimodal Model
  • Post-Training Optimization of PaddleOCR-VL/PaddleOCR-VL-1.5 Models
  • Post-Training Optimization of RAG pipeline
• Requirements:
  • Updated to openvino 2026.1.0 (#4005).
  • Updated to torch 2.10.0 (#3852).
  • Updated to onnx 1.20.1 (#3966).
  • Updated to onnxruntime 1.24.3 (#3977).
  • Moved pandas to optional dependency (#3970).
  • Removed unused pillow dependency (#3929).

Post-training Quantization:

• Breaking changes:
  • Renamed nncf.CompressWeightsMode.CB4_F8E4M3 mode option to nncf.CompressWeightsMode.CB4.
• General:
  • Added nncf.prune API function, which provides a unified interface for pruning algorithms. Currently available for PyTorch backend and supports Magnitude Pruning.
    More details about the new API can be found in the documentation.
  • Added nncf.build_graph API function for building NNCFGraph from a model. This API can be used to inspect and define the ignored scope.
  • Added documentation about using nncf.IgnoredScope.
  • Reworked HWConfig, now using Python-style definition of hardware configuration instead of JSON files.
• Features:
  • Added support for models containing MatMul operations with transposed activation inputs in data-free Weight Compression and data-aware AWQ algorithms.
  • (OpenVINO) Introduced new experimental compression data type ADAPTIVE_CODEBOOK. This compression type calculates a unique codebook for each MatMul or block of identical MatMuls (for example, all down_proj could have the same codebook). This approach reduces quality degradation in the case of per-channel weight compression. See example.
  • (TorchFX) Preview support for the new compress_pt2e API has been introduced, enabling quantization of torch.fx.GraphModule models with the OpenVINOQuantizer. Users now can quantize their models in ExecuTorch for the OpenVINO backend via the nncf compress_pt2e employing Scale Estimation and AWQ.
  • (PyTorch) Added support for linear functions for the Fast Bias Correction algorithm to improve the accuracy of such models after the quantization.
  • (OpenVINO) Added activation profiler tool to collect and visualize tensor statistics.
• Fixes:
  • (ONNX) Fixed compress_quantize_weights_transformation() method by removing names of deleted initializers from graph inputs.
  • (ONNX) Fixed incorrect insertion of MatMulNBits nodes.
• Improvements:
  • Added support for the compression of 3D weights in AWQ, Scale Estimation, and GPTQ algorithms. Models with MoE (Mixture of Experts), such as GPT-OSS-20B and Qwen3-30B-A3B, can be compressed with data-aware methods now.
• Tutorials:
  • Post-Training Quantization of YOLO26 OpenVINO Model
  • Post-Training Optimization of Wan2.2 Model
  • Post-Training Optimization of DeepSeek-OCR Model
  • Post-Training Optimization of Z-Image-Turbo Model
  • Post-Training Optimization of Qwen-Image Model
  • Post-Training Optimization of Qwen3-TTS Model
  • Post-Training Optimization of Qwen3-ASR Model
  • Post-Training Optimization of Fun-ASR-Nano Model
  • Post-Training Optimization of Fun-CosyVoice 3.0 Model

Deprecations/Removals:

• (TensorFlow) Removed support for TensorFlow backend.
• (PyTorch) Removed legacy create_compressed_model API for PyTorch backend, which was previously marked as deprecated.
• (PyTorch) Removed legacy algorithms for PyTorch that were based on using NNCFNetwork: NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization, and Movement Sparsity.

Requirements:

• Dropped jsonschema, natsort, and pymoo from dependencies as they are no longer required.
• Updated numpy to >=1.24.0, <2.5.0.

Acknowledgements

Thanks for contributions from the OpenVINO developer community:
@avolkov-intel @Shehrozkashif @ruro @mostafafaheem

Post-training Quantization:

• Breaking changes:
  • (OpenVINO) nncf.CompressWeightsMode.E2M1 mode option is renamed to nncf.CompressWeightsMode.MXFP4.
• Features:
  • The Histogram Aggregator was introduced: it records the running histogram of tensor values and computes the quantization range to minimize the L2 norm of histogram bin quantization error. The Histogram Aggregator improved accuracy metrics for a number of PTQ classification models. It can be activated using RangeEstimatorParametersSet.HISTOGRAM through the AdvancedQuantizationParameters in nncf.quantize().
  • (OpenVINO) Introduced several new compression modes in nncf.CompressWeightsMode: MXFP8, FP8, and FP4. These can be used as the mode option in nncf.compress_weights() to apply the corresponding MXFP8, FP8, or FP4 precisions (experimental).
  • Now weight compression bitwidth distribution table also displays group size value for each of the compression data type.
  • (ONNX) Support for the SmoothQuant algorithm has been added to the ONNX backend for INT8 quantization.
  • (ONNX) A new transformation has been added to optimize models by folding QuantizeLinear nodes with constant inputs into precomputed, quantized initializers. This behavior is controlled by the COMPRESS_WEIGHTS backend parameter in nncf.quantize(), which is now enabled (True) by default.
  • (ONNX) Support has been added for applying the Fast Bias/Bias Correction algorithm to MatMul + Add subgraphs where one of the inputs to the Add operation is a constant. Previously, these cases were skipped because the MatMul operation was not recognized as having a bias, preventing the algorithm from being applied.
• Fixes:
  • Added an ignored pattern for position embedding layer in Segment Anything model.
  • (ONNX) Fixed incorrect input handling for the MatMulNBits operation that previously caused graph breaks.
  • (ONNX) Resolved an issue with INT4 weight compression in the Gemm operation when transB=1.
  • Fixed a typo in the _get_smooth_quant_param_grid() method reported in #3613.
• Improvements:
  • Maximum memory consumption during statistic collection has been reduced by releasing model output memory before the next statistic collection inference call.
  • Reduced peak memory footprint for Bias Correction algorithm.
  • (OpenVINO) Reduced time (by up to 3x) and memory (by up to 1.5x) it takes to compress models to MXFP4 data type.
• Tutorials:
  • Deployment of a Post-Training Optimized Qwen3 Agent
  • Post-Training Optimization of ACE Step Model
  • Post-Training Optimization of Qwen3-VL Model
• Other:
  • Refined the handling of layers that don't have channel size divisible by group size during weight compression. Now the default behavior in such case is that an error will be raised and in the error message users are suggested to provide a different group size value or use GroupSizeFallbackMode.ADJUST to automatically adjust group size for problematic layers.

Compression-aware training:

• Improvements:
  • Optimized nncf.strip() for StripFormat.IN_PLACE and example_input is no longer required.

Deprecations/Removals:

• (TensorFlow) The TensorFlow backend is now deprecated and will be removed in future releases. It is recommended to use PyTorch analogous models for training-aware optimization methods and OpenVINO IR, PyTorch, and ONNX models for post-training optimization methods from NNCF.
• The following experimental NNCF methods are deprecated and will be removed in future releases: NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization, Movement Sparsity.

Requirements:

• Updated PyTorch (2.9.0) and Torchvision (0.24.0) versions.
• Dropped support for Python 3.9.

Post-training Quantization:

• Features:
  • (OpenVINO) Introduced new compression data types CB4_F8E4M3 and CODEBOOK. CB4_F8E4M3 is a fixed codebook with 16 fp8 values based on NF4 data type values. CODEBOOK is an arbitrary user-selectable codebook that can be used to experiment with different data types. Both data types are used for weight compression. The AWQ and scale estimation algorithms are supported for these data types.
  • (OpenVINO) Added support for compressing FP8 (f8e4m3 and f8e5m2) weights to 4-bit data types, which is particularly beneficial for models like DeepSeek-R1.
  • Added group_size_fallback_mode parameter for advanced weight compression. It controls how nodes that do not support the default group size are handled. By default (IGNORE), such nodes are skipped. With ERROR, an exception is raised if the channel size is not divisible by the group size, while ADJUST attempts to modify the group size so it becomes valid.
  • (TorchFX) Added support for external quantizers in the quantize_pt2e API, including XNNPACKQuantizer and CoreMLQuantizer. Users now can quantize their models in ExecuTorch for the XNNPACK and CoreML backends via the nncf quantize_pt2e employing smooth quant, bias correction algorithms and a wide range of statistic collectors.
  • (ONNX) Added support for data-aware weight compression in the ONNX backend, including the AWQ and Scale Estimation algorithms. Provided an example demonstrating the data-aware weight compression pipeline using the TinyLlama/TinyLlama-1.1B-Chat-v1.0 model in ONNX format.
• Improvements:
  • Support of weight compression for models with the Rotary Positional Embedding block.
  • Support of weight compression for models with stateful self-attention blocks.
• Tutorials:
  • Post-Training Optimization of Qwen-Agent
  • Post-Training Optimization of FLUX.1 Kontext Model
  • Post-Training Optimization of Qwen3 Embedding Model
  • Post-Training Optimization of GLM-4.1V-9B-Thinking Model

Compression-aware training:

• Features:
  • (PyTorch) Enhanced initialization for "QAT with absorbable LoRA" using advanced compression methods (AWQ + Scale Estimation). This improvement replaces the previous basic data-free compression approach, enabling QAT to start with a more accurate model baseline and achieve superior final accuracy.
• Improvements:
  • (PyTorch) Streamlined "QAT with absorbable LoRA" by removing checkpoint selection based on validation set. This change significantly reduces overall tuning time and maximum allocated memory. While the results on Wikitext are slightly worse, it provides a more efficient and faster tuning pipeline (e.g. reduced from 32 minutes to 25 minutes for SmoLM-1.7B).
• Tutorials:
  • (TorchFX) Added example for compression of TinnyLama-1.1B.
  • Updated example to meet NPU implementation.
  • Implemented fast evaluation and improved output in example.

Deprecations/Removals:

• Removed examples that used create_compressed_model API.

Requirements:

• Updated PyTorch (2.8.0) and Torchvision (0.23.0) versions.
• Set require setuptools>=77 to build package.

Acknowledgements

Thanks for contributions from the OpenVINO developer community:
@bopeng1234 @jpablomch

Post-training Quantization:

• General:
  • (PyTorch) The function_hook module is now the default mechanism for model tracing. It has moved out from experimental status and has been moved to the core nncf.torch namespace.
• Features:
  • (OpenVINO, PyTorch, TorchFX) Added 4-bit data-free AWQ (Activation-aware Weight Quantization) based on the per-column magnitudes of the weights making it possible to apply AWQ without a dataset for more accurate compression.
  • (OpenVINO) Added support for quantizing of the value input for ScaledDotProductAttention for FP8.
  • (ONNX) Added support for data-free weight compression using INT4 (INT8) in the ONNX backend. Added an example for LLM weight compression in the ONNX backend. This example showcases the optimization of the TinyLlama-1.1B-Chat-v0.3 model in ONNX format using the NNCF weight compression API.
  • (ONNX) Added the BackendParameters.EXTERNAL_DATA_DIR parameter for the ONNX backend. This parameter specifies the absolute path to the directory where the model's external data files are stored. All external data files must be located in the same directory. It should be used when the model is loaded without external data using onnx.load("model.onnx", load_external_data=False), and the external data files are not in the current working directory of the process. This parameter can be omitted if the external data files are located in the current working directory of the process.
  • (TorchFX, Experimental) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce accuracy loss.
• Fixes:
  • (TorchFX, Experimental) To simplify usage, the nncf.torch.disable_patching() context manager has been made redundant and is no longer required (example).
  • Fixed BiasCorrection failures with models without a batch dimension.
  • Aligned quantile centers for NF4 with OpenVINO implementation.
  • Weights compression statistics collection have been fixed to show the data types of ignored weights.
• Improvements:
  • (OpenVINO) Added the version of NNCF to rt_info.
  • Optimized weight compression for NF4 (up to 10x speed up).
  • Support for transformer>4.52 by nncf.data.generate_text_data.
• Tutorials:
  • Post-Training Optimization of MiniCPM-o 2.6 Model
  • Post-Training Optimization of Qwen2.5-Omni Model
  • Post-Training Optimization of InternVideo2 Model
  • Post-Training Optimization of OpenVoice2 and MeloTTS Models
  • Post-Training Optimization of Flex.2 Model
  • Post-Training Optimization of Wan2.1 Model
  • Post-Training Optimization of Phi-4-mini Model
  • Post-Training Optimization of Torch.FX Stable Diffusion v3 Model

Compression-aware training:

• Features:
  • (PyTorch) For downstream tasks, we introduce Quantization-Aware Training (QAT) with absorbable elastic LoRA adapters and neural low-rank search (NLS). This novel weight compression method enhances the accuracy of Large Language Models (LLMs) with int4 weights on downstream tasks, achieving a reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The nncf.compress_weights API now includes a new compression_format option, nncf.CompressionFormat.FQ_LORA_NLS. A sample QAT compression pipeline with preview support is available here. Building on our previous work with absorbable LoRA adapters, this new pipeline is specifically designed for downstream tasks. In contrast, the pipeline from the previous release was tailored to enhance general accuracy through knowledge distillation using static rank settings. For a more comprehensive understanding of both approaches, please refer to "Weight-Only Quantization Aware Training with LoRA and NLS" in the "Training-Time Compression Algorithms" section of the main README in the repository.
• Fixes:
  • (PyTorch) Minimized the disparity in accuracy between the Torch model and its exported OpenVINO equivalent for "Weight-Only Quantization Aware Training with LoRA and NLS".
• Improvements:
  • (Pytorch) The evaluation and selection process for the best checkpoint in "QAT + absorbable LoRA" with knowledge distillation has been revised. The tuned Torch model is now evaluated using the validation split of Wikitext, while the final results are measured on the test split with the OpenVINO model. The results table for Wikitext has been updated accordingly and now includes three additional models.

Requirements:

• Updated ONNX Runtime (1.21.1).
• Updated PyTorch (2.7.1) and Torchvision (0.22.1) versions.
• Removed jstyleson from requirements.

Post-training Quantization:

Features:

• (PyTorch) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce quality loss.
• (PyTorch, Experimental) Introduced TorchFunctionMode support for MinMax, FastBiasCorrection, SmoothQuant, WeightCompression algorithms.

Fixes:

• Fixed occasional failures of the weights compression algorithm on ARM CPUs.
• Fixed GPTQ fails with per-channel int4 weights compression.
• Fixed weight compression fails for models with fp8 weights.
• (PyTorch, Experimental) Fixed weights compression for float16/bfloat16 models.
• (PyTorch, Experimental) Fixed several memory leak issues: non-detached tensors, extracted modules & graph building with gradients.

Improvements:

• Reduced the run time and peak memory of the mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in the mixed precision case is about 20-40%; peak memory reduction is about 20%.
• The NNCF hardware config has been extended with the narrow_range parameter, enabling more combinations of quantization configurations in the MinMax quantization algorithm.
• (TorchFX, Experimental) Added quantization support for TorchFX models exported with dynamic shapes.
• (TorchFX, Experimental) The constant folding step is removed from the quantize_pt2e function and the transform_for_annotation method of the OpenVINOQuantizer to align with the torch.ao quantization implementation.
• Optimized GPTQ algorithm behavior to decrease memory & time consumption by 2.71x and 1.16x, respectively.
• Added general support for optimization of models with FP8 and NF4 weights.
• Disable applying overflow fix for non 8-bit quantization.

Tutorials:

• Post-Training Optimization of Gemma3 Model
• Post-Training Optimization of GLM4-V Model
• Post-Training Optimization of Llasa Model
• Post-Training Optimization of YOLOv12 Model
• Post-Training Optimization of Phi-4-multimodal Model
• Post-Training Optimization of Qwen2.5VL Model
• Post-Training Optimization of DeepSeek-VL2 Model
• Post-Training Optimization of FLUX.1 Fill Model
• Post-Training Optimization of olmOCR Model
• Post-Training Optimization of SmolDocling Model
• Post-Training Optimization of SmolVLM2 Model
• Post-Training Optimization of GOT-OCR 2.0 Model
• Post-Training Optimization of LTX-Video Model
• Post-Training Optimization of OuteTTS Model
• Post-Training Optimization of SigLIP2 Model
• Post-Training Optimization of OpenCLIP Model

Compression-aware training:

Features:

• (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The nncf.compress_weights API now includes a new compression_format option, nncf.CompressionFormat.FQ_LORA, for this QAT method, a sample compression pipeline with preview support is available here.
• (PyTorch) Changed compression modules serialization API: compressed_model.nncf.get_config was changed to nncf.torch.get_config. The documentation was updated to use the new API.

Requirements:

• Updated PyTorch (2.6.0) and Torchvision (0.21.0) versions.
• Updated Transformers (>=4.48.0) version.
• Updated NumPy (<2.3.0) version support.
• Updated NetworkX (<3.5.0) version support.

Acknowledgements

Thanks for contributions from the OpenVINO developer community:
@shumaari

Post-training Quantization:

Features:

• (TensorFlow) The nncf.quantize() method is now the recommended API for Quantization-Aware Training. Please refer to an example for more details about how to use a new approach.
• (TensorFlow) Compression layers placement in the model now can be serialized and restored with new API functions: nncf.tensorflow.get_config() and nncf.tensorflow.load_from_config(). Please see the documentation for the saving/loading of a quantized model for more details.
• (OpenVINO) Added example with LLM quantization to FP8 precision.
• (TorchFX, Experimental) Preview support for the new quantize_pt2e API has been introduced, enabling quantization of torch.fx.GraphModule models with the OpenVINOQuantizer and the X86InductorQuantizer quantizers. quantize_pt2e API utilizes MinMax algorithm statistic collectors, as well as SmoothQuant, BiasCorrection and FastBiasCorrection Post-Training Quantization algorithms.
• Added unification of scales for ScaledDotProductAttention operation.

Fixes:

• (ONNX) Fixed sporadic accuracy issues with the BiasCorrection algorithm.
• (ONNX) Fixed GroupConvolution operation weight quantization, which also improves performance for a number of models.
• Fixed AccuracyAwareQuantization algorithm to solve #3118 issue.
• Fixed issue with NNCF usage with potentially corrupted backend frameworks.

Improvements:

• (TorchFX, Experimental) Added YoloV11 support.
• (OpenvINO) The performance of the FastBiasCorrection algorithm was improved.
• Significantly faster data-free weight compression for OpenVINO models: INT4 compression is now up to 10x faster, while INT8 compression is up to 3x faster. The larger the model the higher the time reduction.
• AWQ weight compression is now up to 2x faster, improving overall runtime efficiency.
• Peak memory usage during INT4 data-free weight compression in the OpenVINO backend is reduced by up to 50% for certain models.

Tutorials:

• Post-Training Optimization of GLM-Edge-V Model
• Post-Training Optimization of OmniGen Model
• Post-Training Optimization of Sana Models
• Post-Training Optimization of BGE Models
• Post-Training Optimization of Stable Diffusion Inpainting Model
• Post-Training Optimization of LTX Video Model
• Post-Training Optimization of DeepSeek-R1-Distill Model
• Post-Training Optimization of Janus DeepSeek-LLM-1.3b Model

Deprecations/Removals:

• (TensorFlow) The nncf.tensorflow.create_compressed_model() method is now marked as deprecated. Please use the nncf.quantize() method for the quantization initialization.

Requirements:

• Updated the minimal version for numpy (>=1.24.0).
• Removed tqdm dependency.

Acknowledgements

Thanks for contributions from the OpenVINO developer community:
@rk119
@devesh-2002

Post-training Quantization:

Bugfixes:

• (PyTorch) Fixed the get_torch_compile_wrapper function to match with the torch.compile.
• (OpenVINO) Updated cache statistics functionality to utilize the safetensors approach.

Post-training Quantization:

Features:

• Introduced backup_mode optional parameter in nncf.compress_weights() to specify the data type for embeddings, convolutions and last linear layers during 4-bit weights compression. Available options are INT8_ASYM by default, INT8_SYM, and NONE which retains the original floating-point precision of the model weights.
• Added the quantizer_propagation_rule parameter, providing fine-grained control over quantizer propagation. This advanced option is designed to improve accuracy for models where quantizers with different granularity could be merged to per-tensor, potentially affecting model accuracy.
• Introduced nncf.data.generate_text_data API method that utilizes LLM to generate data for further data-aware optimization. See the example for details.
• (OpenVINO) Extended support of data-free and data-aware weight compression methods for nncf.compress_weights() with NF4 per-channel quantization, which makes compressed LLMs more accurate and faster on NPU.
• (OpenVINO) Introduced a new option statistics_path to cache and reuse statistics for nncf.compress_weights(), reducing the time required to find optimal compression configurations. See the TinyLlama example for details.
• (TorchFX, Experimental) Added support for quantization and weight compression of Torch FX models. The compressed models can be directly executed via torch.compile(compressed_model, backend="openvino") (see details here). Added INT8 quantization example. The list of supported features:
  • INT8 quantization with SmoothQuant, MinMax, FastBiasCorrection, and BiasCorrection algorithms via nncf.quantize().
  • Data-free INT8, INT4, and mixed-precision weights compression with nncf.compress_weights().
• (PyTorch, Experimental) Added model tracing and execution pre-post hooks based on TorchFunctionMode.

Fixes:

• Resolved an issue with redundant quantizer insertion before elementwise operations, reducing noise introduced by quantization.
• Fixed type mismatch issue for nncf.quantize_with_accuracy_control().
• Fixed BiasCorrection algorithm for specific branching cases.
• (OpenVINO) Fixed GPTQ weight compression method for Stable Diffusion models.
• (OpenVINO) Fixed issue with the variational statistics processing for nncf.compress_weights().
• (PyTorch, ONNX) Scaled dot product attention pattern quantization setup is aligned with OpenVINO.

Improvements:

• Reduction in peak memory by 30-50% for data-aware nncf.compress_weights() with AWQ, Scale Estimation, LoRA and mixed-precision algorithms.
• Reduction in compression time by 10-20% for nncf.compress_weights() with AWQ algorithm.
• Aligned behavior for ignored subgraph between different networkx versions.
• Extended ignored patterns with RoPE block for nncf.ModelType.TRANSFORMER scheme.
• (OpenVINO) Extended to the ignored scope for nncf.ModelType.TRANSFORMER scheme with GroupNorm metatype.
• (ONNX) SE-block ignored pattern variant for torchvision mobilenet_v3 has been extended.

Tutorials:

• Post-Training Optimization of Llama-3.2-11B-Vision Model
• Post-Training Optimization of YOLOv11 Model
• Post-Training Optimization of Whisper in Automatic speech recognition with OpenVINO Generate API
• Post-Training Optimization of Pixtral Model
• Post-Training Optimization of LLM ReAct Agent Model
• Post-Training Optimization of CatVTON Model
• Post-Training Optimization of Stable Diffusion v3 Model in Torch FX Representation

Known issues:

• (ONNX) nncf.quantize() method can generate inaccurate INT8 results for MobileNet models with the BiasCorrection algorithm.

Deprecations/Removals:

• Migrated from using setup.py to pyproject.toml for the build and package configuration. It is aligned with Python packaging standards as outlined in PEP 517 and PEP 518. The installation through setup.py does not work anymore. No impact on the installation from PyPI and Conda.
• Removed support for Python 3.8.
• (PyTorch) nncf.torch.create_compressed_model() function has been deprecated.

Requirements:

• Updated ONNX (1.17.0) and ONNXRuntime (1.19.2) versions.
• Updated PyTorch (2.5.1) and Torchvision (0.20.1) versions.
• Updated NumPy (<2.2.0) version support.
• Updated Ultralytics (8.3.22) version.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @rk119 @zina-cs

Post-training Quantization:

Features:

• (OpenVINO) Added support for combining GPTQ with AWQ and Scale Estimation (SE) algorithms in nncf.compress_weights() for more accurate weight compression of LLMs. Thus, the following combinations with GPTQ are now supported: AWQ+GPTQ+SE, AWQ+GPTQ, GPTQ+SE, GPTQ.
• (OpenVINO) Added LoRA Correction Algorithm to further improve the accuracy of int4 compressed models on top of other algorithms - AWQ and Scale Estimation. It can be enabled via the optional lora_correction parameter of the nncf.compress_weights() API. The algorithm increases compression time and incurs a negligible model size overhead. Refer to accuracy/footprint trade-off for different int4 compression methods.
• (PyTorch) Added implementation of the experimental Post-training Activation Pruning algorithm. Refer to Activation Sparsity for details.
• Added a memory monitoring tool for logging the memory a piece of python code or a script allocates. Refer to NNCF tools for details.

Fixes:

• (OpenVINO) Fixed the quantization of Convolution and LSTMSequence operations in cases where some inputs are part of a ShapeOF subgraph.
• (OpenVINO) Fixed issue with the FakeConvert duplication for FP8.
• Fixed Smooth Quant algorithm issue in case of the incorrect shapes.
• Fixed non-deterministic layer-wise scheduling.

Improvements:

• (OpenVINO) Increased hardware-fused pattern coverage.
• Improved progress bar logic during weights compression for more accurate remaining time estimation.
• Extended Scale estimation bitness range support for the nncf.compress_weights().
• Removed extra logging for the algorithm-generated ignored scope.

Tutorials:

• Post-Training Optimization of Flux.1 Model
• Post-Training Optimization of PixArt-α Model
• Post-Training Optimization of InternVL2 Model
• Post-Training Optimization of Qwen2Audio Model
• Post-Training Optimization of NuExtract Model
• Post-Training Optimization of MiniCPM-V2 Model

Compression-aware training:

Fixes:

• (PyTorch) Fixed some scenarios of NNCF patching interfering with torch.compile.

Requirements:

• Updated PyTorch (2.4.0) and Torchvision (0.19.0) versions.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @rk119

Post-training Quantization:

Features:

• (OpenVINO, PyTorch, ONNX) Excluded comparison operators from the quantization scope for nncf.ModelType.TRANSFORMER.
• (OpenVINO, PyTorch) Changed the representation of symmetrically quantized weights from an unsigned integer with a fixed zero-point to a signed data type without a zero-point in the nncf.compress_weights() method.
• (OpenVINO) Extended patterns support of the AWQ algorithm as part of nncf.compress_weights(). This allows apply AWQ for the wider scope of the models.
• (OpenVINO) Introduced nncf.CompressWeightsMode.E2M1 mode option of nncf.compress_weights() as the new MXFP4 precision (Experimental).
• (OpenVINO) Added support for models with BF16 precision in the nncf.quantize() method.
• (PyTorch) Added quantization support for the torch.addmm.
• (PyTorch) Added quantization support for the torch.nn.functional.scaled_dot_product_attention.

Fixes:

• (OpenVINO, PyTorch, ONNX) Fixed Fast-/BiasCorrection algorithms with correct support of transposed MatMul layers.
• (OpenVINO) Fixed nncf.IgnoredScope() functionality for models with If operation.
• (OpenVINO) Fixed patterns with PReLU operations.
• Fixed runtime error while importing NNCF without Matplotlib package.

Improvements:

• Reduced the amount of memory required for applying nncf.compress_weights() to OpenVINO models.
• Improved logging in case of the not empty nncf.IgnoredScope().

Tutorials:

• Post-Training Optimization of Stable Audio Open Model
• Post-Training Optimization of Phi3-Vision Model
• Post-Training Optimization of MiniCPM-V2 Model
• Post-Training Optimization of Jina CLIP Model
• Post-Training Optimization of Stable Diffusion v3 Model
• Post-Training Optimization of HunyuanDIT Model
• Post-Training Optimization of DDColor Model
• Post-Training Optimization of DynamiCrafter Model
• Post-Training Optimization of DepthAnythingV2 Model
• Post-Training Optimization of Kosmos-2 Model

Compression-aware training:

Fixes:

• (PyTorch) Fixed issue with wrapping for operator without patched state.

Requirements:

• Updated Tensorflow (2.15) version. This version requires Python 3.9-3.11.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @Lars-Codes

Post-training Quantization:

Features:

• (OpenVINO) Added Scale Estimation algorithm for 4-bit data-aware weights compression. The optional scale_estimation parameter was introduced to nncf.compress_weights() and can be used to minimize accuracy degradation of compressed models (note that this algorithm increases the compression time).
• (OpenVINO) Added GPTQ algorithm for 8/4-bit data-aware weights compression, supporting INT8, INT4, and NF4 data types. The optional gptq parameter was introduced to nncf.compress_weights() to enable the GPTQ algorithm.
• (OpenVINO) Added support for models with BF16 weights in the weights compression method, nncf.compress_weights().
• (PyTorch) Added support for quantization and weight compression of the custom modules.

Fixes:

• (OpenVINO) Fixed incorrect node with bias determination in Fast-/BiasCorrection and ChannelAlighnment algorithms.
• (OpenVINO, PyTorch) Fixed incorrect behaviour of nncf.compress_weights() in case of compressed model as input.
• (OpenVINO, PyTorch) Fixed SmoothQuant algorithm to work with Split ports correctly.

Improvements:

• (OpenVINO) Aligned resulting compression subgraphs for the nncf.compress_weights() in different FP precisions.
• Aligned 8-bit scheme for NPU target device with the CPU.

Examples:

• (OpenVINO, ONNX) Updated ignored scope for YOLOv8 examples utilizing a subgraphs approach.

Tutorials:

• Post-Training Optimization of Stable Video Diffusion Model
• Post-Training Optimization of YOLOv10 Model
• Post-Training Optimization of LLaVA Next Model
• Post-Training Optimization of S3D MIL-NCE Model
• Post-Training Optimization of Stable Cascade Model

Compression-aware training:

Features:

• (PyTorch) nncf.quantize method is now the recommended path for the quantization initialization for Quantization-Aware Training.
• (PyTorch) Compression modules placement in the model now can be serialized and restored with new API functions: compressed_model.nncf.get_config() and nncf.torch.load_from_config. The documentation for the saving/loading of a quantized model is available, and Resnet18 example was updated to use the new API.

Fixes:

• (PyTorch) Fixed compatibility with torch.compile.

Improvements:

• (PyTorch) Base parameters were extended for the EvolutionOptimizer (LeGR algorithm part).
• (PyTorch) Improved wrapping for parameters which are not tensors.

Examples:

• (PyTorch) Added an example for STFPM model from Anomalib.

Tutorials:

• Quantization-Sparsity Aware Training of PyTorch ResNet-50 Model

Deprecations/Removals:

• Removed extra dependencies to install backends from setup.py (like [torch] are [tf], [onnx] and [openvino]).
• Removed openvino-dev dependency.

Requirements:

• Updated PyTorch (2.3.0) and Torchvision (0.18.0) versions.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @DaniAffCH @UsingtcNower @anzr299 @AdiKsOnDev @Viditagarwal7479 @truhinnm

Post-training Quantization:

Features:

• Introduced the subgraph defining functionality for the nncf.IgnoredScope() option.
• Introduced limited support for the batch size of more than 1. MobilenetV2 PyTorch example was updated with batch support.

Fixes:

• Fixed issue with the nncf.OverflowFix parameter absence in some scenarios.
• Aligned the list of correctable layers for the FastBiasCorrection algorithm between PyTorch, OpenVINO and ONNX backends.
• Fixed issue with the nncf.QuantizationMode parameters combination.
• Fixed MobilenetV2 (PyTorch, ONNX, OpenVINO) examples for the Windows platform.
• (OpenVINO) Fixed Anomaly Classification example for the Windows platform.
• (PyTorch) Fixed bias shift magnitude calculation for fused layers.
• (OpenVINO) Fixed removing the ShapeOf graph which led to an error in the nncf.quantize_with_accuracy_control() method.
• Improvements:
• OverflowFix, AdvancedSmoothQuantParameters and AdvancedBiasCorrectionParameters were exposed into the nncf.* namespace.
• (OpenVINO, PyTorch) Introduced scale compression to FP16 for weights in nncf.compress_weights() method, regardless of model weights precision.
• (PyTorch) Modules that NNCF inserted were excluded from parameter tracing.
• (OpenVINO) Extended the list of correctable layers for the BiasCorrection algorithm.
• (ONNX) Aligned BiasCorrection algorithm behaviour with OpenVINO in specific cases.

Tutorials:

• Post-Training Optimization of PhotoMaker Model
• Post-Training Optimization of Stable Diffusion XL Model
• Post-Training Optimization of KerasCV Stable Diffusion Model
• Post-Training Optimization of Paint By Example Model
• Post-Training Optimization of aMUSEd Model
• Post-Training Optimization of InstantID Model
• Post-Training Optimization of LLaVA Next Model
• Post-Training Optimization of AnimateAnyone Model
• Post-Training Optimization of YOLOv8-OBB Model
• Post-Training Optimization of LLM Agent

Compression-aware training:

Features:

• (PyTorch) nncf.quantize method now may be used as quantization initialization for Quantization-Aware Training. Added a Resnet18-based example with the transition from the Post-Training Quantization to a Quantization-Aware Training algorithm.
• (PyTorch) Introduced extractors for the fused Convolution, Batch-/GroupNorm, and Linear functions.

Fixes:

• (PyTorch) Fixed apply_args_defaults function issue.
• (PyTorch) Fixed dtype handling for the compressed torch.nn.Parameter.
• (PyTorch) Fixed is_shared parameter propagation.

Improvements:

• (PyTorch) Updated command creation behaviour to reduce the number of adapters.
• (PyTorch) Added option to insert point for models that wrapped with replace_modules=False.

Deprecations/Removals:

• (PyTorch) Removed the binarization algorithm.
• NNCF installation via pip install nncf[<framework>] option is now deprecated.

Requirements:

• Updated PyTorch (2.2.1) and CUDA (12.1) versions.
• Updated ONNX (1.16.0) and ONNXRuntime (1.17.1) versions.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @Candyzorua @clinty @UsingtcNower @DaniAffCH

Post-training Quantization:

Features:

• (OpenVINO) Added modified AWQ algorithm for 4-bit data-aware weights compression. This algorithm applied only for patterns MatMul->Multiply->Matmul. For that awq optional parameter has been added to nncf.compress_weights() and can be used to minimize accuracy degradation of compressed models (note that this option increases the compression time).
• (ONNX) Introduced support for the ONNX backend in the nncf.quantize_with_accuracy_control() method. Users can now perform quantization with accuracy control for onnx.ModelProto. By leveraging this feature, users can enhance the accuracy of quantized models while minimizing performance impact.
• (ONNX) Added an example based on the YOLOv8n-seg model for demonstrating the usage of quantization with accuracy control for the ONNX backend.
• (PT) Added SmoothQuant algorithm for PyTorch backend in nncf.quantize().
• (OpenVINO) Added an example with the hyperparameters tuning for the TinyLLama model.
• Introduced the nncf.AdvancedAccuracyRestorerParameters.
• Introduced the subset_size option for the nncf.compress_weights().
• Introduced TargetDevice.NPU as the replacement for TargetDevice.VPU.

Fixes:

• Fixed API Enums serialization/deserialization issue.
• Fixed issue with required arguments for revert_operations_to_floating_point_precision method.

Improvements:

• (ONNX) Aligned statistics collection with OpenVINO and PyTorch backends.
• Extended nncf.compress_weights() with Convolution & Embeddings compression in order to reduce memory footprint.

Deprecations/Removals:

• (OpenVINO) Removed outdated examples with nncf.quantize() for BERT and YOLOv5 models.
• (OpenVINO) Removed outdated example with nncf.quantize_with_accuracy_control() for SSD MobileNetV1 FPN model.
• (PyTorch) Deprecated the binarization algorithm.
• Removed Post-training Optimization Tool as OpenVINO backend.
• Removed Dockerfiles.
• TargetDevice.VPU was replaced by TargetDevice.NPU.

Tutorials:

• Post-Training Optimization of Stable Diffusion v2 Model
• Post-Training Optimization of DeciDiffusion Model
• Post-Training Optimization of DepthAnything Model
• Post-Training Optimization of Stable Diffusion ControlNet Model

Compression-aware training:

Fixes

• (PyTorch) Fixed issue with NNCFNetworkInterface.get_clean_shallow_copy missed arguments.

Acknowledgements

Thanks for contributions from the OpenVINO developer community: @AishwaryaDekhane @UsingtcNower @Om-Doiphode

Post-training Quantization:

Bugfixes:

• (Common) Fixed issue with nncf.compress_weights() to avoid overflows on 32-bit Windows systems.
• (Common) Fixed performance issue with nncf.compress_weights() on LLama models.
• (Common) Fixed nncf.quantize_with_accuracy_control pipeline with tune_hyperparams=True enabled option.
• (OpenVINO) Fixed issue for stateful LLM models and added state restoring after the inference for it.
• (PyTorch) Fixed issue with nncf.compress_weights() for LLM models with the executing is_floating_point with tracing.

Post-training Quantization:

Breaking changes:

• nncf.quantize signature has been changed to add mode: Optional[nncf.QuantizationMode] = None as its 3-rd argument, between the original calibration_dataset and preset arguments.
• (Common) nncf.common.quantization.structs.QuantizationMode has been renamed to nncf.common.quantization.structs.QuantizationScheme

General:

• (OpenVINO) Changed default OpenVINO opset from 9 to 13.

Features:

• (OpenVINO) Added 4-bit data-aware weights compression. For that dataset optional parameter has been added to nncf.compress_weights() and can be used to minimize accuracy degradation of compressed models (note that this option increases the compression time).
• (PyTorch) Added support for PyTorch models with shared weights and custom PyTorch modules in nncf.compress_weights(). The weights compression algorithm for PyTorch models is now based on tracing the model graph. The dataset parameter is now required in nncf.compress_weights() for the compression of PyTorch models.
• (Common) Renamed the nncf.CompressWeightsMode.INT8 to nncf.CompressWeightsMode.INT8_ASYM and introduce nncf.CompressWeightsMode.INT8_SYM that can be efficiently used with dynamic 8-bit quantization of activations. The original nncf.CompressWeightsMode.INT8 enum value is now deprecated.
• (OpenVINO) Added support for quantizing the ScaledDotProductAttention operation from OpenVINO opset 13.
• (OpenVINO) Added FP8 quantization support via nncf.QuantizationMode.FP8_E4M3 and nncf.QuantizationMode.FP8_E5M2 enum values, invoked via passing one of these values as an optional mode argument to nncf.quantize. Currently, OpenVINO supports inference of FP8-quantized models in reference mode with no performance benefits and can be used for accuracy projections.
• (Common) Post-training Quantization with Accuracy Control - nncf.quantize_with_accuracy_control() has been extended by restore_mode optional parameter to revert weights to int8 instead of the original precision. This parameter helps to reduce the size of the quantized model and improves its performance. By default, it's disabled and model weights are reverted to the original precision in nncf.quantize_with_accuracy_control().
• (Common) Added an all_layers: Optional[bool] = None argument to nncf.compress_weights to indicate whether embeddings and last layers of the model should be compressed to a primary precision. This is relevant to 4-bit quantization only.
• (Common) Added a sensitivity_metric: Optional[nncf.parameters.SensitivityMetric] = None argument to nncf.compress_weights for finer control over the sensitivity metric for assigning quantization precision to layers. Defaults to weight quantization error if a dataset is not provided for weight compression and to maximum variance of the layers' inputs multiplied by inverted 8-bit quantization noise if a dataset is provided. By default, the backup precision is assigned for the embeddings and last layers.

Fixes:

• (OpenVINO) Models with embeddings (e.g. gpt-2, stable-diffusion-v1-5, stable-diffusion-v2-1, opt-6.7b, falcon-7b, bloomz-7b1) are now more accurately quantized.
• (PyTorch) nncf.strip(..., do_copy=True) now actually returns a deepcopy (stripped) of the model object.
• (PyTorch) Post-hooks can now be set up on operations that return torch.return_type (such as torch.max).
• (PyTorch) Improved dynamic graph tracing for various tensor operations from torch namespace.
• (PyTorch) More robust handling of models with disjoint traced graphs when applying PTQ.

Improvements:

• Reformatted the tutorials section in the top-level README.md for better readability.

Deprecations/Removals:

• (Common) The original nncf.CompressWeightsMode.INT8 enum value is now deprecated.
• (PyTorch) The Git patch for integration with HuggingFace transformers repository is marked as deprecated and will be removed in a future release. Developers are advised to use optimum-intel instead.
• Dockerfiles in the NNCF Git repository are deprecated and will be removed in a future release.

Post-training Quantization:

Features:

• (OpenVINO) Added support for data-free 4-bit weights compression through NF4 and INT4 data types (compress_weights(…) pipeline).
• (OpenVINO) Added support for IF operation quantization.
• (OpenVINO) Added dump_intermediate_model parameter support for AccuracyAwareAlgorithm (quantize_with_accuracy_control(…) pipeline).
• (OpenVINO) Added support for SmoothQuant and ChannelAlignment algorithms for HyperparameterTuner algorithm (quantize_with_tune_hyperparams(…) pipeline).
• (PyTorch) Post-training Quantization is now supported with quantize(…) pipeline and the common implementation of quantization algorithms. Deprecated create_compressed_model() method for Post-training Quantization.
• Added new types (AvgPool, GroupNorm, LayerNorm) to the ignored scope for ModelType.Transformer scheme.
• QuantizationPreset.Mixed was set as the default for ModelType.Transformer scheme.

Fixes:

• (OpenVINO, ONNX, PyTorch) Aligned/added patterns between backends (SE block, MVN layer, multiple activations, etc.) to restore performance/metrics.
• Fixed patterns for ModelType.Transformer to align with the quantization scheme.

Improvements:

• Improved UX with the new progress bar for pipeline, new exceptions, and .dot graph visualization updates.
• (OpenVINO) Optimized WeightsCompression algorithm (compress_weights(…) pipeline) execution time for LLM's quantization, added ignored scope support.
• (OpenVINO) Optimized AccuracyAwareQuantization algorithm execution time with multi-threaded approach while calculating ranking score (quantize_with_accuracy_control(…) pipeline).
• (OpenVINO) Added extract_ov_subgraph tool for large IR subgraph extraction.
• (ONNX) Optimized quantization pipeline (up to 1.15x speed up).

Tutorials:

• Post-Training Optimization of BLIP Model
• Post-Training Optimization of DeepFloyd IF Model
• Post-Training Optimization of Grammatical Error Correction Model
• Post-Training Optimization of Dolly 2.0 Model
• Post-Training Optimization of Massively Multilingual Speech Model
• Post-Training Optimization of OneFormer Model
• Post-Training Optimization of InstructPix2Pix Model
• Post-Training Optimization of LLaVA Model
• Post-Training Optimization of Latent Consistency Model
• Post-Training Optimization of Distil-Whisper Model
• Post-Training Optimization of FastSAM Model

Known issues:

• (ONNX) quantize(...) method can generate inaccurate int8 results for models with the BatchNormalization layer that contains biases. To get the best accuracy, use the do_constant_folding=True option during export from PyTorch to ONNX.

Compression-aware training:

Fixes:

• (PyTorch) Fixed Hessian trace calculation to solve #2155 issue.

Requirements:

• Updated PyTorch version (2.1.0).
• Updated numpy version (<1.27).

Deprecations/Removals:

• (PyTorch) Removed legacy external quantizer storage names.
• (PyTorch) Removed torch < 2.0 version support.

Post-training Quantization:

Features:

• Added CPU_SPR device type support.
• Added quantizers scales unification.
• Added quantization scheme for ReduceSum operation.
• Added new types (ReduceL2, ReduceSum, Maximum) to the ignored scope for ModelType.Transformer.
• (OpenVINO) Added SmoothQuant algorithm.
• (OpenVINO) Added ChannelAlignment algorithm.
• (OpenVINO) Added HyperparameterTuner algorithm.
• (PyTorch) Added FastBiasCorrection algorithm support.
• (OpenVINO, ONNX) Added embedding weights quantization.
• (OpenVINO, PyTorch) Added new compress_weights method that provides data-free INT8 weights compression.

Fixes:

• Fixed detection of decomposed post-processing in models.
• Multiple fixes (new patterns, bugfixes, etc.) to solve #1936 issue.
• Fixed model reshaping while quantization to keep original model shape.
• (OpenVINO) Added support for sequential models quanitzation.
• (OpenVINO) Fixed in-place statistics cast to support empty dimensions.
• (OpenVINO, ONNX) Fixed quantization of the MatMul operation with weights rank > 2.
• (OpenVINO, ONNX) Fixed BiasCorrection algorithm to enable CLIP model quantization.

Improvements:

• Optimized quantize(…) pipeline (up to 4.3x speed up in total).
• Optimized quantize_with_accuracy_control(…) pipelilne (up to 8x speed up for 122-quantizing-model-with-accuracy-control notebook).
• Optimized general statistics collection (up to 1.2x speed up for ONNX backend).
• Ignored patterns separated from Fused patterns scheme (with multiple patterns addition).

Tutorials:

• Post-Training Optimization of Segment Anything Model.
• Post-Training Optimization of CLIP Model.
• Post-Training Optimization of ImageBind Model.
• Post-Training Optimization of Whisper Model.
• Post-Training Optimization with accuracy control.

Compression-aware training:

Features:

• Added shape pruning processor for BootstrapNAS algorithm.
• Added KD loss for BootstrapNAS algorithm.
• Added validate_scopes parameter for NNCF configuration.
• (PyTorch) Added PyTorch 2.0 support.
• (PyTorch) Added .strip() option to API.
• (PyTorch) Enabled bfloat data type for quantization kernels.
• (PyTorch) Quantized models can now be torch.jit.traced without calling .strip().
• (PyTorch) Added support for overridden forward instance attribute on model objects passed into create_compressed_model.
• (Tensorflow) Added Tensorflow 2.12 support.

Fixes:

• (PyTorch) Fixed padding adjustment issue in the elastic kernel to work with the different active kernel sizes.
• (PyTorch) Fixed the torch graph tracing in the case the tensors belonging to parallel edges are interleaved in the order of the tensor argument.
• (PyTorch) Fixed recurrent nodes matching (LSTM, GRU cells) condition with the strict rule to avoid adding not necessary nodes to the ignored scope.
• (PyTorch) Fixed torch.jit.script wrapper so that user-side handling exceptions during torch.jit.script invocation do not cause NNCF to be permanently disabled.
• (PyTorch, Tensorflow) Adjusted quantizer propagation algorithm to check if quantizer propagation will result in output quantization.
• (PyTorch) Added redefined __class__ method for ProxyModule that avoids causing error while calling .super() in forward method.

Deprecations/Removals:

• (PyTorch) Removed deprecated NNCFNetwork.__getattr__, NNCFNetwork.get_nncf_wrapped_model methods.

Requirements:

• Updated PyTorch version (2.0.1).
• Updated Tensorflow version (2.12.0).

Post-training Quantization:

Features:

• Official release of OpenVINO framework support.
  • Ported NNCF OpenVINO backend to use the nGraph representation of OpenVINO models.
  • Changed dependecies of NNCF OpenVINO backend. It now depends on openvino package and not on the openvino-dev package.
  • Added GRU/LSTM quantization support.
  • Added quantizer scales unification.
  • Added support for models with 3D and 5D Depthwise convolution.
  • Added FP16 OpenVINO models support.
• Added "overflow_fix" parameter (for quantize(...) & quantize_with_accuracy_control(...) methods) support & functionality. It improves accuracy for optimized model for affected devices. More details in Quantization section.
• (OpenVINO) Added support for in-place statistics collection (reduce memory footprint during optimization).
• (OpenVINO) Added Quantization with accuracy control algorithm.
• (OpenVINO) Added YOLOv8 examples for quantize(...) & quantize_with_accuracy_control(...) methods.
• (PyTorch) Added min-max quantization algorithm as experimental.

Fixes:

• Fixed ignored_scope attribute behaviour for weights. Now, the weighted layers excludes from optimization scope correctly.
• (ONNX) Checking correct ONNX opset version via the nncf.quantize(...). Now, models with opset < 13 are optimized correctly in per-tensor quantization.

Improvements:

• Added improvements for statistic collection process (collect weights statistics only once).
• (PyTorch, OpenVINO, ONNX) Introduced unified quantizer parameters calculation.

Known issues:

• quantize(...) method can generate inaccurate int8 results for models with the DenseNet-like architecture. Use quantize_with_accuracy_control(...) in such case.
• quantize(...) method can hang on models with transformer architecture when fast_bias_correction optional parameter is set to False. Don't set it to False or use quantize_with_accuracy_control(...) in such case.
• quantize(...) method can generate inaccurate int8 results for models with the MobileNet-like architecture on non-VNNI machines.

Compression-aware training:

New Features:

• Introduced automated structured pruning algorithm for JPQD with support for BERT, Wave2VecV2, Swin, ViT, DistilBERT, CLIP, and MobileBERT models.
• Added nncf.common.utils.patcher.Patcher - this class can be used to patch methods on live PyTorch model objects with wrappers such as nncf.torch.dynamic_graph.context.no_nncf_trace when doing so in the model code is not possible (e.g. if the model comes from an external library package).
• Compression controllers of the nncf.api.compression.CompressionAlgorithmController class now have a .strip() method that will return the compressed model object with as many custom NNCF additions removed as possible while preserving the functioning of the model object as a compressed model.

Fixes:

• Fixed statistics computation for pruned layers.
• (PyTorch) Fixed traced tensors to implement the YOLOv8 from Ultralytics.

Improvements:

• Extension of attributes (transpose/permute/getitem) for pruning node selector.
• NNCFNetwork was refactored from a wrapper-approach to a mixin-like approach.
• Added average pool 3d-like ops to pruning mask.
• Added Conv3d for overflow fix.
• nncf.set_log_file(...) can now be used to set location of the NNCF log file.
• (PyTorch) Added support for pruning of torch.nn.functional.pad operation.
• (PyTorch) Added torch.baddbmm as an alias for the matmul metatype for quantization purposes.
• (PyTorch) Added config file for ResNet18 accuracy-aware pruning + quantization on CIFAR10.
• (PyTorch) Fixed JIT-traceable PyTorch models with internal patching.
• (PyTorch) Added __matmul__ magic functions to the list of patched ops (for SwinTransformer by Microsoft).

Requirements:

• Updated ONNX version (1.13)
• Updated Tensorflow version (2.11)

General changes:

• Added Windows support for NNCF.

Target version updates:

• Bump target framework versions to PyTorch 1.13.1, TensorFlow 2.8.x, ONNX 1.12, ONNXRuntime 1.13.1
• Increased target HuggingFace transformers version for the integration patch to 4.23.1

Features:

• Official release of the ONNX framework support. NNCF may now be used for post-training quantization (PTQ) on ONNX models. Added an example script demonstrating the ONNX post-training quantization on MobileNetV2.
• Preview release of OpenVINO framework support. NNCF may now be used for post-training quantization on OpenVINO models. Added an example script demonstrating the OpenVINO post-training quantization on MobileNetV2. pip install nncf[openvino] will install NNCF with the required OV framework dependencies.
• Common post-training quantization API across the supported framework model formats (PyTorch, TensorFlow, ONNX, OpenVINO IR) via the nncf.quantize(...) function. The parameter set of the function is the same for all frameworks - actual framework-specific implementations are being dispatched based on the type of the model object argument.
• (PyTorch, TensorFlow) Improved the adaptive compression training functionality to reduce effective training time.
• (ONNX) Post-processing nodes are now automatically excluded from quantization.
• (PyTorch - Experimental) Joint Pruning, Quantization and Distillation for Transformers enabled for certain models from HuggingFace transformers repo. See description of the movement pruning involved in the JPQD for details.

Bugfixes:

• Fixed a division by zero if every operation is added to ignored scope
• Improved logging output, cutting down on the number of messages being output to the standard logging.INFO log level.
• Fixed FLOPS calculation for linear filters - this impacts existing models that were pruned with a FLOPS target.
• "chunk" and "split" ops are correctly handled during pruning.
• Linear layers may now be pruned by input and output independently.
• Matmul-like operations and subsequent arithmetic operations are now treated as a fused pattern.
• (PyTorch) Fixed a rare condition with accumulator overflow in CUDA quantization kernels, which led to CUDA runtime errors and NaN values appearing in quantized tensors and
• (PyTorch) transformers integration patch now allows to export to ONNX during training, and not only at the end of it.
• (PyTorch) torch.nn.utils.weight_norm weights are now detected correctly.
• (PyTorch) Exporting a model with sparsity or pruning no longer leads to weights in the original model object in-memory to be hard-set to 0.
• (PyTorch - Experimental) improved automatic search of blocks to skip within the NAS algorithm – overlapping blocks are correctly filtered.
• (PyTorch, TensorFlow) Various bugs and issues with compression training were fixed.
• (TensorFlow) Fixed an error with "num_bn_adaptation_samples": 0 in config leading to a TypeError during quantization algo initialization.
• (ONNX) Temporary model file is no longer saved on disk.
• (ONNX) Depthwise convolutions are now quantizable in per-channel mode.
• (ONNX) Improved the working time of PTQ by optimizing the calls to ONNX shape inferencing.

Breaking changes:

• Fused patterns will be excluded from quantization via ignored_scopes only if the top-most node in data flow order matches against ignored_scopes
• NNCF config's "ignored_scopes" and "target_scopes" are now strictly checked to be matching against at least one node in the model graph instead of silently ignoring the unmatched entries.
• Calling setup.py directly to install NNCF is deprecated and no longer guaranteed to work.
• Importing NNCF logger as from nncf.common.utils.logger import logger as nncf_logger is deprecated - use from nncf import nncf_logger instead.
• pruning_rate is renamed to pruning_level in pruning compression controllers.
• (ONNX) Removed CompressionBuilder. Excluded examples of NNCF for ONNX with CompressionBuilder API

New features

• (ONNX) PTQ API support for ONNX.
• (ONNX) Added PTQ examples for ONNX in image classification, object detection, and semantic segmentation.
• (PyTorch) Added BootstrapNAS to find high-performing sub-networks from the super-network optimization.

Bugfixes

• (PyTorch) Returned the initial quantized model when the retraining failed to find out the best checkpoint.
• (Experimental) Fixed weight initialization for ONNXGraph and MinMaxQuantization.

New features

• Pre-production quality
• (TensorFlow) Added TensorFlow 2.5.x support.
• (TensorFlow) The SubclassedConverter class was added to create NNCFGraph for the tf.Graph Keras model.
• (TensorFlow) Added TFOpLambda layer support with TFModelConverter, TFModelTransformer, and TFOpLambdaMetatype.
• (TensorFlow) Patterns from MatMul and Conv2D to BiasAdd and Metatypes of TensorFlow operations with weights TFOpWithWeightsMetatype are added.
• (PyTorch, TensorFlow) Added prunings for Reshape and Linear as ReshapePruningOp and LinearPruningOp.
• (PyTorch) Added mixed precision quantization config with HAWQ for Resnet50 and Mobilenet_v2 for the latest VPU.
• (PyTorch) Splitted NNCFBatchNorm into NNCFBatchNorm1d, NNCFBatchNorm2d, NNCFBatchNorm3d.
• (PyTorch - Experimental) Added the BNASTrainingController and BNASTrainingAlgorithm for BootstrapNAS to search the model's architecture.
• (Experimental) ONNX ModelProto is now converted to NNCFGraph through GraphConverter.
• (Experimental) ONNXOpMetatype and extended patterns for fusing HW config is now available.
• (Experimental) Added ONNXPostTrainingQuantization and MinMaxQuantization supports for ONNX.

Bugfixes

• (PyTorch, TensorFlow) Added exception handling of BN adaptation for zero sample values.
• (PyTorch, TensorFlow) Fixed learning rate after validation step for EarlyExitCompressionTrainingLoop.
• (PyTorch) Fixed FakeQuantizer to make exact zeros.
• (PyTorch) Fixed Quantizer misplacements during ONNX export.
• (PyTorch) Restored device information during ONNX export.
• (PyTorch) Fixed the statistics collection from the pruned model.

New features

• (PyTorch) All PyTorch operations are now NNCF-wrapped automatically.
• (TensorFlow) Scales for concat-affecting quantizers are now unified
• (PyTorch) The pruned filters are now set to 0 in the exported ONNX file instead of removing them from the ONNX definition.
• (PyTorch, TensorFlow) Extended accuracy-aware training pipeline with the early_exit mode.
• (PyTorch, TensorFlow) Added support for quantization presets to be specified in NNCF config.
• (PyTorch, TensorFlow) Extended pruning statistics displayed to the user.
• (PyTorch, TensorFlow) Users may now register a dump_checkpoints_fn callback to control the location of checkpoint saving during accuracy-aware training.
• (PyTorch, TensorFlow) Default pruning schedule is now exponential.
• (PyTorch) SILU activation now supported.
• (PyTorch) Dynamic graph no longer traced during compressed model execution, which improves training performance of models compressed with NNCF.
• (PyTorch) Added BERT-MRPC quantization results and integration instructions to the HuggingFace Transformers integration patch.
• (PyTorch) Knowledge distillation extended with the option to specify temperature for the softmax mode.
• (TensorFlow) Added mixed_min_max option for quantizer range initialization.
• (PyTorch, TensorFlow) ReLU6-based HSwish and HSigmoid activations are now properly fused.
• (PyTorch - Experimental) Added an algorithm to search the model's architecture for basic building blocks.

Bugfixes:

• (TensorFlow) Fixed a bug where an operation with int32 inputs (following a Cast op) was attempted to be quantized.
• (PyTorch, TensorFlow) LeakyReLU now properly handled during pruning
• (PyTorch) Fixed errors with custom modules failing at the determine_subtype stage of metatype assignment.
• (PyTorch) Fix handling modules with torch.nn.utils.weight_norm.WeightNorm applied

Target version updates:

• Relax TensorFlow version requirements to 2.4.x

Target version updates:

• Bump target framework versions to PyTorch 1.9.1 and TensorFlow 2.4.3
• Increased target HuggingFace transformers version for the integration patch to 4.9.1

Bugfixes:

• Fixed statistic collection for the algo mixing scenario
• Increased pruning algorithm robustness in cases of a disconnected NNCF graph
• Fixed the fatality of NNCF graph PNG rendering failures
• Fixed README command lines
• (PyTorch) Fixed a bug with quantizing shared weights multiple times
• (PyTorch) Fixed knowledge distillation failures in CPU-only and DataParallel scenarios
• (PyTorch) Fixed sparsity application for torch.nn.Embedding and EmbeddingBag modules
• (PyTorch) Added GroupNorm + ReLU as a fusable pattern
• (TensorFlow) Fixed gamma fusion handling for pruning TF BatchNorm
• (PyTorch) Fixed pruning for models where operations have multiple convolution predecessors
• (PyTorch) Fixed NNCFNetwork wrapper so that self in the calls to the wrapped model refer to the wrapper NNCFNetwork object and not to the wrapped model
• (PyTorch) Fixed tracing of view operations to handle shape arguments with the torch.Tensor type
• (PyTorch) Added matmul ops to be considered for fusing
• (PyTorch, TensorFlow) Fixed tensorboard logging for accuracy-aware scenarios
• (PyTorch, TensorFlow) Fixed FLOPS calculation for grouped convolutions
• (PyTorch) Fixed knowledge distillation failures for tensors of unsupported shapes - will ignore output tensors with unsupported shapes now instead of crashing.

New features:

• Added TensorFlow 2.4.2 support - NNCF can now be used to apply the compression algorithms to models originally trained in TensorFlow. NNCF with TensorFlow backend supports the following features:

  • Compression algorithms:
    • Quantization (with HW-specific targeting aligned with PyTorch)
    • Sparsity:
      • Magnitude Sparsity
      • RB Sparsity
    • Filter pruning
  • Support for only Keras models consisting of standard Keras layers and created by:
    • Keras Sequential API
    • Keras Functional API
  • Automatic, configurable model graph transformation to obtain the compressed model.
  • Distributed training on multiple GPUs on one machine is supported using tf.distribute.MirroredStrategy.
  • Exporting compressed models to SavedModel or Frozen Graph format, ready to use with OpenVINO™ toolkit.

• Added model compression samples for NNCF with TensorFlow backend:

  • Classification
    • Keras training loop.
    • Models form the tf.keras.applications module (ResNets, MobileNets, Inception and etc.) are supported.
    • TensorFlow Datasets (TFDS) and TFRecords (ImageNet2012, Cifar100, Cifar10) are supported.
    • Compression results are claimed for MobileNet V2, MobileNet V3 small, MobileNet V3 large, ResNet50, Inception V3.
  • Object Detection (Compression results are claimed for RetinaNet, YOLOv4)
    • Custom training loop.
    • TensorFlow Datasets (TFDS) and TFRecords for COCO2017 are supported.
    • Compression results for are claimed for RetinaNet, YOLOv4.
  • Instance Segmentation
    • Custom training loop
    • TFRecords for COCO2017 is supported.
    • Compression results are claimed for MaskRCNN

• Accuracy-aware training available for filter pruning and sparsity in order to achieve best compression results within a given accuracy drop threshold in a fully automated fashion.

• Framework-specific checkpoints produced with NNCF now have NNCF-specific compression state information included, so that the exact compressed model state can be restored/loaded without having to provide the same NNCF config file that was used during the creation of the NNCF-compressed checkpoint

• Common interface for compression methods for both PyTorch and TensorFlow backends (https://github.com/openvinotoolkit/nncf/tree/develop/nncf/api).

• (PyTorch) Added an option to specify an effective learning rate multiplier for the trainable parameters of the compression algorithms via NNCF config, for finer control over which should tune faster - the underlying FP32 model weights or the compression parameters.

• (PyTorch) Unified scales for concat operations - the per-tensor quantizers that affect the concat operations will now have identical scales so that the resulting concatenated tensor can be represented without loss of accuracy w.r.t. the concatenated subcomponents.

• (TensorFlow) Algo-mixing: Added configuration files and reference checkpoints for filter-pruned + qunatized models: ResNet50@ImageNet2012(40% of filters pruned + INT8), RetinaNet@COCO2017(40% of filters pruned + INT8).

• (Experimental, PyTorch) Learned Global Ranking filter pruning mechanism for better pruning ratios with less accuracy drop for a broad range of models has been implemented.

• (Experimental, PyTorch) Knowledge distillation supported, ready to be used with any compression algorithm to produce an additional loss source of the compressed model against the uncompressed version

Breaking changes:

• CompressionLevel has been renamed to CompressionStage
• "ignored_scopes" and "target_scopes" no longer allow prefix matching - use full-fledged regular expression approach via {re} if anything more than an exact match is desired.
• (PyTorch) Removed version-agnostic name mapping for ReLU operations, i.e. the NNCF configs that referenced "RELU" (all caps) as an operation name will now have to reference an exact ReLU PyTorch function name such as "relu" or "relu_"
• (PyTorch) Removed the example of code modifications (Git patches and base commit IDs are provided) for mmdetection repository.
• Batchnorm adaptation "forgetting" step has been removed since it has been observed to introduce accuracy degradation; the "num_bn_forget_steps" parameter in the corresponding NNCF config section has been removed.
• Framework-specific requirements no longer installed during pip install nncf or python setup.py install and are assumed to be present in the user's environment; the pip's "extras" syntax must be used to install the BKC requirements, e.g. by executing pip install nncf[tf], pip install nncf[torch] or pip install nncf[tf,torch]
• "quantizable_subgraph_patterns" option removed from the NNCF config

Bugfixes:

• (PyTorch) Fixed a hang with batchnorm adaptation being applied in DDP mode
• (PyTorch) Fixed tracing of the operations that return NotImplemented

• Fixed a bug with where compressed models that were supposed to return named tuples actually returned regular tuples
• Fixed an issue with batch norm adaptation-enabled compression runs hanging in the DDP scenario

New features:

• Adjust Padding feature to support accurate execution of U4 on VPU - when setting "target_device" to "VPU", the training-time padding values for quantized convolutions will be adjusted to better reflect VPU inference process.
• Weighted layers that are "frozen" (i.e. have requires_grad set to False at compressed model creation time) are no longer considered for compression, to better handle transfer learning cases.
• Quantization algorithm now sets up quantizers without giving an option for requantization, which guarantees best performance, although at some cost to quantizer configuration flexibility.
• Pruning models with FCOS detection heads and instance normalization operations now supported
• Added a mean percentile initializer for the quantization algorithm
• Now possible to additionally quantize model outputs (separate control for each output quantization is supported)
• Models quantized for CPU now use effective 7-bit quantization for weights - the ONNX-exported model is still configured to use 8 bits for quantization, but only the middle 128 quanta of the total possible 256 are actually used, which allows for better OpenVINO inference accuracy alignment with PyTorch on non-VNNI CPUs
• Bumped target PyTorch version to 1.8.1 and relaxed package requirements constraints to allow installation into environments with PyTorch >=1.5.0

Notable bugfixes:

• Fixed bias pruning in depthwise convolution
• Made per-tensor quantization available for all operations that support per-channel quantization
• Fixed progressive training performance degradation when an output tensor of an NNCF-compressed model is reused as its input.
• pip install . path of installing NNCF from a checked-out repository is now supported.
• Nested with no_nncf_trace() blocks now function as expected.
• NNCF compression API now formally abstract to guard against virtual function calls
• Now possible to load AutoQ and HAWQ-produced checkpoints to evaluate them or export to ONNX

Removed features:

• Pattern-based quantizer setup mode for quantization algorithm - due to its logic, it did not guarantee that all required operation inputs are ultimately quantized.

• Added AutoQ - an AutoML-based mixed-precision initialization mode for quantization, which utilizes the power of reinforcement learning to select the best quantizer configuration for any model in terms of quality metric for a given HW architecture type.
• NNCF now supports inserting compression operations as pre-hooks to PyTorch operations, instead of abusing the post-hooking; the flexibility of quantization setups has been improved as a result of this change.
• Improved the pruning algorithm to group together dependent filters from different layers in the network and prune these together
• Extended the ONNX compressed model exporting interface with an option to explicitly name input and output tensors
• Changed the compression scheduler so that the correspondingepoch_step and step methods should now be called in the beginning of the epoch and before the optimizer step (previously these were called in the end of the epoch and after the optimizer step respectively)
• Data-dependent compression algorithm initialization is now specified in terms of dataset samples instead of training batches, e.g. "num_init_samples" should be used in place of "num_init_steps" in NNCF config files.
• Custom user modules to be registered for compression can now be specified to be ignored for certain compression algorithms
• Batch norm adaptation now being applied by default for all compression algorithms
• Bumped target PyTorch version to 1.7.0
• Custom OpenVINO operations such as "FakeQuantize" that appear in NNCF-exported ONNX models now have their ONNX domain set to org.openvinotoolkit
• The quantization algorithm will now quantize nn.Embedding and nn.EmbeddingBag weights when targeting CPU
• Added an option to optimize logarithms of quantizer scales instead of scales themselves directly, a technique which improves convergence in certain cases
• Added reference checkpoints for filter-pruned models: UNet@Mapillary (25% of filters pruned), SSD300@VOC (40% of filters pruned)

• Models with filter pruning applied are now exportable to ONNX
• BatchNorm adaptation now available as a common compression algorithm initialization step - currently disabled by default, see "batchnorm_adaptation" config parameters in compression algorithm documentation (e.g. Quantizer.md) for instructions on how to enable it in NNCF config
• Major performance improvements for per-channel quantization training - now performs almost as fast as the per-tensor quantization training
• nn.Embedding and nn.Conv1d weights are now quantized by default
• Compression level querying is now available to determine current compression level (for purposes of choosing a correct "best" checkpoint during training)
• Generalized initializing data loaders to handle more interaction cases between a model and the associated data loader
• FP16 training supported for quantization
• Ignored scopes can now be set for the propagation-based quantization setup mode
• Per-optimizer stepping enabled as an option for polynomial sparsity scheduler
• Added an example config and model checkpoint for the ResNet50 INT8 + 50% sparsity (RB)