PyPI · #642 most downloaded on PyPI

sentencepiece

0.2.227M downloads/mogoogle/sentencepiece ↗New releases in your daily digest, changelog included.

Unsupervised text tokenizer and detokenizer.

Last release 2 months ago
12 Jul 2026
Release timing varies
gaps range from 3 weeks to 1.5 years
Most releases are documented
notes for 20 of 33 stable releases
1 version withdrawn
withdrawn after publishing
9 years old
34 releases · first in 2017

Release timeline

34 releases since 2017

2017201920212023

Releases

0.2.212 Jul 2026Deprecated
Deprecate Win32 Wheels : Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c ).
Release Notes v0.2.2

This release contains major updates, including the migration of the Python wrapper to pybind11, native NumPy support, enhanced offset mapping, performance optimizations, security hardening, and internal code modernization.

1. Python Wrapper (pybind11 Migration & Features)

SWIG to pybind11 Migration: Fully migrated the Python wrapper from SWIG to pybind11, enabling better performance, safer memory management, and cleaner code integration (#1266, Commit 528dc9e).

Native NumPy Support:

direct encode/decoder from/to numpy object

Enhanced Offset Mapping:

Added new offset mapping features supporting both Unicode character offsets and raw byte offsets.

Added bounds checking to prevent OOB reads in offset mapping (Commit c5b0ce0).

Fixed crash issues and OOB spans in alignment conversions (#1231, #1211).

Type Stubs (PEP 561): Added inline type stubs (.pyi files) and packaged PEP 561 marker to provide auto-completion and static type safety in IDEs (#1258, Commit 8b7f97b).

Parallel Encoding API: Introduced parallel_encode and ThreadPool APIs, allowing Python users to tokenize single large documents across multiple CPU cores in parallel (#1254, Commit 3c37c39).

Direct Proto Loading: Added support for loading processors and normalizers directly from in-memory Proto instances (Commit 12788ba).

Concurrency Enhancements: Added GIL release during C++ execution to improve multi-threaded Python performance (#1251, Commit 0a044ae).

BOS/EOS Behavior Hardening: Aligned BOS/EOS adding behavior with C++ constraints, raising explicit ValueError instead of silent failures or crashes (Commit 780618a).

2. Performance Optimizations

BPE Training Optimization: Ported O(log n) lazy priority queue update optimization to BPE trainer to speed up training (Commit 8db9878).

Encode/Decode Optimizations: Decoding/Encoding optimization for integer/string input case by skipping SentencePieceText conversion, resulting in ~2x speedup (Commit 4b9231b).

Unigram Pruning: Improved unigram pruning using token-normalized frequency calculations (#1135, Commit 1cd0c59).

3. Security & Hardening

Trie Validation at Load: Added verification of darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files (#1224, Commit 6b2f05d).

Bounds Checking in Maps: Added bounds checks to piece accessors and validated value offsets in DecompileCharsMap (#1265, Commit 82b7c6d).

Crash Prevention: Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs (Commit 66b2587).

Null Character Disallowment: Explicitly disallow vocabulary pieces containing null characters (Commit eb3ba49).

NBest Timeout: Implemented a global timeout for NBest A* search to prevent hangs on pathological inputs (Commit ba57749).

4. C++ API & Internals Modernization

Abseil Migration: Extensively migrated from custom utilities to Abseil primitives ([#1270]

Normalizer Maps: Supported loading normalizer from raw maps in C++ (#1273, Commit e476e5e).

5. Build & Packaging

Deprecate Win32 Wheels: Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c).

Modernized Python builds: Switched from calling setup.py directly to the standard pypa/build PEP 517 frontend (#1167, Commit f300c7d).

PEP 639 License: Added PEP 639 license metadata to pyproject.toml (#1173).

6. Documentation

API Cheat Sheet: Added comparison tables, zero-copy, and sampling comparison guides.

Refactored Options: Fully reorganized options.md and special_symbols.md with verified code examples.

One column per quarter.

0.2.112 Aug 2025Security

[ALL]: Security fix to address a heap overflow issue that could occur when using a model containing an invalid precompiled normalization model.

Major changes

[Python] Supported wheels and builds for Python 3.13 and 3.14(rc1) #1134, #1127, #1121, #1111, #1110, #1104, #1103, #1099, #1091
[Python] Added an experimental support for free-threading. #1134, #1127, #1110 https://github.com/google/sentencepiece/tree/master/python#free-threading-support
[Python] Updated the supported Python version to 3.9 or later.

New features

[ALL]: Added new build mode to prevent the precompiled normalization rules being embedded in *.so and *.a. (-DSPM_DISABLE_EMBEDDED_DATA=ON). This reduces the runtime size by approximately 1-2 MB. This mode is enabled to build python wheels. The rules are loaded as the data package.

Bug fixes & minor changes

[ALL]: Security fix to address a heap overflow issue that could occur when using a model containing an invalid precompiled normalization model.
[Python]: Deprecates the wheel package for Linux i686.
[Python]: Supported wheel for Windows Arm64. #1114
[Python]: Fixed the crash issue on batch decoding #1051
[ALL]: Updated the Unicode normalization rule with the latest ICU/Unicode rules.
[ALL]: Unused code and build mode cleanup.

Open source →

0.2.019 Feb 2024

[ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample

Major changes

N/A

New features

[ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample
[ALL] Added SentencePieceProcessor::Normalize method in C++/Python Python Sample
C++ Sample
[ALL] Added functionality to override the normalization spec before the processing. Python Sample

Bug fixes & minor changes

Introduce better support of using external abseil and protobuf #869
Build universal binary in OSX release package #892
Add the set_min_log_level function to python to change the loglevel from the python wrapper. #893
Uses the logsumexp techniques in marginal probabilities of n-best tokenization to avoid underflow.
Support Python 3.12 #932
Improves the thread utilization in batch encoding/decoding.
Fix nasty bug in BPE position encoding.
Fix bugs in the handling of duplicated bigrams

Open source →

0.1.992 May 2023

[ALL] Fixes the NaN issues in unigram model training: #851

Major changes

N/A

New features

N/A

Bug fixes & minor changes

[ALL] Fixes the NaN issues in unigram model training: #851
[ALL] Fixes the bug in unigram loss computation: #628
[ALL] Fixes the minor bug in BPE token extraction algorithm: #318
[ALL] Increase the number of maximum threads from 128 to 1024. #857

Open source →

0.1.9812 Apr 2023

Python 3.11 support (wheel packages for python 3.11 are available)

Major changes

Python 3.11 support (wheel packages for python 3.11 are available)
Includes the entire full sources in the source python package to reduce the pip install troubles.
Improves the algorithm to initialize unigram seed vocabulary. Coverage is improved.

New features

[ALL] Added the feature to train the model with pre-tokenization boundary constraints. (--pretokenization_delimiter) flag

Bug fixes & minor changes

[ALL] Makes the error message more descriptive.
[ALL] Fixes the crash error when std::random_device failed
[ALL] Fixes the build error on Raspberry pi around atomic operation
[ALL] Fixes the minor bugs in nbest enumeration
[ALL] Fixes the build error when using the external protobuf library.
[ALL] Fixes the build error on a big-endian machine.
[Windows] Use /MD build flag instead of /MT.

Open source →

0.1.977 Aug 2022

Migrated the C++ version from C++11 to C++17.

Major changes

Migrated the C++ version from C++11 to C++17.
Migrated the CI environment from Travis-CI to Github actions
Started using cibuildtool to build pypi wheel packages

New features

[ALL] Support differential privacy while training. https://aclanthology.org/2022.findings-acl.171.pdf
[ALL] Introduced APIs that return the struct of ImmutableSentencePieceText, which encodes string-token, id, and utf-8 byte offsets at once. New API is available both from C++ and Python.
[ALL] Allow tab ‘\t’ to be included in user defined symbols.
[ALL] Added NFKD normalization rule. NFKD rule is provided as a TSV file.
[ALL] Added option to emit unknown symbol <unk> instead of raw symbol.
[Python]: Batch encode/decode requests are performed in native multi-threads.
[Python]: Supports to pass a custom log stream during training.
[Python]: Adds module-level version variable: spm.__version__
[Python]: Creates wheel package of Mac universal binary.

Bug fixes & minor changes

Uses the efficient encoding algorithm by default. Removed the functionality to switch the Viterbi tokenization algorithm.
Make the output of Encode and 1-best from NBestEncode same.
Use std::string_view as much as possible.
[Python] Removed pip package for ppc64le and s390x architecture as cibuiltool doesn’t support them.

Open source →

0.1.9618 Jun 2021

Improves the performance of unigram training

Updates

Improves the performance of unigram training
Updated the nfkc normalization with the latest ICU module.
Stop handling zero-width-joiner string as whitespace.

New features

added new sampling algorithm without replacement.
added API for new sampling and perplexity calculation.
added allow_whitespace_only_pieces mode.

Open source →

0.1.9510 Jan 2021

support to build sentencepiece with the external (official) abseil library.

Updates

support to build sentencepiece with the external (official) abseil library.
upgraded protobuf 3.14.0
changed the type of input_sentence_size from int32 to uint64.

Open source →

0.1.9424 Oct 2020

added SetRandomGeneratorSeed function to set the seed value for random generator. This can allow to make reproducible sampling.

Updates

added SetRandomGeneratorSeed function to set the seed value for random generator. This can allow to make reproducible sampling.
Validate the range of the vocab id in Python module.
Change the directory arrangement of python module.
Added protobuf python module.

Bug fixes

Support to build python wheel from source package.

Open source →

0.1.928 Jun 2020withdrawn: Crash bug is reported (confirming)

Fixed the regression bug around the flag --minloglevel

Bug fix

Fixed the regression bug around the flag --minloglevel
Fixed build break on Solaris.

Minor upgrade

upgrade builtin protobuf to 3.12.3
Implmeneted absl::flags port.

Open source →

0.1.9121 May 2020

[Python] Added a feature to feed training data as Python's iterable object. https://github.com/google/sentencepiece/tree/master/python#training-withou

New API

[Python] Added a feature to feed training data as Python's iterable object. https://github.com/google/sentencepiece/tree/master/python#training-without-local-filesystem
[Python] Added a feature to set model writer to emit the output model to any non-local devices. https://github.com/google/sentencepiece/tree/master/python#training-without-local-filesystem
[C++] Add an API to returns the trained model directly as std::string.

Bug Fix

Ignores nbest parameter in BPE-dropout
fixed build error when SPM_ENABLE_NFKC_COMPILE=ON
fixed the cost computation around user_defined_symbol and faster encoding introduced in the previous release.

Open source →

0.1.9013 May 2020

Renamed v0.1.9 to v0.1.90 because PyPI doesn't recognize 0.1.9 as the latest release.

Open source →

0.1.8624 Apr 2020

Support tf 1.5.1 2.0.0 2.0.1 2.1.0 and 2.2.0rc3

Support tf 1.5.1 2.0.0 2.0.1 2.1.0 and 2.2.0rc3
Added python wrapper for Python3.8 on Mac

Open source →

0.1.8515 Dec 2019

Support tf 1.15 and Python3.8 on Windows

Open source →

0.1.8316 Aug 2019

Use the official docker image to build tf_sentencepiece ops

Use the official docker image to build tf_sentencepiece ops
support tf 1.14.0 and tf 2.0.0-beta1.

Open source →

0.1.8213 Apr 2019

Bug fix: fixed the behavior of is_unknown method in Python module.

Open source →

0.1.8122 Mar 2019

Fix: support tensorflow 0.13.1

Open source →

0.1.811 Jan 2019

Feature: Get rid of the dependency to external protobuf Feature: added (Encode|Decode)AsSerializedProto interface so Python module can get full access

Feature: Get rid of the dependency to external protobuf Feature: added (Encode|Decode)AsSerializedProto interface so Python module can get full access to the SentencePieceText proto including the byte offsets/aligments Feature: added --treat_whitespace_as_suffix option to make _ be a suffix of word. Feature: Added normalization rules to remove control characters in the default nmt_* normalizers Minor fix: simplify the error messager Minor fix: do not emit full source path in LOG(INFO)

For more detail: https://github.com/google/sentencepiece/compare/v0.1.7...v0.1.8

Open source →

0.1.726 Dec 2018Deprecated

Deprecated: --mining_sentence_size and --training_sentence_size. Load all sentences by default. --input_sentence_size can be specified to limit the se…

Deprecated: --mining_sentence_size and --training_sentence_size. Load all sentences by default. --input_sentence_size can be specified to limit the sentences to be loaded Feature: added --unk_piece/--bos_piece/--eos_piece/--pad_piece flags to change the surface representations of these special symbols. Bug fix: added third_party directory for cmake's subdirectory.

For more detail: https://github.com/google/sentencepiece/compare/v0.1.6...v0.1.7

Open source →

0.1.612 Nov 2018

Bug fix: do not apply normalization to the user-defined-symbols.

Bug fix: do not apply normalization to the user-defined-symbols.
Bug fix: stop adding extra whitespaces before user-defined symbols
Feature: added --minloglevel flag to suppress LOG(INFO) message
Feature: added --split_by_number flag to allow numbers to attach other symbols.
Feature: added --max_sentence_length flag to control the maximum byte length of input sentence for training.
used tf-versioned so file for _sentencepiece_processor_ops to minimize ABI incompatibility for tf wapper.

For more detail: https://github.com/google/sentencepiece/compare/v0.1.5...master

Open source →

0.1.529 Oct 2018