NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #659 most downloaded on PyPI
Unsupervised text tokenizer and detokenizer.
Latest stable
0.2.3
Last release 17 days ago
05 Nov 2026
Release timing varies
gaps range from 3 weeks to 1.5 years
Most releases are documented
notes for 21 of 34 stable releases
1 version withdrawn
withdrawn after publishing
9 years old
35 releases · first in 2017
Drag across the ruler to compare, or tap a release to open it.
…and community reports, and a cleanup of long-deprecated APIs and flags.
This release introduces SentencePiece Lite, a zero-dependency C++20 runtime, new vocabulary-selection algorithms for training (Auto-character Coverage and Sparse L1 Unigram pruning), a Bazel (Bzlmod) build, a large number of security and robustness fixes found by OSS-Fuzz and community reports, and a cleanup of long-deprecated APIs and flags.
lite/ runtime: A compact (~45 KB stripped, < 2k LOC) C++20 tokenizer engine that runs directly on memory-mapped FlatBuffers model binaries (.spm.fb) with no runtime dependency on Protobuf or Abseil (#1330, Commit a2cb4d8). See lite/README.md.--auto_character_coverage): New TrainerSpec.auto_character_coverage option for both BPE and Unigram that globally optimizes the allocation between single characters and multi-character subwords within the vocabulary budget, while keeping lossless round-trip via byte fallback (Commits e23ece8, df18339, d34a6ce). See doc/auto_character_coverage.md.--use_sparse_pruning): Optional L1-regularized EM vocabulary selection with Post-Lasso debiased refit for the Unigram model, replacing the discrete heuristic pruning (#1295, Commit fd53b5c). Required when --auto_character_coverage is used with Unigram. See doc/sparse_l1_unigram.md.PruneUnreachableSentencePieces: Remove pieces that can never appear in a Viterbi path during Unigram training, and fix the discrete pruning loop order (Commits b292ce5, 59c3b80).std::function<std::vector<std::string>(absl::string_view)> pretokenizer, exposed to Python via the pretokenizer= training argument; pretokenization_delimiter and callback are mutually exclusive (#1287, Commit d1b6696).unk_id = -1 with byte_fallback: Training with --unk_id=-1 is now allowed when --byte_fallback=true; null-byte pieces are treated as UNUSED when byte fallback is enabled (fixes #1308, Commit c2fbf86).spm_eval metrics: Added Bytes/Tok, Model PPL, Empirical PPL, and Coverage metrics (Commit 01305c9).SentencePieceProcessor now preserves encode options and the full instance dict (including subclasses) when pickled (#1349, Commits 4e66c53, 52b1a47).b3717da).cp314t) and GIL-safety issues in _TrainFromMap (Commits ca0f388, ee1d3dd). Free-threaded wheels are now tested on 3.14t / 3.15t.cibuildwheel 4.3.1 (Commits fa40c16, ab49d75).btree_set with a vector and parallelized initial indexing; removed the 65,535-character sentence length limit by replacing 16-bit packed positions with a Position struct (Commits a44ce92, b3717da).TrainerInterface::LoadSentences and SplitSentencesByWhitespace, with RunBatch-based dynamic load balancing for normalization (Commits a44ce92, d1b6696).StrReplaceAll in Decode() (Commit a44ce92).Lattice::NBest / Lattice::Sample, heap-free score precomputation in BoundedPriorityQueue::Gc, and deduplicated corpus byte counting (Commit ee1d3dd).SentencePieceText allocations in ParallelEncode overloads and skip alignment bookkeeping when not requested (Commit ee1d3dd).unigram::Lattice::NBest to fix timeouts/OOM; added piece-ID bounds checks in IdToPiece, GetScore, IsControl, IsUnknown, IsUnused, IsByte (fixes #1342, #1344, #1347, Commit 2e432ca).Lattice::Insert, verify EOS/begin nodes before Viterbi backtrace, and guard against null prev pointers and cycles (Commit fa40c16).PrefixMatcher::PrefixMatch and Unigram PieceToId (#1326, #1325).DecompileCharsMap; fix Builder::BuildNFC_CFMap and regenerate nfc_cf.tsv (#1329, #1327).RET_CHECK on zero character/word counts, clamp thread pool size to 1024, fix swap-and-pop bug when removing empty normalized sentences, and fix NaN in Gumbel sampling / thread-local RNG re-seeding (Commits b3717da, 70ed5fc, a44ce92).b3717da).5d21631).nmt_nfkc: Normalize U+061C (ARABIC LETTER MARK) to space (#1331).442a7d7):
*AsSerializedProto / *AsImmutableProto variants, ImmutableSentencePieceText, ImmutableNBestSentencePieceText.SampleEncodeAndScore*, CalculateEntropy.SetVocabulary / ResetVocabulary / LoadVocabulary.OverrideNormalizerSpec, __getstate_for_pickle__.TrainerSpec fields (now marked [deprecated = true] and removed from spm_train CLI flags): accept_language, self_test_sample_size, mining_sentence_size, training_sentence_size, train_extremely_large_corpus, enable_differential_privacy, differential_privacy_noise_level, differential_privacy_clipping_threshold (Commits d29cf00, 470b7d5).SelfTestData from sentencepiece_model.proto (Commit 470b7d5).contrib/nlcodec fast BPE trainer and other dead code / unused build dependencies (Commits b6910fd, c91fbef).MODULE.bazel / BUILD.bazel covering the runtime, trainer, CLI binaries, shared libraries, and test suites; darts-clone, libsais, and abseil-cpp are taken from the Bazel Central Registry (#1310, #1311, #1312, #1313, closes #1280).c0232a9).protobuf-lite and builtin_pb (#1296, Commit 443fedc).testharness with GoogleTest / GoogleMock via CMake FetchContent (Commits 87ab362, c5ae4a9).protoc cross-compilation support (CMAKE_CROSSCOMPILING_EMULATOR), and self-contained sdist (Commits 6bcf11d, e9e7600, 16e057b, 4e5215e).a24b577, a705d0a).doc/auto_character_coverage.md, doc/sparse_l1_unigram.md, doc/bazel.md, and lite/README.md.options.md, special_symbols.md, C++ / normalization / CMake docs, and the tokenizer comparison cheat sheet for 0.2.3; README examples now use return_type (Commits ee1d3dd, 70ed5fc, cc5aab4).Your coding agent can read these notes before it upgrades. Set up the MCP server →
Deprecate Win32 Wheels: Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c).
This release contains major updates, including the migration of the Python wrapper to pybind11, native NumPy support, enhanced offset mapping, performance optimizations, security hardening, and internal code modernization.
528dc9e)..pyi files) and packaged PEP 561 marker to provide auto-completion and static type safety in IDEs (#1258, Commit 8b7f97b).parallel_encode and ThreadPool APIs, allowing Python users to tokenize single large documents across multiple CPU cores in parallel (#1254, Commit 3c37c39).12788ba).0a044ae).ValueError instead of silent failures or crashes (Commit 780618a).8db9878).SentencePieceText conversion, resulting in ~2x speedup (Commit 4b9231b).1cd0c59).darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files (#1224, Commit 6b2f05d).DecompileCharsMap (#1265, Commit 82b7c6d).CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs (Commit 66b2587).eb3ba49).ba57749).e476e5e).24fd00c).setup.py directly to the standard pypa/build PEP 517 frontend (#1167, Commit f300c7d).pyproject.toml (#1173).options.md and special_symbols.md with verified code examples.[ALL]: Security fix to address a heap overflow issue that could occur when using a model containing an invalid precompiled normalization model.
-DSPM_DISABLE_EMBEDDED_DATA=ON). This reduces the runtime size by approximately 1-2 MB. This mode is enabled to build python wheels. The rules are loaded as the data package.[ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample
N/A
[ALL] Fixes the NaN issues in unigram model training: https://github.com/google/sentencepiece/issues/851
N/A
N/A
Python 3.11 support (wheel packages for python 3.11 are available)
--pretokenization_delimiter) flagMigrated the C++ version from C++11 to C++17.
spm.__version__Improves the performance of unigram training
allow_whitespace_only_pieces mode.support to build sentencepiece with the external (official) abseil library.
added SetRandomGeneratorSeed function to set the seed value for random generator. This can allow to make reproducible sampling.
Fixed the regression bug around the flag --minloglevel
[Python] Added a feature to feed training data as Python's iterable object. https://github.com/google/sentencepiece/tree/master/python#training-withou
Renamed v0.1.9 to v0.1.90 because PyPI doesn't recognize 0.1.9 as the latest release.
Renamed v0.1.9 to v0.1.90 because PyPI doesn't recognize 0.1.9 as the latest release.
Support tf 1.5.1 2.0.0 2.0.1 2.1.0 and 2.2.0rc3
Support tf 1.15 and Python3.8 on Windows
Support tf 1.15 and Python3.8 on Windows
Use the official docker image to build tf_sentencepiece ops
Bug fix: fixed the behavior of is_unknown method in Python module.
Bug fix: fixed the behavior of is_unknown method in Python module.
Fix: support tensorflow 0.13.1
Fix: support tensorflow 0.13.1
Feature: Get rid of the dependency to external protobuf Feature: added (Encode|Decode)AsSerializedProto interface so Python module can get full access
Feature: Get rid of the dependency to external protobuf Feature: added (Encode|Decode)AsSerializedProto interface so Python module can get full access to the SentencePieceText proto including the byte offsets/aligments Feature: added --treat_whitespace_as_suffix option to make _ be a suffix of word. Feature: Added normalization rules to remove control characters in the default nmt_* normalizers Minor fix: simplify the error messager Minor fix: do not emit full source path in LOG(INFO)
For more detail: https://github.com/google/sentencepiece/compare/v0.1.7...v0.1.8
Deprecated: --mining_sentence_size and --training_sentence_size. Load all sentences by default. --input_sentence_size can be specified to limit the se…
Deprecated: --mining_sentence_size and --training_sentence_size. Load all sentences by default. --input_sentence_size can be specified to limit the sentences to be loaded
Feature: added --unk_piece/--bos_piece/--eos_piece/--pad_piece flags to change the surface representations of these special symbols.
Bug fix: added third_party directory for cmake's subdirectory.
For more detail: https://github.com/google/sentencepiece/compare/v0.1.6...v0.1.7
Bug fix: do not apply normalization to the user-defined-symbols.
For more detail: https://github.com/google/sentencepiece/compare/v0.1.5...master
Nothing published for this version
Initial SentencePiece releases
Initial SentencePiece releases
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per quarter.