NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #155 most downloaded on PyPI
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Last release 1 months ago
17 Aug 2026
Ships fairly regularly
a new release about every 3 months
Nearly every release is documented
notes for 20 of 20 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
20 releases · first in 2022
One column per quarter.
Support looking up more GPT-5 series models
This project publishes no release notes. Between 0.13.0 and 0.14.0 there were 8 commits, 7 of them substantive:
Update fancy-regex for significantly increased performance
pyo3blobfileThis project publishes no release notes. Between 0.12.0 and 0.13.0 there were 4 commits, 4 of them substantive:
Support for free-threaded Python
pyo3 and rustc-hashblobfile for reading local filesgpt-5 model identifierThis project publishes no release notes. Between 0.11.0 and 0.12.0 there were 6 commits, 4 of them substantive:
Fix special token handling in encode_to_numpy
GPT-5pyo3encode_to_numpyThis project publishes no release notes. Between 0.9.0 and 0.11.0 there were 4 commits, 4 of them substantive:
- Support for newer models - Improvements to private APIs
Better error messages when loading invalid vocabulary files
o1 and o3 modelsThis project publishes no release notes. Between 0.8.0 and 0.9.0 there were 6 commits, 6 of them substantive:
Support for o1- and chatgpt-4o- models
o1- and chatgpt-4o- modelsget_encoding__version__ attributepyo3, regex, fancy-regexThis project publishes no release notes. Between 0.7.0 and 0.8.0 there were 5 commits, 5 of them substantive:
- Support for gpt-4o - Performance improvements
gpt-4oThis project publishes no release notes. Between 0.6.0 and 0.7.0 there were 7 commits, 7 of them substantive:
Optimise regular expressions for a 20% performance improvement, thanks to @paplorinc!
text-embedding-3-* models to encoding_for_modelEncoding objects. Registered Encoding will be pickled by referenceThank you to @paplorinc, @mdwelsh, @Praneet460!
This project publishes no release notes. Between 0.5.2 and 0.6.0 there were 7 commits, 7 of them substantive:
Update version of PyO3 to allow multiple imports
This project publishes no release notes. Between 0.5.1 and 0.5.2 there were 2 commits, 2 of them substantive:
Add encoding_name_for_model, undo some renames to variables that are implementation details
encoding_name_for_model, undo some renames to variables that are implementation detailsThis project publishes no release notes. Between 0.5.0 and 0.5.1 there were 2 commits, 2 of them substantive:
<|endoftext|> with constant (#186)Add tiktoken._educational submodule to better document how byte pair encoding works
tiktoken._educational submodule to better document how byte pair encoding worksencoding_for_model knows about several new modelsdecode_with_offetsAdd decode_batch and decode_bytes_batch
decode_batch and decode_bytes_batchtiktoken will now make a best effort attempt to replace surrogate pairs with the corresponding Unicode character and will replace lone surrogates with
tiktoken will now make a best effort attempt to replace surrogate pairs with the corresponding
Unicode character and will replace lone surrogates with the Unicode replacement character.- Add encoding for GPT-4
Make blobfile an optional dependency
blobfile an optional dependencyThank you to @messense for the environment variable that makes cargo not OOM under emulation!
Improve performance by 5-20%; thank you to @nistath!
gpt-3.5-turbo models to encoding_for_modelencoding_for_model to better support future model versionsAdd tiktoken.encoding_for_model to get the encoding for a specific model
tiktoken.encoding_for_model to get the encoding for a specific modelThank you to @fritzo, @arvid220u, @khanhvu207, @henriktorget for various small corrections
Avoid use of blobfile for public files
blobfile for public files- Initial release
Your coding agent can read these notes before it upgrades. Set up the MCP server →