NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
crates.io · #2314 most downloaded on crates.io
Library for encoding and decoding with the tiktoken library in Rust
Last release 14 days ago
24 Sep 2026
Ships fairly regularly
a new release about every 4 months
Some releases are documented
notes for 10 of 36 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
36 releases · first in 2023
One column per quarter.
A maintenance release that refreshes upstream tiktoken and restores compatibility with the declared Rust 1.85 minimum. Existing public function signat
A maintenance release that refreshes upstream tiktoken and restores compatibility with the declared Rust 1.85 minimum. Existing public function signatures are unchanged.
gpt-6* names. This adds context metadata only; GPT-6 tokenizer lookup and token-budget helpers remain unsupported. (#174)lazy_static with the standard library's LazyLock. (#165)The minimum supported Rust version remains 1.85.
Thanks to @Expyron for their first contribution in #165.
Full changelog: v0.12.0...v0.12.1
Users who call lower-level CoreBPE encoding methods directly should review the breaking changes below.
This release backports OpenAI tiktoken 0.13.0 into tiktoken-rs. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.
For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level CoreBPE encoding methods directly should review the breaking changes below.
tiktoken Rust core from 0.9.0 to 0.13.0.CoreBPE::encode to return Result<(Vec<Rank>, usize), EncodeError>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.encode_as and count to return Result because they call encode.EncodeError so callers can handle encode failures directly.tiktoken 0.13.0 while keeping local extra prefixes isolated.If your code calls CoreBPE::encode, unwrap or propagate the result before using the tokens:
let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?;The generic helpers changed similarly:
let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?;
let token_count = bpe.count(text, &allowed)?;encode_ordinary, encode_ordinary_as, encode_with_special_tokens, and count_ordinary remain infallible.
Projects must now build with Rust 1.85 or newer.
get_chat_completion_max_tokens, get_text_completion_max_tokens, bpe_for_model, or singleton tokenizer constructors should not need call-site changes.CoreBPE::encode, encode_as, or count needs a small migration to handle Result.tiktoken 0.13.0: https://github.com/openai/tiktoken/releases/tag/0.13.0Add gpt-5.4 support, fix context sizes for o1-mini/chatgpt-4o/gpt-4.5 by @zurawiki in #156
Full Changelog: v0.10.0...v0.11.0
Rename misleading API functions, deprecate old names by @zurawiki in #154
Full Changelog: v0.9.1...v0.10.0
Support o-series models, gpt-5, and add o200k_harmony tokenizer for gpt-oss in #123
Full Changelog: v0.7.0...v0.9.1
Nothing published for this version
Remove need for Arc<Mutex<>> on singletons by @benbrandt in #107
Full Changelog: v0.6.0...v0.7.0
Minor version release signifies a breaking change in 0.x
Minor version release signifies a breaking change in 0.x
Full Changelog: v0.5.9...v0.6.0
Add gpt-4-0125 and gpt-3.5-turbo-1106 context sizes by @mikaelsouza in #59
gpt-4-0125 and gpt-3.5-turbo-1106 context sizes by @mikaelsouza in #59gpt-3.5-turbo-0125 by @mikaelsouza in #60Full Changelog: v0.5.8...v0.5.9
Add tokenizer prefixes for fine-tuned models by @jbgriesner in #53
Full Changelog: v0.5.7...v0.5.8
Add gpt-4-1106-preview model by @JosephTLyons in #49
gpt-4-1106-preview model by @JosephTLyons in #49Full Changelog: v0.5.6...v0.5.7
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →