tiktoken-rs

Summary

This release backports OpenAI tiktoken 0.13.0 into tiktoken-rs. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.

For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level CoreBPE encoding methods directly should review the breaking changes below.

What Changed

• Backported the vendored OpenAI tiktoken Rust core from 0.9.0 to 0.13.0.
• Added the upstream large-piece BPE merge path. Functionally, this improves behavior for very large or repetitive inputs that previously stressed the merge algorithm.
• Changed CoreBPE::encode to return Result<(Vec<Rank>, usize), EncodeError>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.
• Updated encode_as and count to return Result because they call encode.
• Re-exported EncodeError so callers can handle encode failures directly.
• Aligned the vendored core with Rust 2024 and raised the crate MSRV to Rust 1.85.
• Synced model-to-tokenizer mappings with upstream tiktoken 0.13.0 while keeping local extra prefixes isolated.
• Hardened asset downloads with SHA-256 checks and a repo-root-aware asset path.

Breaking Changes

If your code calls CoreBPE::encode, unwrap or propagate the result before using the tokens:

let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?;

The generic helpers changed similarly:

let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?;
let token_count = bpe.count(text, &allowed)?;

encode_ordinary, encode_ordinary_as, encode_with_special_tokens, and count_ordinary remain infallible.

Projects must now build with Rust 1.85 or newer.

Practical Impact

• Applications processing long repeated text should see more robust tokenization behavior.
• Code that only uses helpers like get_chat_completion_max_tokens, get_text_completion_max_tokens, bpe_for_model, or singleton tokenizer constructors should not need call-site changes.
• Code using low-level CoreBPE::encode, encode_as, or count needs a small migration to handle Result.

Links

• PR: #164
• Upstream tiktoken 0.13.0: https://github.com/openai/tiktoken/releases/tag/0.13.0
• Full changelog: v0.11.0...v0.12.0

What's Changed

Features

• Add gpt-5.4 support, fix context sizes for o1-mini/chatgpt-4o/gpt-4.5 by @zurawiki in #156
• Add generic token type APIs (FromRank trait) by @zurawiki in #157
• Add gpt-5.2/5.3/codex model support by @zurawiki in #159
• Return Option from get_context_size, improve error messages, bump to 0.11.0 by @zurawiki in #160

Other Changes

• Use GitHub auto-generated release notes by @zurawiki in #158

Full Changelog: v0.10.0...v0.11.0

What's Changed

Features

• Use singleton BPE in num_tokens_from_messages and get_completion_max_tokens by @zurawiki in #140
• Count function_call tokens in num_tokens_from_messages by @zurawiki in #145
• Change decode() to accept &[Rank] and make decode_bytes() public by @zurawiki in #146
• Return &'static CoreBPE from get_bpe_from_model and get_bpe_from_tokenizer by @zurawiki in #147
• Rename misleading API functions, deprecate old names by @zurawiki in #154
• Add count-only token counting APIs by @zurawiki in #153

Bug Fixes

• Fix should-publish version check failing in CI by @zurawiki in #143
• Fix gpt-3.5 model version check in num_tokens_from_messages by @zurawiki in #144

Dependencies

• chore(deps): update extractions/setup-just action to v3 by @renovate[bot] in #127
• chore(deps): update actions/checkout action to v5 by @renovate[bot] in #126
• chore(deps): pin swatinem/rust-cache action to 98c8021 by @renovate[bot] in #125
• chore(deps): update actions/checkout v6, pin rust-toolchain, update create-gh-release v1.10.0 by @zurawiki in #142
• chore(deps): update extractions/setup-just action to v4 by @renovate[bot] in #150
• chore(deps): update swatinem/rust-cache action to v2.9.1 - autoclosed by @renovate[bot] in #149
• Sync regex patterns with upstream tiktoken, bump deps, port tests by @zurawiki in #151
• Bump async-openai to 0.34, add tool_calls/refusal token counting, update docs by @zurawiki in #152

Other Changes

• Fix release CI and harden workflow permissions by @zurawiki in #141
• Add release.yml for auto-generated release notes by @zurawiki in #148
• Release v0.10.0 by @zurawiki in #155

Full Changelog: v0.9.1...v0.10.0

What's Changed

• Support o-series models, gpt-5, and add o200k_harmony tokenizer for gpt-oss in #123

Full Changelog: v0.7.0...v0.9.1

What's Changed

• Remove need for Arc<Mutex<>> on singletons by @benbrandt in #107
• Update to latest tiktoken + new model support by @benbrandt in #106
• Dependency and build/CI updates

New Contributors

• @benbrandt made their first contribution in #107

Full Changelog: v0.6.0...v0.7.0

Minor version release signifies a breaking change in 0.x

What's Changed

• Add support for chatgpt-4o-latest by @Congyuwang in #85
• Refactor internals to make future updates to tiktoken easier to merge
• Do not expose tiktoken internal modules and functions
• Update dependencies

New Contributors

• @Congyuwang made their first contribution in #85

Full Changelog: v0.5.9...v0.6.0

What's Changed

• Add gpt-4-0125 and gpt-3.5-turbo-1106 context sizes by @mikaelsouza in #59
• Add gpt-3.5-turbo-0125 by @mikaelsouza in #60
• Update README.md by @imalsogreg in #61
• Add support: o200k_base tokenizer. by @CaffeeLake in #63

New Contributors

• @mikaelsouza made their first contribution in #59
• @imalsogreg made their first contribution in #61
• @CaffeeLake made their first contribution in #63

Full Changelog: v0.5.8...v0.5.9

What's Changed

• Add tokenizer prefixes for fine-tuned models by @jbgriesner in #53

New Contributors

• @jbgriesner made their first contribution in #53

Full Changelog: v0.5.7...v0.5.8

What's Changed

• Add gpt-4-1106-preview model by @JosephTLyons in #49

New Contributors

• @JosephTLyons made their first contribution in #49

Full Changelog: v0.5.6...v0.5.7