dart_sentencepiece_tokenizer

What's New

GitHub Actions CI Pipeline (#19)

Automated continuous integration is now configured for the project:

• Analyze job — Enforces dart format consistency and dart analyze --fatal-infos with zero tolerance for warnings.
• Test job — Runs the full test suite across a matrix of Dart stable and Dart 3.10.7 (minimum supported SDK version).
• Minimal permissions (contents: read) and concurrency groups to cancel stale runs.

Improvements

• Code formatting — Applied dart format to 23 source files for consistent code style across the codebase.
• Static analysis cleanup — Resolved all dart analyze --fatal-infos issues:
  • Removed deprecated avoid_returning_null_for_future lint rule.
  • Added curly braces to if statements, const constructors, and final local variables where required.
• Documentation (#21) — Added inline comments clarifying google/sentencepiece proto spec compliance for default token IDs (unkId=0, bosId=1, eosId=2, padId=-1).

Notes

This is a maintenance release with no API changes or breaking changes. Focus: CI infrastructure, code hygiene, and documentation clarity.

Full Changelog: v1.3.1...v1.3.2

Added

• GitHub Actions CI Pipeline (#9, #19)
  • analyze job - dart format --set-exit-if-changed and dart analyze --fatal-infos
  • test job - Matrix testing across Dart stable and 3.10.7 (minimum supported version)
  • Minimal permissions (contents: read) and concurrency group to cancel stale runs

Changed

• Applied dart format to 23 files for consistent code style
• Resolved all dart analyze --fatal-infos issues
  • Removed deprecated avoid_returning_null_for_future lint rule
  • Added curly braces to if statements, const constructors, final local variables
• Added documentation comments clarifying google/sentencepiece proto spec compliance for default token IDs (unkId=0, bosId=1, eosId=2, padId=-1) (#20)

Load HuggingFace tokenizers directly from tokenizer.json — no conversion step required.

What's New

HuggingFace tokenizer.json Format Support

You can now load any HuggingFace tokenizer.json file without converting it to SentencePiece .model format first. This makes it straightforward to use tokenizers published on the HuggingFace Hub.

// Load from file
final tokenizer = await HuggingFaceTokenizerLoader.fromJsonFile('tokenizer.json');

// Load from a pre-parsed map
final tokenizer = HuggingFaceTokenizerLoader.fromMap(jsonMap);

// Auto-detection — works transparently with TokenizerJsonLoader
final tokenizer = await TokenizerJsonLoader.fromJsonFile('tokenizer.json');

Supported model types:

• Unigram — Llama, T5, ALBERT, XLNet, and other Unigram-based models
• BPE — Gemma, GPT-2, RoBERTa, and other BPE-based models

Automatic configuration inference:

• Special tokens (unk, bos, eos, pad) are detected from the added_tokens section
• Normalizer settings (addDummyPrefix, escapeWhitespaces) are inferred from the HuggingFace normalizer config
• Post-processor flags (addBosToken, addEosToken) are parsed from TemplateProcessing
• Byte fallback behavior is detected from the decoder configuration
• Tokens beyond the base vocabulary are handled automatically

Format detection:

TokenizerJsonLoader.isHuggingFaceFormat() lets you check whether a JSON map uses the HuggingFace format. When you call TokenizerJsonLoader.fromJsonFile(), HuggingFace format is detected and delegated automatically — no code changes needed if you already use TokenizerJsonLoader.

Install / Upgrade

dependencies:
  dart_sentencepiece_tokenizer: ^1.3.1

Full Changelog: https://github.com/brody-0125/dart_sentencepiece_tokenizer/blob/develop/CHANGELOG.md

Added

• HuggingFace tokenizer.json Format Support
  • HuggingFaceTokenizerLoader class for loading HuggingFace tokenizer.json files directly
    • fromJsonString() / fromMap() - Parse from JSON string or pre-parsed map
    • fromJsonFile() / fromJsonFileSync() - Load from file (async/sync)
  • Supports both Unigram (Llama) and BPE (Gemma) model types
  • Automatic detection of special tokens (unk, bos, eos, pad) from added_tokens section
  • Normalizer settings inference (addDummyPrefix, escapeWhitespaces) from HuggingFace normalizer config
  • Post-processor configuration parsing (addBosToken, addEosToken) from TemplateProcessing
  • Byte fallback detection from decoder configuration
  • Added tokens handling beyond base vocabulary
  • TokenizerJsonLoader.isHuggingFaceFormat() - Helper to detect HuggingFace format
  • Auto-detection in TokenizerJsonLoader - Automatically delegates to HuggingFaceTokenizerLoader when HuggingFace format is detected

What's Changed

• feat: add HuggingFace TextStreamer compatible streaming API by @brody-0125 in #8

Full Changelog: 1.2.2...1.3.0

Added

• Streaming API (HuggingFace TextStreamer Compatible)
  • BaseStreamer - Abstract interface for streaming token decoders with put() and end() methods
  • TextStreamer - HuggingFace TextStreamer-compatible class for real-time LLM token decoding
    • put(int tokenId) - Add tokens as they are generated
    • end() - Signal end of generation and flush remaining content
    • onFinalizedText callback for custom text handling
    • skipSpecialTokens option to filter BOS/EOS/PAD tokens
    • skipPrompt option to skip initial prompt tokens
    • promptLength option to skip multiple prompt tokens
    • Word boundary heuristics for clean text emission (newlines, CJK, spaces)
  • SentencePieceTokenizer.createTextStreamer() - Factory for TextStreamer
  • SentencePieceTokenizer.decodeStream() - Stream-based token decoding
  • SentencePieceTokenizer.decodeWithCallback() - Callback-based token decoding

Usage Examples

TextStreamer (HuggingFace-compatible):

final streamer = tokenizer.createTextStreamer();
for (final id in llmOutput) {
  streamer.put(id);
}
streamer.end();

// With custom callback
final streamer = tokenizer.createTextStreamer(
  onFinalizedText: (text, {required streamEnd}) {
    myTextController.append(text);
    if (streamEnd) myTextController.complete();
  },
);

Stream-based decoding:

final textStream = tokenizer.decodeStream(llmTokenStream);
await for (final chunk in textStream) {
  stdout.write(chunk);
}

Callback-based decoding:

tokenizer.decodeWithCallback(
  tokenIds,
  (chunk) => stdout.write(chunk),
);

What's Changed

• feat: optimize memory usage and refactor tests for v1.2.2 by @brody-0125 in #7

Full Changelog: 1.2.1...1.2.2

Changed

• Extracted duplicate surrogate pair decoding logic in Trie into shared _decodeCodePoint helper
• Cached computed sequenceIds in Encoding to avoid O(n) recomputation on repeated access
• Added merge cache size limit (10,000 entries) to BpeAlgorithm and BpeAlgorithmOptimized to prevent unbounded memory growth
• Replaced manual loops with fillRange for padding initialization in Encoding.withPadding()

What's Changed

• feat: add JSON Serialization API, Dynamic Token Addition API and Optimized BPE Algorithm by @brody-0125 in #5

Full Changelog: 1.2.0...1.2.1

Changed

• Optimized batch addTokens() to use single typed array allocation instead of per-token expansion (O(N) instead of O(N²))
• Added input validation and defensive error handling in JSON deserialization (TokenizerJsonLoader)
• Consolidated duplicate _kMaxInputLength constant declarations

Added

• JSON Serialization - HuggingFace-compatible tokenizer.json format

  • toJson() - Serialize tokenizer to JSON string
  • saveToJson() / saveToJsonSync() - Save to file
  • TokenizerJsonLoader.fromJsonString() - Load from JSON string
  • TokenizerJsonLoader.fromJsonFile() / fromJsonFileSync() - Load from file

• Dynamic Token Addition API

  • addTokens(List<String>) - Add new tokens to vocabulary
  • addSpecialTokens(Map<String, String>) - Add special tokens (pad, mask, etc.)
  • getAddedVocab() - Get map of dynamically added tokens
  • isAddedToken(String) - Check if token was added dynamically
  • getVocab({withAddedTokens}) - Get full vocabulary as Map<String, int>

• HuggingFace-compatible Methods

  • tokenize(String) - Returns List<String> of tokens
  • tokenizeBatch(List<String>) - Batch tokenization

• Optimized BPE Algorithm (BpeAlgorithmOptimized)

  • O(n log n) complexity using priority queue (heap)
  • ~35% faster than original algorithm on medium-length text

Changed

• SpVocabulary now uses growable list for dynamic token addition support

What's Changed

• feat: improve BPE Algorithm by @brody-0125 in #1
• feat: improve BPE Algorithm (#1) by @brody-0125 in #2
• feat: add JSON Serialization API, Dynamic Token Addition API and Optimized BPE Algorithm by @brody-0125 in #3

Full Changelog: 1.0.0...1.2.0

Added

• Input length validation (max 500,000 characters) to prevent OOM
• Example usage file (example/example.dart)

Changed

• Improved BPE algorithm efficiency
• Enhanced error messages for input validation

Full Changelog: https://github.com/brody-0125/dart_sentencepiece_tokenizer/commits/1.0.0

Added

• Initial release of dart_sentencepiece_tokenizer
• Pure Dart implementation with zero external dependencies
• Support for BPE (Byte Pair Encoding) algorithm used by Gemma models
• Support for Unigram algorithm used by Llama models
• Viterbi algorithm implementation for optimal Unigram segmentation
• Byte fallback support for handling unknΩown characters
• Unicode-aware Trie for efficient vocabulary lookup
• Memory-efficient typed arrays (Int32List, Uint8List) for encodings

Features

• SentencePieceTokenizer - Main tokenizer class

  • fromBytes() - Load from protobuf bytes
  • fromModelFile() / fromModelFileSync() - Load from .model file
  • encode() - Encode single text
  • encodeBatch() - Encode multiple texts
  • encodeBatchParallel() - Parallel batch encoding using Isolates
  • encodePair() - Encode text pairs for sequence classification
  • encodePairBatch() - Batch encode text pairs
  • decode() / decodeBatch() - Decode token IDs back to text

• Encoding class with:

  • ids - Token IDs (Int32List)
  • tokens - Token strings
  • typeIds - Segment type IDs (Uint8List)
  • attentionMask - Attention mask (Uint8List)
  • specialTokensMask - Special token indicators (Uint8List)
  • offsets - Character offsets for each token
  • withPadding() / withTruncation() - Post-processing methods
  • truncatePair() - Static method for pair truncation

• Predefined configurations:

  • SentencePieceConfig.llama - Llama-style (BOS only)
  • SentencePieceConfig.gemma - Gemma-style (BOS + EOS)

• Truncation strategies:

  • longestFirst - Truncate longer sequence first
  • onlyFirst - Only truncate first sequence
  • onlySecond - Only truncate second sequence
  • doNotTruncate - No truncation

• Padding options:

  • Left/right padding direction
  • Fixed length or pad to longest
  • Pad to multiple of N

Performance

• Efficient Trie-based vocabulary lookup
• Memory-optimized typed arrays reduce memory usage by ~78%
• Parallel batch processing with configurable chunk size
• Lazy evaluation where possible

Compatibility

• Dart SDK 3.10.7+
• Compatible with Llama, Gemma, and other SentencePiece models
• HuggingFace-compatible API design