dart_sentencepiece_tokenizer

Added

• Expanded Hugging Face tokenizer.json compatibility for SentencePiece-based BPE and Unigram tokenizers (#31).
  • Direct and nested Precompiled normalizers with ordered Replace operations.
  • WhitespaceSplit and Metaspace pre-tokenizer pipelines, including current and legacy configuration fields.
  • TemplateProcessing special tokens with declared BOS/EOS IDs and type IDs.
  • Tokenizer-level padding and truncation settings, including post-processor special-token preservation.
  • Unigram consecutive unknown-token fusion (fuse_unk) and declared added-token IDs.
• Added acceptance coverage for official Hugging Face tokenizer.json downloads and MiniLM golden IDs.

Changed

• Exported SpAddedToken for tokenizer JSON metadata interoperability.
• Added configurable padding type IDs and special-token-aware truncation to Encoding.

Fixed

• Malformed precompiled base64 and charsmap data now fail explicitly instead of being silently ignored.
• Corrected SentencePiece protobuf int32 decoding without ZigZag conversion.
• Preserved BOS/EOS tokens when truncating long single, batch, and parallel-batch encodings.

Fixed

• Fixed loading of Hugging Face BPE merges entries represented as [left, right] pairs (#26).

Changed

• Expanded CI matrix coverage for Dart 3.10.7, latest Dart 3.10–3.13 patch releases, and stable.
• Added contributor acknowledgements and a link to the GitHub contributors graph in the README.

Added

• GitHub Actions CI Pipeline (#9, #19)
  • analyze job - dart format --set-exit-if-changed and dart analyze --fatal-infos
  • test job - Matrix testing across Dart stable and 3.10.7 (minimum supported version)
  • Minimal permissions (contents: read) and concurrency group to cancel stale runs

Changed

• Applied dart format to 23 files for consistent code style
• Resolved all dart analyze --fatal-infos issues
  • Removed deprecated avoid_returning_null_for_future lint rule
  • Added curly braces to if statements, const constructors, final local variables
• Added documentation comments clarifying google/sentencepiece proto spec compliance for default token IDs (unkId=0, bosId=1, eosId=2, padId=-1) (#20)

Added

• HuggingFace tokenizer.json Format Support
  • HuggingFaceTokenizerLoader class for loading HuggingFace tokenizer.json files directly
    • fromJsonString() / fromMap() - Parse from JSON string or pre-parsed map
    • fromJsonFile() / fromJsonFileSync() - Load from file (async/sync)
  • Supports both Unigram (Llama) and BPE (Gemma) model types
  • Automatic detection of special tokens (unk, bos, eos, pad) from added_tokens section
  • Normalizer settings inference (addDummyPrefix, escapeWhitespaces) from HuggingFace normalizer config
  • Post-processor configuration parsing (addBosToken, addEosToken) from TemplateProcessing
  • Byte fallback detection from decoder configuration
  • Added tokens handling beyond base vocabulary
  • TokenizerJsonLoader.isHuggingFaceFormat() - Helper to detect HuggingFace format
  • Auto-detection in TokenizerJsonLoader - Automatically delegates to HuggingFaceTokenizerLoader when HuggingFace format is detected

Added

• Streaming API (HuggingFace TextStreamer Compatible)
  • BaseStreamer - Abstract interface for streaming token decoders with put() and end() methods
  • TextStreamer - HuggingFace TextStreamer-compatible class for real-time LLM token decoding
    • put(int tokenId) - Add tokens as they are generated
    • end() - Signal end of generation and flush remaining content
    • onFinalizedText callback for custom text handling
    • skipSpecialTokens option to filter BOS/EOS/PAD tokens
    • skipPrompt option to skip initial prompt tokens
    • promptLength option to skip multiple prompt tokens
    • Word boundary heuristics for clean text emission (newlines, CJK, spaces)
  • SentencePieceTokenizer.createTextStreamer() - Factory for TextStreamer
  • SentencePieceTokenizer.decodeStream() - Stream-based token decoding
  • SentencePieceTokenizer.decodeWithCallback() - Callback-based token decoding

Usage Examples

TextStreamer (HuggingFace-compatible):

final streamer = tokenizer.createTextStreamer();
for (final id in llmOutput) {
  streamer.put(id);
}
streamer.end();

// With custom callback
final streamer = tokenizer.createTextStreamer(
  onFinalizedText: (text, {required streamEnd}) {
    myTextController.append(text);
    if (streamEnd) myTextController.complete();
  },
);

Stream-based decoding:

final textStream = tokenizer.decodeStream(llmTokenStream);
await for (final chunk in textStream) {
  stdout.write(chunk);
}

Callback-based decoding:

tokenizer.decodeWithCallback(
  tokenIds,
  (chunk) => stdout.write(chunk),
);

Changed

• Extracted duplicate surrogate pair decoding logic in Trie into shared _decodeCodePoint helper
• Cached computed sequenceIds in Encoding to avoid O(n) recomputation on repeated access
• Added merge cache size limit (10,000 entries) to BpeAlgorithm and BpeAlgorithmOptimized to prevent unbounded memory growth
• Replaced manual loops with fillRange for padding initialization in Encoding.withPadding()

Changed

• Optimized batch addTokens() to use single typed array allocation instead of per-token expansion (O(N) instead of O(N²))
• Added input validation and defensive error handling in JSON deserialization (TokenizerJsonLoader)
• Consolidated duplicate _kMaxInputLength constant declarations

Added

• Input length validation (max 500,000 characters) to prevent OOM
• Example usage file (example/example.dart)

Changed

• Improved BPE algorithm efficiency
• Enhanced error messages for input validation

Added

• Initial release of dart_sentencepiece_tokenizer
• Pure Dart implementation with zero external dependencies
• Support for BPE (Byte Pair Encoding) algorithm used by Gemma models
• Support for Unigram algorithm used by Llama models
• Viterbi algorithm implementation for optimal Unigram segmentation
• Byte fallback support for handling unknΩown characters
• Unicode-aware Trie for efficient vocabulary lookup
• Memory-efficient typed arrays (Int32List, Uint8List) for encodings

Features

• SentencePieceTokenizer - Main tokenizer class

  • fromBytes() - Load from protobuf bytes
  • fromModelFile() / fromModelFileSync() - Load from .model file
  • encode() - Encode single text
  • encodeBatch() - Encode multiple texts
  • encodeBatchParallel() - Parallel batch encoding using Isolates
  • encodePair() - Encode text pairs for sequence classification
  • encodePairBatch() - Batch encode text pairs
  • decode() / decodeBatch() - Decode token IDs back to text

• Encoding class with:

  • ids - Token IDs (Int32List)
  • tokens - Token strings
  • typeIds - Segment type IDs (Uint8List)
  • attentionMask - Attention mask (Uint8List)
  • specialTokensMask - Special token indicators (Uint8List)
  • offsets - Character offsets for each token
  • withPadding() / withTruncation() - Post-processing methods
  • truncatePair() - Static method for pair truncation

• Predefined configurations:

  • SentencePieceConfig.llama - Llama-style (BOS only)
  • SentencePieceConfig.gemma - Gemma-style (BOS + EOS)

• Truncation strategies:

  • longestFirst - Truncate longer sequence first
  • onlyFirst - Only truncate first sequence
  • onlySecond - Only truncate second sequence
  • doNotTruncate - No truncation

• Padding options:

  • Left/right padding direction
  • Fixed length or pad to longest
  • Pad to multiple of N

Performance

• Efficient Trie-based vocabulary lookup
• Memory-optimized typed arrays reduce memory usage by ~78%
• Parallel batch processing with configurable chunk size
• Lazy evaluation where possible

Compatibility

• Dart SDK 3.10.7+
• Compatible with Llama, Gemma, and other SentencePiece models
• HuggingFace-compatible API design