NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
pub.dev
pub.dev
Unicode text tokenization for Dart — Tokenizer interface, IcuTokenizer (system ICU FFI, UAX #29), and RegExpTokenizer (pure Dart, Latin fallback).
Latest stable
0.1.0
Last release 3 months ago
24 Aug 2026
Ships on a steady schedule
a new release about every 5 weeks
Nearly every release is documented
notes for 1 of 1 stable releases
Nothing withdrawn
no release was ever pulled
5 months old
3 releases · first in 2026
Drag across the ruler to compare, or tap a release to open it.
Prep for 0.1.0 release. Moved to Dart 3.13
Prep for 0.1.0 release. Moved to Dart 3.13
Updated license - original text seemed incorrect
Updated license - original text seemed incorrect
OffsetTokenizer — a Tokenizer that also reports each token's character
offsets in the source text via tokeniseSpans(), returning TokenSpan
(text, start, end). Implemented by IcuTokenizer and RegExpTokenizer
— position data is a natural byproduct of both algorithms' underlying
implementation. Not implemented by BrowserTokenizer. tokenise() on both
implementations is now defined in terms of tokeniseSpans(), with no change
in behaviour.Your coding agent can read these notes before it upgrades. Set up the MCP server →
One column per month.
Tokenizer — abstract interface for text segmentation. Implementations return only word-like tokens (letters and digits); punctuation and whitespace ar
Initial release.
Tokenizer — abstract interface for text segmentation. Implementations return
only word-like tokens (letters and digits); punctuation and whitespace are
discarded.
IcuTokenizer — UAX #29 word-boundary segmentation via the system ICU library
(FFI). Handles non-Latin scripts: CJK, Arabic, Thai, Devanagari, and more.
Supported platforms:
| Platform | Library |
|---|---|
| macOS / iOS | libicucore.dylib (ships with the OS) |
| Android | libicuuc.so (NDK) |
| Linux | libicuuc.so.NN (install libicu-dev/icu) |
| Windows | icu.dll (Windows 10+) |
Probes for ICU symbol renaming (e.g. ubrk_open_76 on Debian Trixie+ and
Fedora) so no manual configuration is needed.
Uses RegExp-based span classification instead of ubrk_getRuleStatus() for
portability across all supported platforms, including Apple's libicucore
which does not export UAX #29 rule-status tags.
RegExpTokenizer — pure-Dart tokenizer using RegExp with Unicode property
escapes (\p{L}, \p{N}). Zero FFI dependencies; suitable for English prose
and common technical identifiers (mTLS, 0x8004210B, etc.). Works on every
platform, including web.
BrowserTokenizer — UAX #29 word-boundary segmentation via the browser's
Intl.Segmenter API (dart:js_interop). Web-only; zero bundle cost. Requires
Chrome 87+, Firefox 125+, or Safari 16.4+. On native targets the conditional
import resolves to a stub that throws UnsupportedError.
Conditional export pattern in lib/betto_icu.dart: the correct IcuTokenizer
implementation (real FFI or stub) and BrowserTokenizer implementation (real
JS interop or stub) are selected at compile time via dart.library.ffi and
dart.library.js_interop.
IcuTokenizer.forPlatform(String platform) — named constructor that loads the
ICU library for an explicit platform string. Enables tests to exercise each
library-loading branch without requiring the native OS.
bin/tokenize.dart — command-line tool that tokenizes a string with either or
both tokenizers (--icu, --regexp).
Flutter integration test app (integration_test_app/) — runs the full
contract and UAX #29 test suite on Android and iOS via
package:integration_test.