betto_lexical

Stemmer previously wired up only English, throwing ArgumentError for
every other locale, even though the underlying snowball_stemmer
dependency already implements 28 languages. Extend the factory to map
all of them (excluding the generic porter variant, not a distinct
language). Each new mapping was verified against the actual stemmer
output for a representative inflected word per language, not just
type-checked.

Also bump the betto_icu dependency to 0.1.0-dev.2 and re-export its
new OffsetTokenizer/TokenSpan types, needed by kmdb's vault chunker
for position-aware tokenisation.

Co-Authored-By: Claude Sonnet 5 [email protected]

Stemming

• Stemmer now supports 28 languages, up from just English: Arabic, Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Tamil, Turkish, and Yiddish — every language package:snowball_stemmer implements (excluding its generic porter variant, an alternate English algorithm rather than a distinct language). Unsupported language codes still throw ArgumentError, as before.

Tokenization

• Re-exports OffsetTokenizer and TokenSpan from betto_icu (now pinned to ^0.1.0-dev.2) — a Tokenizer that also reports each token's character offsets in the source text. Implemented by IcuTokenizer and RegExpTokenizer.

Co-Authored-By: Claude Sonnet 4.6 [email protected]

Initial development release.

Tokenization

• createDefaultTokenizer() returns the best tokenizer for the current platform at compile time — no runtime Platform checks.
• IcuTokenizer — UAX #29 word segmentation via the system ICU library (native: macOS, Linux, Windows, Android, iOS).
• BrowserTokenizer — word segmentation via Intl.Segmenter (web).
• RegExpTokenizer — lightweight Latin-script tokenizer in pure Dart, available on all platforms.

Stemming

• Stemmer — Snowball-based stemmer. Construct with a Locale and call stem(word) to reduce tokens to their base form. English (en) supported.

Stop words

• getStopWords(Locale) — returns a Stopwords enum value containing the stop-word set for the given locale. Throws ArgumentError for unsupported language codes.
• 58 languages sourced from stopwords-iso.