bert_tokenizer

Resolved regex syntax errors by utilizing raw triple quotes for the punctuation character class.

Enforced strict WordPiece standard (entire word becomes [UNK] if any subword fails).

Added toLowerCase parameter for cased/uncased model flexibility.

Added pre-tokenization stripping of invisible control characters.

Robust line ending handling (\r\n and \n) for cross-platform vocab loading.

Fixed Truncation Crash: Rewrote the sequence length handling in prepareNerInput to prevent unhandled StateError exceptions when maxLength is set to small values (like 0 or 1).

Preserved Special Tokens: Adjusted the truncation logic to trim the raw word pieces before injecting the [CLS] and [SEP] tokens, ensuring the structural markers required by BERT models are never overwritten.

Enforced Minimum Bounds: Added an ArgumentError validation check to prepareNerInput ensuring maxLength is at least 2, which is the mathematical minimum required to contain the mandatory start and end tokens.

Added Initialization Validation: Updated fromStringContent to verify that both [UNK] and [PAD] are explicitly defined within the parsed vocabulary map, replacing late-stage runtime crashes (!) with a descriptive FormatException during setup.