NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
npm
npm, #1680 most downloaded on npm
A promise based streaming tokenizer
Latest stable
10.7.0
Last release 20 days ago
02 Nov 2026
Release timing varies
gaps range from 1 weeks to 8 months
Nearly every release is documented
notes for 55 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
9 years old
127 releases · first in 2017
Drag across the ruler to compare, or tap a release to open it.
feat: add TokenWriter for token serialization @Borewit
NPM release: strtok3@10.7.0
fix: reject stream peeks beyond known EOF before allocating lookahead @Borewit
NPM release: strtok3@10.6.3
Your coding agent can read these notes before it upgrades. Set up the MCP server →
fix: preserve caller-owned stream tokenizer options @Borewit
NPM release: strtok3@10.6.2
fix: support Node.js type checks without DOM globals @Borewit
NPM release: strtok3@10.6.1
feat: add finite borrowed tokenizer views @Borewit
NPM release: strtok3@10.6.0
feat: add regular-file-only mode to fromFile @Borewit
NPM release: strtok3@10.5.0
fix: preserve peeked data after normal Node stream close @Borewit
NPM release: strtok3@10.4.1
Add an opt-in tokenizer buffer allocation limit @Borewit
Add an opt-in tokenizer buffer allocation limit @Borewit (#1320)
Applications parsing untrusted input should set a finite maxBufferSize when creating a tokenizer. The cap is opt-in and defaults to Infinity; choose a limit appropriate for your application, for example 64 MiB:
import { fromStream } from 'strtok3';
const tokenizer = await fromStream(stream, {
maxBufferSize: 64 * 1024 * 1024
});
This limits dynamically sized tokenizer buffers and stream lookahead spans. Oversized buffer or lookahead requests reject with BufferSizeError. It does not cap total process memory, caller-owned buffers, allocations inside a token's get() method, or chunks supplied by stream sources.
See the tokenizer options documentation for details.
NPM release: strtok3@10.4.0
Reject negative ignore length @Borewit
NPM release: strtok3@10.3.5
Remove source-map references from distribution @Borewit
NPM release: strtok3@10.3.4
Improve API documentation @Borewit
NPM release: strtok3@10.3.2
Don't throw EndOfStreamError on 0 bytes read, with mayBeLess flag set @Borewit
NPM release: strtok3@10.3.1
Merge peek-readable package into strtok3 @Borewit
NPM release: strtok3@10.3.0
Update to peek-readable 7.0.0 @Borewit
NPM release: strtok3@10.2.2
Bump peek-readable from 6.1.0 to 6.1.1 @[dependabot[bot]]
NPM release: strtok3@10.2.1
Update peek-readable to ^6.1.0. @Borewit
^6.1.0. @Borewit (#1209)NPM release: strtok3@10.2.0
Bump peek-readable from 6.0.1 to 6.0.2 @[dependabot[bot]]
NPM release: strtok3@10.1.2
Fix close ReadStreamTokenizer @Borewit
ReadStreamTokenizer @Borewit (#1204)NPM release: strtok3@10.1.1
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fix random-read in BufferTokenizer @Borewit
NPM release: strtok3@10.0.1
Deprecate option.offset @Borewit
option.offset @Borewit (#1187)NPM release: strtok3@10.0.0
Export type IRandomAccessFileInfo @Borewit
IRandomAccessFileInfo @Borewit (#1185)NPM release: strtok3@9.1.1
Introduction of random access reading @Borewit
NPM release: strtok3@9.1.0
Abort tokenizer on close() @Borewit
NPM release: strtok3@9.0.1
Add function to abort async operations @Borewit
NPM release: strtok3@9.0.0
Update to ECMAScript 2020 (11th Edition) @Borewit
AnyWebByteStream from peek-readable and not from own export @Borewit (#1149)NPM release: strtok3@8.1.0
Fix export typing AnyWebByteStream @Borewit
AnyWebByteStream @Borewit (#1146)NPM release: strtok3@8.0.5
Allow both DOM and Node.js web-byte-streams @Borewit
NPM release: strtok3@8.0.4
Workaround for issue in Bun, which throws an error on 0 byte file read @Araxeus
NPM release: strtok3@8.0.3
Incomplete NPM release, replaced by v8.0.3 Release has been deprecated at NPM.
Incomplete NPM release, replaced by v8.0.3 Release has been deprecated at NPM.
Export AbstractTokenizer and more types @Borewit
AbstractTokenizer and more types @Borewit (#1133)NPM release: strtok3@8.0.1
Be able to pass on-close handler @Borewit
Switch node:fs/promise API @Borewit
node:fs/promise API @Borewit (#1125)NPM release: strtok3@7.1.1
feat: Goodbye, node.js Buffer @bjornstar
NPM release: strtok3@7.1.0
:warning: Breaking change: version 7 is now a pure ECMAScript Module (ESM).
:warning: Breaking change: version 7 is now a pure ECMAScript Module (ESM).
Changes:
NPM: strtok3 7.0.0
ESM: Fixed import of types #787
Nothing published for this version
Update peek-readable to v5.0.0-alpha.5
Nothing published for this version
Explicit node:buffer import https://github.com/Borewit/strtok3/pull/632
Changes:
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Update ECMAScript 2017 (ES8) to ECMAScript 2018 (ES9) #667
Nothing published for this version
Nothing published for this version
Use strictEqual in favour of equal in unit tests #527
Changes:
strictEqual in favour of equal in unit tests #527NPM: strtok3 6.2.2
Fix BufferTokenizer copy https://github.com/Borewit/strtok3/pull/523
Abstract Buffer usage to Uint8Array #522
Changes:
Note: There are sill remaining dependencies on Buffer.
NPM: strtok3 6.2.0
Changes: * #516 Fix dependency @types/debug as dev dependency NPM: strtok3 6.1.3
Bump peek-readable from 3.1.3 to 3.1.4 #515
Downgrade @tokenizer/token to version 0.1.1 #513
Changes: * #506 Update dependency to token-types v3.0.0 using BigInt * #511 Fix ignore documentation NPM: strtok3 6.1.0
Changes:
NPM: strtok3 6.1.0
One column per quarter.