This is the first released version of chumsky's 'zero-copy' rewrite.

This release has no precise changelog, although one will be added when 1.0.0 is eventually released in full form. For now, things are still fluctuating enough that a full changelog would inevitably be out of date in a few weeks.

Thanks

This release has been over a year in development and represents the work of a lot of people. In particular:

• @CraftSpider, who effectively co-developed the rewrite with me and came up with large chunks of the core API
• @wackbyte, who ported many combinators over to the new codebase as well as adding no_std support
• @bew, who reworked many combinators around changes to the core API
• @Zij-IT, who ported all of the text combinators across, as well as the (yet to be merged) pratt parser combinator created by @alvra
• Many other contributors who worked on smaller items

How you can help

This is the first alpha release. Do not expect a finished product: many minor API features are still incomplete, missing, or subject to change. Some documentation is incomplete, or still refers to concepts from past versions of the crate. In particular, the tutorial has not yet been updated. You may experience bugs, API footguns, and more issues besides. That said, we're releasing this version because we believe the core of the rewrite is ready to be exposed to users and we want to find out what problems there are and catch them before a full release.

We'd like folks to open issues if they find:

• Bugs
• API oddities (things that don't look/feel right, or could be expressed more neatly)
• Things that feel like they should work, but don't (lifetime issues, unnecessary cloning, etc.)
• Missing features

If you're an even awesome-er sort of person and you feel like contributing to the crate, there's still a lot of work that needs doing in the following areas:

• Documentation
• Writing/updating examples
• Filling API 'holes'
• Porting old APIs over
• Small improvements to existing combinators
• Writing tests
• API design: there's still work to be done on the context-sensitivity, recovery, and iterable parser APIs

All this aside, you'd be helping us out a bunch just by using this alpha release (especially porting existing chumsky parsers over to it) and telling us how you got on: what worked, what didn't work, what things you got stuck on or confused by, etc. If you'd like to give a more casual report like this, feel free to start a discussion.

What's new?

Needless to say, the crate has received a substantial upgrade, overhauling virtually every aspect of its API. It's substantially more capable than it ever was, and now supports the following:

• Zero-copy parsing: parser outputs can hold references to the input
• Nested parsing: parsers can handle nested data structures like token trees
• Stateful parsing: parsers can be parameterised by state, allowing for the natural integration of arena allocators, string interners, etc.
• Memoisation: parsers can opt into memoisation, allowing you to quickly parse awkward grammars that would normally produce exponential behaviour in a traditional recursive descent parser
• Left recursion: the aforementioned memoisation feature can also properly handle left recursive grammars elegantly
• Context-sensitive parsing: parsers can use built-in context sensitivity to carefully parameterise future parsers, allowing you to parse things like Rust-style raw strings, Pythonic indentation, and other context-sensitive syntax that context-free parsers traditionally struggle with
• Iterable parsers: parsers that produce multiple outputs can now be turned into iterators, similar to logos lexers

Performance

On top of all of that, we've worked really hard to push performance as far as we can using an innovative use of Generic Associated Types (GATs) internally that allows chumsky to automatically detect when an output is never used (such as with .then_ignore(...)) and avoid generating it all in the first place. You can find some technical details about this approach in Niko Matsakis' blog where they discuss chumsky.

Our work on performance has paid off: chumsky's JSON benchmark is now extremely competitive, beating out nom and others, and even banging on the door of more traditional hand-written JSON parsers.

In general, you can probably expect this new release to be several times faster than older releases for similar parsers. The JSON benchmark is about 12x faster.

Conclusion

Pushing zero-copy to the point of a release was always going to be a very long road to walk, but we're finally approaching the end. Thanks for using chumsky, and - if you're fortunate enough to have the resources and kind enough to consider donating them - please support the other contributors I listed at the top of this release!

Added

• select! and select_ref! now support cfg attributes.
• You can now call as_terminal and as_non_terminal on labels that will hide the internals for the debug info
• RichReason can now be provided with an arbitrary RichReason::Custom type parameter
• Implemented ConfigParser for OneOf
• Rich::map_span
• Parser::map_err_with
• Parser::filter_map

Removed

• Unused spin and sync feature flags

Changed

• Debug info now uses the Display impl to render labels with instead of Debug
• Migrated to Codeberg, updated links accordingly
• Removed the unstable Cache API due to soundness issues
• Added DefaultExpected::NothingElse to allow parsers that can never succeed to generate correct errors
• Removed the Container trait in favour of std's FromIterator, allowing .collect() to work with a greater range of types

Fixed

• Labelled -> LabelledWith conversion now carries over the proper debug settings
• Fixed crash when using memoisation with certain kinds of error generation
• DefaultExpected::into_owned now has more permissive lifetime constraints

Added

• MapExtra::emit, which allows emitting secondary errors during mapping operations
• InputRef::emit, which allows emitting secondary errors within custom parsers
• labelled_with, which avoids the need to implement Clone for labels
• Input::split_token_span, a convenience function for splitting (Token, Span) inputs so that chumsky can understand them
• Input::split_spanned, which does the same as above, but for implementors of WrappingSpan
• spanned, a combinator which automatically annotates a parser output with a span
• Experimental:
  • IterParser::parse_iter, a way to turn an IterParser (like x.repeated()) into an iterator
  • Parser::debug, which provides access to various parser debugging utilities.

Changed

• Made nested_in more flexible by allowing it to map between different input types instead of requiring the same input as the outer parser

Fixed

• A prioritisation bug with nested_in

Added

• Implement Default, PartialOrd, and Ord for SimpleSpan
• Implement PartialOrd and Ord for Rich

Fixed

• Patched compilation error that only appeared in release builds

Added

• The set(...) combinator, which may be used to conveniently parse a set of patterns, in any order
• Support for non-associative infix operators are now supported by the .pratt(...) combinator
• Parser::try_foldl, allowing folding operations to short-circuit in a manner similar to Parser::try_map
• Parser::contextual, which allows a parser to be disabled or enabled in a context-sensitive manner
• Implemented ValueInput for IterInput, which was previously missing
• More types that implement the State trait:
  • RollbackState, which reverts parser state when a parser rewinds and may be used to, for example, count the number of times a pattern appears in the input
  • TruncateState, which truncates a vector when a parser rewinds, and may be used to implement an arena allocator for AST nodes
• Implemented IterParser for a.then(b) when both a and b are both IterParsers that produce the same output type

Changed

• Made lifetime bounds on recursive and ParserExtra more permissive
• Improved support for grapheme parsing
• Text parsers now report labels
• Parser::filter now generates a DefaultExpected::SomethingElse label instead of nothing (this can be overridden with the .labelled(...) function)
• Improved areas of documentation
• Make whitespace parsers reject codepoints that are vulnerable to CVE-2021-42574
• Maybe the select! parser more permissive, accepting any implementor of Input instead of requiring ValueInput too

Fixed

• Many minor incorrect debug-only sanity checks have been fixed
• Many minor span and error prioritisation behavioural problems have been fixed (most notably, Parser::try_map)
• The .rewind() parser no longer rewinds any secondary error that were encountered
• Accidental text::ascii::keyword lifetime regression

Added

• Implemented Container for VecDeque
• New section covering recursion in the guide

Changed

• Boxed types now have a default type parameter of extra::Default, like Parser and IterParser
• The tutorial has been updated for 0.10 and has been moved to the guide

Fixed

• Nonsense spans occasionally generated for non-existent tokens
• Improved docs have been added for several items
• Many minor documentation issues have been fixed

The 0.10.0 release of chumsky is a from-scratch rewrite of the crate based on the work that's been ongoing for several years in the 1.0.0 alpha builds. The release of 0.10.0 is, in many ways, a concession to a few inconvenient facts:

• Users were rightly complaining that the latest stable release of chumsky (i.e: that which docs.rs shows by default) was the 0.9.x release, despite the fact that we've been recommending that new users use 1.0.0 alpha builds for a long time now. Many users have accidentally tried to run code taken from the 1.0.0 examples, only to find that they don't work with 0.9.
• 1.0.0 has been in development for a long time, and there are still some breaking changes left to make (although we're getting closer!). Users wants to be able to pull in a stable version and start working productively, and the existing situation was becoming cumbersome.
• Although 1.0.0 is not yet finished, it is moving closer and closer to its final form. It is unlikely that much about the surface API will change between 0.10.0 and 1.0.0, although some changes left to be made are technically breaking.

We recommend that users of chumsky depend on 0.10.0, if they can. Despite that, there are a few things to consider:

• (This has now been resolved) Not all documentation has been properly updated yet. Some docs still reference 0.9.x concepts or are not yet complete.
• Some features are still in a state of partial completeness. We don't anticipate significant breakage going forwards, but some features are explicitly in need of more work and future 0.x releases will address them.

Any help from the community to assist in resolving these points is greatly appreciated!

Here follows the changelog for 0.10.0. I also wrote up an informal migration guide.

Added

• Support for zero-copy parsing (i.e: parser outputs that hold references to the parser input)
• Support for parsing nested inputs like token trees
• Support for parsing context-sensitive grammars such as Python-style indentation, Rust-style raw strings, and much
  more
• Support for parsing by graphemes as well as unicode codepoints
• Support for caching parsers independent of the lifetime of the parser
• A new trait, IterParser, that allows expressing parsers that generate many outputs
• Added the ability to collect iterable parsers into fixed-size arrays, along with a plethora of other container types
• Support for manipulating shared state during parsing, elegantly allowing support for arena allocators, cstrees,
  interners, and much more
• Support for a vast array of new input types: slices, strings, arrays, impl Readers, iterators, etc.
• Experimental support for memoization, allowing chumsky to parse left-recursive grammars and reducing the
  computational complexity of parsing certain grammars
• An extension API, allowing third-party crates to extend chumsky's capabilities and introduce new combinators
• A pratt parser combinator, allowing for conveniently and simply creating expression parsers with precise operator
  precedence
• A regex combinator, allowing the parsing of terms based on a specific regex pattern
• Properly differentiated ASCII and Unicode text parsers

Removed

• Parser::then_with has been removed in favour of the new context-sensitive combinators

Changed

• Performance has radically improved
• Error generation and handling is now significantly more flexible

Note: version 0.10 is a from-scratch rewrite of chumsky with innumerable small changes. To avoid this changelog being longer than the compiled works of Douglas Adams, the following is a high-level overview of the major feature additions and does not include small details.

Added

• Support for zero-copy parsing (i.e: parser outputs that hold references to the parser input)
• Support for parsing nested inputs like token trees
• Support for parsing context-sensitive grammars such as Python-style indentation, Rust-style raw strings, and much more
• Support for parsing by graphemes as well as unicode codepoints
• Support for caching parsers independent of the lifetime of the parser
• A new trait, IterParser, that allows expressing parsers that generate many outputs
• Added the ability to collect iterable parsers into fixed-size arrays, along with a plethora of other container types
• Support for manipulating shared state during parsing, elegantly allowing support for arena allocators, cstrees, interners, and much more
• Support for a vast array of new input types: slices, strings, arrays, impl Readers, iterators, etc.
• Experimental support for memoization, allowing chumsky to parse left-recursive grammars and reducing the computational complexity of parsing certain grammars
• An extension API, allowing third-party crates to extend chumsky's capabilities and introduce new combinators
• A pratt parser combinator, allowing for conveniently and simply creating expression parsers with precise operator precedence
• A regex combinator, allowing the parsing of terms based on a specific regex pattern
• Properly differentiated ASCII and Unicode text parsers

Removed

• Parser::then_with has been removed in favour of the new context-sensitive combinators

Changed

• Performance has radically improved
• Error generation and handling is now significantly more flexible

Fixed

• Properly fixed skip_then_retry_until regression

Fixed

• Regression in skip_then_retry_until recovery strategy

Added

• A spill-stack feature that uses stacker to avoid stack overflow errors for deeply recursive parsers
• The ability to access the token span when using select! like select! { |span| Token::Num(x) => (x, span) }
• Added a skip_parser recovery strategy that allows you to implement your own recovery strategies in terms of other
  parsers. For example, .recover_with(skip_parser(take_until(just(';')))) skips tokens until after the next semicolon
• A not combinator that consumes a single token if it is not the start of a given pattern. For example,
  just("\\n").or(just('"')).not() matches any char that is not either the final quote of a string, and is not the
  start of a newline escape sequence
• A semantic_indentation parser for parsing indentation-sensitive languages. Note that this is likely to be
  deprecated/removed in the future in favour of a more powerful solution
• #[must_use] attribute for parsers to ensure that they're not accidentally created without being used
• Option<Vec<T>> and Vec<Option<T>> now implement Chain<T> and Option<String> implements Chain<char>
• choice now supports both arrays and vectors of parsers in addition to tuples
• The Simple error type now implements Eq

Changed

• text::whitespace returns a Repeated instead of an impl Parser, allowing you to call methods like at_least and
  exactly on it.
• Improved no_std support
• Improved examples and documentation
• Use zero-width spans for EoI by default
• Don't allow defining a recursive parser more than once
• Various minor bug fixes
• Improved Display implementations for various built-in error types and SimpleReason
• Use an OrderedContainer trait to avoid unexpected behaviour for unordered containers in combination with just

Fixed

• Made several parsers (todo, unwrapped, etc.) more useful by reporting the parser's location on panic
• Boxing a parser that is already boxed just gives you the original parser to avoid double indirection
• Improved compilation speeds

Added

• A spill-stack feature that uses stacker to avoid stack overflow errors for deeply recursive parsers
• The ability to access the token span when using select! like select! { |span| Token::Num(x) => (x, span) }
• Added a skip_parser recovery strategy that allows you to implement your own recovery strategies in terms of other parsers. For example, .recover_with(skip_parser(take_until(just(';')))) skips tokens until after the next semicolon
• A not combinator that consumes a single token if it is not the start of a given pattern. For example, just("\\n").or(just('"')).not() matches any char that is not either the final quote of a string, and is not the start of a newline escape sequence
• A semantic_indentation parser for parsing indentation-sensitive languages. Note that this is likely to be deprecated/removed in the future in favour of a more powerful solution
• #[must_use] attribute for parsers to ensure that they're not accidentally created without being used
• Option<Vec<T>> and Vec<Option<T>> now implement Chain<T> and Option<String> implements Chain<char>
• choice now supports both arrays and vectors of parsers in addition to tuples
• The Simple error type now implements Eq

Changed

• text::whitespace returns a Repeated instead of an impl Parser, allowing you to call methods like at_least and exactly on it.
• Improved no_std support
• Improved examples and documentation
• Use zero-width spans for EoI by default
• Don't allow defining a recursive parser more than once
• Various minor bug fixes
• Improved Display implementations for various built-in error types and SimpleReason
• Use an OrderedContainer trait to avoid unexpected behaviour for unordered containers in combination with just

Fixed

• Made several parsers (todo, unwrapped, etc.) more useful by reporting the parser's location on panic
• Boxing a parser that is already boxed just gives you the original parser to avoid double indirection
• Improved compilation speeds

Added

• then_with combinator to allow limited support for parsing nested patterns
• impl From<&[T; N]> for Stream
• SkipUntil/SkipThenRetryUntil::skip_start/consume_end for more precise control over skip-based recovery

Changed

• Allowed Validate to map the output type
• Switched to zero-size End Of Input spans for default implementations of Stream
• Made delimited_by take combinators instead of specific tokens
• Minor optimisations
• Documentation improvements

Fixed

• Compilation error with --no-default-features
• Made default behaviour of skip_until more sensible

Added

• A new tutorial to help new users

• select macro, a wrapper over filter_map that makes extracting data from specific tokens easy

• choice parser, a better alternative to long or chains (which sometimes have poor compilation performance)

• todo parser, that panics when used (but not when created) (akin to Rust's todo! macro, but for parsers)

• keyword parser, that parses exact identifiers

• from_str combinator to allow converting a pattern to a value inline, using std::str::FromStr

• unwrapped combinator, to automatically unwrap an output value inline

• rewind combinator, that allows reverting the input stream on success. It's most useful when requiring that a pattern is followed by some terminating pattern without the first parser greedily consuming it

• map_err_with_span combinator, to allow fetching the span of the input that was parsed by a parser before an error was encountered

• or_else combinator, to allow processing and potentially recovering from a parser error

• SeparatedBy::at_most to require that a separated pattern appear at most a specific number of times

• SeparatedBy::exactly to require that a separated pattern be repeated exactly a specific number of times

• Repeated::exactly to require that a pattern be repeated exactly a specific number of times

• More trait implementations for various things, making the crate more useful

Changed

• Made just, one_of, and none_of significant more useful. They can now accept strings, arrays, slices, vectors, sets, or just single tokens as before
• Added the return type of each parser to its documentation
• More explicit documentation of parser behaviour
• More doc examples
• Deprecated seq (just has been generalised and can now be used to parse specific input sequences)
• Sealed the Character trait so that future changes are not breaking
• Sealed the Chain trait and made it more powerful
• Moved trait constraints on Parser to where clauses for improved readability

Fixed

• Fixed a subtle bug that allowed separated_by to parse an extra trailing separator when it shouldn't
• Filled a 'hole' in the Error trait's API that conflated a lack of expected tokens with expectation of end of input
• Made recursive parsers use weak reference-counting to avoid memory leaks

Added

• skip_until error recovery strategy
• SeparatedBy::at_least and SeparatedBy::at_most for parsing a specific number of separated items
• Parser::validate for integrated AST validation
• Recursive::declare and Recursive::define for more precise control over recursive declarations

Changed

• Improved separated_by error messages
• Improved documentation
• Hid a new (probably) unused implementation details

Added

• take_until primitive

Changed

• Added span to fallback output function in nested_delimiters

Added

• Support for LL(k) parsing
• Custom error recovery strategies
• Debug mode
• Nested input flattening

Changed

• Radically improved error quality