winnow

Fixes

• Correctly point to error location in InputErrors Display for single-line input

Features

• Support Accumulate for BTreeSet and HashSet

Features

• Add more patterns for token::take_until[01]

Performance

• Optimize take_until* when parsing &str

Features

• Add take_till ranged parser

Compatibility

• Deprecated take_till0, take_till1 in favor of take_till

Features

• Add Caseless to make it easier to add case insensitivity to any parser.

Compatibility

• Deprecated tag_no_case in favor of tag(Caseless(...))

Features

• Add separated combinator

Compatibility

• Deprecated separated0 and separated1 in favor of separated

Fixes

• Support Accumulate for HashMap with custom hashers

Compatibility

• Deprecated Uint impls for signed numbers

Documentation

• Provide lexer/parser example with details on handling of custom tokens

Fixes

• Correctly calculate offset_from for non-byte slices

Performance

• Improve build times with debug when closures are used

Performance

• Speed up take_until when simd is enabled

Performance

• ErrMode inlining for improving gitoxide

Performance

• Try inlining more wrapper functions

Features

• Impl Clone for ContextError

Features

• Impl Display for StrContext

Fixes

• Improve rendering of ErrorKind in other errors

Features

• Add TreeError::into_owned
• impl Error for TreeError
• Add back in VerboseError to help with migrations

Features

• Support Display for TreeError

Features

• New TreeError for showing full error path in tests
• InputError::map_input for making input types nicer for test failures

Fixes

• alt([...]) fails, or the errors together, like alt((...))

Internal

• Update dependencies

Features

• implement AddContext for ErrMode for easier direct construction

Features

• Allow [P; N] arrays within alt

Features

• Added ParseError::into_inner

Migration

Preparation: 0. Upgrade to the latest 0.4 release

1. Process all deprecation warnings
2. Replace not-quite deprecated items:

• Replace winnow::branch with winnow::combinator
• Replace winnow::bytes with winnow::token
• Replace Offset::offset_to with Offset::offset_from
• Replace ParseError with ParserError
• Replace ContextError with AddContext
• Replace Error with InputError

1. Ensure parsers return impl Parser over impl FnMut to reduce the size of the upgrade commit since the FnMut signature will change
2. When creating errors within your parser,s switch from doing so directly to using ErrModes trait impls to make the code more independent of the current error type
3. Instrument your parser with winnow::trace::trace to make it easier to debug problems on upgrade
4. Merge the above

For the actual upgrade, there are two approaches that can be mixed.

Switch to new, imperative APIs

1. Upgrade to 0.5
2. Replace IResult<I, O, E> with PResult<O, E> (E is still defaulted but to ContextError)
3. Replace your parsers I parameter with &mut I

• For fn(&mut &str) -> PResult<&str>, a lifetime will be needed: fn<'i>(&mut &'i str) -> PResult<&'i str>
• For embedded closures, move |input| might need to be updated to move |input: &mut _|

1. Update imperative parsers from expecting parse_next to return (input, output) to output

• When matching against ErrMode::Backtrace, you might need to input.reset(checkpoint) to revert the parser state

1. Update error reporting for new error type returned from Parser::parse

Maintain pure-functional APIs

1. Upgrade to 0.5
2. Switch to new names for the old APIs

• Replace Parser::parse_next with Parser::parse_peek
• Replace Stream::next_token with Stream::peek_token
• Replace Stream::next_slice with Stream::peek_slice

1. Wrap calls to FnMut(I) -> IResult<I, O, E> parsers with winnow::unpeek
2. Update error reporting for new error type returned from Parser::parse

Example: toml_edit:

• Pre-upgrade steps
• Upgrade

Example: winnow:

• Maintaining pure-functional API
  • making pure-functional Stream code to still work
  • making pure-functional impl Parser code to still work
  • making pure-functional FnMut code to still work
• Switch to new, imperative APIs from pure-functional API
  • porting to new Parser::parse_next
  • porting to new FnMut

Example: chumskys json bench:

• Pre-upgrade steps
• Upgrade

Compatibility

• Parser::parse_next (and the impl Parser for FnMut) now take &mut I and return PResult
  • e.g. porting to new Parser::parse_next
  • e.g. porting to new FnMut
  • e.g. making pure-functional Stream code to still work
  • e.g. making pure-functional impl Parser code to still work
  • e.g. making pure-functional FnMut code to still work
• Changed error type for Parser::parse to allow quality errors without requiring E to carry I
• Removed impl ContainsToken for &str (e.g. one_of("abc") since it takes 5x more time leading to easily avoidable slow downs
  • Instead use ['a', 'b', 'c'] or 'a'..='c' (see 08b3e57 for examples)
• Changed ParserError and FromExternalError from accepting I to &I
• Renamed ParseError to ParserError
• Renamed ContextError to AddContext
• Renamed Error to InputError
• Removed Offset::offset_to in favor of Offset_offset_from
• Removed hack from trace that allowed parsers to change input to point to a different string (this wasn't present in other parsers relying on Offset::offset_from)
• Removed some Offset bounds, requiring changing a.offset_from(b) to a.offset_from(&b)
• Renamed bytes to token to clarify its scope
• Renamed character to ascii to make it agnostic of char tokens
• Moved all binary-related parsing combinators into the binary module
  • bits -> binary::bits
  • number
  • multi::length_*
• Moved all generic combinators into combinator
  • sequence
  • branch
  • multi
• Deprecated parsers were removed
• Stream trait changed
  • Stream::raw added
  • Stream::next_token renamed to Stream::peek_token
  • Stream::next_slice renamed to Stream::peek_slice
  • New Stream::next_token and Stream::next_slice added
  • Added Stream::finish and Stream::peek_finish
  • Stream::Checkpoint, Stream::checkpoint, and Stream::reset added
  • Offset<Stream::Checkpoint> is a new super trait

Features

• Added ContextError, a lightweight ParserError that supports AddContext (default for PResult)
• Added Stream::finish and Stream::peek_finish for making it easier to capture all remaining content
• Added ErrMode::into_inner for unifying Cut and Backtrack
• Allow ErrorKind as a ParserError

Fixes

• Correctly show BStr and Bytes Debug impl, rather than &[u8]

Performance

• Removed impl ContainsToken for &str (e.g. one_of("abc") since it takes 5x more time leading to easily avoidable slow downs
• Walk the Stream imperatively with &mut I rather than requiring returning the updating error location
  • i.e. switched parsing from parser(I) -> (I, O) to parser(&mut I) -> O
  • Gains are around 10% in select benchmarks
  • It is believed that this reduced the size of the return type, allowing values to be returned through registers more often (as opposed to the stack), and this gain only shows up in parsers that return large data types, like format preserving parsers

chore: Release winnow version 0.4.12

Performance

• Backport v0.5 performance improvements

Performance

• Additional inlining to make v0.4 and v0.5 performance differences focus on architecture and not additional changes.

Features

• Add aliases for renamed APIs in v0.5
  • error::AddContext which wraps error::ContextError
  • error::ParserError which wraps error::ParseError
  • error::InputError which wraps error::Error

Features

• Add aliases for renamed APIs in v0.5
  • Parser::parse_peek which wraps Parser::parse_next
  • Stream::peek_token which wraps Stream::next_token
  • Stream::peek_slice which wraps Stream::next_slice
  • Offset::offset_from which wraps Offset::offset_to
• Add unpeek which is a no-op function in prep for v0.5

Fixes

• Allow ascii::escaped_transform on core
• Update ascii::float to correct parse infinity
• Update special topics to link out to relevant parsers they re-implement

Performance

• Resolve a performance regression in winnow::ascii: parsers (any that use take_while)

Compatibility

• Deprecated count, repeat0, repeat1 in favor of repeat
• Deprecated take_while0, take_while1 in favor of take_while
• Deprecated fold_repeat0, fold_repeat1 in favor of fold_repeat

Performance

• Improve partial-parsing performance by ~15%

Fixes

• Improve color control for debug feature

Fixes

• Reduce risk of breaking when using glob imports

Compatibility

• MSRV raised to 1.64.0
• bits, bytes, character, number, sequence, branch, and multi modules are deprecated
• Parser::map_res is deprecated

Fixes

• Renamed Parser::map_res to Parser::try_map for consistency
• Renamed bytes to token to clarify its scope
• Renamed character to ascii to make it agnostic of char tokens
• Moved all binary-related parsing combinators into the binary module
  • bits -> binary::bits
  • number
  • multi::length_*
• Moved all generic combinators into combinator
  • sequence
  • branch
  • multi
• Renamed the *many* combinators as *repeat* to make the code read more clearly
• Generalized repeat_m_n as repeat which takes any range type
• Generalzied fold_repeat_m_n as fold_repeat which takes any range type
• Generalized take_while_m_n as take_while which takes any range type

Documentation

• Start "Performance" special topic

Features

• Allow array references and char arrays for ContainsToken

Performance

• Revert a regression in space0, space1, multiscape0, multispace1

Compatibility

Breaking:

• All winnow parsers now return impl Parser, rather than impl FnMut
• winnow::prelude::Parser is now named
• Some infinite loop, overflows, and bounds errors were changed to asserts
• ErrorKind was greatly simplified
• Some trait bounds and generic parameters changed
• Deprecated APIs were removed

Other

• Deprecated FinishIResult

Fix

• Improved type inference, especially for "", '' parsers
• winnow::prelude now allows named access to Parser

Features

• Parser::parse added as a replacement for FinishIResult::finish

Performance

• escape: Remove extraneous bounds checks

Internal

• Cleaned up the code so it better serves as examples for user-written parsers

Performance

• Backport v0.5 performance improvements

Performance

Backport v0.4 and v0.5 inlining changes for more even comparison across architectures

Features

• error::Error now implements Copy, Clone

Fixes

• Fix core and alloc support

Fixes

• Ensure Partial can be toggled to complete.

Documentation

• Fix tutorial headers

Documentation

• Improve the partial parsing example

Fixes

• Fix bounds on Parser::flat_map so that it works with closures

v0.3.0 changes are relative to nom v7.1.3

Versioning starts at v0.3.0 due to the historical nature of how winnow was forked from nom. See also the nom8 changelog.

nom Migration Guide

1. Update all references from nom to winnow
2. Replace

• winnow::Err::Error with winnow::Err::Backtrack
• winnow::Err::Fatal with winnow::Err::Cut
• winnow::Input* with winnow::stream::Stream
• E::append with err.append
• E::add_context with err.add_context
• Resolve many* type inference issues by adding .map(|v: Vec<_>| v)

1. Resolve remaining compile errors
2. Resolve deprecations as described by them

Examples:

• git-config-env
• git-conventional
• typos

Breaking Changes

• Parser::into in favor of Parser::output_into and Parser::err_into (#48)
• Combinator structs returned by Parser were moved from winnow to winnow::combinator module (#4)
• Tweaks were made to what input traits are needed for various types / parsers
• Removed pub use bits::*; into root namespace (#52)
• Moved error types and traits into the winnnow::error module (#73, #117)
• winnow::error::Err was renamed to ErrMode (#117)
  • Error variant was renamed to Backtrack
  • Fatal variant was renamed to Cut
• winnow::error::ParserError function parameter order has changed (#92)
• winnow::error::Error's code field was renamed to kind (#92)
• Removed support for (I, ErrorKind) errors in favor of winnow::error::Error (#92, #117)
• winnow::error::ErrorKind for infinite loops, like in winnow::multi, was changed to ErrorKind::Assert (#146)
• winnow::multi parsers will debug_assert by default when encountering infinite loops (#146)
• Removed error::error_to_u32 (#54)
• Changed all parsers to use FnMut (#152, #164)
• Moved input traits from winnow to winnow::stream (#13, #142)
• Merged many input traits into winnow::stream::Stream (#105, #107, #142, #155, #157)
  • InputIter::Iter is now Stream::IterOffsets
  • InputIter::iter_indices is now Stream::iter_offsets
  • InputIter::position is now Stream::offset_for
  • InputIter::slice_index is now Stream::offset_at
• Renamed winnow::stream::InputLen to SliceLen (#105)
• Renamed winnow::stream::FindSubstring to FindSlice (#105)
• Renamed winnow::stream::FindTokens to ContainsToken (#105)
• Renamed winnow::stream::ParseTo to ParseSlice (#136)
• Renamed winnow::stream::Offset::offset to offset_to (#105)
• Split AsBytes into AsBytes and AsBStr (#134)
• Removed seemingly unused winnow::stream trait impls (#105)
• Removed winnow::lib from API (#118)

Compatibility

MSRV was raised to 1.60 (#158, #160, #167)

memchr is no longer used by default; opt-in with the simd feature

Deprecations

• character::is_* functions in favor of AsChar (#25)
• winnow::*::complete::* and winnow::*::streaming::* parsers in favor of the merged ones (#28)
  • Unsigned parsers like u8 were merge into winnow::character::dec_uint (#59)
  • Unsigned parsers like i8 were merge into winnow::character::dec_int (#59)
  • winnow::number::*::hex_u32 was replaced with winnow::character::hex_uint (#59)
  • winnow::bytes::*::take_till was renamed to winnow::bytes::take_till0 (#132)
  • winnow::bytes::*::take_while was renamed to winnow::bytes::take_while0 (#132)
  • winnow::bytes::*::take_until was renamed to winnow::bytes::take_until0 (#132)
  • winnow::multi::*::many_till was renamed to winnow::multi::many_till0 (#132)
  • winnow::multi::*::separated_list0 was renamed to winnow::multi::separated0 (#140)
  • winnow::multi::*::separated_list1 was renamed to winnow::multi::separated1 (#140)
  • winnow::bytes::*::escaped was moved to winnow::character::escaped (#108)
  • winnow::bytes::*::escaped_transform was moved to winnow::character::escaped_transform (#108)
• dbg_dmp in favor of -features debug (#173)
• all_consuming in favor of eof / FinishIResult::finish (#112)
• Freestanding parsers that were in both winnow::combinator to Parser trait (#37)
  • map, see Parser::map
  • flat_map, see Parser::flat_map
  • map_parser, see Parser::and_then
• Freestanding parsers that were moved from winnow::combinator to Parser (#40)
  • map_res as Parser::map_res
  • map_opt as Parser::verify_map (#145)
  • complete as Parser::complete_err (#168)
  • verify as Parser::verify
  • value as Parser::value
  • recognize as Parser::recognize
  • consumed as Parser::with_recognized (note: output order swapped)
  • context as Parser::context
• many0_count, many1_count in favor of many0 and many1 (#127)
• Freestanding into in favor of Parser::output_into and Parser::err_into (#37, #48)
• winnow::combinator::cut in favor of winnow::combinator::cut_err (#117)
• Parser::parse in favor of Parser::parse_next (#76)
• is_a with take_while1 thanks to new FindToken impls (#44)
• is_not with take_till1 thanks to new FindToken impls (#44)
• satisfy with one_of thanks to new FindToken impls (#44)
• Parser::and, Parser::or in favor of impl Parser for <tuples> and alt (#37)
• tuple, pair parser in favor of using tuples (#42, #46)
• ParseError::from_char in favor of ContextError (#45)
• error::make_error and error::append_error (#55)
• error::Finish in favor of FinishIResult (#21)
• error::error_position! and error::error_mode_position! (#92)

Feature

• Provide a winnow::prelude (#21)
• winnow::Stateful for attaching state to the input type (#58)
• winnow::Located for tracking the input's location, along with Parser::span and Parser::with_span to capture it (#61)
• Improved debug experience with
  • New winnow::trace::trace combinator, used by all winnow parsers (#156, #165, #176)
  • Bytes and BStr new types with improved Debug (#144, #147, #149, #177)
• Merged streaming/complete parsers, opting in to streaming with the Partial<I> stream type (#28, #142, #166, #170, #172)
  • Merged ASCII float parsers moved to character and made generic (#38, #59)
  • Moved one_of, none_of from character to bytes as they are more general (#44)
  • Moved character::anychar to bytes::any with a more generic return type (#44)
• Allow generic context in errors (#49)
• impl Parser for <tuples> as an alternative to tuple(()) (#42)
• impl Parser for (char|u8|&str|&[u8]) as an alternative to one_of, char, and tag (#47, #50, #171)
• Parser::by_ref to allow using &mut impl Parser as a Parser (#34, #41)
• New combinators:
  • Parser::void (#113)
  • Parser::parse_to (#136)
  • winnow::combinator::todo (#135)
  • winnow::combinator::backtrack_err (#117)
  • winnow::branch::dispatch! as a match-like alternative to alt (#119)
  • winnow::multi::separated_foldl1 (#140)
  • winnow::multi::separated_foldr1 (#140)
• Add combinators directly to Parser:
  • Parser::map_res (#40)
  • Parser::verify_map (#40, #145)
  • Parser::complete_err (#40, #168)
  • Parser::verify (#40)
  • Parser::value (#40)
  • Parser::with_recognized (#40)
  • Parser::context (#40)
• Allow more containers to be used with winnow::multi parsers (#127)
• Allow u8 and char, ranges of u8 and char, functions, and tuples of the prior wherever ContainsToken is accepted (#44, #105)
• Generalize take_while, take_till with ContainsToken trait (#44, #105)
• Add character::is_* functions to AsChar that were missing (#2)
• Implement Stream for (I, usize), allowing more combinators to work with bit-streams (#153)
• Treat all slices as input, regardless of token type (#111, rust-bakery/nom#1482)
• FinishIResult::finish for dropping the remainder (#30)
• FindSlice now also finds tokens, making take_until0 and friends more flexible (#105)
• Implement ParseError and FromExternalError for ErrMode, making it easier to create the right kind of error (#120, #124)

Fixes

• Correctly count multi-byte chars for take_while_m_n (#130)
• Change fill, iterator, bits::bits, and bits::bytes from taking a function to taking a Parser (#10, #11, #120)
• Allow byte arrays of any size (#14)
• length_data and length_value now auto-detect complete/streaming, before were streaming only (#28)
• Clarify Offset::offset_to` with explicit assert (#109, rust-bakery/nom#1520)
• Prefer Into over From (#110, rust-bakery/nom#1460)
• Use ErrorConvert in ErrMode::convert (#124)

Performance

• Use memchr::memmem on FindSlice (#86, rust-bakery/nom#1375)
• Use Vec::with_capacity for length_count (#87, rust-bakery/nom#1462)

Documentation

• Pulled loose markdown into rustdoc (#1, #174, #183)
  • Pull in the nominomicon (#185, rust-bakery/nom#1525)
  • Pull examples into the "Special Topics" section (#175)
• Add #[doc(alias)]s for easier migration from nom, chumsky, and combine (#178)
• Made examples more interactive (#77, #175)
• Add example for bits::tag (#88, rust-bakery/nom#1501)
• Special topics: improve identifier example (#90, rust-bakery/nom#1334)

Internal

• Now verified with miri (#143, #161)