• Defined pyparsing-specific warning classes so that they can be selectively enabled or disabled without affecting warnings raised by other libraries in the same Python app:

  • PyparsingWarning - base warning for all pyparsing-specific warnings (inherits from UserWarning)
  • PyparsingDeprecationWarning - warning for using deprecated features (inherits from PyparsingWarning and DeprecationWarning)
  • PyparsingDiagnosticWarning - warning raised when pyparsing diagnostics are enabled and a diagnostic feature is used (inherits from PyparsingWarning)

• Added as_datetime parse action to pyparsing.common - a more generalized version of the convert_to_datetime parse action (supports any expression that extracts date/time fields into "year", "month", "day", etc. results names), and validates that the parsed fields represent a valid date and time.

• Added iso8601_date_validated and iso8601_datetime_validated expressions to pyparsing.common, which return a Python datetime.datetime

• Various performance improvements in ParseResults class and core functions, with 10-20% performance overall.

• Added regex_inverter web page (using PyScript) to demonstrate using the inv_regex.py example.

• Expanded regex forms handled by the examples/inv_regex.py example:

  • named capturing groups (?P<name>)
  • partial repetition ({m,} and {,n})
  • negated character classes ([^...])

• Added SPy (Simplified Python) parser to examples.

• Added license info to metadata, following PEP-639. Thanks to Gedalia Pasternak and Marc Mueller for submitted issue and PR. Fixes #626.

=======================================================================================
The version 3.3.0 release will begin emitting DeprecationWarnings for pyparsing methods that have been renamed to PEP8-compliant names (introduced in pyparsing 3.0.0, in August, 2021, with legacy names retained as aliases). In preparation, I added in pyparsing 3.2.2 a utility for finding and replacing the legacy method names with the new names. This utility is located at pyparsing/tools/cvt_pep8_names.py. This script will scan all Python files specified on the command line, and if the -u option is selected, will replace all occurrences of the old method names with the new PEP8-compliant names, updating the files in place.

Here is an example that converts all the files in the pyparsing /examples directory:

  python -m pyparsing.tools.cvt_pyparsing_pep8_names -u examples/*.py

The new names are compatible with pyparsing versions 3.0.0 and later.

=======================================================================================

• Deprecated indentedBlock, when converted using the cvt_pyparsing_pep8_names utility, will emit UserWarnings that additional code changes will be required. This is because the new IndentedBlock class no longer requires the calling code to supply an indent stack, while adding support for nested indentation levels and grouping.

• Deprecated locatedExpr, when converted using the cvt_pyparsing_pep8_names utility, will emit UserWarnings that additional code changes may be required. The new Located class removes the extra grouping level of the parsed values. (If the original locatedExpr parser was defined with a results name, then the extra grouping is retained, so that the results name nesting works properly; in this case, no code changes would be required.)

• Updated all examples and test cases to use PEP8 names (unless the test case is specifically designed to test behavior of a legacy method). Added railroad diagrams for some examples.

• Added exception handling when calling formatted_message(), so that str(exception) always returns at least something.

• All unit tests pass with Python 3.14, including 3.14t. This does not necessarily mean that pyparsing is now thread-safe, just that when run in the free-threaded interpreter, there were no errors. None of the unit tests try to do any parsing with multiple threads - they test the basic functionality of the library, under various versions of packrat and left-recursive parsing.

• Added AI instructions so that AI agents can be prompted with best practices for generating parsers using pyparsing code. These instructions are in the ai/best_practices.md file, and can be accessed programmatically by calling pyparsing.show_best_practices() or running python -m pyparsing.ai.show_best_practices from the command line, after installing the pyparsing package.

• Implemented a TINY language parser/interpreter using pyparsing, in the examples/tiny directory. This is a little tutorial language that I used to demonstrate how to use pyparsing to build a simple interpreter, following a recommended parser+AST+engine+run structure. The docs sub-directory also includes transcripts of the AI session used to create the parser and the interpreter. The samples sub-directory includes a few sample TINY programs.

• Fixed minor formatting bugs in pyparsing.testing.with_line_numbers, found during development of the TINY language example.

• Added test in DelimitedList and nested_expr which auto-suppress delimiting commas to avoid wrapping in a Suppress if delimiter is already a Suppress.

• Added performance benchmarking tools and documentation:

  • tests/perf_pyparsing.py runs a series of benchmark parsing tests, exercising different aspects of the pyparsing package. For cross-version analysis, this script can export results as CSV and append to a consolidated data file.
  • Runner scripts run_perf_all_tags.bat (Windows) and run_perf_all_tags.sh (Ubuntu/bash) execute the benchmark across multiple Python versions (3.9–3.14) and pyparsing versions (3.1.1 through 3.3.0), aggregating results into perf_pyparsing.csv at the repo root.
  • See tests/README.md for usage instructions.

• Used performance benchmarking to identify and revert an inefficient utility method used in transform_string (introduced in pyparsing 3.2.0b2).

(added in 3.3.0b1)

• Implemented a TINY language parser/interpreter using pyparsing, in the examples/tiny directory. This is a little tutorial language that I used to demonstrate how to use pyparsing to build a simple interpreter, following a recommended parser+AST+engine+run structure. The docs sub-directory also includes transcripts of the AI session used to create the parser and the interpreter.

• Fixed minor formatting bugs in pyparsing.testing.with_line_numbers, found during development of the TINY language example.

• Added test in DelimitedList and nested_expr which auto-suppress delimiting commas to avoid wrapping in a Suppress if delimiter is already a Suppress.

• Added performance benchmarking tools and documentation:

  • tests/perf_pyparsing.py runs a series of benchmark parsing tests, exercising different aspects of the pyparsing package. For cross-version analysis, this script can export results as CSV and append to a consolidated data file.
  • Runner scripts run_perf_all_tags.bat (Windows) and run_perf_all_tags.sh (Ubuntu/bash) execute the benchmark across multiple Python versions (3.9–3.14) and pyparsing versions (3.1.1 through 3.3.0), aggregating results into perf_pyparsing.csv at the repo root.
  • See tests/README.md for usage instructions.

• Supports Python 3.14 and free-threaded 3.14t. (Note: this does not mean that pyparsing is thread-safe, simply that pyparsing can safely run using the free-threaded build of Python.)
• The version 3.3.0 release will begin emitting DeprecationWarnings for pyparsing methods that have been renamed to PEP8-compliant names (introduced in pyparsing 3.0.0, in August, 2021, with legacy names retained as aliases).
• AI instructions have been included for AI guidance when generating parsers using pyparsing. They can be accessed using python -m pyparsing.ai.show_best_practices.
• The cvt_pyparsing_pep8_names utility (released in pyparsing 3.2.2) will raise UserWarnings when converting locatedExpr and indentedBlock calls, since the new methods have slightly different call signatures and return types.
• Source code examples have all been upgraded to use new PEP8-compliant names.

Barring any catastrophic bugs in this release, this will be the last release in the 3.2.x line. The next release, 3.3.0, will begin emitting DeprecationWarnings when the pre-PEP8 methods are used (see header notes in CHANGES file for more information, including available automation for converting any existing code using pyparsing using the old names).

• JINX! Well, 3.2.4 had a bug for Word expressions that include a space character, if that expression was then copied, either directly with .copy() or by adding a results name, or including in another construct (like DelimitedList) that makes a copy internally. Issue #618, reported by mstinberg, among others - thanks, and sorry for the inconvenience.

Full Changelog: 3.2.4...3.2.5

Barring any catastrophic bugs in this release, this will be the last release in the 3.2.x line. The next release, 3.3.0, will begin emitting DeprecationWarnings when the pre-PEP8 methods are used (see header notes above for more information, including available automation for converting any existing code using pyparsing with the old names).

• Fixed bug when using a copy of a Word expression (either by using the explicit copy() method, or attaching a results name), and setting a new expression name, a raised ParseException still used the original expression name. Also affected Regex expressions with as_match or as_group_list = True. Reported by Waqas Ilyas, in Issue #612 - good catch!

• Fixed type annotation for replace_with, to accept Any type. Fixes Issue #602, reported by esquonk.

• Added locking around potential race condition in ParserElement.reset_cache, as well as other cache-related methods. Fixes Issue #604, reported by CarlosDescalziIM.

• Substantial update to docstrings and doc generation in preparation for 3.3.0, great effort by FeRD, thanks!

• Notable addition by FeRD to convert docstring examples to work with doctest! This was long overdue, thanks so much!

• Fixed bug released in 3.2.2 in which nested_expr could overwrite parse actions for defined content, and could truncate list of items within a nested list. Fixes Issue #600, reported by hoxbro and luisglft, with helpful diag logs and repro code.

The upcoming version 3.3.0 release will begin emitting DeprecationWarnings for pyparsing methods that have been renamed to PEP8-compliant names (introduced in pyparsing 3.0.0, in August, 2021, with legacy names retained as aliases). In preparation, I have added in pyparsing 3.2.2 a utility for finding and replacing the legacy method names with the new names. This utility is located at pyparsing/tools/cvt_pep8_names.py. This script will scan all Python files specified on the command line, and if the -u option is selected, will replace all occurrences of the old method names with the new PEP8-compliant names, updating the files in place.

Here is an example that converts all the files in the pyparsing /examples directory:

  python -m pyparsing.tools.cvt_pyparsing_pep8_names -u examples/*.py

The new names are compatible with pyparsing versions 3.0.0 and later.

• Released cvt_pyparsing_pep8_names.py conversion utility to upgrade pyparsing-based programs and libraries that use legacy camelCase names to use the new PEP8-compliant snake_case method names. The converter can also be imported into other scripts as

    from pyparsing.tools.cvt_pyparsing_pep8_names import pep8_converter
  

• Fixed bug in nested_expr where nested contents were stripped of whitespace when the default whitespace characters were cleared (raised in this StackOverflow question https://stackoverflow.com/questions/79327649 by Ben Alan). Also addressed bug in resolving PEP8 compliant argument name and legacy argument name.

• Fixed bug in rest_of_line and the underlying Regex class, in which matching a pattern that could match an empty string (such as ".*" or "[A-Z]*" would not raise a ParseException at or beyond the end of the input string. This could cause an infinite parsing loop when parsing rest_of_line at the end of the input string. Reported by user Kylotan, thanks! (Issue #593)

• Enhancements and extra input validation for pyparsing.util.make_compressed_re - see usage in examples/complex_chemical_formulas.py and result in the generated railroad diagram examples/complex_chemical_formulas_diagram.html. Properly escapes characters like "." and "*" that have special meaning in regular expressions.

• Fixed bug in one_of() to properly escape characters that are regular expression markers (such as '*', '+', '?', etc.) before building the internal regex.

• Better exception message for MatchFirst and Or expressions, showing all alternatives rather than just the first one. Fixes Issue #592, reported by Focke, thanks!

• Added return type annotation of "-> None" for all __init__() methods, to satisfy mypy --strict type checking. PR submitted by FeRD, thank you!

• Added optional argument show_hidden to create_diagram to show elements that are used internally by pyparsing, but are not part of the actual parser grammar. For instance, the Tag class can insert values into the parsed results but it does not actually parse any input, so by default it is not included in a railroad diagram. By calling create_diagram with show_hidden = True, these internal elements will be included. (You can see this in the tag_metadata.py script in the examples directory.)

• Fixed bug in number_words.py example. Also added ebnf_number_words.py to demonstrate using the ebnf.py EBNF parser generator to build a similar parser directly from EBNF.

• Fixed syntax warning raised in bigquery_view_parser.py, invalid escape sequence "\s". Reported by sameer-google, nice catch! (Issue #598)

• Added support for Python 3.14.

• Updated generated railroad diagrams to make non-terminal elements links to their related sub-diagrams. This greatly improves navigation of the diagram, especially for large, complex parsers.

• Simplified railroad diagrams emitted for parsers using infix_notation, by hiding lookahead terms. Renamed internally generated expressions for clarity, and improved diagramming.

• Improved performance of cpp_style_comment, c_style_comment, common.fnumber and common.ieee_float Regex expressions. PRs submitted by Gabriel Gerlero,
  nice work, thanks!

• Add missing type annotations to match_only_at_col, replace_with, remove_quotes, with_attribute, and with_class. Issue #585 reported by rafrafrek.

• Added generated diagrams for many of the examples.

• Replaced old examples/0README.html file with examples/README.md file.

Version 3.2.0 - October, 2024

• Discontinued support for Python 3.6, 3.7, and 3.8. Adopted new Python features from Python versions 3.7-3.9:

  • Updated type annotations to use built-in container types instead of names imported from the typing module (e.g., list[str] vs List[str]).
  • Reworked portions of the packrat cache to leverage insertion-preserving ordering in dicts (including removal of uses of OrderedDict).
  • Changed pdb.set_trace() call in ParserElement.set_break() to breakpoint().
  • Converted typing.NamedTuple to dataclasses.dataclass in railroad diagramming code.
  • Added from __future__ import annotations to clean up some type annotations. (with assistance from ISyncWithFoo, issue #535, thanks for the help!)

• POSSIBLE BREAKING CHANGES

  The following bugfixes may result in subtle changes in the results returned or exceptions raised by pyparsing.

  • Fixed code in ParseElementEnhance subclasses that replaced detailed exception messages raised in contained expressions with a less-specific and less-informative generic exception message and location.

    If your code has conditional logic based on the message content in raised ParseExceptions, this bugfix may require changes in your code.

  • Fixed bug in transform_string() where whitespace in the input string was not properly preserved in the output string.

    If your code uses transform_string, this bugfix may require changes in your code.

  • Fixed bug where an IndexError raised in a parse action was incorrectly handled as an IndexError raised as part of the ParserElement parsing methods, and reraised as a ParseException. Now an IndexError that raises inside a parse action will properly propagate out as an IndexError. (Issue #573, reported by August Karlstedt, thanks!)

    If your code raises IndexErrors in parse actions, this bugfix may require changes in your code.

• FIXES AND NEW FEATURES

  • Added type annotations to remainder of pyparsing package, and added mypy run to tox.ini, so that type annotations are now run as part of pyparsing's CI. Addresses Issue #373, raised by Iwan Aucamp, thanks!

  • Exception message format can now be customized, by overriding ParseBaseException.format_message:

    def custom_exception_message(exc) -> str:
        found_phrase = f", found {exc.found}" if exc.found else ""
        return f"{exc.lineno}:{exc.column} {exc.msg}{found_phrase}"
    
    ParseBaseException.formatted_message = custom_exception_message
    

    (PR #571 submitted by Odysseyas Krystalakos, nice work!)

  • run_tests now detects if an exception is raised in a parse action, and will report it with an enhanced error message, with the exception type, string, and parse action name.

  • QuotedString now handles translation of escaped integer, hex, octal, and Unicode sequences to their corresponding characters.

  • Fixed the displayed output of Regex terms to deduplicate repeated backslashes, for easier reading in debugging, printing, and railroad diagrams.

  • Fixed (or at least reduced) elusive bug when generating railroad diagrams, where some diagram elements were just empty blocks. Fix submitted by RoDuth, thanks a ton!

  • Fixed railroad diagrams that get generated with a parser containing a Regex element defined using a verbose pattern - the pattern gets flattened and comments removed before creating the corresponding diagram element.

  • Defined a more performant regular expression used internally by common_html_entity.

  • Regex instances can now be created using a callable that takes no arguments and just returns a string or a compiled regular expression, so that creating complex regular expression patterns can be deferred until they are actually used for the first time in the parser.

  • Added optional flatten Boolean argument to ParseResults.as_list(), to return the parsed values in a flattened list.

  • Added indent and base_1 arguments to pyparsing.testing.with_line_numbers. When using with_line_numbers inside a parse action, set base_1=False, since the reported loc value is 0-based. indent can be a leading string (typically of spaces or tabs) to indent the numbered string passed to with_line_numbers. Added while working on #557, reported by Bernd Wechner.

• NEW/ENHANCED EXAMPLES

  • Added query syntax to mongodb_query_expression.py with:

    • better support for array fields ("contains", "contains all", "contains any", and "contains none")
    • "like" and "not like" operators to support SQL "%" wildcard matching and "=~" operator to support regex matching
    • text search using "search for"
    • dates and datetimes as query values
    • a[0] style array referencing

  • Added lox_parser.py example, a parser for the Lox language used as a tutorial in Robert Nystrom's "Crafting Interpreters" (http://craftinginterpreters.com/). With helpful corrections from RoDuth.

  • Added complex_chemical_formulas.py example, to add parsing capability for formulas such as "3(C₆H₅OH)₂".

  • Updated tag_emitter.py to use new Tag class, introduced in pyparsing 3.1.3.

Changes since 3.2.0b3:

• Fixed handling of IndexError raised in a parse action.
• QuotedString parser now handles \xnn, \ooo, and \unnnn characters when convert_whitespace_escapes is True.
• Reformatted CHANGES file for final release.

All changes in 3.2.0:

Version 3.2.0 - October, 2024

• Discontinued support for Python 3.6, 3.7, and 3.8. Adopted new Python features from Python versions 3.7-3.9:

  • Updated type annotations to use built-in container types instead of names imported from the typing module (e.g., list[str] vs List[str]).
  • Reworked portions of the packrat cache to leverage insertion-preserving ordering in dicts (including removal of uses of OrderedDict).
  • Changed pdb.set_trace() call in ParserElement.set_break() to breakpoint().
  • Converted typing.NamedTuple to dataclasses.dataclass in railroad diagramming code.
  • Added from __future__ import annotations to clean up some type annotations. (with assistance from ISyncWithFoo, issue #535, thanks for the help!)

• POSSIBLE BREAKING CHANGES

  The following bugfixes may result in subtle changes in the results returned or exceptions raised by pyparsing.

  • Fixed code in ParseElementEnhance subclasses that replaced detailed exception messages raised in contained expressions with a less-specific and less-informative generic exception message and location.

    If your code has conditional logic based on the message content in raised ParseExceptions, this bugfix may require changes in your code.

  • Fixed bug in transform_string() where whitespace in the input string was not properly preserved in the output string.

    If your code uses transform_string, this bugfix may require changes in your code.

  • Fixed bug where an IndexError raised in a parse action was incorrectly handled as an IndexError raised as part of the ParserElement parsing methods, and reraised as a ParseException. Now an IndexError that raises inside a parse action will properly propagate out as an IndexError. (Issue #573, reported by August Karlstedt, thanks!)

    If your code raises IndexErrors in parse actions, this bugfix may require changes in your code.

• FIXES AND NEW FEATURES

  • Added type annotations to remainder of pyparsing package, and added mypy run to tox.ini, so that type annotations are now run as part of pyparsing's CI. Addresses Issue #373, raised by Iwan Aucamp, thanks!

  • Exception message format can now be customized, by overriding ParseBaseException.format_message:

    def custom_exception_message(exc) -> str:
        found_phrase = f", found {exc.found}" if exc.found else ""
        return f"{exc.lineno}:{exc.column} {exc.msg}{found_phrase}"
    
    ParseBaseException.formatted_message = custom_exception_message
    

    (PR #571 submitted by Odysseyas Krystalakos, nice work!)

  • run_tests now detects if an exception is raised in a parse action, and will report it with an enhanced error message, with the exception type, string, and parse action name.

  • QuotedString now handles translation of escaped integer, hex, octal, and Unicode sequences to their corresponding characters.

  • Fixed the displayed output of Regex terms to deduplicate repeated backslashes, for easier reading in debugging, printing, and railroad diagrams.

  • Fixed (or at least reduced) elusive bug when generating railroad diagrams, where some diagram elements were just empty blocks. Fix submitted by RoDuth, thanks a ton!

  • Fixed railroad diagrams that get generated with a parser containing a Regex element defined using a verbose pattern - the pattern gets flattened and comments removed before creating the corresponding diagram element.

  • Defined a more performant regular expression used internally by common_html_entity.

  • Regex instances can now be created using a callable that takes no arguments and just returns a string or a compiled regular expression, so that creating complex regular expression patterns can be deferred until they are actually used for the first time in the parser.

  • Added optional flatten Boolean argument to ParseResults.as_list(), to return the parsed values in a flattened list.

  • Added indent and base_1 arguments to pyparsing.testing.with_line_numbers. When using with_line_numbers inside a parse action, set base_1=False, since the reported loc value is 0-based. indent can be a leading string (typically of spaces or tabs) to indent the numbered string passed to with_line_numbers. Added while working on #557, reported by Bernd Wechner.

• NEW/ENHANCED EXAMPLES

  • Added query syntax to mongodb_query_expression.py with:

    • better support for array fields ("contains", "contains all", "contains any", and "contains none")
    • "like" and "not like" operators to support SQL "%" wildcard matching and "=~" operator to support regex matching
    • text search using "search for"
    • dates and datetimes as query values
    • a[0] style array referencing

  • Added lox_parser.py example, a parser for the Lox language used as a tutorial in Robert Nystrom's "Crafting Interpreters" (http://craftinginterpreters.com/). With helpful corrections from RoDuth.

  • Added complex_chemical_formulas.py example, to add parsing capability for formulas such as "3(C₆H₅OH)₂".

  • Updated tag_emitter.py to use new Tag class, introduced in pyparsing 3.1.3.

(This is the final beta release before 3.2.0.)

• QuotedString now handles translation of escaped integer, hex, octal, and Unicode sequences to their corresponding characters.

• Added type annotations to remainder of pyparsing package, and added mypy run to tox.ini, so that type annotations are now run as part of pyparsing's CI. Addresses Issue #373, raised by Iwan Aucamp, thanks!

• Exception message format can now be customized, by overriding ParseBaseException.format_message:

  def custom_exception_message(exc) -> str:
      found_phrase = f", found {exc.found}" if exc.found else ""
      return f"{exc.lineno}:{exc.column} {exc.msg}{found_phrase}"
  
  ParseBaseException.formatted_message = custom_exception_message
  

  (PR #571 submitted by Odysseyas Krystalakos, nice work!)

• POSSIBLE BREAKING CHANGE: Fixed bug in transform_string() where whitespace in the input string was not properly preserved in the output string.

  If your code uses transform_string, this bugfix may require changes in your code.

• Fixed railroad diagrams that get generated with a parser containing a Regex element defined using a verbose pattern - the pattern gets flattened and comments removed before creating the corresponding diagram element.

• Defined a more performant regular expression used internally by common_html_entity.

• Regex instances can now be created using a callable that takes no arguments and just returns a string or a compiled regular expression, so that creating complex regular expression patterns can be deferred until they are actually used for the first time in the parser.

• Added optional flatten Boolean argument to ParseResults.as_list(), to return the parsed values in a flattened list.

• Discontinued support for Python 3.6, 3.7, and 3.8. Adopted new Python features from Python versions 3.7-3.9:

  • Updated type annotations to use built-in container types instead of names imported from the typing module (e.g., list[str] vs List[str]).
  • Reworked portions of the packrat cache to leverage insertion-preserving ordering in dicts.
  • Changed pdb.set_trace() call in ParserElement.set_break() to breakpoint().
  • Converted typing.NamedTuple to dataclasses.dataclass in railroad diagramming code.
  • Added from __future__ import annotations to clean up some type annotations.

• POSSIBLE BREAKING CHANGE: Fixed code in ParseElementEnhance subclasses that replaced detailed exception messages raised in contained expressions with a less-specific and less-informative generic exception message and location.

  If your code has conditional logic based on the message content in raised ParseExceptions, this bugfix may require changes in your code.

• Fixed the displayed output of Regex terms to deduplicate repeated backslashes, for easier reading in debugging, printing, and railroad diagrams.

• Fixed (or at least reduced) elusive bug when generating railroad diagrams, where some diagram elements were just empty blocks. Fix submitted by RoDuth, thanks a ton!

• Added indent and base_1 arguments to pyparsing.testing.with_line_numbers. When using with_line_numbers inside a parse action, set base_1=False, since the reported loc value is 0-based. indent can be a leading string (typically of spaces or tabs) to indent the numbered string passed to with_line_numbers. Added while working on #557, reported by Bernd Wechner.

• Added query syntax to mongodb_query_expression.py with better support for array fields ("contains", "contains all", "contains any", and "contains none"); and "like" and "not like" operators to support SQL "%" wildcard matching and "=~" operator to support regex matching. Also:

  • added support for dates and datetimes as query values
  • added support for a[0] style array referencing

• Added lox_parser.py example, a parser for the Lox language used as a tutorial in Robert Nystrom's "Crafting Interpreters" (http://craftinginterpreters.com/). With helpful corrections from RoDuth.

• Added complex_chemical_formulas.py example, to add parsing capability for formulas such as "3(C₆H₅OH)₂".

• Fixed a regression introduced in pyparsing 3.1.3, addition of a type annotation that referenced re.Pattern. Since this type was introduced in Python 3.7, using this type definition broke Python 3.6 installs of pyparsing 3.1.3. PR submitted by Felix Fontein, nice work!

• Added new Tag ParserElement, for inserting metadata into the parsed results. This allows a parser to add metadata or annotations to the parsed tokens. The Tag element also accepts an optional value parameter, defaulting to True. See the new tag_metadata.py example in the examples directory.

  Example:

    # add tag indicating mood
    end_punc = "." | ("!" + Tag("enthusiastic")))
    greeting = "Hello" + Word(alphas) + end_punc
  
    result = greeting.parse_string("Hello World.")
    print(result.dump())
  
    result = greeting.parse_string("Hello World!")
    print(result.dump())
  

  prints:

    ['Hello', 'World', '.']
  
    ['Hello', 'World', '!']
    - enthusiastic: True
  

• Added example mongodb_query_expression.py, to convert human-readable infix query expressions (such as a==100 and b>=200) and transform them into the equivalent query argument for the pymongo package ({'$and': [{'a': 100}, {'b': {'$gte': 200}}]}). Supports many equality and inequality operators - see the docstring for the transform_query function for more examples.

• Fixed issue where PEP8 compatibility names for ParserElement static methods were not themselves defined as staticmethods. When called using a ParserElement instance, this resulted in a TypeError exception. Reported by eylenburg (#548).

• To address a compatibility issue in RDFLib, added a property setter for the ParserElement.name property, to call ParserElement.set_name.

• Modified ParserElement.set_name() to accept a None value, to clear the defined name and corresponding error message for a ParserElement.

• Updated railroad diagram generation for ZeroOrMore and OneOrMore expressions with stop_on expressions, while investigating #558, reported by user Gu_f.

• Added <META> tag to HTML generated for railroad diagrams to force UTF-8 encoding with older browsers, to better display Unicode parser characters.

• Fixed some cosmetics/bugs in railroad diagrams:

  • fixed groups being shown even when show_groups=False
  • show results names as quoted strings when show_results_names=True
  • only use integer loop counter if repetition > 2

• Some type annotations added for parse action related methods, thanks August Karlstedt (#551).

• Added exception type to trace_parse_action exception output, while investigating SO question posted by medihack.

• Added set_name calls to internal expressions generated in infix_notation, for improved railroad diagramming.

• delta_time, lua_parser, decaf_parser, and roman_numerals examples cleaned up to use latest PEP8 names and add minor enhancements.

• Fixed bug (and corresponding test code) in delta_time example that did not handle weekday references in time expressions (like "Monday at 4pm") when the weekday was the same as the current weekday.

• Minor performance speedup in trim_arity, to benefit any parsers using parse actions.

• Added early testing support for Python 3.13 with JIT enabled.

• Support for Python 3.13.

• Added ieee_float expression to pyparsing.common, which parses float values, plus "NaN", "Inf", "Infinity". PR submitted by Bob Peterson (#538).

• Updated pep8 synonym wrappers for better type checking compatibility. PR submitted by Ricardo Coccioli (#507).

• Fixed empty error message bug, PR submitted by InSync (#534). This should return pyparsing's exception messages to a former, more helpful form. If you have code that parses the exception messages returned by pyparsing, this may require some code changes.

• Added unit tests to test for exception message contents, with enhancement to pyparsing.testing.assertRaisesParseException to accept an expected exception message.

• Updated example select_parser.py to use PEP8 names and added Groups for better retrieval of parsed values from multiple SELECT clauses.

• Added example email_address_parser.py, as suggested by John Byrd (#539).

• Added example directx_x_file_parser.py to parse DirectX template definitions, and generate a Pyparsing parser from a template to parse .x files.

• Some code refactoring to reduce code nesting, PRs submitted by InSync.

• All internal string expressions using '%' string interpolation and str.format() converted to f-strings.

• Fixed regression in Word(min), reported by Ricardo Coccioli, good catch! (Issue #502)

• Fixed bug in bad exception messages raised by Forward expressions. PR submitted by Kyle Sunden, thanks for your patience and collaboration on this (#493).

• Fixed regression in SkipTo, where ignored expressions were not checked when looking for the target expression. Reported by catcombo, Issue #500.

• Fixed type annotation for enable_packrat, PR submitted by Mike Urbach, thanks! (Issue #498)

• Some general internal code cleanup. (Instigated by Michal Čihař, Issue #488)

NOTE: In the future release 3.2.0, use of many of the pre-PEP8 methods (such as ParserElement.parseString) will start to raise DeprecationWarnings. 3.2.0 should get released some time later in 2023. I currently plan to completely drop the pre-PEP8 methods in pyparsing 4.0, though we won't see that release until at least late 2023 if not 2024. So there is plenty of time to convert existing parsers to the new function names before the old functions are completely removed. (Big help from Devin J. Pohly in structuring the code to enable this peaceful transition.)

Version 3.2.0 will also discontinue support for Python versions 3.6 and 3.7.

Version 3.1.0 - June, 2023

API CHANGES

• A slight change has been implemented when unquoting a quoted string parsed using the QuotedString class. Formerly, when unquoting and processing whitespace markers such as \t and \n, these substitutions would occur first, and then any additional '' escaping would be done on the resulting string. This would parse "\\n" as "\<newline>". Now escapes and whitespace markers are all processed in a single pass working left to right, so the quoted string "\\n" would get unquoted to "\n" (a backslash followed by "n"). Fixes issue #474 raised by jakeanq, thanks!

• Reworked delimited_list function into the new DelimitedList class. DelimitedList has the same constructor interface as delimited_list, and in this release, delimited_list changes from a function to a synonym for DelimitedList. delimited_list and the older delimitedList method will be deprecated in a future release, in favor of DelimitedList.

• ParserElement.validate() is deprecated. It predates the support for left-recursive parsers, and was prone to false positives (warning that a grammar was invalid when it was in fact valid). It will be removed in a future pyparsing release. In its place, developers should use debugging and analytical tools, such as ParserElement.set_debug() and ParserElement.create_diagram(). (Raised in Issue #444, thanks Andrea Micheli!)

NEW FEATURES AND ENHANCEMENTS

• Optional(expr) may now be written as expr | ""

  This will make this code:

  "{" + Optional(Literal("A") | Literal("a")) + "}"
  

  writable as:

  "{" + (Literal("A") | Literal("a") | "") + "}"
  

  Some related changes implemented as part of this work:

  • Literal("") now internally generates an Empty() (and no longer raises an exception)
  • Empty is now a subclass of Literal

  Suggested by Antony Lee (issue #412), PR (#413) by Devin J. Pohly.

• Added new class method ParserElement.using_each, to simplify code that creates a sequence of Literals, Keywords, or other ParserElement subclasses.

  For instance, to define suppressible punctuation, you would previously write:

  LPAR, RPAR, LBRACE, RBRACE, SEMI = map(Suppress, "(){};")
  

  You can now write:

  LPAR, RPAR, LBRACE, RBRACE, SEMI = Suppress.using_each("(){};")
  

  using_each will also accept optional keyword args, which it will pass through to the class initializer. Here is an expression for single-letter variable names that might be used in an algebraic expression:

  algebra_var = MatchFirst(
      Char.using_each(string.ascii_lowercase, as_keyword=True)
  )
  

• Added new builtin python_quoted_string, which will match any form of single-line or multiline quoted strings defined in Python. (Inspired by discussion with Andreas Schörgenhumer in Issue #421.)

• Extended expr[] notation for repetition of expr to accept a slice, where the slice's stop value indicates a stop_on expression:

  test = "BEGIN aaa bbb ccc END"
  BEGIN, END = Keyword.using_each("BEGIN END".split())
  body_word = Word(alphas)
  
  expr = BEGIN + Group(body_word[...:END]) + END
  # equivalent to
  # expr = BEGIN + Group(ZeroOrMore(body_word, stop_on=END)) + END
  
  print(expr.parse_string(test))
  

  Prints:

  ['BEGIN', ['aaa', 'bbb', 'ccc'], 'END']
  

• Added named field "url" to pyparsing.common.url, returning the entire parsed URL string.

• Added bool embed argument to ParserElement.create_diagram(). When passed as True, the resulting diagram will omit the <DOCTYPE>, <HEAD>, and <BODY> tags so that it can be embedded in other HTML source. (Useful when embedding a call to create_diagram() in a PyScript HTML page.)

• Added recurse argument to ParserElement.set_debug to set the debug flag on an expression and all of its sub-expressions. Requested by multimeric in Issue #399.

• Added '·' (Unicode MIDDLE DOT) to the set of Latin1.identbodychars.

• ParseResults now has a new method deepcopy(), in addition to the current copy() method. copy() only makes a shallow copy - any contained ParseResults are copied as references - changes in the copy will be seen as changes in the original. In many cases, a shallow copy is sufficient, but some applications require a deep copy. deepcopy() makes a deeper copy: any contained ParseResults or other mappings or containers are built with copies from the original, and do not get changed if the original is later changed. Addresses issue #463, reported by Bryn Pickering.

• Added new class property identifier to all Unicode set classes in pyparsing.unicode, using the class's values for cls.identchars and cls.identbodychars. Now Unicode-aware parsers that formerly wrote:

  ppu = pyparsing.unicode
  ident = Word(ppu.Greek.identchars, ppu.Greek.identbodychars)
  

  can now write:

  ident = ppu.Greek.identifier
  # or
  # ident = ppu.Ελληνικά.identifier
  

• Error messages from MatchFirst and Or expressions will try to give more details if one of the alternatives matches better than the others, but still fails. Question raised in Issue #464 by msdemlei, thanks!

BUG FIXES AND GENERAL CHANGES

• Added support for Python 3.12.

• Updated ci.yml permissions to limit default access to source - submitted by Joyce Brum of Google. Thanks so much!

• Updated create_diagram() code to be compatible with railroad-diagrams package version 3.0. Fixes Issue #477 (railroad diagrams generated with black bars), reported by Sam Morley-Short.

• Fixed bug in NotAny, where parse actions on the negated expr were not being run. This could cause NotAny to incorrectly fail if the expr would normally match, but would fail to match if a condition used as a parse action returned False. Fixes Issue #482, raised by byaka, thank you!

• Fixed create_diagram() to accept keyword args, to be passed through to the template.render() method to generate the output HTML (PR submitted by Aussie Schnore, good catch!)

• Fixed bug in python_quoted_string regex.

• Fixed bug when parse actions returned an empty string for an expression that had a results name, that the results name was not saved. That is:

  expr = Literal("X").add_parse_action(lambda tokens: "")("value")
  result = expr.parse_string("X")
  print(result["value"])
  

  would raise a KeyError. Now empty strings will be saved with the associated results name. Raised in Issue #470 by Nicco Kunzmann, thank you.

• Fixed bug in SkipTo where ignore expressions were not properly handled while scanning for the target expression. Issue #475, reported by elkniwt, thanks (this bug has been there for a looooong time!).

• Fixed bug in Word when max=2. Also added performance enhancement when specifying exact argument. Reported in issue #409 by panda-34, nice catch!

• Word arguments are now validated if min and max are both given, that min <= max; raises ValueError if values are invalid.

• Fixed bug in srange, when parsing escaped '/' and '' inside a range set.

• Fixed exception messages for some ParserElements with custom names, which instead showed their contained expression names.

• Fixed bug in pyparsing.common.url, when input URL is not alone on an input line. Fixes Issue #459, reported by David Kennedy.

• Multiple added and corrected type annotations. With much help from Stephen Rosen, thanks!

• Some documentation and error message clarifications on pyparsing's keyword logic, cited by Basil Peace.

• General docstring cleanup for Sphinx doc generation, PRs submitted by Devin J. Pohly. A dirty job, but someone has to do it - much appreciated!

EXAMPLE UPDATES

• Added tag_emitter.py to examples. This example demonstrates how to insert tags into your parsed results that are not part of the original parsed text.

• Added bf.py Brainf*ck parser/executor example. Illustrates using a pyparsing grammar to parse language syntax, and attach executable AST nodes to the parsed results.

• invRegex.py example renamed to inv_regex.py and updated to PEP-8 variable and method naming. PR submitted by Ross J. Duff, thanks!

• Removed examples sparser.py and pymicko.py, since each included its own GPL license in the header. Since this conflicts with pyparsing's MIT license, they were removed from the distribution to avoid confusion among those making use of them in their own projects.

• Updated the lucene_grammar.py example (better support for '*' and '?' wildcards) and corrected the test cases - brought to my attention by Elijah Nicol, good catch!

• Updated create_diagram() code to be compatible with railroad-diagrams package version 3.0. Fixes Issue #477 (railroad diagrams generated with black bars), reported by Sam Morley-Short.

• Fixed bug in NotAny, where parse actions on the negated expr were not being run. This could cause NotAny to incorrectly fail if the expr would normally match, but would fail to match if a condition used as a parse action returned False. Fixes Issue #482, raised by byaka, thank you!

• Fixed create_diagram() to accept keyword args, to be passed through to the template.render() method to generate the output HTML (PR submitted by Aussie Schnore, good catch!)

• Fixed bug in python_quoted_string regex.

• Added examples/bf.py Brainf*ck parser/executor example. Illustrates using a pyparsing grammar to parse language syntax, and attach executable AST nodes to the parsed results.

• Added support for Python 3.12.

• API CHANGE: A slight change has been implemented when unquoting a quoted string parsed using the QuotedString class. Formerly, when unquoting and processing whitespace markers such as \t and \n, these substitutions would occur first, and then any additional '' escaping would be done on the resulting string. This would parse "\n" as "<newline>". Now escapes and whitespace markers are all processed in a single pass working left to right, so the quoted string "\n" would get unquoted to "\n" (a backslash followed by "n"). Fixes issue #474 raised by jakeanq, thanks!

• Added named field "url" to pyparsing.common.url, returning the entire parsed URL string.

• Fixed bug when parse actions returned an empty string for an expression that had a results name, that the results name was not saved. That is:

  expr = Literal("X").add_parse_action(lambda tokens: "")("value")
  result = expr.parse_string("X")
  print(result["value"])
  

  would raise a KeyError. Now empty strings will be saved with the associated results name. Raised in Issue #470 by Nicco Kunzmann, thank you.

• Fixed bug in SkipTo where ignore expressions were not properly handled while scanning for the target expression. Issue #475, reported by elkniwt, thanks (this bug has been there for a looooong time!).

• Updated ci.yml permissions to limit default access to source - submitted by Joyce Brum of Google. Thanks so much!

• Updated the lucene_grammar.py example (better support for '*' and '?' wildcards) and corrected the test cases - brought to my attention by Elijah Nicol, good catch!

NOTE: In the future release 3.2.0, use of many of the pre-PEP8 methods (such as ParserElement.parseString) will start to raise DeprecationWarnings. 3.2.0 should get released some time later in 2023. I currently plan to completely drop the pre-PEP8 methods in pyparsing 4.0, though we won't see that release until at least late 2023 if not 2024. So there is plenty of time to convert existing parsers to the new function names before the old functions are completely removed. (Big help from Devin J. Pohly in structuring the code to enable this peaceful transition.)

Version 3.2.0 will also discontinue support for Python versions 3.6 and 3.7.

• API ENHANCEMENT: Optional(expr) may now be written as expr | ""

  This will make this code:

  "{" + Optional(Literal("A") | Literal("a")) + "}"
  

  writable as:

  "{" + (Literal("A") | Literal("a") | "") + "}"
  

  Some related changes implemented as part of this work:

  • Literal("") now internally generates an Empty() (and no longer raises an exception)
  • Empty is now a subclass of Literal

  Suggested by Antony Lee (issue #412), PR (#413) by Devin J. Pohly.

• Added new class property identifier to all Unicode set classes in pyparsing.unicode, using the class's values for cls.identchars and cls.identbodychars. Now Unicode-aware parsers that formerly wrote:

  ppu = pyparsing.unicode
  ident = Word(ppu.Greek.identchars, ppu.Greek.identbodychars)
  

  can now write:

  ident = ppu.Greek.identifier
  # or
  # ident = ppu.Ελληνικά.identifier
  

• Reworked delimited_list function into the new DelimitedList class. DelimitedList has the same constructor interface as delimited_list, and in this release, delimited_list changes from a function to a synonym for DelimitedList. delimited_list and the older delimitedList method will be deprecated in a future release, in favor of DelimitedList.

• Added new class method ParserElement.using_each, to simplify code that creates a sequence of Literals, Keywords, or other ParserElement subclasses.

  For instance, to define suppressable punctuation, you would previously write:

  LPAR, RPAR, LBRACE, RBRACE, SEMI = map(Suppress, "(){};")
  

  You can now write:

  LPAR, RPAR, LBRACE, RBRACE, SEMI = Suppress.using_each("(){};")
  

  using_each will also accept optional keyword args, which it will pass through to the class initializer. Here is an expression for single-letter variable names that might be used in an algebraic expression:

  algebra_var = MatchFirst(
      Char.using_each(string.ascii_lowercase, as_keyword=True)
  )
  

• Added new builtin python_quoted_string, which will match any form of single-line or multiline quoted strings defined in Python. (Inspired by discussion with Andreas Schörgenhumer in Issue #421.)

• Extended expr[] notation for repetition of expr to accept a slice, where the slice's stop value indicates a stop_on expression:

  test = "BEGIN aaa bbb ccc END"
  BEGIN, END = Keyword.using_each("BEGIN END".split())
  body_word = Word(alphas)
  
  expr = BEGIN + Group(body_word[:END]) + END
  # equivalent to
  # expr = BEGIN + Group(ZeroOrMore(body_word, stop_on=END)) + END
  
  print(expr.parse_string(test))
  

  Prints:

  ['BEGIN', ['aaa', 'bbb', 'ccc'], 'END']
  

• ParserElement.validate() is deprecated. It predates the support for left-recursive parsers, and was prone to false positives (warning that a grammar was invalid when it was in fact valid). It will be removed in a future pyparsing release. In its place, developers should use debugging and analytical tools, such as ParserElement.set_debug() and ParserElement.create_diagram(). (Raised in Issue #444, thanks Andrea Micheli!)

• Added bool embed argument to ParserElement.create_diagram(). When passed as True, the resulting diagram will omit the <DOCTYPE>, <HEAD>, and <BODY> tags so that it can be embedded in other HTML source. (Useful when embedding a call to create_diagram() in a PyScript HTML page.)

• Added recurse argument to ParserElement.set_debug to set the debug flag on an expression and all of its sub-expressions. Requested by multimeric in Issue #399.

• Added '·' (Unicode MIDDLE DOT) to the set of pp.unicode.Latin1.identbodychars.

• Fixed bug in Word when max=2. Also added performance enhancement when specifying exact argument. Reported in issue #409 by panda-34, nice catch!

• Word arguments are now validated if min and max are both given, that min <= max; raises ValueError if values are invalid.

• Fixed bug in srange, when parsing escaped '/' and '' inside a range set.

• Fixed exception messages for some ParserElements with custom names, which instead showed their contained expression names.

• Fixed bug in pyparsing.common.url, when input URL is not alone on an input line. Fixes Issue #459, reported by David Kennedy.

• Multiple added and corrected type annotations. With much help from Stephen Rosen, thanks!

• Some documentation and error message clarifications on pyparsing's keyword logic, cited by Basil Peace.

• General docstring cleanup for Sphinx doc generation, PRs submitted by Devin J. Pohly. A dirty job, but someone has to do it - much appreciated!

• invRegex.py example renamed to inv_regex.py and updated to PEP-8 variable and method naming. PR submitted by Ross J. Duff, thanks!

• Removed examples sparser.py and pymicko.py, since each included its own GPL license in the header. Since this conflicts with pyparsing's MIT license, they were removed from the distribution to avoid confusion among those making use of them in their own projects.

• Added Unicode set BasicMultilingualPlane (may also be referenced as BMP) representing the Basic Multilingual Plane (Unicode characters up to code point 65535). Can be used to parse most language characters, but omits emojis, wingdings, etc. Raised in discussion with Dave Tapley (issue #392).

• To address mypy confusion of pyparsing.Optional and typing.Optional resulting in error: "_SpecialForm" not callable message reported in issue #365, fixed the import in exceptions.py. Nice sleuthing by Iwan Aucamp and Dominic Davis-Foster, thank you! (Removed definitions of OptionalType, DictType, and IterableType and replaced them with typing.Optional, typing.Dict, and typing.Iterable throughout.)

• Fixed typo in jinja2 template for railroad diagrams, thanks for the catch Nioub (issue #388).

• Removed use of deprecated pkg_resources package in railroad diagramming code (issue #391).

• Updated bigquery_view_parser.py example to parse examples at https://cloud.google.com/bigquery/docs/reference/legacy-sql

Version 3.0.8 -

• API CHANGE: modified pyproject.toml to require Python version 3.6.8 or later for pyparsing 3.x. Earlier minor versions of 3.6 fail in evaluating the version_info class (implemented using typing.NamedTuple). If you are using an earlier version of Python 3.6, you will need to use pyparsing 2.4.7.

• Improved pyparsing import time by deferring regex pattern compiles. PR submitted by Anthony Sottile to fix issue #362, thanks!

• Updated build to use flit, PR by Michał Górny, added BUILDING.md doc and removed old Windows build scripts - nice cleanup work!

• More type-hinting added for all arithmetic and logical operator methods in ParserElement. PR from Kazantcev Andrey, thank you.

• Fixed infix_notation's definitions of lpar and rpar, to accept parse expressions such that they do not get suppressed in the parsed results. PR submitted by Philippe Prados, nice work.

• Fixed bug in railroad diagramming with expressions containing Combine elements. Reported by Jeremy White, thanks!

• Added show_groups argument to create_diagram to highlight grouped elements with an unlabeled bounding box.

• Added unicode_denormalizer.py to the examples as a demonstration of how Python's interpreter will accept Unicode characters in identifiers, but normalizes them back to ASCII so that identifiers print and 𝕡𝓻ᵢ𝓃𝘁 and 𝖕𝒓𝗂𝑛ᵗ are all equivalent.

• Removed imports of deprecated sre_constants module for catching exceptions when compiling regular expressions. PR submitted by Serhiy Storchaka, thank you.

• Fixed bug #345, in which delimitedList changed expressions in place using expr.streamline(). Reported by Kim Gräsman, thanks!

• Fixed bug #346, when a string of word characters was passed to WordStart or WordEnd instead of just taking the default value. Originally posted as a question by Parag on StackOverflow, good catch!

• Fixed bug #350, in which White expressions could fail to match due to unintended whitespace-skipping. Reported by Fu Hanxi, thank you!

• Fixed bug #355, when a QuotedString is defined with characters in its quoteChar string containing regex-significant characters such as ., *, ?, [, ], etc.

• Fixed bug in ParserElement.run_tests where comments would be displayed using with_line_numbers.

• Added optional "min" and "max" arguments to delimited_list. PR submitted by Marius, thanks!

• Added new API change note in whats_new_in_pyparsing_3_0_0, regarding a bug fix in the bool() behavior of ParseResults.

  Prior to pyparsing 3.0.x, the ParseResults class implementation of __bool__ would return False if the ParseResults item list was empty, even if it contained named results. In 3.0.0 and later, ParseResults will return True if either the item list is not empty or if the named results dict is not empty.

  # generate an empty ParseResults by parsing a blank string with
  # a ZeroOrMore
  result = Word(alphas)[...].parse_string("")
  print(result.as_list())
  print(result.as_dict())
  print(bool(result))
  
  # add a results name to the result
  result["name"] = "empty result"
  print(result.as_list())
  print(result.as_dict())
  print(bool(result))
  

  Prints:

  []
  {}
  False
  
  []
  {'name': 'empty result'}
  True
  

  In previous versions, the second call to bool() would return False.

• Minor enhancement to Word generation of internal regular expression, to emit consecutive characters in range, such as "ab", as "ab", not "a-b".

• Fixed character ranges for search terms using non-Western characters in booleansearchparser, PR submitted by tc-yu, nice work!

• Additional type annotations on public methods.

• Added suppress_warning() method to individually suppress a warning on a specific ParserElement. Used to refactor original_text_for to preserve internal results names, which, while undocumented, had been adopted by some projects.

• Fix bug when delimited_list was called with a str literal instead of a parse expression.

• Added return type annotations for col, line, and lineno.

• Fixed bug when warn_ungrouped_named_tokens_in_collection warning was raised when assigning a results name to an original_text_for expression. (Issue #110, would raise warning in packaging.)

• Fixed internal bug where ParserElement.streamline() would not return self if already streamlined.

• Changed run_tests() output to default to not showing line and column numbers. If line numbering is desired, call with with_line_numbers=True. Also fixed minor bug where separating line was not included after a test failure.

• Fixed bug in which Dict classes did not correctly return tokens as nested ParseResults, reported by and fix identified by Bu Sun Kim, many thanks!!!

• Documented API-changing side-effect of converting ParseResults to use __slots__ to pre-define instance attributes. This means that code written like this (which was allowed in pyparsing 2.4.7):

  result = Word(alphas).parseString("abc")
  result.xyz = 100
  

  now raises this Python exception:

  AttributeError: 'ParseResults' object has no attribute 'xyz'
  

  To add new attribute values to ParseResults object in 3.0.0 and later, you must assign them using indexed notation:

  result["xyz"] = 100
  

  You will still be able to access this new value as an attribute or as an indexed item.

• Fixed bug in railroad diagramming where the vertical limit would count all expressions in a group, not just those that would create visible railroad elements.

• Fixed regex typo in one_of fix for as_keyword=True.

• Fixed a whitespace-skipping bug, Issue #319, introduced as part of the revert of the LineStart changes. Reported by Marc-Alexandre Côté, thanks!

• Added header column labeling > 100 in with_line_numbers - some input lines are longer than others.

• Reverted change in behavior with LineStart and StringStart, which changed the interpretation of when and how LineStart and StringStart should match when a line starts with spaces. In 3.0.0, the xxxStart expressions were not really treated like expressions in their own right, but as modifiers to the following expression when used like LineStart() + expr, so that if there were whitespace on the line before expr (which would match in versions prior to 3.0.0), the match would fail.

  3.0.0 implemented this by automatically promoting LineStart() + expr to AtLineStart(expr), which broke existing parsers that did not expect expr to necessarily be right at the start of the line, but only be the first token found on the line. This was reported as a regression in Issue #317.

  In 3.0.2, pyparsing reverts to the previous behavior, but will retain the new AtLineStart and AtStringStart expression classes, so that parsers can chose whichever behavior applies in their specific instance. Specifically:

  # matches expr if it is the first token on the line (allows for leading whitespace)
  LineStart() + expr
  
  # matches only if expr is found in column 1
  AtLineStart(expr)
  

• Performance enhancement to one_of to always generate an internal Regex, even if caseless or as_keyword args are given as True (unless explicitly disabled by passing use_regex=False).

• IndentedBlock class now works with recursive flag. By default, the results parsed by an IndentedBlock are grouped. This can be disabled by constructing the IndentedBlock with grouped=False.

• Fixed bug where Word(max=n) did not match word groups less than length 'n'. Thanks to Joachim Metz for catching this!

• Fixed bug where ParseResults accidentally created recursive contents. Joachim Metz on this one also!

• Fixed bug where warn_on_multiple_string_args_to_oneof warning is raised even when not enabled.

Version 3.0.0 -

• A consolidated list of all the changes in the 3.0.0 release can be found in docs/whats_new_in_3_0_0.rst. (https://github.com/pyparsing/pyparsing/blob/master/docs/whats_new_in_3_0_0.rst)

Version 3.0.0.final -

• Added support for python -W warning option to call enable_all_warnings() at startup. Also detects setting of PYPARSINGENABLEALLWARNINGS environment variable to any non-blank value.

• Fixed named results returned by url to match fields as they would be parsed using urllib.parse.urlparse.

• Early response to with_line_numbers was positive, with some requested enhancements: . added a trailing "|" at the end of each line (to show presence of trailing spaces); can be customized using eol_mark argument . added expand_tabs argument, to control calling str.expandtabs (defaults to True to match parseString) . added mark_spaces argument to support display of a printing character in place of spaces, or Unicode symbols for space and tab characters . added mark_control argument to support highlighting of control characters using '.' or Unicode symbols, such as "␍" and "␊".

• Modified helpers common_html_entity and replace_html_entity() to use the HTML entity definitions from html.entities.html5.

• Updated the class diagram in the pyparsing docs directory, along with the supporting .puml file (PlantUML markup) used to create the diagram.

• Added global method autoname_elements() to call set_name() on all locally defined ParserElements that haven't been explicitly named using set_name(), using their local variable name. Useful for setting names on multiple elements when creating a railroad diagram.

        a = pp.Literal("a")
        b = pp.Literal("b").set_name("bbb")
        pp.autoname_elements()
  

  a will get named "a", while b will keep its name "bbb".

• Added url expression to pyparsing_common. (Sample code posted by Wolfgang Fahl, very nice!)

  This new expression has been added to the urlExtractorNew.py example, to show how it extracts URL fields into separate results names.

• Added method to pyparsing_testing to help debugging, with_line_numbers. Returns a string with line and column numbers corresponding to values shown when parsing with expr.set_debug():

  data = """\
     A
        100"""
  expr = pp.Word(pp.alphanums).set_name("word").set_debug()
  print(ppt.with_line_numbers(data))
  expr[...].parseString(data)
  

  prints:

                1
       1234567890
     1:   A
     2:      100
    Match word at loc 3(1,4)
         A
         ^
    Matched word -> ['A']
    Match word at loc 11(2,7)
            100
            ^
    Matched word -> ['100']
  

• Added new example cuneiform_python.py to demonstrate creating a new Unicode range, and writing a Cuneiform->Python transformer (inspired by zhpy).

• Fixed issue #272, reported by PhasecoreX, when LineStart() expressions would match expressions that were not necessarily at the beginning of a line.

  As part of this fix, two new classes have been added: AtLineStart and AtStringStart. The following expressions are equivalent:

  LineStart() + expr      and     AtLineStart(expr)
  StringStart() + expr    and     AtStringStart(expr)
  

• Fixed ParseFatalExceptions failing to override normal exceptions or expression matches in MatchFirst expressions. Addresses issue #251, reported by zyp-rgb.

• Fixed bug in which ParseResults replaces a collection type value with an invalid type annotation (changed behavior in Python 3.9). Addresses issue #276, reported by Rob Shuler, thanks.

• Fixed bug in ParseResults when calling __getattr__ for special double-underscored methods. Now raises AttributeError for non-existent results when accessing a name starting with '__'. Addresses issue #208, reported by Joachim Metz.

• Modified debug fail messages to include the expression name to make it easier to sync up match vs success/fail debug messages.

• Railroad diagrams have been reformatted: . creating diagrams is easier - call

    expr.create_diagram("diagram_output.html")
  

  create_diagram() takes 3 arguments: . the filename to write the diagram HTML . optional 'vertical' argument, to specify the minimum number of items in a path to be shown vertically; default=3 . optional 'show_results_names' argument, to specify whether results name annotations should be shown; default=False

  . every expression that gets a name using setName() gets separated out as a separate subdiagram . results names can be shown as annotations to diagram items . Each, FollowedBy, and PrecededBy elements get [ALL], [LOOKAHEAD], and [LOOKBEHIND] annotations . removed annotations for Suppress elements . some diagram cleanup when a grammar contains Forward elements . check out the examples make_diagram.py and railroad_diagram_demo.py

• Type annotations have been added to most public API methods and classes.

• Better exception messages to show full word where an exception occurred.

  Word(alphas)[...].parseString("abc 123", parseAll=True)
  

  Was:

  pyparsing.ParseException: Expected end of text, found '1'  (at char 4), (line:1, col:5)
  

  Now:

  pyparsing.exceptions.ParseException: Expected end of text, found '123'  (at char 4), (line:1, col:5)
  

• Suppress can be used to suppress text skipped using "...".

  source = "lead in START relevant text END trailing text"
  start_marker = Keyword("START")
  end_marker = Keyword("END")
  find_body = Suppress(...) + start_marker + ... + end_marker
  print(find_body.parseString(source).dump())
  

  Prints:

  ['START', 'relevant text ', 'END']
  - _skipped: ['relevant text ']
  

• New string constants identchars and identbodychars to help in defining identifier Word expressions

  Two new module-level strings have been added to help when defining identifiers, identchars and identbodychars.

  Instead of writing::

  import pyparsing as pp
  identifier = pp.Word(pp.alphas + "_", pp.alphanums + "_")
  

  you will be able to write::

  identifier = pp.Word(pp.indentchars, pp.identbodychars)
  

  Those constants have also been added to all the Unicode string classes::

  import pyparsing as pp
  ppu = pp.pyparsing_unicode
  
  cjk_identifier = pp.Word(ppu.CJK.identchars, ppu.CJK.identbodychars)
  greek_identifier = pp.Word(ppu.Greek.identchars, ppu.Greek.identbodychars)
  

• Added a caseless parameter to the CloseMatch class to allow for casing to be ignored when checking for close matches. (Issue #281) (PR by Adrian Edwards, thanks!)

• Fixed bug in Located class when used with a results name. (Issue #294)

• Fixed bug in QuotedString class when the escaped quote string is not a repeated character. (Issue #263)

• parseFile() and create_diagram() methods now will accept pathlib.Path arguments.

• PEP-8 compatible names are being introduced in pyparsing version 3.0! All methods such as parseString have been replaced with the PEP-8 compliant name parse_string. In addition, arguments such as parseAll have been renamed to parse_all. For backward-compatibility, synonyms for all renamed methods and arguments have been added, so that existing pyparsing parsers will not break. These synonyms will be removed in a future release.

  In addition, the Optional class has been renamed to Opt, since it clashes with the common typing.Optional type specifier that is used in the Python type annotations. A compatibility synonym is defined for now, but will be removed in a future release.

• HUGE NEW FEATURE - Support for left-recursive parsers! Following the method used in Python's PEG parser, pyparsing now supports left-recursive parsers when left recursion is enabled.

    import pyparsing as pp
    pp.ParserElement.enable_left_recursion()
  
    # a common left-recursion definition
    # define a list of items as 'list + item | item'
    # BNF:
    #   item_list := item_list item | item
    #   item := word of alphas
    item_list = pp.Forward()
    item = pp.Word(pp.alphas)
    item_list <<= item_list + item | item
  
    item_list.run_tests("""\
        To parse or not to parse that is the question
        """)
  

  Prints:

    ['To', 'parse', 'or', 'not', 'to', 'parse', 'that', 'is', 'the', 'question']
  

  Great work contributed by Max Fischer!

• delimited_list now supports an additional flag allow_trailing_delim, to optionally parse an additional delimiter at the end of the list. Contributed by Kazantcev Andrey, thanks!

• Removed internal comparison of results values against b"", which raised a BytesWarning when run with python -bb. Fixes issue #271 reported by Florian Bruhin, thank you!

• Fixed STUDENTS table in sql2dot.py example, fixes issue #261 reported by legrandlegrand - much better.

• Python 3.5 will not be supported in the pyparsing 3 releases. This will allow for future pyparsing releases to add parameter type annotations, and to take advantage of dict key ordering in internal results name tracking.

• API CHANGE locatedExpr is being replaced by the class Located. Located has the same constructor interface as locatedExpr, but fixes bugs in the returned ParseResults when the searched expression contains multiple tokens, or has internal results names.

  locatedExpr is deprecated, and will be removed in a future release.

• API CHANGE Diagnostic flags have been moved to an enum, pyparsing.Diagnostics, and they are enabled through module-level methods:

  • pyparsing.enable_diag()
  • pyparsing.disable_diag()
  • pyparsing.enable_all_warnings()

• API CHANGE Most previous SyntaxWarnings that were warned when using pyparsing classes incorrectly have been converted to TypeError and ValueError exceptions, consistent with Python calling conventions. All warnings warned by diagnostic flags have been converted from SyntaxWarnings to UserWarnings.

• To support parsers that are intended to generate native Python collection types such as lists and dicts, the Group and Dict classes now accept an additional boolean keyword argument aslist and asdict respectively. See the jsonParser.py example in the pyparsing/examples source directory for how to return types as ParseResults and as Python collection types, and the distinctions in working with the different types.

  In addition parse actions that must return a value of list type (which would normally be converted internally to a ParseResults) can override this default behavior by returning their list wrapped in the new ParseResults.List class:

  # this parse action tries to return a list, but pyparsing
  # will convert to a ParseResults
  def return_as_list_but_still_get_parse_results(tokens):
      return tokens.asList()
  
  # this parse action returns the tokens as a list, and pyparsing will
  # maintain its list type in the final parsing results
  def return_as_list(tokens):
      return ParseResults.List(tokens.asList())
  

  This is the mechanism used internally by the Group class when defined using aslist=True.

• A new IndentedBlock class is introduced, to eventually replace the current indentedBlock helper method. The interface is largely the same, however, the new class manages its own internal indentation stack, so it is no longer necessary to maintain an external indentStack variable.

• API CHANGE Added cache_hit keyword argument to debug actions. Previously, if packrat parsing was enabled, the debug methods were not called in the event of cache hits. Now these methods will be called, with an added argument cache_hit=True.

  If you are using packrat parsing and enable debug on expressions using a custom debug method, you can add the cache_hit=False keyword argument, and your method will be called on packrat cache hits. If you choose not to add this keyword argument, the debug methods will fail silently, behaving as they did previously.

• When using setDebug with packrat parsing enabled, packrat cache hits will now be included in the output, shown with a leading '*'. (Previously, cache hits and responses were not included in debug output.) For those using custom debug actions, see the previous item regarding an optional API change for those methods.

• setDebug output will also show more details about what expression is about to be parsed (the current line of text being parsed, and the current parse position):

    Match integer at loc 0(1,1)
      1 2 3
      ^
    Matched integer -> ['1']
  

  The current debug location will also be indicated after whitespace has been skipped (was previously inconsistent, reported in Issue #244, by Frank Goyens, thanks!).

• Modified the repr() output for ParseResults to include the class name as part of the output. This is to clarify for new pyparsing users who misread the repr output as a tuple of a list and a dict. pyparsing results will now read like:

  ParseResults(['abc', 'def'], {'qty': 100}]
  

  instead of just:

  (['abc', 'def'], {'qty': 100}]
  

• Fixed bugs in Each when passed OneOrMore or ZeroOrMore expressions: . first expression match could be enclosed in an extra nesting level . out-of-order expressions now handled correctly if mixed with required expressions . results names are maintained correctly for these expressions

• Fixed traceback trimming, and added ParserElement.verbose_traceback save/restore to reset_pyparsing_context().

• Default string for Word expressions now also include indications of min and max length specification, if applicable, similar to regex length specifications:

    Word(alphas)             -> "W:(A-Za-z)"
    Word(nums)               -> "W:(0-9)"
    Word(nums, exact=3)      -> "W:(0-9){3}"
    Word(nums, min=2)        -> "W:(0-9){2,...}"
    Word(nums, max=3)        -> "W:(0-9){1,3}"
    Word(nums, min=2, max=3) -> "W:(0-9){2,3}"
  

  For expressions of the Char class (similar to Word(..., exact=1), the expression is simply the character range in parentheses:

    Char(nums)               -> "(0-9)"
    Char(alphas)             -> "(A-Za-z)"
  

• Removed copy() override in Keyword class which did not preserve definition of ident chars from the original expression. PR #233 submitted by jgrey4296, thanks!

• In addition to pyparsing.__version__, there is now also a pyparsing.__version_info__, following the same structure and field names as in sys.version_info.

Version 3.0.0a2 - June, 2020

• Summary of changes for 3.0.0 can be found in "What's New in Pyparsing 3.0.0" documentation.

• API CHANGE Changed result returned when parsing using countedArray, the array items are no longer returned in a doubly-nested list.

• An excellent new enhancement is the new railroad diagram generator for documenting pyparsing parsers:

    import pyparsing as pp
    from pyparsing.diagram import to_railroad, railroad_to_html
    from pathlib import Path
  
    # define a simple grammar for parsing street addresses such
    # as "123 Main Street"
    #     number word...
    number = pp.Word(pp.nums).setName("number")
    name = pp.Word(pp.alphas).setName("word")[1, ...]
  
    parser = number("house_number") + name("street")
    parser.setName("street address")
  
    # construct railroad track diagram for this parser and
    # save as HTML
    rr = to_railroad(parser)
    Path('parser_rr_diag.html').write_text(railroad_to_html(rr))
  

  Very nice work provided by Michael Milton, thanks a ton!

• Enhanced default strings created for Word expressions, now showing string ranges if possible. Word(alphas) would formerly print as W:(ABCD...), now prints as W:(A-Za-z).

• Added ignoreWhitespace(recurse:bool = True) and added a recurse argument to leaveWhitespace, both added to provide finer control over pyparsing's whitespace skipping. Also contributed by Michael Milton.

• The unicode range definitions for the various languages were recalculated by interrogating the unicodedata module by character name, selecting characters that contained that language in their Unicode name. (Issue #227)

  Also, pyparsing_unicode.Korean was renamed to Hangul (Korean is also defined as a synonym for compatibility).

• Enhanced ParseResults dump() to show both results names and list subitems. Fixes bug where adding a results name would hide lower-level structures in the ParseResults.

• Added new __diag__ warnings:

  "warn_on_parse_using_empty_Forward" - warns that a Forward has been included in a grammar, but no expression was attached to it using '<<=' or '<<'

  "warn_on_assignment_to_Forward" - warns that a Forward has been created, but was probably later overwritten by erroneously using '=' instead of '<<=' (this is a common mistake when using Forwards) (currently not working on PyPy)

• Added ParserElement.recurse() method to make it simpler for grammar utilities to navigate through the tree of expressions in a pyparsing grammar.

• Fixed bug in ParseResults repr() which showed all matching entries for a results name, even if listAllMatches was set to False when creating the ParseResults originally. Reported by Nicholas42 on GitHub, good catch! (Issue #205)

• Modified refactored modules to use relative imports, as pointed out by setuptools project member jaraco, thank you!

• Off-by-one bug found in the roman_numerals.py example, a bug that has been there for about 14 years! PR submitted by Jay Pedersen, nice catch!

• A simplified Lua parser has been added to the examples (lua_parser.py).

• Added make_diagram.py to the examples directory to demonstrate creation of railroad diagrams for selected pyparsing examples. Also restructured some examples to make their parsers importable without running their embedded tests.

• Removed Py2.x support and other deprecated features. Pyparsing now requires Python 3.5 or later. If you are using an earlier version of Python, you must use a Pyparsing 2.4.x version

  Deprecated features removed: . ParseResults.asXML() - if used for debugging, switch to using ParseResults.dump(); if used for data transfer, use ParseResults.asDict() to convert to a nested Python dict, which can then be converted to XML or JSON or other transfer format

  . operatorPrecedence synonym for infixNotation - convert to calling infixNotation

  . commaSeparatedList - convert to using pyparsing_common.comma_separated_list

  . upcaseTokens and downcaseTokens - convert to using pyparsing_common.upcaseTokens and downcaseTokens

  . compat.collect_all_And_tokens will not be settable to False to revert to pre-2.3.1 results name behavior - review use of names for MatchFirst and Or expressions containing And expressions, as they will return the complete list of parsed tokens, not just the first one. Use __diag__.warn_multiple_tokens_in_named_alternation to help identify those expressions in your parsers that will have changed as a result.

• Removed support for running python setup.py test. The setuptools maintainers consider the test command deprecated (see https://github.com/pypa/setuptools/issues/1684). To run the Pyparsing test, use the command tox.

• API CHANGE: The staticmethod ParseException.explain has been moved to ParseBaseException.explain_exception, and a new explain instance method added to ParseBaseException. This will make calls to explain much more natural:

  try:
      expr.parseString("...")
  except ParseException as pe:
      print(pe.explain())
  

• POTENTIAL API CHANGE: ZeroOrMore expressions that have results names will now include empty lists for their name if no matches are found. Previously, no named result would be present. Code that tested for the presence of any expressions using "if name in results:" will now always return True. This code will need to change to "if name in results and results[name]:" or just "if results[name]:". Also, any parser unit tests that check the asDict() contents will now see additional entries for parsers having named ZeroOrMore expressions, whose values will be [].

• POTENTIAL API CHANGE: Fixed a bug in which calls to ParserElement.setDefaultWhitespaceChars did not change whitespace definitions on any pyparsing built-in expressions defined at import time (such as quotedString, or those defined in pyparsing_common). This would lead to confusion when built-in expressions would not use updated default whitespace characters. Now a call to ParserElement.setDefaultWhitespaceChars will also go and update all pyparsing built-ins to use the new default whitespace characters. (Note that this will only modify expressions defined within the pyparsing module.) Prompted by work on a StackOverflow question posted by jtiai.

• Expanded diag and compat to actual classes instead of just namespaces, to add some helpful behavior:

  • enable() and .disable() methods to give extra help when setting or clearing flags (detects invalid flag names, detects when trying to set a compat flag that is no longer settable). Use these methods now to set or clear flags, instead of directly setting to True or False.

    import pyparsing as pp
    pp.__diag__.enable("warn_multiple_tokens_in_named_alternation")
    

  • diag.enable_all_warnings() is another helper that sets all "warn*" diagnostics to True.

    pp.__diag__.enable_all_warnings()
    

  • added new warning, "warn_on_match_first_with_lshift_operator" to warn when using '<<' with a '|' MatchFirst operator, which will create an unintended expression due to precedence of operations.

    Example: This statement will erroneously define the fwd expression as just expr_a, even though expr_a | expr_b was intended, since '<<' operator has precedence over '|':

    fwd << expr_a | expr_b
    

    To correct this, use the '<<=' operator (preferred) or parentheses to override operator precedence:

    fwd <<= expr_a | expr_b
             or
    fwd << (expr_a | expr_b)
    

• Cleaned up default tracebacks when getting a ParseException when calling parseString. Exception traces should now stop at the call in parseString, and not include the internal traceback frames. (If the full traceback is desired, then set ParserElement.verbose_traceback to True.)

• Fixed FutureWarnings that sometimes are raised when '[' passed as a character to Word.

• New namespace, assert methods and classes added to support writing unit tests.

  • assertParseResultsEquals
  • assertParseAndCheckList
  • assertParseAndCheckDict
  • assertRunTestResults
  • assertRaisesParseException
  • reset_pyparsing_context context manager, to restore pyparsing config settings

• Enhanced error messages and error locations when parsing fails on the Keyword or CaselessKeyword classes due to the presence of a preceding or trailing keyword character. Surfaced while working with metaperl on issue #201.

• Enhanced the Regex class to be compatible with re's compiled with the re-equivalent regex module. Individual expressions can be built with regex compiled expressions using:

  import pyparsing as pp import regex

  would use regex for this expression

  integer_parser = pp.Regex(regex.compile(r'\d+'))

  Inspired by PR submitted by bjrnfrdnnd on GitHub, very nice!

• Fixed handling of ParseSyntaxExceptions raised as part of Each expressions, when sub-expressions contain '-' backtrack suppression. As part of resolution to a question posted by John Greene on StackOverflow.

• Potentially huge performance enhancement when parsing Word expressions built from pyparsing_unicode character sets. Word now internally converts ranges of consecutive characters to regex character ranges (converting "0123456789" to "0-9" for instance), resulting in as much as 50X improvement in performance! Work inspired by a question posted by Midnighter on StackOverflow.

• Improvements in select_parser.py, to include new SQL syntax from SQLite. PR submitted by Robert Coup, nice work!

• Fixed bug in PrecededBy which caused infinite recursion, issue #127 submitted by EdwardJB.

• Fixed bug in CloseMatch where end location was incorrectly computed; and updated partial_gene_match.py example.

• Fixed bug in indentedBlock with a parser using two different types of nested indented blocks with different indent values, but sharing the same indent stack, submitted by renzbagaporo.

• Fixed bug in Each when using Regex, when Regex expression would get parsed twice; issue #183 submitted by scauligi, thanks!

• BigQueryViewParser.py added to examples directory, PR submitted by Michael Smedberg, nice work!

• booleansearchparser.py added to examples directory, PR submitted by xecgr. Builds on searchparser.py, adding support for '*' wildcards and non-Western alphabets.

• Fixed bug in delta_time.py example, when using a quantity of seconds/minutes/hours/days > 999.

• Fixed bug in regex definitions for real and sci_real expressions in pyparsing_common. Issue #194, reported by Michael Wayne Goodman, thanks!

• Fixed FutureWarning raised beginning in Python 3.7 for Regex expressions containing '[' within a regex set.

• Minor reformatting of output from runTests to make embedded comments more visible.

• And finally, many thanks to those who helped in the restructuring of the pyparsing code base as part of this release. Pyparsing now has more standard package structure, more standard unit tests, and more standard code formatting (using black). Special thanks to jdufresne, klahnakoski, mattcarmody, and ckeygusuz, to name just a few.

Version 2.4.7 - April, 2020

• Backport of selected fixes from 3.0.0 work: . Each bug with Regex expressions . And expressions not properly constructing with generator . Traceback abbreviation . Bug in delta_time example . Fix regexen in pyparsing_common.real and .sci_real . Avoid FutureWarning on Python 3.7 or later . Cleanup output in runTests if comments are embedded in test string

Version 2.4.6 - December, 2019

• Fixed typos in White mapping of whitespace characters, to use correct "\u" prefix instead of "u".

• Fix bug in left-associative ternary operators defined using infixNotation. First reported on StackOverflow by user Jeronimo.

• Backport of pyparsing_test namespace from 3.0.0, including TestParseResultsAsserts mixin class defining unittest-helper methods: . def assertParseResultsEquals( self, result, expected_list=None, expected_dict=None, msg=None) . def assertParseAndCheckList( self, expr, test_string, expected_list, msg=None, verbose=True) . def assertParseAndCheckDict( self, expr, test_string, expected_dict, msg=None, verbose=True) . def assertRunTestResults( self, run_tests_report, expected_parse_results=None, msg=None) . def assertRaisesParseException(self, exc_type=ParseException, msg=None)

  To use the methods in this mixin class, declare your unittest classes as:

  from pyparsing import pyparsing_test as ppt class MyParserTest(ppt.TestParseResultsAsserts, unittest.TestCase): ...

Version 2.4.5 - November, 2019

• Fixed encoding when setup.py reads README.rst to include the project long description when uploading to PyPI. A stray unicode space in README.rst prevented the source install on systems whose default encoding is not 'utf-8'.

Check-in bug in Pyparsing 2.4.3 that raised UserWarnings was masked by stdout buffering in unit tests - fixed.

Version 2.4.3 - November, 2019

(Backport of selected critical items from 3.0.0 development branch.)

• Fixed a bug in ParserElement.__eq__ that would for some parsers create a recursion error at parser definition time. Thanks to Michael Clerx for the assist. (Addresses issue #123)

• Fixed bug in indentedBlock where a block that ended at the end of the input string could cause pyparsing to loop forever. Raised as part of discussion on StackOverflow with geckos.

• Backports from pyparsing 3.0.0: . __diag__.enable_all_warnings() . Fixed bug in PrecededBy which caused infinite recursion, issue #127 . support for using regex-compiled RE to construct Regex expressions

Version 2.4.2 - July, 2019

• Updated the shorthand notation that has been added for repetition expressions: expr[min, max], with '...' valid as a min or max value:

  • expr[...] and expr[0, ...] are equivalent to ZeroOrMore(expr)
  • expr[1, ...] is equivalent to OneOrMore(expr)
  • expr[n, ...] or expr[n,] is equivalent to expr*n + ZeroOrMore(expr) (read as "n or more instances of expr")
  • expr[..., n] is equivalent to expr*(0, n)
  • expr[m, n] is equivalent to expr*(m, n) Note that expr[..., n] and expr[m, n] do not raise an exception if more than n exprs exist in the input stream. If this behavior is desired, then write expr[..., n] + ~expr.

  Better interpretation of [...] as ZeroOrMore raised by crowsonkb, thanks for keeping me in line!

  If upgrading from 2.4.1 or 2.4.1.1 and you have used expr[...] for OneOrMore(expr), it must be updated to expr[1, ...].

• The defaults on all the __diag__ switches have been set to False, to avoid getting alarming warnings. To use these diagnostics, set them to True after importing pyparsing.

  Example:

  import pyparsing as pp
  pp.__diag__.warn_multiple_tokens_in_named_alternation = True
  

• Fixed bug introduced by the use of getitem for repetition, overlooking Python's legacy implementation of iteration by sequentially calling getitem with increasing numbers until getting an IndexError. Found during investigation of problem reported by murlock, merci!

Release candidate for 2.4.2:

• FIxes incorrect implementation of expr[…] as OneOrMore, changed to ZeroOrMore
• Fixes __getitem__-induced iterability for ParserElement class
• __diag__ flags are now all False by default

This is a re-release of version 2.4.1 to restore the release history in PyPI, since the 2.4.1 release was deleted.

There are 3 known issues in this release, which are fixed in the upcoming 2.4.2:

• API change adding support for expr[...] - the original code in 2.4.1 incorrectly implemented this as OneOrMore. Code using this feature under this relase should explicitly use expr[0, ...] for ZeroOrMore and expr[1, ...] for OneOrMore. In 2.4.2 you will be able to write expr[...] equivalent to ZeroOrMore(expr).

• Bug if composing And, Or, MatchFirst, or Each expressions using an expression. This only affects code which uses explicit expression construction using the And, Or, etc. classes instead of using overloaded operators '+', '^', and so on. If constructing an And using a single expression, you may get an error that "cannot multiply ParserElement by 0 or (0, 0)" or a Python IndexError. Change code like

  cmd = Or(Word(alphas))
  

  to

  cmd = Or([Word(alphas)])
  

  (Note that this is not the recommended style for constructing Or expressions.)

• Some newly-added __diag__ switches are enabled by default, which may give rise to noisy user warnings for existing parsers. You can disable them using:

  import pyparsing as pp
  pp.__diag__.warn_multiple_tokens_in_named_alternation = False
  pp.__diag__.warn_ungrouped_named_tokens_in_collection = False
  pp.__diag__.warn_name_set_on_empty_Forward = False
  pp.__diag__.warn_on_multiple_string_args_to_oneof = False
  pp.__diag__.enable_debug_on_named_expressions = False
  

  In 2.4.2 these will all be set to False by default.

• Well, it looks like the API change that was introduced in 2.3.1 was more drastic than expected, so for a friendlier forward upgrade path, this release: . Bumps the current version number to 2.4.0, to reflect this incompatible change. . Adds a pyparsing.__compat__ object for specifying compatibility with future breaking changes. . Conditionalizes the API-breaking behavior, based on the value pyparsing.__compat__.collect_all_And_tokens. By default, this value will be set to True, reflecting the new bugfixed behavior. To set this value to False, add to your code:

    import pyparsing
    pyparsing.__compat__.collect_all_And_tokens = False
  

  . User code that is dependent on the pre-bugfix behavior can restore it by setting this value to False.

  In 2.5 and later versions, the conditional code will be removed and setting the flag to True or False in these later versions will have no effect.

• Updated unitTests.py and simple_unit_tests.py to be compatible with python setup.py test. To run tests using setup, do:

  python setup.py test
  python setup.py test -s unitTests.suite
  python setup.py test -s simple_unit_tests.suite
  

  Prompted by issue #83 and PR submitted by bdragon28, thanks.

• Fixed bug in ParserElement.runTests handling '\n' literals in quoted strings.

• Added tag_body attribute to the start tag expressions generated by makeHTMLTags, so that you can avoid using SkipTo to roll your own tag body expression:

  a, aEnd = pp.makeHTMLTags('a')
  link = a + a.tag_body("displayed_text") + aEnd
  for t in s.searchString(html_page):
      print(t.displayed_text, '->', t.startA.href)
  

• indentedBlock failure handling was improved; PR submitted by TMiguelT, thanks!

• Address Py2 incompatibility in simple_unit_tests, plus explain() and Forward str() cleanup; PRs graciously provided by eswald.

• Fixed docstring with embedded '\w', which creates SyntaxWarnings in Py3.8, issue #80.

• Examples:

  • Added example parser for rosettacode.org tutorial compiler.

  • Added example to show how an HTML table can be parsed into a collection of Python lists or dicts, one per row.

  • Updated SimpleSQL.py example to handle nested selects, reworked 'where' expression to use infixNotation.

  • Added include_preprocessor.py, similar to macroExpander.py.

  • Examples using makeHTMLTags use new tag_body expression when retrieving a tag's body text.

  • Updated examples that are runnable as unit tests:

    python setup.py test -s examples.antlr_grammar_tests
    python setup.py test -s examples.test_bibparse
    

New features in Pyparsing 2.3.1 -

• ParseException.explain() method, to convert a raw Python traceback into a list of the parse expressions leading up to a parse mismatch.

• New unicode sets Latin-A and Latin-B, and the ability to define custom sets using multiple inheritance.

    class Turkish_set(pp.pyparsing_unicode.Latin1, pp.pyparsing_unicode.LatinA):
        pass
  
    turkish_word = pp.Word(Turkish_set.alphas)
  

• State machine examples, showing how to extend Python with your own pyparsing-enabled syntax. The examples implement a 'statemachine' keyword to define a set of classes and transition attribute to implement a State pattern:

    statemachine TrafficLightState:
        Red -> Green
        Green -> Yellow
        Yellow -> Red
  

  Transitions can be named also:

    statemachine LibraryBookState:
        New -(shelve)-> Available
        Available -(reserve)-> OnHold
        OnHold -(release)-> Available
        Available -(checkout)-> CheckedOut
        CheckedOut -(checkin)-> Available
  

• Example parser for decaf language. This language is commonly used in university CS compiler classes.

• Fixup of docstrings to Sphinx format, so pyparsing docs are now available on readthedocs.com! (https://pyparsing-docs.readthedocs.io/en/latest/)

• NEW SUPPORT FOR UNICODE CHARACTER RANGES This release introduces the pyparsing_unicode namespace class, defining a series of language character sets to simplify the definition of alphas, nums, alphanums, and printables in the following language sets: . Arabic . Chinese . Cyrillic . Devanagari . Greek . Hebrew . Japanese (including Kanji, Katakana, and Hirigana subsets) . Korean . Latin1 (includes 7 and 8-bit Latin characters) . Thai . CJK (combination of Chinese, Japanese, and Korean sets)

POSSIBLE API CHANGES:

• IndexErrors raised in parse actions are now wrapped in ParseExceptions
• ParseResults have had several bugfixes which remove erroneous nesting levels See the CHANGES file for more details.

New classes:

• PrecededBy - lookbehind match
• Char - single character match (similar to Word(exact=1))

Version 2.2.2 - September, 2018

• Fixed bug in SkipTo, if a SkipTo expression that was skipping to an expression that returned a list (such as an And), and the SkipTo was saved as a named result, the named result could be saved as a ParseResults - should always be saved as a string. Issue #28, reported by seron.

• Added simple_unit_tests.py, as a collection of easy-to-follow unit tests for various classes and features of the pyparsing library. Primary intent is more to be instructional than actually rigorous testing. Complex tests can still be added in the unitTests.py file.

• New features added to the Regex class:

  • optional asGroupList parameter, returns all the capture groups as a list

  • optional asMatch parameter, returns the raw re.match result

  • new sub(repl) method, which adds a parse action calling re.sub(pattern, repl, parsed_result). Simplifies creating Regex expressions to be used with transformString. Like re.sub, repl may be an ordinary string (similar to using pyparsing's replaceWith), or may contain references to capture groups by group number, or may be a callable that takes an re match group and returns a string.

    For instance:

    expr = pp.Regex(r"([Hh]\d):\s*(.*)").sub(r"<\1>\2</\1>")
    expr.transformString("h1: This is the title")
    

    will return

    <h1>This is the title</h1>
    

• Fixed omission of LICENSE file in source tarball, also added CODE_OF_CONDUCT.md per GitHub community standards. Issue #31

• Updates to migrate source repo to GitHub
• Fix deprecation warning in Python 3.7 re: importing collections.abc
• Fix Literal/Keyword bug raising IndexError instead of ParseException

• Bumped minor version number to reflect compatibility issues with OneOrMore and ZeroOrMore bugfixes in 2.1.10. (2.1.10 fixed a bug that was introduced in 2.1.4, but the fix could break code written against 2.1.4 - 2.1.9.)

• Updated setup.py to address recursive import problems now that pyparsing is part of 'packaging' (used by setuptools). Patch submitted by Joshua Root, much thanks!

• Fixed KeyError issue reported by Yann Bizeul when using packrat parsing in the Graphite time series database, thanks Yann!

• Fixed incorrect usages of '' in literals, as described in https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior Patch submitted by Ville Skyttä - thanks!

• Minor internal change when using '-' operator, to be compatible with ParserElement.streamline() method.

• Expanded infixNotation to accept a list or tuple of parse actions to attach to an operation.

• New unit test added for dill support for storing pyparsing parsers. Ordinary Python pickle can be used to pickle pyparsing parsers as long as they do not use any parse actions. The 'dill' module is an extension to pickle which does support pickling of attached parse actions.

• Fixed bug in reporting named parse results for ZeroOrMore expressions, thanks Ethan Nash for reporting this!

• Fixed behavior of LineStart to be much more predictable. LineStart can now be used to detect if the next parse position is col 1, factoring in potential leading whitespace (which would cause LineStart to fail). Also fixed a bug in col, which is used in LineStart, where '\n's were erroneously considered to be column 1.

• Added support for multiline test strings in runTests.

• Fixed bug in ParseResults.dump when keys were not strings. Also changed display of string values to show them in quotes, to help distinguish parsed numeric strings from parsed integers that have been converted to Python ints.

• Added class CloseMatch, a variation on Literal which matches "close" matches, that is, strings with at most 'n' mismatching characters.

• Fixed bug in Keyword.setDefaultKeywordChars(), reported by Kobayashi Shinji - nice catch, thanks!

• Minor API change in pyparsing_common. Renamed some of the common expressions to PEP8 format (to be consistent with the other pyparsing_common expressions): . signedInteger -> signed_integer . sciReal -> sci_real

  Also, in trying to stem the API bloat of pyparsing, I've copied some of the global expressions and helper parse actions into pyparsing_common, with the originals to be deprecated and removed in a future release: . commaSeparatedList -> pyparsing_common.comma_separated_list . upcaseTokens -> pyparsing_common.upcaseTokens . downcaseTokens -> pyparsing_common.downcaseTokens

  (I don't expect any other expressions, like the comment expressions, quotedString, or the Word-helping strings like alphas, nums, etc. to migrate to pyparsing_common - they are just too pervasive. As for the PEP8 vs camelCase naming, all the expressions are PEP8, while the parse actions in pyparsing_common are still camelCase. It's a small step - when pyparsing 3.0 comes around, everything will change to PEP8 snake case.)

• Fixed Python3 compatibility bug when using dict keys() and values() in ParseResults.getName().

• After some prodding, I've reworked the unitTests.py file for pyparsing over the past few releases. It uses some variations on unittest to handle my testing style. The test now: . auto-discovers its test classes (while maintining their order of definition) . suppresses voluminous 'print' output for tests that pass

• Fixed issue in the optimization to _trim_arity, when the full stacktrace is retrieved to determine if a TypeError is raised in pyparsing or in the caller's parse action. Code was traversing the full stacktrace, and potentially encountering UnicodeDecodeError.

• Fixed bug in ParserElement.inlineLiteralsUsing, causing infinite loop with Suppress.

• Fixed bug in Each, when merging named results from multiple expressions in a ZeroOrMore or OneOrMore. Also fixed bug when ZeroOrMore expressions were erroneously treated as required expressions in an Each expression.

• Added a few more inline doc examples.

• Improved use of runTests in several example scripts.

• Fixed regression reported by Andrea Censi (surfaced in PyContracts tests) when using ParseSyntaxExceptions (raised when using operator '-') with packrat parsing.

• Minor fix to oneOf, to accept all iterables, not just space-delimited strings and lists. (If you have a list or set of strings, it is not necessary to concat them using ' '.join to pass them to oneOf, oneOf will accept the list or set or generator directly.)

• Major packrat upgrade, inspired by patch provided by Tal Einat - many, many, thanks to Tal for working on this! Tal's tests show faster parsing performance (2X in some tests), and memory reduction from 3GB down to ~100MB! Requires no changes to existing code using packratting. (Uses OrderedDict, available in Python 2.7 and later. For Python 2.6 users, will attempt to import from ordereddict backport. If not present, will implement pure-Python Fifo dict.)

• Minor API change - to better distinguish between the flexible numeric types defined in pyparsing_common, I've changed "numeric" (which parsed numbers of different types and returned int for ints, float for floats, etc.) and "number" (which parsed numbers of int or float type, and returned all floats) to "number" and "fnumber" respectively. I hope the "f" prefix of "fnumber" will be a better indicator of its internal conversion of parsed values to floats, while the generic "number" is similar to the flexible number syntax in other languages. Also fixed a bug in pyparsing_common.numeric (now renamed to pyparsing_common.number), integers were parsed and returned as floats instead of being retained as ints.

• Fixed bug in upcaseTokens and downcaseTokens introduced in 2.1.5, when the parse action was used in conjunction with results names. Reported by Steven Arcangeli from the dql project, thanks for your patience, Steven!

• Major change to docs! After seeing some comments on reddit about general issue with docs of Python modules, and thinking that I'm a little overdue in doing some doc tuneup on pyparsing, I decided to following the suggestions of the redditor and add more inline examples to the pyparsing reference documentation. I hope this addition will clarify some of the more common questions people have, especially when first starting with pyparsing/Python.

• Deprecated ParseResults.asXML. I've never been too happy with this method, and it usually forces some unnatural code in the parsers in order to get decent tag names. The amount of guesswork that asXML has to do to try to match names with values should have been a red flag from day one. If you are using asXML, you will need to implement your own ParseResults->XML serialization. Or consider migrating to a more current format such as JSON (which is very easy to do: results_as_json = json.dumps(parse_result.asDict()) Hopefully, when I remove this code in a future version, I'll also be able to simplify some of the craziness in ParseResults, which IIRC was only there to try to make asXML work.

• Updated traceParseAction parse action decorator to show the repr of the input and output tokens, instead of the str format, since str has been simplified to just show the token list content.

  (The change to ParseResults.str occurred in pyparsing 2.0.4, but it seems that didn't make it into the release notes - sorry! Too many users, especially beginners, were confused by the "([token_list], {names_dict})" str format for ParseResults, thinking they were getting a tuple containing a list and a dict. The full form can be seen if using repr().)

  For tracing tokens in and out of parse actions, the more complete repr form provides important information when debugging parse actions.