Fixed

• Parsing slow when there are many table-like lines on input (#254).
• sdist is missing test-requirements.txt (#253).

Fixed

• LaTeXRenderer: Render thematic break on a new line (#224).
• LaTeXRenderer: Properly escape caret ("^") within raw text (#234).
• Incomplete configuration for tox and tests (#220, #221).
• Add types to __init__.py's markdown function (#251).

Documentation

• Fix typos in the in-line documentation, while also resolving problems within the docs folder (#226).
• Describe Document's constructor and how end-of-lines are treated (#64).

Added

• Add parent attribute/property (#71 via #206).

  COMPATIBILITY REMARKS:

  • As the children attribute changed to property, existing code needs to be changed like this:

    -hasattr(token, 'children')
    +token.children is not None
    
    # ...
    
    -'children' in vars(token)
    +token.children is not None
    
    # ...
    
    -getattr(token, 'children', [])
    +token.children or []

Fixed

• Incomplete sdist content (#219).

Changed

• MarkdownRenderer: Keep the original content spacing before the list marker (#213 via #215).
• Bump build/CI scripts.

Added

• Add line numbers (line_number attribute) on all block tokens during parsing (#188).
• More thorough documentation and explanation of AST and tokens in the Developer's Guide (within #188).
• HtmlRenderer: Option to skip HTML tokens parsing (#74 via #204). Just pass process_html_tokens=False to the renderer's constructor.
• LaTeXRenderer: Add AMS packages for Math (#207).

Fixed

• MarkdownRenderer: Penultimate lines of multiline fragments being ignored (#201).
• MathJaxRenderer: Output inline math (in: $...$) correctly (out: \(...\)) (#195).
• Make local installation work again with pip 23.2 and above (fb231b1).

Changed

• MarkdownRenderer: Keep the original content spacing after the list marker (#196 via #197).

  COMPATIBILITY REMARKS:

  • The old behavior can be enforced by passing normalize_whitespace=True to the renderer's constructor (#202).
  • Provided you create ListItem's tokens directly via its constructor, you need to pass it a new parameter called indentation (number of spaces before the item marker):

    -def __init__(self, parse_buffer, prepend, leader):
    +def __init__(self, parse_buffer, indentation, prepend, leader):

• Allow just 1 dash in Table delimiter cells (#131 via #203).

For contributors:

• Improvements in the Python styling checks, setup and documentation - see diff.

Fixed

• In v1.2.0, due to #182, using HTMLRenderer (the old class name) directly from the mistletoe package stopped working.

Fixed

• FileWrapper's methods anchor/reset are DEPRECATED, use a more versatile get_pos/set_pos approach instead (#186).
• LaTeXRenderer: Escape special characters in URLs of generated links (#114 via #190).

Changed

• Do not escape double or single quotes by default (#181).
• As in GFM, tables are allowed to interrupt paragraphs by default (#166 via #187). Set mistletoe.block_token.Table.interrupt_paragraph to false if you need the old behavior.
• PygmentsRenderer no longer throws a ClassNotFound error if there is a code language specified which is not supported by the Pygments highlighter (#183). If you still need to get the error, simply pass fail_on_unsupported_language=True to the renderer's constructor.
• Use strict PascalCase for class names (#182). The old names like HTMLRenderer are marked as deprecated.

For contributors:

• Unify assertEqual() calls in all the tests (#185).
• Starting to use requirements.txt.
• Scheme renderer's __main__ section turned into a proper unit test (#189).
• The possibility of a block token to interrupt a paragraph was generalized (#187). A token class just needs to define the check_interrupts_paragraph(cls, lines) -> bool method.

Added

• MarkdownRenderer - the long awaited renderer for outputting parsed AST back to the Markdown syntax (#4, #162). 🎉
  For this, parsing was extended to store additional information about the input markdown.
• New options to control whether HTMLRenderer should escape double and/or single quotes (#176).

Fixed

• Token.__repr__() no longer outputs class attributes (#172).
• Make Scheme (scheme.py) work again (7883f58).
• Make CI work again (#121). Good news for both the maintainers and the contributors.

Changed

• "Simpler is better!": Revert back to HTMLRenderer not escaping single quotes by default (#176).
  Note that we plan to extend this to not escaping double quotes in the next version.
• As a side effect of #162, the AST token classes and parsing process was changed for more uniform behavior and API
  (#163).
• ASTRenderer no longer outputs all token attributes, but only those listed in Token.repr_attributes (#172).

Special thanks go to @anderskaplan as the most active contributor of this release. 💪

Fixed:

• Do include package "mistletoe.contrib" in the built artifacts (#177).

Updated:

• Make CONTRIBUTING.md more up-to-date.

WARNING - Backwards compatibility changes:

• #167: For practical reasons and for following common packaging practices, contrib folder got moved under the mistletoe folder / package. So if you reference a renderer from that folder, you need to reference it as mistletoe.contrib.<renderer> now.
• See "change handling of (white)space characters in code spans" below. We keep processing extra whitespace characters at the parsing level, so that all renderers can benefit from it out-of-the-box. Provided that a custom renderer, for whatever reason, relied on all the spaces being collapsed, it needs to do that collapsing itself now (e.g. ' '.join(re.split('[ \n]+', content.strip())), or re.sub('[ \n]+', ' ', content.strip())).

Added:

• JIRARenderer: Support link title notation, i.e. [label](url "title") gets transformed to [label|url|title] (#161)

Fixed:

• Make the traverse() function actually work with various input parameters:
  • filtering by class (#157, #158)
  • filtering by depth (#159)
• Compatibility with the latest CommonMark specification v0.30 (#108):
  • Change handling of (white)space characters in code spans (#156; in line with commonmark/commonmark-spec#532 and commonmark/commonmark-spec#569)
  • Fix handling of tabs and parsing continuation lines within list items (#89 via #164)
  • Fix the other examples, mostly edge cases, from the spec (see #165 and #168 for details; #173 seems to be fixed as well)
• Make parsing of link reference definitions (a.k.a. Footnotes) more strict - spec compliant (#132)
• Fix handling of special punctuation characters when parsing emphasis (#96, #175)

Updated:

• Smaller inner working refactorings, like #171.

WARNING - Backwards compatibility changes:

• Python versions below 3.5 are no longer supported (Python 3.6 end-of-life: December 2021)
  • html module (available since Python 3.4) is no longer included
• As unescaping of HTML character references (entities) is now correctly done in parsing phase already, prospective custom renderers should be altered accordingly provided they do the unescaping themselves now.
• HTMLRenderer: single quote is no longer rendered as ', but as ' (see #115; let us know if you would need the old behavior)
• BaseRenderer.__getattr__() is removed and replaced by explicit render_*() methods definitions for clearer API (#133)

Added:

• Add __repr__() methods to all token classes (#140)
• Add type hints for HTMLRenderer methods (#133; supported since Python 3.0)

Fixed:

• Correctly unescape HTML character references (entities) for LaTeXRenderer - refactored globally (#135)
• Ensure LaTeX renderer uses valid \verb delimiter - not always just | (#149)
• GithubWiki unit test failing when run via pytest (#142)

Others:

• Simplify implementation of escaping special HTML characters (#135)
• Remove unused imports and variables (#146)
• Document (for maintainers) how to create a new mistletoe release
• Fix and extend docstring documentation of various token types (#154)
• This version is about 0,5% faster according to the benchmark test. :)

Fixed:

• Support emphasized inline code (#70)
• Failure to parse paragraph containing just "[" (#130) (a side-effect of the fix of #124 in v0.8.1)
• IndexError when parsing unfinished-link-like text (#98)

Others:

• Small documentation improvements.

Added:

• Documentation (#122 - covering #56, #99 and some other basic topics)

Fixed:

• Avoid infinite loop when parsing specific Footnotes (#124)
• Read and write to files / console in UTF-8 (in all remaining locations)

Testing:

• Benchmark tests made up-to-date (#119)

Fixed:

• Fixed incorrect handling of loose list (#54, #65, thanks @Rogdham and @Vallentin)
• Fixed FileWrapper backstep after StopIteration (#58, thanks @Rogdham)
• Allow more than one level of token subclass (#62, thanks @Rogdham)
• Tables can handle rows with missing columns (#67, thanks @Grollicus)
• Fixed unresolved reference (#73, thanks @Vallentin)
• Fixed EOL markers in LaTeX tables (#79, thanks @liuq)

Testing:

• Add Python 3.7 to integration testing (#63, thanks @nikolas)

Fixed:

• only matching the first instance of InlineCode (#50, thanks @huettenhain);
• normalize newlines after every line (#51, thanks @elebow and @rsrdesarrollo);
• trailing characters after reference definition.

Performance:

• small speed boost to ParseToken.append_child.

Warning: this is a release that breaks backwards compatibility in non-trivial ways (hopefully for the last time!) Read the full release notes if you are updating from a previous version.

Features:

• all tests passing in CommonMark test suite (finally! :tada:)
• allow specifying span token precedence levels;
• new and shiny span_tokenizer.tokenize.

Fixed:

• well, all the CommonMark test cases..
• ASTRenderer crashes on tables with headers (#48, thanks @timfox456!)

Where I break backwards compatibility:

Previously span-level tokens need to have their children attribute manually specified. This is no longer the case, as the children attribute will automatically be set based on the class variable parse_group, which correspond to the regex match group in which child tokens might occur.

As an example, previously GithubWiki is implemented as this:

from mistletoe.span_token import SpanToken, tokenize_inner
import re

class GithubWiki(SpanToken):
    pattern = re.compile(r'...')
    def __init__(self, match_obj):
        super().__init__(match_obj) 
        # alternatively, self.children = tokenize_inner(match_obj.group(1))
        self.target = match_obj.group(2)

Now we can write:

from mistletoe.span_token import SpanToken
import re

class GithubWiki(SpanToken):
    pattern = re.compile(r'...')
    parse_inner = True  # default value, can be omitted
    parse_group = 1  # default value, can be omitted
    precedence = 5  # default value, can be omitted
    def __init__(self, match_obj):
        self.target = match_obj.group(2)

If we have a span token that does not need further parsing, we can write:

class Foo(SpanToken):
    pattern = re.compile(r'(foo)')
    parse_inner = False
    def __init__(self, match_obj):
        self.content = match_obj.group(1)

See the readme for more details.

Features:

• CommonMark compliant CodeFence;
• CommonMark compliant BlockCode;
• CommonMark compliant HTMLBlock;
• CommonMark compliant HTMLSpan;
• CommonMark compliant AutoLink;
• CommonMark compliant InlineCode;
• CommonMark compliant Heading;
• CommonMark compliant SetextHeading;
• added span-level token LineBreak;
• better handling of lazy-continuation in Quote;
• Footnotes can be defined in any block-level containers.

Fixes:

• loose lists conform to CommonMark spec (#44, thanks @huettenhain);
• not parsing sub-lists deeper than two levels (#46, thanks @daerhu);
• FileWrapper._index should not go below -1.

Development:

• refactored handling of SetextHeading;
• removed block_tokenizer.MismatchException;
• removed _children attribute, using children directly; (potentially breaking change?)
• renamed Separator to ThematicBreak;
• renamed FootnoteBlock to Footnote;
• tokenize and tokenize_inner returns lists of tokens;
• refactored CommonMark testing script.

Features:

• CommonMark compliant CodeFence (#41);
• allow multiple backticks for InlineCode;
• strips whitespace around InlineCode;

Fixed:

• Separator needs at least three characters;
• indented code blocks should not interrupt paragraphs (#40, thanks @joncass);
• crashes when sublists have different marker type (#42, thanks @JBartlett86);
• typo in Paragraph.read (#43, thanks @NatTupper);
• preliminary fixes for handling loose lists (#44, thanks @huettenhain);
• removed corrupted block_token.until function;
• html code language tags starts with "language-".

Features:

• added Pygments renderer to contrib (#35, thanks to @Bridouz);
• HTMLSpan now supports comments (#37);
• (more or less) Commonmark compliant List implementation (#40).

Fixes:

• changed logo to an actual mistletoe (#21, thanks to @liuq);
• allow lists after block tokens without newlines (#34, thanks to @huettenhain);
• recognize headings within paragraphs (#36);
• disallow opening space in html tag (#37).

Performance:

• removed FileWrapper.normalize;
• utilized universal newline mode.

Breaking changes:

• BlockToken.start does not advance file iterator.

Special shout-out to @joncass for raising the unattributed issues above, and giving me the motivation to finally fix the list implementation!

Note that this is a release with major changes. If you notice any rough edges (as there will certainly be), please do not hesitate to open an issue.

Features:

• added default render methods for all tokens;
• added reset_tokens function to block_token and span_token;
• allowed BlockToken.read to return any iterable;
• BaseRenderer is now available at mistletoe.BaseRenderer;
• added Scheme.

Fixes:

• throw better AttributeError when accessing RawText.children (#31, thanks @jabdoa2);
• disallow whitespace in span_token.Link (#32, thanks @DMRobertson);
• allowed empty alt text in Image and FootnoteImage (#33, thanks @joncass).

Features:

• md2jira: read from stdin if no input file is given (#27, thanks @alexkolson!);
• better command line options and help messages;
• auto-splitlines when mistletoe.markdown is given a string;
• inline tokens can span multiple lines (#30, thanks @duckwork!).

Fixes:

• TableRow now supports table shorthand (#29, thanks @huettenhain!);
• normalize line breaks.

... plus various refactors and documentation improvements.

Features:

• shortened mistletoe.markdown keyword argument name (renderer_cls to renderer);
• removed List reference lookup;
• list items can contain paragraphs (CM5.2);
• shorthand syntax added for tables (#26).

Fixed:

• ignored invisible characters at line end for CodeFence (#24);
• fixed extra newlines for headings in JIRARenderer (#25, thanks @huettenhain!);

Development:

• moved documentation to docs directory;
• solved the biggest mystery in the codebase.

Fixed:

• contrib/md2jira.py was importing from the wrong directory (#20, thanks to @cctile);
• characters in LaTeX lstlisting environment should not be escaped (#23, thanks to @liuq).

Features:

• added JIRA Markdown support (thanks to @cctile);
• Strong / Emphasis elements must open with non-whitespace characters;
• no more than 6 levels of Heading;

Fixed:

• render_table crashing when iterating token.children (#12);
• FootnoteLink engulfing trailing spaces (#14);
• Paragraph.read not stopping before CodeFence (#15);

Development:

• added testing for CommonMark compliance;
• merged plugins directory into contrib (thanks to @huettenhain);

Lastly, I miss cheeseburgers. 🍔

Features:

• BlockToken is a hell lot more flexible now;
• add_token accepts an additional position argument;
• Newlines are now preserved in Paragraph tokens.

Fixed:

• ASTRenderer fails to serialize FootnoteAnchor.

Where I broke backwards compatibility:

• BlockToken now has start and read methods, instead of match method. This allows for much more granular control of parsing when defining custom block-level tokens.
• Heading and SetextHeading are now different token classes, though their renderer functions are still the same.
• CodeFence and BlockCode are now different token classes, though their renderer functions are still the same.

What has been in my life for the past few weeks:

❄️

Features:

• added support for empty or self-closing HTMLSpan;
• added --renderer flag for command line usage;
• token.children now has idempotent behavior!

Development:

• refactored command line functionalities;
• testing and error handling.

Merry Christmas! 🎄

Features:

• removed argument footnotes from render functions;
• make custom tokens usable without invoking renderer as context manager (#5).

Development:

• documentation updates and bug fixes (more to come!);
• refactoring and slight performance gains.

Now for a beer emoji: 🍺

Features:

• auto-closes unclosed code fences;
• adds support for "[footnote]"-style links;
• interactive mode adds keyboard control support;
• accepts multiple filenames from the command line.

Fixed:

• render_image function missing argument;
• mistletoe crashes with empty list items;
• removes redundant whitespace for empty lines in code fences;
• fixed performance issues on PyPy: very, very fast now.

Features:

• span-level token constructors now accept match objects;
• simplified adding custom tokens to the parsing process;
• simplified creating new renderer classes.

Development:

• refactored (most of the) tests;
• cleaned up benchmarking script;
• cleaned up some spaghetti in tokenizer modules.

Features:

• added table-of-contents plugin;
• added rudimentary MathJax support in mathjax plugin;

Fixed:

• mistletoe crashes with text between underscores;
• incorrect handling of hashes in code blocks;

Relicensed under MIT.

Features:

• added support for footnote-style images and links;
• added support for LaTeX renderer;
• renderer classes are now context managers (see README).

Development:

• added test suite for LaTeX renderer;
• added benchmarking script for performance comparison;
• added scripts to compare render output across commits;
• added CI testing for versions up to Python 3.3.

Fixed:

• a bunch of regex craziness;
• outdated documentation.

This release is mainly to celebrate that I shaved. Other than that:

Block-level token support:

• heading (ATX and setext);
• quote;
• paragraph;
• block code (code fence and indented code);
• lists and nested lists;
• tables;
• horizontal rule;

Span-level token support:

• strong (with asterisks or underscore);
• emphasis (with asterisks or underscore);
• inline code;
• strikethrough;
• images (inline link only);
• links (inline link only) and autolinks;

Output format support:

• render to HTML;
• render to mdast-like AST;
• render to really janky LaTeX;

Lastly, hello world!