syntect

Improvements

• Eliminate syntect library's dependency on serde's "derive" feature. Increases build parallelism.
• Add parsing of gutterSettings.

Dependencies

• Update regex-syntax to 0.8.

Improvements

• Fix how syntect::util::split_at() handles multi-byte characters
• Allow case insensitive search for .tmtheme paths
• impl PartialEq for Theme

Dependencies

• Upgrade fancy-regex to 0.11
• Upgrade regex-syntax to 0.7
• Replace lazy_static with once_cell

Other

• Change MSRV policy to "last three stable versions"
• Make Debug impl of syntect::highlighting::Color less verbose

Breaking changes

• Lazy-load syntaxes to significantly improve startup time. This changes the binary format of syntax dump files.
• Remove ContextId::new() from public API to support lazy-loading of syntaxes
• Rename HighlightLines::highlight() to HighlightLines::highlight_line() to make it clear that the function takes one line at a time
• Make plist dependency (used for loading themes) optional via new plist-load feature
• Remove obsolete dump-load-rs and dump-create-rs features that has been identical to dump-load and dump-create for two years
• Remove deprecated items ThemeSettings::highlight_foreground, ThemeSettings::selection_background, ClassedHTMLGenerator::new, ClassedHTMLGenerator::parse_html_for_line, html::css_for_theme, html::tokens_to_classed_html and html::tokens_to_classed_spans
• Mark all error enums as #[non_exhaustive]
• These functions have been changed to return a Result to allow propagation of errors:
  • html::ClassedHTMLGenerator::parse_html_for_line_which_includes_newline
  • html::append_highlighted_html_for_styled_line
  • html::css_for_theme_with_class_style
  • html::highlighted_html_for_string
  • html::line_tokens_to_classed_spans
  • html::styled_line_to_highlighted_html
  • parsing::ParseState::parse_line
  • parsing::ScopeStack::apply
  • parsing::ScopeStack::apply_with_hook
  • parsing::syntax_definition::Context::match_at
  • parsing::syntax_definition::ContextReference::id
  • parsing::syntax_definition::ContextReference::resolve

Other changes

• Fall back to Plain Text if a referenced syntax is missing
• Add support for hidden_file_extensions key in syntaxes.
• Implement Error and Display for all error enums by using thiserror
• Replace lazycell with once_cell to fix crash on lazy initialization
• Add ScopeRangeIterator
• Add CI check for Minimum Supported Rust Version. This is currently Rust 1.53.
• Make looking up a syntax by extension use case-insensitive comparison
• Make from_dump_file() ~15% faster
• Blend alpha value on converting colors to ANSI color sequences
• Fix sample code in documentation to avoid double newlines
• Fix lots of build warnings and lints
• Add Criterion benchmarks for a whole syntect pipeline and for from_dump_file()

This version was yanked from crates.io due to a semver violation issue.

See the release notes for v4.7.0, this release removes a new Cargo feature which constituted a semver violation:

• Remove 'plist-load' feature again due to semver violation. #403

• Add html::line_tokens_to_classed_spans to also take a mutable ScopeStack, deprecate tokens_to_classed_spans, to avoid panics and incorrect highlighting.
• Derive Hash for Color and Style
• Add find_unlinked_contexts to SyntaxSet
• Add syntaxes method to SyntaxSetBuilder
• Bump fancy-regex to v0.7 and yaml-rust to v0.4.5

• Added a new function for producing classed HTML which handles newlines correctly and deprecated old one. #307

• Errors are now Send + Sync + 'static #304

• Fixes unnecesary dependency of the html feature on the assets feature. #300
• Adds ability to add prefixes to html module CSS class names. #296

• Updates to new versions of onig and plist. The new onig version doesn't require bindgen thus making compilation easier. #293

• Properly handle backreferences in included contexts #288

• Make sure errors implement Send #285
• Fix errors to not use the deprecated description() #286

Thanks @sharkdp for the bug fixes! Bumping second part of semver since Send is adding functionality (back).

Headline feature: pure-Rust fancy-regex engine option

Users can now opt in to a pure-Rust regex engine using Cargo features, making compilation easier in general. People experiencing difficulty compiling for Windows and Wasm should try switching to fancy-regex. Note this currently approximately halves highlighting speed.

See the Readme and #270 for details. Thanks to @robinst for implementing this!

Other changes

• Ability to generate CSS for a theme for use with classed HTML generation (won't always be correct) #274
• Don't generate empty spans in classed HTML #276
• Miscellaneous dependency bumps and cleanup

Breaking changes and upgrading

Upgrading should cause no errors for nearly all users. Users using more unusual APIs may have a small amount of tweaking to do.

• If you use default-features = false you may need to update your features to choose a regex engine
• A bunch of technically public APIs that I don't know if anyone uses changed due to the regex engine refactor, common uses shouldn't break

Bug fixes and new utilities

• Fixes multiple bugs
• Add RangedHighlightIterator
• Add as_latex_escaped util

• Bump onig dependency
• inconsequential patches

• Actually make tokens_to_classed_spans public like intended

Metadata and new classed HTML generation

• Add support for loading metadata (#223 #225 #230)
• Improve support for generating classed HTML and fix a bug, old function is deprecated because it's impossible to use correctly (#235)
• Update plist to v0.4 and pretty_assertions to v0.6 (#232 #236)

Bug fixes

• Fix application of multiple with_prototypes (#220, fixes #160, #178, ASP highlighting)
• Fix prototype marking logic (#221, fixes #219)

Minor bug fixes

• Fix a bug with syntaxes that used captures in lookarounds (#176 #215)
• Fix the precedence order of syntaxes to match Sublime (#217 #216)

Breaking changes and major new features

This is a major release with multiple breaking API changes, although upgrading shouldn't be too difficult. It fixes bugs and comes with some nice new features.

Breaking changes and upgrading

• The SyntaxSet API has been revamped to use a builder and an arena of contexts. See example usage.
• Many functions now need to be passed the SyntaxSet that goes with the rest of their arguments because of this new arena.
• Filename added to LoadingError::ParseSyntax
• Many functions in the html module now take the newlines version of syntaxes.
  • These methods have also been renamed, partially so that code that needs updating doesn't break without a compile error.
  • The HTML they output also treats newlines slightly differently and I think more correctly but uglier when you look at the HTML.

Breaking rename upgrade guide

• SyntaxSet::add_syntax -> SyntaxSetBuilder::add
• SyntaxSet::load_syntaxes -> SyntaxSetBuilder::add_from_folder
• SyntaxSet::load_plain_text_syntax -> SyntaxSetBuilder::add_plain_text_syntax
• html::highlighted_snippet_for_string -> html::highlighted_html_for_string: also change to newlines SyntaxSet
• html::highlighted_snippet_for_file -> html::highlighted_html_for_file: also change to newlines SyntaxSet
• html::styles_to_coloured_html -> html::styled_line_to_highlighted_html: also change to newlines SyntaxSet
• html::start_coloured_html_snippet -> html::start_highlighted_html_snippet: return type also changed

Major changes and new features

• Use arena for contexts (#182 #186 #187 #190 #195): This makes the code cleaner, enables use of syntaxes from multiple threads, and prevents accidental misuse.
  • This involves a new SyntaxSetBuilder API for constructing new SyntaxSets
  • See the revamped parsyncat example.
• Encourage use of newlines (#197 #207 #196): The nonewlines mode is often buggy so we made it easier to use the newlines mode.
  • Added a LinesWithEndings utility for iterating over the lines of a string with \n characters.
  • Reengineer the html module to use newlines syntaxes.
• Add helpers for modifying highlighted lines (#198): For use cases like highlighting a piece of text in a blog code snippet or debugger. This allows you to reach into the highlighted spans and add styles.
  • Check out split_at and modify_range in the util module.
• New ThemeSet::add_from_folder function (#200): For modifying existing theme sets.

Bug Fixes

• Improve nonewlines regex rewriting: #212 #211
• Reengineer theme application to match Sublime: #209
• Also mark contexts referenced by name as "no prototype" (same as ST): #180
• keep with_prototype when switching contexts with set: #177 #166
• Fix unused import warning: #174
• Ignore trailing dots in selectors: #173
• Fix embed to not include prototypes: #172 #160

Upgraded dependencies

• plist: 0.2 -> 0.3
• regex: 0.2 -> 1.0
• onig: 3.2.1 -> 4.1

Regex checking and plain file names

• Check regexes compile upon loading from YAML (There's technically a small breaking change here if you match on the previously unused regex error, but I don't think anyone does)
• Can detect the correct syntax on full file names like CMakeLists.txt
• Make nonewlines mode marginally less buggy (still prefer using newlines mode)
• Better error types
• Better examples and tests

More robust parsing

• Parsing now abandons a regex after reaching a recursion depth limit instead of taking forever
• Loop detection better matches Sublime Text
• Parsing is faster!
• Dependency upgrades
• Other minor tweaks

Thanks to @robinst for the headline features of this release!

Breaking Changes and New Stuff

Breaking changes

• The static-onig feature was removed, static linking is now the default
• Font styles and color constants now use associated consts because of bitflags upgrade
• SyntaxDefinition::load_from_str now has an extra parameter

Other notable changes

• Support for new embed syntax, see #124
• Updates to many dependencies
• Updated dumps
• More compact HTML output

New Inspired GitHub and libonig

Update bitflags & packages

This release changes how the constants for FontStyle and Color, relying on the new associated consts feature in Rust 1.20. The old constants are still available but are deprecated and will be removed in v2.0.

Packages were also updated to newer versions.

Enable comparison of parse states

Fixes comparisons of parse states so they are fast and don't recurse infinitely. Thanks @raphlinus

Bug fixes and package updates

• Fixes #101, which caused some syntaxes like PHP to behave incorrectly.
• Updates Packages with new syntax versions
• Adds new handy flags to the syncat example

Pure Rust dump loading / creation features

Helper methods and more theme attributes

Highlighting stacks

Small release, adds a convenience method for highlighting an entire stack, and derives some more things on Scope.

Serde and optional parsing

This release switches the dump format from rustc-serialize to Serde, anyone using custom dumps will have to update them.

It also makes the parsing part of the library optional behind a feature flag, anyone not using the default feature flags probably will want to add the parsing flag.

Bug fixes, tests, updates and feature flags

• Syntax tests: there is a new syntest example for running Sublime Text syntax tests
• Bug fixes: there's a ton of bugs fixed in this release, mostly found via the syntax tests. These mostly affected certain syntaxes which pushed/set multiple contexts at once.
• Updated packages: The Sublime packages have been updated to the latest version
• Feature flags: there's now Cargo feature flags for disabling some parts of syntect if you don't want unnecessary binary and dependency bloat.