elasticlunr-rs

Added

• Default implementations for every Language type, delegating to their existing new().
• TokenizerFn type alias for the boxed tokenizer accepted by IndexBuilder::add_field_with_tokenizer.
• A cargo-fuzz target for the English pipeline (#58)

Changed

• Cleaned up all Clippy warnings.

Fixed

• Fixed tokens from earlier fields leaking into later fields' indices, inflating term and document frequencies (#60)
• Fixed a panic (subtract with overflow) when the English stemmer was given the word "ion" (#57)
• Fixed a panic (subtract with overflow) when the English stemmer was given a word ending in "eing" (#58)
• Fixed a panic when the English stemmer's doubled-consonant rule truncated a word in the middle of a multi-byte character, such as "त" in "aतed" (#63)
• Fixed the terminal "y" to "i" rule disagreeing with elasticlunr.js when the preceding character is itself a "y", so that "leeshyy" now stems to "leeshyi" (#63)

Compatibility

• The token leak fix changes the generated index: terms no longer appear in fields they were not in, and term/document frequencies in every field after the first are lower. Search results and ranking will shift, and any test that snapshots a generated index will need regenerating.

This release was yanked because the rewritten English stemmer could panic on certain inputs. Use 3.0.2 or a later release instead.

Changed

• Rewrote the English stemmer for improved performance (#48)
• Changed the rust-version in Cargo.toml to match the Minimum Supported Rust Version (1.60.0).
• Added support for either Criterion 0.4.0 or 0.5.0
• Updated the Cargo.toml license specification to better match the situation described in the README.

Changed

• Updated Minimum Supported Rust Version to 1.60.0.
• Updated Criterion to 0.4.0.

Added

• Language support for Korean (#50)
• Language support for Hungarian (#51)

Changed

• Updated dependencies and MSRVs to fix builds. (#47 et. al.)

See the changelog for changes.

Added

• Language support for Arabic (#40).
• Add the Language trait to make it easier to implement languages outside the crate.
• Add IndexBuilder::add_field_with_tokenizer to specify the tokenizer for a field.

Changed

• Update to 2018 edition, and bump MSRV to 1.54.0.
• Change benchmarks to use Criterion.
• Remove dependency on lazy_static.
• Update dependencies.
• Use Unicode character classes for trimmer.
• IndexBuilder functions which add fields will now panic if the same field is added multiple times.
• Fix IndexBuilder not respecting field insertion order.

Removed

• Remove the default feature. You now need to opt-in to the languages feature.
• Remove the deprecated function Pipeline::for_language.
• Remove the pipeline::tokenize* functions, which are now implemented as part of the Language trait.
• Remove Index::add_doc_with_tokenizer(s), replaced by IndexBuilder::add_field_with_tokenizer.
• Remove the Language enum. Use the Language trait implementations in the lang modules, and the free functions lang::from_name, lang::from_code, and lang::languages.