tantivy-query-grammar

• API Changes. Using Range instead of (start, end) in the API and internals (FileSlice, OwnedBytes, Snippets, ...) This change is breaking but migration is trivial.
• Added an Histogram collector. (@fulmicoton) #994
• Added support for Option<TCollector>. (@fulmicoton)
• DocAddress is now a struct (@scampi) #987
• Bugfix consistent tie break handling in facet's topk (@hardikpnsp) #357
• Date field support for range queries (@rihardsk) #516
• Added lz4-flex as the default compression scheme in tantivy (@PSeitz) #1009
• Renamed a lot of symbols to avoid all uppercasing on acronyms, as per new clippy recommendation. For instance, RAMDirectory -> RamDirectory. (@fulmicoton)
• Simplified positions index format (@fulmicoton) #1022
• Moved bitpacking to bitpacker subcrate and add BlockedBitpacker, which bitpacks blocks of 128 elements (@PSeitz) #1030
• Added support for more-like-this query in tantivy (@evanxg852000) #1011
• Added support for sorting an index, e.g presorting documents in an index by a timestamp field. This can heavily improve performance for certain scenarios, by utilizing the sorted data (Top-n optimizations)(@PSeitz). #1026
• Add iterator over documents in doc store (@PSeitz). #1044
• Fix log merge policy (@PSeitz). #1043
• Add detection to avoid small doc store blocks on merge (@PSeitz). #1054
• Make doc store compression dynamic (@PSeitz). #1060
• Switch to json for footer version handling (@PSeitz). #1060
• Updated TermMerger implementation to rely on the union feature of the FST (@scampi) #469
• Add boolean marking whether position is required in the query_terms API call (@fulmicoton). #1070

• Remove dependency to atomicwrites #833 .Implemented by @fulmicoton upon suggestion and research from @asafigan).
• Migrated tantivy error from the now deprecated failure crate to thiserror #760. (@hirevo)
• API Change. Accessing the typed value off a Schema::Value now returns an Option instead of panicking if the type does not match.
• Large API Change in the Directory API. Tantivy used to assume that all files could be somehow memory mapped. After this change, Directory return a FileSlice that can be reduced and eventually read into an OwnedBytes object. Long and blocking io operation are still required by they do not span over the entire file.
• Added support for Brotli compression in the DocStore. (@ppodolsky)
• Added helper for building intersections and unions in BooleanQuery (@guilload)
• Bugfix in Query::explain
• Removed dependency on notify #924. Replaced with FileWatcher struct that polls meta file every 500ms in background thread. (@halvorboe @guilload)
• Added FilterCollector, which wraps another collector and filters docs using a predicate over a fast field (@barrotsteindev)
• Simplified the encoding of the skip reader struct. BlockWAND max tf is now encoded over a single byte. (@fulmicoton)
• FilterCollector now supports all Fast Field value types (@barrotsteindev)
• FastField are not all loaded when opening the segment reader. (@fulmicoton)
• Added an API to merge segments, see tantivy::merge_segments #1005. (@evanxg852000)

This version breaks compatibility and requires users to reindex everything.

Tantivy 0.13 introduce a change in the index format that will require you to reindex your index (BlockWAND information are added in the skiplist). The index size increase is minor as this information is only added for full blocks. If you have a massive index for which reindexing is not an option, please contact me so that we can discuss possible solutions.

• Bugfix in FuzzyTermQuery not matching terms by prefix when it should (@Peachball)
• Relaxed constraints on the custom/tweak score functions. At the segment level, they can be mut, and they are not required to be Sync + Send.
• MMapDirectory::open does not return a Result anymore.
• Change in the DocSet and Scorer API. (@fulmicoton). A freshly created DocSet point directly to their first doc. A sentinel value called TERMINATED marks the end of a DocSet. .advance() returns the new DocId. Scorer::skip(target) has been replaced by Scorer::seek(target) and returns the resulting DocId. As a result, iterating through DocSet now looks as follows

let mut doc = docset.doc();
while doc != TERMINATED {
   // ...
   doc = docset.advance();
}

The change made it possible to greatly simplify a lot of the docset's code.

• Misc internal optimization and introduction of the Scorer::for_each_pruning function. (@fulmicoton)
• Added an offset option to the Top(.*)Collectors. (@robyoung)
• Added Block WAND. Performance on TOP-K on term-unions should be greatly increased. (@fulmicoton, and special thanks to the PISA team for answering all my questions!)

• Removing static dispatch in tokenizers for simplicity. (#762)
• Added backward iteration for TermDictionary stream. (@halvorboe)
• Fixed a performance issue when searching for the posting lists of a missing term (@audunhalland)
• Added a configurable maximum number of docs (10M by default) for a segment to be considered for merge (@hntd187, landed by @halvorboe #713)
• Important Bugfix #777, causing tantivy to retain memory mapping. (diagnosed by @poljar)
• Added support for field boosting. (#547, @fulmicoton)

How to update?

Crates relying on custom tokenizer, or registering tokenizer in the manager will require some minor changes. Check https://github.com/quickwit-oss/tantivy/blob/main/examples/custom_tokenizer.rs to check for some code sample.

• Added f64 field. Internally reuse u64 code the same way i64 does (@fdb-hiroshima)
• Various bugfixes in the query parser.
  • Better handling of hyphens in query parser. (#609)
  • Better handling of whitespaces.
• Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. "title:>hello", "weight:>=70.5", "height:<200" (@petr-tik)
• API change around Box<BoxableTokenizer>. See detail in #629
• Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)
• Add footer with some metadata to index files. #605 (@fdb-hiroshima)
• Add a method to check the compatibility of the footer in the index with the running version of tantivy (@petr-tik)
• TopDocs collector: ensure stable sorting on equal score. #671 (@brainlock)
• Added handling of pre-tokenized text fields (#642), which will enable users to load tokens created outside tantivy. See usage in examples/pre_tokenized_text. (@kkoziara)
• Fix crash when committing multiple times with deleted documents. #681 (@brainlock)

How to update?

• The index format is changed. You are required to reindex your data to use tantivy 0.11.
• Box<dyn BoxableTokenizer> has been replaced by a BoxedTokenizer struct.
• Regex are now compiled when the RegexQuery instance is built. As a result, it can now return an error and handling the Result is required.
• tantivy::version() now returns a Version object. This object implements ToString()