org.apache.lucene:lucene-monitor

Lucene 10.4 brings some significant performance improvements and a brand new knn vectors format.

Many lucene queries should see a performance improvement of 10-15%, some might even see a 35% improvement! This is due to increasing the the block size of the terms postings and better utilization of SIMD optimized code.

Additionally, there is a new scalar quantized format for dense vectors and knn search. Lucene104ScalarQuantizedVectorsFormat and Lucene104HnswScalarQuantizedVectorsFormat allow custom quantized bits for 1, 2, 4, 7, and 8. The recall has improved significantly and for many vector types, quantizing to 2 bits will achieve even better recall than older formats at the 4 bit level. This improves latency while increasing recall for various vector work loads.

New Features

• Provides a new and improved scalar quantized formats Lucene104ScalarQuantizedVectorsFormat  and Lucene104HnswScalarQuantizedVectorsFormat  for dense vectors. Allowing for quantizing to 1, 2, 4, 7, and 8 bits. For reference, the new 2 bit quantization technique provides better recall and speed than the old 4 bit.

API Changes

• New bulk operation APIs for dense vectors and numeric doc values

Improvements and Optimizations

• HNSW graphs can now delay being built for tiny segments and will prevent completely rebuilding the graphs when handling deletes
• Handling of deletes in general got much faster and cheaper, improving storage costs significantly when there are very few deleted docs
• Block size increased for terms postings, significantly improving query latency for many types of queries
• Use a coarser-grained competitive iterator with lower construction costs for numeric sorts against fields with DocValuesSkippers.

Runtime Behavior Changes and Bug Fixes

• Fix tessellator failure by preferring the shared vertex that is the leftmost vertex of the hole
• The reverse field of SortField is now final. If you have subclassed SortField,
  you should set reverse in the super constructor.
• Align float vectors on disk to 64 bytes, for optimal performance on Arm Neoverse
  machines

Bug fixes

• Fix potential EOF introduced by optimized filter iterations in MaxScoreBulkScorer.

What's Changed

• [10.3] Utilize iterator and not approximation in MaxScoreBulkScorer by @benwtrent in #15380

Full Changelog: releases/lucene/10.3.1...releases/lucene/10.3.2

Bug fixes

• Fix the Impact returned from Lucene103PostingsReader when frequencies are not indexed.

What's Changed

• [10.3] Fix returned Impacts when frequencies are not indexed by @iverase in #15263

Full Changelog: releases/lucene/10.3.0...releases/lucene/10.3.1

Lucene 10.3 brings major performance improvements.

Lexical search is now vectorized to better take advantage of SIMD instructions, more efficient memory access patterns, CPU pipelining and amortize the cost of virtual function calls. Lucene's nightly benchmarks report a 40% speedup compared with Lucene 10.2 when computing top-100 hits by score on disjunctive and conjunctive queries.

Vector search now better parallelizes fetching vectors into the CPU cache. Lucene's nightly benchmarks report a 15%-20% speedup compared with Lucene 10.2.

The terms dictionary performs about 30% faster than in Lucene 10.2 on primary-key lookups according to Lucene's nightly benchmarks. This should help speed up workloads that rely on terms dictionary lookup performance including primary-key lookups, indexing operations that specify an ID and TermInSet queries.

New Features

• Supports reranking with late interaction model multi-vectors, full precision vector similarity scores, or any provided DoubleValuesSource, enabling improved ranking of search results.
• Adds a MultiIndexMergeScheduler – a multi-tenant wrapper that allows sharing a common merge scheduler across multiple instances.

API Changes

• Adds API to fetch the size of off-heap memory required by a KNN field. This size can be used to help determine the memory requirements for optimal search performance, which can be greatly affected by page faults when not enough memory is available.
• RandomVectorScorer now supports a bulk scoring interface.
• LeafReader#searchNearestVectors now accepts an AcceptDocs instance instead of a Bits instance to identify document IDs to filter.
• Collectors can now take advantage of pre-aggregated data to speed up faceting using LeafCollector#collectRange.

Improvements and Optimizations

• Adds optimistic knn search to vector queries. Optimistic knn search addresses a major issue where we return inconsistent results due to race conditions in the shared queue previously used over multi-segment search.
• Faster vector search on HNSW graphs through GroupVarInt encoding.
• Searcher managers now support 'Adaptive Refresh', enabling users to control the commit points it refreshes on. This helps with graceful handling of large replication payloads in segment-replicated systems.

Runtime Behavior Changes and Bug Fixes

• The default ReadAdvice has been changed from RANDOM to NORMAL. MMapDirectory will no longer set any specific read advice out-of-the-box.
• Default RefCountedSharedArena.DEFAULT_MAX_PERMITS are reduced to 64. Also fixes the infinite loop when RefCountedSharedArena's underlying Arena#close fails due to concurrent usage of segments.
• Uses READONCE when reading segment infos, to fix mmap leaks on segment info files. Includes fixes for multiple other resource leaks.

Bug fixes

• Reduce NeighborArray on-heap memory during HNSW graph building
• Fix IndexSortSortedNumericDocValuesRangeQuery for int sort
• ValueSource.fromDoubleValuesSource(dvs).getSortField() would throw errors when used if the DoubleValuesSource needed scores

This patch release contains bug fixes that are highlighted below.

• Fix DISIDocIdStream::count so that it does not try to count beyond max.
• Correct TermOrdValComparator competitive iterator so that it forces sparse field iteration to be at least scoring window baseline when doing intoBitSet.
• Provide better impacts for fields indexed with IndexOptions.DOCS
• Fixed lead cost computations for bulk scorers of conjunctive queries that mix MUST and FILTER clauses, and disjunctive queries that configure a minimum number of matching SHOULD clauses.

Bug fixes

• Support for JDK24+
• Reduce sharedArenaMaxPermits from 1024 to 64
• Use READONCE to read segment infos
• Fix rare spin-loop in RefCountedSharedArena
• Ensure vector queries handle advanceShallow correctly
• Fix failure due to hole bridge being coplanar with polygon edge

Bug fixes

• Reduce NeighborArray on-heap memory during HNSW graph building
• Fix IndexSortSortedNumericDocValuesRangeQuery for int sort
• ValueSource.fromDoubleValuesSource(dvs).getSortField() would throw errors when used if the DoubleValuesSource needed scores
• Disable connectedComponents logic in HNSW graph building.