teamtnt/tntsearch

Faster indexing and a couple of correctness fixes.

Performance

• SQLite index creation is ~3× faster. saveWordlist() no longer inserts each term and catches a UNIQUE-constraint exception (re-preparing every statement per duplicate term); it uses a single INSERT … ON CONFLICT(term) DO UPDATE upsert and resolves term ids without SELECT MAX(id).
• insert() is ~25× faster for large documents. Each document is now indexed inside a single transaction instead of one implicit commit (fsync) per wordlist/doclist write — so indexing e.g. a PDF per insert() call no longer pays thousands of fsyncs. (#360)

Fixed

• total_documents was left at 0 when the number of indexed rows was an exact multiple of the 10,000-row commit batch, because the final transaction was left open and the info-table update rolled back. This corrupted BM25 idf = log(count / df) for such indexes.
• MysqlEngine wordlist upsert accumulated num_hits using the document count instead of the hit count, giving multi-document terms the wrong total occurrence count.

Added

• A benchmark/ harness (seeded Zipfian dataset generator + timers) for measuring index-creation speed.

Full Changelog: v5.2.0...v5.3.0

Added

• SqliteEngine::createIndex() now creates the storage directory if it doesn't exist (#304)

Fixed

• Preserve the exact match in fuzzy search results — SqliteEngine & MysqlEngine (#302)
• Multibyte-aware Levenshtein distance so fuzzy search works correctly for Cyrillic/UTF-8 terms (#193)
• Boolean -term negation now excludes results (AND-NOT) instead of unioning the NOT set, and no longer crashes on two negations (#263, #246, #267)
• searchBoolean() now returns a docScores key, matching search() (fixes "Undefined index: docScores" in Laravel Scout) (#278)
• filesystemMapIdsToPaths() returns null for an id with no filemap row instead of silently coercing to $res[0] (#130)
• Highlighter no longer re-highlights the markup it injected for earlier terms (#274)
• Clamp acos() domain in TNTGeoSearch to avoid NAN distances silently dropping the closest result (#357)
• RedisEngine::createIndex() no longer throws "must not be accessed before initialization" on the typed $dbh property
• Fix selectIndex()/createIndex() writing terms to the previously selected index when switching indexes in one process (Laravel Scout with multiple models)

Full Changelog: v5.1.0...v5.2.0

Fix selectIndex()/createIndex() writing terms to the wrong index (#367).
Switching indexes in a single process (e.g. Laravel Scout with multiple
searchable models) no longer corrupts the previously selected index.

What's Changed

• Skip 'non-string' and empty values by @leonardfischer in #342
• Unifying names of interfaces, keep old code for backwards compatibility by @leonardfischer in #341
• Remove duplicate unique index for wordlist table by @BlackbitDevs in #319

Full Changelog: v5.0.0...v5.0.1