github.com/moov-io/watchman

Watchman 1.1.0 adds opt-in field-level debug by match threshold, and tighter name preparation for vessels, former names, and legal forms.

Search

• search: GET /v2/search?debug=<float> attaches field-level score pieces only for returned hits at or above that score. debug=true still attaches to every returned hit. Cheap scoring stays on the full candidate set; DebugSimilarity runs after ranking. by @adamdecaf in #927
• search: treat f.k.a., formerly, and (ex-Name) parentheticals as former names
• search: strip vessel prefixes (MV, M/T, SS), trailing port/place, and trade-document party labels (SHIPPER:, FIELD 59:)
• search: canonicalize English legal-form phrases (Limited Liability Company → llc) and collapse spaced L.L.C. letters
• search: penalize disagreeing name numerals (NAME_NUMERAL_CONFLICT_PENALTY_MULTIPLIER, default 0.70)
• test: vendor Cascade company and vessel name-pair fixtures

Docs

• docs: debug=true and debug=0.80 on Using Watchman, the Search Guide, and the compliance briefing

Fixes

• deps: renovate updates (openrouter go-sdk, jekyll-feed, and grouped bumps)

Full Changelog: v1.0.2...v1.1.0

Watchman 1.0.2 is the first 1.0 cut with working GitHub release artifacts. v1.0.0 and v1.0.1 did not finish Create Release (no binaries or images).

Watchman 1.0 is the v2 sanctions screening engine. Screening is GET /v2/search. Each result is the list record with match (0 to 1) on the same object.

What 1.0 freezes

• API: GET /v2/search with type, name, IDs (gov_passport=COUNTRY:NUMBER), dates, and addresses. There is no JSON POST body on that path.
• Identity vs evidence: a matching passport, national ID, IMO/MMSI, aircraft serial, or crypto address (with country) can score 1.0. A matching tax number, company registration, email, or phone raises the score; it does not force 1.0. Two IDs of the same type that disagree lower the blended score (ID_CONFLICT_PENALTY_MULTIPLIER, default 0.70).
• Cutoff: minMatch is policy. 0.80 is the usual production queue (precision ~0.99 on OpenSanctions subject pairs). Default minMatch remains 0 so 0.x callers keep the same page of results unless they pass a cutoff.
• Ingest: every uploaded row is loaded into the in-memory search index. A per-source checksum avoids scanning the table on every query.
• Ops: authentication belongs at a reverse proxy. Embeddings stay off until you configure a provider (qwen3-embedding:0.6b via Ollama is the measured pick). MCP is experimental.

Using Watchman: https://moov-io.github.io/watchman/using-watchman/
For compliance and risk: https://moov-io.github.io/watchman/methodology/for-compliance/

Search

• search: stop forcing 1.0 on tax IDs and contact matches
• search: penalize conflicting government IDs
• search: recast person/business/organization type mismatches (vessel/aircraft stay a hard zero)
• ingest: index all ingested rows with a checksum cache by @adamdecaf in #912
• pkg/recordlink: public Keys and Prefixes for hashed blocking by @adamdecaf in #909

Docs and evaluation

• docs: Using Watchman guide, first-search at minMatch=0.80 (OFAC SDN 48603 + passport → 1.0)
• docs: compliance briefing, OpenSanctions Pairs tables (precision/recall; drop F1 from public pages)
• research: OpenSanctions Pairs evaluator and results by @adamdecaf in #903
• build: ship Linux and macOS GitHub releases with libpostal

Fixes

• makefile: DIST_GOTAGS is a make variable, not a clean recipe (Docker Hub / OpenShift make clean on Create Release)
• makefile: sudo make for libpostal so data can be written under /usr/local/share on Ubuntu
• ci: install autoconf/automake on macOS before libpostal (autoreconf)
• ci: Create Release matrices use fail-fast: false
• webui: omit request path from wasm compression log (CWE-117)
• deps: rubyzip 3.7.0 (GHSA-47m2-wp7j-p9vc)
• test: split geocoding Success tests into mock and live; keep OFAC methodology page nav
• ci: GitHub releases are latest, not prerelease

Full Changelog: v0.69.0...v1.0.2

This tag did not finish Create Release (libpostal make could not write /usr/local/share/libpostal on Ubuntu). There are no binaries or images for v1.0.1.

Use v1.0.2.

This tag did not finish Create Release (Docker Hub / OpenShift make clean, macOS libpostal). There are no binaries or images for v1.0.0.

Use v1.0.2.

Search result impact

High-confidence hits for the usual screening shape stay in the result set. The rest of the page is not identical to v0.68.0. Re-check golden files, minMatch=0 top-N lists, and identifier-only queries.

• Name queries intersect distinctive tokens using document frequency in this partition (not a language-specific suffix list). Extra common legal-form words (Limited, ООО, GmbH, …) are not required, so a DBA of Ocean Shipping still matches Ocean Shipping Limited. Tokens with no postings are skipped; typos still scan the type partition. Disjoint tokens fall back to their union.
• Partial-name rows drop. John Smith no longer scores every John and every Smith. Those were one-token overlaps, usually below a typical minMatch of 0.8.
• Identifier-only IMO / MMSI / serial / email / phone match the exact value, a prefix (min 3–4 characters), or a single QWERTY-adjacent typo. With default limit=10 and minMatch=0, the exact hit stays on top; the other weak fillers from a full partition scan are gone. Crypto and government IDs stay exact. Unknown identifiers still scan the partition.
• Scores: Jaro-Winkler math is unchanged. If a primary name is already ≥ 0.99, remaining AKAs are not scored, so a reported match can be 0.99 instead of 1.0.

IMPROVEMENTS

• search: distinctive-token candidate intersection; legal-form optionality from corpus frequency by @adamdecaf in #901
• search: score corpus indices and copy Entity values only for the top-N
• search: plaintext IMO/MMSI/serial/email/phone index for prefix and QWERTY-near typed queries
• search: pooled Jaro-Winkler scratch buffers

Full Changelog: v0.68.0...v0.69.0

ADDITIONS

• address: optional deepparse HTTP parser by @adamdecaf in #885
  • Opt-in alternative to libpostal / usaddress; default parsers are unchanged
  • Configure Watchman.Deepparse (Enabled, BaseURL or DEEPPARSE_URL, Model, Timeout)
  • make setup-deepparse starts ghcr.io/graal-research/deepparse:0.11.0 on :8000 via a compose profile (not part of make setup)
  • Search ?address= uses deepparse when enabled; if both Deepparse and PostalPool are on, Deepparse wins
  • Client: https://github.com/adamdecaf/deepparse-go

IMPROVEMENTS

• docs: clarify that pre-built Watchman releases use libpostal/PostalPool by @adamdecaf in bf299ede

BUILD

• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.156 by @renovate in #886
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.157 by @renovate in #887
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.158 by @renovate in #888

Full Changelog: v0.67.0...v0.68.0

ADDITIONS

• search: optional per-request name-matching algorithms by @adamdecaf in #878 and #884
  • GET /v2/search?algorithm=, MCP search_entities, and Go SearchOpts.Algorithm
  • Default is unchanged: Jaro-Winkler (jaro-winkler / omitted)
  • Opt-in values: soundex, soft-bidist, soft-bisim, editex, nsim, nsim-3, double-metaphone, beider-morse
  • Inner scorers (soft-bidist, soft-bisim, editex, nsim, nsim-3) replace only the token pair metric. Phonetic options (soundex, double-metaphone, beider-morse) keep Jaro-Winkler and boost pairs whose encodings match
  • Beider-Morse is generic BMPM v1 only (not language-specific)
  • Generated score matrix (markdown + CSV): https://moov-io.github.io/watchman/algorithm-comparison/
• feat: add PII-safe composite blocking keys for record linkage by @adamdecaf in #654
  • Hashed hierarchical keys (name Soundex, government IDs, address prefixes, contact) for candidate blocking without storing raw PII in the index
  • In-memory corpus uses government-ID and address-only keys; name search still uses the token inverted index
  • Docs: https://moov-io.github.io/watchman/record-linkage/

IMPROVEMENTS

• ingest: cap POST /v2/ingest request body size by @adamdecaf in #877
  • Default 32 MiB, override with INGEST_MAX_BODY_BYTES
• docs: Watchman is not designed for the public internet by @adamdecaf in 6844934
• docs: download URLs are operator configuration by @adamdecaf in 7070a31
• test: skip live OFAC search tests in short mode by @adamdecaf in #874
• fix: omit raw algorithm from unknown-algorithm errors by @adamdecaf in 52fb25e
  • Avoids CodeQL go/log-injection on /v2/search error logs

BUILD

• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.148 by @renovate in #869
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.149 by @renovate in #873
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.150 by @renovate in #883

Full Changelog: v0.66.3...v0.67.0

IMPROVEMENTS

• geocoding: read GEOCODING_API_KEY from environment by @toruiwasa in #867
  • The env var was documented but only YAML Geocoding.Provider.APIKey was honored
  • When set, GEOCODING_API_KEY takes precedence over YAML (same pattern as OPENSANCTIONS_API_KEY)
• embeddings: read EMBEDDINGS_* from environment by @toruiwasa in #868
  • EMBEDDINGS_ENABLED, CACHE_SIZE, CROSS_SCRIPT_ONLY, BATCH_SIZE, and INDEX_BUILD_TIMEOUT were documented but ignored
  • Env overrides now apply before the Enabled check, so embeddings can be turned on or off from the environment
• bench: keep skip-fetch-gh-pages on master by @adamdecaf in c099c42
  • github-action-benchmark fetches master:master, which git refuses when master is already checked out
• bench: write data.js in github-action-benchmark format by @adamdecaf in 7c1a563
  • A trailing semicolon made the action treat stored history as missing
• fix(ci): don't inspect docker tags at makefile parse time by @adamdecaf in c7e9e81
  • Move the existing-tag guard into the manifest recipes so OpenShift builds are not blocked by Docker Hub tags
• bench: reset stored benchmark history by @adamdecaf in 6cd1e5e

BUILD

• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.135 by @renovate in #865
• fix(deps): update all by @renovate in #866
  • github.com/openrouterteam/go-sdk v0.7.135 → v0.7.137
  • benchmark-action/github-action-benchmark v1.22.1 → v1.22.2

Full Changelog: v0.66.2...v0.66.3

IMPROVEMENTS

• csl_eu: skip rows with a non-numeric EntityLogicalID by @SashaMIT in #861
  • Failed Atoi of EntityLogicalID stored as 0, so those rows merged into one record
• search: read SEARCH_MAX_IN_FLIGHT from environment by @toruiwasa in #862
  • The env var was documented but only YAML Search.MaxInFlight was honored (server never calls DefaultConfig())
  • When set to a positive integer, SEARCH_MAX_IN_FLIGHT takes precedence over YAML (same pattern as INITIAL_DATA_DIRECTORY)
• docs: correct environment variable tables in config.md by @toruiwasa in #860
• fix(ci): upload Windows release assets with bash so $TAG expands by @adamdecaf in 328f7c3
  • PowerShell is the default shell on windows-latest, so $TAG/$REPO were empty and gh release upload failed

BUILD

• fix(deps): update all by @renovate in #863
  • github.com/openrouterteam/go-sdk v0.7.132 → v0.7.133
  • github.com/modelcontextprotocol/go-sdk v1.7.0 → v1.8.0
• fix(deps): update module github.com/urfave/cli/v3 to v3.12.0 by @renovate in #864

New Contributors

• @SashaMIT made their first contribution in #861

Full Changelog: v0.66.1...v0.66.2

IMPROVEMENTS

• download: read ERROR_ON_EMPTY_LIST from environment by @toruiwasa in #858
  • The env var was documented but only YAML ErrorOnEmptyList was honored
  • When set, ERROR_ON_EMPTY_LIST takes precedence over YAML (same pattern as INITIAL_DATA_DIRECTORY)
• ci: stabilize TestMinMaxMed and benchmark failure logs by @adamdecaf in #845
• Add zizmor GitHub Actions security analysis by @adamdecaf in #818
• ci: push docker images only on official moov-io releases by @adamdecaf in #826
• ci: drop advanced CodeQL workflow by @adamdecaf in #825
• ci: skip code owners on lockfiles, workflows, and go.mod/go.sum by @adamdecaf in #827, #831, #832
• fix(ci): clear release workflow deprecation warnings by @adamdecaf in #813

BUILD

• fix(deps): bump google.golang.org/grpc to v1.83.2 by @adamdecaf in #844
• fix(deps): update module github.com/urfave/cli/v3 to v3.11.0 by @renovate in #816
• fix(deps): update opentelemetry-go monorepo to v1.46.0 by @renovate in #824
• fix(deps): update module github.com/openrouterteam/go-sdk (v0.7.40 → v0.7.132)
• chore(deps): bump ubi9/ubi-minimal in /build by @dependabot in #822
• fix(deps): update all by @renovate in #812, #814, #815, #817, #819, #820, #821, #823, #829, #840, #842, #850, #851, #852

New Contributors

• @toruiwasa made their first contribution in #858

Full Changelog: v0.66.0...v0.66.1

ADDITIONS

• feat: add Prometheus duration histogram for HTTP API requests by @naemono in #811
  • Exposes watchman_http_request_duration_seconds (HistogramVec by method, route, code) on the existing admin /metrics endpoint
  • Route labels use the registered path template (e.g. /v2/ingest/{fileType}), not the raw request path

IMPROVEMENTS

• fix: resolve open CodeQL findings on master by @adamdecaf in #800
  • Avoid log injection in the ingest API by keeping user-controlled fileType on the structured field only
  • Pin third-party Actions in release.yml by commit SHA

BUILD

• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.40 by @renovate in #810
• fix(deps): update all by @renovate in #808
• chore(deps): update all (major) by @renovate in #786
• fix(deps): update all by @renovate in #807
• fix(deps): update module github.com/moov-io/base to v0.63.0 by @renovate in #806
• chore(deps): update registry.access.redhat.com/ubi9/ubi-minimal docker tag to v9.8-1786323074 by @renovate in #805
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.32 by @renovate in #804
• chore(deps): bump github.com/moov-io/base to v0.62.1 by @adamdecaf in #803
• build(deps-dev): bump json from 2.20.0 to 2.21.2 in /docs by @dependabot in #802
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.26 by @renovate in #799
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.25 by @renovate in #798
• fix(deps): update opentelemetry-go monorepo to v1.45.0 by @renovate in #796
• fix(deps): update all by @renovate in #797
• fix(deps): update all by @renovate in #795
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.18 by @renovate in #794

New Contributors

• @naemono made their first contribution in #811

Full Changelog: v0.65.2...v0.66.0

IMPROVEMENTS

• fix: reduce refresh memory peak after corpus swap by @adamdecaf in #791
  • Search corpus owns the entity slice; latestStats keeps metadata only
  • After index swap, run GC + FreeOSMemory so heap returns to single-corpus steady when Update returns (instead of lingering near ~2.5× until a later GC)
  • Local bench (8 lists / 39,837 entities): post-swap heap 2.48× → 1.00× steady; ~294 MB less retained at swap vs prior behavior
  • Live prod dumps (US lists: ofac/csl/non_sdn/fincen_311): ~3.5 GB RSS / ~100 MB Go heap — cgroup sizing is driven by RSS (libpostal in image), not Go corpus alone

BUILD

• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.10 by @renovate in #788
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.7.17 by @renovate in #789

Full Changelog: v0.65.1...v0.65.2

IMPROVEMENTS

• fix: keep serving data when scheduled refresh fails by @adamdecaf in #787

What's Changed

• fix(deps): update all by @renovate[bot] in #780
• fix(deps): update all by @renovate[bot] in #782
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.6.0 by @renovate[bot] in #783
• fix(deps): update module github.com/openrouterteam/go-sdk to v0.6.2 by @renovate[bot] in #784
• build(deps): bump google.golang.org/grpc from 1.81.1 to 1.82.1 by @dependabot[bot] in #781
• fix(deps): update all by @renovate[bot] in #785

Full Changelog: v0.65.0...v0.65.1

IMPROVEMENTS

• perf: candidate index, admission control, and hot-path allocation cuts by @adamdecaf in #779

What's Changed

• chore(deps): update all (major) by @renovate[bot] in #777
• fix(deps): update all by @renovate[bot] in #776
• fix(search): clean up errors from Client by @adamdecaf in #778

Full Changelog: v0.64.0...v0.65.0

ADDITIONS

• build: Compile for linux/arm64 as well by @elliotcourant in #772
• feat: add US Terrorist Exclusion List (TEL) by @cacheflow in #774
• feat: Allow data to be refreshed via an API call by @elliotcourant in #771

What's Changed

• build(deps-dev): bump nokogiri from 1.19.3 to 1.19.4 in /docs by @dependabot[bot] in #768
• build(deps-dev): bump faraday from 2.14.2 to 2.14.3 in /docs by @dependabot[bot] in #769
• build(deps): bump concurrent-ruby from 1.3.6 to 1.3.7 in /docs by @dependabot[bot] in #770
• chore(deps): update all (major) by @renovate[bot] in #766
• fix(deps): update all by @renovate[bot] in #765

New Contributors

• @cacheflow made their first contribution in #774

Full Changelog: v0.63.3...v0.64.0

Full Changelog: v0.63.2...v0.63.3

What's Changed

• perf: warm up libpostal concurrently with the initial data load by @akamick86 in #763

Full Changelog: v0.63.1...v0.63.2

IMPROVEMENTS

• fix: init libpostal before accepting requests by @adamdecaf in #758

What's Changed

• chore(deps): update all by @renovate[bot] in #757
• chore(deps): update all to v0.14.0 by @renovate[bot] in #759
• chore(deps): update all to v0.14.0 by @renovate[bot] in #760
• chore(deps): update all to v0.14.0 by @renovate[bot] in #761

Full Changelog: v0.63.0...v0.63.1

ADDITIONS

• feat(download): watchman ignored download errors by @bpross in #748

IMPROVEMENTS

• address: switch to moov-io/gopostal by @adamdecaf in #750
• perf: skip non-matching OFAC government ID regexes during preparation by @akamick86 in #752
• perf: skip language detection when it cannot change stopword removal by @akamick86 in #753

What's Changed

• fix(deps): update opentelemetry-go monorepo to v1.44.0 by @renovate[bot] in #749
• fix(deps): update all by @renovate[bot] in #751
• chore(config): migrate Renovate config - autoclosed by @renovate[bot] in #754

New Contributors

• @akamick86 made their first contribution in #752
• @bpross made their first contribution in #748

Full Changelog: v0.62.0...v0.63.0

NOTICE

The switch to using Senzing's libpostal data increases memory usage to over 3GB. Adjust your deployments accordingly.

ADDITIONS

• feat: add MCP server by @adamdecaf in #724

IMPROVEMENTS

• Make final score penalties configurable by @pvpacheco in #718
• feat: switch libpostal data to Senzing updated dataset by @adamdecaf in #735
• stringscore: add Soundex phonetic matching algorithm by @SamOluwadamilare300 in #743

What's Changed

• fix(deps): update opentelemetry-go monorepo to v1.41.0 by @renovate[bot] in #714
• fix(deps): update all by @renovate[bot] in #716
• fix(deps): update module go.opentelemetry.io/otel/trace to v1.42.0 by @renovate[bot] in #717
• chore(deps): update all (major) by @renovate[bot] in #715
• fix(deps): update opentelemetry-go monorepo to v1.43.0 by @renovate[bot] in #727
• feat: add MCPS message signing to MCP server by @razashariff in #730
• build(deps-dev): bump activesupport from 7.2.0 to 7.2.3.1 in /docs by @dependabot[bot] in #722
• build(deps-dev): bump json from 2.18.1 to 2.19.2 in /docs by @dependabot[bot] in #721
• chore(deps): update all by @renovate[bot] in #719
• build(deps): bump addressable from 2.8.7 to 2.9.0 in /docs by @dependabot[bot] in #728
• build(deps): bump github.com/jackc/pgx/v5 from 5.8.0 to 5.9.0 by @dependabot[bot] in #731
• fix(deps): update all by @renovate[bot] in #732
• feat(mcp): AgentPass agent-identity verification for MCP endpoint by @razashariff in #733
• fix(deps): update all by @renovate[bot] in #734
• build(deps): bump github.com/jackc/pgx/v5 from 5.9.0 to 5.9.2 by @dependabot[bot] in #736
• chore(deps): update all by @renovate[bot] in #737
• feat(mcp): stamp agent identity into search_entities responses by @MorganaFuture in #739
• fix(docs): Updating API spec by @elliotcourant in #742
• build(deps-dev): bump nokogiri from 1.19.1 to 1.19.3 in /docs by @dependabot[bot] in #741
• fix(deps): update all by @renovate[bot] in #738
• build(deps-dev): bump faraday from 2.14.1 to 2.14.2 in /docs by @dependabot[bot] in #744
• chore(deps): update all by @renovate[bot] in #745
• fix(deps): update all by @renovate[bot] in #746
• fix(deps): update all by @renovate[bot] in #747

New Contributors

• @pvpacheco made their first contribution in #718
• @razashariff made their first contribution in #730
• @elliotcourant made their first contribution in #742
• @SamOluwadamilare300 made their first contribution in #743

Full Changelog: v0.61.1...v0.62.0