ioc-finder

Changed

• Sped up domain name parsing by bypassing pyparsing for the standalone domain parser: candidate spans from the cheap regex are now validated against a TLD set directly, picking the rightmost TLD that leaves ≥1 preceding label. The domain_name grammar is still used as a sub-grammar inside email/url/xmpp parsers, where its inner label loop has been collapsed into a single Regex (replacing the OneOrMore(label + "." + FollowedBy(...)) construction).
• The scheme_less_url and scheme_less_url_complete grammars no longer also match URLs with a scheme; they only match URLs without one (#244). parse_urls and parse_urls_complete now run the scheme-ful grammar first and mask each matched URL before running the scheme-less grammar, so user-facing output of find_iocs is unchanged but embedded scheme-less hosts inside a scheme-ful URL's query (e.g. https://shortener.com/?url=foo.com/bar) no longer surface as a second URL.

Fixed

• urls_complete now accepts @ in URL paths, matching the RFC 3986 pchar definition (e.g. https://example.com/users/@alice is now captured in full).
• parse_user_agents no longer absorbs a trailing token like TLP from input such as Mozilla/4.0 (...) TLP:RED. A bare (version-less) platform token must not be immediately followed by :. Versioned platforms (e.g. Chrome/91.0) are unaffected. (#227)

Added

• New socket_addresses IOC type with a matching parse_socket_addresses helper (#248). Accepts ipv4:port and bracketed [ipv6]:port shapes with a 1..65535 port range. IPv6 host validation reuses the existing _is_valid_ipv6 helper so the same shortened forms accepted by parse_ipv6_addresses are accepted here too. Observed text is preserved verbatim (no leading-zero normalization). Port 0 is intentionally excluded as IANA-reserved.
• New ipv6_cidrs IOC type with a matching parse_ipv6_cidrs helper (#121). It accepts the same shortened/::/trailing-:: forms as the existing IPv6 address parser plus a /0..128 bit range, and integrates with the existing CIDR-vs-URL and parse_address_from_cidr handling. CIDR parsing runs against the pre-fang text so the standard fanger (which rewrites ::/ to ://) cannot silently swallow indicators like 2001:db8::/32.
• Added structured logging throughout the library using the standard logging module (#287). The package follows the standard library convention of attaching a NullHandler so consumers see no output unless they configure logging. find_iocs emits INFO at start/finish (text length, IOC type count), WARNING when unsupported types are passed via included_ioc_types (they are now ignored rather than silently treated as misses), and DEBUG for per-type result counts and lower-level transforms (text fanging, URL cleaning). Apps can opt in with e.g. logging.getLogger("ioc_finder").setLevel(logging.DEBUG).

Added

• Added regex prefiltering to all observable types (both default and non-default)

Changed (Internal)

• Switched fully over to UV.
  • Changed Dependabot to the uv ecosystem so pyproject.toml and uv.lock are updated atomically.
  • Removed the generated requirements.txt / requirements_dev.txt compatibility exports and the make export-requirements target. Contributors should use uv sync --locked --group dev to set up a dev environment.

Fixed

• parse_imphashes_ and parse_authentihashes_ no longer return the same hash multiple times when the input labels it with different separators (e.g. imphash: <h>, imphash <h>, imphash - <h>). The deduplication now runs on the extracted hash itself rather than the full keyword+separator+hash match string. (#213)

Added

• Added regex pre-filter for all default indicator types to speed parsing of defaults.

Fixed

• IPv6 parser now accepts :: (the unspecified address) and trailing-:: forms (e.g. 1::), which were previously rejected by the grammar.

Changed

• Extended the regex pre-filter approach (introduced for the domain and ATT&CK parsers) to the remaining hot-path parsers. parse_complete_email_addresses, parse_email_addresses, parse_ipv6_addresses, parse_imphashes_, parse_authentihashes_, parse_urls, parse_urls_complete, parse_mac_addresses, parse_user_agents, and parse_file_paths now use cheap candidate regexes to narrow where the pyparsing grammar runs, dropping benchmark mean from ~7.1s to ~2.4s on the long benchmark article. The parse_domain_names and ATT&CK helpers were folded onto the same shared _scan_candidates helper (no behavior change).
• Added scripts/find_hotspots.py to reproduce per-grammar timings and a cProfile drill-down for future tuning.
• Breaking: DEFAULT_IOC_TYPES has been reduced from 30 entries to 9 common indicator types (domains, urls, ipv4s, ipv6s, email_addresses, md5s, sha1s, sha256s, cves) to speed up parsing in the typical case. Callers that relied on default-parsing of obscure types (e.g. bitcoin_addresses, attack_techniques, imphashes, urls_complete, etc.) must now pass included_ioc_types=SUPPORTED_IOC_TYPES (or an explicit subset) to get them. The CLI parses the common defaults by default; pass --all to parse every supported type. (#340)

Added

• New SUPPORTED_IOC_TYPES constant exposing the full list of parseable IOC types. Both DEFAULT_IOC_TYPES and SUPPORTED_IOC_TYPES are now exported from the ioc_finder package. (#340)
• New --all CLI flag to parse every supported indicator type instead of the common defaults. (#340)

Removed

• Breaking: Removed the deprecated parse_imphashes and parse_authentihashes keyword arguments from find_iocs(), along with the --no_import_hashes and --no_authentihashes CLI flags. To disable parsing of these hash types, omit "imphashes" / "authentihashes" from included_ioc_types instead. (#333)

Changed

• The domain parser has been sped up with a regex pre-filter to identify possible candidate domains before using the grammar (#335)
• The enterprise/mobile/pre-attack mitigation, tactic, and technique parsers now use a regex pre-filter to identify candidate ATT&CK-ID spans before running the grammar, significantly improving performance on large inputs (#238)