OpenSERP v0.8.6

Full diff: v0.8.3...v0.8.6

Added

• Added structured CLI search output via openserp search <engine> <query>.
  • Supports json, markdown, text, and ndjson with --format.
  • Supports --limit, --lang, --region, --start, --site, --file, --extract, --features, --full, and --search-timeout.
• Added a shared engine registry used by both the CLI and server.
• Added X-Proxy-Attempts response header for proxy retry visibility.
• Added Ecosia Cloudflare captcha detection for browser and raw modes.
• Added test coverage for CLI output, extraction parsing, proxy rotation, ranking, raw HTTP fingerprinting, and SERP features.

Changed

• CLI output now matches the HTTP API v2.1 response format.
• CLI now defaults to quiet mode, sending logs to stderr and keeping stdout clean.
• Replaced extract_top with extract=N.
• Raw HTTP requests now use tls-client with Chrome-like TLS fingerprints, Client Hints, and header ordering.
• Reworked raw HTTP client pooling to prevent idle client growth.
• Unified browser and raw ranking logic using shared RankState.
• Normalized SERP feature links and text before returning results.
• features=false now consistently disables feature extraction.
• Improved Google and Bing SERP feature extraction, including AI summaries, answer boxes, related questions, and related searches.
• Updated README, OpenAPI spec, examples, and CI configuration.

Fixed

• CLI logs no longer pollute stdout.
• Chrome processes are now cleaned up after CLI browser searches.
• Invalid app.block_resources values no longer cause request panics.
• Challenged proxies are rotated out before retrying.
• Browser health checks no longer run immediately after captcha detection.
• Authenticated SOCKS proxies are now rejected in browser mode instead of failing later.
• Raw and library search paths now share the same rate limiter state.
• Extraction now falls back to nearby results when top candidates fail.
• Google AI Overview placeholders no longer leak into serp_features.
• Improved Google related searches and People Also Ask extraction for newer layouts.
• Improved Bing title/description fallbacks, AI answers, answer boxes, related questions, and related searches.
• Bing click-tracking URLs are stripped from normalized links.
• features=false is now respected by Baidu, Ecosia, Google, and Yandex raw search.
• Yandex browser parsing now reuses the raw HTML parser.
• Fixed ranking edge cases where ads could affect organic result positions.

feat(cli): clean search, add extract/format flags; harden engines & p…

OpenSERP v0.8.3

Full diff: v0.7.15...v0.8.3

Added

• Added URL content extraction via GET /extract and POST /extract.
  • Supports json, markdown, text, and ndjson responses.
  • Supports extraction modes: auto, fast, and rendered.
  • Includes page metadata such as title, description, headings, links, canonical URL, Open Graph tags, schema.org data, and extraction metadata.
• Added search result enrichment with extracted page content using:
  • extract=true
  • extract_top
  • extract_mode
  • min_runes
• Added Markdown/text rendering for extracted result content.
• Added llms.txt / llms-full.txt probing for root URL extraction when enabled.
• Added SSRF-style request protection for extraction: private, loopback, link-local, multicast, and CGNAT targets are blocked by default.
• Added JavaScript and Python usage examples for quickstart, search, SEO, AI, content extraction, and image workflows.
• Added documentation links for official SDKs and integrations: JS/TS SDK, Python package, MCP server, and n8n node.
• Added contributor onboarding docs:
  • docs/ADDING_AN_ENGINE.md
  • docs/GOOD_FIRST_ISSUES.md
  • expanded docs/CONTRIBUTING.md
• Added GitHub issue templates and pull request template.
• Added Makefile targets for build, test, integration tests, lint, run, and fmt.
• Added Docker image workflow for validation and multi-arch publishing.
• Added GitHub release workflow for building and uploading release binaries.

Changed

• Bumped CLI version to 0.8.3.
• Bumped OpenAPI spec version to 2.2.0.
• Default search limit is now 10.
• Small/default limits now parse only the first SERP page; pagination is used for larger limits when supported by the engine.
• features now defaults to true, enabling SERP feature extraction unless explicitly disabled.
• Docker usage docs now use the prebuilt karust/openserp:latest image.
• CI now pins golangci-lint and uses the repository lint config.

Fixed

• Fixed default search limit behavior across Google, Baidu, Bing, DuckDuckGo, Ecosia, and Yandex.
• Fixed Google result parsing by matching organic selectors more carefully across different SERP layouts.
• Hardened Google answer-box parsing to avoid panics on short answer text.
• Fixed Baidu empty-result handling: pages that never hydrate now return ErrSearchTimeout instead of a successful empty result.
• Centralized engine panic recovery in the resilient search layer.
• Fixed request deadlines so the server timeout is derived from the retry budget instead of cutting retries short.
• Fixed rate limiter behavior by reusing configured limiters instead of creating new ones per call.
• Improved circuit breaker behavior so context cancellations and deadline errors are not recorded as engine failures.
• Hardened extraction timeouts and batch enrichment behavior so slow or failed extraction targets do not turn the whole search request into a 500.
• Stabilized integration tests and deduplicated engine smoke-test setup.

extract region/UULE resolution into dependency-free core/region subpa…

…ckage

This release larger API refresh: Ecosia support, v2.1 response envelopes, HTML parsing endpoints, region targeting, faster mega search modes, and more observable browser/proxy behavior.

Highlights

• Added Ecosia as a supported search engine.

  • New endpoints: /ecosia/search and /ecosia/image.
  • Added Ecosia support to the CLI and server engine registry.
  • Added raw-mode Ecosia web search support.

• Added HTML SERP parser endpoints.

  • Engines can now expose POST /{engine}/parse for parsing raw SERP HTML into the normalized OpenSERP response envelope.
  • Parser logic is shared with browser/raw search paths, reducing selector drift between modes.
  • Supported parser implementations were added for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia.

• Introduced the v2.1 search response shape.

  • meta.version is now 2.1.
  • Results now use explicit result type values such as organic, ad, answer_box, people_also_ask, and related SERP modules.
  • Ads and organic results are ranked separately while position.absolute preserves the mixed on-page SERP position.
  • meta.engines_responded and meta.engine_errors are available for multi-engine requests.

• Added region targeting.

  • New region query parameter is accepted on search and image endpoints.
  • Google, Bing, DuckDuckGo, and Yandex now map the region hint into engine-specific market/location parameters where supported.
  • Cache keys now include normalized region and market data.

• Added mega search execution modes.

  • mode=balanced keeps the existing parallel aggregation behavior and remains the default.
  • mode=any tries engines sequentially until the first successful response.
  • mode=fast chooses the fastest available engine based on circuit-breaker latency history.
  • New dedupe and merge controls tune aggregation behavior.
  • New app.mega_timeout bounds total mega request time and returns partial results when slower engines miss the deadline.

API Changes

• Added response headers:

  • X-Network-Bytes reports counted network bytes for the request.
  • X-Browser-Profile-Id reports the browser profile IDs used during browser-mode requests.

• Added request header:

  • X-Use-Profile lets debug/controlled requests force a known browser profile ID.

• Error responses now include request_id when available.

• Search errors now expose more precise machine-readable codes, including:

  • all_engines_failed
  • circuit_open
  • request_timeout
  • request_canceled

• Mega search failures now include sanitized per-engine error details in meta.engine_errors.

Compatibility Notes

• The public JSON response schema changed from v1 to v2.1.

  • is_ad was removed; use type == "ad" instead.
  • position.page and position.on_page were removed; use pagination plus position.absolute.
  • domain_info booleans such as is_gov, is_edu, and is_social were replaced by domain_info.category.
  • classification fields are now omitted when empty.

• limit now means the number of organic results requested. Ads may be returned in addition to the requested organic limit.

• Query parameters are now trimmed before validation and cache key generation, so whitespace-only queries are rejected consistently and equivalent query inputs share cache entries.

• Mega search cache keys now include mode, dedupe, merge, region, and normalized engine lists. mode=fast responses are not cached.

Search Quality and Parsing

• Improved SERP result classification and ad positioning.
• Centralized selectors across browser, raw, and parse flows.
• Added parser implementations and tests for multiple engines.
• Improved Bing redirect URL unwrapping and result normalization.
• Improved Baidu, Yandex, Google, Bing, DuckDuckGo, and Ecosia parsing coverage.
• Raw search paths now share common HTTP helpers, status classification, TLS behavior, Accept-Language handling, and parser reuse.

Browser and Proxy Improvements

• Reworked browser profile selection with a larger profile catalog, weighted profile selection, runtime browser-version adaptation, locale/timezone matching, and per-session profile stability.
• Added browser profile reporting in logs and response headers.
• Added X-Use-Profile validation for forced profile selection.
• Improved headless browser fingerprint patching, including plugins/mime types, permissions, WebGL/profile surfaces, and Chrome runtime stubs.
• Fixed request-blocking cleanup so browser resource blocking no longer panics during shutdown/cancellation.
• Improved authenticated proxy handling under concurrent browser requests.
• Browser pool stats and proxy lane stats are exposed more consistently in stats responses.

Reliability and Observability

• Added network byte telemetry for browser and raw HTTP paths.
• Circuit breaker stats now track average successful response latency.
• Dedicated and mega endpoints report richer per-engine failure details.
• Mega search can return partial results instead of waiting indefinitely for slow engines.
• Error detail sanitization masks credentialed proxy URLs before returning metadata.

Documentation

• Updated README examples for Ecosia, region targeting, mega modes, and v2.1 response examples.
• Updated OpenAPI version to 2.1.0.
• Documented new headers, error codes, region parameter, mega mode controls, and updated response schemas.
• Updated configuration example with app.mega_timeout, Ecosia engine settings, and current rate-limit defaults.

Fixes

• Fixed Ecosia proxy policy, validation, raw mode, and OpenAPI integration.
• Fixed browser request-blocking cleanup panic.
• Fixed Baidu raw search to reuse the shared parser and classify captcha/empty-result states more reliably.
• Fixed raw HTTP proxy error wrapping so proxy failures are categorized correctly.
• Fixed cache key normalization for query inputs and proxy market signals.

New Contributors

• @miku made their first contribution in #26

Full Changelog: v0.7.2...v0.7.13

OpenSERP 0.7.2

Highlights

• New API response envelope for search and image endpoints.
• Unified normalized result model with stable IDs, domain enrichment, classification, metadata, and pagination.
• New output formats: json, markdown, text, ndjson.
• Better locale, language, region, and date-filter handling across engines.
• Major browser runtime improvements: persistent browser connection, isolated contexts, profiles, smart waits, resource blocking.
• Better proxy health handling and captcha isolation.
• New OpenAPI spec, architecture docs, contributing guide, CI, and integration tests.

Added

API response model

• Added JSON response envelopes with:

  • query
  • meta
  • results
  • pagination
  • engines_responded
  • engines_failed

• Added normalized Result model with:

  • stable result IDs
  • result type
  • position metadata
  • engine metadata
  • domain enrichment

• Added dedicated image response envelope and image result structure.

• Added result clustering for /mega/search, grouping same URLs across engines.

• Added Bing redirect URL unwrapping before result hashing.

Search features

• Added locale propagation into browser profile and Accept-Language.
• Added improved query handling with locale support and date filters.
• Improved URL builders and added edge-case coverage.

Browser/runtime

• Replaced generic stealth mode with custom browser profile functionality.
• Added browser profiles and profile coherence tests.
• Added persistent browser connection reuse.
• Added isolated browser contexts per request.
• Added configurable resource/tracker blocking.
• Added smart waits for SERP loading.
• Added fingerprint debug tests and endpoint.
• Added graceful shutdown with drain signal and browser cleanup.

Error handling

• Added typed 400 validation errors with stable reason codes.
• Added limit validation range: 1..100.
• Improved error handling across search engines.
• Added safe HTTP response draining/closing.

Proxy and captcha handling

• Isolated proxy health from captcha/parser errors.
• Replaced proxy re-enable thrashing with quarantine behavior.
• Improved captcha detection selectors.
• Added single source of truth for captcha selectors.
• Captcha solver misconfiguration now exits non-zero.

Docs and release assets

• Added OpenAPI spec.
• Added Swagger UI/docs embedding.
• Added architecture documentation.
• Added contributing guide.
• Updated README to match new API format and features.
• Clarified OpenSERP open-source status and third-party affiliation wording.

CI and tests

• Added GitHub CI workflow.

• Added integration workflow.

• Added integration tests for:

  • Baidu
  • Bing
  • DuckDuckGo
  • Google
  • Yandex

• Added sanitized real SERP HTML fixtures.

• Added raw HTML parser tests.

• Added server endpoint coverage.

• Added browser, profile, captcha, middleware, retry, proxy, locale, and response tests.

Docker

• Refactored Docker setup.
• Switched to chromedp/headless.
• Updated Dockerfile and docker-compose configuration.

Changed

• Search internals now distinguish raw engine results from normalized API results.
• Multi-engine search now returns responded/failed engine metadata.
• Non-JSON output formats bypass JSON response cache to avoid cache pollution.
• Logging is now more structured and includes more useful request/search context.
• Request context now propagates through search, retry, limiter, and browser navigation.
• README examples were updated for the new response shape.

Fixed

• Fixed API response format issues.
• Fixed captcha selector detection problems.
• Fixed CI issues.
• Fixed linter errors.
• Fixed unsafe/unclear engine error handling.
• Fixed proxy health behavior where captcha/parser failures could incorrectly affect proxy status.
• Fixed HTTP response cleanup with safe drain-and-close logic.

Breaking / migration notes

• Search endpoints no longer return bare arrays; they now return response envelopes.
• Consumers should read results from results instead of treating the whole response as an array.
• Error responses now include stable machine-readable reason values.
• limit is now validated and must be between 1 and 100.
• Clients relying on old response structs should update to the new v1 schema.

Full Changelog: v0.6.0...v0.7.2

OpenSERP v0.6.0

This release focuses on server reliability, proxy control, and deployment readiness.

Highlights

• Added /health endpoint for lightweight service checks.
• Added retries, circuit breaker protection, optional dedicated-endpoint fallback, and CORS controls for server mode.
• Added bounded TTL response cache for dedicated and mega endpoints.
• Added proxy support with tagged proxy pools, health-based rotation, fail-closed behavior, and runtime proxy selection.
• Added per-request proxy override via X-Use-Proxy header.
• Added proxy and resilience observability through /stats, /stats/cache, /stats/proxy, and /stats/cb.
• Added custom browser binary path support.
• Expanded query controls with Google filter support and broader start pagination support.

Proxy and Runtime Improvements

• Reworked proxy configuration around proxies.global, proxies.entries, and per-engine proxy tags.
• Added lazy browser reuse per effective proxy to avoid duplicate browser launches across engines.
• Added runtime proxy override support for API requests without restarting the server.
• Improved fail-closed behavior when a required proxy tag has no healthy candidates.
• Added masked proxy reporting so response headers and stats do not expose credentials.

Caching and Resilience

• Added cache support for dedicated endpoints and mega endpoints with bounded size and TTL.
• Avoid fallback-response caching on dedicated endpoints so the primary engine can recover naturally.
• Added retry and circuit-breaker behavior around engine execution.
• Added resilience stats endpoints for easier troubleshooting in server deployments.

Packaging and Config

• Updated the default shipped config.yaml to use safe direct-traffic defaults instead of local proxy test endpoints.
• Fixed Docker image config discovery by setting the app working directory to the config location.
• Fixed docker-compose.yaml environment keys to the current server.* config names.
• Fixed explicit --config loading so a provided config file path is actually used.

Acknowledgements

• Parts of the ideas and implementation direction for this release were originally explored in PR #21 by Sai-Prashanth123: #21
• That PR contributed useful groundwork around production-readiness areas such as health checks, caching, CORS, structured errors, request logging, and resilience-related server improvements.
• The final 0.6.0 implementation was reviewed, reworked, debugged, and adapted further in the main codebase to match OpenSERP's current project vision and configuration model.

Full Changelog: v0.5.4...v0.6.0

• Fix duckduck date and limit filtering.
• Deduplicate duckduck and bing search results.

• Bing search + images
• Duckduckgo search + images
• New Megasearch + Megaimage APIs
• Antibot measurments for non-stealth headless
• Updated logs
• Bugfix: CLI arguments prioritization
• Several other fixes and optimizations
• Update readme

v0.4.1

Compare

Choose a tag to compare