github.com/Anakin-Inc/anakin

Added

• Anonymous telemetry — event-level usage telemetry with hourly batch reporting (opt-out: TELEMETRY=off). See TELEMETRY.md
• Transparency endpoint — GET /v1/telemetry/status shows exactly what telemetry data will be sent
• Startup banner notice — telemetry status displayed on server boot
• Privacy documentation — TELEMETRY.md details what is and isn't collected
• Failed domains tracking — domains where scraping fails are reported to help improve the product

Added

• Anonymous telemetry — event-level usage telemetry with hourly batch reporting (opt-out: TELEMETRY=off). See TELEMETRY.md
• Transparency endpoint — GET /v1/telemetry/status shows exactly what telemetry data will be sent
• Startup banner notice — telemetry status displayed on server boot
• Privacy documentation — TELEMETRY.md details what is and isn't collected

AnakinScraper OSS v0.1.0

The open-source web scraping API for AI. Turn any website into LLM-ready markdown or structured data.

Features

• Sync + async scraping — POST /v1/scrape (blocks) or /v1/url-scraper (poll)
• Batch scraping — up to 10 URLs in one request
• Handler chain — fast HTTP fetch with automatic fallback to Camoufox anti-detect browser
• Structured JSON extraction — Gemini 2.5 Flash AI extraction (bring your own API key)
• Domain configs — per-domain handler selection, timeouts, retries, content validation, custom headers
• Proxy auto-selection — Thompson Sampling picks the best proxy per domain
• HTML to Markdown — intelligent content extraction with boilerplate removal

Quick Start

git clone https://github.com/AnakinAI/anakinscraper-oss.git && cd anakinscraper-oss && make up
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

CLI

pip install anakin-cli
anakin scrape "https://example.com" --api-url http://localhost:8080

Infrastructure

• Single Go binary + Camoufox browser + PostgreSQL
• Docker Compose for one-command deployment
• 35 unit tests
• AGPL-3.0 license

Initial open-source release.

Features

• Sync scraping — POST /v1/scrape returns results directly (30s timeout)
• Async scraping — POST /v1/url-scraper with polling
• Batch scraping — up to 10 URLs in one request
• Handler chain — HTTP fetch with automatic fallback to Camoufox anti-detect browser
• Structured JSON extraction — Gemini 2.5 Flash AI extraction (bring your own API key)
• Domain configs — per-domain handler selection, timeouts, retries, content validation, custom headers
• Proxy auto-selection — Thompson Sampling picks the best proxy per domain
• HTML to Markdown — intelligent content extraction with boilerplate removal

Tools

• CLI — anakin scrape/batch/health commands (anakin-cli)
• Web dashboard — React UI for scraping, job tracking, domain configs, proxy monitoring
• OpenClaw skill — AI agent integration

Infrastructure

• Single Go binary + Camoufox browser + PostgreSQL
• Docker Compose for one-command deployment
• GitHub Actions CI
• 35 unit tests