0.9.3 is a security release. It closes five coordinated-disclosure advisories in the PDF processing path and the Docker Playground UI, and ships the 33 bug fixes that accumulated on develop since 0.9.2, most of them in the Docker server. There are no new features and no breaking changes. Users who accept untrusted URLs on the Docker server, or who open PDFs from sources they do not control, should upgrade.

Security

The PDF path was the common thread. PDFContentScrapingStrategy is selectable from an untrusted Docker API request body, and it fetches with requests outside the browser, so none of the Chromium-side egress or resource controls applied to it.

• Arbitrary file write via PDF image-write fields (CWE-22, high): PDFContentScrapingStrategy had no field allowlist, so an untrusted request body could set save_images_locally and image_save_dir and make the server write extracted images to a path of the caller's choosing. Those fields are now filtered at the trust boundary and extract_images is forced off for untrusted bodies. Credit: Zhixi "Jace" Sun (manus-use). (GHSA-xpp7-j28w-2gvx)
• SSRF via PDF download redirects (CWE-918, high): the PDF download followed redirects without consulting any destination policy, so a public URL that redirected to an internal address reached it. Redirects are now resolved by hand with a per-hop destination check, bounded at five hops, and the peer IP of the response actually read back is validated to close DNS rebinding. The Docker server installs its egress policy into this path at boot. Credit: Nguyen Tran Thanh Lam (c240030). (GHSA-q5rj-45vw-vp2g)
• Denial of service via unbounded PDF size and page count (CWE-400, medium): a remote PDF was streamed to disk and parsed with no cap on bytes or pages. Downloads now stop at max_pdf_bytes (100 MiB default), enforced on the running total rather than the caller-supplied content-length, and parsing stops at max_pdf_pages (2000 default). Untrusted bodies cannot raise their own caps. The Docker config now ships a non-zero limits.wall_clock_s of 300 seconds. Credit: Nguyen Tran Thanh Lam (c240030). (GHSA-v2rm-hvrj-2x9q)
• XSS via unescaped PDF text in cleaned_html (CWE-79, medium): paragraph text taken verbatim from a PDF was written into cleaned_html without escaping, so markup embedded in a PDF survived into the result and executed when rendered. Paragraph text is now escaped like every other sink in that function. Credit: Nguyen Tran Thanh Lam (c240030). (GHSA-7g3g-vhm6-79f3)
• DOM-based XSS in the Docker Playground leading to API token theft (CWE-79, high): the result viewer reset syntax highlighting with element.innerHTML = element.textContent, which re-parsed attacker-controlled crawled content as live HTML in the operator's session. The round trip is removed; highlight.js renders safely from textContent. Credit: e1codes. (GHSA-m446-hp3q-qfxp)

All reporters are credited in SECURITY-CREDITS.md. GitHub Security Advisories accompany this release.

Fixed

This release also carries the bug fixes that accumulated on develop since 0.9.2.

Docker server

• PDF scraping is supported by default, and requests selecting PDFContentScrapingStrategy are routed to PDFCrawlerStrategy so the pairing works without extra configuration. (#2094, #2150)
• The egress proxy chains through an upstream HTTP_PROXY / HTTPS_PROXY instead of ignoring it. (#2142)
• Junk proxy environment values fall through to the next candidate, and non-http proxy schemes are refused. (#2094)
• Compose v5 compatibility, clearer warnings on legacy fields, and better playground error handling. (#2094)
• CRAWL4AI_API_TOKEN is forwarded through compose, and .llm.env is optional rather than required. (#2094)
• The disabled-hooks 403 distinguishes the removed hooks.code field from other rejections. (#2094)
• output_path is declared a deprecated no-op rather than silently ignored. (#2094)
• GET /monitor redirects to the dashboard UI. (#2157, issue #2091)
• Failed crawl results are preserved instead of dropped, for both batch and single-URL requests. (#2094, #2134, issue #2133)
• An unavailable IPv6 loopback no longer breaks startup. (#2081, issue #2078)
• mcp is capped below 2 so the v1 low-level API used by mcp_bridge keeps working. (#2148, thanks @weike-zhang)
• Commented environment variable lines in compose are aligned with the environment list. (#2156)

Crawler and core

• ManagedBrowser no longer leaks a Playwright driver process when the browser fails to launch inside __aenter__. (#2160)
• PDFCrawlerStrategy placeholder responses are no longer vetoed as anti-bot blocks, which previously failed every PDF crawl and burned the retry budget. (#2138, issue #2135)
• Cookies are carried across manual PDF redirect hops, so gated and CDN-signed PDFs download correctly. (#2159)
• Unconditional setTimeout waits are removed from the overlay and consent removal scripts, which could hang a crawl under a restrictive CSP. (#2139)
• remove_overlay_elements no longer removes <body> when the body carries a global popup class. (#2163, thanks @Nalhin)
• The body-visibility timeout is configurable and validated, and the timeout warning is emitted even with verbose=False. (#2117, #2131, #2145, issues #2116, #2129, #2144)

Documentation

• Self-hosting and migration guides updated for 0.9.x. (#2093)
• The PDFCrawlerStrategy plus PDFContentScrapingStrategy pairing requirement is documented.

Tests

• tests/unit/test_pdf_download_limits.py: 22 tests covering per-hop destination validation, DNS rebinding, redirect bounds, byte and page caps, and untrusted-body clamping.
• tests/unit/test_pdf_html_escaping.py: escaping of PDF paragraph text in cleaned_html.
• deploy/docker/tests/test_security_pdf_image_write.py: rejection of image-write fields from untrusted bodies.
• Docker endpoint coverage for crawl failures and for the per-URL crawler_configs PDF guard.

Breaking Changes

None.

Crawl4AI v0.9.2: Maintenance Bug Fixes

July 2026 - 2 min read

---

I'm releasing Crawl4AI v0.9.2, a small maintenance patch that cleans up a dispatcher resource leak and fixes a handful of Docker and GPU build issues. No new features, no breaking changes.

If you're on v0.9.1, upgrade freely.

What's new at a glance

• Dispatcher cleanup: MemoryAdaptiveDispatcher no longer leaves crawl tasks and browser pages running after a streaming crawl is closed
• Docker Playground: "Advanced Config" no longer returns a 400
• Docker Monitor: the dashboard WebSocket no longer returns a 500 under JWT auth
• Docker image: Playwright headless shell is now included
• GPU builds: ENABLE_GPU=true Docker builds no longer fail on the CUDA toolkit

Bug fixes

Dispatcher (1 fix)

• Streaming cleanup leak: When the async generator from MemoryAdaptiveDispatcher.run_urls_stream() was closed or cancelled mid-stream, its per-URL crawl tasks kept running in the background. A later arun_many() on the same crawler could then overlap the old tasks while they still held the same browser, contexts, and pages — leaking pages and emitting stray Playwright TargetClosedErrors. Closing the stream now cancels and awaits every in-flight task and drains any queued-but-unstarted URLs before returning. (#2071, thanks @reallav0)

Docker (3 fixes)

• Playground "Advanced Config" 400: /config/dump requires a type, but the playground's pyConfigToJson only sent code. The request now includes the config type, aligns the stream fallback with the dump shape, and teaches shouldUseStream both nestings. (#2059, thanks @Pitchfork-and-Torch)
• Monitor WebSocket 500: the monitor dashboard's /monitor/ws endpoint 500'd because a router-level token_dep is HTTP-request-only and can't inject into WebSocket scopes. Auth is now enforced solely by AuthGateMiddleware; admin routes keep require_admin. (#2060, thanks @Pitchfork-and-Torch)
• Headless shell packaging: the Playwright headless shell is now installed in the Docker image. (#2067, thanks @reallav0)

GPU / Build (1 fix)

• CUDA toolkit on Bookworm: ENABLE_GPU=true builds failed with "Package has no installation candidate" because nvidia-cuda-toolkit lives in Debian Bookworm's non-free component, which the python:3.12-slim-bookworm base image omits. The Dockerfile now adds the non-free (and contrib) apt sources before the GPU install block. (#2020, thanks @harshmathurx)

Tests

• Regression test added for the dispatcher stream-close cleanup, asserting in-flight tasks are cancelled, the queue is drained, and session count returns to zero.
• Updated the monitor auth test for the WebSocket-safe router config, and corrected a regex group name and the Playwright cache-copy target in the Docker tests.

Breaking changes

None.

Upgrade

pip install -U crawl4ai
crawl4ai-doctor  # verify installation

Docker users: pull the latest image once the Docker release workflow finishes.

Verification

python docs/releases_review/demo_v0.9.2.py

Acknowledgments

Thanks to the community contributors who made this release possible: @reallav0 (#2071, #2067), @Pitchfork-and-Torch (#2059, #2060), @harshmathurx (#2020).

Support & Resources

• Documentation
• GitHub Issues
• Discord Community

Crawl4AI v0.9.1: Bug Fixes & PruningContentFilter Whitelist

July 2026 - 3 min read

---

I'm releasing Crawl4AI v0.9.1, a patch release that ships 12 bug fixes across Docker, browser, core, and extraction, plus one new feature for PruningContentFilter.

No breaking changes. If you're on v0.9.0, upgrade freely.

What's new at a glance

• PruningContentFilter whitelist: New preserve_classes / preserve_tags parameters to protect specific elements from density-based pruning
• Windows browser fix: Default channel='chromium' no longer crashes Playwright on Windows
• Docker hardening: 6 fixes for auth gate, supervisord, redis, tmpfs, and FastAPI compatibility
• HTTP timeout fix: page_timeout was passed in milliseconds to aiohttp (which expects seconds), effectively disabling timeouts in HTTP mode
• Dependency: lxml ceiling widened to allow 6.x

PruningContentFilter whitelist

PruningContentFilter's density-based scoring is great at stripping boilerplate, but it sometimes takes short metadata elements — author names, timestamps, attribution lines — along with it. The new preserve_classes and preserve_tags parameters let you whitelist specific CSS classes or HTML tags that should never be pruned, regardless of their density score.

from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

filter = PruningContentFilter(
    threshold=0.48,
    preserve_classes=["author", "byline", "dateline"],
    preserve_tags=["time", "address"],
)
generator = DefaultMarkdownGenerator(content_filter=filter)

config = CrawlerRunConfig(markdown_generator=generator)

Whitelisted nodes skip scoring entirely. Default is empty sets — no behavior change for existing users. (#1900, thanks @hafezparast)

Bug fixes

Docker (6 fixes)

• Auth gate UI: Dashboard and playground now load when JWT auth is active. Token input bar added to both UIs; all fetch calls use authFetch() with Bearer token. API routes remain fail-closed. (#2037)
• Supervisord/Redis dirs: Pidfile and RDB snapshot dirs moved to writable paths for read-only rootfs deployments. (#2047, thanks @TobiasWallura-xitaso)
• tmpfs writable: Read-only tmpfs mounts made writable. (#2027, thanks @nightcityblade)
• FastAPI cap: Pinned FastAPI below 0.137 to avoid compatibility breakage. (#2025, thanks @nightcityblade)
• Redis auth: Rate-limit Redis storage now authenticates with the configured password. (#2040, thanks @harshmathurx)
• Posture tests updated: Dashboard/playground moved to public UI assertions to match auth gate fix.

Browser (2 fixes)

• Windows channel crash: channel='chromium' (the default) caused Playwright to look for a system Chrome install instead of the bundled binary, crashing on Windows with TargetClosedError. The default channel is no longer passed to Playwright. (#2051, thanks @fstark96)
• Context snapshot leak: Browser contexts from snapshot are now properly closed. (#1999, thanks @nightcityblade)

Core (2 fixes)

• HTTP timeout: page_timeout (60000ms) was passed directly to aiohttp.ClientTimeout which expects seconds, making the effective timeout 16.7 hours. Now correctly divided by 1000. (#1894, thanks @hafezparast)
• Best-first ordering: Batch ordering in BestFirstCrawlingStrategy stabilized for deterministic crawl order. (#1998, thanks @nightcityblade)

Extraction (1 fix)

• Table attributes: html2text now preserves all attributes on table tags when bypass_tables is enabled. (#2007)

Dependencies (1 fix)

• lxml 6.x: Widened lxml ceiling from <6 to <7 so crawl4ai can co-install with packages requiring lxml 6.x (e.g. scrapling). (#2019)

Housekeeping

• Removed dead normalize_url duplicates and accidental adaptive_crawler copy. (thanks @RajanChavada)
• Sponsor logos hosted locally to fix broken GitHub rendering.

Upgrade

pip install -U crawl4ai
crawl4ai-doctor  # verify installation

Docker users: pull the latest image once the Docker release workflow finishes.

Acknowledgments

Thanks to the community contributors who made this release possible: @hafezparast (#1894, #1900), @nightcityblade (#1998, #1999, #2025, #2027), @fstark96 (#2051), @TobiasWallura-xitaso (#2047), @harshmathurx (#2040), @RajanChavada (#2042).

Support & Resources

• Documentation
• GitHub Issues
• Discord Community

0.9.0 is a major, secure-by-default release of the Crawl4AI Docker API server. The out-of-the-box deployment is now hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary. This release contains breaking changes for the self-hosted HTTP server only. The core pip library (SDK / in-process use) is unchanged.

What changed: the Docker server moved from an open, trust-the-caller posture to a closed, secure-by-default one. Defaults that used to be permissive (open bind, no auth, request-supplied browser internals, TLS verification off, Redis with no password) are now safe by default and gated behind explicit configuration.

What you must do: set CRAWL4AI_API_TOKEN and re-issue any tokens, then review whether you relied on any of the request fields or features that are now configured server-side. Most plain "crawl these URLs" users only need the two steps in the "Everyone" section of the migration guide. The full guide is at deploy/docker/MIGRATION.md.

Security

This release completes the secure-by-default hardening of the Docker API server begun in 0.8.7 and 0.8.8. It moves the worst remaining issues from mitigation to architecture: unauthenticated access and request-supplied code/config are eliminated by design rather than patched in place. Every change is hardening; users self-hosting the Docker server should upgrade and follow the migration guide.

• Authentication on by default, loopback bind: the server no longer serves an unauthenticated API on 0.0.0.0. With no token it binds 127.0.0.1 and prints a one-off local token; exposing it requires CRAWL4AI_API_TOKEN and Authorization: Bearer <token> on every request except GET /health.
• Request trust boundary: a crawl request body now carries declarative, scalar options only. Fields that previously let a caller drive browser internals or arbitrary code are rejected at the network boundary.
• Declarative hooks replace hook code: arbitrary Python hook strings are replaced by a fixed set of declarative actions, removing request-supplied code from the server entirely.
• Strengthened JWT, admin-scoped monitor actions, deny-by-default CORS, strict security headers, TLS verification on, password-protected loopback-only Redis, bounded job queue, generic error responses with correlation ids, and validated webhook headers round out the defense-in-depth posture. See the migration guide for the full list.
• Download path confinement (CWE-22): both download sinks now confine writes with basename plus realpath plus O_NOFOLLOW, closing a path-traversal-to-file-write class. Credit: Y4tacker.
• SSRF destination validation on the streaming crawl path (CWE-918): /crawl/stream and /crawl with stream=true now validate the destination and return HTTP 400 for disallowed targets, matching the non-streaming handlers. Credit: KOH Jun Sheng.
• Request-supplied browser_config.extra_args rejected (CWE-94): launch arguments can no longer be supplied over the network, closing a Chromium launch-arg injection class. Credit: Y4tacker, UDU_RisePho (hoanggxyuuki).

All reporters are credited in SECURITY-CREDITS.md. GitHub Security Advisories accompany this release.

Breaking Changes

These apply to the self-hosted Docker API server only. The pip library is unaffected. See deploy/docker/MIGRATION.md for the step-by-step migration and deploy/docker/SECURITY-VERIFY.md for the deployment checklist.

• Auth is on by default: set CRAWL4AI_API_TOKEN and send Authorization: Bearer <token>. With no token the server binds loopback only.
• Loopback bind by default: the server no longer binds 0.0.0.0 without a token; put a TLS-terminating reverse proxy in front when you expose it.
• Tokens must be re-issued: the JWT implementation changed and tokens from older versions are no longer valid. Re-mint via POST /token.
• Request trust boundary: js_code, js_code_before_wait, c4a_script, proxy / proxy_config, extra_args, user_data_dir, cdp_url, cookies, headers, init_scripts, base_url, deep_crawl_strategy, simulate_user, magic, process_in_browser, and nested LLM config objects are rejected with HTTP 400 when sent over the network. Configure them server-side or use the in-process SDK. Unknown fields are dropped; timeouts, viewport, and scroll counts are clamped.
• Hooks are declarative: hooks.code is replaced by a fixed action set (block_resources, add_cookies, set_headers, scroll_to_bottom, wait_for_timeout). See GET /hooks/info.
• output_path removed, replaced by an artifact id: /screenshot and /pdf store the result and return artifact_id + URL; fetch via authenticated GET /artifacts/{artifact_id} (TTL and quota apply).
• LLM base_url removed: /md, /llm, and /llm/job select a provider by name only; endpoint and key are configured server-side and constrained by config.llm.allowed_providers.
• Monitor actions require an admin token: POST /monitor/actions/* and /monitor/stats/reset need an admin-scope principal.
• CORS deny-by-default: cross-origin browser requests are denied unless listed in security.cors_allow_origins.
• TLS verification on: self-signed / internal TLS targets fail by default. Escape hatches for trusted internal testing: CRAWL4AI_ALLOW_INSECURE_TLS=true, CRAWL4AI_ALLOW_INTERNAL_URLS=true.
• Webhook headers validated: malformed or hop-by-hop / sensitive headers are rejected with HTTP 422.
• Redis requires a password: in-container Redis is loopback-only, password-protected, and its port is no longer published. For external Redis set REDIS_PASSWORD.
• Bounded background job queue: request body size, per-crawl wall clock, queue size, and per-principal concurrency are now capped (configurable; 0 = unbounded).
• Generic 5xx responses: server errors return {"error": "Internal server error", "correlation_id": "…"}; match the id in the logs for detail.

Security Credits

Y4tacker, KOH Jun Sheng, and UDU_RisePho (hoanggxyuuki). See SECURITY-CREDITS.md.

0.8.9 is a follow-up, backward-compatible security patch for the self-hosted Docker API server, closing an SSRF path that 0.8.8 did not cover. Upgrade in place; no configuration changes required.

Security

A security advisory accompanies this release.

• SSRF via proxy settings (CWE-918): the SSRF destination check was applied only to the crawl target URL, not to the proxy address. An unauthenticated /crawl, /crawl/stream, or /crawl/job request could set browser_config.proxy_config.server (or the deprecated browser_config.proxy, or crawler_config.proxy_config, or a --proxy-server / --host-resolver-rules flag in extra_args) to an internal address and route the browser through it, reaching internal services and cloud-metadata endpoints. All proxy destinations are now validated with the same global-routability check before the browser is built, and proxy/DNS-redirecting flags are stripped from extra_args. A legitimate public proxy still works. Credit: Geo (geo-chen).

Backward compatible. Note: raw --proxy-server / --host-resolver-rules / --proxy-bypass-list / --proxy-pac-url flags passed via extra_args are now ignored; configure proxies through proxy_config (which is validated).

0.8.8 is a focused, backward-compatible security patch for the self-hosted Docker API server. Upgrade in place; no configuration changes are required. If you run the Docker server, upgrade. If it is exposed to a network, also set CRAWL4AI_API_TOKEN.

Security

Security advisories accompany this release.

• SSRF filter gaps closed (CWE-918): the Docker server's SSRF protection now rejects any resolved address that is not globally routable, evaluated on IPv6 transition forms too (NAT64 64:ff9b::/96, 6to4 2002::/16, IPv4-mapped, and the unspecified ::), which previously bypassed the explicit blocklist and could reach internal services and cloud-metadata endpoints. SSRF errors no longer echo the resolved address. Credit: internal security audit.
• Arbitrary file write via output_path hardened (CWE-59/22): /screenshot and /pdf now resolve symlinks and re-check containment before writing, and write with O_NOFOLLOW, closing a symlink/TOCTOU bypass of the directory restriction. output_path behavior is unchanged for normal use. Credit: internal security audit.
• LLM credential exfiltration closed (CWE-522/200): the LLM endpoints (/md, /llm, /llm/job) ignore a request-supplied base_url, so the configured provider key can no longer be redirected to an attacker endpoint. LLMConfig additionally refuses to resolve protected environment variables via the env: token form. The base_url field is still accepted but no longer honored. Credit: Geo (geo-chen); the env: hardening from internal security audit.
• CRLF-safe logging (CWE-117) and webhook request-header validation (CWE-93): log records are stripped of CR/LF/control characters, and user-supplied webhook headers are validated (name pattern, no control characters, hop-by-hop/sensitive headers denied).

All changes are backward compatible.

Coming next: secure-by-default Docker server (~1-2 weeks)

The next release is a larger, secure-by-default update for the self-hosted Docker API server, with intentional breaking changes. We are giving advance notice so you can prepare. If you run the Docker server, start planning now and test in staging before upgrading:

• Authentication will be on by default. The server binds loopback unless a credential (CRAWL4AI_API_TOKEN) is configured.
• Request bodies are validated more strictly and safer defaults apply (TLS verification on, stricter outbound egress controls, declarative hook actions instead of inline code).
• A few request options move server-side: /screenshot and /pdf return an artifact id instead of a file path, and the LLM endpoint is selected by provider name.
• Hardened container defaults (least-privilege compose, Redis authentication, loopback bind).

A full migration guide will accompany the pre-announcement on Discord and X.

0.8.7 is a security-hardening release. It bundles every responsibly-disclosed vulnerability patched since 0.8.6, plus the new DomainMapper feature and a batch of scraping, deep-crawl, and LLM fixes.

Security

This release fixes multiple critical vulnerabilities in the Docker API server. If you self-host the Docker API, upgrade immediately. Two GitHub Security Advisories accompany this release.

• CRITICAL: AST Sandbox Escape leading to Pre-Auth RCE (CVSS 9.8, CWE-94/913): a gi_frame.f_back frame-chain escape in the computed-field eval() path. Removed eval() from computed fields entirely and deleted _safe_eval_expression. Credit: Song Binglin (q1uf3ng).
• CRITICAL: Hook Sandbox Escape RCE (CVSS 9.8, CWE-94): injected module objects (asyncio, json, re) carried a full __builtins__, bypassing the __import__ block. Stripped injected builtins and removed dangerous allowlist entries. Credit: by111 (August829).
• CRITICAL: Hardcoded JWT Secret (CVSS 9.8, CWE-798): the default signing key "mysecret" allowed token forgery. Removed the default, reject weak/short secrets, and auto-generate an ephemeral key when JWT is enabled with no key set. Credit: by111 (August829).
• HIGH: Arbitrary File Write via output_path (CVSS 9.1, CWE-22): /screenshot and /pdf wrote to any path. Restricted writes to CRAWL4AI_OUTPUT_DIR and reject .. traversal. Credit: Jeongbean Jeon, wulonchia.
• HIGH: SSRF via Webhook URL (CVSS 8.6, CWE-918): webhook URLs on /crawl/job and /llm/job could reach internal and cloud-metadata IPs. Added a blocklist and follow_redirects=False. Credit: Jeongbean Jeon.
• HIGH: SSRF via Direct Crawl Endpoints (CVSS 8.6, CWE-918): /crawl, /md, and /llm fetched arbitrary URLs, and IPv6-mapped IPv4 addresses ([::ffff:169.254.169.254]) bypassed naive checks. Added destination validation on all entry points and normalize IPv6-mapped IPv4 before the blocklist check. Credit: secsys_codex, Velayutham Selvaraj, IcySun.
• HIGH: Arbitrary JavaScript Execution via /execute_js (CVSS 8.1, CWE-94): disabled by default via CRAWL4AI_EXECUTE_JS_ENABLED, removed --disable-web-security from default browser args, and added an SSRF blocklist on the destination. Credit: by111 (August829).
• MEDIUM: Monitor Endpoint Auth Bypass (CVSS 6.5, CWE-306): /monitor/* routes, including destructive actions, were unauthenticated. Added token_dep to the router and an explicit token check on the WebSocket endpoint. Credit: Jeongbean Jeon.
• MEDIUM: Stored XSS in Monitor Dashboard (CVSS 6.1, CWE-79): URLs and errors were rendered via innerHTML without escaping. Added server-side html.escape() and a client-side escapeHtml() wrapper. Credit: Jeongbean Jeon.
• eval() removed from /config/dump: replaced with JSON input validated by Pydantic.
• Config hardening: validate the markdown_generator type in CrawlerRunConfig to reject malformed JSON (#1880).

Added

• DomainMapper: comprehensive domain URL discovery, with an include_subdomains flag and a per-source timeout.
• arun_many config-list support in the Docker API: per-URL configs (#1837).
• Docker server can listen on all addresses.

Fixed

• Markdown and scraping fidelity:
  • Preserve mermaid diagram text from SVGs and prevent nested fences (#1043)
  • Preserve table rowspan/colspan in cleaned HTML (#1920)
  • Preserve .tail text when removing empty elements (#1938)
  • Keep sentence order in NlpSentenceChunking (#1909)
• Deep crawl and dispatcher:
  • Fix the deep-crawl streaming ContextVar bug, using set(False) instead of reset(token) (#1917)
  • Wire semaphore_count into the auto-created MemoryAdaptiveDispatcher and default it to 10 (#1927)
• LLM and providers:
  • Add Bedrock to the provider prefixes so AWS credential auth works
  • Default LLMExtractionStrategy.extraction_type to schema
  • Add LLMTableExtraction to the Docker deserialization allowlist
• Crawler and downloads:
  • Return success=True for binary downloads and skip the block check when downloaded_files is set
  • Honor <base href> in prefetch quick_extract_links (#752)
• Logging and MCP:
  • Route AsyncLogger output to stderr by default (#1968) and use Console(width=200) for non-TTY contexts
  • Use ensure_ascii=False in the MCP bridge to preserve CJK characters (#1967)
• Browser and misc:
  • browser_adapter now uses the Stealth import, fixing a stealth import mismatch (#1960)
  • Correct the arun() return type to CrawlResultContainer (#1898)
  • Log the real failure reason before COMPLETE, fixing a misleading "SCRAPE ok" line (#1949)
  • Assistant toolbar scroll fix and issue-1973 fix

Docs

• Added Privacy Policy, Terms of Service, and Support pages.

Security Credits

Song Binglin (q1uf3ng), by111 (August829), Jeongbean Jeon, wulonchia, secsys_codex, Velayutham Selvaraj, and IcySun. See SECURITY-CREDITS.md.

Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes

March 2026 • 10 min read

---

I'm releasing Crawl4AI v0.8.5—our biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.

What's New at a Glance

• Anti-Bot Detection & Proxy Escalation: 3-tier detection with automatic retry, proxy chain, and fallback
• Shadow DOM Flattening: Extract content hidden inside shadow DOM components
• Deep Crawl Cancellation: Stop long crawls gracefully with cancel() or should_cancel callback
• Config Defaults API: Set once, apply everywhere with set_defaults() / get_defaults() / reset_defaults()
• Source/Sibling Selector: Extract data spanning sibling elements in JSON extraction schemas
• Consent Popup Removal: Auto-dismiss cookie banners from 40+ CMP platforms
• Resource Filtering: Block ads and CSS at the network level with avoid_ads / avoid_css
• Browser Recycling: Memory-saving mode and automatic browser restart for long sessions
• GFM Table Compliance: Proper | col1 | col2 | pipe delimiters in markdown output
• 60+ Bug Fixes: Security patches, browser stability, extraction accuracy, and more

---

New Features

1. Anti-Bot Detection, Retry & Fallback

This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes action—retrying with different proxies or falling back to an alternative fetch method.

The detection uses three tiers:

• Tier 1: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
• Tier 2: Generic block indicators on small pages
• Tier 3: Structural integrity checks (empty shells, script-heavy pages with no content)

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.async_configs import ProxyConfig

config = CrawlerRunConfig(
    # Try direct first, then proxy on bot detection
    proxy_config=[
        ProxyConfig.DIRECT,
        ProxyConfig(server="http://my-proxy:8080"),
    ],
    max_retries=2,
    # Optional: fallback when all proxies fail
    fallback_fetch_function=my_web_unlocker_function,
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://protected-site.com", config=config)

    # Check what happened
    stats = result.crawl_stats
    print(f"Resolved by: {stats['resolved_by']}")  # "direct", "proxy", or "fallback_fetch"
    print(f"Proxies tried: {len(stats['proxies_used'])}")

The system errs on the side of caution—false positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.

2. Shadow DOM Flattening

Web components with shadow DOM hide their content from regular DOM traversal. The new flatten_shadow_dom option serializes shadow DOM content into the light DOM before extraction.

config = CrawlerRunConfig(flatten_shadow_dom=True)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://some-web-component-site.com", config=config)
    # Shadow DOM content is now visible in result.html, cleaned_html, and markdown

The implementation patches attachShadow to force-open closed shadow roots, recursively resolves <slot> projections, and strips only shadow-scoped <style> tags. It also reorders the JS execution pipeline—js_code now runs after wait_for + delay_before_return_html so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new js_code_before_wait parameter.

3. Deep Crawl Cancellation

All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:

from crawl4ai.deep_crawling import DFSDeepCrawlStrategy

pages_found = 0

def should_stop():
    return pages_found >= 50  # Stop after finding enough pages

async def on_state(state):
    nonlocal pages_found
    pages_found = state["pages_crawled"]

strategy = DFSDeepCrawlStrategy(
    max_depth=3,
    max_pages=1000,
    should_cancel=should_stop,      # Sync or async callback
    on_state_change=on_state,
)

config = CrawlerRunConfig(deep_crawl_strategy=strategy)

async with AsyncWebCrawler() as crawler:
    results = await crawler.arun("https://example.com", config=config)
    print(f"Cancelled: {strategy.cancelled}")

You can also call strategy.cancel() directly from another thread or coroutine.

4. Config Defaults API

Tired of repeating the same parameters? Set defaults once and they apply to every new instance:

from crawl4ai import BrowserConfig, CrawlerRunConfig

# Set organization-wide defaults
BrowserConfig.set_defaults(headless=True, text_mode=True)
CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)

# All new instances inherit defaults
bc = BrowserConfig()                   # headless=True, text_mode=True
rc = CrawlerRunConfig()                # verbose=False, remove_consent_popups=True

# Explicit params always override
bc2 = BrowserConfig(text_mode=False)   # text_mode=False, headless still True

# Inspect and reset
print(BrowserConfig.get_defaults())    # {"headless": True, "text_mode": True}
BrowserConfig.reset_defaults()         # Back to normal

5. Source/Sibling Selector in JSON Extraction

Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate <tr> rows). The new "source" field navigates to a sibling before extracting:

from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "HackerNewsItems",
    "baseSelector": "tr.athing",
    "fields": [
        {"name": "title", "selector": ".titleline > a", "type": "text"},
        {"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
        # Navigate to the NEXT sibling <tr> to get the score
        {"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
        {"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
    ]
}

strategy = JsonCssExtractionStrategy(schema=schema)

Works in both JsonCssExtractionStrategy and JsonXPathExtractionStrategy. Falls back gracefully when siblings don't exist.

6. Consent Popup Removal

A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:

config = CrawlerRunConfig(remove_consent_popups=True)

Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.

7. Resource Filtering: avoid_ads / avoid_css

Block ad trackers and CSS resources at the network level for faster, leaner crawls:

config = BrowserConfig(
    avoid_ads=True,   # Blocks doubleclick, google-analytics, etc.
    avoid_css=True,   # Blocks .css, .less, .scss resources
)

8. Browser Recycling & Memory-Saving Mode

For long-running crawl sessions:

config = BrowserConfig(
    memory_saving_mode=True,          # Aggressive cache/V8 heap flags
    max_pages_before_recycle=100,     # Auto-restart browser after N pages
)

This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent load—we fixed three separate deadlock bugs to get this right.

9. GFM Table Compliance

Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:

Before (v0.8.0):

Name | Age | City
---|---|---
Alice | 30 | NYC

After (v0.8.5):

| Name | Age | City |
| --- | --- | --- |
| Alice | 30 | NYC |

---

Minor Features

• query_llm_config: Separate LLM config for adaptive crawler query expansion (#1682)
• force_viewport_screenshot: Screenshot only the viewport, not the full page
• device_scale_factor: Configurable screenshot DPI via BrowserConfig (#1463)
• redirected_status_code: Now available on CrawlResult (#1435)
• wait_for_images: Wait for images to load before taking screenshots (#1792)
• score_threshold: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
• link_preview_timeout: Configurable timeout in AdaptiveConfig (#1793)
• --json-ensure-ascii: CLI flag for Unicode preservation in JSON output (#1668)
• type-list pipeline: Chained extraction like ["attribute", "regex"] in JsonCssExtractionStrategy (#1290)

---

Security Fixes

Critical: RCE via Deserialization in Docker /crawl Endpoint

Severity: CRITICAL Affected: Docker API deployment (v0.8.0 and earlier)

The /crawl endpoint's deserialization logic used eval() for certain object types. I removed this entirely and added an allowlist (ALLOWED_DESERIALIZE_TYPES) so only known config classes can be instantiated.

Critical: Redis CVE-2025-49844 (CVSS 10.0)

Affected: Docker deployments using Redis

Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.

Additional Security

• XSS prevention: Use DOMParser instead of innerHTML in iframe processing (#1796)
• API token enforcement: /token endpoint now requires api_token when configured (#1795)
• Stealth improvements: sec-ch-ua synced with User-Agent, WebGL kept alive in stealth mode

---

Bug Fixes

Browser & Page Management

• Fix page reuse race condition when create_isolated_context=False
• Fix browser context memory leak — signature shrink + LRU eviction (#943)
• Fix cascading context crash from duplicate add_init_script (#1768)
• Fix simulate_user destroying page content via ArrowDown keypress
• Fix browser recycling deadlock under sustained concurrent load (#1640)
• Fix Docker monitor LOCK contention causing pod deadlock (#1754)

Proxy & Network

• Fix proxy auth ERR_INVALID_AUTH_CREDENTIALS (#1281)
• Fix proxy auth for persistent browser contexts
• Fix proxy escalation not re-raising on first exception when chain has alternatives
• Fix fallback fetch: run when all proxies crash, skip re-check, never return None

Deep Crawling

• Fix can_process_url() to receive normalized URL
• Fix total_score not calculated for links that fail head extraction
• Fix FilterChain.add_filter AttributeError on tuple immutability
• Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
• Fix is_external_url port comparison (#1783)
• Prevent AdaptiveCrawler from crawling external domains (#1805)

Extraction & Content

• Fix <base> tag ignored in html2text relative link resolution (#1721)
• Fix script tag removal losing adjacent text in cleaned_html (#1364)
• Preserve class and id attributes in cleaned_html (#1782)
• Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
• Strip markdown fences in force_json_response path for LLM extraction
• Guard against None LLM content, propagate finish_reason (#1788)
• Fix agenerate_schema() JSON parsing for Anthropic models
• Fix from_serializable_dict ignoring plain data dicts with "type" key
• Fix MediaItem crash on non-numeric width values like "100%" (#1635)
• Fix BM25ContentFilter returning duplicate chunks (#1213)
• Fix css_selector ignored in LXML scraping for raw:// URLs (#1484)

CLI & Docker

• Fix deep-crawl CLI outputting only the first page (#1667)
• Fix VersionManager ignoring CRAWL4_AI_BASE_DIRECTORY env var (#1296)
• Fix Docker health endpoint to use dynamic version (#1686)
• Add explicit UTF-8 encoding to CLI file output (#1789)
• Handle UnicodeEncodeError in URL seeder, strip zero-width chars (#1784)
• Add TTL expiry for Redis task data to prevent memory growth (#1730)
• Add Windows support for crawler monitor keyboard input (#1794)
• Fix scroll_delay ignored in full-page screenshot scroller
• Fix MCP SSE endpoint crash — mount via raw ASGI Route (#1594)
• Fix /llm per-request provider override, Redis config from host/port/password (#1611, #1817)
• Fix screenshot respects scan_full_page=False (#1750)
• Fix screenshot distortion on Elementor sites (#1370)
• Fix deep crawl timeout and arun_many dispatcher bypass (#1818, #1509)

Other

• Replace tf-playwright-stealth with playwright-stealth (#1553)
• Allow local embeddings by removing OpenAI fallback (#1658)
• Include GoogleSearchCrawler script.js in package distribution (#1711)
• Fix bs4 deprecation warning (text → string) (#1077)
• Run blocking chardet.detect in thread executor (#1751)
• Wire mean_delay/max_range from CrawlerRunConfig into dispatcher rate limiter (#1786)

---

Tests

Added a comprehensive 291-test regression suite covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.

---

Breaking Changes

cleaned_html Now Preserves class and id Attributes

If you have downstream code that parses cleaned_html and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.

Docker: Redis Upgraded to 7.2.7

If you pin Redis versions in your deployment, update to 7.2.7 or later.

---

Upgrade Instructions

Python Package

pip install --upgrade crawl4ai
# or
pip install crawl4ai==0.8.5

Docker

docker pull unclecode/crawl4ai:0.8.5

docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5

---

Verification

Run the verification tests to confirm all features are working:

python docs/releases_review/demo_v0.8.5.py

This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.

---

Acknowledgments

This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in CONTRIBUTORS.md.

Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824

---

Support & Resources

• Documentation: docs.crawl4ai.com
• GitHub: github.com/unclecode/crawl4ai
• Discord: discord.gg/crawl4ai
• Twitter: @unclecode

---

This is a massive release—10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!

Happy crawling!

- unclecode

Security

• 🔒 CRITICAL: Remote Code Execution Fix: Removed __import__ from hook allowed builtins
  • Prevents arbitrary module imports in user-provided hook code
  • Hooks now disabled by default via CRAWL4AI_HOOKS_ENABLED environment variable
  • Credit: Neo by ProjectDiscovery
• 🔒 HIGH: Local File Inclusion Fix: Added URL scheme validation to Docker API endpoints
  • Blocks file://, javascript:, data: URLs on /execute_js, /screenshot, /pdf, /html
  • Only allows http://, https://, and raw: URLs
  • Credit: Neo by ProjectDiscovery

Breaking Changes

• Docker API: Hooks disabled by default: Set CRAWL4AI_HOOKS_ENABLED=true to enable
• Docker API: file:// URLs blocked: Use Python library directly for local file processing

Added

• 🚀 init_scripts for BrowserConfig: Pre-page-load JavaScript injection for stealth evasions
• 🔄 CDP Connection Improvements: WebSocket URL support, proper cleanup, browser reuse
• 💾 Crash Recovery for Deep Crawl: resume_state and on_state_change for BFS/DFS/Best-First strategies
• 📄 PDF/MHTML for raw:/file:// URLs: Generate PDFs and MHTML from cached HTML content
• 📸 Screenshots for raw:/file:// URLs: Render cached HTML and capture screenshots
• 🔗 base_url Parameter: Proper URL resolution for raw: HTML processing
• ⚡ Prefetch Mode: Two-phase deep crawling with fast link extraction
• 🔀 Enhanced Proxy Support: Improved proxy rotation and sticky sessions
• 🌐 HTTP Strategy Proxy Support: Non-browser crawler now supports proxies
• 🖥️ Browser Pipeline for raw:/file://: New process_in_browser parameter
• 📋 Smart TTL Cache for Sitemap Seeder: cache_ttl_hours and validate_sitemap_lastmod parameters
• 📚 Security Documentation: Added SECURITY.md with vulnerability reporting guidelines

Fixed

• raw: URL Parsing: Fixed truncation at # character (CSS color codes like #eee)
• Caching System: Various improvements to cache validation and persistence

Documentation

• Multi-sample schema generation section
• URL seeder smart TTL cache parameters
• v0.8.0 migration guide
• Security policy and disclosure process

Crawl4AI v0.8.0 Release Notes

Release Date: January 2026 Previous Version: v0.7.6 Status: Release Candidate

---

Highlights

• Critical Security Fixes for Docker API deployment
• 11 New Features including crash recovery, prefetch mode, and proxy improvements
• Breaking Changes - see migration guide below

---

Breaking Changes

1. Docker API: Hooks Disabled by Default

What changed: Hooks are now disabled by default on the Docker API.

Why: Security fix for Remote Code Execution (RCE) vulnerability.

Who is affected: Users of the Docker API who use the hooks parameter in /crawl requests.

Migration:

# To re-enable hooks (only if you trust all API users):
export CRAWL4AI_HOOKS_ENABLED=true

2. Docker API: file:// URLs Blocked

What changed: The endpoints /execute_js, /screenshot, /pdf, and /html now reject file:// URLs.

Why: Security fix for Local File Inclusion (LFI) vulnerability.

Who is affected: Users who were reading local files via the Docker API.

Migration: Use the Python library directly for local file processing:

# Instead of API call with file:// URL, use library:
from crawl4ai import AsyncWebCrawler
async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url="file:///path/to/file.html")

---

Security Fixes

Critical: Remote Code Execution via Hooks (CVE Pending)

Severity: CRITICAL (CVSS 10.0) Affected: Docker API deployment (all versions before v0.8.0) Vector: POST /crawl with malicious hooks parameter

Details: The __import__ builtin was available in hook code, allowing attackers to import os, subprocess, etc. and execute arbitrary commands.

Fix:

1. Removed __import__ from allowed builtins
2. Hooks disabled by default (CRAWL4AI_HOOKS_ENABLED=false)

High: Local File Inclusion via file:// URLs (CVE Pending)

Severity: HIGH (CVSS 8.6) Affected: Docker API deployment (all versions before v0.8.0) Vector: POST /execute_js (and other endpoints) with file:///etc/passwd

Details: API endpoints accepted file:// URLs, allowing attackers to read arbitrary files from the server.

Fix: URL scheme validation now only allows http://, https://, and raw: URLs.

Credits

Discovered by Neo by ProjectDiscovery (projectdiscovery.io) - December 2025

---

New Features

1. init_scripts Support for BrowserConfig

Pre-page-load JavaScript injection for stealth evasions.

config = BrowserConfig(
    init_scripts=[
        "Object.defineProperty(navigator, 'webdriver', {get: () => false})"
    ]
)

2. CDP Connection Improvements

• WebSocket URL support (ws://, wss://)
• Proper cleanup with cdp_cleanup_on_close=True
• Browser reuse across multiple connections

3. Crash Recovery for Deep Crawl Strategies

All deep crawl strategies (BFS, DFS, Best-First) now support crash recovery:

from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

strategy = BFSDeepCrawlStrategy(
    max_depth=3,
    resume_state=saved_state,  # Resume from checkpoint
    on_state_change=save_callback  # Persist state in real-time
)

4. PDF and MHTML for raw:/file:// URLs

Generate PDFs and MHTML from cached HTML content.

5. Screenshots for raw:/file:// URLs

Render cached HTML and capture screenshots.

6. base_url Parameter for CrawlerRunConfig

Proper URL resolution for raw: HTML processing:

config = CrawlerRunConfig(base_url='https://example.com')
result = await crawler.arun(url='raw:{html}', config=config)

7. Prefetch Mode for Two-Phase Deep Crawling

Fast link extraction without full page processing:

config = CrawlerRunConfig(prefetch=True)

8. Proxy Rotation and Configuration

Enhanced proxy rotation with sticky sessions support.

9. Proxy Support for HTTP Strategy

Non-browser crawler now supports proxies.

10. Browser Pipeline for raw:/file:// URLs

New process_in_browser parameter for browser operations on local content:

config = CrawlerRunConfig(
    process_in_browser=True,  # Force browser processing
    screenshot=True
)
result = await crawler.arun(url='raw:<html>...</html>', config=config)

11. Smart TTL Cache for Sitemap URL Seeder

Intelligent cache invalidation for sitemaps:

config = SeedingConfig(
    cache_ttl_hours=24,
    validate_sitemap_lastmod=True
)

---

Bug Fixes

raw: URL Parsing Truncates at # Character

Problem: CSS color codes like #eee were being truncated.

Before: raw:body{background:#eee} → body{background: After: raw:body{background:#eee} → body{background:#eee}

Caching System Improvements

Various fixes to cache validation and persistence.

---

Documentation Updates

• Multi-sample schema generation documentation
• URL seeder smart TTL cache parameters
• Security documentation (SECURITY.md)

---

Upgrade Guide

From v0.7.x to v0.8.0

1. Update the package:

   pip install --upgrade crawl4ai
   

2. Docker API users:

   • Hooks are now disabled by default
   • If you need hooks: export CRAWL4AI_HOOKS_ENABLED=true
   • file:// URLs no longer work on API (use library directly)

3. Review security settings:

   # config.yml - recommended for production
   security:
     enabled: true
     jwt_enabled: true
   

4. Test your integration before deploying to production

Breaking Change Checklist

• [ ] Check if you use hooks parameter in API calls
• [ ] Check if you use file:// URLs via the API
• [ ] Update environment variables if needed
• [ ] Review security configuration

---

Full Changelog

See CHANGELOG.md for complete version history.

---

Contributors

Thanks to all contributors who made this release possible.

Special thanks to Neo by ProjectDiscovery for responsible security disclosure.

---

For questions or issues, please open a GitHub Issue.

Crawl4AI v0.7.8: Stability & Bug Fix Release

December 2025

---

I'm releasing Crawl4AI v0.7.8—a focused stability release that addresses 11 bugs reported by the community. While there are no new features in this release, these fixes resolve important issues affecting Docker deployments, LLM extraction, URL handling, and dependency compatibility.

What's Fixed at a Glance

• Docker API: Fixed ContentRelevanceFilter deserialization, ProxyConfig serialization, and cache folder permissions
• LLM Extraction: Configurable rate limiter backoff, HTML input format support, and proper URL handling for raw HTML
• URL Handling: Correct relative URL resolution after JavaScript redirects
• Dependencies: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility
• AdaptiveCrawler: Fixed query expansion to actually use LLM instead of hardcoded mock data

Bug Fixes

Docker & API Fixes

ContentRelevanceFilter Deserialization (#1642)

The Problem: When sending deep crawl requests to the Docker API with ContentRelevanceFilter, the server failed to deserialize the filter, causing requests to fail.

The Fix: I added ContentRelevanceFilter to the public exports and enhanced the deserialization logic with dynamic imports.

# This now works correctly in Docker API
import httpx

request = {
    "urls": ["https://docs.example.com"],
    "crawler_config": {
        "deep_crawl_strategy": {
            "type": "BFSDeepCrawlStrategy",
            "max_depth": 2,
            "filter_chain": [
                {
                    "type": "ContentRelevanceFilter",
                    "query": "API documentation",
                    "threshold": 0.3
                }
            ]
        }
    }
}

async with httpx.AsyncClient() as client:
    response = await client.post("http://localhost:11235/crawl", json=request)
    # Previously failed, now works!

ProxyConfig JSON Serialization (#1629)

The Problem: BrowserConfig.to_dict() failed when proxy_config was set because ProxyConfig wasn't being serialized to a dictionary.

The Fix: ProxyConfig.to_dict() is now called during serialization.

from crawl4ai import BrowserConfig
from crawl4ai.async_configs import ProxyConfig

proxy = ProxyConfig(
    server="http://proxy.example.com:8080",
    username="user",
    password="pass"
)

config = BrowserConfig(headless=True, proxy_config=proxy)

# Previously raised TypeError, now works
config_dict = config.to_dict()
json.dumps(config_dict)  # Valid JSON

Docker Cache Folder Permissions (#1638)

The Problem: The .cache folder in the Docker image had incorrect permissions, causing crawling to fail when caching was enabled.

The Fix: Corrected ownership and permissions during image build.

# Cache now works correctly in Docker
docker run -d -p 11235:11235 \
    --shm-size=1g \
    -v ./my-cache:/app/.cache \
    unclecode/crawl4ai:0.7.8

---

LLM & Extraction Fixes

Configurable Rate Limiter Backoff (#1269)

The Problem: The LLM rate limiting backoff parameters were hardcoded, making it impossible to adjust retry behavior for different API rate limits.

The Fix: LLMConfig now accepts three new parameters for complete control over retry behavior.

from crawl4ai import LLMConfig

# Default behavior (unchanged)
default_config = LLMConfig(provider="openai/gpt-4o-mini")
# backoff_base_delay=2, backoff_max_attempts=3, backoff_exponential_factor=2

# Custom configuration for APIs with strict rate limits
custom_config = LLMConfig(
    provider="openai/gpt-4o-mini",
    backoff_base_delay=5,           # Wait 5 seconds on first retry
    backoff_max_attempts=5,          # Try up to 5 times
    backoff_exponential_factor=3     # Multiply delay by 3 each attempt
)

# Retry sequence: 5s -> 15s -> 45s -> 135s -> 405s

LLM Strategy HTML Input Support (#1178)

The Problem: LLMExtractionStrategy always sent markdown to the LLM, but some extraction tasks work better with HTML structure preserved.

The Fix: Added input_format parameter supporting "markdown", "html", "fit_markdown", "cleaned_html", and "fit_html".

from crawl4ai import LLMExtractionStrategy, LLMConfig

# Default: markdown input (unchanged)
markdown_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
    instruction="Extract product information"
)

# NEW: HTML input - preserves table/list structure
html_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
    instruction="Extract the data table preserving structure",
    input_format="html"
)

# NEW: Filtered markdown - only relevant content
fit_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
    instruction="Summarize the main content",
    input_format="fit_markdown"
)

Raw HTML URL Variable (#1116)

The Problem: When using url="raw:<html>...", the entire HTML content was being passed to extraction strategies as the URL parameter, polluting LLM prompts.

The Fix: The URL is now correctly set to "Raw HTML" for raw HTML inputs.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

html = "<html><body><h1>Test</h1></body></html>"

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url=f"raw:{html}",
        config=CrawlerRunConfig(extraction_strategy=my_strategy)
    )
    # extraction_strategy receives url="Raw HTML" instead of the HTML blob

---

URL Handling Fix

Relative URLs After Redirects (#1268)

The Problem: When JavaScript caused a page redirect, relative links were resolved against the original URL instead of the final URL.

The Fix: redirected_url now captures the actual page URL after all JavaScript execution completes.

from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    # Page at /old-page redirects via JS to /new-page
    result = await crawler.arun(url="https://example.com/old-page")

    # BEFORE: redirected_url = "https://example.com/old-page"
    # AFTER:  redirected_url = "https://example.com/new-page"

    # Links are now correctly resolved against the final URL
    for link in result.links['internal']:
        print(link['href'])  # Relative links resolved correctly

---

Dependency & Compatibility Fixes

PyPDF2 Replaced with pypdf (#1412)

The Problem: PyPDF2 was deprecated in 2022 and is no longer maintained.

The Fix: Replaced with the actively maintained pypdf library.

# Installation (unchanged)
pip install crawl4ai[pdf]

# The PDF processor now uses pypdf internally
# No code changes required - API remains the same

Pydantic v2 ConfigDict Compatibility (#678)

The Problem: Using the deprecated class Config syntax caused deprecation warnings with Pydantic v2.

The Fix: Migrated to model_config = ConfigDict(...) syntax.

# No more deprecation warnings when importing crawl4ai models
from crawl4ai.models import CrawlResult
from crawl4ai import CrawlerRunConfig, BrowserConfig

# All models are now Pydantic v2 compatible

---

AdaptiveCrawler Fix

Query Expansion Using LLM (#1621)

The Problem: The EmbeddingStrategy in AdaptiveCrawler had commented-out LLM code and was using hardcoded mock query variations instead.

The Fix: Uncommented and activated the LLM call for actual query expansion.

# AdaptiveCrawler query expansion now actually uses the LLM
# Instead of hardcoded variations like:
# variations = {'queries': ['what are the best vegetables...']}

# The LLM generates relevant query variations based on your actual query

---

Code Formatting Fix

Import Statement Formatting (#1181)

The Problem: When extracting code from web pages, import statements were sometimes concatenated without proper line separation.

The Fix: Import statements now maintain proper newline separation.

# BEFORE: "import osimport sysfrom pathlib import Path"
# AFTER:
# import os
# import sys
# from pathlib import Path

---

Breaking Changes

None! This release is fully backward compatible.

• All existing code continues to work without modification
• New parameters have sensible defaults matching previous behavior
• No API changes to existing functionality

---

Upgrade Instructions

Python Package

pip install --upgrade crawl4ai
# or
pip install crawl4ai==0.7.8

Docker

# Pull the latest version
docker pull unclecode/crawl4ai:0.7.8

# Run
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.7.8

---

Verification

Run the verification tests to confirm all fixes are working:

python docs/releases_review/demo_v0.7.8.py

This runs actual tests that verify each bug fix is properly implemented.

---

Acknowledgments

Thank you to everyone who reported these issues and provided detailed reproduction steps. Your bug reports make Crawl4AI better for everyone.

Issues fixed: #1642, #1638, #1629, #1621, #1412, #1269, #1268, #1181, #1178, #1116, #678

---

Support & Resources

• Documentation: docs.crawl4ai.com
• GitHub: github.com/unclecode/crawl4ai
• Discord: discord.gg/crawl4ai
• Twitter: @unclecode

---

This stability release ensures Crawl4AI works reliably across Docker deployments, LLM extraction workflows, and various edge cases. Thank you for your continued support and feedback!

Happy crawling!

- unclecode

🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update

November 14, 2025 • 10 min read

---

Today I'm releasing Crawl4AI v0.7.7—the Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.

🎯 What's New at a Glance

• 📊 Real-time Monitoring Dashboard: Interactive web UI with live system metrics and browser pool status
• 🔌 Comprehensive Monitor API: Complete REST API for programmatic access to all monitoring data
• ⚡ WebSocket Streaming: Real-time updates every 2 seconds for custom dashboards
• 🎮 Control Actions: Manual browser management (kill, restart, cleanup)
• 🔥 Smart Browser Pool: 3-tier architecture (permanent/hot/cold) with automatic promotion
• 🧹 Janitor Cleanup System: Automatic resource management with event logging
• 📈 Production Metrics: 6 critical metrics for operational excellence
• 🏭 Integration Ready: Prometheus, alerting, and log aggregation examples
• 🐛 Critical Bug Fixes: Async LLM extraction, DFS crawling, viewport config, and more

📊 Real-time Monitoring Dashboard: Complete Visibility

The Problem: Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the container—memory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.

My Solution: I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.

The Self-Hosting Value Proposition

Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:

• 🔒 Data Privacy: Your data never leaves your infrastructure
• 💰 Cost Control: No per-request pricing or rate limits
• 🎯 Full Customization: Complete control over configurations and strategies
• 📊 Complete Transparency: Real-time visibility into every aspect
• ⚡ Performance: Direct access without network overhead
• 🛡️ Enterprise Security: Keep workflows behind your firewall

Interactive Monitoring Dashboard

Access the dashboard at http://localhost:11235/dashboard to see:

• System Health Overview: CPU, memory, network, and uptime in real-time
• Live Request Tracking: Active and completed requests with full details
• Browser Pool Management: Interactive table with permanent/hot/cold browsers
• Janitor Events Log: Automatic cleanup activities
• Error Monitoring: Full context error logs

The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.

🔌 Monitor API: Programmatic Access

The Problem: Monitoring dashboards are great for humans, but automation and integration require programmatic access.

My Solution: A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.

System Health Endpoint

import httpx
import asyncio

async def monitor_system_health():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/health")
        health = response.json()

        print(f"Container Metrics:")
        print(f"  CPU: {health['container']['cpu_percent']:.1f}%")
        print(f"  Memory: {health['container']['memory_percent']:.1f}%")
        print(f"  Uptime: {health['container']['uptime_seconds']}s")

        print(f"\nBrowser Pool:")
        print(f"  Permanent: {health['pool']['permanent']['active']} active")
        print(f"  Hot Pool: {health['pool']['hot']['count']} browsers")
        print(f"  Cold Pool: {health['pool']['cold']['count']} browsers")

        print(f"\nStatistics:")
        print(f"  Total Requests: {health['stats']['total_requests']}")
        print(f"  Success Rate: {health['stats']['success_rate_percent']:.1f}%")
        print(f"  Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")

asyncio.run(monitor_system_health())

Request Tracking

async def track_requests():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/requests")
        requests_data = response.json()

        print(f"Active Requests: {len(requests_data['active'])}")
        print(f"Completed Requests: {len(requests_data['completed'])}")

        # See details of recent requests
        for req in requests_data['completed'][:5]:
            status_icon = "✅" if req['success'] else "❌"
            print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")

Browser Pool Management

async def monitor_browser_pool():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/browsers")
        browsers = response.json()

        print(f"Pool Summary:")
        print(f"  Total Browsers: {browsers['summary']['total_count']}")
        print(f"  Total Memory: {browsers['summary']['total_memory_mb']} MB")
        print(f"  Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")

        # List all browsers
        for browser in browsers['permanent']:
            print(f"🔥 Permanent: {browser['browser_id'][:8]}... | "
                  f"Requests: {browser['request_count']} | "
                  f"Memory: {browser['memory_mb']:.0f} MB")

Endpoint Performance Statistics

async def get_endpoint_stats():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/endpoints/stats")
        stats = response.json()

        print("Endpoint Analytics:")
        for endpoint, data in stats.items():
            print(f"  {endpoint}:")
            print(f"    Requests: {data['count']}")
            print(f"    Avg Latency: {data['avg_latency_ms']:.0f}ms")
            print(f"    Success Rate: {data['success_rate_percent']:.1f}%")

Complete API Reference

The Monitor API includes these endpoints:

• GET /monitor/health - System health with pool statistics
• GET /monitor/requests - Active and completed request tracking
• GET /monitor/browsers - Browser pool details and efficiency
• GET /monitor/endpoints/stats - Per-endpoint performance analytics
• GET /monitor/timeline?minutes=5 - Time-series data for charts
• GET /monitor/logs/janitor?limit=10 - Cleanup activity logs
• GET /monitor/logs/errors?limit=10 - Error logs with context
• POST /monitor/actions/cleanup - Force immediate cleanup
• POST /monitor/actions/kill_browser - Kill specific browser
• POST /monitor/actions/restart_browser - Restart browser
• POST /monitor/stats/reset - Reset accumulated statistics

⚡ WebSocket Streaming: Real-time Updates

The Problem: Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.

My Solution: WebSocket streaming with 2-second update intervals for building custom real-time dashboards.

WebSocket Integration Example

import websockets
import json
import asyncio

async def monitor_realtime():
    uri = "ws://localhost:11235/monitor/ws"

    async with websockets.connect(uri) as websocket:
        print("Connected to real-time monitoring stream")

        while True:
            # Receive update every 2 seconds
            data = await websocket.recv()
            update = json.loads(data)

            # Access all monitoring data
            print(f"\n--- Update at {update['timestamp']} ---")
            print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
            print(f"Active Requests: {len(update['requests']['active'])}")
            print(f"Total Browsers: {update['browsers']['summary']['total_count']}")

            if update['errors']:
                print(f"⚠️  Recent Errors: {len(update['errors'])}")

asyncio.run(monitor_realtime())

Expected Real-World Impact:

• Custom Dashboards: Build tailored monitoring UIs for your team
• Real-time Alerting: Trigger alerts instantly when metrics exceed thresholds
• Integration: Feed live data into monitoring tools like Grafana
• Automation: React to events in real-time without polling

🔥 Smart Browser Pool: 3-Tier Architecture

The Problem: Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.

My Solution: A smart 3-tier browser pool that automatically adapts to usage patterns.

How It Works

import httpx

async def demonstrate_browser_pool():
    async with httpx.AsyncClient() as client:
        # Request 1-3: Default config → Uses permanent browser
        print("Phase 1: Using permanent browser")
        for i in range(3):
            await client.post(
                "http://localhost:11235/crawl",
                json={"urls": [f"https://httpbin.org/html?req={i}"]}
            )
            print(f"  Request {i+1}: Reused permanent browser")

        # Request 4-6: Custom viewport → Cold pool (first use)
        print("\nPhase 2: Custom config creates cold pool browser")
        viewport_config = {"viewport": {"width": 1280, "height": 720}}
        for i in range(4):
            await client.post(
                "http://localhost:11235/crawl",
                json={
                    "urls": [f"https://httpbin.org/json?v={i}"],
                    "browser_config": viewport_config
                }
            )
            if i < 2:
                print(f"  Request {i+1}: Cold pool browser")
            else:
                print(f"  Request {i+1}: Promoted to hot pool! (after 3 uses)")

        # Check pool status
        response = await client.get("http://localhost:11235/monitor/browsers")
        browsers = response.json()

        print(f"\nPool Status:")
        print(f"  Permanent: {len(browsers['permanent'])} (always active)")
        print(f"  Hot: {len(browsers['hot'])} (frequently used configs)")
        print(f"  Cold: {len(browsers['cold'])} (on-demand)")
        print(f"  Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")

asyncio.run(demonstrate_browser_pool())

Pool Tiers:

• 🔥 Permanent Browser: Always-on, default configuration, instant response
• ♨️ Hot Pool: Browsers promoted after 3+ uses, kept warm for quick access
• ❄️ Cold Pool: On-demand browsers for variant configs, cleaned up when idle

Expected Real-World Impact:

• Memory Efficiency: 10x reduction in memory usage vs creating browsers per request
• Performance: Instant access to frequently-used configurations
• Automatic Optimization: Pool adapts to your usage patterns
• Resource Management: Janitor automatically cleans up idle browsers

🧹 Janitor System: Automatic Cleanup

The Problem: Long-running crawlers accumulate idle browsers and consume memory over time.

My Solution: An automatic janitor system that monitors and cleans up idle resources.

async def monitor_janitor_activity():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
        logs = response.json()

        print("Recent Cleanup Activities:")
        for log in logs:
            print(f"  {log['timestamp']}: {log['message']}")

# Example output:
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)

🎮 Control Actions: Manual Management

The Problem: Sometimes you need to manually intervene—kill a stuck browser, force cleanup, or restart resources.

My Solution: Manual control actions via the API for operational troubleshooting.

Force Cleanup

async def force_cleanup():
    async with httpx.AsyncClient() as client:
        response = await client.post("http://localhost:11235/monitor/actions/cleanup")
        result = response.json()

        print(f"Cleanup completed:")
        print(f"  Browsers cleaned: {result.get('cleaned_count', 0)}")
        print(f"  Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")

Kill Specific Browser

async def kill_stuck_browser(browser_id: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:11235/monitor/actions/kill_browser",
            json={"browser_id": browser_id}
        )

        if response.status_code == 200:
            print(f"✅ Browser {browser_id} killed successfully")

Reset Statistics

async def reset_stats():
    async with httpx.AsyncClient() as client:
        response = await client.post("http://localhost:11235/monitor/stats/reset")
        print("📊 Statistics reset for fresh monitoring")

📈 Production Integration Patterns

Prometheus Integration

# Export metrics for Prometheus scraping
async def export_prometheus_metrics():
    async with httpx.AsyncClient() as client:
        health = await client.get("http://localhost:11235/monitor/health")
        data = health.json()

        # Export in Prometheus format
        metrics = f"""
# HELP crawl4ai_memory_usage_percent Memory usage percentage
# TYPE crawl4ai_memory_usage_percent gauge
crawl4ai_memory_usage_percent {data['container']['memory_percent']}

# HELP crawl4ai_request_success_rate Request success rate
# TYPE crawl4ai_request_success_rate gauge
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}

# HELP crawl4ai_browser_pool_count Total browsers in pool
# TYPE crawl4ai_browser_pool_count gauge
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
"""
        return metrics

Alerting Example

async def check_alerts():
    async with httpx.AsyncClient() as client:
        health = await client.get("http://localhost:11235/monitor/health")
        data = health.json()

        # Memory alert
        if data['container']['memory_percent'] > 80:
            print("🚨 ALERT: Memory usage above 80%")
            # Trigger cleanup
            await client.post("http://localhost:11235/monitor/actions/cleanup")

        # Success rate alert
        if data['stats']['success_rate_percent'] < 90:
            print("🚨 ALERT: Success rate below 90%")
            # Check error logs
            errors = await client.get("http://localhost:11235/monitor/logs/errors")
            print(f"Recent errors: {len(errors.json())}")

        # Latency alert
        if data['stats']['avg_latency_ms'] > 5000:
            print("🚨 ALERT: Average latency above 5s")

Key Metrics to Track

CRITICAL_METRICS = {
    "memory_usage": {
        "current": "container.memory_percent",
        "target": "<80%",
        "alert_threshold": ">80%",
        "action": "Force cleanup or scale"
    },
    "success_rate": {
        "current": "stats.success_rate_percent",
        "target": ">95%",
        "alert_threshold": "<90%",
        "action": "Check error logs"
    },
    "avg_latency": {
        "current": "stats.avg_latency_ms",
        "target": "<2000ms",
        "alert_threshold": ">5000ms",
        "action": "Investigate slow requests"
    },
    "browser_reuse_rate": {
        "current": "browsers.summary.reuse_rate_percent",
        "target": ">80%",
        "alert_threshold": "<60%",
        "action": "Check pool configuration"
    },
    "total_browsers": {
        "current": "browsers.summary.total_count",
        "target": "<15",
        "alert_threshold": ">20",
        "action": "Check for browser leaks"
    },
    "error_frequency": {
        "current": "len(errors)",
        "target": "<5/hour",
        "alert_threshold": ">10/hour",
        "action": "Review error patterns"
    }
}

🐛 Critical Bug Fixes

This release includes significant bug fixes that improve stability and performance:

Async LLM Extraction (#1590)

The Problem: LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).

The Fix: Resolved the blocking issue to enable true parallel processing for LLM extraction.

# Before v0.7.7: Sequential processing
# After v0.7.7: True parallel processing

async with AsyncWebCrawler() as crawler:
    urls = ["url1", "url2", "url3", "url4"]

    # Now processes truly in parallel with LLM extraction
    results = await crawler.arun_many(
        urls,
        config=CrawlerRunConfig(
            extraction_strategy=LLMExtractionStrategy(...)
        )
    )
    # 4x faster for parallel LLM extraction!

Expected Impact: Major performance improvement for batch LLM extraction workflows.

DFS Deep Crawling (#1607)

The Problem: DFS (Depth-First Search) deep crawl strategy had implementation issues.

The Fix: Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.

Browser & Crawler Config Documentation (#1609)

The Problem: Documentation didn't match the actual async_configs.py implementation.

The Fix: Updated all configuration documentation to accurately reflect the current implementation.

Sitemap Seeder (#1598)

The Problem: Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).

The Fix: Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.

Remove Overlay Elements (#1529)

The Problem: The remove_overlay_elements functionality wasn't working (issue #1396).

The Fix: Fixed by properly calling the injected JavaScript function.

Viewport Configuration (#1495)

The Problem: Viewport configuration wasn't working in managed browsers (issue #1490).

The Fix: Added proper viewport size configuration support for browser launch.

Managed Browser CDP Timing (#1528)

The Problem: CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).

The Fix: Added exponential backoff for CDP endpoint verification to handle timing variations.

Security Updates

• pyOpenSSL: Updated from >=24.3.0 to >=25.3.0 to address security vulnerability
• Added verification tests for the security update

Docker Fixes

• Port Standardization: Fixed inconsistent port usage (11234 vs 11235) - now standardized to 11235
• LLM Environment: Fixed LLM API key handling for multi-provider support (PR #1537)
• Error Handling: Improved Docker API error messages with comprehensive status codes
• Serialization: Fixed fit_html property serialization in /crawl and /crawl/stream endpoints

Other Important Fixes

• arun_many Returns: Fixed function to always return a list, even on exception (PR #1530)
• Webhook Serialization: Properly serialize Pydantic HttpUrl in webhook config
• LLMConfig Documentation: Fixed casing and variable name consistency (issue #1551)
• Python Version: Dropped Python 3.9 support, now requires Python >=3.10

📊 Expected Real-World Impact

For DevOps & Infrastructure Teams

• Full Visibility: Know exactly what's happening inside your crawling infrastructure
• Proactive Monitoring: Catch issues before they become problems
• Resource Optimization: Identify memory leaks and performance bottlenecks
• Operational Control: Manual intervention when automated systems need help

For Production Deployments

• Enterprise Observability: Prometheus, Grafana, and alerting integration
• Debugging: Real-time logs and error tracking
• Capacity Planning: Historical metrics for scaling decisions
• SLA Monitoring: Track success rates and latency against targets

For Development Teams

• Local Monitoring: Understand crawler behavior during development
• Performance Testing: Measure impact of configuration changes
• Troubleshooting: Quickly identify and fix issues
• Learning: See exactly how the browser pool works

🔄 Breaking Changes

None! This release is fully backward compatible.

• All existing Docker configurations continue to work
• No API changes to existing endpoints
• Monitoring is additive functionality
• No migration required

🚀 Upgrade Instructions

Docker

# Pull the latest version
docker pull unclecode/crawl4ai:0.7.7

# Or use the latest tag
docker pull unclecode/crawl4ai:latest

# Run with monitoring enabled (default)
docker run -d \
  -p 11235:11235 \
  --shm-size=1g \
  --name crawl4ai \
  unclecode/crawl4ai:0.7.7

# Access the monitoring dashboard
open http://localhost:11235/dashboard

Python Package

# Upgrade to latest version
pip install --upgrade crawl4ai

# Or install specific version
pip install crawl4ai==0.7.7

🎬 Try the Demo

Run the comprehensive demo that showcases all monitoring features:

python docs/releases_review/demo_v0.7.7.py

The demo includes:

1. System health overview with live metrics
2. Request tracking with active/completed monitoring
3. Browser pool management (permanent/hot/cold)
4. Complete Monitor API endpoint examples
5. WebSocket streaming demonstration
6. Control actions (cleanup, kill, restart)
7. Production metrics and alerting patterns
8. Self-hosting value proposition

📚 Documentation

New Documentation

• Self-Hosting Guide - Complete self-hosting documentation with monitoring
• Demo Script: docs/releases_review/demo_v0.7.7.py - Working examples

Updated Documentation

• Docker Deployment → Self-Hosting (renamed for better positioning)
• Added comprehensive monitoring sections
• Production integration patterns
• WebSocket streaming examples

💡 Pro Tips

1. Start with the dashboard - Visit /dashboard to get familiar with the monitoring system
2. Track the 6 key metrics - Memory, success rate, latency, reuse rate, browser count, errors
3. Set up alerting early - Use the Monitor API to build alerts before issues occur
4. Monitor browser pool efficiency - Aim for >80% reuse rate for optimal performance
5. Use WebSocket for custom dashboards - Build tailored monitoring UIs for your team
6. Leverage Prometheus integration - Export metrics for long-term storage and analysis
7. Check janitor logs - Understand automatic cleanup patterns
8. Use control actions judiciously - Manual interventions are for exceptional cases

🙏 Acknowledgments

Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.

The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.

📞 Support & Resources

• 📖 Documentation: docs.crawl4ai.com
• 🐙 GitHub: github.com/unclecode/crawl4ai
• 💬 Discord: discord.gg/crawl4ai
• 🐦 Twitter: @unclecode
• 📊 Dashboard: http://localhost:11235/dashboard (when running)

---

Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platform—it's a game changer for operational excellence!

Happy crawling with full visibility! 🕷️📊

- unclecode

Crawl4AI v0.7.6 Release Notes

Release Date: October 22, 2025

I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.

🎯 What's New

Webhook Support for Docker Job Queue API

The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!

Key Capabilities:

• ✅ Universal Webhook Support: Both /crawl/job and /llm/job endpoints now support webhooks
• ✅ Flexible Delivery Modes: Choose notification-only or include full data in the webhook payload
• ✅ Reliable Delivery: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
• ✅ Custom Authentication: Add custom headers for webhook authentication
• ✅ Global Configuration: Set default webhook URL in config.yml for all jobs
• ✅ Task Type Identification: Distinguish between crawl and llm_extraction tasks

How It Works

Instead of constantly checking job status:

OLD WAY (Polling):

# Submit job
response = requests.post("http://localhost:11235/crawl/job", json=payload)
task_id = response.json()['task_id']

# Poll until complete
while True:
    status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
    if status.json()['status'] == 'completed':
        break
    time.sleep(5)  # Wait and try again

NEW WAY (Webhooks):

# Submit job with webhook
payload = {
    "urls": ["https://example.com"],
    "webhook_config": {
        "webhook_url": "https://myapp.com/webhook",
        "webhook_data_in_payload": True
    }
}
response = requests.post("http://localhost:11235/crawl/job", json=payload)

# Done! Webhook will notify you when complete
# Your webhook handler receives the results automatically

Crawl Job Webhooks

curl -X POST http://localhost:11235/crawl/job \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "browser_config": {"headless": true},
    "crawler_config": {"cache_mode": "bypass"},
    "webhook_config": {
      "webhook_url": "https://myapp.com/webhooks/crawl-complete",
      "webhook_data_in_payload": false,
      "webhook_headers": {
        "X-Webhook-Secret": "your-secret-token"
      }
    }
  }'

LLM Extraction Job Webhooks (NEW!)

curl -X POST http://localhost:11235/llm/job \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "q": "Extract the article title, author, and publication date",
    "schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
    "provider": "openai/gpt-4o-mini",
    "webhook_config": {
      "webhook_url": "https://myapp.com/webhooks/llm-complete",
      "webhook_data_in_payload": true
    }
  }'

Webhook Payload Structure

Success (with data):

{
  "task_id": "llm_1698765432",
  "task_type": "llm_extraction",
  "status": "completed",
  "timestamp": "2025-10-22T10:30:00.000000+00:00",
  "urls": ["https://example.com/article"],
  "data": {
    "extracted_content": {
      "title": "Understanding Web Scraping",
      "author": "John Doe",
      "date": "2025-10-22"
    }
  }
}

Failure:

{
  "task_id": "crawl_abc123",
  "task_type": "crawl",
  "status": "failed",
  "timestamp": "2025-10-22T10:30:00.000000+00:00",
  "urls": ["https://example.com"],
  "error": "Connection timeout after 30s"
}

Simple Webhook Handler Example

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    payload = request.json

    task_id = payload['task_id']
    task_type = payload['task_type']
    status = payload['status']

    if status == 'completed':
        if 'data' in payload:
            # Process data directly
            data = payload['data']
        else:
            # Fetch from API
            endpoint = 'crawl' if task_type == 'crawl' else 'llm'
            response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
            data = response.json()

        # Your business logic here
        print(f"Job {task_id} completed!")

    elif status == 'failed':
        error = payload.get('error', 'Unknown error')
        print(f"Job {task_id} failed: {error}")

    return jsonify({"status": "received"}), 200

app.run(port=8080)

📊 Performance Improvements

• Reduced Server Load: Eliminates constant polling requests
• Lower Latency: Instant notification vs. polling interval delay
• Better Resource Usage: Frees up client connections while jobs run in background
• Scalable Architecture: Handles high-volume crawling workflows efficiently

🐛 Bug Fixes

• Fixed webhook configuration serialization for Pydantic HttpUrl fields
• Improved error handling in webhook delivery service
• Enhanced Redis task storage for webhook config persistence

🌍 Expected Real-World Impact

For Web Scraping Workflows

• Reduced Costs: Less API calls = lower bandwidth and server costs
• Better UX: Instant notifications improve user experience
• Scalability: Handle 100s of concurrent jobs without polling overhead

For LLM Extraction Pipelines

• Async Processing: Submit LLM extraction jobs and move on
• Batch Processing: Queue multiple extractions, get notified as they complete
• Integration: Easy integration with workflow automation tools (Zapier, n8n, etc.)

For Microservices

• Event-Driven: Perfect for event-driven microservice architectures
• Decoupling: Decouple job submission from result processing
• Reliability: Automatic retries ensure webhooks are delivered

🔄 Breaking Changes

None! This release is fully backward compatible.

• Webhook configuration is optional
• Existing code continues to work without modification
• Polling is still supported for jobs without webhook config

📚 Documentation

New Documentation

• WEBHOOK_EXAMPLES.md - Comprehensive webhook usage guide
• docker_webhook_example.py - Working code examples

Updated Documentation

• Docker README - Added webhook sections
• API documentation with webhook examples

🛠️ Migration Guide

No migration needed! Webhooks are opt-in:

1. To use webhooks: Add webhook_config to your job payload
2. To keep polling: Continue using your existing code

Quick Start

# Just add webhook_config to your existing payload
payload = {
    # Your existing configuration
    "urls": ["https://example.com"],
    "browser_config": {...},
    "crawler_config": {...},

    # NEW: Add webhook configuration
    "webhook_config": {
        "webhook_url": "https://myapp.com/webhook",
        "webhook_data_in_payload": True
    }
}

🔧 Configuration

Global Webhook Configuration (config.yml)

webhooks:
  enabled: true
  default_url: "https://myapp.com/webhooks/default"  # Optional
  data_in_payload: false
  retry:
    max_attempts: 5
    initial_delay_ms: 1000
    max_delay_ms: 32000
    timeout_ms: 30000
  headers:
    User-Agent: "Crawl4AI-Webhook/1.0"

🚀 Upgrade Instructions

Docker

# Pull the latest image
docker pull unclecode/crawl4ai:0.7.6

# Or use latest tag
docker pull unclecode/crawl4ai:latest

# Run with webhook support
docker run -d \
  -p 11235:11235 \
  --env-file .llm.env \
  --name crawl4ai \
  unclecode/crawl4ai:0.7.6

Python Package

pip install --upgrade crawl4ai

💡 Pro Tips

1. Use notification-only mode for large results - fetch data separately to avoid large webhook payloads
2. Set custom headers for webhook authentication and request tracking
3. Configure global default webhook for consistent handling across all jobs
4. Implement idempotent webhook handlers - same webhook may be delivered multiple times on retry
5. Use structured schemas with LLM extraction for predictable webhook data

🎬 Demo

Try the release demo:

python docs/releases_review/demo_v0.7.6.py

This comprehensive demo showcases:

• Crawl job webhooks (notification-only and with data)
• LLM extraction webhooks (with JSON schema support)
• Custom headers for authentication
• Webhook retry mechanism
• Real-time webhook receiver

🙏 Acknowledgments

Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.

📞 Support

• Documentation: https://docs.crawl4ai.com
• GitHub Issues: https://github.com/unclecode/crawl4ai/issues
• Discord: https://discord.gg/crawl4ai

---

Happy crawling with webhooks! 🕷️🪝

- unclecode

🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update

September 29, 2025 • 8 min read

---

Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.

🎯 What's New at a Glance

• Docker Hooks System: Custom Python functions at key pipeline points with function-based API
• Function-Based Hooks: New hooks_to_string() utility with Docker client auto-conversion
• Enhanced LLM Integration: Custom providers with temperature control
• HTTPS Preservation: Secure internal link handling
• Bug Fixes: Resolved multiple community-reported issues
• Improved Docker Error Handling: Better debugging and reliability

🔧 Docker Hooks System: Pipeline Customization

Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.

Real Example: Authentication & Performance

import requests

# Real working hooks for httpbin.org
hooks_config = {
    "on_page_context_created": """
async def hook(page, context, **kwargs):
    print("Hook: Setting up page context")
    # Block images to speed up crawling
    await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
    print("Hook: Images blocked")
    return page
""",

    "before_retrieve_html": """
async def hook(page, context, **kwargs):
    print("Hook: Before retrieving HTML")
    # Scroll to bottom to load lazy content
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    await page.wait_for_timeout(1000)
    print("Hook: Scrolled to bottom")
    return page
""",

    "before_goto": """
async def hook(page, context, url, **kwargs):
    print(f"Hook: About to navigate to {url}")
    # Add custom headers
    await page.set_extra_http_headers({
        'X-Test-Header': 'crawl4ai-hooks-test'
    })
    return page
"""
}

# Test with Docker API
payload = {
    "urls": ["https://httpbin.org/html"],
    "hooks": {
        "code": hooks_config,
        "timeout": 30
    }
}

response = requests.post("http://localhost:11235/crawl", json=payload)
result = response.json()

if result.get('success'):
    print("✅ Hooks executed successfully!")
    print(f"Content length: {len(result.get('markdown', ''))} characters")

Available Hook Points:

• on_browser_created: Browser setup
• on_page_context_created: Page context configuration
• before_goto: Pre-navigation setup
• after_goto: Post-navigation processing
• on_user_agent_updated: User agent changes
• on_execution_started: Crawl initialization
• before_retrieve_html: Pre-extraction processing
• before_return_html: Final HTML processing

Function-Based Hooks API

Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!

Option 1: Using the hooks_to_string() Utility

from crawl4ai import hooks_to_string
import requests

# Define hooks as regular Python functions (with full IDE support!)
async def on_page_context_created(page, context, **kwargs):
    """Block images to speed up crawling"""
    await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
    await page.set_viewport_size({"width": 1920, "height": 1080})
    return page

async def before_goto(page, context, url, **kwargs):
    """Add custom headers"""
    await page.set_extra_http_headers({
        'X-Crawl4AI': 'v0.7.5',
        'X-Custom-Header': 'my-value'
    })
    return page

# Convert functions to strings
hooks_code = hooks_to_string({
    "on_page_context_created": on_page_context_created,
    "before_goto": before_goto
})

# Use with REST API
payload = {
    "urls": ["https://httpbin.org/html"],
    "hooks": {"code": hooks_code, "timeout": 30}
}
response = requests.post("http://localhost:11235/crawl", json=payload)

Option 2: Docker Client with Automatic Conversion (Recommended!)

from crawl4ai.docker_client import Crawl4aiDockerClient

# Define hooks as functions (same as above)
async def on_page_context_created(page, context, **kwargs):
    await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
    return page

async def before_retrieve_html(page, context, **kwargs):
    # Scroll to load lazy content
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    await page.wait_for_timeout(1000)
    return page

# Use Docker client - conversion happens automatically!
client = Crawl4aiDockerClient(base_url="http://localhost:11235")

results = await client.crawl(
    urls=["https://httpbin.org/html"],
    hooks={
        "on_page_context_created": on_page_context_created,
        "before_retrieve_html": before_retrieve_html
    },
    hooks_timeout=30
)

if results and results.success:
    print(f"✅ Hooks executed! HTML length: {len(results.html)}")

Benefits of Function-Based Hooks:

• ✅ Full IDE support (autocomplete, syntax highlighting)
• ✅ Type checking and linting
• ✅ Easier to test and debug
• ✅ Reusable across projects
• ✅ Automatic conversion in Docker client
• ✅ No breaking changes - string hooks still work!

🤖 Enhanced LLM Integration

Enhanced LLM integration with custom providers, temperature control, and base URL configuration.

Multi-Provider Support

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy

# Test with different providers
async def test_llm_providers():
    # OpenAI with custom temperature
    openai_strategy = LLMExtractionStrategy(
        provider="gemini/gemini-2.5-flash-lite",
        api_token="your-api-token",
        temperature=0.7,  # New in v0.7.5
        instruction="Summarize this page in one sentence"
    )

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://example.com",
            config=CrawlerRunConfig(extraction_strategy=openai_strategy)
        )

        if result.success:
            print("✅ LLM extraction completed")
            print(result.extracted_content)

# Docker API with enhanced LLM config
llm_payload = {
    "url": "https://example.com",
    "f": "llm",
    "q": "Summarize this page in one sentence.",
    "provider": "gemini/gemini-2.5-flash-lite",
    "temperature": 0.7
}

response = requests.post("http://localhost:11235/md", json=llm_payload)

New Features:

• Custom temperature parameter for creativity control
• base_url for custom API endpoints
• Multi-provider environment variable support
• Docker API integration

🔒 HTTPS Preservation

The Problem: Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.

Solution: HTTPS preservation maintains secure protocols throughout crawling.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy

async def test_https_preservation():
    # Enable HTTPS preservation
    url_filter = URLPatternFilter(
        patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
    )

    config = CrawlerRunConfig(
        exclude_external_links=True,
        preserve_https_for_internal_links=True,  # New in v0.7.5
        deep_crawl_strategy=BFSDeepCrawlStrategy(
            max_depth=2,
            max_pages=5,
            filter_chain=FilterChain([url_filter])
        )
    )

    async with AsyncWebCrawler() as crawler:
        async for result in await crawler.arun(
            url="https://quotes.toscrape.com",
            config=config
        ):
            # All internal links maintain HTTPS
            internal_links = [link['href'] for link in result.links['internal']]
            https_links = [link for link in internal_links if link.startswith('https://')]

            print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
            for link in https_links[:3]:
                print(f"  → {link}")

🛠️ Bug Fixes and Improvements

Major Fixes

• URL Processing: Fixed '+' sign preservation in query parameters (#1332)
• Proxy Configuration: Enhanced proxy string parsing (old proxy parameter deprecated)
• Docker Error Handling: Comprehensive error messages with status codes
• Memory Management: Fixed leaks in long-running sessions
• JWT Authentication: Fixed Docker JWT validation issues (#1442)
• Playwright Stealth: Fixed stealth features for Playwright integration (#1481)
• API Configuration: Fixed config handling to prevent overriding user-provided settings (#1505)
• Docker Filter Serialization: Resolved JSON encoding errors in deep crawl strategy (#1419)
• LLM Provider Support: Fixed custom LLM provider integration for adaptive crawler (#1291)
• Performance Issues: Resolved backoff strategy failures and timeout handling (#989)

Community-Reported Issues Fixed

This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:

• Fixed browser configuration reference errors
• Resolved dependency conflicts with cssselect
• Improved error messaging for failed authentications
• Enhanced compatibility with various proxy configurations
• Fixed edge cases in URL normalization

Configuration Updates

# Old proxy config (deprecated)
# browser_config = BrowserConfig(proxy="http://proxy:8080")

# New enhanced proxy config
browser_config = BrowserConfig(
    proxy_config={
        "server": "http://proxy:8080",
        "username": "optional-user",
        "password": "optional-pass"
    }
)

🔄 Breaking Changes

1. Python 3.10+ Required: Upgrade from Python 3.9
2. Proxy Parameter Deprecated: Use new proxy_config structure
3. New Dependency: Added cssselect for better CSS handling

🚀 Get Started

# Install latest version
pip install crawl4ai==0.7.5

# Docker deployment
docker pull unclecode/crawl4ai:latest
docker run -p 11235:11235 unclecode/crawl4ai:latest

Try the Demo:

# Run working examples
python docs/releases_review/demo_v0.7.5.py

Resources:

• 📖 Documentation: docs.crawl4ai.com
• 🐙 GitHub: github.com/unclecode/crawl4ai
• 💬 Discord: discord.gg/crawl4ai
• 🐦 Twitter: @unclecode

Happy crawling! 🕷️

Added

• 🕵️ Undetected Browser Support: New browser adapter pattern with stealth capabilities

  • browser_adapter.py with undetected Chrome integration
  • Bypass sophisticated bot detection systems (Cloudflare, Akamai, custom solutions)
  • Support for headless stealth mode with anti-detection techniques
  • Human-like behavior simulation with random mouse movements and scrolling
  • Comprehensive examples for anti-bot strategies and stealth crawling
  • Full documentation guide for undetected browser usage

• 🎨 Multi-URL Configuration System: URL-specific crawler configurations for batch processing

  • Different crawling strategies for different URL patterns in a single batch
  • Support for string patterns with wildcards ("*.pdf", "*/blog/*")
  • Lambda function matchers for complex URL logic
  • Mixed matchers combining strings and functions with AND/OR logic
  • Fallback configuration support when no patterns match
  • First-match-wins configuration selection with optional fallback

• 🧠 Memory Monitoring & Optimization: Comprehensive memory usage tracking

  • New memory_utils.py module for memory monitoring and optimization
  • Real-time memory usage tracking during crawl sessions
  • Memory leak detection and reporting
  • Performance optimization recommendations
  • Peak memory usage analysis and efficiency metrics
  • Automatic cleanup suggestions for memory-intensive operations

• 📊 Enhanced Table Extraction: Improved table access and DataFrame conversion

  • Direct result.tables interface replacing generic result.media approach
  • Instant pandas DataFrame conversion with pd.DataFrame(table['data'])
  • Enhanced table detection algorithms for better accuracy
  • Table metadata including source XPath and headers
  • Improved table structure preservation during extraction

• 💰 GitHub Sponsors Integration: 4-tier sponsorship system

  • Supporter ($5/month): Community support + early feature previews
  • Professional ($25/month): Priority support + beta access
  • Business ($100/month): Direct consultation + custom integrations
  • Enterprise ($500/month): Dedicated support + feature development
  • Custom arrangement options for larger organizations

• 🐳 Docker LLM Provider Flexibility: Environment-based LLM configuration

  • LLM_PROVIDER environment variable support for dynamic provider switching
  • .llm.env file support for secure configuration management
  • Per-request provider override capabilities in API endpoints
  • Support for OpenAI, Groq, and other providers without rebuilding images
  • Enhanced Docker documentation with deployment examples

Fixed

• URL Matcher Fallback: Resolved edge cases in URL pattern matching logic
• Memory Management: Fixed memory leaks in long-running crawl sessions
• Sitemap Processing: Improved redirect handling in sitemap fetching
• Table Extraction: Enhanced table detection and extraction accuracy
• Error Handling: Better error messages and recovery from network failures

Changed

• Architecture Refactoring: Major cleanup and optimization
  • Moved 2,450+ lines from main async_crawler_strategy.py to backup
  • Cleaner separation of concerns in crawler architecture
  • Better maintainability and code organization
  • Preserved backward compatibility while improving performance

Documentation

• Comprehensive Examples: Added real-world URLs and practical use cases
• API Documentation: Complete CrawlResult field documentation with all available fields
• Migration Guides: Updated table extraction patterns from result.media to result.tables
• Undetected Browser Guide: Full documentation for stealth mode and anti-bot strategies
• Multi-Config Examples: Detailed examples for URL-specific configurations
• Docker Deployment: Enhanced Docker documentation with LLM provider configuration

🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update

August 6, 2025 • 5 min read

---

Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.

🎯 What's New at a Glance

• Multi-URL Configurations: Different crawling strategies for different URL patterns in a single batch
• Flexible Docker LLM Providers: Configure LLM providers via environment variables
• Bug Fixes: Resolved several critical issues for better stability
• Documentation Updates: Clearer examples and improved API documentation

🎨 Multi-URL Configurations: One Size Doesn't Fit All

The Problem: You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.

My Solution: I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.

Technical Implementation

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode

# Define specialized configs for different content types
configs = [
    # Documentation sites - aggressive caching, include links
    CrawlerRunConfig(
        url_matcher=["*docs*", "*documentation*"],
        cache_mode="write",
        markdown_generator_options={"include_links": True}
    ),
    
    # News/blog sites - fresh content, scroll for lazy loading
    CrawlerRunConfig(
        url_matcher=lambda url: 'blog' in url or 'news' in url,
        cache_mode="bypass",
        js_code="window.scrollTo(0, document.body.scrollHeight/2);"
    ),
    
    # API endpoints - structured extraction
    CrawlerRunConfig(
        url_matcher=["*.json", "*api*"],
        extraction_strategy=LLMExtractionStrategy(
            provider="openai/gpt-4o-mini",
            extraction_type="structured"
        )
    ),
    
    # Default fallback for everything else
    CrawlerRunConfig()  # No url_matcher = matches everything
]

# Crawl multiple URLs with appropriate configs
async with AsyncWebCrawler() as crawler:
    results = await crawler.arun_many(
        urls=[
            "https://docs.python.org/3/",      # → Uses documentation config
            "https://blog.python.org/",        # → Uses blog config  
            "https://api.github.com/users",    # → Uses API config
            "https://example.com/"             # → Uses default config
        ],
        config=configs
    )

Matching Capabilities:

• String Patterns: Wildcards like "*.pdf", "*/blog/*"
• Function Matchers: Lambda functions for complex logic
• Mixed Matchers: Combine strings and functions with AND/OR logic
• Fallback Support: Default config when nothing matches

Expected Real-World Impact:

• Mixed Content Sites: Handle blogs, docs, and downloads in one crawl
• Multi-Domain Crawling: Different strategies per domain without separate runs
• Reduced Complexity: No more if/else forests in your extraction code
• Better Performance: Each URL gets exactly the processing it needs

🐳 Docker: Flexible LLM Provider Configuration

The Problem: Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.

My Solution: Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.

Deployment Flexibility

# Option 1: Direct environment variables
docker run -d \
  -e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
  -e GROQ_API_KEY="your-key" \
  -p 11235:11235 \
  unclecode/crawl4ai:latest

# Option 2: Using .llm.env file (recommended for production)
# Create .llm.env file:
# LLM_PROVIDER=openai/gpt-4o-mini
# OPENAI_API_KEY=your-openai-key
# GROQ_API_KEY=your-groq-key

docker run -d \
  --env-file .llm.env \
  -p 11235:11235 \
  unclecode/crawl4ai:latest

Override per request when needed:

# Use default provider from .llm.env
response = requests.post("http://localhost:11235/crawl", json={
    "url": "https://example.com",
    "extraction_strategy": {"type": "llm"}
})

# Override to use different provider for this specific request
response = requests.post("http://localhost:11235/crawl", json={
    "url": "https://complex-page.com",
    "extraction_strategy": {
        "type": "llm",
        "provider": "openai/gpt-4"  # Override default
    }
})

Expected Real-World Impact:

• Cost Optimization: Use cheaper models for simple tasks, premium for complex
• A/B Testing: Compare provider performance without deployment changes
• Fallback Strategies: Switch providers on-the-fly during outages
• Development Flexibility: Test locally with one provider, deploy with another
• Secure Configuration: Keep API keys in .llm.env file, not in commands

🔧 Bug Fixes & Improvements

This release includes several important bug fixes that improve stability and reliability:

• URL Matcher Fallback: Fixed edge cases in URL pattern matching logic
• Memory Management: Resolved memory leaks in long-running crawl sessions
• Sitemap Processing: Fixed redirect handling in sitemap fetching
• Table Extraction: Improved table detection and extraction accuracy
• Error Handling: Better error messages and recovery from network failures

📚 Documentation Enhancements

Based on community feedback, we've updated:

• Clearer examples for multi-URL configuration
• Improved CrawlResult documentation with all available fields
• Fixed typos and inconsistencies across documentation
• Added real-world URLs in examples for better understanding
• New comprehensive demo showcasing all v0.7.3 features

🙏 Acknowledgments

Thanks to our contributors and the entire community for feedback and bug reports.

📚 Resources

• Full Documentation
• GitHub Repository
• Discord Community
• Feature Demo

---

Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!

Happy Crawling! 🕷️

- The Crawl4AI Team

🚀 Crawl4AI v0.7.0: The Adaptive Intelligence Update

January 28, 2025 • 10 min read

---

Today I'm releasing Crawl4AI v0.7.0—the Adaptive Intelligence Update. This release introduces fundamental improvements in how Crawl4AI handles modern web complexity through adaptive learning, intelligent content discovery, and advanced extraction capabilities.

🎯 What's New at a Glance

• Adaptive Crawling: Your crawler now learns and adapts to website patterns
• Virtual Scroll Support: Complete content extraction from infinite scroll pages
• Link Preview with Intelligent Scoring: Intelligent link analysis and prioritization
• Async URL Seeder: Discover thousands of URLs in seconds with intelligent filtering
• Performance Optimizations: Significant speed and memory improvements

🧠 Adaptive Crawling: Intelligence Through Pattern Learning

The Problem: Websites change. Class names shift. IDs disappear. Your carefully crafted selectors break at 3 AM, and you wake up to empty datasets and angry stakeholders.

My Solution: I implemented an adaptive learning system that observes patterns, builds confidence scores, and adjusts extraction strategies on the fly. It's like having a junior developer who gets better at their job with every page they scrape.

Technical Deep-Dive

The Adaptive Crawler maintains a persistent state for each domain, tracking:

• Pattern success rates
• Selector stability over time
• Content structure variations
• Extraction confidence scores

from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
import asyncio

async def main():
    
    # Configure adaptive crawler
    config = AdaptiveConfig(
        strategy="statistical",  # or "embedding" for semantic understanding
        max_pages=10,
        confidence_threshold=0.7,  # Stop at 70% confidence
        top_k_links=3,  # Follow top 3 links per page
        min_gain_threshold=0.05  # Need 5% information gain to continue
    )
    
    async with AsyncWebCrawler(verbose=False) as crawler:
        adaptive = AdaptiveCrawler(crawler, config)
        
        print("Starting adaptive crawl about Python decorators...")
        result = await adaptive.digest(
            start_url="https://docs.python.org/3/glossary.html",
            query="python decorators functions wrapping"
        )
        
        print(f"\n✅ Crawling Complete!")
        print(f"• Confidence Level: {adaptive.confidence:.0%}")
        print(f"• Pages Crawled: {len(result.crawled_urls)}")
        print(f"• Knowledge Base: {len(adaptive.state.knowledge_base)} documents")
        
        # Get most relevant content
        relevant = adaptive.get_relevant_content(top_k=3)
        print(f"\nMost Relevant Pages:")
        for i, page in enumerate(relevant, 1):
            print(f"{i}. {page['url']} (relevance: {page['score']:.2%})")

asyncio.run(main())

Expected Real-World Impact:

• News Aggregation: Maintain 95%+ extraction accuracy even as news sites update their templates
• E-commerce Monitoring: Track product changes across hundreds of stores without constant maintenance
• Research Data Collection: Build robust academic datasets that survive website redesigns
• Reduced Maintenance: Cut selector update time by 80% for frequently-changing sites

🌊 Virtual Scroll: Complete Content Capture

The Problem: Modern web apps only render what's visible. Scroll down, new content appears, old content vanishes into the void. Traditional crawlers capture that first viewport and miss 90% of the content. It's like reading only the first page of every book.

My Solution: I built Virtual Scroll support that mimics human browsing behavior, capturing content as it loads and preserving it before the browser's garbage collector strikes.

Implementation Details

from crawl4ai import VirtualScrollConfig

# For social media feeds (Twitter/X style)
twitter_config = VirtualScrollConfig(
    container_selector="[data-testid='primaryColumn']",
    scroll_count=20,                    # Number of scrolls
    scroll_by="container_height",       # Smart scrolling by container size
    wait_after_scroll=1.0              # Let content load
)

# For e-commerce product grids (Instagram style)
grid_config = VirtualScrollConfig(
    container_selector="main .product-grid",
    scroll_count=30,
    scroll_by=800,                     # Fixed pixel scrolling
    wait_after_scroll=1.5              # Images need time
)

# For news feeds with lazy loading
news_config = VirtualScrollConfig(
    container_selector=".article-feed",
    scroll_count=50,
    scroll_by="page_height",           # Viewport-based scrolling
    wait_after_scroll=0.5              # Wait for content to load
)

# Use it in your crawl
async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        "https://twitter.com/trending",
        config=CrawlerRunConfig(
            virtual_scroll_config=twitter_config,
            # Combine with other features
            extraction_strategy=JsonCssExtractionStrategy({
                "tweets": {
                    "selector": "[data-testid='tweet']",
                    "fields": {
                        "text": {"selector": "[data-testid='tweetText']", "type": "text"},
                        "likes": {"selector": "[data-testid='like']", "type": "text"}
                    }
                }
            })
        )
    )
    
    print(f"Captured {len(result.extracted_content['tweets'])} tweets")

Key Capabilities:

• DOM Recycling Awareness: Detects and handles virtual DOM element recycling
• Smart Scroll Physics: Three modes - container height, page height, or fixed pixels
• Content Preservation: Captures content before it's destroyed
• Intelligent Stopping: Stops when no new content appears
• Memory Efficient: Streams content instead of holding everything in memory

Expected Real-World Impact:

• Social Media Analysis: Capture entire Twitter threads with hundreds of replies, not just top 10
• E-commerce Scraping: Extract 500+ products from infinite scroll catalogs vs. 20-50 with traditional methods
• News Aggregation: Get all articles from modern news sites, not just above-the-fold content
• Research Applications: Complete data extraction from academic databases using virtual pagination

🔗 Link Preview: Intelligent Link Analysis and Scoring

The Problem: You crawl a page and get 200 links. Which ones matter? Which lead to the content you actually want? Traditional crawlers force you to follow everything or build complex filters.

My Solution: I implemented a three-layer scoring system that analyzes links like a human would—considering their position, context, and relevance to your goals.

Intelligent Link Analysis and Scoring

import asyncio
from crawl4ai import CrawlerRunConfig, CacheMode, AsyncWebCrawler
from crawl4ai.adaptive_crawler import LinkPreviewConfig

async def main():
    # Configure intelligent link analysis
    link_config = LinkPreviewConfig(
        include_internal=True,
        include_external=False,
        max_links=10,
        concurrency=5,
        query="python tutorial",  # For contextual scoring
        score_threshold=0.3,
        verbose=True
    )
    # Use in your crawl
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://www.geeksforgeeks.org/",
            config=CrawlerRunConfig(
                link_preview_config=link_config,
                score_links=True,  # Enable intrinsic scoring
                cache_mode=CacheMode.BYPASS
            )
        )

        # Access scored and sorted links
        if result.success and result.links:
            for link in result.links.get("internal", []):
                text = link.get('text', 'No text')[:40]
                print(
                    text,
                    f"{link.get('intrinsic_score', 0):.1f}/10" if link.get('intrinsic_score') is not None else "0.0/10",
                    f"{link.get('contextual_score', 0):.2f}/1" if link.get('contextual_score') is not None else "0.00/1",
                    f"{link.get('total_score', 0):.3f}" if link.get('total_score') is not None else "0.000"
                )

asyncio.run(main())

Scoring Components:

1. Intrinsic Score: Based on link quality indicators

   • Position on page (navigation, content, footer)
   • Link attributes (rel, title, class names)
   • Anchor text quality and length
   • URL structure and depth

2. Contextual Score: Relevance to your query using BM25 algorithm

   • Keyword matching in link text and title
   • Meta description analysis
   • Content preview scoring

3. Total Score: Combined score for final ranking

Expected Real-World Impact:

• Research Efficiency: Find relevant papers 10x faster by following only high-score links
• Competitive Analysis: Automatically identify important pages on competitor sites
• Content Discovery: Build topic-focused crawlers that stay on track
• SEO Audits: Identify and prioritize high-value internal linking opportunities

🎣 Async URL Seeder: Automated URL Discovery at Scale

The Problem: You want to crawl an entire domain but only have the homepage. Or worse, you want specific content types across thousands of pages. Manual URL discovery? That's a job for machines, not humans.

My Solution: I built Async URL Seeder—a turbocharged URL discovery engine that combines multiple sources with intelligent filtering and relevance scoring.

Technical Architecture

import asyncio
from crawl4ai import AsyncUrlSeeder, SeedingConfig

async def main():
    async with AsyncUrlSeeder() as seeder:
        # Discover Python tutorial URLs
        config = SeedingConfig(
            source="sitemap",  # Use sitemap
            pattern="*python*",  # URL pattern filter
            extract_head=True,  # Get metadata
            query="python tutorial",  # For relevance scoring
            scoring_method="bm25",
            score_threshold=0.2,
            max_urls=10
        )
        
        print("Discovering Python async tutorial URLs...")
        urls = await seeder.urls("https://www.geeksforgeeks.org/", config)
        
        print(f"\n✅ Found {len(urls)} relevant URLs:")
        for i, url_info in enumerate(urls[:5], 1):
            print(f"\n{i}. {url_info['url']}")
            if url_info.get('relevance_score'):
                print(f"   Relevance: {url_info['relevance_score']:.3f}")
            if url_info.get('head_data', {}).get('title'):
                print(f"   Title: {url_info['head_data']['title'][:60]}...")

asyncio.run(main())

Discovery Methods:

• Sitemap Mining: Parses robots.txt and all linked sitemaps
• Common Crawl: Queries the Common Crawl index for historical URLs
• Intelligent Crawling: Follows links with smart depth control
• Pattern Analysis: Learns URL structures and generates variations

Expected Real-World Impact:

• Migration Projects: Discover 10,000+ URLs from legacy sites in under 60 seconds
• Market Research: Map entire competitor ecosystems automatically
• Academic Research: Build comprehensive datasets without manual URL collection
• SEO Audits: Find every indexable page with content scoring
• Content Archival: Ensure no content is left behind during site migrations

⚡ Performance Optimizations

This release includes significant performance improvements through optimized resource handling, better concurrency management, and reduced memory footprint.

What We Optimized

# Optimized crawling with v0.7.0 improvements
results = []
for url in urls:
    result = await crawler.arun(
        url,
        config=CrawlerRunConfig(
            # Performance optimizations
            wait_until="domcontentloaded",  # Faster than networkidle
            cache_mode=CacheMode.ENABLED    # Enable caching
        )
    )
    results.append(result)

Performance Gains:

• Startup Time: 70% faster browser initialization
• Page Loading: 40% reduction with smart resource blocking
• Extraction: 3x faster with compiled CSS selectors
• Memory Usage: 60% reduction with streaming processing
• Concurrent Crawls: Handle 5x more parallel requests

🔧 Important Changes

Breaking Changes

• link_extractor renamed to link_preview (better reflects functionality)
• Minimum Python version now 3.9
• CrawlerConfig split into CrawlerRunConfig and BrowserConfig

Migration Guide

# Old (v0.6.x)
from crawl4ai import CrawlerConfig
config = CrawlerConfig(timeout=30000)

# New (v0.7.0)
from crawl4ai import CrawlerRunConfig, BrowserConfig
browser_config = BrowserConfig(timeout=30000)
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

🤖 Coming Soon: Intelligent Web Automation

I'm currently working on bringing advanced automation capabilities to Crawl4AI. This includes:

• Crawl Agents: Autonomous crawlers that understand your goals and adapt their strategies
• Auto JS Generation: Automatic JavaScript code generation for complex interactions
• Smart Form Handling: Intelligent form detection and filling
• Context-Aware Actions: Crawlers that understand page context and make decisions

These features are under active development and will revolutionize how we approach web automation. Stay tuned!

🚀 Get Started

pip install crawl4ai==0.7.0

Check out the updated documentation.

Questions? Issues? I'm always listening:

• GitHub: github.com/unclecode/crawl4ai
• Discord: discord.gg/crawl4ai
• Twitter: @unclecode

Happy crawling! 🕷️

---

P.S. If you're using Crawl4AI in production, I'd love to hear about it. Your use cases inspire the next features.

Added

• Browser pooling with page pre‑warming and fine‑grained geolocation, locale, and timezone controls
• Crawler pool manager (SDK + Docker API) for smarter resource allocation
• Network & console log capture plus MHTML snapshot export
• Table extractor: turn HTML <table>s into DataFrames or CSV with one flag
• High‑volume stress‑test framework in tests/memory and API load scripts
• MCP protocol endpoints with socket & SSE support; playground UI scaffold
• Docs v2 revamp: TOC, GitHub badge, copy‑code buttons, Docker API demo
• “Ask AI” helper button (work‑in‑progress, shipping soon)
• New examples: geo‑location usage, network/console capture, Docker API, markdown source selection, crypto analysis
• Expanded automated test suites for browser, Docker, MCP and memory benchmarks

Changed

• Consolidated and renamed browser strategies; legacy docker strategy modules removed
• ProxyConfig moved to async_configs
• Server migrated to pool‑based crawler management
• FastAPI validators replace custom query validation
• Docker build now uses Chromium base image
• Large‑scale repo tidy‑up (≈36 k insertions, ≈5 k deletions)

Fixed

• Async crawler session leak, duplicate‑visit handling, URL normalisation
• Target‑element regressions in scraping strategies
• Logged‑URL readability, encoded‑URL decoding, middle truncation for long URLs
• Closed issues: #701, #733, #756, #774, #804, #822, #839, #841, #842, #843, #867, #902, #911

Removed

• Obsolete modules under crawl4ai/browser/* superseded by the new pooled browser layer

Deprecated

• Old markdown generator names now alias DefaultMarkdownGenerator and emit warnings

---

Upgrade notes

1. Update any direct imports from crawl4ai/browser/* to the new pooled browser modules
2. If you override AsyncPlaywrightCrawlerStrategy.get_page, adopt the new signature
3. Rebuild Docker images to pull the new Chromium layer
4. Switch to DefaultMarkdownGenerator (or silence the deprecation warning)

---

121 files changed, ≈36 223 insertions, ≈4 975 deletions :contentReference[oaicite:0]{index=0}​:contentReference[oaicite:1]{index=1}

[Feature] 2025-04-21

• Implemented MCP protocol for machine-to-machine communication
  • Added WebSocket and SSE transport for MCP server
  • Exposed server endpoints via MCP protocol
  • Created tests for MCP socket and SSE communication
• Enhanced Docker server with file handling and intelligent search
  • Added PDF and screenshot endpoints with file saving capability
  • Added JavaScript execution endpoint for page interaction
  • Implemented advanced context search with BM25 and code chunking
  • Added file path output support for generated assets
• Improved server endpoints and API surface
  • Added intelligent context search with query filtering
  • Added syntax-aware code function chunking
  • Implemented efficient HTML processing pipeline
• Added support for controlling browser geolocation via new GeolocationConfig class
  • Added locale and timezone configuration options to CrawlerRunConfig
  • Added example script demonstrating geolocation and locale usage
  • Added documentation for location-based identity features

[Refactor] 2025-04-20

• Replaced crawler_manager.py with simpler crawler_pool.py implementation
• Added global page semaphore for hard concurrency cap
• Implemented browser pool with idle cleanup
• Added playground UI for testing and stress testing
• Updated API handlers to use pooled crawlers
• Enhanced logging levels and symbols
• Added memory tests and stress test utilities

[Added] 2025-04-17

• Added content source selection feature for markdown generation
  • New content_source parameter allows choosing between cleaned_html, raw_html, and fit_html
  • Provides flexibility in how HTML content is processed before markdown conversion
  • Added examples and documentation for the new feature
  • Includes backward compatibility with default cleaned_html behavior

Crawl4AI v0.6.0 Release Notes

We're excited to announce the release of Crawl4AI v0.6.0, our biggest and most feature-rich update yet. This version introduces major architectural upgrades, brand-new capabilities for geo-aware crawling, high-efficiency scraping, and real-time streaming support for scalable deployments.

---

Highlights

1. World-Aware Crawlers

Crawl as if you’re anywhere in the world. With v0.6.0, each crawl can simulate:

• Specific GPS coordinates
• Browser locale
• Timezone

Example:

CrawlerRunConfig(
    url="https://browserleaks.com/geo",
    locale="en-US",
    timezone_id="America/Los_Angeles",
    geolocation=GeolocationConfig(
        latitude=34.0522,
        longitude=-118.2437,
        accuracy=10.0
    )
)

Great for accessing region-specific content or testing global behavior.

---

2. Native Table Extraction

Extract HTML tables directly into usable formats like Pandas DataFrames or CSV with zero parsing hassle. All table data is available under result.media["tables"].

Example:

raw_df = pd.DataFrame(
    result.media["tables"][0]["rows"],
    columns=result.media["tables"][0]["headers"]
)

This makes it ideal for scraping financial data, pricing pages, or anything tabular.

---

3. Browser Pooling & Pre-Warming

We've overhauled browser management. Now, multiple browser instances can be pooled and pages pre-warmed for ultra-fast launches:

• Reduces cold-start latency
• Lowers memory spikes
• Enhances parallel crawling stability

This powers the new Docker Playground experience and streamlines heavy-load crawling.

---

4. Traffic & Snapshot Capture

Need full visibility? You can now capture:

• Full network traffic logs
• Console output
• MHTML page snapshots for post-crawl audits and debugging

No more guesswork on what happened during your crawl.

---

5. MCP API and Streaming Support

We’re exposing MCP socket and SSE endpoints, allowing:

• Live streaming of crawl results
• Real-time integration with agents or frontends
• A new Playground UI for interactive crawling

This is a major step towards making Crawl4AI real-time ready.

---

6. Stress-Test Framework

Want to test performance under heavy load? v0.6.0 includes a new memory stress-test suite that supports 1,000+ URL workloads. Ideal for:

• Load testing
• Performance benchmarking
• Validating memory efficiency

---

Core Improvements

• Robots.txt compliance
• Proxy rotation support
• Improved URL normalization and session reuse
• Shared data across crawler hooks
• New page routing logic

---

Breaking Changes & Deprecations

• Legacy crawl4ai/browser/* modules are removed. Update imports accordingly.
• AsyncPlaywrightCrawlerStrategy.get_page now uses a new function signature.
• Deprecated markdown generator aliases now point to DefaultMarkdownGenerator with warning.

---

Miscellaneous Updates

• FastAPI validators replaced custom validation logic
• Docker build now based on a Chromium layer
• Repo-wide cleanup: ~36,000 insertions, ~5,000 deletions

---

New Examples Included

• Geo-location crawling
• Network + console log capture
• Docker MCP API usage
• Markdown selector usage
• Crypto project data extraction

---

Watch the Release Video

Want a visual walkthrough of all these updates? Watch the video: 🔗 https://youtu.be/9x7nVcjOZks

If you're new to Crawl4AI, start here: 🔗 https://www.youtube.com/watch?v=xo3qK6Hg9AA&t=15s

---

Join the Community

We’ve just opened up our Discord for the public. Join us to:

• Ask questions
• Share your projects
• Get help or contribute

💬 https://discord.gg/wpYFACrHR4

---

Install or Upgrade

pip install -U crawl4ai

---

Live long and import crawl4ai. 🖖

Crawl4AI v0.5.0 Release Notes

Release Theme: Power, Flexibility, and Scalability

Crawl4AI v0.5.0 is a major release focused on significantly enhancing the library's power, flexibility, and scalability. Key improvements include a new deep crawling system, a memory-adaptive dispatcher for handling large-scale crawls, multiple crawling strategies (including a fast HTTP-only crawler), Docker deployment options, and a powerful command-line interface (CLI). This release also includes numerous bug fixes, performance optimizations, and documentation updates.

Important Note: This release contains several breaking changes. Please review the "Breaking Changes" section carefully and update your code accordingly.

Key Features

1. Deep Crawling

Crawl4AI now supports deep crawling, allowing you to explore websites beyond the initial URLs. This is controlled by the deep_crawl_strategy parameter in CrawlerRunConfig. Several strategies are available:

• BFSDeepCrawlStrategy (Breadth-First Search): Explores the website level by level. (Default)
• DFSDeepCrawlStrategy (Depth-First Search): Explores each branch as deeply as possible before backtracking.
• BestFirstCrawlingStrategy: Uses a scoring function to prioritize which URLs to crawl next.

import time
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BFSDeepCrawlStrategy
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
from crawl4ai.deep_crawling import DomainFilter, ContentTypeFilter, FilterChain, URLPatternFilter, KeywordRelevanceScorer, BestFirstCrawlingStrategy
import asyncio

# Create a filter chain to filter urls based on patterns, domains and content type
filter_chain = FilterChain(
    [
        DomainFilter(
            allowed_domains=["docs.crawl4ai.com"],
            blocked_domains=["old.docs.crawl4ai.com"],
        ),
        URLPatternFilter(patterns=["*core*", "*advanced*"],),
        ContentTypeFilter(allowed_types=["text/html"]),
    ]
)

# Create a keyword scorer that prioritises the pages with certain keywords first
keyword_scorer = KeywordRelevanceScorer(
    keywords=["crawl", "example", "async", "configuration"], weight=0.7
)

# Set up the configuration
deep_crawl_config = CrawlerRunConfig(
    deep_crawl_strategy=BestFirstCrawlingStrategy(
        max_depth=2,
        include_external=False,
        filter_chain=filter_chain,
        url_scorer=keyword_scorer,
    ),
    scraping_strategy=LXMLWebScrapingStrategy(),
    stream=True,
    verbose=True,
)

async def main():
    async with AsyncWebCrawler() as crawler:
        start_time = time.perf_counter()
        results = []
        async for result in await crawler.arun(url="https://docs.crawl4ai.com", config=deep_crawl_config):
            print(f"Crawled: {result.url} (Depth: {result.metadata['depth']}), score: {result.metadata['score']:.2f}")
            results.append(result)
        duration = time.perf_counter() - start_time
        print(f"\n✅ Crawled {len(results)} high-value pages in {duration:.2f} seconds")

asyncio.run(main())

Breaking Change: The max_depth parameter is now part of CrawlerRunConfig and controls the depth of the crawl, not the number of concurrent crawls. The arun() and arun_many() methods are now decorated to handle deep crawling strategies. Imports for deep crawling strategies have changed. See the Deep Crawling documentation for more details.

2. Memory-Adaptive Dispatcher

The new MemoryAdaptiveDispatcher dynamically adjusts concurrency based on available system memory and includes built-in rate limiting. This prevents out-of-memory errors and avoids overwhelming target websites.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MemoryAdaptiveDispatcher
import asyncio

# Configure the dispatcher (optional, defaults are used if not provided)
dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=80.0,  # Pause if memory usage exceeds 80%
    check_interval=0.5,  # Check memory every 0.5 seconds
)

async def batch_mode():
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun_many(
            urls=["https://docs.crawl4ai.com", "https://github.com/unclecode/crawl4ai"],
            config=CrawlerRunConfig(stream=False),  # Batch mode
            dispatcher=dispatcher,
        )
        for result in results:
            print(f"Crawled: {result.url} with status code: {result.status_code}")

async def stream_mode():
    async with AsyncWebCrawler() as crawler:
        # OR, for streaming:
        async for result in await crawler.arun_many(
            urls=["https://docs.crawl4ai.com", "https://github.com/unclecode/crawl4ai"],
            config=CrawlerRunConfig(stream=True),
            dispatcher=dispatcher,
        ):
            print(f"Crawled: {result.url} with status code: {result.status_code}")

print("Dispatcher in batch mode:")
asyncio.run(batch_mode())
print("-" * 50)
print("Dispatcher in stream mode:")
asyncio.run(stream_mode())

Breaking Change: AsyncWebCrawler.arun_many() now uses MemoryAdaptiveDispatcher by default. Existing code that relied on unbounded concurrency may require adjustments.

3. Multiple Crawling Strategies (Playwright and HTTP)

Crawl4AI now offers two crawling strategies:

• AsyncPlaywrightCrawlerStrategy (Default): Uses Playwright for browser-based crawling, supporting JavaScript rendering and complex interactions.
• AsyncHTTPCrawlerStrategy: A lightweight, fast, and memory-efficient HTTP-only crawler. Ideal for simple scraping tasks where browser rendering is unnecessary.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
import asyncio

# Use the HTTP crawler strategy
http_crawler_config = HTTPCrawlerConfig(
        method="GET",
        headers={"User-Agent": "MyCustomBot/1.0"},
        follow_redirects=True,
        verify_ssl=True
)

async def main():
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy(browser_config =http_crawler_config)) as crawler:
        result = await crawler.arun("https://example.com")
        print(f"Status code: {result.status_code}")
        print(f"Content length: {len(result.html)}")

asyncio.run(main())

4. Docker Deployment

Crawl4AI can now be easily deployed as a Docker container, providing a consistent and isolated environment. The Docker image includes a FastAPI server with both streaming and non-streaming endpoints.

# Build the image (from the project root)
docker build -t crawl4ai .

# Run the container
docker run -d -p 8000:8000 --name crawl4ai crawl4ai

API Endpoints:

• /crawl (POST): Non-streaming crawl.
• /crawl/stream (POST): Streaming crawl (NDJSON).
• /health (GET): Health check.
• /schema (GET): Returns configuration schemas.
• /md/{url} (GET): Returns markdown content of the URL.
• /llm/{url} (GET): Returns LLM extracted content.
• /token (POST): Get JWT token

Breaking Changes:

• Docker deployment now requires a .llm.env file for API keys.
• Docker deployment now requires Redis and a new config.yml structure.
• Server startup now uses supervisord instead of direct process management.
• Docker server now requires authentication by default (JWT tokens).

See the Docker deployment documentation for detailed instructions.

5. Command-Line Interface (CLI)

A new CLI (crwl) provides convenient access to Crawl4AI's functionality from the terminal.

# Basic crawl
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# Use a configuration file
crwl https://example.com -B browser.yml -C crawler.yml

# Use LLM-based extraction
crwl https://example.com -e extract.yml -s schema.json

# Ask a question about the crawled content
crwl https://example.com -q "What is the main topic?"

# See usage examples
crwl --example

See the CLI documentation for more details.

6. LXML Scraping Mode

Added LXMLWebScrapingStrategy for faster HTML parsing using the lxml library. This can significantly improve scraping performance, especially for large or complex pages. Set scraping_strategy=LXMLWebScrapingStrategy() in your CrawlerRunConfig.

Breaking Change: The ScrapingMode enum has been replaced with a strategy pattern. Use WebScrapingStrategy (default) or LXMLWebScrapingStrategy.

7. Proxy Rotation

Added ProxyRotationStrategy abstract base class with RoundRobinProxyStrategy concrete implementation.

import re
from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    CacheMode,
    RoundRobinProxyStrategy,
)
import asyncio
from crawl4ai import ProxyConfig
async def main():
    # Load proxies and create rotation strategy
    proxies = ProxyConfig.from_env()
    #eg: export PROXIES="ip1:port1:username1:password1,ip2:port2:username2:password2"
    if not proxies:
        print("No proxies found in environment. Set PROXIES env variable!")
        return
        
    proxy_strategy = RoundRobinProxyStrategy(proxies)
    
    # Create configs
    browser_config = BrowserConfig(headless=True, verbose=False)
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        proxy_rotation_strategy=proxy_strategy
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        urls = ["https://httpbin.org/ip"] * (len(proxies) * 2)  # Test each proxy twice

        print("\n📈 Initializing crawler with proxy rotation...")
        async with AsyncWebCrawler(config=browser_config) as crawler:
            print("\n🚀 Starting batch crawl with proxy rotation...")
            results = await crawler.arun_many(
                urls=urls,
                config=run_config
            )
            for result in results:
                if result.success:
                    ip_match = re.search(r'(?:[0-9]{1,3}\.){3}[0-9]{1,3}', result.html)
                    current_proxy = run_config.proxy_config if run_config.proxy_config else None
                    
                    if current_proxy and ip_match:
                        print(f"URL {result.url}")
                        print(f"Proxy {current_proxy.server} -> Response IP: {ip_match.group(0)}")
                        verified = ip_match.group(0) == current_proxy.ip
                        if verified:
                            print(f"✅ Proxy working! IP matches: {current_proxy.ip}")
                        else:
                            print("❌ Proxy failed or IP mismatch!")
                    print("---")

asyncio.run(main())

Other Changes and Improvements

• Added: LLMContentFilter for intelligent markdown generation. This new filter uses an LLM to create more focused and relevant markdown output.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import LLMContentFilter
from crawl4ai import LLMConfig
import asyncio

llm_config = LLMConfig(provider="gemini/gemini-1.5-pro", api_token="env:GEMINI_API_KEY")

markdown_generator = DefaultMarkdownGenerator(
    content_filter=LLMContentFilter(llm_config=llm_config, instruction="Extract key concepts and summaries")
)

config = CrawlerRunConfig(markdown_generator=markdown_generator)
async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://docs.crawl4ai.com", config=config)
        print(result.markdown.fit_markdown)

asyncio.run(main())

• Added: URL redirection tracking. The crawler now automatically follows HTTP redirects (301, 302, 307, 308) and records the final URL in the redirected_url field of the CrawlResult object. No code changes are required to enable this; it's automatic.

• Added: LLM-powered schema generation utility. A new generate_schema method has been added to JsonCssExtractionStrategy and JsonXPathExtractionStrategy. This greatly simplifies creating extraction schemas.

from crawl4ai import JsonCssExtractionStrategy
from crawl4ai import LLMConfig

llm_config = LLMConfig(provider="gemini/gemini-1.5-pro", api_token="env:GEMINI_API_KEY")

schema = JsonCssExtractionStrategy.generate_schema(
    html="<div class='product'><h2>Product Name</h2><span class='price'>$99</span></div>",
    llm_config = llm_config,
    query="Extract product name and price"
)
print(schema)

Expected Output (may vary slightly due to LLM)

{
  "name": "ProductExtractor",
  "baseSelector": "div.product",
  "fields": [
      {"name": "name", "selector": "h2", "type": "text"},
      {"name": "price", "selector": ".price", "type": "text"}
    ]
 }

• Added: robots.txt compliance support. The crawler can now respect robots.txt rules. Enable this by setting check_robots_txt=True in CrawlerRunConfig.

  config = CrawlerRunConfig(check_robots_txt=True)
  

• Added: PDF processing capabilities. Crawl4AI can now extract text, images, and metadata from PDF files (both local and remote). This uses a new PDFCrawlerStrategy and PDFContentScrapingStrategy.

  from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
  from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy
  import asyncio
  
  async def main():
      async with AsyncWebCrawler(crawler_strategy=PDFCrawlerStrategy()) as crawler:
          result = await crawler.arun(
              "https://arxiv.org/pdf/2310.06825.pdf",
              config=CrawlerRunConfig(
                  scraping_strategy=PDFContentScrapingStrategy()
              )
          )
          print(result.markdown)  # Access extracted text
          print(result.metadata)  # Access PDF metadata (title, author, etc.)
  
  asyncio.run(main())
  

• Added: Support for frozenset serialization. Improves configuration serialization, especially for sets of allowed/blocked domains. No code changes required.

• Added: New LLMConfig parameter. This new parameter can be passed for extraction, filtering, and schema generation tasks. It simplifies passing provider strings, API tokens, and base URLs across all sections where LLM configuration is necessary. It also enables reuse and allows for quick experimentation between different LLM configurations.

  from crawl4ai import LLMConfig
  from crawl4ai import LLMExtractionStrategy
  from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
  
  # Example of using LLMConfig with LLMExtractionStrategy
  llm_config = LLMConfig(provider="openai/gpt-4o", api_token="YOUR_API_KEY")
  strategy = LLMExtractionStrategy(llm_config=llm_config, schema=...)
  
  # Example usage within a crawler
  async with AsyncWebCrawler() as crawler:
      result = await crawler.arun(
          url="https://example.com",
          config=CrawlerRunConfig(extraction_strategy=strategy)
      )
  

  Breaking Change: Removed old parameters like provider, api_token, base_url, and api_base from LLMExtractionStrategy and LLMContentFilter. Users should migrate to using the LLMConfig object.

• Changed: Improved browser context management and added shared data support. (Breaking Change: BrowserContext API updated). Browser contexts are now managed more efficiently, reducing resource usage. A new shared_data dictionary is available in the BrowserContext to allow passing data between different stages of the crawling process. Breaking Change: The BrowserContext API has changed, and the old get_context method is deprecated.

• Changed: Renamed final_url to redirected_url in CrawledURL. This improves consistency and clarity. Update any code referencing the old field name.

• Changed: Improved type hints and removed unused files. This is an internal improvement and should not require code changes.

• Changed: Reorganized deep crawling functionality into dedicated module. (Breaking Change: Import paths for DeepCrawlStrategy and related classes have changed). This improves code organization. Update imports to use the new crawl4ai.deep_crawling module.

• Changed: Improved HTML handling and cleanup codebase. (Breaking Change: Removed ssl_certificate.json file). This removes an unused file. If you were relying on this file for custom certificate validation, you'll need to implement an alternative approach.

• Changed: Enhanced serialization and config handling. (Breaking Change: FastFilterChain has been replaced with FilterChain). This change simplifies config and improves the serialization.

• Added: Modified the license to Apache 2.0 with a required attribution clause. See the LICENSE file for details. All users must now clearly attribute the Crawl4AI project when using, distributing, or creating derivative works.

• Fixed: Prevent memory leaks by ensuring proper closure of Playwright pages. No code changes required.

• Fixed: Make model fields optional with default values (Breaking Change: Code relying on all fields being present may need adjustment). Fields in data models (like CrawledURL) are now optional, with default values (usually None). Update code to handle potential None values.

• Fixed: Adjust memory threshold and fix dispatcher initialization. This is an internal bug fix; no code changes are required.

• Fixed: Ensure proper exit after running doctor command. No code changes are required.

• Fixed: JsonCss selector and crawler improvements.

• Fixed: Not working long page screenshot (#403)

• Documentation: Updated documentation URLs to the new domain.

• Documentation: Added SERP API project example.

• Documentation: Added clarifying comments for CSS selector behavior.

• Documentation: Add Code of Conduct for the project (#410)

Breaking Changes Summary

• Dispatcher: The MemoryAdaptiveDispatcher is now the default for arun_many(), changing concurrency behavior. The return type of arun_many depends on the stream parameter.
• Deep Crawling: max_depth is now part of CrawlerRunConfig and controls crawl depth. Import paths for deep crawling strategies have changed.
• Browser Context: The BrowserContext API has been updated.
• Models: Many fields in data models are now optional, with default values.
• Scraping Mode: ScrapingMode enum replaced by strategy pattern (WebScrapingStrategy, LXMLWebScrapingStrategy).
• Content Filter: Removed content_filter parameter from CrawlerRunConfig. Use extraction strategies or markdown generators with filters instead.
• Removed: Synchronous WebCrawler, CLI, and docs management functionality.
• Docker: Significant changes to Docker deployment, including new requirements and configuration.
• File Removed: Removed ssl_certificate.json file which might affect existing certificate validations
• Renamed: final_url to redirected_url for consistency
• Config: FastFilterChain has been replaced with FilterChain
• Deep-Crawl: DeepCrawlStrategy.arun now returns Union[CrawlResultT, List[CrawlResultT], AsyncGenerator[CrawlResultT, None]]
• Proxy: Removed synchronous WebCrawler support and related rate limiting configurations

Migration Guide

1. Update Imports: Adjust imports for DeepCrawlStrategy, BreadthFirstSearchStrategy, and related classes due to the new deep_crawling module structure.
2. CrawlerRunConfig: Move max_depth to CrawlerRunConfig. If using content_filter, migrate to an extraction strategy or a markdown generator with a filter.
3. arun_many(): Adapt code to the new MemoryAdaptiveDispatcher behavior and the return type.
4. BrowserContext: Update code using the BrowserContext API.
5. Models: Handle potential None values for optional fields in data models.
6. Scraping: Replace ScrapingMode enum with WebScrapingStrategy or LXMLWebScrapingStrategy.
7. Docker: Review the updated Docker documentation and adjust your deployment accordingly.
8. CLI: Migrate to the new crwl command and update any scripts using the old CLI.
9. Proxy:: Removed synchronous WebCrawler support and related rate limiting configurations.
10. Config:: Replace FastFilterChain to FilterChain

Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features!

This post was generated with the help of ChatGPT, take everything with a grain of salt. 🧂

Hi everyone,

I just finished putting together version 0.4.1 of Crawl4AI, and there are a few changes in here that I think you’ll find really helpful. I’ll explain what’s new, why it matters, and exactly how you can use these features (with the code to back it up). Let’s get into it.

---

Handling Lazy Loading Better (Images Included)

One thing that always bugged me with crawlers is how often they miss lazy-loaded content, especially images. In this version, I made sure Crawl4AI waits for all images to load before moving forward. This is useful because many modern websites only load images when they’re in the viewport or after some JavaScript executes.

Here’s how to enable it:

await crawler.crawl(
    url="https://example.com",
    wait_for_images=True  # Add this argument to ensure images are fully loaded
)

What this does is:

1. Waits for the page to reach a "network idle" state.
2. Ensures all images on the page have been completely loaded.

This single change handles the majority of lazy-loading cases you’re likely to encounter.

---

Text-Only Mode (Fast, Lightweight Crawling)

Sometimes, you don’t need to download images or process JavaScript at all. For example, if you’re crawling to extract text data, you can enable text-only mode to speed things up. By disabling images, JavaScript, and other heavy resources, this mode makes crawling 3-4 times faster in most cases.

Here’s how to turn it on:

crawler = AsyncPlaywrightCrawlerStrategy(
    text_mode=True  # Set this to True to enable text-only crawling
)

When text_mode=True, the crawler automatically:

• Disables GPU processing.
• Blocks image and JavaScript resources.
• Reduces the viewport size to 800x600 (you can override this with viewport_width and viewport_height).

If you need to crawl thousands of pages where you only care about text, this mode will save you a ton of time and resources.

---

Adjusting the Viewport Dynamically

Another useful addition is the ability to dynamically adjust the viewport size to match the content on the page. This is particularly helpful when you’re working with responsive layouts or want to ensure all parts of the page load properly.

Here’s how it works:

1. The crawler calculates the page’s width and height after it loads.
2. It adjusts the viewport to fit the content dimensions.
3. (Optional) It uses Chrome DevTools Protocol (CDP) to simulate zooming out so everything fits in the viewport.

To enable this, use:

await crawler.crawl(
    url="https://example.com",
    adjust_viewport_to_content=True  # Dynamically adjusts the viewport
)

This approach makes sure the entire page gets loaded into the viewport, especially for layouts that load content based on visibility.

---

Simulating Full-Page Scrolling

Some websites load data dynamically as you scroll down the page. To handle these cases, I added support for full-page scanning. It simulates scrolling to the bottom of the page, checking for new content, and capturing it all.

Here’s an example:

await crawler.crawl(
    url="https://example.com",
    scan_full_page=True,   # Enables scrolling
    scroll_delay=0.2       # Waits 200ms between scrolls (optional)
)

What happens here:

1. The crawler scrolls down in increments, waiting for content to load after each scroll.
2. It stops when no new content appears (i.e., dynamic elements stop loading).
3. It scrolls back to the top before finishing (if necessary).

If you’ve ever had to deal with infinite scroll pages, this is going to save you a lot of headaches.

---

Reusing Browser Sessions (Save Time on Setup)

By default, every time you crawl a page, a new browser context (or tab) is created. That’s fine for small crawls, but if you’re working on a large dataset, it’s more efficient to reuse the same session.

I added a method called create_session for this:

session_id = await crawler.create_session()

# Use the same session for multiple crawls
await crawler.crawl(
    url="https://example.com/page1",
    session_id=session_id  # Reuse the session
)
await crawler.crawl(
    url="https://example.com/page2",
    session_id=session_id
)

This avoids creating a new tab for every page, speeding up the crawl and reducing memory usage.

---

Other Updates

Here are a few smaller updates I’ve made:

• Light Mode: Use light_mode=True to disable background processes, extensions, and other unnecessary features, making the browser more efficient.
• Logging: Improved logs to make debugging easier.
• Defaults: Added sensible defaults for things like delay_before_return_html (now set to 0.1 seconds).

---

How to Get the Update

You can install or upgrade to version 0.4.1 like this:

pip install crawl4ai --upgrade

As always, I’d love to hear your thoughts. If there’s something you think could be improved or if you have suggestions for future versions, let me know!

Enjoy the new features, and happy crawling! 🕷️

---