NewYour coding agent can read the release notes before it upgrades.Set up the MCP server β
PyPI, #3610 most downloaded on PyPI
ππ€ Crawl4AI: Open-source LLM Friendly Web Crawler & scraper
Latest stable
0.9.5
Last release 1 months ago
14 Oct 2026
Ships fairly regularly
a new release about every 3 weeks
Most releases are documented
notes for 37 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
2 years old
73 releases Β· first in 2024
Drag across the ruler to compare, or tap a release to open it.
Removed deprecated: crawl4ai/content_cleaning_strategy.py.
This changelog details the updates and changes introduced in Crawl4AI version 0.3.74. It's designed to inform developers about new features, modifications to existing components, removals, and other important information.
downloads_path parameter in the AsyncWebCrawler constructor or the arun method. If not specified, downloads are saved to a "downloads" folder within the .crawl4ai directory.CrawlResult object. Successfully downloaded files are listed in the downloaded_files attribute, providing their paths.accept_downloads parameter to the crawler strategies (defaults to False). If set to True you can add JS code and wait_for parameter for file download.Example:
import asyncio
import os
from pathlib import Path
from crawl4ai import AsyncWebCrawler
async def download_example():
downloads_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
os.makedirs(downloads_path, exist_ok=True)
async with AsyncWebCrawler(
accept_downloads=True,
downloads_path=downloads_path,
verbose=True
) as crawler:
result = await crawler.arun(
url="https://www.python.org/downloads/",
js_code="""
const downloadLink = document.querySelector('a[href$=".exe"]');
if (downloadLink) { downloadLink.click(); }
""",
wait_for=5 # To ensure download has started
)
if result.downloaded_files:
print("Downloaded files:")
for file in result.downloaded_files:
print(f"- {file}")
asyncio.run(download_example())
RelevanceContentFilter strategy (and its implementation BM25ContentFilter) for extracting relevant content from web pages, replacing Fit Markdown and other content cleaning strategy. This new strategy leverages the BM25 algorithm to identify chunks of text relevant to the page's title, description, keywords, or a user-provided query.fit_markdown flag in the content scraper is used to filter content based on title, meta description, and keywords.Example:
from crawl4ai import AsyncWebCrawler
from crawl4ai.content_filter_strategy import BM25ContentFilter
async def filter_content(url, query):
async with AsyncWebCrawler() as crawler:
content_filter = BM25ContentFilter(user_query=query)
result = await crawler.arun(url=url, extraction_strategy=content_filter, fit_markdown=True)
print(result.extracted_content) # Or result.fit_markdown for the markdown version
print(result.fit_html) # Or result.fit_html to show HTML with only the filtered content
asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple", "fruit nutrition health"))
file:// prefix for local file paths.raw: prefix for raw HTML strings.Example:
async def crawl_local_or_raw(crawler, content, content_type):
prefix = "file://" if content_type == "local" else "raw:"
url = f"{prefix}{content}"
result = await crawler.arun(url=url)
if result.success:
print(f"Markdown Content from {content_type.title()} Source:")
print(result.markdown)
# Example usage with local file and raw HTML
async def main():
async with AsyncWebCrawler() as crawler:
# Local File
await crawl_local_or_raw(
crawler, os.path.abspath('tests/async/sample_wikipedia.html'), "local"
)
# Raw HTML
await crawl_raw_html(crawler, "<h1>Raw Test</h1><p>This is raw HTML.</p>")
asyncio.run(main())
ManagedBrowser class introduced for improved browser session handling, offering features like persistent browser sessions between requests (using session_id parameter) and browser process monitoring.use_managed_browser, use_persistent_context, and chrome_channel parameters to AsyncPlaywrightCrawlerStrategy.Example:
async def browser_management_demo():
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "user-data-dir")
os.makedirs(user_data_dir, exist_ok=True) # Ensure directory exists
async with AsyncWebCrawler(
use_managed_browser=True,
user_data_dir=user_data_dir,
use_persistent_context=True,
verbose=True
) as crawler:
result1 = await crawler.arun(
url="https://example.com", session_id="my_session"
)
result2 = await crawler.arun(
url="https://example.com/anotherpage", session_id="my_session"
)
asyncio.run(browser_management_demo())
CacheMode enum (ENABLED, DISABLED, READ_ONLY, WRITE_ONLY, BYPASS) and always_bypass_cache parameter in AsyncWebCrawler for fine-grained cache control. This replaces bypass_cache, no_cache_read, no_cache_write, and always_by_pass_cache.crawl4ai/content_cleaning_strategy.py.bypass_cache, disable_cache, no_cache_read, no_cache_write, and always_by_pass_cache. These have been superseded by cache_mode.crawl4ai/__version__.py.crawl4ai/cache_context.py.crawl4ai/version_manager.py.crawl4ai/migrations.py.crawl4ai-migrate entry point.NEED_MIGRATION and SHOW_DEPRECATION_WARNINGS.CRAWL4AI_API_TOKEN environment variable. This enhances API security./crawl_sync for immediate result retrieval, and direct crawl endpoint /crawl_direct bypassing the task queue.WebCrawler is being phased out. While still available via crawl4ai[sync], it will eventually be removed. Transition to AsyncWebCrawler is strongly recommended. Boolean cache control flags in arun are also deprecated, migrate to using the cache_mode parameter. See examples in the "New Features" section above for correct usage.__del__ method and ensuring the browser context is closed explicitly using context managers.WebScrapingStrategy. More detailed error messages and suggestions for debugging will minimize frustration when running into unexpected issues.Old way:
crawler = AsyncWebCrawler(always_by_pass_cache=True)
result = await crawler.arun(url="https://example.com", bypass_cache=True)
New way:
from crawl4ai import CacheMode
crawler = AsyncWebCrawler(always_bypass_cache=True)
result = await crawler.arun(url="https://example.com", cache_mode=CacheMode.BYPASS)
Your coding agent can read these notes before it upgrades. Set up the MCP server β
Added comprehensive system diagnostics tool
New Doctor Feature
Dockerized API Server
Managed Browser Integration
ManagedBrowser class for better browser lifecycle managementEnhanced HTML Processing
Browser Handling
Database Management
Resource Management
When upgrading to v0.3.73, be aware of the following changes:
Docker Deployment:
If using custom browser management:
For database operations:
Using the Doctor:
crawl4ai doctorNew ContentCleaningStrategy class:
ContentCleaningStrategy class:
proxy_config option for authenticated proxy connectionsfit_markdown: Optimized markdown output with main content focusfit_html: Clean HTML with only essential contentsrc, data-src, srcset, etc.)ContentCleaningStrategy uses configurable thresholds for customizationOverlappingWindowChunking: Allows for overlapping chunks of text, useful for maintaining context between chunks.
OverlappingWindowChunking: Allows for overlapping chunks of text, useful for maintaining context between chunks.SlidingWindowChunking: Improved to handle edge cases and last chunks more effectively.CHUNK_TOKEN_THRESHOLD in config to 2048 tokens (2^11) for better compatibility with most LLM models.AsyncPlaywrightCrawlerStrategy.close() method to use a shorter sleep time (0.5 seconds instead of 500), significantly reducing wait time when closing the crawler.CosineStrategy:
load_HF_embedding_model function, allowing for easier swapping of embedding models.JsonCssExtractionStrategy and JsonXPathExtractionStrategy for better JSON-based extraction.CosineStrategy, now using the automatically detected device.quickstart_async.py for generating a knowledge graph from crawled content.These updates aim to provide more flexibility in text processing, improve performance, and enhance the overall capabilities of the crawl4ai library. The new chunking strategies, in particular, offer more options for handling large texts in various scenarios.
Nothing published for this version
Implemented playwright_stealth for improved bot detection avoidance.
Enhanced Browser Stealth:
playwright_stealth for improved bot detection avoidance.StealthConfig for fine-tuned control over stealth parameters.User Simulation:
simulate_user option to mimic human-like interactions (mouse movements, clicks, keyboard presses).Navigator Override:
override_navigator option to modify navigator properties, further improving bot detection evasion.Improved iframe Handling:
process_iframes parameter to extract and integrate iframe content into the main page.Flexible Browser Selection:
Include Links in Markdown:
include_links_on_markdown in crawl method.Better Error Handling:
Image Processing Enhancements:
Crawling Flexibility:
delay_before_return_html parameter.Performance Optimization:
crawl_with_user_simulation() demonstrating the use of user simulation and navigator override features.New Hook: Added before_retrieve_html hook in AsyncPlaywrightCrawlerStrategy.
before_retrieve_html hook in AsyncPlaywrightCrawlerStrategy.delay_before_return_html parameter to allow waiting before retrieving HTML content.
smart_wait function now uses page_timeout (default 60 seconds) instead of a fixed 30-second timeout.
page_timeout=your_desired_timeout (in milliseconds) when calling crawler.arun().browser_type="firefox" or browser_type="webkit" when initializing AsyncWebCrawler.screenshot=True when calling crawler.arun().extra_args parameter.LLMExtractionStrategy.process_iframes=True in the crawl method.get_delayed_content method in AsyncCrawlResponse.result.get_delayed_content(delay_in_seconds) after crawling.perform_completion_with_backoff function now supports additional arguments for more customized API calls to LLM providers.quickstart_async.py with examples of:
These updates significantly enhance the flexibility, accuracy, and robustness of crawl4ai, providing users with more control and options for their web crawling and content extraction tasks.
Enhance AsyncWebCrawler with smart waiting and screenshot capabilities
Enhance AsyncWebCrawler with smart waiting and screenshot capabilities
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per month.