crawl4ai

Added

• New ContentCleaningStrategy class:
  • Smart content extraction based on text density and element scoring
  • Automatic removal of boilerplate content
  • DOM tree analysis for better content identification
  • Configurable thresholds for content detection
• Advanced proxy support:
  • Added proxy_config option for authenticated proxy connections
  • Support for username/password in proxy configuration
• New content output formats:
  • fit_markdown: Optimized markdown output with main content focus
  • fit_html: Clean HTML with only essential content

Enhanced

• Image source detection:
  • Support for multiple image source attributes (src, data-src, srcset, etc.)
  • Automatic fallback through potential source attributes
  • Smart handling of srcset attribute
• External content handling:
  • Made external link exclusion optional (disabled by default)
  • Improved detection and handling of social media links
  • Better control over external image filtering

Fixed

• Image extraction reliability with multiple source attribute checks
• External link and image handling logic for better accuracy

Developer Notes

• The new ContentCleaningStrategy uses configurable thresholds for customization
• Proxy configuration now supports more complex authentication scenarios
• Content extraction process now provides both regular and optimized outputs

Added

• New chunking strategies:
  • OverlappingWindowChunking: Allows for overlapping chunks of text, useful for maintaining context between chunks.
  • Enhanced SlidingWindowChunking: Improved to handle edge cases and last chunks more effectively.

Changed

• Updated CHUNK_TOKEN_THRESHOLD in config to 2048 tokens (2^11) for better compatibility with most LLM models.
• Improved AsyncPlaywrightCrawlerStrategy.close() method to use a shorter sleep time (0.5 seconds instead of 500), significantly reducing wait time when closing the crawler.
• Enhanced flexibility in CosineStrategy:
  • Now uses a more generic load_HF_embedding_model function, allowing for easier swapping of embedding models.
• Updated JsonCssExtractionStrategy and JsonXPathExtractionStrategy for better JSON-based extraction.

Fixed

• Addressed potential issues with the sliding window chunking strategy to ensure all text is properly chunked.

Developer Notes

• Added more comprehensive docstrings to chunking strategies for better code documentation.
• Removed hardcoded device setting in CosineStrategy, now using the automatically detected device.
• Added a new example in quickstart_async.py for generating a knowledge graph from crawled content.

These updates aim to provide more flexibility in text processing, improve performance, and enhance the overall capabilities of the crawl4ai library. The new chunking strategies, in particular, offer more options for handling large texts in various scenarios.

New Features

1. Enhanced Browser Stealth:

   • Implemented playwright_stealth for improved bot detection avoidance.
   • Added StealthConfig for fine-tuned control over stealth parameters.

2. User Simulation:

   • New simulate_user option to mimic human-like interactions (mouse movements, clicks, keyboard presses).

3. Navigator Override:

   • Added override_navigator option to modify navigator properties, further improving bot detection evasion.

4. Improved iframe Handling:

   • New process_iframes parameter to extract and integrate iframe content into the main page.

5. Flexible Browser Selection:

   • Support for choosing between Chromium, Firefox, and WebKit browsers.

6. Include Links in Markdown:

   • Added support for including links in Markdown content, by definin g a new flag include_links_on_markdown in crawl method.

Improvements

1. Better Error Handling:

   • Enhanced error reporting in WebScrapingStrategy with detailed error messages and suggestions.
   • Added console message and error logging for better debugging.

2. Image Processing Enhancements:

   • Improved image dimension updating and filtering logic.

3. Crawling Flexibility:

   • Added support for custom viewport sizes.
   • Implemented delayed content retrieval with delay_before_return_html parameter.

4. Performance Optimization:

   • Adjusted default semaphore count for parallel crawling.

Bug Fixes

• Fixed an issue where the HTML content could be empty after processing.

Examples

• Added new example crawl_with_user_simulation() demonstrating the use of user simulation and navigator override features.

Developer Notes

• Refactored code for better maintainability and readability.
• Updated browser launch arguments for improved compatibility and performance.

1. Improved Crawling Control

• New Hook: Added before_retrieve_html hook in AsyncPlaywrightCrawlerStrategy.
• Delayed HTML Retrieval: Introduced delay_before_return_html parameter to allow waiting before retrieving HTML content.
  • Useful for pages with delayed content loading.
• Flexible Timeout: smart_wait function now uses page_timeout (default 60 seconds) instead of a fixed 30-second timeout.
  • Provides better handling for slow-loading pages.
• How to use: Set page_timeout=your_desired_timeout (in milliseconds) when calling crawler.arun().

2. Browser Type Selection

• Added support for different browser types (Chromium, Firefox, WebKit).
• Users can now specify the browser type when initializing AsyncWebCrawler.
• How to use: Set browser_type="firefox" or browser_type="webkit" when initializing AsyncWebCrawler.

3. Screenshot Capture

• Added ability to capture screenshots during crawling.
• Useful for debugging and content verification.
• How to use: Set screenshot=True when calling crawler.arun().

4. Enhanced LLM Extraction Strategy

• Added support for multiple LLM providers (OpenAI, Hugging Face, Ollama).
• Custom Arguments: Added support for passing extra arguments to LLM providers via extra_args parameter.
• Custom Headers: Users can now pass custom headers to the extraction strategy.
• How to use: Specify the desired provider and custom arguments when using LLMExtractionStrategy.

5. iframe Content Extraction

• New feature to process and extract content from iframes.
• How to use: Set process_iframes=True in the crawl method.

6. Delayed Content Retrieval

• Introduced get_delayed_content method in AsyncCrawlResponse.
• Allows retrieval of content after a specified delay, useful for dynamically loaded content.
• How to use: Access result.get_delayed_content(delay_in_seconds) after crawling.

Improvements and Optimizations

1. AsyncWebCrawler Enhancements

• Flexible Initialization: Now accepts arbitrary keyword arguments, passed directly to the crawler strategy.
• Allows for more customized setups.

2. Image Processing Optimization

• Enhanced image handling in WebScrapingStrategy.
• Added filtering for small, invisible, or irrelevant images.
• Improved image scoring system for better content relevance.
• Implemented JavaScript-based image dimension updating for more accurate representation.

3. Database Schema Auto-updates

• Automatic database schema updates ensure compatibility with the latest version.

4. Enhanced Error Handling and Logging

• Improved error messages and logging for easier debugging.

5. Content Extraction Refinements

• Refined HTML sanitization process.
• Improved handling of base64 encoded images.
• Enhanced Markdown conversion process.
• Optimized content extraction algorithms.

6. Utility Function Enhancements

• perform_completion_with_backoff function now supports additional arguments for more customized API calls to LLM providers.

Bug Fixes

• Fixed an issue where image tags were being prematurely removed during content extraction.

Examples and Documentation

• Updated quickstart_async.py with examples of:
  • Using custom headers in LLM extraction.
  • Different LLM provider usage (OpenAI, Hugging Face, Ollama).
  • Custom browser type usage.

Developer Notes

• Refactored code for better maintainability, flexibility, and performance.
• Enhanced type hinting throughout the codebase for improved development experience.
• Expanded error handling for more robust operation.

These updates significantly enhance the flexibility, accuracy, and robustness of crawl4ai, providing users with more control and options for their web crawling and content extraction tasks.

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

• Implement smart_wait function in AsyncPlaywrightCrawlerStrategy
• Add screenshot support to AsyncCrawlResponse and AsyncWebCrawler
• Improve error handling and timeout management in crawling process
• Fix typo in CrawlResult model (responser_headers -> response_headers)