pub.dev

ai_webscraper

0.2.1SamsonTobi/ai_webscraper ↗New releases in your daily digest, changelog included.

A comprehensive AI-powered web scraper for Dart with OpenAI and Google Gemini integration, featuring response caching and enhanced logging

Last release 1 years ago
11 Aug 2025
Too new to tell
only 1 release windows
Nearly every release is documented
notes for 5 of 5 stable releases
Nothing withdrawn
no release was ever pulled
1 years old
5 releases · first in 2025

Release timeline

5 releases since 2025

20252026

One column per month.

Releases

0.2.111 Aug 2025
### Added - Solved Dart Analysis Issues
Added

Solved Dart Analysis Issues
Open source →
0.2.010 Aug 2025
Response Caching System: Comprehensive caching with in-memory and file-based storage
Added

Response Caching System: Comprehensive caching with in-memory and file-based storage

Configurable cache expiration and size limits

SHA-256 based cache keys for reliable deduplication

Automatic cache cleanup and persistence

Cache statistics and debugging support

Enhanced Logging: Detailed logging throughout the scraping pipeline

Raw AI response logging for debugging

HTML content preview logging

Schema processing and field normalization logs

Scoped loggers for different components

Gemini ResponseSchema Support: Proper implementation of Gemini's structured output

Native responseSchema integration instead of text-based prompts

Support for complex types: objects, typed arrays (array<string>, array<number>)

Required field marking with type! syntax

Automatic normalization of string "null" artifacts

Improved JavaScript Scraping: Enhanced dynamic content handling

Better React/SPA content detection and waiting

Network idle waiting for complete page loads

Comprehensive page data extraction including structured data

Debug helpers for troubleshooting dynamic content issues

Enhanced

AI Client Architecture: More robust and feature-rich AI integration

Centralized response caching across all AI providers

Detailed response parsing with fallback extraction

Better error handling and provider-specific exceptions

Usage metadata tracking (where supported)

Schema Validation: Extended type support and validation

Object type support for complex nested data

Typed array definitions for better structured output

Email, URL, and date/datetime specialized types

Flexible required field configuration

Web Scraper Core: Improved reliability and debugging

Enhanced fallback logic between HTTP and JavaScript scraping

Better content formatting for AI processing

Comprehensive extraction data formatting

Improved error reporting and context

Fixed

Gemini API object schema validation errors (empty properties requirement)

String "null" artifacts in AI responses converted to proper null values

JavaScript scraping timeout and waiting issues for dynamic content

Cache file corruption handling and error recovery

Schema normalization ensuring all fields are present in responses

Dependencies

Added crypto package for secure cache key generation

0.1.03 Aug 2025

Initial release of AI WebScraper package

Added

Initial release of AI WebScraper package
Support for OpenAI GPT integration
Support for Google Gemini integration
Basic web scraping with HTTP requests
JavaScript rendering with Puppeteer fallback
Schema-based data extraction
Batch processing with configurable concurrency
Comprehensive error handling
Type-safe result objects
Automatic fallback from HTTP to JavaScript scraping
Timeout configuration
Basic logging and debugging support

Core Features

AIWebScraper main class with provider selection
ScrapingResult for consistent result handling
AIProvider enum for provider selection
SchemaType enum for data type definitions
Abstract AI client architecture for extensibility
Factory pattern for AI client creation
Batch processing utilities with semaphore-based concurrency control
Schema validation and URL validation utilities

AI Providers

OpenAI GPT-3.5-turbo integration with Chat Completions API
Google Gemini Pro integration with generative AI capabilities
Structured JSON response handling for both providers
Provider-specific error handling and retry logic

Web Scraping

HTTP-based scraping with configurable timeouts
HTML content parsing and extraction
JavaScript rendering support via Puppeteer
Automatic fallback between scraping methods
Content extraction utilities for clean data processing

Testing

Comprehensive unit test coverage
Integration tests for end-to-end workflows
Mock HTTP server for controlled testing
AI API mocking for reliable testing
Performance and concurrency testing

Documentation

Comprehensive README with usage examples
API documentation with detailed method descriptions
Example implementations for common use cases
Configuration and setup guides
Troubleshooting and best practices

Open source →

ai_webscraper

Release timeline

Releases

Added

Added

Enhanced

Fixed

Dependencies

Fixed

Fixed

Enhanced

Added

Core Features

AI Providers

Web Scraping

Testing

Documentation