crawlee

1.7.3 (2026-06-22)

🐛 Bug Fixes

• memory-storage: Apply skip_empty after pagination in MemoryDatasetClient.get_data (#1937) (7e807aa) by @vdusek
• redis: Return all items from Redis dataset get_data with desc=True and limit=None (#1939) (f7cde2e) by @vdusek
• Retry sitemap fetching on error and raise when retries are exhausted (#1943) (76927d7) by @vdusek
• Validate storage name and alias values (#1950) (0cac092) by @vdusek
• Constrain default sitemap loading (#1956) (3fd5ace) by @Pijukatel
• Remove extra sleep in ImpitHttpClient.stream (#1980) (149a6aa) by @Mantisus
• Retry unprocessed requests in RequestQueue.add_request (#1976) (cfee910) by @vdusek
• Gracefully close sitemap stream on SitemapRequestLoader abort (#1979) (202726d) by @Mantisus

1.7.2 (2026-06-04)

🐛 Bug Fixes

• templates: Install pinned playwright into system env in uv Dockerfile (#1922) (0fd0f3a) by @vdusek
• memory-storage: Avoid duplicate processed requests in memory request queue client (#1941) (d343d0e) by @vdusek
• redis: Preserve shared Redis index hashes when dropping a storage (#1942) (4729dd1) by @vdusek
• Do not raise KeyError in parse_sitemap when partial options are provided (#1940) (8ab3f95) by @vdusek
• Reset private state correctly in sitemap parsers (#1938) (7db517a) by @vdusek

1.7.1 (2026-05-26)

🐛 Bug Fixes

• Include sql_mysql in the all extra (#1895) (4023314) by @vdusek
• Update push_data and user_data annotation with JsonSerializable instead of Any (#1889) (662b93b) by @Mantisus
• stagehand: Inject --no-sandbox into Stagehand's Chromium launch when sandbox is disabled (#1906) (041b92a) by @vdusek
• templates: Pin playwright to base image version in uv Dockerfile template (#1904) (8d902c9) by @vdusek

1.7.0 (2026-05-12)

🚀 Features

• Add use to Router for middleware support with pre-handler execution (#1857) (23d7d6c) by @Mantisus
• Add opt-in per-domain request throttling for HTTP 429 backoff (#1762) (c17f4d5) by @MrAliHasan
• Add pre/post launch hooks to BrowserPool (#1879) (00ffb7e) by @Mantisus
• Add StagehandCrawler with AI-powered browser automation (#1854) (da84db1) by @Mantisus
• cli: Add Adaptive and Stagehand crawler templates (#1888) (39b2d24) by @vdusek

🐛 Bug Fixes

• Reject non-http(s) URL schemes in HTTP clients (#1862) (ac66b2a) by @vdusek
• Filter sitemap-derived URLs by enqueue strategy (#1864) (b3db0dc) by @vdusek
• Bump BrowserPool default operation_timeout to 60 seconds (#1877) (38e7dd2) by @vdusek
• redis: Prevent counter corruption from concurrent mark handled in Redis RQ (#1878) (50d70f0) by @Mantisus
• Fall back to drop+recreate when RequestQueue.purge is unsupported (#1883) (cd15dce) by @vdusek

1.6.3 (2026-04-27)

🐛 Bug Fixes

• Fix potential deadlocks in SitemapRequestLoader and RequestManagerTandem (#1843) (6226d93) by @Mantisus
• Add retry logic for RedisStorageClient and SqlStorageClient (#1838) (b80f562) by @Mantisus
• Fix StorageInstanceManager cache eviction (#1855) (983f14f) by @janbuchar
• Report integer count in 'Experiencing problems' status log (#1860) (40170a6) by @vdusek
• Preserve forefront flag on RequestQueue retry path (#1861) (dc1073a) by @vdusek

1.6.2 (2026-04-08)

🐛 Bug Fixes

• file-system: Reclaim orphaned in-progress requests on RQ recovery (#1825) (e86794a) by @vdusek
• Prevent premature EventManager shutdown when multiple crawlers share it (#1810) (2efb668) by @Mantisus
• Apply SQLite optimizations to the custom connection_string in SqlStorageClient (#1837) (8b53e27) by @Mantisus
• Apply SharedTimeout to post-navigation hooks (#1839) (88bd05a) by @vdusek

1.6.1 (2026-03-30)

🐛 Bug Fixes

• Handle invalid URLs in RequestList (#1803) (0b2e3fc) by @Mantisus
• playwright: Filter unsupported context options in persistent browser (#1796) (69ad22e) by @sushant-mutnale
• Remove double usage_count increment in Session.retire() (#1816) (c40d411) by @vdusek
• Defer page object cleanup to make it accessible in error handlers (#1814) (7eeb500) by @janbuchar

⚡ Performance

• Offload BeautifulSoup parsing to a thread via asyncio.to_thread (#1817) (d612ffa) by @vdusek

1.6.0 (2026-03-20)

🚀 Features

• Allow non-href links extract & enqueue (#1781) (6db365d) by @kozlice
• Add post_navigation_hooks to crawlers (#1795) (38ceda6) by @Mantisus
• Add page lifecycle hooks to BrowserPool (#1791) (6f2ac13) by @Mantisus
• Expose BrowserType and CrawleePage (#1798) (b50b9f2) by @Mantisus
• Expose use_state in BasicCrawler (#1799) (d121873) by @Mantisus

🐛 Bug Fixes

• redis: Do not remove handled request data from request queue (#1787) (3008c61) by @kozlice
• redis: Update actual Request state in request queue Redis storage client (#1789) (787231c) by @Mantisus

1.5.0 (2026-03-06)

🚀 Features

• Use specialized Playwright docker images in templates (#1757) (747c0cf) by @Pijukatel
• Add discover_valid_sitemaps utility (#1777) (872447b) by @Mantisus

🐛 Bug Fixes

• Prevent list modification during iteration in BrowserPool (#1703) (70309d9) by @vdusek
• Fix max_requests_per_crawl excluding failed requests (#1766) (d6bb0b4) by @Pijukatel
• playwright: Dispose of APIResponse body for send_request (#1771) (29d301b) by @kozlice
• Return None from add_request when storage client fails to enqueue request (#1775) (944753a) by @Mantisus
• Re-use pre-existing browser context in PlaywrightBrowserController (#1778) (4487543) by @Pijukatel

1.4.0 (2026-02-17)

🚀 Features

• Dynamic memory snapshots (#1715) (568a7b1) by @Pijukatel
• Add MySQL and MariaDB support for SqlStorageClient (#1749) (202b500) by @Mantisus

🐛 Bug Fixes

• Make log levels consistent in ServiceLocator (#1746) (4163413) by @janbuchar
• Fix PlaywrightCrawler unintentionally setting the global configuration (#1747) (fa58438) by @Pijukatel
• Fix Snapshotter handling of out of order samples (#1735) (387c712) by @Pijukatel

⚡ Performance

• Optimize metadata records processing in SqlStorageClient (#1551) (df1347a) by @Mantisus