NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #3874 most downloaded on PyPI
Crawlee for Python
Latest stable
1.11.2
Last release today
17 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
215 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Add use to Router for middleware support with pre-handler execution (#1857) (23d7d6c) by @Mantisus
use to Router for middleware support with pre-handler execution (#1857) (23d7d6c) by @MantisusBrowserPool (#1879) (00ffb7e) by @MantisusStagehandCrawler with AI-powered browser automation (#1854) (da84db1) by @MantisusBrowserPool default operation_timeout to 60 seconds (#1877) (38e7dd2) by @vdusekRequestQueue.purge is unsupported (#1883) (cd15dce) by @vdusekYour coding agent can read these notes before it upgrades. Set up the MCP server →
use to Router for middleware support with pre-handler execution (#1857) (23d7d6c) by @Mantisus, closes #1742BrowserPool (#1879) (00ffb7e) by @Mantisus, closes #1741StagehandCrawler with AI-powered browser automation (#1854) (da84db1) by @Mantisus, closes #1738BrowserPool default operation_timeout to 60 seconds (#1877) (38e7dd2) by @vdusekRequestQueue.purge is unsupported (#1883) (cd15dce) by @vdusekNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fix potential deadlocks in SitemapRequestLoader and RequestManagerTandem (#1843) (6226d93) by @Mantisus
SitemapRequestLoader and RequestManagerTandem (#1843) (6226d93) by @MantisusRedisStorageClient and SqlStorageClient (#1838) (b80f562) by @MantisusStorageInstanceManager cache eviction (#1855) (983f14f) by @janbucharforefront flag on RequestQueue retry path (#1861) (dc1073a) by @vdusekNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
file-system: Reclaim orphaned in-progress requests on RQ recovery (#1825) (e86794a) by @vdusek
EventManager shutdown when multiple crawlers share it (#1810) (2efb668) by @Mantisusconnection_string in SqlStorageClient (#1837) (8b53e27) by @MantisusSharedTimeout to post-navigation hooks (#1839) (88bd05a) by @vdusekEventManager shutdown when multiple crawlers share it (#1810) (2efb668) by @Mantisus, closes #1805, #1808connection_string in SqlStorageClient (#1837) (8b53e27) by @MantisusSharedTimeout to post-navigation hooks (#1839) (88bd05a) by @vdusekNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Handle invalid URLs in RequestList (#1803) (0b2e3fc) by @Mantisus
RequestList (#1803) (0b2e3fc) by @Mantisususage_count increment in Session.retire() (#1816) (c40d411) by @vdusekRequestList (#1803) (0b2e3fc) by @Mantisus, closes #1802usage_count increment in Session.retire() (#1816) (c40d411) by @vdusekNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Allow non-href links extract & enqueue (#1781) (6db365d) by @kozlice
post_navigation_hooks to crawlers (#1795) (38ceda6) by @MantisusBrowserPool (#1791) (6f2ac13) by @MantisusBrowserType and CrawleePage (#1798) (b50b9f2) by @Mantisususe_state in BasicCrawler (#1799) (d121873) by @MantisusNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Use specialized Playwright docker images in templates (#1757) (747c0cf) by @Pijukatel
discover_valid_sitemaps utility (#1777) (872447b) by @MantisusBrowserPool (#1703) (70309d9) by @vdusek max_requests_per_crawl excluding failed requests (#1766) (d6bb0b4) by @PijukatelAPIResponse body for send_request (#1771) (29d301b) by @kozliceNone from add_request when storage client fails to enqueue request (#1775) (944753a) by @MantisusPlaywrightBrowserController (#1778) (4487543) by @Pijukateldiscover_valid_sitemaps utility (#1777) (872447b) by @Mantisus, closes #1740BrowserPool (#1703) (70309d9) by @vdusek max_requests_per_crawl excluding failed requests (#1766) (d6bb0b4) by @Pijukatel, closes #1765APIResponse body for send_request (#1771) (29d301b) by @kozliceNone from add_request when storage client fails to enqueue request (#1775) (944753a) by @MantisusPlaywrightBrowserController (#1778) (4487543) by @Pijukatel, closes #1776Dynamic memory snapshots (#1715) (568a7b1) by @Pijukatel
MySQL and MariaDB support for SqlStorageClient (#1749) (202b500) by @MantisusServiceLocator (#1746) (4163413) by @janbucharPlaywrightCrawler unintentionally setting the global configuration (#1747) (fa58438) by @PijukatelSnapshotter handling of out of order samples (#1735) (387c712) by @PijukatelMySQL and MariaDB support for SqlStorageClient (#1749) (202b500) by @MantisusServiceLocator (#1746) (4163413) by @janbucharPlaywrightCrawler unintentionally setting the global configuration (#1747) (fa58438) by @PijukatelSnapshotter handling of out of order samples (#1735) (387c712) by @Pijukatel, closes #1734Use max() instead of min() for request_max_duration statistic (#1701) (85c4335) by @vdusek
max() instead of min() for request_max_duration statistic (#1701) (85c4335) by @vdusekblock_requests (#1702) (fcf9adb) by @vdusekNone values for user_data in Request (#1707) (3c575bc) by @Mantisusmax_open_pages_per_browser limit for PlaywrightBrowserController on concurrent new_page calls (#1712) (2e5534b) by @MantisusReset all counter in metadata with purge for RequestQueue (#1686) (ee09260) by @Mantisus
purge for RequestQueue (#1686) (ee09260) by @Mantisushttp3=False for ImpitHttpClient (#1685) (3f390f6) by @Mantisusget_request from permanently blocking requests (#1684) (da416f9) by @Mirza-Samad-Ahmed-Baigpurge for RequestQueue (#1686) (ee09260) by @Mantisus, closes #1682http3=False for ImpitHttpClient (#1685) (3f390f6) by @Mantisus, closes #1683get_request from permanently blocking requests (#1684) (da416f9) by @Mirza-Samad-Ahmed-BaigExpose AdaptivePlaywrightCrawlerStatisticState for AdaptivePlaywrightCrawler (#1635) (1bb4bcb) by @Mantisus
AdaptivePlaywrightCrawlerStatisticState for AdaptivePlaywrightCrawler (#1635) (1bb4bcb) by @MantisusAdaptivePlaywrightCrawler on initialization with a custom parser (#1637) (bff7260) by @Mantisus, closes #1630EnqueueLinksFunction with limit (#1674) (71d7867) by @Mantisus, closes #1673Fix short error summary (#1605) (b751208) by @Pijukatel
Request fields (#1603) (ae6d86b) by @Mantisusenqueue_strategy after redirects in enqueue_links (#1607) (700df91) by @Mantisus, closes #1606Request from partial mutations on request handler failure (#1585) (a69caf8) by @Mantisus, closes #1514Add additional kwargs to Crawler's export_data (#1597) (5977f37) by @vdusek
Unify separators in unique_key construction (#1569) (af46a37) by @vdusek
unique_key construction (#1569) (af46a37) by @vduseksame-domain strategy ignoring public suffix (#1572) (3d018b2) by @PijukatelFailedRequestHandler and ErrorHandler (#1570) (b830019) by @PijukatelFileSystemStorageClient on systems without UTF-8 default encoding (#1580) (f179f86) by @Mantisus<base> when enqueuing (#1590) (de517a1) by @Mantisusunique_key construction (#1569) (af46a37) by @vdusek, closes #1512same-domain strategy ignoring public suffix (#1572) (3d018b2) by @Pijukatel, closes #1571FailedRequestHandler and ErrorHandler (#1570) (b830019) by @Pijukatel, closes #1532FileSystemStorageClient on systems without UTF-8 default encoding (#1580) (f179f86) by @Mantisus, closes #1579<base> when enqueuing (#1590) (de517a1) by @Mantisus, closes #1589Add chrome BrowserType for PlaywrightCrawler to use the Chrome browser (#1487) (b06937b) by @Mantisus
chrome BrowserType for PlaywrightCrawler to use the Chrome browser (#1487) (b06937b) by @MantisusRedisStorageClient based on Redis v8.0+ (#1406) (d08d13d) by @Mantisustransform_request_function parameter for SitemapRequestLoader (#1525) (dc90127) by @Mantisusrequest_queue_records table for SqlRequestQueueClient (#1527) (6509534) by @MantisusRobotsTxtFile.load (#1524) (596a311) by @Mantisuscrawler_runtime not being updated during run and only in the end (#1540) (0d6c3f6) by @PijukatelEventManager context (#1562) (6a44f17) by @Pijukatelchrome BrowserType for PlaywrightCrawler to use the Chrome browser (#1487) (b06937b) by @Mantisus, closes #1071RedisStorageClient based on Redis v8.0+ (#1406) (d08d13d) by @Mantisustransform_request_function parameter for SitemapRequestLoader (#1525) (dc90127) by @Mantisusrequest_queue_records table for SqlRequestQueueClient (#1527) (6509534) by @Mantisus, closes #1526RobotsTxtFile.load (#1524) (596a311) by @Mantisuscrawler_runtime not being updated during run and only in the end (#1540) (0d6c3f6) by @Pijukatel, closes #1541EventManager context (#1562) (6a44f17) by @Pijukatel, closes #1560Respect enqueue_strategy in enqueue_links (#1505) (6ee04bc) by @Mantisus
enqueue_strategy in enqueue_links (#1505) (6ee04bc) by @Mantisusrobots.txt (#1502) (3273da5) by @MantisusSqlStorageClient and AdaptivePlaywrightCrawler (#1496) (ce172c4) by @MantisusBasicCrawler statistics persistence (#1490) (1eb1c19) by @PijukatelAdaptivePlaywrightCrawler after isolated processing in SubCrawler (#1488) (62b7c70) by @Mantisusenqueue_strategy in enqueue_links (#1505) (6ee04bc) by @Mantisus, closes #1504robots.txt (#1502) (3273da5) by @Mantisus, closes #1499SqlStorageClient and AdaptivePlaywrightCrawler (#1496) (ce172c4) by @Mantisus, closes #1495BasicCrawler statistics persistence (#1490) (1eb1c19) by @Pijukatel, closes #1501AdaptivePlaywrightCrawler after isolated processing in SubCrawler (#1488) (62b7c70) by @Mantisus, closes #1483Add support for Pydantic v2.12 (#1471) (35c1108) by @Mantisus
reclaim_request in SqlRequestQueueClient to correctly update the request state (#1486) (1502469) by @MantisusKeyValueStore.auto_saved_value failing in some scenarios (#1438) (b35dee7) by @Pijukatelreclaim_request in SqlRequestQueueClient to correctly update the request state (#1486) (1502469) by @Mantisus, closes #1484KeyValueStore.auto_saved_value failing in some scenarios (#1438) (b35dee7) by @Pijukatel, closes #1354Use Self type in the open() method of storage clients (#1462) (4ec6f6c) by @janbuchar
Fix memory leak in PlaywrightCrawler on browser context creation (#1446) (bb181e5) by @Pijukatel
PlaywrightCrawler on browser context creation (#1446) (bb181e5) by @Pijukatel, closes #1443Check out the Release blog post for more details.
SitemapRequestLoader (#1169) (66599f8) by @Mantisusstatus_message_callback parameter for customization (#1265) (b992fb2) by @MantisusCrawlee CLI help text (#1297) (afbe10f) by @PijukatelOpenTelemetry instrumentation (#1255) (a92d8b3) by @PijukatelImpitHttpClient http-client client using the impit library (#1151) (0d0d268) by @MantisusPlaywrightPersistentBrowser class (#1314) (b5fa955) by @Mantisusimpit option for Crawlee CLI (#1312) (508d7ce) by @MantisusRequestList state (#1274) (cc68014) by @janbucharDefaultRenderingTypePredictor state (#1340) (fad4c25) by @MantisusSitemapRequestLoader state (#1347) (27ef9ad) by @Mantisusadd_requests and enqueue_links methods (#1413) (1cae2bc) by @MantisusSqlStorageClient based on sqlalchemy v2+ (#1339) (07c75a0) by @Mantisusstats field from RequestQueueMetadata (#1331) (0a63bef) by @vdusektimeout for stream method in ImpitHttpClient (#1352) (54b693b) by @MantisusMemoryRequestQueueClient (#1384) (3af326c) by @MantisusRequestQueueState for FileSystemRequestQueueClient in default KVS (#1411) (6ee60a0) by @MantisusBrowserType literal into two different literals based on context (#1070) (72b5698) by @PijukatelHttpResponse.read from sync to async (#1296) (83fa8a4) by @MantisusHttpxHttpClient with ImpitHttpClient as default HTTP client (#1307) (c803a97) by @MantisusDataset unwind parameter to accept list of strings (#1357) (862a203) by @vdusekRequest.id field (#1366) (32f3580) by @PijukatelSitemapRequestLoader (#1169) (66599f8) by @Mantisus, closes #1161status_message_callback parameter for customization (#1265) (b992fb2) by @Mantisus, closes #96Crawlee CLI help text (#1297) (afbe10f) by @Pijukatel, closes #1295OpenTelemetry instrumentation (#1255) (a92d8b3) by @Pijukatel, closes #1254ImpitHttpClient http-client client using the impit library (#1151) (0d0d268) by @MantisusPlaywrightPersistentBrowser class (#1314) (b5fa955) by @Mantisusimpit option for Crawlee CLI (#1312) (508d7ce) by @MantisusRequestList state (#1274) (cc68014) by @janbuchar, closes #99DefaultRenderingTypePredictor state (#1340) (fad4c25) by @Mantisus, closes #1272SitemapRequestLoader state (#1347) (27ef9ad) by @Mantisus, closes #1269add_requests and enqueue_links methods (#1413) (1cae2bc) by @Mantisus, closes #1402SqlStorageClient based on sqlalchemy v2+ (#1339) (07c75a0) by @Mantisus, closes #307stats field from RequestQueueMetadata (#1331) (0a63bef) by @vdusektimeout for stream method in ImpitHttpClient (#1352) (54b693b) by @MantisusMemoryRequestQueueClient (#1384) (3af326c) by @Mantisus, closes #1383RequestQueueState for FileSystemRequestQueueClient in default KVS (#1411) (6ee60a0) by @Mantisus, closes #1410BrowserType literal into two different literals based on context (#1070) (72b5698) by @PijukatelHttpResponse.read from sync to async (#1296) (83fa8a4) by @MantisusHttpxHttpClient with ImpitHttpClient as default HTTP client (#1307) (c803a97) by @Mantisus, closes #1079Dataset unwind parameter to accept list of strings (#1357) (862a203) by @vdusekRequest.id field (#1366) (32f3580) by @Pijukatel, closes #1358Nothing published for this version
Add retire_browser_after_page_count parameter for BrowserPool (#1266) (603aa2b) by @Mantisus
perf_counter_ns for request duration tracking (#1260) (9e92f6b) by @Pijukatel, closes #1256Add stream method for HttpClient (#1241) (95c68b0) by @Mantisus
ClientSnapshot overload calculation (#1228) (a4fc1b6) by @Pijukatel, closes #1207PSS instead of RSS to estimate children process memory usage on Linux (#1210) (436032f) by @Pijukatel, closes #1206Allow config change on PlaywrightCrawler (#1186) (f17bf31) by @mylank
PlaywrightCrawler (#1186) (f17bf31) by @mylankpayload to SendRequestFunction to support POST request (#1202) (e7449f2) by @MantisusWindowsSelectorEventLoopPolicy only for curl-impersonate template without playwright (#1209) (f3b839f) by @MantisusPlaywrightCrawler (#1208) (dbb9f44) by @MantisusEnqueueLinksKwargs for extract_links function (#1213) (c9907d6) by @MantisusPlaywrightCrawler (#1186) (f17bf31) by @mylank, closes #1185payload to SendRequestFunction to support POST request (#1202) (e7449f2) by @MantisusWindowsSelectorEventLoopPolicy only for curl-impersonate template without playwright (#1209) (f3b839f) by @Mantisus, closes #1204PlaywrightCrawler (#1208) (dbb9f44) by @Mantisus, closes #1201EnqueueLinksKwargs for extract_links function (#1213) (c9907d6) by @Mantisus, closes #1212Add an internal HttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context (#1134) (e794f49) by
HttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context (#1134) (e794f49) by @Mantisuson_skipped_request decorator, to process links skipped according to robots.txt rules (#1166) (bd16f14) by @MantisusHttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context (#1134) (e794f49) by @Mantisus, closes #928on_skipped_request decorator, to process links skipped according to robots.txt rules (#1166) (bd16f14) by @Mantisus, closes #1160Handle unprocessed requests in add_requests_batched (#1159) (7851175) by @Pijukatel
add_requests_batched (#1159) (7851175) by @Pijukatelrespect_robots_txt_file option (#1162) (c23f365) by @MantisusUnprocessedRequest to match actual data (#1155) (a15a1f3) by @PijukatelSessionCookies and the handler is executed for PlaywrightCrawler (#1163) (82ff69a) by @Mantisusfailed_request_handler for SessionError when session rotation count exceeds maximum (#1147) (b3637b6) by @Mantisusadd_requests_batched (#1159) (7851175) by @Pijukatel, closes #456respect_robots_txt_file option (#1162) (c23f365) by @MantisusUnprocessedRequest to match actual data (#1155) (a15a1f3) by @Pijukatel, closes #1150SessionCookies and the handler is executed for PlaywrightCrawler (#1163) (82ff69a) by @Mantisusfailed_request_handler for SessionError when session rotation count exceeds maximum (#1147) (b3637b6) by @MantisusAdd ErrorSnapshotter to ErrorTracker (#1125) (9666092) by @Pijukatel
ErrorSnapshotter to ErrorTracker (#1125) (9666092) by @Pijukatel, closes #151Add statistics_log_format parameter to BasicCrawler (#1061) (635ae4a) by @Mantisus
Update to browserforge workaround (#1075) (2378cf8) by @Pijukatel
Add a check thread before set add_signal_handler (#1068) (6983bda) by @Mantisus
Add project template with uv package manager (#1057) (9ec06e5) by @Mantisus
uv package manager (#1057) (9ec06e5) by @Mantisus, closes #1053PlaywrightCrawler by default (#1060) (09cec53) by @Pijukatel, closes #1054Extend ErrorTracker with error grouping (#1014) (561de5c) by @Pijukatel
Add browserforge to mandatory dependencies (#1044) (ddfbde8) by @Pijukatel
c487d89
Pijukatel
One column per month.