NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #1779 most downloaded on PyPI
A high-level Web Crawling and Web Scraping framework
Latest stable
2.20.0
Last release 10 days ago
06 Nov 2026
Release timing varies
gaps range from 2 weeks to 5 months
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
17 years old
116 releases · first in 2009
Drag across the ruler to compare, or tap a release to open it.
- Scrapy contracts: python2.6 compat ( commit a4a9199 )
Scrapy contracts: python2.6 compat ( commit a4a9199 )
Scrapy contracts verbose option ( commit ec41673 )
proper unittest-like output for Scrapy contracts ( commit 86635e4 )
added open_in_browser to debugging doc ( commit c9b690d )
removed reference to global Scrapy stats from settings doc ( commit dd55067 )
Fix SpiderState bug in Windows platforms ( commit 58998f4 )
- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release ( commit 8c780fd )
fixed LogStats extension, which got broken after a wrong merge before the 0.16 release ( commit 8c780fd )
better backward compatibility for scrapy.conf.settings ( commit 3403089 )
extended documentation on how to access crawler stats from extensions ( commit c4da0b5 )
removed .hgtags (no longer needed now that Scrapy uses git) ( commit d52c188 )
fix dashes under rst headers ( commit fa4f7f9 )
set release date for 0.16.0 in news ( commit e292246 )
Your coding agent can read these notes before it upgrades. Set up the MCP server →
- added Spiders Contracts , a mechanism for testing spiders in a formal/reproducible way
Scrapy changes:
added Spiders Contracts , a mechanism for testing spiders in a formal/reproducible way
added options -o and -t to the runspider command
documented AutoThrottle extension and added to extensions installed by default. You still need to enable it with AUTOTHROTTLE_ENABLED
major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals ( stats_spider_opened , etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.
added a process_start_requests() method to spider middlewares
dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.
dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.
documented Core API
lxml is now the default selectors backend instead of libxml2
ported FormRequest.from_response() to use lxml instead of ClientForm
removed modules: scrapy.xlib.BeautifulSoup and scrapy.xlib.ClientForm
SitemapSpider: added support for sitemap urls ending in .xml and .xml.gz, even if they advertise a wrong content type ( commit 10ed28b )
StackTraceDump extension: also dump trackref live references ( commit fe2ce93 )
nested items now fully supported in JSON and JSONLines exporters
added cookiejar Request meta key to support multiple cookie sessions per spider
decoupled encoding detection code to w3lib.encoding , and ported Scrapy code to use that module
dropped support for Python 2.5. See https://www.zyte.com/blog/scrapy-0-15-dropping-support-for-python-2-5/
dropped support for Twisted 2.5
added REFERER_ENABLED setting, to control referer middleware
changed default user agent to: Scrapy/VERSION (+http://scrapy.org)
removed (undocumented) HTMLImageLinkExtractor class from scrapy.contrib.linkextractors.image
removed per-spider settings (to be replaced by instantiating multiple crawler objects)
USER_AGENT spider attribute will no longer work, use user_agent attribute instead
DOWNLOAD_TIMEOUT spider attribute will no longer work, use download_timeout attribute instead
removed ENCODING_ALIASES setting, as encoding auto-detection has been moved to the w3lib library
promoted DjangoItem to main contrib
LogFormatter method now return dicts(instead of strings) to support lazy formatting ( #164 , commit dcef7b0 )
downloader handlers ( DOWNLOAD_HANDLERS setting) now receive settings as the first argument of the init method
replaced memory usage accounting with (more portable) resource module, removed scrapy.utils.memory module
removed signal: scrapy.mail.mail_sent
removed TRACK_REFS setting, now trackrefs is always enabled
DBM is now the default storage backend for HTTP cache middleware
number of log messages (per level) are now tracked through Scrapy stats (stat name: log_count/LEVEL )
number received responses are now tracked through Scrapy stats (stat name: response_received_count )
removed scrapy.log.started attribute
- added precise to supported Ubuntu distros ( commit b7e46df )
added precise to supported Ubuntu distros ( commit b7e46df )
fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion . also removed no longer supported ‘run’ command from extras/scrapy-ws.py ( commit 340fbdb )
meta tag attributes for content-type http equiv can be in any order. #123 ( commit 0cb68af )
replace “import Image” by more standard “from PIL import Image”. closes #88 ( commit 4d17048 )
return trial status as bin/runtests.sh exit value. #118 ( commit b7b2e7f )
- forgot to include pydispatch license. #118 ( commit fd85f9c )
forgot to include pydispatch license. #118 ( commit fd85f9c )
include egg files used by testsuite in source distribution. #118 ( commit c897793 )
update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 ( commit 2548dcc )
added note to docs/topics/firebug.rst about google directory being shut down ( commit 668e352 )
don’t discard slot when empty, just save in another dict in order to recycle if needed again. ( commit 8e9f607 )
do not fail handling unicode xpaths in libxml2 backed selectors ( commit b830e95 )
fixed minor mistake in Request objects documentation ( commit bf3c9ee )
fixed minor defect in link extractors documentation ( commit ba14f38 )
removed some obsolete remaining code related to sqlite support in Scrapy ( commit 0665175 )
- move buffer pointing to start of file before computing checksum. refs #92 ( commit 6a5bef2 )
move buffer pointing to start of file before computing checksum. refs #92 ( commit 6a5bef2 )
Compute image checksum before persisting images. closes #92 ( commit 9817df1 )
remove leaking references in cached failures ( commit 673a120 )
fixed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) ( commit 11133e9 )
fixed struct.error on http compression middleware. closes #87 ( commit 1423140 )
ajax crawling wasn’t expanding for unicode urls ( commit 0de3fb4 )
Catch start_requests() iterator errors. refs #83 ( commit 454a21d )
Speed-up libxml2 XPathSelector ( commit 2fbd662 )
updated versioning doc according to recent changes ( commit 0a070f5 )
scrapyd: fixed documentation link ( commit 2b4e4c3 )
extras/makedeb.py: no longer obtaining version from git ( commit caffe0e )
- extras/makedeb.py: no longer obtaining version from git ( commit caffe0e )
extras/makedeb.py: no longer obtaining version from git ( commit caffe0e )
bumped version to 0.14.1 ( commit 6cb9e1c )
fixed reference to tutorial directory ( commit 4b86bd6 )
doc: removed duplicated callback argument from Request.replace() ( commit 1aeccdd )
fixed formatting of scrapyd doc ( commit 8bf19e6 )
Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension ( commit 14a8e6e )
added comment about why we disable ssl on boto images upload ( commit 5223575 )
SSL handshaking hangs when doing too many parallel connections to S3 ( commit 63d583d )
change tutorial to follow changes on dmoz site ( commit bcb3198 )
Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 ( commit 98f3f87 )
allow spider to set autothrottle max concurrency ( commit 175a4b5 )
Nothing published for this version
Nothing published for this version
- Added web service for controlling Scrapy process (this also deprecates the web console. ( r2053 | #167)
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
Added SMTP-AUTH support to scrapy.mail
New settings added: MAIL_USER , MAIL_PASS ( r2065 | #149)
Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy ( r2039 )
Added web service for controlling Scrapy process (this also deprecates the web console. ( r2053 | #167)
Support for running Scrapy as a service, for production systems ( r1988 , r2054 , r2055 , r2056 , r2057 | #168)
Added wrapper induction library (documentation only available in source code for now). ( r2011 )
Simplified and improved response encoding support ( r1961 , r1969 )
Added LOG_ENCODING setting ( r1956 , documentation available)
Added RANDOMIZE_DOWNLOAD_DELAY setting (enabled by default) ( r1923 , doc available)
MailSender is no longer IO-blocking ( r1955 | #146)
Linkextractors and new Crawlspider now handle relative base tag urls ( r1960 | #148)
Several improvements to Item Loaders and processors ( r2022 , r2023 , r2024 , r2025 , r2026 , r2027 , r2028 , r2029 , r2030 )
Added support for adding variables to telnet console ( r2047 | #165)
Support for requests without callbacks ( r2050 | #166)
Change Spider.domain_name to Spider.name (SEP-012, r1975 )
Response.encoding is now the detected encoding ( r1961 )
HttpErrorMiddleware now returns None or raises an exception ( r2006 | #157)
scrapy.command modules relocation ( r2035 , r2036 , r2037 )
Added ExecutionQueue for feeding spiders to scrape ( r2034 )
Removed ExecutionEngine singleton ( r2039 )
Ported S3ImagesStore (images pipeline) to use boto and threads ( r2033 )
Moved module: scrapy.management.telnet to scrapy.telnet ( r2047 )
Changed default SCHEDULER_ORDER to DFO ( r1939 )
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
Added SMTP-AUTH support to scrapy.mail
New settings added: MAIL_USER, MAIL_PASS (2065 | #149)
Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy (2039)
Added web service for controlling Scrapy process (this also deprecates the web console. (2053 | #167)
Support for running Scrapy as a service, for production systems (1988, 2054, 2055, 2056, 2057 | #168)
Added wrapper induction library (documentation only available in source code for now). (2011)
Simplified and improved response encoding support (1961, 1969)
Added LOG_ENCODING setting (1956, documentation available)
Added RANDOMIZE_DOWNLOAD_DELAY setting (enabled by default) (1923, doc available)
MailSender is no longer IO-blocking (1955 | #146)
Linkextractors and new Crawlspider now handle relative base tag urls (1960 | #148)
Several improvements to Item Loaders and processors (2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030)
Added support for adding variables to telnet console (2047 | #165)
Support for requests without callbacks (2050 | #166)
Change Spider.domain_name to Spider.name (SEP-012, 1975)
Response.encoding is now the detected encoding (1961)
HttpErrorMiddleware now returns None or raises an exception (2006 | #157)
scrapy.command modules relocation (2035, 2036, 2037)
Added ExecutionQueue for feeding spiders to scrape (2034)
Removed ExecutionEngine singleton (2039)
Ported S3ImagesStore (images pipeline) to use boto and threads (2033)
Moved module: scrapy.management.telnet to scrapy.telnet (2047)
Changed default SCHEDULER_ORDER to DFO (1939)
- Removed deprecated scrapy.item.ScrapedItem class - use scrapy.item.Item instead ( r1838 )
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
Added DEFAULT_RESPONSE_ENCODING setting ( r1809 )
Added dont_click argument to FormRequest.from_response() method ( r1813 , r1816 )
Added clickdata argument to FormRequest.from_response() method ( r1802 , r1803 )
Added support for HTTP proxies ( HttpProxyMiddleware ) ( r1781 , r1785 )
Offsite spider middleware now logs messages when filtering out requests ( r1841 )
Changed scrapy.utils.response.get_meta_refresh() signature ( r1804 )
Removed deprecated scrapy.item.ScrapedItem class - use scrapy.item.Item instead ( r1838 )
Removed deprecated scrapy.xpath module - use scrapy.selector instead. ( r1836 )
Removed deprecated core.signals.domain_open signal - use core.signals.domain_opened instead ( r1822 )
Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the spider argument and pass spider references. If you really want to pass a string, use the component argument instead.
Changed core signals domain_opened , domain_closed , domain_idle
The domain argument of process_item() item pipeline method was changed to spider , the new signature is: process_item(spider, item) ( r1827 | #105)
To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain .
StatsCollector was changed to receive spider references (instead of domains) in its methods ( set_value , inc_value , etc).
added StatsCollector.iter_spider_stats() method
removed StatsCollector.list_domains() method
Also, Stats signals were renamed and now pass around spider references (instead of domains). Here’s a summary of the changes:
To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain . spider_stats contains exactly the same data as domain_stats .
CLOSEDOMAIN_TIMEOUT to CLOSESPIDER_TIMEOUT
CLOSEDOMAIN_ITEMCOUNT to CLOSESPIDER_ITEMCOUNT
Removed deprecated SCRAPYSETTINGS_MODULE environment variable - use SCRAPY_SETTINGS_MODULE instead ( r1840 )
Renamed setting: REQUESTS_PER_DOMAIN to CONCURRENT_REQUESTS_PER_SPIDER ( r1830 , r1844 )
Renamed setting: CONCURRENT_DOMAINS to CONCURRENT_SPIDERS ( r1830 )
Refactored HTTP Cache middleware
HTTP Cache middleware has been heavily refactored, retaining the same functionality except for the domain sectorization which was removed. ( r1843 )
Renamed exception: DontCloseDomain to DontCloseSpider ( r1859 | #120)
Renamed extension: DelayedCloseDomain to SpiderCloseDelay ( r1861 | #121)
Removed obsolete scrapy.utils.markup.remove_escape_chars function - use scrapy.utils.markup.replace_escape_chars instead ( r1865 )
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
Added DEFAULT_RESPONSE_ENCODING setting (1809)
Added dont_click argument to FormRequest.from_response() method (1813, 1816)
Added clickdata argument to FormRequest.from_response() method (1802, 1803)
Added support for HTTP proxies (HttpProxyMiddleware) (1781, 1785)
Offsite spider middleware now logs messages when filtering out requests (1841)
Changed scrapy.utils.response.get_meta_refresh() signature (1804)
Removed deprecated scrapy.item.ScrapedItem class - use scrapy.item.Item instead (1838)
Removed deprecated scrapy.xpath module - use scrapy.selector instead. (1836)
Removed deprecated core.signals.domain_open signal - use core.signals.domain_opened instead (1822)
Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the spider argument and pass spider references. If you really want to pass a string, use the component argument instead.
Changed core signals domain_opened, domain_closed, domain_idle
The domain argument of process_item() item pipeline method was changed to spider, the new signature is: process_item(spider, item) (1827 | #105)
To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain.
StatsCollector was changed to receive spider references (instead of domains) in its methods (set_value, inc_value, etc).
added StatsCollector.iter_spider_stats() method
removed StatsCollector.list_domains() method
Also, Stats signals were renamed and now pass around spider references (instead of domains). Here's a summary of the changes:
To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain. spider_stats contains exactly the same data as domain_stats.
CLOSEDOMAIN_TIMEOUT to CLOSESPIDER_TIMEOUT
CLOSEDOMAIN_ITEMCOUNT to CLOSESPIDER_ITEMCOUNT
Removed deprecated SCRAPYSETTINGS_MODULE environment variable - use SCRAPY_SETTINGS_MODULE instead (1840)
Renamed setting: REQUESTS_PER_DOMAIN to CONCURRENT_REQUESTS_PER_SPIDER (1830, 1844)
Renamed setting: CONCURRENT_DOMAINS to CONCURRENT_SPIDERS (1830)
Refactored HTTP Cache middleware
HTTP Cache middleware has been heavily refactored, retaining the same functionality except for the domain sectorization which was removed. (1843 )
Renamed exception: DontCloseDomain to DontCloseSpider (1859 | #120)
Renamed extension: DelayedCloseDomain to SpiderCloseDelay (1861 | #121)
Removed obsolete scrapy.utils.markup.remove_escape_chars function - use scrapy.utils.markup.replace_escape_chars instead (1865)
First release of Scrapy.
One column per quarter.