spatie/crawler

What's Changed

• Fix SitemapUrlParser extracting no URLs from namespaced sitemaps by @witheez in #512

New Contributors

• @witheez made their first contribution in #512

Full Changelog: 9.4.1...9.4.2

What's Changed

• Fix SitemapUrlParser extracting no URLs from namespaced sitemaps by @witheez in https://github.com/spatie/crawler/pull/512

New Contributors

• @witheez made their first contribution in https://github.com/spatie/crawler/pull/512

Full Changelog: https://github.com/spatie/crawler/compare/9.4.1...9.4.2

What's Changed

• Fix Guzzle 8 exception handling for responses and retries by @GrahamCampbell in #510

New Contributors

• @GrahamCampbell made their first contribution in #510

Full Changelog: 9.4.0...9.4.1

What's Changed

• Fix Guzzle 8 exception handling for responses and retries by @GrahamCampbell in https://github.com/spatie/crawler/pull/510

New Contributors

• @GrahamCampbell made their first contribution in https://github.com/spatie/crawler/pull/510

Full Changelog: https://github.com/spatie/crawler/compare/9.4.0...9.4.1

Added support for Guzzle 8 alongside Guzzle 7.

Added support for Guzzle 8 alongside Guzzle 7.

What's Changed

• Skip signal handler registration when pcntl functions are disabled by @poldixd in #508

New Contributors

• @poldixd made their first contribution in #508

Full Changelog: 9.3.1...9.3.2

What's Changed

• Skip signal handler registration when pcntl functions are disabled by @poldixd in https://github.com/spatie/crawler/pull/508

New Contributors

• @poldixd made their first contribution in https://github.com/spatie/crawler/pull/508

Full Changelog: https://github.com/spatie/crawler/compare/9.3.1...9.3.2

What's Changed

• Skip fnmatch for URLs longer than FILENAME_MAX by @mattiasgeniar in #505

Full Changelog: 9.3.0...9.3.1

What's Changed

• Skip fnmatch for URLs longer than FILENAME_MAX by @mattiasgeniar in https://github.com/spatie/crawler/pull/505

Full Changelog: https://github.com/spatie/crawler/compare/9.3.0...9.3.1

What's Changed

• Add a shouldStopCallback hook for graceful external stops by @kissifrot in #504

Full Changelog: 9.2.1...9.3.0

What's Changed

• Add a shouldStopCallback hook for graceful external stops by @kissifrot in https://github.com/spatie/crawler/pull/504

Full Changelog: https://github.com/spatie/crawler/compare/9.2.1...9.3.0

What's Changed

• Close response body streams after processing by @freekmurze in #503

Full Changelog: 9.0.1...9.2.1

What's Changed

• Close response body streams after processing by @freekmurze in https://github.com/spatie/crawler/pull/503

Full Changelog: https://github.com/spatie/crawler/compare/9.0.1...9.2.1

Full Changelog: 9.1.0...9.2.0

Full Changelog: https://github.com/spatie/crawler/compare/9.1.0...9.2.0

• Add urlParser() method to allow setting a custom URL parser

Co-Authored-By: Claude Opus 4.6 (1M context) [email protected]

• Fix styling

Co-authored-by: Claude Opus 4.6 (1M context) [email protected]
Co-authored-by: freekmurze [email protected]

What's fixed

When the crawler reached its time limit, crawl limit, or was interrupted via a signal, any HTTP requests that were still in-flight in the Guzzle pool would eventually fail (e.g. due to their per-request timeout) and get reported to crawlFailed. These URLs were dispatched but never truly crawled, so reporting them as failures was misleading.

The CrawlRequestFailed handler now checks whether the crawler has reached its limits before reporting a failure. If it has, the failure is silently ignored.

Regular request failures (timeouts, connection errors, server errors) that happen during normal crawling are still reported as before.

Fixed

• Fixed allow_redirects default: changed from false to ['track_redirects' => true] so redirects are followed and the redirect history header is populated correctly
• Non-parseable responses (e.g. binary files filtered by allowedMimeTypes) now notify observers via crawled() with an empty body instead of being silently skipped
• URLs containing control characters are now detected and reported as malformed
• Dot segments (/../, /./) in extracted URLs are now normalized per RFC 3986
• Custom client options passed to Crawler::create() now merge with defaults instead of replacing them (pass null to remove a default)
• CrawlRequestFailed now wraps non-RequestException errors so observers always receive a RequestException
• Removed unused classes: Url, ResponseWithCachedBody, InvalidUrl

Added

• stream() method to opt-in to streaming HTTP responses for reduced memory usage
• matchWww() method to treat www.example.com and example.com as equivalent when using internalOnly()
• includeSubdomains() now works as a flag on internalOnly() and composes with matchWww()
• CrawlResponse::redirectHistory() and CrawlResponse::wasRedirected() for inspecting redirect chains
• CrawlObserver::crawlFailed() now receives a ?TransferStatistics parameter for detecting timeouts
• addObserver() now accepts variadic arguments: addObserver($obs1, $obs2)
• CrawlRequestFailed now preserves the original request from ConnectException (retaining custom headers like X-Started-At)

Major rewrite. See UPGRADING.md for a full list of breaking changes.

Changed

• Replace UriInterface with plain string URLs throughout the API
• Replace ResponseInterface with CrawlResponse in observer callbacks
• CrawlProfile is now an interface instead of an abstract class
• CrawlObserverCollection no longer implements ArrayAccess or Iterator
• Default scheme changed from http to https
• JavaScript rendering is now driver-based (Browsershot moved to suggest)
• UrlParser interface redesigned to return ExtractedUrl[] instead of adding to queue directly
• CrawlQueue::has() now accepts string instead of CrawlUrl|UriInterface
• start() now returns a FinishReason enum
• URL is now required in Crawler::create()

Added

• CrawlResponse object with status(), body(), dom(), header(), transferStats(), and more
• CrawlProgress tracking with urlsCrawled, urlsFailed, urlsFound, urlsPending
• FinishReason enum: Completed, CrawlLimitReached, TimeLimitReached, Interrupted
• Closure callbacks: onCrawled(), onFailed(), onFinished(), onWillCrawl()
• foundUrls() to collect all URLs as CrawledUrl objects
• fake() for testing without HTTP requests
• Scope helpers: internalOnly(), includeSubdomains(), shouldCrawl()
• Shorter method names: depth(), concurrency(), delay(), limit(), userAgent()
• Throttling: FixedDelayThrottle and AdaptiveThrottle
• Resource type extraction: alsoExtract(), extractAll(), ResourceType enum
• URL normalization in ArrayCrawlQueue
• Graceful shutdown via SIGINT/SIGTERM
• alwaysCrawl() and neverCrawl() pattern overrides
• retry() for automatic retries on connection errors and 5xx responses
• TransferStatistics with typed timing accessors
• CloudflareRenderer for JavaScript rendering
• JavaScriptRenderer interface for custom renderers
• Request configuration: basicAuth(), token(), withoutVerifying(), proxy(), cookies(), queryParameters(), middleware()

Removed

• CrawlUrl::create() static factory (use new CrawlUrl(...) instead)
• Spatie\Crawler\Url class
• ResponseWithCachedBody (replaced by CrawlResponse)
• nicmart/tree dependency
• spatie/browsershot as a required dependency (moved to suggest)
• setBrowsershot() and getBrowsershot() methods
• startCrawling() method (use start())
• setUrlParserClass() (use parseSitemaps() or pass a UrlParser directly)

Add Laravel 13 support

What's Changed

• Update nicmart/tree dependency version to ^0.10 by @robinmiau in https://github.com/spatie/crawler/pull/497

New Contributors

• @robinmiau made their first contribution in https://github.com/spatie/crawler/pull/497

Full Changelog: https://github.com/spatie/crawler/compare/8.4.6...8.4.7

What's Changed

• When fetching robots.txt, use the same User-Agent as defined by the user by @mattiasgeniar in https://github.com/spatie/crawler/pull/491

Full Changelog: https://github.com/spatie/crawler/compare/8.4.4...8.4.5

What's Changed

• Update issue template by @AlexVanderbist in https://github.com/spatie/crawler/pull/488
• fix(CrawlUrl): Use before initialization is now impossible by @Voltra in https://github.com/spatie/crawler/pull/490

New Contributors

• @Voltra made their first contribution in https://github.com/spatie/crawler/pull/490

Full Changelog: https://github.com/spatie/crawler/compare/8.4.3...8.4.4

What's Changed

• Do not try robots.txt when ignored by @kissifrot in https://github.com/spatie/crawler/pull/485

New Contributors

• @kissifrot made their first contribution in https://github.com/spatie/crawler/pull/485

Full Changelog: https://github.com/spatie/crawler/compare/8.4.2...8.4.3

What's Changed

• set spatie/browsershot minimal version to 5.0.5 by @grafst in https://github.com/spatie/crawler/pull/484

New Contributors

• @grafst made their first contribution in https://github.com/spatie/crawler/pull/484

Full Changelog: https://github.com/spatie/crawler/compare/8.4.1...8.4.2

Full Changelog: https://github.com/spatie/crawler/compare/8.4.0...8.4.1

What's Changed

• Add execution time limit by @VincentLanglet in https://github.com/spatie/crawler/pull/480

New Contributors

• @VincentLanglet made their first contribution in https://github.com/spatie/crawler/pull/480

Full Changelog: https://github.com/spatie/crawler/compare/8.3.1...8.4.0

What's Changed

• Upgrade spatie/browsershot to 5.0 by @hasansoyalan in https://github.com/spatie/crawler/pull/478

New Contributors

• @hasansoyalan made their first contribution in https://github.com/spatie/crawler/pull/478

Full Changelog: https://github.com/spatie/crawler/compare/8.3.0...8.3.1

What's Changed

• Add support for PHP 8.4 by @pascalbaljet in https://github.com/spatie/crawler/pull/477

Full Changelog: https://github.com/spatie/crawler/compare/8.2.3...8.3.0

What's Changed

• Fix setParsableMimeTypes() by @superpenguin612 in https://github.com/spatie/crawler/pull/470

Full Changelog: https://github.com/spatie/crawler/compare/8.2.2...8.2.3

What's Changed

• Check original URL against depth tree when visited link is a redirect by @superpenguin612 in https://github.com/spatie/crawler/pull/467

New Contributors

• @superpenguin612 made their first contribution in https://github.com/spatie/crawler/pull/467

Full Changelog: https://github.com/spatie/crawler/compare/8.2.0...8.2.1

What's Changed

• Fix wording in documentation by @adamtomat in https://github.com/spatie/crawler/pull/460
• Add Laravel/Illuminate 11 Support by @Jubeki in https://github.com/spatie/crawler/pull/461

New Contributors

• @adamtomat made their first contribution in https://github.com/spatie/crawler/pull/460
• @Jubeki made their first contribution in https://github.com/spatie/crawler/pull/461

Full Changelog: https://github.com/spatie/crawler/compare/8.1.0...8.2.0

What's Changed

• feat: custom link parser by @Velka-DEV in https://github.com/spatie/crawler/pull/458

New Contributors

• @Velka-DEV made their first contribution in https://github.com/spatie/crawler/pull/458

Full Changelog: https://github.com/spatie/crawler/compare/8.0.4...8.1.0

• allow Browsershot v4

What's Changed

• Fix return type by @riesjart in https://github.com/spatie/crawler/pull/452

New Contributors

• @riesjart made their first contribution in https://github.com/spatie/crawler/pull/452

Full Changelog: https://github.com/spatie/crawler/compare/8.0.2...8.0.3

What's Changed

• Define only needed methods in observer implementation by @buismaarten in https://github.com/spatie/crawler/pull/449

New Contributors

• @buismaarten made their first contribution in https://github.com/spatie/crawler/pull/449

Full Changelog: https://github.com/spatie/crawler/compare/8.0.1...8.0.2

What's Changed

• Check if rel attribute contains nofollow by @robbinbenard in https://github.com/spatie/crawler/pull/445

New Contributors

• @robbinbenard made their first contribution in https://github.com/spatie/crawler/pull/445

Full Changelog: https://github.com/spatie/crawler/compare/8.0.0...8.0.1

• add linkText to crawl observer methods
• upgrade dependencies

• support Laravel 10

What's Changed

• Feat/convert phpunit tests to pest by @mansoorkhan96 in https://github.com/spatie/crawler/pull/401
• Add the ability to change the default baseUrl scheme by @arnissolle in https://github.com/spatie/crawler/pull/402

New Contributors

• @arnissolle made their first contribution in https://github.com/spatie/crawler/pull/402

Full Changelog: https://github.com/spatie/crawler/compare/7.1.1...7.1.2

• allow Laravel 9 collections

• allow psr7 v2

• change response type hint (#371)

• require PHP 8+
• drop support for PHP 7.x
• convert syntax to PHP 8
• no API changes have been made

• bugfix: infinite loops when a CrawlProfile prevents crawling (#358)

• add setCurrentCrawlLimit and setTotalCrawlLimit
• internal refactors

• add support for PHP 8.0

• tweak variable naming in ArrayCrawlQueue (#326)

• improve chucked reading of response
• move observer / profiles / queues to separate namespaces
• typehint all the things
• use laravel/collections instead of tightenco package
• remove support for anything below PHP 7.4
• remove all deprecated functions and classes

• treat connection exceptions as request exceptions

• fix: method and property name error (#311)

• add crawler option to allow crawl links with rel="nofollow" (#310)

• only crawl links that are completely parsed

• fix curl streaming responses (#295)

• add setParseableMimeTypes() (#293)

• fix LinkAdder not receiving the updated DOM (#292)

• allow tightenco/collect 7 (#282)

• respect maximum response size when checking Robots Meta tags (#281)

• allow Guzzle 7

• allow symfony 5 components