NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Packagist
Packagist, #705 most downloaded on Packagist
Determine if a page may be crawled from robots.txt and robots meta tags
Latest stable
2.6.0
Last release 16 days ago
06 Nov 2026
Ships fairly regularly
a new release about every 3 months
Nearly every release is documented
notes for 30 of 33 stable releases
Nothing withdrawn
no release was ever pulled
9 years old
33 releases · first in 2018
Drag across the ruler to compare, or tap a release to open it.
Full local suite on PHP 8.4.27: 57 tests, 172 assertions, no skips or PHPUnit deprecations, with both PHPUnit 13.4.5 and 13.4.6.
^8.4.1). Support for PHP 8.1, 8.2, 8.3, and 8.4.0 has been dropped.Full Changelog: https://github.com/spatie/robots-txt/compare/2.5.5...2.6.0
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Allow rules containing wildcards (*) or end anchors ($) can now override less-specific Disallow rules. Fixes #88.
*) or end anchors ($) can now override less-specific Disallow rules. Fixes #88.These corrections can change crawl decisions in either direction: some previously denied URLs become allowed, and some previously allowed URLs become denied. There are no public API or dependency changes.
Full Changelog: https://github.com/spatie/robots-txt/compare/2.5.4...2.5.5
Send user agent header with file_get_contents requests to prevent Cloudflare and similar WAFs from blocking requests
Send user agent header with file_get_contents requests to prevent Cloudflare and similar WAFs from blocking requests
Bump stefanzweifel/git-auto-commit-action from 6 to 7 by @dependabot[bot] in https://github.com/spatie/robots-txt/pull/78
Full Changelog: https://github.com/spatie/robots-txt/compare/2.5.2...2.5.3
Bump aglipanci/laravel-pint-action from 2.5 to 2.6 by @dependabot[bot] in https://github.com/spatie/robots-txt/pull/73
Full Changelog: https://github.com/spatie/robots-txt/compare/2.5.1...2.5.2
Bump stefanzweifel/git-auto-commit-action from 5 to 6 by @dependabot in https://github.com/spatie/robots-txt/pull/68
Full Changelog: https://github.com/spatie/robots-txt/compare/2.5.0...2.5.1
Implement RobotsTxt->whyDisallows($path, $userAgent) to find why a path was disallowed by @gvlasov in https://github.com/spatie/robots-txt/pull/66
Full Changelog: https://github.com/spatie/robots-txt/compare/2.4.0...2.5.0
Bump aglipanci/laravel-pint-action from 2.4 to 2.5 by @dependabot in https://github.com/spatie/robots-txt/pull/63
Full Changelog: https://github.com/spatie/robots-txt/compare/2.3.1...2.4.0
Bump dependabot/fetch-metadata from 2.2.0 to 2.3.0 by @dependabot in https://github.com/spatie/robots-txt/pull/62
Full Changelog: https://github.com/spatie/robots-txt/compare/2.3.0...2.3.1
Update phpunit/phpunit requirement from ^9.0|^10.0 to ^11.5.2 by @dependabot in https://github.com/spatie/robots-txt/pull/59
Full Changelog: https://github.com/spatie/robots-txt/compare/2.2.3...2.3.0
avoid expensive queries that do not change certain results by @elcapo in https://github.com/spatie/robots-txt/pull/53
Full Changelog: https://github.com/spatie/robots-txt/compare/2.2.2...2.2.3
Bump actions/checkout from 2 to 4 by @dependabot in https://github.com/spatie/robots-txt/pull/47
Full Changelog: https://github.com/spatie/robots-txt/compare/2.2.1...2.2.2
Add missing use statement for InvalidArgumentException by @remcom in https://github.com/spatie/robots-txt/pull/44
Full Changelog: https://github.com/spatie/robots-txt/compare/2.2.0...2.2.1
optionally allow partial matches and global groups by @resohead in https://github.com/spatie/robots-txt/pull/43
Full Changelog: https://github.com/spatie/robots-txt/compare/2.1.0...2.2.0
Add some efficiencies to prevent unnecessary requests by @tsjason in https://github.com/spatie/robots-txt/pull/42
Full Changelog: https://github.com/spatie/robots-txt/compare/2.0.3...2.1.0
Fix deprecation message by @buismaarten in https://github.com/spatie/robots-txt/pull/41
Full Changelog: https://github.com/spatie/robots-txt/compare/2.0.2...2.0.3
Match meta tag for single quotes, often used by Wordpress sites by @gjportegies in https://github.com/spatie/robots-txt/pull/35
Full Changelog: https://github.com/spatie/robots-txt/compare/2.0.1...2.0.2
- added x-robots-tag: none
use php-cs-fixer & github workflow
- handle multiple user-agent
add support for PHP 8.0 + move to GitHub actions
fix find robots meta tag line if minified code
fix headers checking (nofollow, noindex) for custom userAgent
improvements around handling of wildcards, end-of-string, query string
- improve readability
fix parsing robotstxt urls with keywords
make robots.txt check case insensitive
prevent exception if the domain has no robots.txt
- initial release
Even more header parsing improvements.
Even more header parsing improvements.
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per quarter.