What's Changed

🎉 New Features

• Added Context field to Options for external lifecycle control by @idanavishai in #1593
• Added secrets extractor to knowledgebase by @dogancanbakir in #1667
• Added endpoints extractor to knowledgebase by @dogancanbakir in #1669
• Added callbacks to headless engine by @evanrolfe in #1662
• Added content similarity detection / deduplication by @geeknik in #915

🐞 Bug Fixes

• Fixed require label boundary for rdn scope to block look-alike domains by @Mubashirrrr in #1685
• Fixed concurrent map access on global custom fields by @dogancanbakir in #1702
• Fixed cap redirect by @Mzack9999 in #1697

Other Changes

• Added documentation for dit knowledge base + auto-download by @dogancanbakir in #1684

New Contributors

• @idanavishai made their first contribution in #1593
• @evanrolfe made their first contribution in #1662
• @Mubashirrrr made their first contribution in #1685

Full Changelog: v1.6.1...v1.7.0

What's Changed

bumped version

Full Changelog: v1.6.0...v1.6.1

What's Changed

🎉 New Features

• Added automatic login support by @dogancanbakir in #1620
• Added per-domain page budget to limit pages crawled per domain (0 = unlimited) by @dogancanbakir in #1612
• Added per-host rate limiting with adaptive backoff by @dogancanbakir in #1614
• Made jsluice cgo-free by @Mzack9999 in #1570
• Added page load strategy by @fail-open in #1514

-pls, -page-load-strategy string page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")

🐞 Bug Fixes

• Fixed cleanup of crawler resource lifecycles by @x-stp in #1634
• Fixed headless Chrome profile persistence with -no-incognito and -chrome-data-dir by @iamkunal9 in #1608
• Fixed gracefully handling DOM retrieval timeout in headless mode by @trasnake87 in #1592
• Fixed ignored headless options since v1.4.0 by @dogancanbakir in #1622
• Fixed onclick link discovery for dynamic JS navigations by @dogancanbakir in #1624
• Fixed preserving response body/raw and waiting after form submission in headless mode by @XTeam-Wing in #1585
• Fixed output of unvisited urls by @Mzack9999 in #1595

Other Changes

• Added none to filters by @Mzack9999 in #1594
• Added helper functions + tests by @Mzack9999 in #1596
• Added deduplication of headless mode output by @dogancanbakir in #1602
• Updated Ubuntu installation instructions by @ecdeveloper in #1628

New Contributors

• @trasnake87 made their first contribution in #1592
• @iamkunal9 made their first contribution in #1608
• @ecdeveloper made their first contribution in #1628
• @x-stp made their first contribution in #1634
• @XTeam-Wing made their first contribution in #1585

Full Changelog: v1.5.0...v1.6.0

What's Changed

🎉 New Features

• Added -filter-similar flag for URL path pattern deduplication by @dogancanbakir in #1550
• Added captcha detection and solving support for headless by @dogancanbakir in #1536
• Added knowledge base classification using dit by @dogancanbakir in #1538

🐞 Bug Fixes

• Fixed runner to prioritize chrome-ws-url over headless flag by @fail-open in #1506

New Contributors

• @fail-open made their first contribution in #1506

Full Changelog: v1.4.0...v1.5.0

What's Changed

🎉 New Features

• Added headless mode support by @Ice3man543 in #1396

You can try out the headless crawler today with the below steps:

# Simple headless crawl, show the browser as well
katana -u https://cloud.projectdiscovery.io -headless -show-browser

# Deeper exploration
katana -u https://ginandjuice.shop -headless -depth 5 -aff

# With debug logging (for investigating state failures, etc) writes diagnostic information in a new directory for debugging purposes along with more verbose logging.
katana -debug -u https://www.github.com -headless -enable-diagnostics

🐞 Bug Fixes

• Fixed concurrency not working as expected in the hybrid crawler by @majiayu000 in #1488

New Contributors

• @majiayu000 made their first contribution in #1488

Full Changelog: v1.3.0...v1.4.0

What's Changed

🎉 New Features

• Added -no-default-ext-filter option to disable default extension filter by @mjkim610 in #1386
• Added -eof and -lof flags for JSON output filtering by @hanXen in #1390

🐞 Bug Fixes

• Fixed -kf option validation by guarding it with pre-defined values by @mjkim610 in #1359
• Fixed nil ptr dereference in response handling by @dogancanbakir in #1407
• Fixed onclick event detection in javascript by @zsbahtiar in #1427
• Fixed nil ptr dereference in response handling in makeRequest function by @zzdboy in #1460
• Fixed -no-scope behaviour by @Deamhan in #1454

Other Changes

• Added crawl depth for each entry in verbose logs by @mjkim610 in #1418
• Updated .gitignore by @mjkim610 in #1395
• Updated formatJSON function to use OrderedMap to preserve order of output fields by @hanXen in #1417
• Improved program exit status handling by @dogancanbakir in #1388
• Improved error handling for for empty response by @mjkim610 in #1409
• Improved depth filtering by updating order of operations by @Deamhan in #1455

New Contributors

• @hanXen made their first contribution in #1390
• @mjkim610 made their first contribution in #1386
• @zsbahtiar made their first contribution in #1427
• @zzdboy made their first contribution in #1460
• @Deamhan made their first contribution in #1455

Full Changelog: v1.2.2...v1.3.0

What's Changed

• Added option to disable unique content filter by @nirohfeld in #1356
• Added cookie support with crawl by @vbisbest in #1345
• Added stale workflow by @dogancanbakir in #1382

New Contributors

• @nirohfeld made their first contribution in #1356
• @vbisbest made their first contribution in #1345

Full Changelog: v1.2.1...v1.2.2

What's Changed

Other Changes

• Bumped Dockerfile golang version by @dogancanbakir in #1339

Full Changelog: v1.2.0...v1.2.1

What's Changed

🎉 New Features

• Added -output-template option to allow users to define custom output formats using templates, providing flexibility in structuring results by @dwisiswant0 in #1197

$ katana -silent -u https://www.google.com/appserve/security-bugs/m2/new -ot "{{email}} - {{url}}"
for.someone.else@gmail.com - https://www.google.com/appserve/security-bugs/m2/new

• Introduced OnSkipURL to allow more control over SDK behavior by @dogancanbakir in #1329
• Introduced -path-climb option to allow auto discovering parent paths to crawl by @dogancanbakir in #1334

$ katana -silent -u https://scanme.sh/test/path/to/file.html -path-climb
https://scanme.sh/test/path/to/file.html
https://scanme.sh/test/path
https://scanme.sh/test/path/to
https://scanme.sh/test

🐞 Bug Fixes

• Fixed DSL matcher by @dogancanbakir in #1283

Other Changes

• Replaced global parser state with instance-based parsers to enabling more flexible and isolated parsing behavior by @awerqo in #1275
• Improved network request capturing to include XHR, Fetch, and Script resource types when extraction is enabled by @dwisiswant0 in #1277
• Improved JavaScript file endpoint detection to better handle varying relative path formats, ensuring more accurate matching across a broader range of file paths. by @Autumn-27 in #1242
• Improved error guarding for crawl session initialization by @dogancanbakir in #1300
• Updated release test action to use composite action by @dogancanbakir in #1308

New Contributors

• @awerqo made their first contribution in #1275
• @Autumn-27 made their first contribution in #1242

Full Changelog: v1.1.3...v1.2.0

What's Changed

🐞 Bug Fixes

• Fixed the issue where the "-ef" parameter only filtered at the output level, which increased memory usage by @dogancanbakir in #1203
• Fixed crawl issue by replacing WaitLoad/Idle with WaitStable to ensure all requests are completed by @alban-stourbe-wmx in #1217
• Fixed Dockerfile by adjusting the build environment setup by @LukasForst in #1122

Other Changes

• Made -store-response feature to properly save URLs with ports by @dogancanbakir in #1229
• Fixed typos in both the README and code documentation by @kianmeng in #1208
• Added note to documentation about Go version required by @adaminfinitum in #1216

New Contributors

• @LukasForst made their first contribution in #1122
• @kianmeng made their first contribution in #1208
• @adaminfinitum made their first contribution in #1216

Full Changelog: v1.1.2...v1.1.3