cloudscraper

• Add new captcha provider Captcha AI

• Whats the point in even trying to detect v1 challenges anymore...

• Updated v2 detection

• Old code sitting in dev, pushed to prod..

• Cloudflare changing minor things that broke the regex.
• Changed debug to support non printable UTF-8

• The arms race continues, updated Cloudflare's changes ... agaaaaaaaaiiiin

• Refactoring code and move away from supporting python 2
• Bug fix Cloudflare v2 detection

• Added in "Bot Fight Mode" detection

• hCaptcha fix for deathbycaptcha
• Added new captcha provider CapMonster Cloud
• Replaced package polling with polling2 in captcha modules

• Removed debug from 2captcha (ooops my bad).
• Added in no_proxy to captcha parameters if you dont want to send proxy to 2captcha / anticaptcha.
• Added in platform filtering to browser (User-Agent) via platform parameter.
• added doubleDown parameter to control if re-request is to be performed when Captcha is detected.

• Initial update to Captcha providers to support proxies (anti-captcha, 2captcha).
• Some re-wording, comments and general house cleaning.

~12 days have passed and Cloudflare updated again... they keeping to the schedule 👍

• Fixed: Cloudflare V1 challenge change (broke regex by introducing blank a.value).
• Fixed: string -> float -> string causes issues in py2 str() rounding precision
• Enhancement: Added Pre/Posting Hooking into request function.

• Update regex for new Cloudflare changes in numerous places.
• Updated JSFuck challenge for new dynamic k variable.
• Updated interpreters to account for new dynamic k allocation from subset list.

• Update regex for Cloudflare form challenge
• Overwrite auto_set_ecdh by manually setting elliptic curve
• Rewrote native interpreter for JSFuck due to nested calculations
• Added exception if new Cloudflare challenge detected.
• Added support for hCaptcha in 9KW

• Add ability for custom ssl context to be passed
• Added new timer to anticaptcha module
• Fixed Cloudflare's challenge form change
• Removed DNT from headers causing reCaptcha on some sites
• Updated cipher suite for browsers

• Refactored Exceptions classes.
• Updated cipherSuites for Cloudflare.

The good folks over at Cloudflare have changed something... yet again... and explicitly setting ALPN now causes challenge issues on Ubuntu and Windows.

Cloudflare made a change, my change should circumvent theirs.

• Force SSL Context ALPN to HTTP/1.1

Just some refactoring / bug fixes

• Refactored 302 Redirect on localized path with no schema.
• @dipu-bd submitted PR for User_Agent.loadUserAgent() to close browser.json.

Thanks to @Fran008 , @TheYoke @paulitap88 , @vrayv and anyone else I missed for raising the tickets and testing the dev branches for me ❤️

cheers guys.

Hotfix 1.2.23

• Fix reCaptcha html entities
• Fix 302 Redirect after challenge solve, fix redirecting to existing path via `./

• Custom Exceptions
• Unescape HTML Entities Cloudflare introduced on the challenge request.

• Changed openSSL warning to a print instead of a raised exception.
• Removed Brotli as a required dependency.
• Updated cipher suites for User-Agents
• Fixed a bug in matching custom User-Agents

Hohoho Merry Christmas. …

• Improve / re-implement redirection support
• Also support http -> https protocol scheme switch on challenge solve
• Re-word Cloudflare 1020 block message
• Add cookie test
• Updated README.md

Has a number of fixes

• New native python Cloudflare challenge solver (now default interpreter).
• Cloudflare sometimes redirects instead of passthrough after challenge solve, re-request if is redirect.
• Alert/Raise Error if Cloudflare 1020 firewall block detected.
• Removed the requirement for pyopenssl
• Split out pytests into dev requirements
• Added custom parameter to browser param for custom User-Agent
• Added a tryMatch code in conjuction with custom parameter, which will try find and load ciphers && headers if custom is found in browsers.json otherwise a custom default will be used.

IUAM

• Endpoints have changed to detect parameter __cf_chl_jschl_tk__ with UUID, for the challenge solve
• Method is now a POST, no longer a GET
• Parameter's have been removed, and are now instead data in the POST form

reCaptcha

• Changes in IUAM apply here as well as the additional listed below
• Endpoints have changed to detect parameter __cf_chl_captcha_tk__ with UUID, for the challenge solve
• New id param in payload added, id derived from CF-RAY header, which is also in the variable data-ray

Testing

• testing is disabled till I write some new tests.

Fixed an issue with reCaptcha where if urllib3 < 1.25.1 and content was brotli compressed, it was not decompressing the brotli content.

• Removed cipher ECDHE-RSA-CHACHA20-POLY1305 to mitigate reCaptcha generation from Cloudflare

Removed Nothinng -> RuntimeError ReCaptcha
Removed Nothinng -> RuntimeError ReCaptcha
Removed TLS_CHACHA20_POLY1305_SHA256 -> RuntimeError ReCaptcha
Removed TLS_AES_128_GCM_SHA256 -> RuntimeError ReCaptcha
Removed TLS_AES_256_GCM_SHA384 -> RuntimeError ReCaptcha
Removed ECDHE-RSA-AES128-GCM-SHA256 -> 200
Removed AES128-GCM-SHA256 -> 200
Removed AES256-GCM-SHA384 -> 200
Removed AES256-SHA -> 200
Removed ECDHE-ECDSA-AES256-GCM-SHA384 -> 200
Removed ECDHE-ECDSA-CHACHA20-POLY1305 -> 200
Removed ECDHE-RSA-CHACHA20-POLY1305 -> 200
Removed ECDHE-ECDSA-AES128-GCM-SHA256 -> 200
Removed TLS_AES_128_GCM_SHA256 -> RuntimeError ReCaptcha
Removed TLS_AES_256_GCM_SHA384 -> RuntimeError ReCaptcha
Removed TLS_CHACHA20_POLY1305_SHA256 -> RuntimeError ReCaptcha
Removed ECDHE-ECDSA-AES128-GCM-SHA256 -> 200
Removed ECDHE-ECDSA-AES256-SHA -> 200
Removed ECDHE-RSA-CHACHA20-POLY1305 -> 200
Removed ECDHE-ECDSA-AES128-SHA -> 200
Removed ECDHE-RSA-AES128-GCM-SHA256 -> 200
Removed ECDHE-ECDSA-CHACHA20-POLY1305 -> 200
Removed DHE-RSA-AES256-SHA -> 200
Removed ECDHE-ECDSA-AES256-GCM-SHA384 -> 200
Removed AES256-SHA -> 200
Removed DHE-RSA-AES128-SHA -> 200

* Working list, by removing one of these ciphers in both browsers:
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
AES256-SHA

+-------------------------------+--------+---------+------------+
|             Cipher            | Chrome | Firefox | Removable? |
+-------------------------------+--------+---------+------------+
|     TLS_AES_128_GCM_SHA256    |   X    |    X    |            |
|     TLS_AES_256_GCM_SHA384    |   X    |    X    |            |
|  TLS_CHACHA20_POLY1305_SHA256 |   X    |    X    |            |
| ECDHE-ECDSA-AES128-GCM-SHA256 |   X    |    X    |    Yes     |
|  ECDHE-RSA-AES128-GCM-SHA256  |   X    |    X    |    Yes     |
| ECDHE-ECDSA-AES256-GCM-SHA384 |   X    |    X    |    Yes     |
| ECDHE-ECDSA-CHACHA20-POLY1305 |   X    |    X    |    Yes     |
|  ECDHE-RSA-CHACHA20-POLY1305  |   X    |    X    |    Yes     |
|       AES128-GCM-SHA256       |   X    |         |            |
|       AES256-GCM-SHA384       |   X    |         |            |
|           AES256-SHA          |   X    |    X    |    Yes     |
|     ECDHE-ECDSA-AES256-SHA    |        |    X    |            |
|     ECDHE-ECDSA-AES128-SHA    |        |    X    |            |
|       DHE-RSA-AES128-SHA      |        |    X    |            |
|       DHE-RSA-AES256-SHA      |        |    X    |            |
+-------------------------------+--------+---------+------------+

• Removed cipher ECDHE-RSA-AES256-GCM-SHA384 to mitigate reCaptcha generation from Cloudflare

Fix reCaptcha class solveCaptcha params.

Passed resp.text instead of resp to is_reCaptcha_Challenge() in Challenge_Response by mistake.

Cloudflare triggering reCapthca on firefox with the cipher DHE-RSA-AES128-SHA on Ubuntu and Windows systems

Cloudflare triggering reCapthca on the cipher AES256-GCM-SHA384 with chrome User-Agent on Ubuntu and Windows systems

Cloudflare triggering reCapthca or addtional challenge requests on the cipher AES256-SHA on Ubuntu 16.04, other OS's seem fine with the cipher...

Cloudflare causing issues for User-Agents with the following ciphers:

• AES128-GCM-SHA256 in Chrome
• DHE-RSA-AES256-SHA in Firefox