niquests

Fixed

• Accessing a lazy response (multiplexed enabled) that have multiple redirects did not work appropriately.

Changed

• Response iter_content and iter_line read chunks as they arrive by default. The default chunk size is now -1. -1 mean to instruct that the chunks can be of variable sizes, depending on how packets arrives. It improves overall performances.
• urllib3.future lower bound constraint has been raised to version 2.4.904 in order to accept -1 as a chunk size.

Fixed

• Connection information kept targeting its original copy, thus always keeping the latest timings inside while expecting the historical ones.

Added

• AsyncSession now returns a AsyncResponse when stream is set to True in order to handle properly streams in an async context.

Fixed

• CaseInsensibleDict did not properly convert HTTPHeaderDict from urllib3 thus only letting the last entry in.
• Redirect chain lead to a non waited coroutine in AsyncSession.

Added

• Support for specifying a custom DNS resolver in Session.
• Support for passing the source address in Session.
• Support for disabling either IPv4 or IPv6 within a Session.

Changed

• PySocks is no longer used for SOCKS proxies. Replaced by python-socks instead.
• urllib3.future minimal version raised to 2.4+ to leverage newly added features.
• Improve compatibility when end-user inadvertently pass a Timeout or Retry instance from the legacy urllib3 instead of urllib3_future.

Fixed

• Blocking the event loop when closing the AsyncSession using with.
• Rare exception on older PyPy interpreters due to Generic having unsupported type variable in extensions._sync_to_async module.

Misc

• Project extras aligned with urllib3.future.
• Using nox instead of tox.
• Switch to ruff instead of black and isort.

Fixed

• Overall static typing experience have been improved.
• Code HTTP 425 is now registered as too_early in addition to the legacy unordered_collection.

Removed

• Private module niquests._internal_utils has been removed as it no longer serve its purposes.

Added

• Hook on_upload that allows you to monitor/track the upload progress.
• Model TransferProgress that is used in PreparedRequest as public property upload_progress.

Fixed

• Hooks that does not accept keyword arguments are rejected.
• Applying max_fetch to Session.gather(...) did not prevent the adapter to drain all pending responses.
• Closed session having unconsumed multiplexed requests leaked an exception from urllib3.future.

Changed

• Aligned qh3 version constraint in http3 extra with urllib3.future.

Fixed

• Warning filter (ignore) for DependencyWarning within urllib3.future wasn't applied in time.

Added

• Maximum of (lazy) response(s) to be resolved when calling Session.gather(..., max_fetch = ...). Specifying a valid int to max_fetch will stop right after having resolved the right amount of responses.

Changed

• urllib3.future minimal version raised to 2.3.900 to leverage the fallback top-level package urllib3_future.

Fixed

• Error when having accidentally overridden urllib3.future by an external dependency.
• Undesirable warning yielded by cryptography because of a Microsoft root certificate. "Parsed a negative serial number, which is disallowed by RFC 5280."
• Take into account newly registered custom CA when using wassima.register_ca(...).

Removed

• Dependency check at runtime for urllib3. There's no more check and warnings at runtime for that subject. Ever.

Fixed

• Compatibility with some third-party mock tools.

Changed

• Relax IllegalHeader constraint when value is an integer, or float.

Fixed

• Overall performance improvement.
• Racing condition in HTTPAdapter with multiplexed enabled while in threads.

Removed

• Checks for header validity as it is duplicated (upstream does it already). Warning: Function check_header_validity has been removed. It was not public in the first place.

Changed

• Cached proxies from environment to avoid unnecessary scan on each request.

Changed

• Enforced a maximum in-flight request when using multiplexed connections. Default to 124 per connections so, actually 1240 per Session (default is 10 connections). This can be overridden in our HTTPAdapter for advanced users. This limit was changed due to constraint in qh3, for now we have no way to dynamically set this. We choose the safest lowest common value in h2, and qh3.

Fixed

• Performance issues in get_environ_proxies().

Fixed

• Performance issues in HTTP/2, and HTTP/3, with or without multiplexed connections.

Changed

• Enforced a maximum in-flight request when using multiplexed connections. Default to 200 per connections so, actually 2000 per Session (default is 10 connections). This can be overridden in our HTTPAdapter for advanced users.

Changed

• Changed method raise_for_status in class Response to return self in order to make the call chainable. Idea taken from upstream https://github.com/psf/requests/issues/6215
• Bump minimal version supported for urllib3.future to 2.2.901 for recently introduced added features (below).

Added

• Support for multiplexed connection in HTTP/2 and HTTP/3. Concurrent requests per connection are now a thing, in synchronous code. This feature is the real advantage of using binaries HTTP protocols. It is disabled by default and can be enabled through Session(multiplexed=True), each Response object will be 'lazy' loaded. Accessing anything from returned Response will block the code until target response is retrieved. Use Session.gather() to efficiently receive responses. You may also give a list of responses that you want to load.

  Example A) Emitting concurrent requests and loading them via Session.gather()

  from niquests import Session
  from time import time
  
  s = Session(multiplexed=True)
  
  before = time()
  responses = []
  
  responses.append(
    s.get("https://pie.dev/delay/3")
  )
  
  responses.append(
    s.get("https://pie.dev/delay/1")
  )
  
  s.gather()
  
  print(f"waited {time() - before} second(s)")  # will print 3s
  

  Example B) Emitting concurrent requests and loading them via direct access

  from niquests import Session
  from time import time
  
  s = Session(multiplexed=True)
  
  before = time()
  responses = []
  
  responses.append(
    s.get("https://pie.dev/delay/3")
  )
  
  responses.append(
    s.get("https://pie.dev/delay/1")
  )
  
  # internally call gather with self (Response)
  print(responses[0].status_code)  # 200! :! Hidden call to s.gather(responses[0])
  print(responses[1].status_code)  # 200!
  
  print(f"waited {time() - before} second(s)")  # will print 3s
  

  You have nothing to do, everything from streams to connection pooling are handled automagically!

• Support for in-memory intermediary/client certificate (mTLS). Thanks for support within urllib3.future. Unfortunately this feature may not be available depending on your platform. Passing cert=(a, b, c) where a or/and b contains directly the certificate is supported. See https://urllib3future.readthedocs.io/en/latest/advanced-usage.html#in-memory-client-mtls-certificate for more information. It is proposed to circumvent recent pyOpenSSL complete removal.

• Detect if a new (stable) version is available when invoking python -m niquests.help and propose it for installation.

• Add the possibility to disable a specific protocol (e.g. HTTP/2, and/or HTTP/3) when constructing Session. Like so: s = Session(disable_http2=..., disable_http3=...) both options are set to False, thus letting them enabled. urllib3.future does not permit to disable HTTP/1.1 for now.

• Support passing a single str to auth=... in addition to actually supported types. It will be treated as a Bearer token, by default to the Authorization header. It's a shortcut. You may keep your own token prefix in given string (e.g. if not Bearer).

• Added MultiplexingError exception for anything related to failure with a multiplexed connection.

• Added async support through AsyncSession that utilize an underlying thread pool.

  from niquests import AsyncSession
  import asyncio
  from time import time
  
  async def emit() -> None:
      responses = []
  
      async with AsyncSession(multiplexed=True) as s:
          responses.append(await s.get("https://pie.dev/get"))
          responses.append(await s.get("https://pie.dev/head"))
  
          await s.gather()
  
      print(responses)
  
  async def main() -> None:
      foo = asyncio.create_task(emit())
      bar = asyncio.create_task(emit())
      await foo
      await bar
  
  if __name__ == "__main__":
      before = time()
      asyncio.run(main())
      print(time() - before)
  

  Or without multiplexing if you want to keep multiple connections open per host per request.

  from niquests import AsyncSession
  import asyncio
  from time import time
  
  async def emit() -> None:
      responses = []
  
      async with AsyncSession() as s:
          responses.append(await s.get("https://pie.dev/get"))
          responses.append(await s.get("https://pie.dev/head"))
  
      print(responses)
  
  async def main() -> None:
      foo = asyncio.create_task(emit())
      bar = asyncio.create_task(emit())
      await foo
      await bar
  
  if __name__ == "__main__":
      before = time()
      asyncio.run(main())
      print(time() - before)
  

  You may disable concurrent threads by setting AsyncSession.no_thread = True.

Security

• Certificate revocation verification may not be fired for subsequents requests in a specific condition (redirection).

Fixed

• Static type checker not accepting iterable[str] for data. A fix in urllib3.future allows it since v2.1.902.
• Unattended override of manually provided Authorization if .netrc existed with an eligible entry. Taken from closed PR https://github.com/psf/requests/pull/6555 and initially raised in https://github.com/psf/requests/issues/3929

Added

• oheaders property in Request, and PreparedRequest in addition to Response.

Fixed

• oheaders from a Response contains Set-Cookie entries when it should not.
• Static type checker not accepting list[str] in values for argument param.
• Static type checker not accepting Iterable[bytes] for data.
• Function proxy_bypass_registry for Windows may be fooled by insufficient control on our end. Patch taken from idle upstream PR https://github.com/psf/requests/pull/6302
• SSLError message related to the certificate revocation could print None instead of unspecified for the reason.

Changed

• Allow setting None in max_size for SharableLimitedDict to remove limits.
• Using RLock instead of Lock in SharableLimitedDict, and InMemoryRevocationStatus classes.

Misc

• Missing assert statements for test test_header_validation.
• Unrelated warnings are now silent in our test suite.
• Unexpected warning now trigger an error in our test suite.
• Removed tests.compat.
• Removed test-readme, flake8, and publish from Makefile.

Added

• Extra-dist install http3 to force install HTTP/3 support in your environment if not present.
• Extra-dist install ocsp to force install certificate revocation support in your environment if not present.

Fixed

• Static type checker not accepting list[str] in values for argument data.

Misc

• Changed the documentation theme by furo.

Added

• IPv6 support in the NO_PROXY environment variable or in the proxies (key no_proxy) argument. Patch taken from idle upstream PR https://github.com/psf/requests/pull/5953

• Preemptively register a website to be HTTP/3 capable prior to the first TLS over TCP handshake. You can do so by doing like:

  from niquests import Session
  
  s = Session()
  s.quic_cache_layer.add_domain("cloudflare.com")
  

• Passed data will be converted to form-data if headers have a Content-Type header and is set to multipart/form-data. Otherwise, by default, it is still urlencoded. If you specified a boundary, it will be used, otherwise, a random one will be generated.

Fixed

• Fixed Transfer-Encoding wrongfully added to headers when body is actually of length 0. Due to ambiguous return of super_len in niquests internals.
• Fixed accepting three-valued tuple for Timeout (connect, read, total) in addition of known (connect, read) tuple.

Misc

• Static typing has been improved to provide a better development experience.

Added

• Certificate revocation verification via the OCSP protocol.

  This feature is broadly available and is enabled by default when verify=True. We decided to follow what browsers do by default, so Niquests follows by being non-strict. OCSP responses are expected to arrive in less than 200ms, otherwise ignored (e.g. OCSP is dropped). Niquests keeps in-memory the results until the size exceed 2,048 entries, then an algorithm choose an entry to be deleted (oldest request or the first one that ended in error).

  You can at your own discretion enable strict OCSP checks by passing the environment variable NIQUESTS_STRICT_OCSP with anything inside but 0. In strict mode the maximum delay for response passes from 200ms to 1,000ms and raises an error or explicit warning.

  In non-strict mode, this security measure will be deactivated automatically if your usage is unreasonable. e.g. Making a hundred of requests to a hundred of domains, thus consuming resources that should have been allocated to browser users. This was made available for users with a limited target of domains to get a complementary security measure.

  Unless in strict-mode, the proxy configuration will be respected when given, as long as it specify a plain http proxy. This is meant for people who want privacy.

  This feature may not be available if the cryptography package is missing from your environment. Verify the availability after Niquests upgrade by running python -m niquests.help.

  There is several downside of using OCSP, Niquests knows it. It is not a silver bullet solution. But better than nothing. It does not apply to HTTPS proxies themselves. For now.

• Add property ocsp_verified in both PreparedRequest, and Response to have a clue on the post handshake verification.

  Will be None if no verification took place, True if the verification leads to a confirmation from the OCSP server that the certificate is valid, False otherwise.

Changed

• Bump lower version requirement for urllib3.future to 2.1.900 to ensure compatibility with newer features.
• Internal in-memory QUIC capabilities is now thread safe and limited to 12,288 entries.
• Pickling a Session object no-longer dump adapters or the QUIC in-memory capabilities, they are reset on setstate.

Fixed

• conn_info was unset if the response came after a redirect.

Changed

• niquests.help show more information about direct dependencies.
• urllib3.future minimal version raised to 2.0.936 due to an important fixes on the QUIC layer.
• wassima minimal version raised to 1.0.1 in order to support certifi as fallback in rare case.

Fixed

• Method head not accepting allow_redirect named argument.
• PreparedRequest did not uppercase the HTTP verb when constructed manually.

Changed

• Minor code refactor in the text, and json methods within the Response class to be more concise.

Added

• Static type annotations thorough the whole package.
• cert argument for client authentication with certificate can now pass the password/passphrase using a 3-values tuple (cert, key, password). The three parameters in the tuple must be of type str.
• verify argument behavior has been extended and now accept your CA bundle as str instead of a path. It also accepts your CA bundle as bytes directly. This help when you do not have access to the fs.
• Operating system truststore will be used instead of certifi. Root CAs are automatically grabbed from your computer configuration.
• Oriented-object headers. Access them through the new property oheaders in your Response.
• Propagated the argument retries in niquests.api for all functions.
• Added argument retries in the Session constructor.
• Property conn_info to the PreparedRequest and Response that hold a reference to a ConnectionInfo. This class exposes the following properties: certificate_der (bytes), certificate_dict (dict) as provided by the standard library (ssl), destination_address (tuple[ipAddress, portNumber]), cipher (str), tls_version (TLSVersion), and http_version.
• Two hooks, namely pre_send and pre_request. The pre_request event is fired just after the initial construction of a PreparedRequest instance. Finally, the pre_send will be triggered just after picking a (live) connection for your request. The two events receive a PreparedRequest instance.

Changed

• Calling the method json from Response when no encoding was provided no longer relies on internal encoding inference. We fall back on charset-normalizer with a limited set of charsets allowed (UTF-8/16/32 or ASCII).
• No longer will the text method from Response return str if content cannot be decoded. It returns None instead.
• If specified charset in content-type does not exist (LookupError) the text method from Response will rely on charset detection.
• If specified charset in content-type is not made for text decoding (e.g. base64), the text method from Response returns None.
• With above four changes, the json method will raise RequestsJSONDecodeError when the payload (body) cannot be decoded.
• Passing invalid files description no longer just skip invalid entries, it raises ValueError from now on.
• Non-str HTTP-Verb are refused.
• Passing files with minimal description (meaning no tuple but just the fp) no longer guess its name when fp.name return bytes.
• No longer will the default timeout be unset, thus making you waiting indefinitely. Functions get, head, and options ships with a default of 30 seconds. Then put, post, patch and delete uses a default of 120 seconds. Finally, the request function also have 120 seconds.
• Basic authorization username and password are now encoded using utf-8 instead of latin-1 prior to being base64 encoded.

Removed

• Property apparent_encoding in favor of a discrete internal inference.
• Support for the legacy chardet detector in case it was present in environment. Extra chardet_on_py3 is now unavailable.
• requests.compat no longer hold reference to chardet.
• Deprecated requests.packages that was meant to avoid breakage from people importing urllib3 or chardet within this package. They were vendored in early versions of Requests. A long time ago.
• Deprecated function get_encodings_from_content from utils.
• Deprecated function get_unicode_from_response from utils.
• BasicAuth middleware no-longer support anything else than bytes or str for username and password.
• requests.compat is stripped of every reference that no longer vary between supported interpreter version.
• Charset fall back ISO-8859-1 when content-type is text and no charset was specified.
• Main function get, post, put, patch, delete, and head no longer accept kwargs. They have a fixed list of typed argument. It is no longer possible to specify non-supported additional keyword argument from a Session instance or directly through requests.api functions. e.g. function delete no-longer accept json, or files arguments. as per RFCs specifications. You can still override this behavior through the request function.
• Mixin classes RequestEncodingMixin, and RequestHooksMixin due to OOP violations. Now deported directly into child classes.
• Function unicode_is_ascii as it is part of the stable str stdlib on Python 3 or greater.
• Alias function session for Session context manager that was kept for BC reasons since the v1.
• pyOpenSSL/urllib3 injection in case built-in ssl module does not have SNI support as it is not the case anymore for every supported interpreters.
• Constant DEFAULT_CA_BUNDLE_PATH, and submodule certs due to dropping certifi.
• Function extract_zipped_paths because rendered useless as it was made to handle an edge case where certifi is "zipped".
• Extra security when installing this package. It was previously emptied in the previous major.
• Warning emitted when passing a file opened in text-mode instead of binary. urllib3.future can overrule the content-length if it detects an error. You should not encounter broken request being sent.
• Support for simplejson if was present in environment.
• Submodule compat.

Fixed

• An invalid content-type definition would cause the charset being evaluated to True, thus making the program crash.
• Given proxies could be mutated when environment proxies were evaluated and injected. This package should not modify your inputs. For context see https://github.com/psf/requests/issues/6118
• A server could specify a Location header that does not comply to HTTP specifications and could lead to an unexpected exception. We try to fall back to Unicode decoding if the typical and expected Latin-1 would fail. If that fails too, a proper exception is raised. For context see https://github.com/psf/requests/issues/6026
• Top-level init now specify correctly the exposed api. Fixes mypy error .. does not explicitly export attribute ...

Bugfixes

• Fix QUIC cache when using requests.request without persistent Session

Dependencies

• urllib3.future minimal version supported raised to 2.0.932 (ship with critical fixes)

Dependencies

• ⚠️ Switch urllib3 for urllib3.future ⚠️ This may contain minor breaking changes, so we advise careful testing and reviewing.

Features

• Support for HTTP/2.
• Support for HTTP/3 over QUIC.
• Added property http_version that return an integer (11, 20, and 30) within Response to identify which protocol has been negotiated.
• Added quic_cache_layer mutable mapping optional argument to Session for optional external caching of QUIC server capabilities.

Improvements

• HTTP/1.1 no longer relies on the standard library http.client but h11.
• Response representation __repr__ now yield the HTTP version.

Removed

• Warning filter for charset_normalizer as it was removed long ago.