pex

[!NOTE] Although 2.6.1 successfully released to PyPI, it failed to release to GitHub Releases (neither the Pex PEX nor the pex.pdf were published.) You can use Pex 2.6.3 instead which has no Pex code changes over this release.

This release improves error messages when attempting to read invalid metadata from distributions such that the problematic distribution is always identified.

• Improve errors for invalid distribution metadata. (#2443)

This release adds support for PEP-723 script metadata in --exes. For such a script with metadata describing its dependencies or Python version requirements, running the script is as simple as pex --exe <script> -- <script args> and building a PEX encapsulating it as simple as pex --exe <script> --output <PEX file>.

• Add support for PEP-723 script metadata to --exe. (#2436)

This release brings support for Python 3.13 and --pip-version 24.1, which is the first Pip version to support it.

• Support --pip-version 24.1 and Python 3.13. (#2435)

This release fixes pex --only-binary X --lock ... to work with lock files also created with --only-binary X. The known case here is a --style universal lock created with --only-binary X to achieve a partially wheel-only universal lock.

• Fix pex --only-binary X --lock .... (#2433)

This release brings new support for preserving arguments passed to the Python interpreter (like -u or -W ignore) either via running a PEX via Python from the command line like python -u my.pex or via a shebang with embedded Python arguments like #!/usr/bin/python -u.

In addition, PEXes can now be built with --inject-python-args similar to the existing --inject-args but sealing in arguments to pass to Python instead. When both explicitly passed Python interpreter arguments and injected Python interpreter arguments are specified, the injected arguments appear first on the synthesized command line and the explicitly passed arguments appear last so that the explicit arguments can trump (which is how Python handles this).

Several bugs existing in the --exclude implementation since its introduction are now fixed and the feature is greatly improved to act on excludes eagerly, never traversing them in the resolve process; thus avoiding downloads associated with them as well as potentially failing metadata extraction & wheel builds for ill-behaved sdists.

Finally, a bug was fixed in pex3 lock export for lock files containing either locked VCS requirements or locked local project directories. Previously, these were exported with a <project name>==<version> requirement, which lost fidelity with the input requirement. Now they are exported with their original requirement form. Further, since the --hash of these styles of locked requirement are unuseable outside Pex, a new --format option of pip-no-hashes is introduced for the adventurous.

• Implement support for preserving and injecting Python args. (#2427)
• Fix --exclude. (#2409)
• Fix pex3 lock export handling of exotic reqs. (#2423)

This release fixes pex3 lock create support for --pip-versions 23.3.1 and newer. Previously, when locking using indexes that serve artifacts via re-directs, the resulting lock file would contain the final re-directed URL instead of the originating index artifact URL. This could lead to issues when the indexes re-direction scheme changed or else if authentication parameters in the original index URL were stripped in the Pip logs.

• Fix artifact URL recording for pip>=23.3. (#2421)

This release fixes a regression for users of gevent monkey patching. The fix in #2356 released in Pex 2.1.163 lead to these users receiving spurious warnings from the gevent monkey patch system about ssl being patched too late.

• Delay import of ssl in pex.fetcher. (#2417)

This release fixes Pex to respect lock file interpreter constraints and target systems when downloading artifacts.

• Fix lock downloads to use all lock info. (#2396)

This release introduces pex3 lock sync as a higher-level tool that can be used to create and maintain a lock as opposed to using a combination of pex3 lock create and pex3 lock update. When there is no existing lock file, pex3 lock sync --lock lock.json ... is equivalent to pex3 lock create --output lock.json ..., it creates a new lock. On subsequent uses however, pex3 lock sync --lock lock.json ... updates the lock file minimally to meet any changed requirements or other changed lock settings.

This release also fixes pex --no-build --lock ... to work with lock files also created with --no-build. The known case here is a --style universal lock created with --no-build to achieve a wheel-only universal lock.

This release includes a fix to clarify the conditions under which --requierements-pex can be used to combine the third party dependencies from a pre-built PEX into a new PEX; namely, that the PEXes must use the same value for the --pre-install-wheels option.

Finally, this release fixes pex3 venv to handle venvs created by Virtualenv on systems that distinguish purelib and platlib site-packages directories. Red Hat distributions are a notable example of this.

• Implement pex3 lock sync. (#2373)
• Guard against mismatched --requirements-pex. (#2392)
• Fix pex --no-build --lock .... (#2390)
• Fix Pex to handle venvs with multiple site-packages dirs. (#2383)

This release fixes pex3 lock create to handle .tar.bz2 and .tgz sdists in addition to the officially sanctioned .tar.gz and (less officially so) .zip sdists.

• Handle .tar.bz2 & .tgz sdists when locking. (#2380)

This release trims down the size of the Pex wheel on PyPI and the released Pex PEX by about 20KB by consolidating image resources.

This release also fixes the release process to remove a window of time when several links would be dead on at https://docs.pex-tool.org that pointed to release artifacts that were not yet fully deployed.

• Fix release ordering of the doc site deploy. (#2369)
• Trim embedded doc image assets. (#2368)

This release adds tools to interact with Pex's new embedded offline documentation. You can browse those docs with pex --docs or, more flexibly, with pex3 docs. See pex3 docs --help for all the options available.

This release also returns to SemVer versioning practices. Simply, you can expect 3 things from Pex version numbers:

• The first component (the major version) will remain 2 as long as possible. Pex tries very hard to never break existing users and to allow them to upgrade without fear of breaking. This includes not breaking Python compatibility. In Pex 2, Python 2.7 is supported as well as Python 3.5+ for both CPython and PyPy. Pex will only continue to add support for new CPython and PyPy releases and never remove support for already supported Python versions while the major version remains 2.
• The second component (the minor version) will be incremented whenever a release adds a feature. Since Pex is a command line tool only (not a library), this means you can expect a new subcommand, a new option, or a new allowable option value was added. Bugs might also have been fixed.
• The third component (the patch version) indicates only bugs were fixed.

You can expect the minor version to get pretty big going forward!

• Add pex --docs and several pex3 docs options. (#2365)

This release moves Pex documentation from https://pex.readthedocs.io to https://docs.pex-tool.org. While legacy versioned docs will remain available at RTD in perpetuity, going forward only the latest Pex release docs will be available online at the https://docs.pex-tool.org site. If you want to see the Pex docs for the version you are currently using, Pex now supports the pex3 docs command which will serve the docs for your Pex version locally, offline, but with full functionality, including search.

• Re-work Pex documentation. (#2362)

This release fixes Pex to work in certain OS / SSL environments where it did not previously. In particular, under certain Fedora distributions using certain Python Build Standalone interpreters.

• Create SSLContexts in the main thread. (#2356)

This release adds support for --pip-version 24.0 as well as fixing a bug in URL encoding for artifacts in lock files. Notably, torch's use of local version identifiers (+cpu) combined with their find links page at https://download.pytorch.org/whl/torch_stable.html would lead to pex3 lock create errors.

• Add support for Pip 24.0. (#2350)
• Fix URL escaping for lock artifacts. (#2349)

This release adds support for --only-wheel <project name> and --only-build <project name> to allow finer control over which distribution artifacts are resolved when building a PEX or creating or updating a lock file. These options correspond to Pip's --only-binary and --no-binary options with project name arguments.

• Plumb Pip's --{no,only}-binary. (#2346)

This release adds the ability for pex3 lock update to replace requirements in a lock or delete them from the lock using -R / --replace-project and -d / --delete-project, respectively.

• Lock updates support deleting & replacing reqs. (#2335)

This release brings a fix for leaks of Pex's vendored attrs onto the sys.path of PEXes during boot in common usage scenarios.

• Fix vendored attrs sys.path leak. (#2328)

This release adds support for tab completion to all PEX repls running under Pythons with the readline module available. This tab completion support is on-par with newer Python REPL out of the box tab completion support.

• Add tab-completion support to PEX repls. (#2321)

This release fixes a bug in pex3 lock update for updates that leave projects unchanged whose primary artifact is an sdist.

• Fix lock updates for locks with sdist bystanders. (#2325)

This release optimizes wheel install overhead for warm caches. Notably, this speeds up warm boot for PEXes containing large distributions like PyTorch as well as creating venvs from them.

• Lower noop wheel install overhead. (#2315)

This release brings support for --pip-version 23.3.2 along with optimizations that reduce built PEX size for both --include-tools and --venv PEXes (which includes the Pex PEX) as well as reduce PEX build time for --pre-install-wheels PEXes (the default) and PEX cold first boot time for --no-pre-install-wheels PEXes that use more than one parallel install job.

• Add support for Pip 23.3.2. (#2307)
• Remove Pip.spawn_install_wheel & optimize. (#2305)
• Since we no longer use wheel code, remove it. (#2302)

This release brings three new features:

1. When creating PEXes without specifying an explicit --python-shebang, an appropriate shebang is chosen correctly in more cases than previously and a warning is emitted when the shebang chosen cannot be guaranteed to be correct. The common case this helps select the appropriate shebang for is PEXes built using --platform or --complete-platform.
2. PEXes can now be created with --no-pre-install-wheels to cut down PEX build times with a tradeoff of roughly 10% greater boot overhead upon the 1st execution of the PEX file. For PEXes with very large dependency sets (machine learning provides common cases), the build time savings can be dramatic.
3. PEXes can now be told to install dependencies at runtime on 1st execution using parallel processes using --max-install-jobs at PEX build time or by setting the PEX_MAX_INSTALL_JOBS environment variable at runtime.

The last two features come with complicated tradeoffs and are turned off by default as a result. If you think they might help some of your use cases, there is more detail in the command line help for --no-pre-install-wheels and --max-install-jobs as well as in the pex --help-variables output for PEX_MAX_INSTALL_JOBS. You can also find a detailed performance analysis in #2292 for the extreme cases of very small and very large PEXes. In the end though, experimenting is probably your best bet.

• Use appropriate shebang for multi-platform PEXes. (#2296)
• Add support for --no-pre-install-wheels and --max-install-jobs. (#2298)

This release fixes Pex runtime sys.path scrubbing to do less work and thus avoid errors parsing system installed distributions with bad metadata.

• Remove Pex runtime scrubbing dist discovery. (#2290)

This release fixes the computation of the hash of the code within a PEX when nested within directories, a bug introduced in 2.1.149.

• Exclude pyc dirs, not include, when hashing code (#2286)

This release brings support for a new --exclude <req> PEX build option that allows eliding selected resolved distributions from the final PEX. This is an advanced feature that will, in general, lead to broken PEXes out of the box; so read up on the --exclude command line help to make sure you understand the consequences.

This release also brings a fix for --inject-env that ensures the specified environment variables are always injected to the PEX at runtime regardless of the PEX entry point exercised.

• Implement support for --exclude <req>. (#2281)
• Relocate environment variable injection to before the interpreter is run (#2260)

This release brings support for --pip-version 23.3.1.

• Add support for Pip 23.3.1. (#2276)

Fix --style universal lock handing of none ABI wheels with a specific Python minor version expressed in their wheel tag. There are not many of these in the wild, but a user discovered the case of python-forge 18.6.0 which supplies 1 file on PyPI: python_forge-18.6.0-py35-none-any.whl.

• Fix universal lock handling of the none ABI. (#2270)

Add support to the Pex for checking if built PEXes are valid Python zipapps. Currently, Python zipapps must reside in 32 bit zip files due to limitations of the stdlib zipimport module's zipimporter; so this check amounts to a check that the built PEX zip does not use ZIP64 extensions. The check is controlled with a new --check {none,warn,error} option, defaulting to warn.

• Add --check support for zipapps. (#2253)

Add support for --use-pip-config to allow the Pip Pex calls to read PIP_* env vars and Pip configuration files. This can be particularly useful for picking up custom index configuration (including auth).

• Add support for --use-pip-config. (#2243)

This release brings a fix by new contributor @yjabri for the __pex__ import hook that gets it working properly for --venv mode PEXes.

• Fix non executable venv sys path bug (#2236)

This release broadens the range of the flit-core build system Pex uses to include 3.x, which is known to work for modern Python versions and Pex's existing build configuration.

• Raise the flit-core limit for Python 3 (#2229)

This release fixes Pex to build PEX files with deterministic file order regardless of the operating system / file system the PEX was built on.

• Traverse directories in stable order when building a PEX (#2220)

This release fixes Pex to work by default under eCryptFS home dirs.

• Guard against too long filenames on eCryptFS. (#2217)

This release fixes Pex to handle Pip backtracking due to sdist build errors when attempting to extract metadata.

• Handle backtracking due to sdist build errors. (#2213)

This release fixes the Pex CLI to work when run from a read-only installation. A prominent example of this comes in certain nix setups.

• Fix the Pex CLI to work when installed read-only. (#2205)

This release fixes several spurious warnings emitted for Python 3.11 and 3.12 users and fixes a leak of Pex's vendored attrs when using the __pex__ import hook.

• Eliminate warnings for default use. (#2188)
• Cleanup sys.path after pex is imported. (#2189)

This release brings support for Python 3.12 and Pip 23.2 which is the minimum required Pip version for Python 3.12. N.B.: Since Pip 23.2 requires Python 3.7 or newer, multiplatform PEX files and locks that support Python 3.12 will not also be able to support Python 2.7, 3.5 or 3.6 even though Pex continues to support those versions generally.

In addition, two new options for adding local project source files to a pex are added: -P/--package and -M/--module. Importantly, you can use the options instead of the existing -D/--sources-directory when you have a project with code at the top level (i.e.: not in a src/ subdirectory for example) intermixed with other files you prefer not to include in the PEX. See pex --help for more details on using these new options.

Finally, an internal API is fixed that allows for Lambdex files to include versions of attrs incompatible with Pex's own vendored version.

• Add official support for Python 3.12 / Pip 23.2. (#2176)
• Add support for selecting packages and modules. (#2181)
• Fix pex.pex_bootstrapper.bootstrap_pex_env leak. (#2184)

This release brings fixes for two obscure corner cases.

Previously, if you used --venv PEXes in the default symlinked site-packages mode that contained first party code in a namespace package shared with 3rd-party dependencies the first party code would contaminate the Pex installed wheel cache for one of the 3rd-party dependencies in PEX.

Even more obscure (the only known issue was in Pex's own CI), if you ran the Pex CLI concurrently using two different --pip-version arguments, you may have seen spurious Pip HTTP errors that found an invalid Content-Type: Unknown header.

• Isolate the Pip cache per Pip version. (#2164)
• Fix symlinked venv ns-package calcs. (#2165)

This release fixes a long-standing bug in lock file creation for exotic locking scenarios pulling the same project from multiple artifact sources (any mix of URLs, VCS and local project directories).

• Fix inter-artifact comparisons. (#2152)

This release adds the pex3 lock export-subset command. This is a version of pex3 lock export that also accepts requirements arguments allowing just a subset of the lock satisfying the given requirements to be exported.

• Add pex3 lock export-subset. (#2145)

This release brings support for pex3 venv {inspect,create} for working with venvs directly using Pex. Previously, a PEX built with --include-tools (or --venv) had the capability of turning itself into a venv but the new pex3 venv create command can do this for any PEX file with the addition of a few new features:

1. The venv can now be created directly from requirements producing no intermediate PEX file.
2. The venv can be created either from a PEX file or a lock file. A subset of either of those can be chosen by also supplying requirements.
3. Instead of creating a full-fledged venv, just the site-packages can be exported (without creating an intermediate venv). This "flat" layout is used by several prominent runtimes - notably AWS Lambda -and emulates pip install --target. This style layout can also be zipped and prefixed. Additionally, it supports --platform and --complete-platform allowing creation of, for example, an AWS Lambda (or Lambda Layer) deployment zip on a non-Linux host.

Additionally, this release adds support for Pip 23.1.1 and 23.1.2.

• Add Support for Pip 23.1.1. (#2133)
• Introduce pex3 venv inspect. (#2135)
• Introduce pex3 venv create. (#2140)
• Add support for Pip 23.1.2. (#2142)

This release fixes pex3 lock create gathering of sdist metadata for PEP-517 build backends with non-trivial get-requires-for-build-wheel requirements.

• Use get_requires_for_build_wheel for metadata prep. (#2129)

This release fixes --venv mode PEX venv script shebangs for some scenarios using Python <=3.7 interpreters.

• Fix venv script shebangs. (#2122)

This release brings support for the latest Pip release with --pip-version 23.1 or by using new support for pinning to the latest version of Pip supported by Pex with --pip-version latest.

• Add support for Pip 23.1 (#2114)
• Add support for --pip-version latest. (#2116)

This release fixes some inconsistencies in Pex JSON output across the Python 2/3 boundary and in handling of venv collisions when using the venv Pex tool.

• Stabilize JSON output format across Python 2/3. (#2106)
• Support --pip overrides via PEX deps. (#2107)

This release fixes a regression locking certain complex cases of direct and transitive requirement interactions as exemplified in #2098.

• Guard lock analysis against Pip-cached artifacts. (#2103)

This release fixes a bug downloading a VCS requirement from a lock when the ambient Python interpreter used to run Pex does not meet the Requires-Python constraint of the VCS requirement.

• Fix VCS lock downloads to respect target. (#2094)

This release fixes a regression introduced in Pex 2.1.120 that caused --no-venv-site-packages-copies (the default when using --venv) to be ignored for both zipapp PEXes (the default) and --layout packed PEXes.

• Fix regression in venv symlinking. (#2090)

This release fixes --lock resolve sub-setting for local project requirements.

• Fix lock subsetting for local projects. (#2085)

This release fixes a long-standing (> 4 years old!) concurrency bug when building the same sdist for the 1st time and racing another Pex process doing the same sdist build.

• Guard against racing sdist builds. (#2080)

This release makes --platform and --complete-platform resolves and locks as permissive as possible. If such a resolve or lock only has an sdist available for a certain project, that sdist will now be used if it builds to a wheel compatible with the specified foreign platform(s).

• Attempt "cross-builds" of sdists for foreign platforms. (#2075)

This release adds support for specifying --non-hermetic-venv-scripts when building a --venv PEX. This can be useful when integrating with frameworks that do setup via PYTHONPATH manipulation.

Support for Pip 23.0.1 and setuptools 67.4.0 is added via --pip-version 23.0.1.

Additionally, more work towards hardening Pex against rare concurrency issues in its atomic directory handling is included.

• Introduce --non-hermetic-venv-scripts. (#2068)
• Wrap inter-process locks in in-process locks. (#2070)
• Add support for Pip 23.0.1. (#2072)

This release fixes a few pex3 lock create bugs.

There was a regression introduced in Pex 2.1.122 where projects that used a PEP-518 [build-system] requires but specified no corresponding build-backend would fail to lock.

There were also two long-standing issues handling more exotic direct reference URL requirements. Source archives with names not following the standard Python sdist naming scheme of <project name>-<version>.{zip,tar.gz} would cause a lock error. An important class of these is provided by GitHub's magic source archive download URLs. Also, although local projects addressed with Pip proprietary support for pure local path requirements would lock, the same local projects addressed via <project name> @ file://<local project path> would also cause a lock error. Both of these cases are now fixed and can be locked successfully.

When locking with an --interpreter-constraint, any resolve traversing wheels using the pypyXY or cpythonXY python tags would cause the lock to error. Wheels with this form of python tag are now handled correctly.

• Handle [build-system] with no build-backend. (#2064)
• Handle locking all direct reference URL forms. (#2060)
• Fix python tag handling in IC locks. (#2061)

This release fixes posix file locks used by Pex internally and enhances lock creation to support locking sdist-only C extension projects that do not build on the current platform. Pex is also updated to support --pip-version 22.3.1 and --pip-version 23.0, bringing it up to date with the latest Pip's available.

• Support the latest Pip releases: 22.3.1 & 23.0 (#2056)
• Lock sdists with prepare-metadata-for-build-wheel. (#2053)
• Fix execute_parallel "leaking" a thread. (#2052)

This release fixes two bugs brought to light trying to interoperate with Poetry projects.

• Support space separated markers in URL reqs. (#2039)
• Handle file:// URL deps in distributions. (#2041)

This release completes the --complete-platform fix started in Pex 2.1.116 by #1991. That fix did not work in all cases but now does.

PEXes run in interpreter mode now support command history when the underlying interpreter being used to run the PEX does; use the PEX_INTERPRETER_HISTORY bool env var to turn this on.

Additionally, PEXes built with the combination --layout loose --venv --no-venv-site-packages-copies are fixed to be robust to moves of the source loose PEX directory.

• Fix loose --venv PEXes to be robust to moves. (#2033)
• Fix interpreter resolution when using --complete-platform with --resolve-local-platforms (#2031)
• Support REPL command history. (#2018)