pex

This release fixes a bug introduced in Pex 2.1.109 where the released Pex PEX could not be executed by PyPy interpreters. More generally, any PEX created with interpreter constraints that did not specify the Python implementation, e.g.: ==3.8.*, were interpreted as being CPython specific, i.e.: CPython==3.8.*. This is now fixed, but if the intention of a constraint like ==3.8.* was in fact to restrict to CPython only, interpreter constraints need to say so now and use CPython==3.8.* explicitly.

• Fix interpreter constraint parsing. (#1998)

This release fixes a bug in --resolve-local-platforms when --complete-platform was used.

• Check for --complete-platforms match when --resolve-local-platforms (#1991)

This release brings some attention to the pex3 lock export subcommand to make it more useful when interoperating with pip-tools.

• Sort requirements based on normalized project name when exporting (#1992)
• Use raw version when exporting (#1990)

This release brings two fixes for --venv mode PEXes.

• Only insert "" to head of sys.path if a venv PEX runs in interpreter mode (#1984)
• Map pex python path interpreter to realpath when creating venv dir hash. (#1972)

This is a hotfix release that fixes errors installing wheels when there is high parallelism in execution of Pex processes. These issues were a regression introduced by #1961 included in the 2.1.112 release.

• Restore AtomicDirectory non-locked good behavior. (#1974)

This release brings support for the latest Pip release and includes some internal changes to help debug intermittent issues some users are seeing that implicate what may be file locking related bugs.

• Add support for --pip-version 22.3. (#1953)

This release fixes resolving requirements from a lock using arbitrary equality (===).

In addition, you can now "inject" runtime environment variables and arguments into PEX files such that, when run, the PEX runtime ensures those environment variables and command line arguments are passed to the PEXed application. See PEX Recipes for more information.

• Fix lock resolution to handle arbitrary equality. (#1951)
• Support injecting args and env vars in a PEX. (#1948)

This release fixes Pex runtime sys.path scrubbing for cases where Pex is not the main entry point. An important example of this is in Lambdex where the AWS Lambda Python runtime packages (boto3 and botocore) are leaked into the PEX runtime sys.path.

• Fix sys.path scrubbing. (#1946)

This release brings musllinux wheel support and a fix for a regression introduced in Pex 2.1.105 by #1902 that caused PEX_PATH= (an exported PEX_PATH with an empty string value) to raise an error in almost all use cases.

• Vendor latest packaging; support musllinux wheels. (#1937)
• Don't treat PEX_PATH= as . like other PATHS. (#1938)

This release fixes a latent PEX boot performance bug triggered by requirements with large extras sets.

• Fix slow PEX boot time when there are many extras. (#1929)

This release fixes an issue handling credentials in git+ssh VCS urls when creating locks.

• Fix locks for git+ssh with credentials. (#1923)

This release fixes a long-standing bug in handling direct reference requirements with a local version component.

• Unquote path component of parsed url requirements (#1920)

This is a fix release which addresses issues related to build time work_dir creation, virtualenv, and sh_boot support.

In the unlikely event of a UUID collision in atomic workdir creation, pex could overwrite an existing directory and cause a corrupt state. When building a shell bootable --sh-boot pex the --runtime-pex-root was not always respected based on the condition of the build environment, and the value of the PEX_ROOT.

• Fail on atomic_directory work_dir collision. (#1905)
• Use raw_pex_root when constructing sh_boot pexes. (#1906)
• Add support for offline downloads (#1898)

This release brings a long-awaited upgrade of the Pip Pex uses, but behind a --pip-version 22.2.2 flag you must opt in to. Pex will then use that version of Pip if it can (your Pex operations target Python >=3.7) and warn and fall back to the older vendored Pip (20.3.4) if it can't. To turn the need to fall back to older Pip from a warning into a hard error you can also specify --no-allow-pip-version-fallback.

The pex3 lock update command now gains the ability to update just the index and find links repos the lock's artifacts originate from by using a combination of --no-pypi, --index & --find-links along with --pin to ensure the project versions stay pinned as they are in the lockfile and just the repos they are downloaded from is altered. Consult the CLI --help for --fingerprint-mismatch {ignore,warn,error} to gain more control over repo migration behavior.

There are several bug fixes as well dealing with somewhat esoteric corner cases involving changing a PEX --layout from one form to another and building artifacts using certain interpreters on macOS 11.0 (aka: 10.16).

• Add support for Pip 22.2.2. (#1893)
• Make lock update sensitive to artifacts. (#1887)
• Ensure locally built wheel is consumable locally. (#1886)
• Ensure --output always overwrites destination. (#1883)

This release fixes things such that pex lockfiles can be created and updated using the Pex PEX when local projects are involved.

• Fix pex3 lock ... when run from the Pex PEX. (#1874)

This is a hotfix release that fixes a further corner missed by #1863 in the Pex 2.1.101 release whereby Pex would fail to install platform-specific packages on Red Hat based OSes.

In addition, an old but only newly discovered bug in --inherit-path={prefer,fallback} handling is fixed. Previously only using PEX_INHERIT_PATH={prefer,fallback} at runtime worked properly.

In the process of fixing the old --inherit-path={prefer,fallback} bug, also fix another old bug handling modern virtualenv venvs under Python 2.7 during zipapp execution mode PEX boots.

• Fix wheel installs: account for purelib & platlib. (#1867)
• Fix --inhert-path handling. (#1871)
• Error using pex + virtualenv>=20.0.0 + python 2.7 (#992)

This release fixes a corner-case revealed by python-certifi-win32 1.6.1 that was not previously handled when installing certain distributions.

• Make wheel install site-packages detection robust. (#1863)

This release fixes a hole in the lock creation --target-system feature added in #1823 in Pex 2.1.95.

• Fix lock creation --target-system handling. (#1858)

This release fixes a concurrency bug in the pex --lock ... artifact downloading.

• Fix pex --lock ... concurrent download errors. (#1854)

This releases fixes regressions in foreign --platform handling and artifact downloading introduced by #1787 in Pex 2.1.91 and #1811 in 2.1.93.

In addition, PEXes can now be used as sys.path entries. Once on the sys.path, via PYTHONPATH or other means, the code in the PEX can be made importable by first importing __pex__ either as its own stand-alone import statement; e.g.: import __pex__; import psutil or as a prefix of the code to import from the PEX; e.g.: from __pex__ import psutil.

• Tags should be patched for --platform. (#1846)
• Add support for importing from PEXes. (#1845)
• Fix artifact downloads for foreign platforms. #1851

This release patches a hole left by #1828 in the Pex 2.1.95 release whereby, although you could run a PEX under a too-long PEX_ROOT you could not build a PEX under a tool-long PEX_ROOT.

• Avoid ENOEXEC for Pex internal --venvs. (#1843)

This is a hotfix release that fixes --venv mode PEX_EXTRA_SYS_PATH propagation introduced in Pex 2.1.95 to only apply to sys.executable and not other Pythons.

• Fix --venv PEX PEX_EXTRA_SYS_PATH propagation. (#1837)

This release brings two new pex3 lock features for --style universal locks.

By default, universal locks are created to target all operating systems. This can cause problems when you only target a subset of operating systems and a lock transitive dependency that is conditional on an OS you do not target is not lockable. The new --target-system {linux,mac,windows} option allows you to restrict the set of targeted OSes to work around this sort of issue. Since PEX files currently only support running on Linux and Mac, specifying --target-system linux --target-system mac is a safe way to pre-emptively avoid these sorts of locking issues when creating a universal lock.

Previously you could not specify the --platforms or --complete-platforms you would be using later to build PEXes with when creating a universal lock. You now can, and Pex will verify the universal lock can support all the specified platforms.

As is usual there are also several bug fixes including properly propagating PEX_EXTRA_SYS_PATH additions to forked Python processes, fixing pex3 lock export to only attempt to export for the selected target and avoiding too long shebang errors for --venv mode PEXes in a robust way.

• Fix PEX_EXTRA_SYS_PATH propagation. (#1832)
• Fix pex3 lock export: re-use --lock resolver. (#1831)
• Avoid ENOEXEC for --venv shebangs. (#1828)
• Check lock can resolve platforms at creation time. (#1824)
• Support restricting universal lock target os. (#1823)

This is a hotfix release that fixes a regression introduced in Pex 2.1.93 downloading certain sdists when using pex --lock ....

• Fix pex --lock ... handling of sdists. (#1818)

This release brings several new features in addition to bug fixes.

When creating a PEX the entry point can now be any local python script by passing --exe path/to/python-script.

The pex3 lock update command now supports a -dry-dun check mode that exits non-zero to indicate that a lock needs updating and the -p / --project targeted update arguments can now be new projects to attempt to add to the lock.

On the bug fix front, traditional zipapp mode PEX files now properly scrub sys.displayhook and sys.excepthook and their teardown sequence has now been simplified fixing logging to stderr late in teardown.

Finally, pex3 lock create now logs when requirement resolution is taking a long time to provide some sense of progress and suggest generic remedies and pex --lock now properly handles authentication.

• Support adding new requirements in a lock update. (#1797)
• Add pex3 lock update --dry-run check mode. (#1799)
• Universal locks no longer record a platform_tag. (#1800)
• Support python script file executable. (#1807)
• Fix PEX scrubbing to account for sys.excepthook. (#1810)
• Simplify PEX teardown / leave stderr in tact. (#1813)
• Surface pip download logging. (#1808)
• Use pip download instead or URLFetcher. (#1811)

This release adds support for locking local projects.

• Add support for local project locking. #1792

This release fixes --sh-boot mode PEXes to have an argv0 and exported PEX environment variable consistent with standard Python boot PEXes; namely the absolute path of the originally invoked PEX.

• Fix --sh-boot argv0. (#1785)

This release fixes Pex handling of sdists to be atomic and also fixes lock files to be emitted ending with a newline. In addition, many typos in Pex documentation were fixed in a contribution by Kian-Meng Ang.

• Ensure Pip cache operations are atomic. (#1778)
• Ensure that lockfiles end in newlines. (#1774)
• Fix typos (#1773)

This release brings official support for CPython 3.11 and PyPy 3.9 as well as long needed robust runtime interpreter selection.

• Select PEX runtime interpreter robustly. (#1770)
• Upgrade PyPy checking to latest. (#1767)
• Add 3.11 support. (#1766)

This release is a hotfix for 2.1.86 that handles unparseable ~/.netrc files gracefully.

• Just warn when ~/.netrc can't be loaded. (#1763)

This release fixes pex3 lock create to handle relative --tmpdir.

• Fix lock save detection to be more robust. (#1760)

This release fixes an oversight in lock file use against secured custom indexes and find links repos. Previously credentials were passed during the lock creation process via either ~/.netrc or via embedded credentials in the custom indexes and find links URLs Pex was configured with. But, at lock use time, these credentials were not used. Now ~/.netrc entries are always used and embedded credentials passed via custom URLS at lock creation time can be passed in the same manner at lock use time.

• Support credentials in URLFetcher. (#1754)

This PyCon US 2022 release brings full support for Python interpreter emulation when a PEX is run in interpreter mode (without an entry point or else when forced via PEX_INTERPRETER=1).

A special thank you to Loren Arthur for contributing the fix in the Pantsbuild sprint at PyCon.

• PEX interpreters should support all underlying Python interpreter options. (#1745)

This release fixes a bug creating a PEX from a --lock when pre-release versions are involved.

• Fix --lock handling of pre-release versions. (#1742)

This releases fixes a bug creating --style universal locks with --interpreter-constraint configured when the ambient interpreter does not match the constraints and the resolved lock includes sdist primary artifacts.

• Fix universal lock creation for ICs. (#1738)

This is a hotfix release for a regression in prerelease version handling introduced in the 2.1.81 release by #1727.

• Fix prerelease handling when checking resolves. (#1732)

This release brings a fix to Pex resolve checking for distributions built by setuptools whose Requires-Dist metadata does not match a distibutions project name exactly (i.e.: no PEP-503 [._-] normalization was performed).

• Fix Pex resolve checking. (#1727)

This release brings another fix for pathologically slow cases of lock creation as well as a new --sh-boot feature for creating PEXes that boot via /bin/sh for more resilience across systems with differing Python installations as well as offering lower boot latency.

• Support booting via /bin/sh with --sh-boot. (#1721)
• Fix more pathologic lock creation slowness. (#1723)

This release fixes --lock resolving for certain cases where extras are involved as well as introducing support for generating and consuming portable --find-links locks using -path-mapping.

• Fix --lock resolver extras handling. (#1719)
• Support canonicalizing absolute paths in locks. (#1716)

This release fixes missing artifacts in non-strict locks.

• Don't clear lock link database during analysis. (#1712)

This release fixes pathologically slow cases of lock creation as well as introducing support for --no-compression to allow picking the time-space tradeoff you want for your PEX zips.

• Fix pathologic lock creation slowness. (#1707)
• Support uncompressed PEXes. (#1705)

This release finalizes spurious deadlock handling in --lock resolves worked around in #1694 in Pex 2.1.75.

• Fix lock_resolver to use BSD file locks. (#1702)

This release fixes a deadlock when building PEXes in parallel via the new --lock flag.

• Avoid deadlock error when run in parallel. (#1694)

This release fixes multiplatform --lock resolves for sdists that are built to multiple platform specific wheels, and it also introduces support for VCS requirements in locks.

• Add support for locking VCS requirements. (#1687)
• Fix --lock for multiplatform via sdists. (#1689)

This is a hotfix for various PEX issues:

1. --requirements-pex handling was broken by #1661 in the 2.1.71 release and is now fixed.
2. Creating universal locks now works using any interpreter when the resolver version is the pip-2020-resolver.
3. Building PEXes with --lock resolves that contain wheels with build tags in their names now works.

• Fix --requirements-pex. (#1684)
• Fix universal locks for the pip-2020-resolver. (#1682)
• Fix --lock resolve wheel tag parsing. (#1678)

This release fixes an old bug with --venv PEXes initially executed with either PEX_MODULE or PEX_SCRIPT active in the environment.

• Fix venv creation to ignore ambient PEX env vars. (#1669)

This release fixes the instability introduced in 2.1.68 by switching to a more robust means of determining venv layouts. Along the way it upgrades Pex internals to cache all artifacts with strong hashes ( previously sha1 was used). It's strongly recommended to upgrade or use the exclude !=2.1.68,!=2.1.69,!=2.1.70 when depending on an open-ended Pex version range.

• Switch Pex installed wheels to --prefix scheme. (#1661)

This is another hotfix release for 2.1.68 that fixes a bug in *.data/* file handling for installed wheels which is outlined in PEP 427

• Handle *.data/* RECORD entries not existing. (#1644)

This is a hotfix release for a regression introduced in 2.1.68 for a narrow class of --venv --no-venv-site-packages-copies mode PEXes with special contents on the PEX_PATH.

• Fix venv creation for duplicate symlinked dists. (#1639)

This release brings a fix for installation of additional data files in PEX venvs (More on additional data files here) as well as a new venv install --scope that can be used to create fully optimized container images with PEXed applications (See how to use this feature here).

• Support splitting venv creation into deps & srcs. (#1634)
• Fix handling of data files when creating venvs. (#1632)

This release brings support for --platform arguments with a 3-component PYVER portion. This supports working around python_full_version environment marker evaluation failures for --platform resolves by changing, for example, a platform of linux_x86_64-cp-38-cp38 to linux_x86_64-cp-3.8.10-cp38. This is likely a simpler way to work around these issues than using the --complete-platform facility introduced in 2.1.66 by #1609.

• Expand --platform syntax: support full versions. (#1614)

This release brings a new --complete-platform Pex CLI option that can be used instead of --platform when more detailed foreign platform specification is needed to satisfy a resolve (most commonly, when python_full_version environment markers are in-play). This, paired with the new pex3 interpreter inspect command that can be used to generate complete platform data on the foreign platform machine being targeted, should allow all foreign platform PEX builds to succeed exactly as they would if run on that foreign platform as long as pre-built wheels are available for that foreign platform.

Additionally, PEXes now know how to set a usable process name when the PEX contains the setproctitle distribution. See here for more information.

• Add support for --complete-platform. (#1609)
• Introduce pex3 interpreter inspect. (#1607)
• Use setproctitle to sanitize ps info. (#1605)
• Respect PEX_ROOT in PEXEnvironment.mount. (#1599)

This release really brings support for mac universal2 wheels. The fix provided by 2.1.64 was partial; universal2 wheels could be resolved at build time, but not at runtime.

• Upgrade vendored packaging to 20.9. (#1591)

This release brings support for mac universal2 wheels.

• Update vendored Pip to 386a54f0. (#1589)

This release fixes spurious collision warnings & errors when building venvs from PEXes that contain multiple distributions contributing to the same namespace package.

• Allow for duplicate files in venv population. (#1572)

This release exposes three Pip options as Pex options to allow building PEXes for more of the Python distribution ecosystem:

1. --prefer-binary: To prefer older wheels to newer sdists in a resolve which can help avoid problematic builds.
2. --[no]-use-pep517: To control how sdists are built: always using PEP-517, always using setup.py or the default, always using whichever is appropriate.
3. --no-build-isolation: To allow distributions installed in the environment to be seen during builds of sdists. This allows working around distributions with undeclared build dependencies by pre-installing them in the environment before running Pex.

• Expose more Pip options. (#1561)

This release fixes a regression in Pex --venv mode compatibility with distributions that are members of a namespace package that was introduced by #1532 in the 2.1.57 release.

• Merge packages for --venv-site-packages-copies. (#1557)

This release fixes a bug that prevented creating PEXes when duplicate compatible requirements were specified using the pip-2020-resolver.

• Fix Pex to be duplicate requirement agnostic. (#1551)