execnet

• Re-release without code changes, just to include tox.ini into the source distribution.

• Re-release without code changes, just to include docs and tests into the source distribution.

• Removed support for Python < 3.7.

  • Applied pyupgrade --py37-plus.

  • Minimal mypy fixes and dropped Python 2 support code.

• Migrated packaging to hatch.

• Dropped deprecated APIs of old makegateway names.

• Removed py testing dependency.

• Explicitly pass encoding when opening files in the gateway to get rid of warnings when using PYTHONWARNDEFAULTENCODING=1 (#195).

• Fixed error when loading source code files from a path containing non-ascii characters.

• Removed the apipkg dependency.

Update CHANGELOG for 1.8.1

• Dropped support for Python 3.4.

• #118: Fixed internal leak that should make execnet execute remote code in the main thread more often; previously it would sometimes spawn a thread to execute a remote_exec call, even when the caller didn't issue multiple remote_exec calls at the same time. Some frameworks require code to execute in the main thread, so the previous behavior would break them on occasion (see pytest-dev/pytest-xdist#620 for an example).

• #115: Current working directory is now restored when calling script/socketserver.py. The script now also loops by default when called from the command-line.

Update CHANGELOG for #108

• #102: Show paths in stack traces generated by remote_exec().

• #100: Fix flaky hangs in workerpool.waitall.

• #98: Internal change to avoid using deprecated funcargs name in pytest 5+.

• execnet no longer supports Python 2.6 and 3.3 (#85). Users of those Python versions using a recent enough pip should not be affected, as pip will only install 1.5.0 for them.

• Update test suite to support pytest>4.

• support shell escaping in python pathnames of popen.

  Eugene Ciurana discovered that execnet breaks if you use pathnames with spaces in a "python=" part of a spec. We now use shlex.split to split the string. There is a potential for regressions if you used quote or escape sequences as part of your python command.

• Only insert importdir into sys.path if it is not already in the path.

  This prevents a bug when using enum34 with python 3.6 and pytest-xdist.

  The issue is that enum34 installs an 'enum' module in site-packages which is normally shadowed by the stdlib version of enum, however in gateway_bootstrap.py site-packages is added at the front the the search path. This means on the workers enum34 is hit for import enum which in turn causes import re to fail (as it makes use of the new enum features in 3.6).

• fix #49 - use inspect.getfullargspec if possible to avoid deprecationwarnings

• fix #56 - use partials in safe_terminate to avoid a bad carried binding

• fix spec parsing on Windows due to path containing '' characters.

• fix a regression of the Serializer created by the implied opcode ordering which resulted in a incompatible opcode mapping

  warning stored serialized objects created with 1.4.0 are incompatible with previous versions and future versions additionally stored serialized objects containing complex objects will have a incompatible opcode when read with execnet < 1.4.0 and won't be loadable with execnet 1.4.0 either

  its strongly suggested to avoid using the Serializer of execnet 1.4.0 this affects devpi and the external pytest-cache plugin

• introduce execnet.dumps/loads providing serialization between python interpreters.

• group.remote_exec now supports kwargs as well

• support per channel string coercion configuration

• Popen2IO.read now reads correct amounts of bytes from nonblocking fd's

• added a dont_write_bytecode option to Popen gateways, this sets the sys.dont_write_bytecode flag on the spawned process, this only works on CPython 2.6 and higher. Thanks to Alex Gaynor.

• added a pytest --broken-isp option to skip tests that assume DNS queries for unknown hosts actually are resolved as such (Thanks Alex Gaynor)

• fix issue 1 - decouple string coercion of channels and gateway

• fix issue #2 - properly reconfigure the channels string coercion for rsync, so it can send from python2 to python3

• fix issue #9 - properly terminate the worker threadpools in safe_terminate

• fix issue #8 - no longer kill remote pids locally on jython ssh gateways

• refactor socketserver, so it can be directly remote_exec'd for starting a socket gateway on a remote

• add gw.reconfigure() to configure per gateway options. Currently supported: py2str_as_py3str and py3str_as_py2str to configure string deserialization

• channel.makefile() objects now have a isatty() returning False

• group.allocate_id(spec) allows to early-determine an (automatic) id

• internal refactorings and cleanups (thanks Ronny Pfannschmidt): - refactor message types into received handler functions - refactor b(chr(opcode)) to bchr(opcode) - reorder Message ctor args, rename msgtype to msgcode - refactor gateway.send to take message's init args instead of a message - inline and remove Message.writeto/readfrom - refactor collection loading to avoid the indirection over tuple - remove the unused NamedThreadPool

• new gateway.remote_exec(func, **kwargs) style for executing a pure function with parameters. The function on the remote side also needs to accept a channel which allows it to communicate back and forth. Thanks to Ronny Pfannschmidt for implementing it with special kudos to Maciej Fijalkowski for writing a "pure-function" checker so that on Python2.6 onwards non-pure functions will be rejected.

• enhance rsyncing to also sync permissions (stat().st_mode) of directories and files. (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/68/)

• fix rsyncing of symlinks, thanks to Charles Solar (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/70/)

• update internal usage of apipkg to 1.0b6

• remote_exec(module) now makes sure that the linecache is updated before reading and sending the source. thanks Ronny, Matt.

• removed all trailing whitespace from source files

• try to avoid a random KeyboardInterrupt Error when threads are ending.

• extend xspec syntax to allow for one or multiple "env:NAME=value" environment variable settings which will be set on the remote side. (thanks Jakub Gustak)

• fix jython/windows interactions

• fix waitclose/callback-with-endmarker race condition

• fix race condition where multiple threads sending data over channels would crash the serializer and process

• more care during receiver-thread finalization during interp-shutdown, should get rid of annoying and meaningless exceptions

• fix glitch in ssh-fileserver example

• experimentally add "setup.py test" support - will run py.test

• try to deal more cleanly with interpreter shutdown setting globals to None. this avoids (hopefully) some bogus tracebacks at process exit.

• refine termination some more: CTRL-C and gateway.exit will now try harder to interrupt remote execution. this helps to avoid left-over ssh-processes.

• fix read-on-non-blocking-files issue probably related to jython only: the low-level read on subprocess pipes may be non-blocking, returning less bytes than requested - so we now loop.

• Windows/python2.4: fix bug that killing subprocesses would fail

• make RemoteError and TimeoutError available directly on execnet namespace

• fix some doc and test issues (thanks thm and ronny), add ssh_fileserver example

• update internal copy of apipkg

• always skip remote tests if no ssh specs given

• generalize channel-over-channel sending: you can now have channels anywhere in a data structure (i.e. as an item of a container type). Add according examples.

• automatically close a channel when a remote callback raises an exception, makes communication more robust because until now failing callbacks rendered the receiverthread unusable leaving the remote side in-accessible.

• internally split socket gateways, speeds up popen-gateways by 10% (now at <50 milliseconds per-gateway on a 1.5 GHZ machine)

• fix bug in channel.receive() that would wrongly raise a TimeoutError after 1000 seconds (thanks Ronny Pfannschmidt)

• revamp and better structure documentation

• new method: gateway.hasreceiver() returns True if the gateway is still receive-active. remote_status now only carries information about remote execution status.

• new: execnet.MultiChannel provides basic iteration/contain interface

• new: execnet.Group can be indexed by integer

• new: group.makegateway() uses group.default_spec if no spec is given and the execnet.default_group uses popen as a default spec.

• have popen-gateways use imports instead of source-strings, also improves debugging/tracebacks, as a side effect popen-gateway startup can be substantially faster (>30%)

• refine internal gateway exit/termination procedure and introduce group.terminate(timeout) which will attempt to kill all subprocesses that did not terminate within time.

• EOFError on channel.receive/waitclose if the other side unexpectedly went away. When a gateway exits it now internally sends an explicit termination message instead of abruptly closing.

• introduce a timeout parameter to channel.receive() and default to periodically internally wake up to let KeyboardInterrupts pass through.

• EXECNET_DEBUG=2 will cause tracing to go to stderr, which with popen worker gateways will relay back tracing to the instantiator process.

• introduce execnet.Group for managing gateway creation and termination. Introduce execnet.default_group through which all "global" calls are routed. cleanup gateway termination. All Gateways get an id through which they can be retrieved from a group object.

• deprecate execnet.XYZGateway in favour of direct makegateway() calls.

• refine socketserver-examples, experimentally introduce a way to indirectly setup a socket server ("installvia") through a gateway url.

• refine and automatically test documentation examples

• fix EXECNET_DEBUG to work with win32

• add support for serializing longs, sets and frozensets (thanks Benjamin Peterson)

• introduce remote_status() method which on the low level gives information about the remote side of a gateway

• disallow explicit close in remote_exec situation

• perform some more detailed tracing with EXECNET_DEBUG

• make internal protocols more robust against serialization failures

• fix a serialization bug with nested tuples containing empty tuples (thanks to ronny for discovering it)

• setting the environment variable EXECNET_DEBUG will generate per process trace-files for debugging

• added new examples for NumPy, Jython, IronPython

• improved documentation

• include apipkg.py for lazy-importing

• integrated new serializer code from Benjamin Peterson

• improved support for Jython-2.5.1

• improve documentation, new website

• use sphinx for documentation, added boilerplate files and setup.py

• fixes for standalone usage, adding boilerplate files

• imported py/execnet and made it work standalone