awkward

Another batch of November bug-fixes:

@nsmith- made mixin classes derived from a decorator pickleable (#542).

@ianna fixed a bug in sort/argsort (#524) and concatenate (#548).

@reikdas added a script to clean up generated files (#556).

@jpivarski generalized ak.where to take a non-NumPy condition (#544), implemented __contains__ in and out of Numba (#547), np.array(ak.Array) in Numba (#550), removed the right-broadcast from ak.with_field (#553), implemented ak.to_regular and ak.from_regular (#551), updated the GitHub issue templates, and fixed a bug affecting Coffea (https://github.com/CoffeaTeam/coffea/issues/372, PR #558).

Mostly bug-fixes, but a few new functions:

• PR #495: @reikdas added a test to ensure kernel signatures match specification, are in order
• PR #517: masked __getitem__ on empty arrays
• PR #518: fixed ak.to_pandas with IndexedArrays and other nodes
• PR #519: ak.local_index
• PR #520: actually removed long-deprecated Pandas code
• PR #525: fixed the definition of Form::getitem_field
• PR #526: infinite recursion bug in Arrow translation
• PR #527: definition of "mergeable" with parameters (different __doc__ is mergeable, different __record__ is not)
• PR #535: ak.zeros_like, ak.ones_like, ak.full_like
• PR #536: enforce minimum pyarrow version (2.0.0)
• PR #537: fixed matrix multiplication (multiply many differently sized matrices with one @!)
• PR #539: @ianna implemented ak.concatenate for axis > 0!
• PR #540: performance issues in __setitem__

Calling ufuncs on any custom types (not just records) will raise errors unless a customization is defined for exactly that ufunc and type. Some custom types, like "categorical," need to handle arbitrary ufuncs, so I also added a "apply_ufunc" interface to define behaviors across the board. (#513)

Added copy and deepcopy, both with the Python interface (copy module) and with the NumPy interface (np.copy). (#514)

Fixed the cache-handling of version 0.4.2. (#515)

Mutable ak.Array.cache has been replaced with immutable ak.Array.caches, which always contains all the VirtualArray caches in the whole layout; it no longer needs to be maintained by users.

This is an interface-breaking change (with deprecation warnings), and a new release is needed to bring uproot4 up to date with the change.

Broadcasting ufuncs across records is now deprecated, and will be removed in version 1.0.0 (December 1, 2020).

Fixed two bugs (#501 and #499).

Supports Python 3.9. (Now MacOS only supports Pythons 2.7, 3.7-3.9 (3.6 dropped). All other operating systems support Pythons 2.7, 3.5-3.9)

Based on pybind 2.6.0. This is the first jump in pybind11 version in a while.

As discussed in #350, Awkward arrays can no longer be used as Pandas columns. Also part of this API change (#464),

• array.tojson has been removed: use ak.to_json(array).
• array.columns has been renamed: use array.fields.
• ak.keys(array) has been renamed: use ak.fields(array). (ak.keys still exists with a "deprecated until 0.4.0" message)
• array.ndim (property) is now the dimension of the array, rather than 1 (a Pandas requirement)
• array.type has been added; it is equivalent to ak.type(array). If you had fields named "type", they will now have to be accessed as array["type"], rather than array.type.

Thanks to @reikdas, this is the first release with a formal specification for the cpu-kernels/cuda-kernels library! (The bottom layer of the 3-layer architecture, documented here.)

Furthermore, ufunc behavioral matching has been generalized (#463) to allow:

ak.behavior[np.multiply, numbers.Number, "Vector"] = scalar_multiply
ak.behavior[np.multiply, "Vector", numbers.Number] = scalar_multiply

Two mistakes were fixed in the Windows builds (#465 by @chrisburr and #467), allowing them to be used without debug-version system libraries (#430). The first of these enabled @lgray to deploy it on conda-forge!

Still built on pybind 2.4.3, though we might want to update that, especially if pybind supports the NumPy datetime dtypes.

PR #147, built on pybind 2.4.3.

PR #140, built on pybind 2.4.3.

By incrementing the minor version number, we break the rough connection we had between PR numbers and patch version numbers. This represents the fact that we're entering a phase in which the work is driven by issues (which are not sequential), rather than non-issue PRs (which can be sequential enough).

Up through PR #4: NumpyArray and ListOffsetArray usable in C++, Python, and Numpy. C++ and Python have an Identity framework for particle kinematics (see combinatorics language). The dummy1 and dummy3 functions tested in deployments 0.0.x have been removed (Identity has some real code in cpu-kernels).