tibs

Releases

1. ▸▾2.0.113 Aug 20262 sources agree

   Lots of new features added. A few caused some small backwardly incompatible changes, so as the user base is still small I just accepted the better API

   Lots of new features added. A few caused some small backwardly incompatible changes, so as the user base is still small I just accepted the better API and upped the major version number.

   Backwardly incompatible changes

   • Dtype is now a base class and parsing factory rather than a concrete class. Dtype("u8") returns a DtypeSingle; see DtypeArray and DtypeTuple below. Dtype strings are unchanged, but code that uses from_params, checks a dtype's exact type, or relies on its repr will need updating.

   • The View constructor and View.from_indices no longer accept a Mutibs. Use Mutibs.view() for a live MutableView, or Mutibs.to_tibs() for an immutable copy to view.

   • The minimum Python version is now 3.11 instead of 3.10.

   • Renamed set_at and unset_at to with_set and with_unset on both Tibs and Mutibs.

   • Renamed Mutibs.as_tibs to Mutibs.take_tibs and Mutibs.as_raw_data to Mutibs.take_raw_data, as both empty the object they are called on.

   • Removed to_raw_data from Tibs and Mutibs. Use to_padded_bytes() with len() to round-trip the same value, or encode(Codec.Raw) to persist it.

   • Mutibs.capacity is now a property rather than a method.

   • Mutibs.__iter__ is now None, so isinstance(m, collections.abc.Iterable) is False where it used to be True.

   • Views now compare by the bits they present rather than by their whole source, so a View and a MutableView over the same bits and layout are now equal.

   • The buffer protocol now needs a length that is a multiple of 8, and raises BufferError otherwise. It used to export a partial final byte, whose bits past the logical length are not padding but whatever the storage was sliced out of, so Tibs('0xffff')[0:4] and Tibs('0b1111') presented different bytes despite being equal. Use to_padded_bytes() for those.

   • The offset and length parameters of Tibs.from_bytes and Mutibs.from_bytes are now bit_offset and bit_length. They always counted bits, which the old names left to the reader to guess on a method that takes bytes.

   • Assigning to a single index now takes only True, False, 0 or 1, and raises TypeError for anything else, rather than testing the value for truthiness. This is the rule append and the implicit bit patterns already followed. Use bool(x) at the call site to keep the old behaviour.

   Added

   • Added the Reader class, which wraps a Tibs or Mutibs with a bit position for reading fields in sequence.

     >>> r = Reader(Tibs('0x47ff10'))
     >>> r.read_value('u8')
     71
     >>> r.read_value('(bool, u7)')
     (True, 127)
     >>> r.remaining
     8
     

   • Added DtypeArray and DtypeTuple alongside the scalar dtype (now DtypeSingle), so that a single Dtype can describe a structured, multi-field value: Dtype("[u8; 4]") for repeated fields and Dtype("(u8, u16_le, bool)") for differently-typed ones. Both nest to any depth.

     >>> Dtype("(u8, [bool; 3])").pack((5, [True, False, True]))
     Tibs('0b00000101101')
     

   • Added the bf16 dtype for bfloat16, the non-IEEE 16-bit float that keeps the 8-bit exponent of an f32 and truncates the mantissa to 7 bits.

     >>> Tibs.from_value("bf16", 1.0).hex
     '3f80'
     >>> Tibs("0x3f80").to_value("f16")
     1.875
     

   • Added eleven fixed-width floating point dtypes: the draft P3109 binary8p3 and binary8p4 formats; OCP ocp_e4m3 and ocp_e5m2, each with separate _saturate and _overflow packing policies; ocp_e3m2, ocp_e2m3, ocp_e2m1, ocp_e8m0 and ocp_int8. Packing rounds directly from Python binary64, to nearest with ties to even. The P3109 formats are provisional and may be updated as the draft standard develops. These are raw scalar encodings only: shared scales and MX block behaviour are not yet included.

     >>> Tibs.from_values("ocp_e2m1", [0.5, 1.0, 3.0, 6.0]).hex
     '1257'
     >>> Tibs.from_value("ocp_e4m3_overflow", 1000.0).hex
     'ff'
     

   • A DtypeKind whose length is intrinsic — the eleven narrow formats above, plus bool and bf16 — is now accepted anywhere a dtype is, and its length may be omitted from DtypeSingle.from_params.

     >>> Tibs("0x1257").to_values(DtypeKind.OcpE2M1)
     [0.5, 1.0, 3.0, 6.0]
     >>> DtypeSingle.from_params(DtypeKind.OcpE2M1)
     DtypeSingle('ocp_e2m1')
     

   • Added eight methods for comparing two containers of the same length without building an intermediate object: count_and, count_or, count_xor, count_andnot, intersects, is_disjoint, is_subset_of and is_superset_of. The counts are equivalent to (a & b).count(1) and so on, and count_xor is the Hamming distance. Skipping the intermediate makes them faster, increasingly so as the containers get longer, and the four predicates stop as soon as they know the answer.

     >>> a, b = Tibs('0b1100'), Tibs('0b1010')
     >>> a.count_xor(b)      # the Hamming distance
     2
     >>> Tibs('0b1010').is_subset_of('0b1011')
     True
     >>> Tibs('0b1100').is_disjoint('0b0011')
     True
     

   • The value parameter of count is now optional, and counts the set bits when it's not given, so t.count() is the same as t.count(1).

   • count now takes a byte_aligned parameter, so that a pattern can be counted on byte boundaries only, matching find and find_all.

     >>> Tibs('0x1f2e3f').count('0x0f', mask='0x0f', byte_aligned=True)
     2
     

   • Added extracted, deposit and deposited for reading and writing bit fields whose bits are scattered through a container by a mask, rather than being contiguous like the ones field handles. These are the bit-level equivalents of the x86 PEXT and PDEP instructions.

     >>> Tibs('0b11010110').extracted('0b10110000')
     Tibs('0b101')
     >>> Tibs('0b11010110').deposited('0b111', '0b10110000').bin
     '11110110'
     

   • Searches can now take a mask, so that patterns can contain don't-care bits: only the bits set in the mask need to match. It's available on find, rfind, find_all, find_all_iter, rfind_all_iter, count, replace and replaced, and combines with start, end, count and byte_aligned as you'd expect. For replace the mask affects only the matching, and the whole of each match is still replaced.

     >>> Tibs('0x1f2e3f').find_all('0x0f', mask='0x0f', byte_aligned=True)
     [0, 16]
     

   • The u and i interpretations no longer have a 128 bit limit. Any positive length now works, for from_u / from_i, the u and i properties and to_u / to_i, the uN / iN dtypes, views, and the u and i format codes.

     >>> Tibs.from_u(2 ** 200 - 1, 200).u == 2 ** 200 - 1
     True
     

   • Added View.to_value, MutableView.to_value and MutableView.write_value, so that any dtype is reachable through a view rather than only the nine interpretations that have their own view property. The view's byte order and bit order are applied first, so view.to_value(dtype, start, end) is always view.to_tibs().to_value(dtype, start, end).

     >>> m = Mutibs.from_bytes(bytes.fromhex("07 01 00 00 44 33 22 11"))
     >>> m.lsb0.le.field(63, 32).to_value("u32")
     287454020
     >>> m.lsb0.le.field(63, 32).write_value("u32", 0xdeadbeef)
     >>> m.hex
     '07010000efbeadde'
     

     Byte order is stated in one place only, so an le, be or lsb0 view refuses a dtype that names a byte order, at any nesting depth. The plain view() claims no layout and still passes any dtype through.

   • Added __format__ to Tibs, Mutibs, View and MutableView, so that they can be used directly in f-strings and with str.format(). The type codes are b, o, x and X for the bit representations (equivalent to the bin, oct and hex properties, so leading zeros are kept), u and i for the unsigned and signed integer interpretations, and e, f and g with their uppercase forms for the float one, which needs a length of 16, 32 or 64. The # flag adds a prefix, and _ groups the digits, with the group size taken from the otherwise unused precision field.

     >>> f"{Tibs('0xac804f4b'):#_.2x}"
     '0xac_80_4f_4b'
     >>> pi = Tibs.from_f(3.14159, 32)
     >>> f"{pi:f}", f"{pi:.2f}", f"{pi:.3e}", f"{pi:g}"
     ('3.141590', '3.14', '3.142e+00', '3.14159')
     

     Fill, alignment and width work as they do elsewhere in Python, except that zero padding is rejected for b, o, x and X, as it would silently change the apparent length of the value. Groups are counted from bit zero, so it's the last group that comes up short rather than the first, and padding is never itself grouped.

   • Tibs now exports the buffer protocol, so that it can be passed straight to anything that takes a bytes-like object — memoryview(t), array.array, numpy.frombuffer, a socket or file write, and so on — without copying its data first.

     >>> bytes(memoryview(Tibs('0xff00')))
     b'\xff\x00'
     

     The buffer is read-only and covers whole bytes, so a final partial byte includes its padding bits, unmasked. Exporting needs the underlying storage to start on a byte boundary; when it doesn't, a BufferError is raised and to_bytes or to_padded_bytes will give you an owned copy instead. This is deliberately not offered on Mutibs, whose storage moves as it is edited.

   • Tibs and Mutibs can now be pickled, so that they can be sent through multiprocessing, stored in a cache, or deep copied with copy.deepcopy, which previously raised a TypeError. The pickled state is the Codec.Raw encoding, so any bit length is restored exactly and pickling costs about what copying costs; pickle the result of encode() yourself if you want the compression that Codec.Auto can give.

     >>> pickle.loads(pickle.dumps(Tibs('0b110101')))
     Tibs('0b110101')
     

   • Added a tibs.__version__ string, alongside the tibs.__author__ that was already there.

   • Free-threaded Python (3.14t) is now a supported configuration, with wheels built for it and the test suite run against them. Tibs declares that it does not need the global interpreter lock, so importing it doesn't switch the GIL back on, and work on separate containers runs on separate threads in parallel rather than taking turns. A Tibs is immutable and can be shared freely, with no lock taken at all. A Mutibs can be shared too: every method runs inside CPython's per-object critical section, which makes a single call atomic but not a sequence of calls, exactly as for a list. There's a new appendix in the docs covering this in full.

   Changed

   • Inverting an empty Tibs or Mutibs with ~ now returns an empty container instead of raising a ValueError, matching Tibs.inverted and Mutibs.invert.

   Performance improvements

   • Compound dtype packing and unpacking are much faster. A DtypeTuple operation equivalent to a struct call ("(i16_be, i16_be, i32_be)" against ">hhl") is now in the same ballpark as the handwritten struct code, where it was an order of magnitude slower when compound dtypes were first implemented.
   • Parsed dtype specs are now cached, so that passing a string where a dtype is expected no longer re-parses it on every call. This mattered most for short operations, where parsing could dominate; passing a prebuilt Dtype is quicker still. Bulk calls were never parse-bound and are unchanged.
   • A number of other operations — string/hex/oct/bin conversion, from_values, Mutibs.reverse and others — moved from bit-at-a-time to byte-at-a-time or whole-buffer implementations. See tests/performance_comparison.py for how tibs measures up against bitarray and against equivalent standard library code.