lexical-core

Increment high-level lexical version for release.

Fixes #191.
Fixes apache/datafusion#13686

Fixed

• Only require 19 digits for writing i64 and not 20 (#191).

Fixed

• Only require 19 digits for writing i64 and not 20 (#191).

Changed

• Improved performance of number formatting with non-decimal radices (#169).

Fixed

• Inaccurate number formatting with non-decimal radices (#169).

Increment versions for release.

Changed

• Improved performance of number formatting with non-decimal radices (#169).

Fixed

• Inaccurate number formatting with non-decimal radices (#169).

Changed

• Higher performance when parsing floats with digit separators.

Fixed

• Inlining inconsistency between public API methods (credit to @zheland)
• Incorrectly accepting leading zeros when no_integer_leading_zeros was enabled.
• Have consistent errors when an invalid leading digit is found for floating point numbers to always be Error::InvalidDigit.
• Incorrect parsing of consecutive digit separators.
• Inaccuracies when parsing digit separators at various positions leading to incorect errors being returned.
• Selecting only a subset of parse and/or write features would cause compilation errors.
• Fixed bug with writing integers with custom radices.

Fixed

• A correctness regression.
• Regressions where parsing digit separators without the compact panicked.

Added

• Added fuzzing and miri code safety analysis to our CI pipelines.
• Removed requirement of alloc in no_std ennvironments without the write feature.
• Make multi-digit optimizations in integer parsing optional.
• Much higher miri coverage including for proptests and our corner cases from the golang test suite.

Changed

• Updated the MSRV to 1.63.0 (1.65.0 for development).
• Improved performance due to compiler regressions in rustc 1.81.0 and above.

Fixed

• Removed use of undefined behavior in MaybeUninit.
• Provide better safety documentation.
• Parsing of Ruby float literals.
• Performance regressions in Rust 1.81.0+.
• Removed incorrect bounds checking in reading from iterators.
• Overflow checking with integer parsing.
• Writing -0.0 with a leading -.
• Reduced binary siezes when the compact feature was enabled.
• Improved performance of integer and float parsing, particularly with small integers.
• Removed almost all unsafety in lexical-util and clearly documented the preconditions to use safely.
• Removed almost all unsafety in lexical-write-integer and clearly documented the preconditions to use safely.
• Writing special numbers even with invalid float formats is now always memory safe.

Removed

• Support for mips (MIPS), mipsel (MIPS LE), mips64 (MIPS64 BE), and mips64el (MIPS64 LE) on Linux.
• All _unchecked API methods, since the performance benefits are dubious and it makes safety invariant checking much harder.
• The safe and nightly features, since ASM is now supported by the MSRV on stable and opt-in for memory-safe indexing is no longer relevant.

Changed

• Fixed the partial integer parser to correctly return negative values if parsing partial input.

Added

• Added support for -Zmiri-tag-raw-pointers in miri correctness checks.
• Added the documented functions format_error and format_is_valid to determine if a format packed struct is valid, since the previous checks relied on undocumented behavior.
• Added from_radix to ParseFloatOptions and WriteFloatOptions, to simplify creating the default options with a different radix.

Changed

• Fixed no_std in lexical-parse-float when default features are disabled.
• Fixed issue in parsing integers and floats with the power-of-two feature enabled for radixes 16 and 32.

Changed

• Reproduce all dependent licenses in lexical, and document extensively what features are dependent on what licensing terms.

This is a very large release that involved a full re-write from the ground-up.

Added

• Added the Options API
  • ParseFloatOptions
  • ParseIntegerOptions
  • WriteFloatOptions
  • WriteIntegerOptions
• Added parse_with_options, parse_partial_with_options
• Added write_with_options.
• Added a faster float writer and parser for power of 2 radixes.
• Added the required_exponent_notation flag to NumberFormat.
• Added the power-of-two feature, for conversion to and from strings with power-of-two bases.
• Added the compact feature, optimized for binary size rather than performance.
• Added extensive documentation on benchmarks, algorithms used, and internal implementation details.

Changed

• Made the Eisel-Lemire algorithm the default float-parser, leading to substantial improvements in performance.
• Made Dragonbox the default float writer.
• Removed pre-computed float tables of powers-of-two, and recreated stable powers of 2 through bit manipulations.
• Refactored all numeric conversions into separate crates, using workspaces.
• Improved the algorithms to write and parse 128-bit integers, using faster division and multiplication algorithms.
• Updated the MSRV to 1.51.0.
• Updated the benchmarks to note the significant algorithm changes.
• Added automatic formatting via Rustfmt, linting via Clippy, as well as increased documentation of safety guarantees.
• Re-wrote NumberFormat to allow more extensive configuration and use const generics.
• Added support for base prefixes and suffixes.
• Reduced static storage required for extended-float algorithms.
• Fixed a bug with trailing digit separators in special values.
• Updated the fuzz handlers and conformance tests.
• Simplified the big-integer arithmetic, improving performance and simplifying maintenance for float parsing algorithms.

Removed

• Remove the write_format, write_radix, and similar functions.
• Remove the parse_format, parse_radix, and similar functions.
• Removed the dtoa feature.
• Removed the rounding feature.
• Removed the lexical-capi. Existing, high-performance C/C++ libraries exist, so it's no longer practical.

Changed

• Updated cfg_if version.
• Downgraded rand to fix proptests.
• Fixed a bug on newer Rustc versions where slice::sort is not present in no_std.
• Added a feature libm which enables stable no_std use.
• Patched an implementation of insert_many due to a security advisory which does not affect lexical.

Changed

• Fixed issue with integer::BITS conflicting with new compilers.

Changed

• Changed NumberFormat to use 64-bit flags.
• Added NO_INTEGER_LEADING_ZEROS and NO_FLOAT_LEADING_ZEROS NumberFormat flags.
• Added InvalidLeadingZeros to ErrorCode enum.
• Added validate_no_leading_zeros to reject invalid inputs.
• Added new leading zero checks to all pre-defined NumberFormat constants.

Changed

• Updated the format bitflags to make matches more efficient, allowing jumptables to be used for most situations.

Changed

• Fixed a regression in parsing special values.

Added

• Added format feature to control parsing integers and floats from number specifications.
• Added the NumberFormat bitflags to control Number format specifications. These flags control how a number is parsed, including enabling the use of digit separators, requiring integer or fraction digits, and more.
• Added pre-defined constants for NumberFormat (RUST_STRING, PYTHON_LITERAL) to avoid compiling formats when not needed.
• Added the FromLexicalFormat and FromLexicalLossyFormat traits with the format feature, which enable you to specify the number format during parsing.
• Implemented algorithms in terms of generic iterators, to allow skipping digit separators.
• Implemented data interfaces to simplify parsing and validating number format.
• Added more values to ErrorCode to signify more parse failures.

Changed

• Changed ErrorCode::MissingFraction to be ErrorCode::MissingMantissa, to differentiate between missing integers, missing fractions, and missing significant digits (mantissa).
• Remove RawFloatState and FloatState and replaced it logically with FastDataInterface and SlowDataInterface, allowing format-defined parsing.

Changed

• Updated arrayvec and static-assertions versions.
• Removed support for Rustc versions below 1.37.0.

Changed

• Fixed a bug causing compilation issues on MSVC.

Changed

• Removed panic handler, allowing use in no_std environments.

Added

• Added get/set_exponent_default_char, get/set_exponent_backup_char, and get/set_float_rounding, to allow validation logic for config variables.
• Added the write* functions, which replace the *toa methods.
• Added parse* functions, which replace the ato* methods.
• Added tests to ensure lexical-core works with proc-macros.
• Substituted rust-stackvector with arrayvec, which has better support.

Removed

• Removed EXPONENT_DEFAULT_CHAR, EXPONENT_BACKUP_CHAR, and FLOAT_ROUNDING.
• Removed all expanded API signatures for ato* to *toa, since these were for compatibility with the C-API and were therefore redundant.
• Removed the FFI submodule, and moved all functionality to the lexical-capi crate.

Added

• Added ffi::Result, an FFI-compatible tagged union, to store results for FFI-compatible functions.
• Added MAX_*_SIZE_BASE10 constants, to determine the maximum size in bytes a formatted type will take in base 10.
• Added ato*_partial functions, which parse until an invalid digit is found, returning the value and number of processed digits.
• Added unittests for itoa designed to test uniform, sequential, and heterogeneous data, aimed to test both optimal performance and branch prediction misses.

Changed

• Improved code generation and reduced binary bloat. There are a few algorithmic differences that make this possible, but the parsers now generally extract the float subcomponents first, then parsed known-good data after for enhanced performance.
• Updated the benchmarks and benchmark results.
• Moved lexical::Error to lexical_core::Error. Error is re-exported in lexical.
• Add variants for lexical_core::ErrorCode. Added Underflow, to detect numerical underflow, EmptyFraction, to detect floats with an empty integer and fraction components, and EmptyExponent, to detect floats with no value following the exponent character.
• Changed Result to use std::result::Result.
• Fixed proptests and added more comprehensive proptests thanks to @dangrabcad.
• Moved FFI-compatible code to the public ffi module.
• Removed *_slice from functions in the public API, since there is no naming conflict with FFI code.
• Removed *_ffi and *_FFI from public functions and constants.
• Optimized the itoa exporters using specialized optimizations for the type size and number of digits. The resulting performance excels for all values, providing 2-3x performance improvements, with exceptional performance for values with a small number of digits.
• Optimized integer division algorithm based off the compiler-builtins crate to avoid redundant calls to an expensive compiler intrinsic (__udivmodti4).
• Optimized the atoi algorithm for 128-bit integers by using intermediary 64-bit integers, leading to performance boosts of up to 30x.

Removed

• Removed the unchecked parsers from the public API. try_parse* has been replaced with parse*.
• Removed ErrorCode::Success, since with the new result types it was redundant.

• Fixed a bug (issue #20) leading to incorrect float parsing (1 ULP error) for slow-path algorithms containing floats with a trailing 0-digit in the fraction component (discovery by @dangrabcad). Added in comprehensive unittests to avoid future regressions.
• Fixed CI for older Rustc versions due to issues with the edition keyword.
• Updated dependencies (credit to @junhoo).
• Updated the benchmarks for float and integer formatting to use Write::write_fmt rather than to_string to avoid heap allocation leading to misleading results (credit to @RazrFalcon).

Added

• Comprehensive continuous integration unittests for numerous platforms, based off the trust templates.
• Added known issue for a non-default, lossy setting on armv6 architectures.

Changed

• Bug fix for 32-bit targets.

Added

• Backwards compatible support for Rustc 1.24.0.
• Added parse_partial* parsers, which parse until an invalid digit is found, returning the value and number of processed digits.

Changed

• Worked around a bug in the internal float formatter where &arr[idx] creates a local copy in Rust versions before 1.28.0, creating a dangling reference after the scope ends. For more details, see cached_grisu_power in /lexical-core/src/ftoa/grisu2.rs .
• Changed public trait names for FromBytes, FromBytesLossy, and ToBytes to FromLexical, FromLexicalLossy, and ToLexical.
• Changed method names for trait ToLexical from to_bytes and to_bytes_radix to to_lexical and to_lexical_radix to avoid conflict with unstable feature int_to_from_bytes.
• Changed method names for trait FromLexical from from_bytes, from_bytes_radix, try_from_bytes, and try_from_bytes_radix to from_lexical, from_lexical_radix, try_from_lexical, and try_from_lexical_radix to avoid conflict with unstable feature int_to_from_bytes.
• Changed method names for trait FromLexicalLossy from from_bytes_lossy, from_bytes_lossy_radix, try_from_bytes_lossy, and try_from_bytes_lossy_radix to from_lexical_lossy, from_lexical_lossy_radix, try_from_lexical_lossy, and try_from_lexical_lossy_radix to avoid conflict with unstable feature int_to_from_bytes.