symbolic-demangle

Updating

• Require python 3.8+ for python bindings (#793)

Fixes:

• sourcemapcache: Improved scope name resolution. (#786)

Fixes:

• Optimize SourceBundle/DebugSession (#787)

Fixes:

• Correctly resolve sourceMappingURL references for "source" files.
• Allow parsing the DebugId of a PS4 executable.

Fixes:

• PPDB files report has_sources() = true even when they only contain source links (#774)
• Make SourceBundle writer deterministic (#778)
• Increase IL2CPPP source mapping boundary (#776)

Features:

• Skip hidden sequence points when creating PortablePDB cache. (#769)

Features:

• Replace internal usage of LazyCell by OnceCell and make SourceBundleDebugSession: Send, Sync and AsSelf. (#767)

Features:

• Added debug IDs to source bundle JavaScript files and source maps. (#762)
• Add support for embedded debug IDs in minified files (#765)

Breaking changes:

• Change DebugSession::source_by_path() to return a SourceFileDescriptor containing either file content or a URL to fetch it from. (#758,#762)
• Change SourceBundleWriter::write_object_with_filter() filter callback to take SourceFileDescriptor, allowing to include/exclude embedded sources. (#764,#762)

Fixes:

• Make sure to parse PortablePdb streams in the correct order. (#760)

Features:

• Correctly read the debug_id of Deterministic PE files (#658)
• Gracefully handle invalid PPDBs (#751)
• Support embedded PortablePDB in DLLs (#752, #757)

Fixes:

• Discard invalid line records and inlinees when parsing functions. (#747)

Features:

• PortablePdbDebugSession now returns files referenced in the Portable PDB file. (#729)
• PortablePdbDebugSession now returns source files embedded in the Portable PDB file. (#734)
• Implement symbolic_common::AsSelf for SourceMapCache (#742)
• Debug information can now be retrieved from PE's with DWARF debug info. (#744)

Breaking changes:

• Demangling functionality is removed from C and Python bindings. (#730)
• The fields of FileInfo and the compilation_dir field on FileEntry are now private. (#729)
• PortablePdbDebugSession now has a lifetime parameter. (#729)
• PeDebugSession placeholder has been removed. (#744)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @vaind
• @casept

Features:

• PortablePdbCache now implements Clone, PartialEq, Eq, and AsSelf. (#726, #727)

Features:

• Add inlinee overlap checks in symbolic-debuginfo. (#718)

Fixed:

• Ensure proper inlinee nesting in symbolic-debuginfo. (#720)
• Split lines in more cases in symbolic-debuginfo. (#717)

Internal

• Add a debuginfo_debug example. (#716)
• Update js-source-scopes to 0.3.1. (#721)

Internal

• Update js-source-scopes to 0.3.0. (#715)

Internal

• Update js-source-scopes to 0.2.2. (#714)

Internal

• Python/CABI bindings will use a tempfile instead of holding SourceMapCache in memory. (#711)

Internal

• Updated goblin, cpp_demangle and wasmparser dependencies. (#709)

Fixed:

• Rebuild C headers for SourceMapCache to expose token name (#704)

Features:

• Add name to SourceMapCacheToken Python bindings (#703)

Features:

• Added an Object type for Portable PDB files. (#696)
• Version 2 of the sourcemapcache format additionally saves the names of source locations. (#698)

Fixes:

• Fixed a bug in Unreal Engine log parsing by updating the anylog dependency. (#695)

Features:

• Added a new sub-crate for working with JavaScript SourceMaps: symbolic-sourcemapcache. (#688)
• This new sourcemap handling is also exposed via the symbolic Python bindings.
• The new version 8 of the symcache format saves strings slightly more compactly. Reading of version 7 is still supported. (#670)
• symbolic-ppdb incorporated into symbolic (behind the ppdb feature flag) (#687)

Internal:

• Use the watto library for de/serialization of symcache and ppdbcache. (#670)

Fixes:

• Be more lenient when eagerly parsing DWARF Abbreviations. (#685)

Fixes:

• Be stricter about demangling only _Z prefixed C++ names. (#681)
• Work around a pathological case in DWARF processing that could lead to slowness and high memory usage. (#683)

Fixes:

• Correctly resolve the DW_AT_producer attribute of DWARF files (#676)
• Improve _sigtramp workaround and explanation (#662)
• Slightly lower demangling recursion limit (#655)

Fixes:

• Allow underflows in Windows x64 CFI to allow restoring registers from outside of the current stack frame. (#645)
• Rework the SymCache Writer for correctness. (#648)
• Undecorate Windows symbols when writing SymCache. (#649)
• Correctly use the .debug_addr section needed for DWARF5. (#652)

Internal:

• Remove outdated and unused il2cpp/usym-related code. (#650)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @mstange

Features:

• Added a new subcrate for working with Portable PDB files: symbolic-ppdb (#621)
• Added support for breakpad sym files containing INLINE_ORIGIN and INLINE records. (#605)
• Most function names from PDBs will now include function arguments. (#426)

Fixes:

• Improved DWARF handling to create more consistent inlinee and line information. (#607, #633)
• Improved PDB handling to include more symbols and create more consistent inlinee and line information. (#622, #627, #636, #641)
• Fixed a typo in DWARF 5 addrx parsing which caused incorrect address ranges. (#608)
• symcache generation now handles more cases of line / inlinee information correctly. (#637)
• Improved arm64 CFI for macOS system libraries by translating UnwindOp::None into trivial CFI. (#619)
• Improved arm64 CFI for macOS _sigtramp by restricting a workaround to x86_64. (#625)
• Improved MIPS CFI by always generating .ra CFI. (#612)
• Fixed il2cpp lookup to discard information for unrelated lines. (#617)
• Leading underscores are no longer stripped from PDB symbols. (#642)
• MSVC demangled functions now look more visually pleasing. (#640)
• Updated wasmparser dependency to 0.85.0. (#624)

Breaking changes:

• Updated the debugid dependency to 0.8.
• Updated the uuid dependency to 1.0.
• Updated the pdb dependency to 0.8.
• Removed the public method symbolic_common::CpuFamily::cfi_register_name.
• The symbolic-minidump crate has been dropped. The CFI functionality that was contained in symbolic-minidump now resides in its own crate, symbolic-cfi.
• The symbolic-unwind crate has been dropped.
• The symbolic-sourcemap crate has been dropped. Since it was only used in symbolic-cabi, its functionality has been incorporated into symbolic-cabi.
• Support for symcache versions before v7 has been dropped. This entails a number of changes in the public API of symbolic-symcache:
  • Removed support for symcache binary formats prior to v7.
  • Removed SymCacheWriter.
  • Removed SymCacheError.
  • Removed SymCacheErrorKind.
  • Removed Line.
  • Removed Lines.
  • Removed LineInfo.
  • Removed Lookup.
  • Removed Function::id.
  • Removed Function::parent_id.
  • Removed Function::address.
  • Removed Function::symbol.
  • Removed Function::compilation_dir.
  • Removed Function::lines.
  • Removed SymCache::has_line_info.
  • Removed SymCache::has_file_info.
  • Changed return type of Function::name to string slice.
  • Changed return type of SymCache::lookup to SourceLocations.
  • Added Function::name_for_demangling with the previous signature and behavior of Function::name.
  • Added Function::entry_pc.
  • Added SymCacheConverter.
  • Added Error.
  • Added ErrorKind.
  • Added File.
  • Added Files.
  • Added FilesDebug.
  • Added FunctionsDebug.
  • Added SourceLocation.
  • Added SourceLocations.
  • Added SymCache::files.
  • Added lifetime parameter to Transformers.
  • Undeprecated Function and Functions.
  • Undeprecated SymCache::functions.
• Some C and Python bindings have been dropped or adjusted. Concretely:
  • symbolic-cabi::minidump and the corresponding Python functionality has been removed. The CFI functionality that was contained therein now resides in symbolic-cabi::cfi and symbolic.cfi, respectively.
  • symbolic-cabi::unreal and the corresponding Python functionality has been removed.
  • symbolic-cabi::symcache::SymbolicLineInfo has been replaced with SymbolicSourceLocation, which has a different interface. Likewise, symbolic.symcache.LineInfo has been replaced with SourceLocation.
  • symbolic-cabi::symcache::symbolic_symcache_has_file_info and symbolic_symcache_has_line_info have been removed, likewise for symbolic.symcache.SymCache.has_line_info and has_file_info.

Features:

• Optionally collect referenced C# file sources when creating a source bundle. (#516)

Fixes:

• Only skip one function when encountering unknown Unwind Codes on Windows x64. (#588)
• Skip over low_pc sentinels instead of erroring. (#590)

Fixes:

• Make CFI generation for Windows x64 more accurate, restoring all possible registers and supporting frame pointer unwinding. (#549)

Fixes:

• Make sure to correctly parse Unreal crash reports with zero-length files (#565)

Fixes:

• Updated wasmparser dependency to 0.83.0 (#557)
• Updated rust-sourcemap dependency to hopefully speed up sourcemap parsing (#559)
• Match symbol names by exact addresses from symbol table (#510)
• Return a more correct function_size when dealing with split functions (#522)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @shuoli84
• @bnjbvr

Features:

• Added a new SymCache Transformer, which can be used to apply Function or SourceLocation transformations. (#496)
• Turn the breakpad-based minidump processor into an optional feature flag. (#519)

Fixes:

• Fixed CFI STACK WIN records being written correctly. (#513)
• Do not consider empty files as valid BcSymbolMaps anymore. (#523)
• Fix wasm parsing rejecting valid wasm files with non-default features. (#520)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @bnjbvr

Fixes:

• Update goblin which received fixes to avoid panics and unreasonable memory allocations based on invalid input. (#503)
• Fix wrong instruction addresses of the first frame in ARM and ARM64 minidumps. The addresses were incorrectly incremented by one instruction size. (#504)
• Correctly skip ELF sections with an offset of 0 instead of ignoring all following sections. This bug may have lead to missing unwind or debug information. (#505)
• Detect unwind information when linking with gold. (#505)

Features:

• Added a new SymCache binary format which is fundamentally based around instruction addr ranges.
• Add ElfObject::debug_link that allows recovering the debug link from an Elf if present. (#450)
• Updated Swift demangler to 5.5.1. (#465)
• Support split functions. (#441)
• Refactor symbolic-debuginfo feature flags. (#470)
• Rewrite wasm parser. (#474)

Fixes:

• Make SourceBundle ordering deterministic. (#489)
• Replace unmaintained dependencies.
• Better guard against invalid input that could lead to unreasonable memory allocations, panics or infinite loops.

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @dureuill
• @Jake-Shadle

Features:

• Add ByteView::map_file_ref constructor which does not consume the File passed to it. (#448)

Fixes:

• Support Unreal Engine 5 crash reporter. (#449)

Features:

• Add Unreal4Crash::parse_with_limit which allows specifying a maximum allocation size when extracting compressed UE4 crash archives. (#447)

Fixes:

• Apply speculative handling of stackless functions only on amd64 when creating CFI caches. (#445)

Features:

• Build and publish binary wheels for arm64 / aarch64 on macOS and Linux. (#442)

Fixes:

• Don’t prefix ARM registers with $ for CFI files. (#443)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @Gankra

Fixes:

• Avoid panic when looking for hex suffixes in multibyte character strings in the demangler. (#430)
• Allow processing of ELF files as long as they have valid program and section headers. (#434)
• Expose dynamic symbols in ELF files. (#421)
• Make dsym_parent accept .framework.dSYM. (#425)

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @goffrie
• @gabrielesvelto
• @luser
• @mstange

Features:

• Write versioned CFI Cache files. Reading those files is only supported with symbolic versions >= 8.2.1, so trying to use a CFI Cache file with an older version of symbolic will fail with a CfiErrorKind::BadFileMagic error.

Fixes:

• Correctly restore callee saves registers when using compact unwind info.
• Correctly map all DWARF information when using BcSymbolMaps.
• Allow processing of PDB files that have broken inlinee file references.
• Skip duplicated DWARF functions which can lead to inline parent offset overflows.

Features:

• Add support for reading versioned CFI Cache files.

Fixes:

• Avoid quadratic slowdown when using compact unwind info on macOS.

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @Gankra

Caution:

• Relevant dependencies such as gimli, goblim, and wasm-related libraries were updated.

Features:

• Support for compact unwind info in MachO files was added, along with special casing of some well known macOS system functions.
• The parser of the Breakpad Format was rewritten.

Bug Fixes:

• All valid STACK WIN record types are being parsed correctly. This did add new variants to the BreakpadStackWinRecordType enum. Technically a breaking change, but we do not consider the Breakpad Parser types as adhering to strict SemVer rules.

Thank you:

Features, fixes and improvements in this release have been contributed by:

• @Gankra

Features:

• Add support for loading BCSymbolMaps into MachObjects to un-obfuscate symbol names in bitcode builds. (#336)

Bug Fixes:

• Handle too many files more gracefully. (#374)
• Parse .pdb files containing modules without symbols. (pdb#102)

Bug Fixes:

• Fix detecting hidden Swift symbols in MachObject::requires_symbolmap. Additionally, the MachO symbol iterator no longer strips underscores from __hidden# symbols. (#316)

Manylinux2010 has dropped support for Python 2.7. As a result, we're no longer building or testing the Python package with Python 2.7. This and future releases require at least Python 3.6.

Bug Fixes:

• Compute correct line offsets in symcaches with large gaps between line records. (#319)
• Support symcache lookups for public symbols larger than 65k. (#320)
• Fixed bug that caused functions to have a length of 0 in symcaches. (#324)
• Support debug_addr indexes in DWARF functions. (#326)

Bug Fixes:

• Support DWARF information from MIPS compilers in SHT_MIPS_DWARF sections. (#317)
• Remove a duplicate dependency to two versions of walrus for WASM parsing. (#312)

Bug Fixes:

• Include third-party submodules to allow the Python sdist to build again. (#310)

Bug Fixes:

• Compute correct debug identifiers when proessing a Minidump from a machine with opposite endianness. This particularly allows to process MIPS minidumps on little-endian hosts. (#281)
• Update the breakpad processor with better stack scanning heuristics. Some false-positive frames are avoided during stack scanning now. (#281)
• Avoid panics when processing UE4 crash logs containing ambiguous local times. (#307)

Breaking Changes:

• Usage of failure was removed, and all error types were changed to only implement std::error::Error and related traits.
• symbolic-proguard was removed in favor of the proguard crate. Proguard is still supported via symbolic-cabi and the python API however.
• Deprecated APIs have been removed:
  • InstructionInfo's fields are no longer public.
  • pointer_size, instruction_alignment and ip_register_name have moved from Arch to CpuFamily.
  • Arch::register_name as been moved to CpuFamily::cfi_register_name.
  • Dwarf::raw_data and Dwarf::section_data have been replaced with the raw_section and section APIs.
  • Unreal4ContextRuntimeProperties::misc_primary_cpu_brand is has been removed.
• Deprecated Python APIs have been removed:
  • CodeModule.id and CodeModule.name Use debug_id and code_file, respectively.
• DemangleFormat and public fields of DemangleOptions have been removed in favor of builder methods on DemangleOptions.
• Name::new now takes both the NameMangling state, and the Language explicitly.

Features:

• Add support for the wasm32 architecture. (#166)
• Support demangling for Swift 5.3. (#282)

Bug Fixes:

• Detect mangled anonymous namespaces in PDB inlinees (#261)
• Fix a panic due to undefined behavior. (#287)
• Skip line program sequences at 0. (#291)
• Prefer DWARF names for Dart functions. (#293)

Changes:

• Add missing unreal data attributes (EngineData and GameData). (#257)
• Expose binary names for ELF and MachO (#252)
• Mark enums as non_exhaustive. (#256)
• Add method to create Archive from bytes. (#250)

Bug Fixes:

• Fix compilation errors on nightly Rust due to a lifetime mismatch. This is temporarily solved with a statically verified unsafe transmute, which will be replaced in an upcoming breaking change. (#258)

Deprecations:

• pointer_size, instruction_alignment and ip_register_name have moved from Arch to CpuFamily.
• Arch::register_name as been moved to CpuFamily::cfi_register_name.
• Field access on InstructionInfo has been deprecated and replaced with a builder.

Changes:

• More detailed documentation and examples on all types and functions in symbolic-common. (#246)

Bug Fixes:

• CpuFamily::cfi_register_name returns None instead of Some("") for some unknown registers.
• Update cpp_demangle again after the previous release was yanked. (#247)

Bug Fixes:

• Update the cpp_demangle dependency to fix broken builds after a breaking change. (#244, thanks @o0Ignition0o)

Bug Fixes:

• Update the proguard dependency to fix line info detection. (#242)

Deprecations:

• symbolic-proguard is now deprecated and will be removed in the next major release. Use the proguard crate directly. The C-bindings and Python interface will remain. (#240)

Python:

• Switch the C-ABI and python to proguard 4.0.0 which supports frame remapping. (#240)

Bug Fixes:

• Fix broken links in docs on ByteView, SelfCell and AsSelf. (#241)

Bug Fixes:

• Update broken doc comments for SelfCell and debuginfo::dwarf (#238)
• Fix holes in line records of inline parents in DWARF (#239)

Bug Fixes:

• Fix line information of inline parents in DWARF (#237). Many thanks to @calixteman!

Bug Fixes:

• Skip invalid PE runtime function entries (#230)
• Support demangling of block invocation functions (#229)
• Skip invalid CFI entries instead of erroring out (#232)
• Detect stub DLLs and skip CFI generation (#233)
• Skip functions with unknown unwind codes (#234)
• Update goblin to fix panics in PE unwinding (#231)
• Update gimli to to support eh_frame CIE version 3 (#231)
• Update cpp_demangle (#231)

Build Changes:

To build with the demangle feature, a C++14 compiler is now required.

Features:

• Support Swift 5.2 mangling (#208)
• Inline docs from sub-crates (#209)
• Add Path utilities for dSYM structures (#212)
• Updated C++ demangler (#215)

Bug Fixes:

• Do not error in functions iterator on name errors (#201)
• Avoid infinite recursion in DWARF because of self references (#202)
• Do not skip symbols from SHT_NOBIT sections (#207)
• Do not assume sorted DWARF compilation units (#214)
• Skip eliminated functions in linked objects (#216)
• Avoid panics for UTF-8 characters in paths (#217)
• Get CFI info from eh_frame even if err in debug_frame (#218)
• Avoid TooManyRegisterRules errors in CFI (#219)
• Calculate correct line record sizes for DWARF (#220)
• Detect all scopes to fix incorrect inlinee hierarchies (#221)
• Patch all parent line records of inlinees in DWARF (#223)
• Fix broken compilation with -Clink-dead-code (#225)
• Return the same instruction address for inlinees in symcaches (#226)