github.com/wader/fq

Changes

• Format specific options
  • Formats can now have own options
  • Example to skip decoding of samples in a mp4 file use:
  • fq -d decode_samples=false d file.mp4 or ... | mp4({decode_samples: false}})
  • To see supported options for a formats see formats documentation, use fq -h mp4 or help(mp4) in the REPL.
• gojq fork rebase:
  • Many performance improvements from upstream
  • Assign to a JQValue will now shallowly turn it into a jq value and then be assigned.
  • Refactor and rewrote large parts to make it easier to rebase and maintain in the future.

Decoder changes

• amf0 Add Action Message Format 0 decoder #214
• hevc_pps Add H.265/HEVC Picture Parameter Set decoder #210
• hevc_sps Add H.265/HEVC Sequence Parameter Set decoder #210
• hevc_vpc Add H.265/HEVC Video Parameter Set decoder #210
• mp3 Add max_unique_header_config and max_sync_seek options #242
• mp4 Simplify granule structure a bit #242
• mp4 Add decode_samples and allow_truncate options #242
• flac_frame Add bits_per_sample option #242
• icmpv6 Add Internet Control Message Protocol v6 decoder #216
• id3v2 Add v2.0 PIC support
• ipv6_packet Add Internet protocol v6 packet decoder #216
• macho Remove redundant arch struct level and cleanup some sym values #226
• macho Add raw fields for section and encryption info #238
• mp4 Add more HEIF boxes support #221
• mpeg_pes Support MPEG1 #219
• rtmp Add Real-Time Messaging Protocol decoder. Only plain RTMP for now. #214
• matroska Symbol name cleanup #220
• tcp Better port matching and make it possible to know if byte stream has start/end. #223
• udp Better port matching #223

Changelog

• 010f6430 Update docker-golang from 1.17.8 to 1.18.0
• 05096f50 Update docker-golang from 1.18.0 to 1.18.1
• e5f61e22 Update github-go-version from 1.17.7, 1.17.7, 1.17.7 to 1.18.0
• fdfc5c5b Update github-go-version from 1.18.0, 1.18.0, 1.18.0 to 1.18.1
• 4ea362e3 Update github-golangci-lint from 1.44.2 to 1.45.0
• 2a90485b Update github-golangci-lint from 1.45.0 to 1.45.2
• d9195ac4 Update gomod-mapstructure from 1.4.3 to 1.5.0
• cf88bc11 Update make-golangci-lint from 1.44.2 to 1.45.0
• 3a0799cb Update make-golangci-lint from 1.45.0 to 1.45.2
• 34cbe487 amf0: Decode strings in more detail
• b2a865ea avc_sps: Add chroma format name mapping
• b35b1804 decode,format: Add d.FieldFormatOrRaw(Len)
• f4480c6f decode,interp: Support for format specific options
• 5ff67e4c formats: Sym and field name cleanup to be more jq friendly
• 3c029925 github: Update action versions
• 02a97fa3 gojq: Rebase fq fork
• 2e240447 gojq: Rebase fq fork
• 518f6af4 gojq: Rebase fq fork
• 88f791e0 gojq: Rebase fq fork
• 8c918702 gojq: Rebase fq fork
• adde8c70 gojq: Rebase fq fork
• d79afeb3 gojq: Rebase fq fork
• dd0d97ea gojq: Rebase fq fork (speedup and fix range with JQValue)
• afd724bf gojq: Rebase fq fork. Fixes JQValue path tracking when iterating
• 74978c9d hevc: Add hevc_vps, hevc_sps and hevc_pps decoders
• c0202483 hevc_vpc,hevc_sps: Use same naming for profile as in spec
• 09385c61 id3v2: Add 2.0 PIC support
• 9cb4b57a interp,cli: Handle ctrl-c properly
• 607202bb interp: Don't truncate last display column
• 6f03471d interp: Paths with a array as root was missing start dot
• dabad850 interp: Proper display column truncate
• e8678ca8 interp: Remove opts refactor leftover
• d376520f interp: Remove to*range pad argument and fix stdout padding issue
• 087d1241 interp: Simpler and more efficient hexdump
• 21ad628a interp: dump: Show field name for compound values in arrays
• e8dc7112 ipv6,icmpv6: Add decoder
• d6c31dac macho: Add section and encryption_info raw data fields
• 5424eed7 macho: Cleanup syms and remove redundant fat_arch struct
• f8d79a57 matroska: More sym cleanup
• f34ebd83 mp4: Add more HEIF boxes
• f8fd6b7f mp4: Add more HEIF boxes
• 39ba5c4d mpeg_pes: Support mpeg1 and some cleanup
• d8aaf303 rtmp,amf0: Add decoders
• 788b0ac1 rtmp,amf0: Improve decoders, aac asc, chunk stream interrupt, fix amf0 ecma arrays
• 5d25bbc2 tcp,udp: Refactor and make port matching better

Added macho decoder (thanks @Akaame), nicer REPL interrupt, error and prompt, add slurp/spew functions and explode for binary.

Added fq talk slides from Binary Tools Summit 2022 to README.md.

Changes

• Major query rewrite refactor to share code for slurp-ish functions repl, slurp and future help system. #178
• REPL improvements:
  • Much improved eval and output interrupt. Should fix more or less all issue with un-interruptable long outputs. It is still possible to get "hangs" if some decode value ends up being expanded into a huge string etc. #191 #192
  • Prompt paths now has colors support. #181
  • Shows an arrow on parse error.
  • Faster on multi inputs. #165
• Speedup interpreter by skipping redundant includes. #172
• gojq fork rebase: #179
  • Fixes try ... | ... catch precedence issue.
  • tonumber now supports non-base-10 numbers.
• Add slurp/spew to collect outputs and outputs them later.
  • 1,2,3 | slurp("a") collects, later do spew("a") to output them. Also a global array $a will be available. #178
• Add explode for binary. #188
  • "a" | tobits | explode return bits [0,1,1,0,0,0,0,1].
  • "åäö" | tobytes | explode return utf8 bytes instead of codepoints [195,165,195,164,195,182].
• Add optional sub topic to --help: #177
  • Replace --formats with --help formats. #181
  • Add --help options to see all default option values. #181
• Remove var, use slupr instead.

Decoder changes

• macho Add decoder. Thanks @Akaame #43
• mp4 Support colr box. #176

Changelog

• ee5e4718 Update docker-golang from 1.17.7 to 1.17.8
• ca04cc20 Update github-golangci-lint from 1.44.0 to 1.44.1
• 5c6e1d32 Update github-golangci-lint from 1.44.1 to 1.44.2
• 1b8e6936 Update make-golangci-lint from 1.44.0 to 1.44.1
• 9d5ba826 Update make-golangci-lint from 1.44.1 to 1.44.2
• cd2cbef6 decode: Some cleanup
• 9e4f2641 dev: Add .jq-lsp.jq to add additional builtins for jq-lsp
• c6a90cfc doc,asn1_ber: Add more documentation
• c53bd777 doc: Add bts2022 video
• b97776c9 doc: Add fq bts2022 presentation
• d334c2d4 doc: Add href in supported format list
• c95b0d6d doc: Forgot make doc
• a202df9a doc: Improve and fix some typos
• 9ec1d357 doc: Improve project description
• 758b2d0e doc: Regenerate after macho merge
• 920629f5 doc: Regenerate and fix macho section size
• d3397cf9 doc: Tweak format diagram
• d47e04c4 fixup! macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff
• 27e76157 format: Simplify torepr, no need for _f function
• 206dcd02 fuzz: Include more testdata seed files
• be6f0093 gojq: Rebase fq fork and add support for non-10 base for tonumber
• 33efb02a interp,repl: Add path and value colors to prompt
• 41551de3 interp,repl: Improved eval and output interrupt
• dff7e7da interp: Cleanup binary regexp overloading and add explode
• fe8183b5 interp: Color parse in jq
• 6f10745a interp: Fix interrupt regression after query rewrite refactor
• f66f115e interp: Make _finally handle null, call fin once and last
• eeb59152 interp: Make help output less wide
• 9dc59e5d interp: Move _is_decode_value to jq
• 0bc11719 interp: Move opts eval to options.jq
• 3f50bb90 interp: Rework formats and options help
• 03f450f8 interp: Skip redundant includes
• c5918d23 macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff
• 5c974209 macho: TS string to UTC
• 04eae939 macho: add basic docs
• 5e95d1c3 macho: add cpuSubTypes
• 2638f419 macho: add darwin_amd64 test
• 5c5bd879 macho: add fqtest actualization
• bf214d5e macho: add nolint suppression to const defs
• 333a3243 macho: add scalar.Hex mapper to addr fields
• a86e7043 macho: add section type parsing
• 90b94631 macho: adopt plural-singular scheme for FieldStructArrayLoop
• b78ed02f macho: barebones decoder impl
• e199d219 macho: basic impl for ar and fat file parsing
• 66feebc5 macho: change parseFlags impl for ordered results
• b5fe9ce6 macho: change registry description
• 20e5be3f macho: delete ar decoder code
• efdd0bf5 macho: discard lc_ and lowercase command names
• b0911af2 macho: docs review changes
• a29bfca5 macho: expand filetypes and header flags
• fb0654ec macho: fix FieldUTF8NullFixedLen for segname
• d1f093ce macho: fix fat header decode bug
• 0d648928 macho: fix null in segname sectname
• 9eb71dc6 macho: generate doc via make doc
• 3991c51a macho: handle unknown lc_commands better
• ef2919b3 macho: introduce arm and fat tests
• 98c9840d macho: linting changes for ar parse
• 1feb81c9 macho: little-endian to little_endian
• 141a8e84 macho: mach_header_X to header
• 9206d9d8 macho: magicToHex to scalar.Hex
• 2021b054 macho: make actual
• 70b84cde macho: ntools fix LC_MAIN fix
• 78699f3a macho: parse flags individually
• 4016ad0b macho: parse segment section flags
• 5a48cb30 macho: refactor prebound_dylib
• 2e7767cd macho: remake docs
• 33347503 macho: reuse ar decoder
• 228757b9 macho: review fixes
• 5ee9a23c macho: review fixes
• e3daee7d macho: simplify thread state decoder
• 70c9d519 macho: thread state visualization
• a4789dc1 macho: timestamp mapper
• 2ccb8087 macho: update test cases v to dv
• 74abe990 macho: update tests
• 12eb7cc5 macho: use FieldUTF8NullFixedLen
• 5f4ad410 macho: use FieldUTF8NullFixedLen for segname
• f8690e6c mp4: Add colr box support
• b157751a mp4: Reformat and use dv in test
• 0a043f90 repl,interp: Refactor repl and slurp
• ca8cdadb repl: Add comments and query from/to helper
• 9cb4205b repl: Correct error arrow position in color mode
• e238f292 repl: Speedup multi input to sub-repl
• 56ae4a0c test: Make expect cli test more robust