github.com/wader/fq

# or help(matroska) in the REPL
$ fq -h matroska
matroska: Matroska file decoder

Decode examples
===============

  # Decode file as matroska
  $ fq -d matroska . file
  # Decode value as matroska
  ... | matroska

Lookup element using path
=========================

  $ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv

Get path to element
===================

  $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv

References
==========
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus

$ fq '.objects.entries[0] | .key, .value' Info.plist
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}:
0x32│               5c            │     \    │  type: "ascii_string" (5) (ASCII encoded string)
0x32│               5c            │     \    │  size_bits: 12
    │                             │          │  size: 12
0x32│                  43 46 42 75│      CFBu│  value: "CFBundleName"
0x3c│6e 64 6c 65 4e 61 6d 65      │ndleName  │
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}:
0x1ea│         5f                  │   _      │  type: "ascii_string" (5) (ASCII encoded string)
0x1ea│         5f                  │   _      │  size_bits: 15
0x1ea│            10               │    .     │  large_size_marker: 1 (valid)
0x1ea│            10               │    .     │  exponent: 0
0x1ea│               18            │     .    │  size_bigint: 24
    │                             │          │  size: 24
0x1ea│                  41 70 70 6c│      Appl│  value: "AppleProResCodecEmbedded"
0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│
0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│

• Supports torepr

$ fq torepr.CFBundleName Info.plist
"AppleProResCodecEmbedded"

• More robust decoding when string tables are missing. #417<br>

  # extract entry opcodes and disassemble with ndisasm
  $ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \
  | ndisasm -b 64 -
  00000000  B03C              mov al,0x3c
  00000002  66BF0600          mov di,0x6
  00000006  0F05              syscall
  

• Now decodes program header notes. #421

# array with all level 1 and 2 headers
$ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md
[
  "fq",
  "Usage",
  "Presentations",
  "Install",
  "TODO and ideas",
  "Development and adding a new decoder",
  "Thanks and related projects",
  "License"
]

# show details of first two track in file
$ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track
0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│  samples[0:1295]:
0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│
*       │until 0x541697.7 (5473678)   │          │
        │                             │          │  id: 1
        │                             │          │  data_format: "mp4a" (MPEG-4 Audio)
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track
0x00a5e6│                           00│         .│  samples[0:1440]:
0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│
0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│
*       │until 0x540959.7 (5464939)   │          │
        │                             │          │  id: 2
        │                             │          │  data_format: "avc1" (Advanced Video Coding)

# show part of code_section
$ fq '.sections[4].content.code.x[0].code.e | d' add.wasm
    │00 01 02 03 04 05 06 07 08 09│0123456789│.sections[4].content.code.x[0].code.e[0:4]:
    │                             │          │  [0]{}: in
0x3c│                           20│          │    opcode: "local.get" (0x20)
0x46│01                           │.         │    x: 1 (valid)
    │                             │          │  [1]{}: in
0x46│   20                        │          │    opcode: "local.get" (0x20)
0x46│      00                     │  .       │    x: 0 (valid)
    │                             │          │  [2]{}: in
0x46│         6a                  │   j      │    opcode: "i32.add" (0x6a)
    │                             │          │  [3]{}: in
0x46│            0b               │    .     │    opcode: "end" (0xb)

# count opcode usage
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' add.wasm
{
  "end": 1,
  "i32.add": 1,
  "local.get": 2
}

# list exports and imports
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' add.wasm
{
  "export": [
    "memory",
    "add"
  ],
  "import": []
}

• │until 0x1bc.7 (end) (445) │ │ ... </pre>

• For example useful when piping to less and you want fancy unicode and colors:<br> fq -cU d file | less -r

• Normalize to strings is done so that non-string scalars can be used:

  $ fq -n '[[1,true,null,"a"]] | tocsv'
  "1,true,,a\n"
  

• Add 32 bit samples support. #378 Thanks @ktmf01
• Properly decode/checksum samples in partitions with raw samples. #379 Thanks @ktmf01<br> Now successfully decodes all test cases from https://github.com/ietf-wg-cellar/flac-test-files

• Split FAT Macho decoding into macho_fat format which also fixed handling of file offsets in sections. #362
• Decode symbol and string sections. #352

• Support LSF (low sampling frequency) frames. #376
• Skip trying to figure out what main data is for current frame and not. Was incorrect and doing it properly probably requires hoffman decoding. #376

• Decodes to a tree following the WASM binary grammar specification

• Don't require PK header as there seems to be zip files with prepended data. #359
• Correctly limit amount of backwards search for EOCD (end of content directory). #363

• Change attribute prefix to @ instead of - and make it an option attribute_prefix. #401
• Skip default namespace in element names. #389
• Always include attributes and children even when empty in array mode. Makes it a lot easier to work with as you can assume .[1] will be attributes and so on. #357
• Normalize to strings is done so that non-string scalars can be used:

  $ fq -nr '{a: {"-boolean": true, "-number": 123, "-null": null}} | toxml'
  <a boolean="true" null="" number="123"></a>
  

• Allow and ignore trailing <?procinstr?> and improve trailing data error message. #368
• Correctly sort if any #seq is found and also properly sort negative #seq. #384