github.com/wader/fq

Adds decoders for PostgreSQL btree, control and heap files. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka

Adds new option skip gaps and output binary as hex string.

Make bits/bytes formats work a bit more intuitive.

Bug fixes to to_hex, to_base64 and trim functions.

Bug fixes and additions to bson, bitcoin_transaction, mp4, id3v2, html, and matroska formats.

Changes

• bits,bytes now are real binaries and not raw decode value. This means they behave more like you would expect. #666

  # build your own strings(1)-like tool:
  # scan matches range in a binary using a regexp and output ranges as new binaries
  # \w\s looks for whitespace and alpha/numeric characters
  # uses `...` raw string literal to not have to escape
  # select/test to only include strings containing "thread"
  # dd to display with no truncation
  $ fq -d bytes 'scan(`[\w\s]{16,}`) | select(test("thread")) | dd' file.mp4
       │00 01 02 03 04 05 06 07│01234567│
  0x250│36 20 6c 6f 6f 6b 61 68│6 lookah│.: raw bits 0x250-0x262.7 (19)
  0x258│65 61 64 5f 74 68 72 65│ead_thre│
  0x260│61 64 73               │ads     │
       │00 01 02 03 04 05 06 07│01234567│
  0x260│            31 20 73 6c│    1 sl│.: raw bits 0x264-0x273.7 (16)
  0x268│69 63 65 64 5f 74 68 72│iced_thr│
  0x270│65 61 64 73            │eads    │
  

• to_hex,to_base64 now correctly handles raw decode values, before the raw bits would be turned into codepoints and then binary UTF-8 possibly introducing invalid codepoints (0xfffd). Thanks @Rogach #672

  $ fq -r '.uncompressed | to_hex' file.gz
  f6f2074cf77d449d
  
  # with the change to add hex bits format you can also do this now
  $ fq -Vr -o bits_format=hex .uncompressed file.gz
  f6f2074cf77d449d
  

• tovalue Now output a "deep" jq value, before it was shallowly a jq value which could be confusing, ex tovalue | .header could be a decode value.
• New option skip_gaps for -V/tovalue can be used to filter out gap fields when represented as JSON. Gaps are bit ranges that no decoder added any field for. #649
  • Can for example be used to remove trailing data that will end up as gap0 etc.
  • Note that gaps in an array that gets filtered out might affect index numbers.
• Add bits_format=hex to represent raw bits as hex string in JSON. #673

  # output decode tree, JSON with binaries as strings and JSON with binaries as hex strings
  $ fq '.packets[0].packet | ., tovalue, tovalue({bits_format:"hex"})' file.pcap
      │00 01 02 03 04 05 06 07│01234567│.packets[0].packet{}: (ether8023_frame)
  0x28│8c 85 90 74 b8 3b      │...t.;  │  destination: "8c:85:90:74:b8:3b" (0x8c859074b83b)
  0x28│                  e8 de│      ..│  source: "e8:de:27:c8:9a:6e" (0xe8de27c89a6e)
  0x30│27 c8 9a 6e            │'..n    │
  0x30│            08 06      │    ..  │  ether_type: "arp" (0x806) (Address Resolution Protocol)
  0x30│                  00 01│      ..│  payload: raw bits
  0x38│08 00 06 04 00 01 e8 de│........│
  0x40│27 c8 9a 6e c0 a8 01 01│'..n....│
  0x48│00 00 00 00 00 00 c0 a8│........│
  0x50│01 e6                  │..      │
  {
    "destination": "8c:85:90:74:b8:3b",
    "ether_type": "arp",
    "payload": "\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\ufffd\ufffd'Țn\ufffd\ufffd\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\ufffd\u0001\ufffd",
    "source": "e8:de:27:c8:9a:6e"
  }
  {
    "destination": "8c:85:90:74:b8:3b",
    "ether_type": "arp",
    "payload": "0001080006040001e8de27c89a6ec0a80101000000000000c0a801e6",
    "source": "e8:de:27:c8:9a:6e"
  }
  

• Explicit use of d/display on a binary will now always hexdump, this feels more intuitive. Before it could output raw binary as display is used as an implicit output function. Implicit (you don't mention d at all) can still output raw binary. #665

  # outputs raw binary (if stdout is not a tty)
  $ fq -n '[1,2,3] | tobytes' | xxd
  00000000: 0102 03
  
  # outputs hexdump (even if stdout is not a tty) as we explicitly use d
  $ fq -n '[1,2,3] | tobytes | d' | cat
     |00 01 02 03 04 05 06 07|01234567|
  0x0|01 02 03|              |...|    |.: raw bits 0x0-0x2.7 (3)
  

• trim can now handle multi-line strings. #668
• Help texts originated from markdown are now rendered a bit more nicely and compact. #661
• Fix weirdly rendered monospace font in demo SVG when using Japanese locale and Chrome on macOS. Thanks @acevif helping with investigation. #655 #662
• Fix a bunch of typos. Thanks @kianmeng and @castilma. #644 #670

Decoder changes

• bits,bytes Is a proper binary not a raw decode value. #666
• bitcoin_transaction Properly decode witness items array. #671 Thanks @Rogach
• bson Add javascript, decimal128, minkey and maxkey support. Fix decoding of datetime and use the correct size type for binary and document size. Thanks Matt Dale @matthewdale. #650
• id3v2
  • Add WXXX frame support. #656
  • Add CTOC flags support. #657
• html Is now probeable. #667
• matroska Fallback raw is probe fail or file_data #645
• mp4
  • Better description for terminator atom (is a QuickTime thing) #651
  • ctts,infe,iinf,trun more proper decoding based on version. #643
  • Use correct QuickTime epoch for timestamps. #674
• pg_btree,pg_control,pg_heap Add PostgreSQL btree, control and heap support. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka. #415

Changelog

• d4f8dfa2 Add flavour arg to postgres parser
• 3c6ea870 Add heap infomask flags parser to PostgreSQL
• 0107d122 Add pgproee14 flavour to postgres
• 9a96da86 Add postgres pg_control parser
• 3b81d99f DBState enum for postgres
• 51878dcd PostgreSQL heap page parser implememtation.
• 6ed02639 PostgreSQL: accept only normal item pointers
• ffc08cfc PostgreSQL: add heap for pgpro14
• b4ae1d58 PostgreSQL: add pg_control for pgproee 12
• a6537107 PostgreSQL: add pg_control to pgpro14
• d9de2d4f PostgreSQL: fix
• 96a86e20 PostgreSQL: fixes
• 96a96a5b PostgreSQL: heap impl for pgproee 12
• 6618e766 PostgreSQL: heap impl for version 11
• ce9ae761 PostgreSQL: heap tuples implementation
• ffd7c9b0 PostgreSQL: implement pgproee 14
• 850dc608 PostgreSQL: lp_flags format
• f5278f38 PostgreSQL: pg_control for ver 12
• 07056c2b PostgreSQL: pg_control impl for pgproee 10
• 972c5a39 PostgreSQL: pg_control, pgheap impl for pgproee13
• d1487278 PostgreSQL: pgheap impl for pgproee10
• 2d3884a3 PostgreSQL: pgproee11 heap impl
• d8b891c0 PostgreSQL: pgproee11 pg_control impl
• b722b219 PostgreSQL: ref
• fdb3b3e4 PostgreSQl: fix offset
• 621c4c4b PostgreSQl: heap impl for version 13
• c273a6c9 PostgreSQl: pg_control impl for version 13
• 01b380e8 PostgrreSQl heap decode refactoring
• bebdfa94 Try to implement pgwal - fail.
• c9d3b8fb Update docker-golang to 1.20.4 from 1.20.3
• 2a5927b5 Update github-go-version to 1.20.4 from 1.20.3
• c405afd4 Update gomod-golang-x-crypto to 0.9.0 from 0.8.0
• ed0cd6d2 Update gomod-golang-x-net to 0.10.0 from 0.9.0
• deaf5ef0 WalLevel for postgres
• f069ddc2 [WIP] initial attempt to add postgres
• 2b1bdfb3 add icu version mapper
• cf1e7b23 add pgpro11 for postgres
• 9570d4df add pgpro12 postgres
• 7bf6b11e add pgpro13 heap
• 2b3035fe add pgpro13 to postgres
• f56c72d3 add postgres tests for mem, cpu profiling
• 2ee01f79 allow to change FillGaps in decoder
• a3361e70 bitcoin: fix witness item structs
• 222cd88b bits,bytes: Behave as binary instead of raw decode value
• 9a982d0a bson: add BSON test file generator module and correct BSON format docs
• 40630d39 bson: fix doc formatting and add author info
• 2017ff87 bson: support all non-deprecated types and fix int/uint bugs
• b08ef00d decode,interp: Refactor format groups into a proper struct
• af68511a dev,doc Clarify some dev docs and rename launch.json to be a template
• 97c952b3 doc: Add some more examples
• 88be3a7f doc: Hopefully fix svg fixed font issue
• dd4fa268 doc: fix typos
• b0e4da28 fix non-ascii characters handling in to_hex and to_base64 functions
• a4a332bf formats: Clenaup naming a bit
• b3b6cd0e gzip.go: fix typo in variablename: delfate
• 2c505fee help,markdown: Rewrote and made text rendering nicer
• e2eb6670 html: Add to probe group
• d010dcec id3v2: Add WXXX (desc/url) frame support
• f237db27 id3v2: Decode CTOC flags
• 684a0838 interp,decode: Support decode group argument
• 8a468f45 interp: Add hex bits format
• c5127139 interp: Add skip_gaps option for tovalue/-V
• 033498b2 interp: Don't output raw binary if display is called explicitly
• ee66fece interp: Make tovalue output behave as jq value
• d5ae1165 interp: trim: Add multi-line support
• 8d317ddf lsn mapper
• fcd7fbcc mappers for postgres
• 8941b139 matroska: file_data: Fallback to raw if probe fails
• 7adc1e70 mp4: Better description for QuickTime terminator atom
• 493848a7 mp4: Use correct epoch for quicktime timestamps
• 3c6d31b0 mp4: ctts,infe,iinf,trun: More ISOMFF version handling
• d6f785c6 pcap: Add forgotten help test
• 8cf83fbc pg_control implementation
• 441eceea pgpro version mapper
• 5ff62735 postgres: add additional checks in pg_heap
• 46765906 postgres: add argument to calc page's check sum correctly
• 7fd41090 postgres: add btree index tests
• 1aa08e33 postgres: add btree, pg_control to how_to.md
• 2423f86b postgres: add how_to.md - how to generate test files for postgres
• 97bbc22a postgres: add page arg in pg_btree, change args names in pg_heap
• 6aed2387 postgres: add pg_wal for pgproee11 as copy of postgres14
• 08eb3034 postgres: add postgres format docs, refactoing postgres flavours
• 90386a65 postgres: add postgres.md to format
• 09c42c35 postgres: add state to wal struct
• c2591ac8 postgres: add test data with specific values
• de68785a postgres: add test files
• 4db1284f postgres: add tests
• 3c7ed5d7 postgres: add tests data
• e66baa75 postgres: add wal checks
• fb7778a5 postgres: add wal tests
• 9f61e637 postgres: allow all flovours to decode btree index
• ccf2edb5 postgres: better versions probing in pg_control, fix holes, better tests
• f3f259af postgres: btree add free space
• efda7b32 postgres: btree handle full file
• 87b7acf3 postgres: btree impl
• d370f5d9 postgres: btree impl
• e8391916 postgres: btree refactored by Mattias Wadman
• 9f1adb2d postgres: change AssertPosBytes to AssertPos (bits)
• 3e09f9f1 postgres: change tuple struct in heap
• e6a9cdbe postgres: doc
• 6281b50d postgres: exclude wal tests for now
• 0ea20e68 postgres: fail on error in pg_heap
• ba8b90ba postgres: fill gap alignment in heap tuple
• 1d9ef300 postgres: first correct read of WAL file
• e5f15c5f postgres: fix compilation, fix tests
• bffa0083 postgres: fix error in tests
• c23bc421 postgres: fix line endings in error messages, simplify code, add comments
• 9508a209 postgres: fix lint
• 85c04228 postgres: fix linter
• e8bb1692 postgres: fix pg_wal when XLogRecord size is more than page size
• 666bbfba postgres: fix some unknown, chanche tests tovalue -> dv
• de3ecf16 postgres: generate docs by embedded md
• 939c7c17 postgres: how_to.md
• dafbf4b7 postgres: lint fixes
• 6fe5f05f postgres: lint, doc
• 9f5036a3 postgres: made root an array
• 46e1e337 postgres: make page size const
• 1e24d70e postgres: move SeekAbs(0) to Probe
• ae838b92 postgres: move postgres.md to formats.md, add btree tests
• 03d8fe1c postgres: page sum impl
• 22a6cfa5 postgres: pg_btree add opaque flags
• dd84d321 postgres: pg_btree begin impl
• b06c9bc2 postgres: pg_control change default flavour to empty string - it uses versions prober. Fix root name in pg_heap.
• edb56502 postgres: pg_control refactoring
• 9deab2ea postgres: pg_heap fix page_begin, page_end
• 7dd7dbee postgres: pg_heap reafactoring
• 00de0a96 postgres: pg_heap refactoring
• 35124bf2 postgres: pg_heap refactoring
• e8d8caca postgres: pgpro wal implementation
• d7a0f930 postgres: pgpro wal refactoring
• 067f8d56 postgres: pgwal checks
• 78583731 postgres: postgres 10 support
• 5664c0a4 postgres: refactor ItemIdData
• 08c53523 postgres: refactoring
• 296ce68e postgres: refactoring
• 7b52149c postgres: refactoring
• 7c92715f postgres: refactoring
• 7dedcbab postgres: refactoring
• a4d904e1 postgres: refactoring
• bedc480a postgres: refactoring
• d7bcca0a postgres: refactoring
• e06fa6e1 postgres: refactoring
• e57c3b98 postgres: refactoring
• f224ed00 postgres: refactoring
• f6f8d5c0 postgres: refactoring
• ff4b6fdf postgres: refactoring - remove GetHeapD
• 7f7f729c postgres: refactoring, tests
• 12b86973 postgres: regenerate docs
• 8f55e177 postgres: remove SeekAbs(0) where it's not used.
• 6e2e44d6 postgres: remove arg in pg_btree
• 5eea605f postgres: remove duplicate tests
• 5bb86544 postgres: remove lsn parameter in pg_wal
• 60709e5a postgres: remove pg_wal. Failed to implement.
• e87d5a6b postgres: remove unused code
• 448c3690 postgres: try to implement pg_wal
• 586c803f postgres: try to implement wal
• 7a89234b postgres: update doc
• c9350de3 postgres: use bit stream instead of masks to get flags
• c9b263e9 postgres: version 15 support
• a4c1c5b8 postgres: wal const
• e311434b postgres: wal decoding implement
• c105fcdd postgres: wal impl
• 7c1dfbd0 postgres: wal implementation
• 26bff144 postgres: wal refactoing
• b09ec2fc postgres: wal refactoing
• 069babbc postgres: wal refactoring
• bd2bdd64 postgres: wal refactoring
• 015b7705 postgres: wal support multiple xlog_body for wal record
• c3ef3411 postgresql: general logic for pg_heap, pg_btree
• 721c1ab3 psotgres: refactoring
• 03274113 ref
• c8ece642 show mock_authentication_nonce as hex
• af0e2207 unix time mapper for postgres

Mostly a bug fix release but adds -V for easy JSON output.

Changes

• Add -V argument to default output JSON instead of decode tree in case of decode value. #385 Thanks @peterwaller-arm for reminding me to merge this.

  # default in case of decode value is to show a hexdump tree
  $ fq '.headers | grep_by(.id=="TSSE").text' file.mp3
      │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
  0x0c│                           4c 61 76│         Lav│.headers[0].frames[0].text: "Lavf58.76.100"
  0x18│66 35 38 2e 37 36 2e 31 30 30 00   │f58.76.100. │
  
  # with -V an implicit "tovalue" is done
  $ fq -V '.headers | grep_by(.id=="TSSE").text' file.mp3
  "Lavf58.76.100"
  
  # and in combination with -r will for strings output a "raw string" without quotes
  # for other types like number, object, array etc -r makes not difference (same as jq)
  $ fq -Vr '.headers | grep_by(.id=="TSSE").text' file.mp3
  Lavf58.76.100
  

  As a side note -V can be used with binary type also. Then the binary data will be interpreted as UTF-8 and turned into a string.

  # trailing null terminator ends up as codepoint zero `\u0000`
  $ fq -V '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3
  "Lavf58.76.100\u0000"
  
  # with -r null terminator and a new line is outputted
  $ fq -Vr '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3 | hexdump -C
  00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00 0a     |Lavf58.76.100..|
  0000000f
  
  # in contrast raw binary output has no new line separator
  $ fq '.headers | grep_by(.id=="TSSE").text | tobytes' doc/file.mp3 | hexdump -C
  00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00        |Lavf58.76.100.|
  0000000e
  

• Fix issue using decode value in object passed as argument to internal function. #638

  # this used to fail but now works
  fq '.tracks[0].samples[10] | avc_au({length_size: <decode value>})' file.mp4
  

• Some typo fixes. Thanks @retokromer and @peterwaller-arm

Decoder changes

• aiff Basic AIFF decoder added. #614
• matroska Update to latest specification. #640
• msgpack Fix bug decoding some fixstr lengths. #636 Thanks @schmee for reporting.

Changelog

• 4ad1cced Update docker-golang to 1.20.3 from 1.20.2
• f7dca477 Update github-go-version to 1.20.3 from 1.20.2
• c9608939 Update github-golangci-lint to 1.52.0 from 1.51.2
• 0a6b46c8 Update github-golangci-lint to 1.52.1 from 1.52.0
• c4eb67d9 Update github-golangci-lint to 1.52.2 from 1.52.1
• 19140a6f Update gomod-creasty-defaults to 1.7.0 from 1.6.0
• 6e5df724 Update gomod-golang-x-crypto to 0.8.0 from 0.7.0
• 6c4aebfe Update gomod-golang-x-net to 0.9.0 from 0.8.0
• f13cc979 Update gomod-golang/text to 0.9.0 from 0.8.0
• e2af57ee Update gomod-gopacket to 1.1.0 from 1.0.0
• a63fd684 Update make-golangci-lint to 1.52.0 from 1.51.2
• d3d1f0e8 Update make-golangci-lint to 1.52.1 from 1.52.0
• f0b08457 Update make-golangci-lint to 1.52.2 from 1.52.1
• dc4a82ee aiff: Add basic decoder
• c5f6809b decode,fuzz,dev: Move recoverable error check to recoverfn.Run
• 980ecdba decode: Add float 80 reader
• a6c4db75 decode: Cleanup old unused help system code
• 87e5bb14 fix typo
• 0b6ef2a9 golangci-lint: Disable revive unused-parameter and update for new default config
• 427ce78d interp: Add --value-output/-V option to do tovalue before output
• 9a1ef84c interp: Allow and convert JQValues:s (ex decode value) in function arg objects
• 3dd2c61d interp: Fix input completion regression in sub-REPLs
• 5415bfca interp: Make completion work again
• 2a2b64dd matroska: Update ebml specification
• 82da99c9 msgpack: Add str, array and object type tests
• 97360d6f msgpack: fixstr length field is 5 bits
• ffc66db0 readline: remove direct access to (*Instance).Config
• e1b02312 wav: Cleanup avi leftovers

TLS decode and decryption, better streaming matroska/webm support, support raw IP in PCAP and bug fixes.

Changes

• Fix panic when interrupting big JSON output. #573
• Support passing options (-o name=value) to nested decoders. #589
  • Allows for example to pass keylog to a TLS decoder inside a PCAP file or to tell a container decoders to not decode samples inside a ZIP file etc.
• Exit with error if -o name=@path fails to read file at path. #597

Decoder changes

• id3v2 Properly decode CTOC subframes. #606
• matroska
  • Now supports streaming matroska and webm better (master elements with unknown size). #576 #581
  • Add decode_samples option. #574
  • Spec update and clean up of symbols and descriptions. #580
• pcap,pcapng Support raw IPv4 and IPv6 link frames. #599 #590
• tls Add Transport layer security decoder and decryption. #603
  • Supports TLS 1.0, 1.1, 1.2 and some SSL 3.0.
  • Decodes records and most messages and extensions.
  • Can decrypt most common cipher suites if a keylog is provided. See documentation for list of supported ciphers suites.

  # show first 50 bytes of decrypted client/server TLS application data stream
  # -o keylog=@file.pcap.keylog is used to read keylog from a file
  # first .stream is TCP stream, second .stream the application data stream
  $ fq -o keylog=@file.pcap.keylog '.tcp_connections[0].["client", "server"].stream.stream | tobytes[0:50] | dd' file.pcap
      │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
  0x00│47 45 54 20 2f 64 75 6d 70 2f 6c 6f│GET /dump/lo│.: raw bits 0x0-0x31.7 (50)
  0x0c│67 20 48 54 54 50 2f 31 2e 31 0d 0a│g HTTP/1.1..│
  0x18│48 6f 73 74 3a 20 69 6e 77 61 64 65│Host: inwade│
  0x24│72 2e 63 6f 6d 0d 0a 55 73 65 72 2d│r.com..User-│
  0x30│41 67                              │Ag          │
      │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
  0x00│48 54 54 50 2f 31 2e 31 20 32 30 30│HTTP/1.1 200│.: raw bits 0x0-0x31.7 (50)
  0x0c│20 4f 4b 0d 0a 41 63 63 65 70 74 2d│ OK..Accept-│
  0x18│52 61 6e 67 65 73 3a 20 62 79 74 65│Ranges: byte│
  0x24│73 0d 0a 43 6f 6e 74 65 6e 74 2d 4c│s..Content-L│
  0x30│65 6e                              │en          │
  
  # show first TLS record from server
  $ fq '.tcp_connections[0].server.stream.records[0] | d' file.pcap
      │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│.tcp_connections[1].server.stream.records[0]{}: record
  0x00│16                                 │.           │  type: "handshake" (22) (valid)
  0x00│   03 03                           │ ..         │  version: "tls1.2" (0x303) (valid)
  0x00│         00 40                     │   .@       │  length: 64
      │                                   │            │  message{}:
  0x00│               02                  │     .      │    type: "server_hello" (2)
  0x00│                  00 00 3c         │      ..<   │    length: 60
  0x00│                           03 03   │         .. │    version: "tls1.2" (0x303)
      │                                   │            │    random{}:
  0x00│                                 86│           .│      gmt_unix_time: 2249760024 (2041-04-16T21:20:24Z)
  0x0c│18 9d 18                           │...         │
  0x0c│         19 92 33 c2 21 ce 4f 97 30│   ..3.!.O.0│      random_bytes: raw bits
  0x18│28 98 b3 fd 1e 15 f4 36 bb e9 14 f4│(......6....│
  0x24│67 61 66 79 d5 3f 06               │gafy.?.     │
  0x24│                     00            │       .    │    session_id_length: 0
      │                                   │            │    session_id: raw bits
  0x24│                        c0 2f      │        ./  │    cipher_suit: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" (0xc02f)
  0x24│                              00   │          . │    compression_method: "null" (0x0)
  0x24│                                 00│           .│    extensions_length: 20
  0x30│14                                 │.           │
      │                                   │            │    extensions[0:2]:
      │                                   │            │      [0]{}: extension
  0x30│   ff 01                           │ ..         │        type: "renegotiation_info" (65281)
  0x30│         00 01                     │   ..       │        length: 1
  0x30│               00                  │     .      │        data: raw bits
      │                                   │            │      [1]{}: extension
  0x30│                  00 10            │      ..    │        type: "application_layer_protocol_negotiation" (16)
  0x30│                        00 0b      │        ..  │        length: 11
  0x30│                              00 09│          ..│        serer_names_length: 9
      │                                   │            │        protocols[0:1]:
      │                                   │            │          [0]{}: protocol
  0x3c│08                                 │.           │            length: 8
  0x3c│   68 74 74 70 2f 31 2e 31         │ http/1.1   │            name: "http/1.1"
  
  # use ja3.jq to calculate ja3 TLS fingerprint
  # https://github.com/wader/fq/blob/master/format/tls/testdata/ja3.jq
  $ fq -L path/to/ja3 'include "ja3"; pcap_ja3' file.pcap
  [
    {
      "client_ip": "192.168.1.193",
      "client_port": 64126,
      "ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
      "ja3_digest": "bc29aa426fc99c0be1b9be941869f88a",
      "server_ip": "46.101.135.150",
      "server_port": 443
    }
  ]
  

• toml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594
• zip Properly decode EOCD record in zip64 files. Thanks @0-wiz-0 for report and spec interpretation. #586 #596
• xml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594

Changelog

• 0581ecea Update docker-golang to 1.20.1 from 1.20.0
• 72870a5a Update docker-golang to 1.20.2 from 1.20.1
• 02e573a9 Update github-go-version to 1.20.1 from 1.20.0, 1.20.0, 1.20.0
• c5130887 Update github-go-version to 1.20.2 from 1.20.1
• ce263726 Update github-golangci-lint to 1.51.1 from 1.51.0
• 75bfdda3 Update github-golangci-lint to 1.51.2 from 1.51.1
• b1d9306b Update gomod-golang-x-crypto to 0.6.0 from 0.5.0
• c03d3ccd Update gomod-golang-x-crypto to 0.7.0 from 0.6.0
• 2430fba7 Update gomod-golang-x-net to 0.6.0 from 0.5.0
• dd8ab799 Update gomod-golang-x-net to 0.7.0 from 0.6.0
• 80a07446 Update gomod-golang-x-net to 0.8.0 from 0.7.0
• 97643b98 Update gomod-golang/text to 0.7.0 from 0.6.0
• e7168b99 Update gomod-golang/text to 0.8.0 from 0.7.0
• 36df57eb Update make-golangci-lint to 1.51.1 from 1.51.0
• 70e08faa Update make-golangci-lint to 1.51.2 from 1.51.1
• 50d26ec7 colorjson: Handle encoding error value
• 5c8e1151 colorjson: Refactor to option struct
• 8e0dde03 decode: Support multiple format args and some rename and refactor
• a1bb630a doc,fq: Improve cli help and some cleanup
• 156aeeca doc: Add FOSDEM 2023 talk
• 3e0ebafa doc: Run make doc
• 3cc83837 gojq: Update fq fork
• dec433fc help,markdown: Fix double line breaks when converting to text
• c75a83c8 help: Show default option value as JSON
• cc52a441 id3v2: Decode subframes for CTOC and add struct for headers
• dc79a73b interp,json: Move error handling to colorjson
• 73db6587 interp: Exit with error if -o name=@path fails to be read, also document
• c8666eeb ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
• b60aceca matroska: Add decode_samples option
• 9aaf2ddf matroska: Add unknown size test and add description to ebml header
• a8d0bf4d matroska: Assume master with unknown size has ended if a valid parent is found
• 0d14d7b4 matroska: Handle unknown size for non-master types a bit better
• c890a289 matroska: Update spec and make refs in descriptions look nicer
• 6c032455 pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6)
• d4ea6632 pcap: Add ipv4 fragments tcp test
• f50bd6ee readline: Update fq fork
• 9852f56b tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
• 56edb59e toml,xml: Fail fast on invalid content
• 5228fdd6 zip: Correctly look for and decode both zip32/64 EOCD record
• bdd6718d zip: Correctly peek for zip64 EOCD