#record #timeline #msgpack #command-line-tool #data #creation #rdump

flow-record

library to create records to be used by the rdump cli utility

14 unstable releases (3 breaking)

0.4.9 Oct 4, 2024
0.4.8 Oct 4, 2024
0.4.4 Sep 25, 2024
0.3.0 Sep 19, 2024
0.1.1 Sep 16, 2024

#582 in Encoding

Download history 270/week @ 2024-09-12 551/week @ 2024-09-19 254/week @ 2024-09-26 638/week @ 2024-10-03 69/week @ 2024-10-10 33/week @ 2024-10-17 10/week @ 2024-10-24 23/week @ 2024-10-31 13/week @ 2024-11-07

84 downloads per month
Used in ntdsextract2

GPL-3.0 license

71KB
740 lines

Crates.io Crates.io Crates.io (latest)

flow-record

Library for the creation of DFIR timelines, to be used by rdump

Record flow format

Basically, the record format uses MsgPack. A record stream is a sequence of tuples, each containing of a 4 byte size field and a msgpack encoded record pack (see below).

The very first of those tuples is a special case; it is some kind of a header.

                                ┌────────[record size] bytes───────┐ 
                               /                                    \
┌──────────────────────────────┬────────────────────────────────────┐
│record size (32bit big endian)│     msgpack encoded content        │
└──────────────────────────────┴────────────────────────────────────┘

Header

The header is formed by the serialized version of the string RECORDSTREAM\n, encoded as bin8:

   ┌───────────────msgpack type: bin8
   │                                 
   │    ┌──────────length: 13 bytes  
   │    │                            
   │    │    ┌─────13 bytes of data  
   ▼    ▼    ▼                       
┌────┬────┬──────────────┐           
│0xc4│0x0d│RECORDSTREAM\n│           
└────┴────┴──────────────┘           

In the following description I omit the fact that every distinct record and every descriptor must be preceded by its length.

Record packs

All data in the record format are specified as a record pack, which is simply a tuple (a fixarray of length 2) consisting of an record pack type and the record pack data.

   ┌──────────────────────────── msgpack type ext8/ext16/ext32
   │    ┌─────────────────────── length of content            
   │    │    ┌────────────────── type id must be 0x0e         
   │    │    │     ┌──────────── array of length 2            
   │    │    │     │    ┌─────── record pack type             
   ▼    ▼    ▼     │    │    ┌── record pack data                      
┌────┬────┬────┬───┼────┼────┼──────────────────────────      
│    │    │    │   ▼    ▼    ▼                                
│    │    │    │┌────┬────┬──────────────────                 
│0xc7│    │0x0e││0x92│    │                                   
│    │    │    │└────┴────┴──────────────────                 
│    │    │    │                                              
└────┴────┴────┴────────────────────────────────────────      

The following record pack types are known:

Object ID Raw value Description
RecordPackTypeRecord 0x1
RecordPackTypeDescriptor 0x2 a record descriptor
RecordPackTypeFieldtype 0x3
RecordPackTypeDatetime 0x10
RecordPackTypeVarint 0x11
RecordPackTypeGroupedrecord 0x12

Descriptor

Every record must have some certain type, which must be specified using a record descriptor first. A record descriptor is a record pack of type RecordPackTypeDescriptor, which is wrapped as an msgpack ext8 (depending on its size). The msgpack type id is 0x0e.

Consider the following type:

struct test_csv_test1 {
       field11: String,
       field12: String,
       field13: String
}

which will the following msgpack encoding:

raw value explanation
0xc7 This is an ext8 record
0x43 length of the containing data
0x0e marker for rdump that this contains an object

The record pack itself will be the msgpack encoded equivalent of the following data:

[
	2,
	[
		"test_csv_test1",
		[
			[
				"string",
				"field11"
			],
			[
				"string",
				"field12"
			],
			[
				"string",
				"field13"
			]
		]
	]
]

It is important to note that every field is encoded as a tuple where the first entry is the datatype, and the second is the field name. The following datatypes are supported:

Datatype Mapped from Rust type Explanation
boolean bool
command
dynamic
datetime DateTime<TZ: TimeZone> UNIX timestamp, encoding as integer in msgpack
filesize flow-record-common::Filesize
uint16 u8, u16
uint32 u32, u64
float f32, f64
string String
stringlist
dictlist
unix_file_mode String the chmod formatted string will be converted to u16 internally
varint i8,i16, i32, i64
wstring
net.ipv4.Address
net.ipv4.Subnet
net.tcp.Port
net.udp.Port
uri
digest
bytes Vec<u8>
record
net.ipaddress
net.ipnetwork
path flow_record_common::types::Path

Identifier

A record descriptor is identified by

  • the name of the record type
  • a hash, which equals the first 32 bit of a SHA256-Hash of the record type name and the names and types (in that order) of the record fields. For example, the above struct would have the following input for the hash function: test_csv_test1field11stringfield12stringfield13string, which would result in the hash 12a9d8d90aa34e5068dbf6692b82baf6fff0143eeaa84d7b2a9c92021f7747c2. Here we take the first 4 bytes 12a9d8d9, interpret them as byte endian integer 313120985 and use this as hash.

Every remaining record can refer to a record descriptor using the name and hash of it.

Record data

A record contains a reference to the record descriptor and a list of values, in the order of fields like specified in the descriptor.

        ┌───────────────────────────────────────────────────────────── record pack type 1           
        │                     ┌─────────────────────────────────────── name of the descriptor       
        │                     │                   ┌─────────────────── hash of the descriptor       
        │                     │                   │       ┌─────────── array of values              
        ▼                     │                   │       │   ┌─────── value of the first data field
┌────┬────┬───────────────────┼───────────────────┼───────┼───┼───────────────────────────          
│    │    │                   │                   │       │   │                                     
│    │    │┌────┬─────────────┼───────────────────┼───┬───┼───┼───────────────────────────          
│    │    ││    │             ▼                   ▼   │   ▼   ▼                                     
│    │    ││    │┌────┬────┬─────────────┬────┬──────┐│┌────┬─────────┬─────────┬─────────          
│0x92│0x01││0x92││0x92│0xa?│<struct name>│0xce│<hash>│││0x9?│<field 1>│<field 2>│...                
│    │    ││    │└────┴────┴─────────────┴────┴──────┘│└────┴─────────┴─────────┴─────────          
│    │    ││    │                                     │                                             
│    │    │└────┴─────────────────────────────────────┴───────────────────────────────────          
│    │    │                                                                                         
└────┴────┴───────────────────────────────────────────────────────────────────────────────          

License: GPL-3.0

Dependencies

~5–7MB
~124K SLoC