56 releases (14 stable)

1.4.3 Jun 9, 2023
1.4.2 Feb 15, 2021
1.4.1 Dec 20, 2020
1.4.0 Nov 21, 2019
0.8.0 Jul 28, 2017

#406 in Encoding

Download history 5/week @ 2024-07-26 1/week @ 2024-08-02 2/week @ 2024-08-23 2/week @ 2024-08-30 114/week @ 2024-09-20 15/week @ 2024-09-27 2/week @ 2024-10-04

217 downloads per month

MIT license

35KB
722 lines

pq license Crates.io

protobuf to json deserializer, written in Rust

pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited/i32be streams, and it can connect to Kafka.

pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.

Download

pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.

Usage

new You can now pass in a proto file and have pq compile it on the fly using protoc:

$ pq --protofile ./tests/protos/dog.proto  --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "breed": "gsd",
  "age": 3,
  "temperament": "excited"
}

Use PROTOC and PROTOC_INCLUDE to control the executed protoc binary and configure the -I=/proto/path argument (design copied from prost).

To set up, put your *.fdset files in ~/.pq, /etc/pq, or a different directory specified with the FDSET_PATH env var:

$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/

You can specify additional fdset directories or files via options:

$ pq --msgtype com.example.dog.Dog --fdsetdir ./tests/fdsets <./tests/samples/dog
$ pq --msgtype com.example.dog.Dog --fdsetfile ./tests/fdsets/dog.fdset <./tests/samples/dog

Pipe a single compiled protobuf message:

$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "age": 4,
  "breed": "poodle",
  "temperament": "excited"
}

Pipe a varint or leb128 delimited stream:

$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Consume from a Kafka stream:

$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Convert a Kafka stream to varint-delimited:

$ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
> pq --msgtype com.example.dog.Dog --stream varint
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Pipe kafkacat output to it:

$ kafkacat -b 192.168.0.1:9092 -C -u -q -f "%R%s" -t my_topic |\
> pq --msgtype=com.example.dog.Dog --stream i32be
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Compile without kafka

To compile pq without kafka support, run:

$ cargo build --no-default-features

Compile for Windows

  1. Install Visual Studio Installer Community edition
  2. Run the installer and install Visual Studio Build Tools 2019. You need the C++ Build Tools workload. Note that you can't just install the minimal package, you also need MSVC C++ x64/86 Build Tools and Windows 10 SDK.
  3. Open x64 Native Tools Command Prompt from the start menu.
  4. Download and run rustup-init.exe
  5. Close and reopen your terminal (so cargo will be in your path)
  6. Run cargo install --no-default-features pq

Note that this will disable the Kafka feature, which is not currently supported on Windows.

Dependencies

~8–18MB
~293K SLoC