56 releases (14 stable)
1.4.3 | Jun 9, 2023 |
---|---|
1.4.2 | Feb 15, 2021 |
1.4.1 | Dec 20, 2020 |
1.4.0 | Nov 21, 2019 |
0.8.0 | Jul 28, 2017 |
#406 in Encoding
217 downloads per month
35KB
722 lines
pq
protobuf to json deserializer, written in Rust
pq
is a tool which deserializes protobuf messages given a set of pre-compiled .fdset
files. It can understand varint/leb128-delimited/i32be streams, and it can connect to Kafka.
pq
will pretty-print when outputting to a tty, but you should pipe it to jq
for more fully-featured json handling.
Download
pq is on crates.io: cargo install pq
. You can also download a static binary from the releases page.
Usage
new You can now pass in a proto file and have pq compile it on the fly using protoc
:
$ pq --protofile ./tests/protos/dog.proto --msgtype com.example.dog.Dog <./tests/samples/dog
{
"breed": "gsd",
"age": 3,
"temperament": "excited"
}
Use PROTOC and PROTOC_INCLUDE to control the executed protoc binary and configure the -I=/proto/path
argument (design copied from prost).
To set up, put your *.fdset
files in ~/.pq
, /etc/pq
, or a different directory specified with the FDSET_PATH
env var:
$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/
You can specify additional fdset directories or files via options:
$ pq --msgtype com.example.dog.Dog --fdsetdir ./tests/fdsets <./tests/samples/dog
$ pq --msgtype com.example.dog.Dog --fdsetfile ./tests/fdsets/dog.fdset <./tests/samples/dog
Pipe a single compiled protobuf message:
$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
"age": 4,
"breed": "poodle",
"temperament": "excited"
}
Pipe a varint
or leb128
delimited stream:
$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Consume from a Kafka stream:
$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Convert a Kafka stream to varint-delimited:
$ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
> pq --msgtype com.example.dog.Dog --stream varint
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Pipe kafkacat
output to it:
$ kafkacat -b 192.168.0.1:9092 -C -u -q -f "%R%s" -t my_topic |\
> pq --msgtype=com.example.dog.Dog --stream i32be
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Compile without kafka
To compile pq
without kafka support, run:
$ cargo build --no-default-features
Compile for Windows
- Install Visual Studio Installer Community edition
- Run the installer and install
Visual Studio Build Tools 2019
. You need theC++ Build Tools
workload. Note that you can't just install the minimal package, you also needMSVC C++ x64/86 Build Tools
andWindows 10 SDK
. - Open
x64 Native Tools Command Prompt
from the start menu. - Download and run
rustup-init.exe
- Close and reopen your terminal (so
cargo
will be in your path) - Run
cargo install --no-default-features pq
Note that this will disable the Kafka feature, which is not currently supported on Windows.
Dependencies
~8–18MB
~293K SLoC