2 releases
0.1.1 | Aug 27, 2019 |
---|---|
0.1.0 | Jul 15, 2019 |
#8 in #first-line
16KB
111 lines
Multiline
multiline
is a CLI utility that can be used to transform realtime log message streams (e.g. log entries from application log file) that contain messages spanning multiple lines (e.g. stack traces) into streams with one message per line.
It can be useful as pre-processing step before sending log file data to syslog which would otherwise interpret messages spanning over multiple lines as separate messages one per line.
Usage
multiline
program will read UTF-8 data from from standard input, one line at a time and write messages spawning potentially multiple lines joined by --join
string to standard output.
To configure message beginning or end matching algorithm --message-pattern
regex, --negate
and --match-last
flags can be used. --strip-pattern
flag can by used to remove matched message pattern from output.
Additionally multiline
can handle multiple streams of log data interlaced together in standard input with use of --stream-id-pattern
regex to match each stream identifying pattern.
Real time processing parameters can be configured with --max-size
flag which limits maximum number of lines that each stream will accumulate and --max-duration
which limits time duration that each message will continue to aggregate lines for. After one of this limits is reached concatenated message is flushed to the output.
Specifying --timestamp
flag will enable printing of timestamps (in ISO 8601 UTC format) for each message constructed based on time when first line of the message was read.
Examples
Joining stack traces
Match every line that starts with space as belonging to previous message with negate
.
multiline --message-pattern '^\s+' --negate --strip-pattern --join "; " <<EOF
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
EOF
Produces:
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Joining lines ending with special escape
Use match-last
with negate
to anchor on the lines without the escape.
Add strip-pattern
to ensure that the escape is not part of the output.
multiline --message-pattern ' >$' --negate --match-last --strip-pattern --join "; " <<EOF
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
EOF
Produces:
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Join messages from multiple interlaced streams of log data
The stream-id-pattern
regex will match the stream name. For each stream a separate concatenation buffer will be maintained.
Timestamp is used to match the beginning of the message in each stream.
multiline --message-pattern '^20' --stream-id-pattern '.*: ' --join " " <<EOF
bar: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet,
foo: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet,
foo: consectetur adipiscing elit
bar: consectetur adipiscing elit
foo: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem,
foo: a placerat ex dictum iaculis.
foo: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
foo: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem,
foo: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: a placerat ex dictum iaculis.
bar: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
bar: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
foo: 2019-07-05 15:42:21 Phasellus sed nulla faucibus,
foo: bibendum dolor vitae,
foo: faucibus dui.
bar: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: 2019-07-05 15:42:21 Phasellus sed nulla faucibus,
bar: bibendum dolor vitae,
bar: faucibus dui.
EOF
Produces:
foo: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet, consectetur adipiscing elit
foo: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem, a placerat ex dictum iaculis.
foo: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
bar: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet, consectetur adipiscing elit
foo: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem, a placerat ex dictum iaculis.
bar: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
foo: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:20 Cras suscipit lacus ex.
foo: 2019-07-05 15:42:21 Phasellus sed nulla faucibus, bibendum dolor vitae, faucibus dui.
bar: 2019-07-05 15:42:21 Phasellus sed nulla faucibus, bibendum dolor vitae, faucibus dui.
Timestamp messages
Enable timestamps with --timestamp
flag.
seq 1 10 | while read I; do echo $I; sleep 0.2; done | multiline --message-pattern '.' --timestamp
Produces:
2019-07-12T13:29:12.872375Z 1
2019-07-12T13:29:13.071095Z 2
2019-07-12T13:29:13.273488Z 3
2019-07-12T13:29:13.475585Z 4
2019-07-12T13:29:13.677506Z 5
2019-07-12T13:29:13.879438Z 6
2019-07-12T13:29:14.082132Z 7
2019-07-12T13:29:14.284793Z 8
2019-07-12T13:29:14.487547Z 9
2019-07-12T13:29:14.690270Z 10
Dependencies
~13MB
~226K SLoC