1 unstable release
0.1.3 | Jan 21, 2024 |
---|
#311 in Command-line interface
33KB
312 lines
geoipsed
Fast, inline geolocation decoration of IPv4 and IPv6 addresses written in Rust
Concept
IP address metadata is essential to network defense and incident response. City and country-level geolocation can be clues to rule in or rule out IPs of interest. ASN metadata provides insights of network ownership as well as simply identifying which IPs are internet-routable vs which are internal, private, or bogons.
In command line log analysis, IP geolocation metadata is most useful in addition to the already available log metadata on each line. geoipsed
enriches IP addresses in place leaving the existing context intact.
Features
- IPv4 and IPv6 address support
- City, Country, ASN, time zone fields among the available metadata
- Flexible templating to customize how
geoipsed
decorates matching IPs - Coloring to more readily spot the IPs in the logs
- Optional mode to just emit the matching IPs just like grep/ripgrep
-o
parameter - Spaces are removed from decoration labels so as not to mess up column numbering in your logs
Install
geoipsed
uses Maxmind's GeoLite2-ASN.mmdb and GeoLite2-City.mmdb files. Follow the instructions to obtain these files here: https://dev.maxmind.com/geoip/updating-databases
Currently, geoipsed
looks exclusively to /usr/share/GeoIP
for the mmdb files, however you can override this with the -I
command line option or the environment variable MAXMIND_MMDB_DIR
.
Build and install with cargo:
cargo install --git https://github.com/erichutchins/geoipsed
Usage
geoipsed 0.1.3
Inline decoration of IPv4 and IPv6 address geolocations
USAGE:
geoipsed [OPTIONS] [FILE]...
ARGS:
<FILE>... Input file(s) to process. Leave empty or use "-" to read from stdin
OPTIONS:
-C, --color <COLOR> Use markers to highlight the matching strings [default: auto]
[possible values: always, never, auto]
-h, --help Print help information
-I <DIR> Specify directory containing GeoLite2-ASN.mmdb and
GeoLite2-City.mmdb [env: MAXMIND_MMDB_DIR=]
-L, --list-templates Display a list of available template substitution parameters to use
in --template format string
-o, --only-matching Show only nonempty parts of lines that match
-t, --template <TEMPLATE> Specify the format of the IP address decoration. Use the
--list-templates option to see which fields are available. Field
names are enclosed in {}, for example "{field1} any fixed string
{field2} & {field3}"
-V, --version Print version information
Available fields to use in customizing the -t
parameter:
; geoipsed -L
Available template geoip field names are:
{ip}
{asnnum}
{asnorg}
{city}
{continent}
{country_iso}
{country_full}
{latitude}
{longitude}
{timezone}
Benchmark
Comparing the Rust implementation to a basic Python version against 30,000 lines (~23MB decompressed) of Suricata json eve logs:
Benchmark 1: zstdcat ../30k.log.zst | target/release/geoipsed
Time (mean ± σ): 157.1 ms ± 16.7 ms [User: 167.8 ms, System: 28.9 ms]
Range (min … max): 137.1 ms … 209.9 ms 21 runs
Benchmark 2: zstdcat ../30k.log.zst | python python/geoipsed.py
Time (mean ± σ): 15.209 s ± 0.929 s [User: 15.213 s, System: 0.210 s]
Range (min … max): 14.312 s … 17.076 s 10 runs
Summary
'zstdcat ../30k.log.zst | target/release/geoipsed' ran
96.80 ± 11.84 times faster than 'zstdcat ../30k.log.zst | python python/geoipsed.py'
- Note that a significant factor of this speed difference is the regular expression matching, specifically the IPv6 pattern. If you match just on IPv4, it was only ~5x faster.
Background & Gratitude
-
The historical inspiration for geoipsed was when a sensei taught me long ago that perl's
s/find/replace/g
sed interface can interpret the replace pattern as a perl expression! For example, decoding hexascii, xor'ing, and printing the characters:perl -pe 's/%([a-f0-9]{2})/chr(hex($1)^0x5e)/ieg
. If I can run perl chr and hex functions, can I import geoip libraries, and run those functions? It turns out, yes! Thus geoipsed was born. -
Writing this was an endeavor to learn more Rust, and I'm grateful for burntsushi and sstadick whose awesome crates and well documented code are sources of inspiration and regular references.
Dependencies
~7–16MB
~196K SLoC