3 releases (1 stable)
1.0.0 | May 25, 2022 |
---|---|
0.1.1 | May 16, 2017 |
0.1.0 | May 15, 2017 |
#14 in #concatenate
10KB
122 lines
catcsv
: Concatenate directories of possibly-compressed CSV files
This is a small utility that we use to reassemble many small CSV files into much larger ones. In our case, the small CSV files are generated by highly-parallel by Pachyderm pipelines doing map/reduce-style operations.
Usage:
catcsv - Combine many CSV files into one
Usage:
catcsv <input-file-or-dir>...
catcsv (--help | --version)
Options:
--help Show this screen.
--version Show version.
Input files must have the extension *.csv or *.csv.sz. The latter are assumed
to be in Google's "snappy framed" format: https://github.com/google/snappy
If passed a directory, this will recurse over all files in that directory.
Wish list
If you'd like to add support for other common compression formats, such as *.gz
,
we'll happily accept PRs that depend on either pure Rust crates, or which
include C code in the crate but still cross-compile easily with musl.
Related utilities
If you're interested in this utility, you might also be interested in:
- BurntSushi's excellent xsv utility, which features a wide variety of
subcommands for working with CSV files. Among these is a powerful
xsv cat
command, which has many options thatcatcsv
doesn't (but which doesn't do directory walking or automatic decompression as far as I know). - Faraday's scrubcsv utility, which attempts to normalize non-standard CSV files.
Dependencies
~7–18MB
~221K SLoC