13 releases
0.2.0 | Dec 26, 2021 |
---|---|
0.1.11 | Dec 4, 2021 |
0.1.9 | Nov 24, 2021 |
0.1.8 | Sep 6, 2021 |
0.1.5 | Aug 22, 2021 |
#8 in #data-engineering
219 downloads per month
Used in 19 crates
180KB
5.5K
SLoC
pipebase
is a tokio
based runtime library for data integration app. It provides basic pipe
s implemented with rust standard library.
Basic Pipes
list of basic pipes built in pipebase
pipe type | implementation | example |
---|---|---|
Exporter |
Printer |
printer |
Poller |
Timer |
timer |
Listener |
LocalFilePathVisitor |
file_path |
Streamer |
FileLineReader |
stateless_word_count |
Streamer |
IteratorReader |
stateless_word_count |
Mapper |
FieldVisit |
field_filter |
Mapper |
FilterMap |
field_filter |
Mapper |
StringSplitter |
stateless_word_count |
Mapper |
Projection |
project_file |
Mapper |
FileReader |
project_file |
Mapper |
FileWriter |
convert_csv |
Mapper |
Conversion |
convert_csv |
Selector |
RandomSelector |
ingest_redis_parallel |
Collector |
TextCollector |
ingest_elasticsearch |
Collector |
InMemoryWindowCollector |
moving_average |
Dependencies
~7–15MB
~171K SLoC