5 releases (breaking)

0.8.0 May 18, 2024
0.7.0 May 5, 2024
0.6.0 Dec 29, 2023
0.5.0 Jun 25, 2023
0.4.0 May 3, 2023

#39 in #stage

Download history 17/week @ 2024-06-02 11/week @ 2024-06-09 6/week @ 2024-06-16 12/week @ 2024-06-23 15/week @ 2024-06-30 37/week @ 2024-07-07 37/week @ 2024-07-14 13/week @ 2024-07-21 103/week @ 2024-07-28 12/week @ 2024-08-04 9/week @ 2024-08-18 23/week @ 2024-08-25 3/week @ 2024-09-01 17/week @ 2024-09-08 28/week @ 2024-09-15

72 downloads per month
Used in gasket

Apache-2.0

8KB
143 lines

Gasket

An opinionated Rust library for building application using data pipeline semantics. It uses a similar approach to the "Actor model", but fine-tuned for heavy, composable data processing.

Why

  • spliting data processing logic into small stages facilitates development (my own experience)
  • stages are easily composable
  • pipelines are a good abstraction for implementing observability
  • robust loops with error handling, backoff retries, etc are ubiquitous and hard to implement correctly
  • traditional async loops hide too much of the complexity, hard to optimice at scale

Goals

  • zero-cost abstraction (sort of)
  • staticly typed stages, dynamic composition of pipeline at runtime
  • developer has full-control over each work unit
  • robust work loop provided by the library
  • error handling out-of-the-box, with different policies (retry, exit, backoff)
  • fully observable out-of-the-box (metrics, traces)
  • multi-thread pipeline, async option inside stage
  • ergonomic pipeline setup
  • back-pressure out-of-the-box
  • hot swap of stages (add / remove)

Status

Current state of the lib is: "only I understand it". I'm experimenting with the api surface, so there are constant breaking changes happening frequently.

Dependencies

~220–650KB
~16K SLoC