2 unstable releases
0.2.0 | Sep 13, 2024 |
---|---|
0.1.0 | Jan 14, 2023 |
#847 in Database interfaces
215KB
1K
SLoC
Delta Sharing client library for Rust
Please note that this project is currently experimental.
This is a simple library for Rust to access data published via Delta Sharing. Under the hood, it uses HTTP APIs exposed by Delta Sharing.
Delta Sharing is an open protocol for secure data sharing, making it simple to share data with other organizations regardless of which computing platforms they use.
Features
- Retrieve Delta Sharing information (shares, schemas, tables and files).
- Query shared table data using Polars.
get_dataframe
downloads the table's parquet files (and caches then locally for subsequent queries) and returns a lazy abstraction (logical plan) over an eager DataFrame. This lazy abstraction provides methods for incrementally modifying that logical plan until output is requested (viacollect
). - Provides both an async Client (
delta_sharing::Client
) and a blocking one (delta_sharing::blocking::Client
).
Pre-requisites
-
Rust is installed, e.g. as described here
-
Delta Sharing is set up with at least one shared table
This library uses profile files, which are JSON files containing a user's credentials to access a Delta Sharing Server. There are several ways to get started:
- Download the profile file to access an open, example Delta Sharing Server that we're hosting here. You can try the connectors with this sample data.
- Start your own Delta Sharing Server and create your own profile file following profile file format to connect to this server.
- Download a profile file from your data provider.
Quick start
- Clone the repo
- Set the
bearerToken
andendpoint
values in theconfig.json
to match your Delta Sharing information. - Run a simple example included with the library that uses an async client:
cargo run --example async
. When executed, it will get and display all the data from the first Data Sharing table it finds. - For an example of using a blocking version of the client to do the same, try
cargo run --example blocking --features blocking
.
Using in your own project
Add delta-sharing
to your Cargo.toml
Development
- Run all tests:
cargo test --features blocking
(orRUST_LOG=debug cargo test --features blocking
for extra troubleshooting) - Run async client tests only:
cargo test
- Style check:
cargo fmt -- --check
Dependencies
~47–79MB
~1.5M SLoC