#page #offset #reference #data-structures #disk #tree #traits

no-std bin+lib sanakirja-core-async

Copy-on-write datastructures, storable on disk (or elsewhere) with a stable format

2 releases

0.0.2 Apr 1, 2023
0.0.1 Nov 11, 2022

#35 in #data-structure

MIT/Apache

225KB
4.5K SLoC

This crate defines tools to implement datastructures that can live in main memory or on-disk, meaning that their natural habitat is memory-mapped files, but if that environment is threatened, they might seek refuge in lower-level environments.

One core building block of this library is the notion of virtual memory pages, which are allocated and freed by an externally-provided allocator (see how the sanakirja crate does this). The particular implementation used here is meant to allow a transactional system with readers reading the structures concurrently with one writer at a time.

At the moment, only B trees are implemented, as well as the following general traits:

  • LoadPage is a trait used to get a pointer to a page. In the most basic version, this may just return a pointer to the file, offset by the requested offset. In more sophisticated versions, this can be used to encrypt and compress pages.
  • AllocPage allocates and frees pages, because as datastructures need to be persisted on disk, we can't rely on Rust's memory management to do it for us. Users of this crate don't have to worry about this though.

Moreover, two other traits can be used to store things on pages: Storable is a simple trait that all Sized + Ord types without references can readily implement (the direct_repr! macro does that). For types containing references to pages allocated in the database, the comparison function can be customised. Moreover, these types must supply an iterator over these references, in order for reference-counting to work properly when the datastructures referencing these types are forked.

Dynamically-sized types, or types that need to be represented in a dynamically-sized way, can use the UnsizedStorable format.

Dependencies

~0.2–1.1MB
~24K SLoC