#chunks #epochfs #epoch-fs

epochfs

EpochFS is a versioned cloud file system with git-like branching, transaction support

7 releases

0.1.5 Mar 11, 2025
0.1.4 Feb 21, 2025
0.0.1 Jan 24, 2025

#55 in #chunks

Download history 101/week @ 2025-01-23 7/week @ 2025-01-30 7/week @ 2025-02-06 124/week @ 2025-02-13 341/week @ 2025-02-20 38/week @ 2025-02-27 108/week @ 2025-03-06 40/week @ 2025-03-13

67 downloads per month

Apache-2.0

29KB
438 lines

EpochFS

EpochFS is a versioned cloud file system with git-like branching, transaction support.

Motivation

EpochFS is inspired by delta Lake and lakeFS, it is a data lake but for files, offering git-like branching, transaction support, and scaling to handle exabyte-level data effortlessly.

Features in minds

In my minds, EpochFS will support the following features:

  • A library that allows users to integrate and operate EpochFS directly within their applications.
  • A FUSE implementation enabling users to mount EpochFS as a local file system.
  • Support for both serverless mode and meta-server mode:
    • Serverless Mode: EpochFS stores all metadata within the storage.
    • Meta-Server Mode: EpochFS offloads all metadata to a separate server and commits changes to the storage layer, ensuring eventual consistency.
  • Capability to seamlessly convert an existing bucket into EpochFS at no cost.
  • Functionality to export and restore an EpochFS instance.
  • Support for multiple storage backends, including S3, GCS, and Azure Blob Storage.

Current Plan

The current plan is implement the basic functionalities of EpochFS and provide a library that allows users to integrate and operate EpochFS directly within their applications.

License

Licensed under Apache License, Version 2.0.

Dependencies

~40–55MB
~1M SLoC