#file-transfer #directory #tokio #io-uring #networking #rkyv #machine

app rcopy

A fast file and directory transfer tool powered by Tokio, RKYV, and io_uring

1 unstable release

0.1.0 May 22, 2023

#11 in #rkyv

MIT license

5.5MB
1K SLoC

rcopy

This program does one thing and one thing only! It remotely copies files or directories as fast as I can figure out how over the network from one machine to another. On my network, it hits several hundred MB/s (or 30GB in 90s).

rcopy uses a few key technologies to accomplish this speed:

  • Asynchronous concurrent Rust using Tokio
  • Zero-copy serialization and deserialization using RKYV
  • Fast-as-possible file reads and writes using io_uring

If you know how to make it faster, please PR!

Install

Install with cargo install rcopy

Run

On the machine that has the file or directory (in my case, ~/hub/models/...Vicuna may be smaller than GPT-4 but the file is still pretty huge!) you want to copy, run:

rcopy send ~/hub/models

send

On the machine you want the files to copy to (we'll copy them to ~/Downloads/models-copy/), run:

rcopy receive -a 192.168.0.185:3120 ~/Downloads/models-copy/

receive

FAQ

Q: What's the use case for this?

A: Already mentioned above, but I literally am tired of moving Linux ISOs and 9GB ggml files between my machines. I realized SFTP and RSync are horrifically slow, so I figured it'd be fun to make my own.

Q: Is this secure?

A: No! Secure your network, this is for moving big model files that you don't care about from one machine to another (or Linux ISOs, or whatever). In the future, I plan on adding an encrypted mode that'll use the SSH keys already on your machines to negotiate AES.

Planned Features

  • Encryption (for obvious reasons)
  • Holepunching (for non-LAN transactions)
  • Data compression (for slower networks)
  • Integrity checks (for careful people)
  • Experiments with multiple TCP streams to find saturation limits

Dependencies

~11–22MB
~289K SLoC