#download #io #version #source #registry #cargo-registry #checksum

app get-all-crates

Download .crate files of all version of all crates from crates.io

19 releases

0.1.18 Oct 7, 2024
0.1.17 Jul 14, 2024
0.1.16 Jun 26, 2024
0.1.13 Mar 21, 2024
0.1.1 Dec 27, 2022

#81 in Development tools

Download history 142/week @ 2024-07-08 53/week @ 2024-07-15 51/week @ 2024-07-22 27/week @ 2024-07-29 23/week @ 2024-08-05 18/week @ 2024-08-12 18/week @ 2024-08-19 21/week @ 2024-08-26 16/week @ 2024-09-02 19/week @ 2024-09-09 41/week @ 2024-09-16 62/week @ 2024-09-23 29/week @ 2024-09-30 219/week @ 2024-10-07 39/week @ 2024-10-14 25/week @ 2024-10-21

316 downloads per month

MIT license

23KB
441 lines

get-all-crates

github crates.io build status

Download .crate files of all versions of all crates from crates.io.

Useful for things like noisy-clippy which need to analyze the source code of all crates.

I wrote this tool to saturate a 1000 Mbps connection. From scratch, it can finish downloading in under 20 minutes. It can also check the checksums of a directory of downloaded crates in 40 seconds on 64 cores. For a more fully featured tool, which some of this code is based on, but which is much slower in my experience, check out https://git.shipyard.rs/jstrong/registry-backup.


Usage

$ cargo install get-all-crates
$ git clone https://github.com/rust-lang/crates.io-index /path/to/index
$ get-all-crates --index /path/to/index --out /path/to/crates

Warning: as of 2022 this writes out more than 100 GB of crates.

The output directory structure is similar to how Cargo's registry index is structured:

/path/to/crates
 ├── 1
 │  └── m
 │     └── m-0.1.1.crate
 ├── 2
 │  └── nu
 │     └── nu-0.73.0.crate
 ├── 3
 │  └── s
 │     └── syn
 │        └── syn-1.0.107.crate
 └── se
    └── rd
       ├── serde
       │  └── serde-1.0.151.crate
       └── serde_json
          └── serde_json-1.0.91.crate

License

MIT license.

Dependencies

~17–32MB
~506K SLoC