19 releases
0.1.18 | Oct 7, 2024 |
---|---|
0.1.17 | Jul 14, 2024 |
0.1.16 | Jun 26, 2024 |
0.1.13 | Mar 21, 2024 |
0.1.1 | Dec 27, 2022 |
#81 in Development tools
316 downloads per month
23KB
441 lines
get-all-crates
Download .crate files of all versions of all crates from crates.io.
Useful for things like noisy-clippy which need to analyze the source code of all crates.
I wrote this tool to saturate a 1000 Mbps connection. From scratch, it can finish downloading in under 20 minutes. It can also check the checksums of a directory of downloaded crates in 40 seconds on 64 cores. For a more fully featured tool, which some of this code is based on, but which is much slower in my experience, check out https://git.shipyard.rs/jstrong/registry-backup.
Usage
$ cargo install get-all-crates
$ git clone https://github.com/rust-lang/crates.io-index /path/to/index
$ get-all-crates --index /path/to/index --out /path/to/crates
Warning: as of 2022 this writes out more than 100 GB of crates.
The output directory structure is similar to how Cargo's registry index is structured:
/path/to/crates ├── 1 │ └── m │ └── m-0.1.1.crate ├── 2 │ └── nu │ └── nu-0.73.0.crate ├── 3 │ └── s │ └── syn │ └── syn-1.0.107.crate └── se └── rd ├── serde │ └── serde-1.0.151.crate └── serde_json └── serde_json-1.0.91.crate
License
Dependencies
~17–32MB
~506K SLoC