#io #dump #concatenation #download #crates-io #single

app db-dump-concat

Tool to concatenate crates.io database dumps

5 releases

0.1.3 Feb 4, 2024
0.1.2 Jan 2, 2024
0.1.1 Sep 28, 2022
0.1.0 Jul 6, 2022
0.0.0 Jul 6, 2022

#589 in Database interfaces

Download history 23/week @ 2024-06-08 23/week @ 2024-06-15 22/week @ 2024-06-22 33/week @ 2024-06-29 18/week @ 2024-07-06 15/week @ 2024-07-13 16/week @ 2024-07-20 40/week @ 2024-07-27 19/week @ 2024-08-03 14/week @ 2024-08-10 15/week @ 2024-08-17 14/week @ 2024-08-24 14/week @ 2024-08-31 16/week @ 2024-09-07 17/week @ 2024-09-14 35/week @ 2024-09-21

84 downloads per month

MIT/Apache

11KB
214 lines

crates.io database dumps

github crates.io docs.rs build status

Library for scripting analyses against crates.io's database dumps.

These database dumps contain all information exposed by the crates.io API packaged into a single download. An updated dump is published every 24 hours. The latest dump is available at https://static.crates.io/db-dump.tar.gz.


Examples

The examples/ directory of this repo contains several runnable example analyses.

total‑downloads Computes time series of total downloads by day across all crates on crates.io
crate‑downloads Computes time series of downloads of one specific crate
top‑crates Computes the top few most directly depended upon crates
user‑dependencies Computes the percentage of crates.io which depends directly on at least one crate by the specified user
user‑downloads Computes time series of the fraction of crates.io downloads attributed to a single given user's crates

Each of these examples can be run using Cargo once you've downloaded a recent database dump:

$ wget https://static.crates.io/db-dump.tar.gz
$ cargo run --release --example total-downloads

Here is the implementation of the most basic example, total-downloads, and graph of the resulting table. It shows crates.io download rate doubling every 9 months, or equivalently 10× every 2.5 years!

use chrono::Utc;
use db_dump::Date;
use std::collections::BTreeMap as Map;

fn main() -> db_dump::Result<()> {
    let mut downloads = Map::<Date<Utc>, u64>::new();
    db_dump::Loader::new()
        .version_downloads(|row| {
            *downloads.entry(row.date).or_default() += row.downloads;
        })
        .load("./db-dump.tar.gz")?;

    for (date, count) in downloads {
        println!("{},{}", date, count);
    }

    Ok(())
}
Crates.io downloads per day (log scale)

Here is a graph from the user-downloads example:

Fraction of crates.io downloads that are dtolnay's crates

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies

~5–15MB
~193K SLoC