#optimization #hyper-parameters #estimator #algorithm #categorical #tree-structured #parzen

tpe

A library that implements TPE, a hyperparameter optimization algorithm

3 unstable releases

0.2.0 Jan 10, 2022
0.1.1 Mar 14, 2021
0.1.0 Sep 27, 2020

#380 in Science

Download history 24/week @ 2024-03-14 23/week @ 2024-03-21 30/week @ 2024-03-28 27/week @ 2024-04-04 25/week @ 2024-04-11 20/week @ 2024-04-18 24/week @ 2024-04-25 19/week @ 2024-05-02 19/week @ 2024-05-09 20/week @ 2024-05-16 13/week @ 2024-05-23 16/week @ 2024-05-30 14/week @ 2024-06-06 105/week @ 2024-06-13 76/week @ 2024-06-20 12/week @ 2024-06-27

209 downloads per month
Used in serenade

MIT license

28KB
496 lines

tpe

tpe Documentation Actions Status Coverage Status License: MIT

This crate provides a hyperparameter optimization algorithm using TPE (Tree-structured Parzen Estimator).

Examples

Minimize the result of a quadratic function

An example optimizing a simple quadratic function which has one numerical and one categorical parameters.

use rand::SeedableRng as _;

let choices = [1, 10, 100];
let mut optim0 =
    tpe::TpeOptimizer::new(tpe::parzen_estimator(), tpe::range(-5.0, 5.0)?);
let mut optim1 =
    tpe::TpeOptimizer::new(tpe::histogram_estimator(), tpe::categorical_range(choices.len())?);

fn objective(x: f64, y: i32) -> f64 {
    x.powi(2) + y as f64
}

let mut best_value = std::f64::INFINITY;
let mut rng = rand::rngs::StdRng::from_seed(Default::default());
for _ in 0..100 {
   let x = optim0.ask(&mut rng)?;
   let y = optim1.ask(&mut rng)?;

   let v = objective(x, choices[y as usize]);
   optim0.tell(x, v)?;
   optim1.tell(y, v)?;
   best_value = best_value.min(v);
}

assert_eq!(best_value, 1.000098470725203);

kurobako benchmark

There is an example examples/tpe-solver.rs which implements the kurobako solver interface, so you can run a benchmark using TPE as follows:

$ PROBLEMS=$(kurobako problem-suite sigopt auc)
$ SOLVERS="$(kurobako solver command -- cargo run --release --example tpe-solver) $(kurobako solver optuna)"
$ kurobako studies --solvers $SOLVERS --problems $PROBLEMS --repeats 30 --budget 80 | kurobako run > result.json
$ cat result.json | kurobako report > report.md

The result (report.md) of the above commands is shown here.

References

Please refer to the following papers about the details of TPE:

Dependencies

~5.5MB
~116K SLoC