Pikmin

An extensible downloader for obtaining trade data simultaneously from exchanges' API.

pikmin is a trade (execution) data downloader for crypto-currency exchanges, such as BitMex, bitFlyer, Liquid, etc. This library provides not only some pre-composed downloaders, but also ability to build a custom downloader for users' demand.

Pre-composed downloaders

Currently, this library has the following downloaders:

• BitMex (specify time, chronologically)
• bitFlyer (specify id, reverse-chronologically)
• Liquid (specify time, chronologically)

Built-in Writer

Writer is a processor between trade data and destination (typically DB). Pikmin has some built-in writers:

• MySQL
• stdout

You can create your own writer easily.

Example

A simple downloader for Liquid with writing to stdout. This program creates ./qn-progress.txt for recording progress, so delete it if you want to run again from the starting point.

use std::path::PathBuf;
use std::thread;
use std::time::Duration;

use chrono::offset::TimeZone;
use chrono::Utc;

use pikmin::downloader::Downloader;
use pikmin::FileRecorder;
use pikmin::LiquidDownloader;
use pikmin::StdOutWriter;

fn main() {
    // by using thread, you can run multiple downloaders
    let liquid = thread::spawn(move || {
        while {
            // download data from 2019-01-01T01:01:01Z to 2019-01-01T01:03:01Z
            // output the downloaded data to standard out (println!)
            let downloader = LiquidDownloader::new(
                Utc.ymd(2019, 1, 1).and_hms(1, 1, 1),
                Utc.ymd(2019, 1, 1).and_hms(1, 3, 1),
            );

            // Locate progress file to `/tmp/qn-progress.txt`.
            // You can control the starting point of downloading
            // by preparing this file before you run.
            let mut recorder = FileRecorder::new(PathBuf::from("/tmp/qn-progress.txt"));

            // write out to standard out. simple writer for debugging
            let mut writer = StdOutWriter::default();

            println!("start QnDownloader");

            // run!
            match downloader.run(&mut writer, &mut recorder) {
                Ok(_) => {
                    println!("finished");
                    false
                }
                Err(e) => {
                    println!("error: {}", e);
                    println!("retry...");
                    thread::sleep(Duration::from_secs(5));
                    true
                }
            }
        } {}
    });

    match liquid
        .join()
        {
            Ok(_) => println!("finish all"),
            Err(_) => println!("threading error"),
        }
}

Other examples can be found in ./examples.

Future work

• create pre-composed downloaders for other exchanges
• parameterize the direction of downloading (chronologically or not)
• abstraction of the progress writer