RLlama

RLlama is a Rust implementation of the quantized Llama 7B language model.

Llama 7B is a very small but performant language model that can be easily run on your local machine.

This library uses Candle to run Llama.

Usage

use kalosm_llama::prelude::*;

#[tokio::main]
async fn main() {
    let mut model = Llama::new().await.unwrap();
    let prompt = "The capital of France is ";
    let mut stream = model(prompt);

    print!("{prompt}");
    while let Some(token) = stream.next().await {
        print!("{token}");
    }
}