2 releases
0.0.2 | Oct 4, 2024 |
---|---|
0.0.1 | Oct 4, 2024 |
#75 in #cuda
366 downloads per month
Used in 2 crates
59KB
1.5K
SLoC
llm_devices: Managing Devices and Builds for LLMs
This crate is part of the llm_client crate.
The llm_interface crate uses it as a dependency for building llama.cpp.
It's functionality includes:
-
Cloning the specified tag, and building llama.cpp.
-
Checking for device availabilty (CUDA, MacOS) to determine what platform to build for.
-
Fetching available VRAM or system RAM for estimating the correct model to load.
-
Offloading model layers to memory.
-
Logging tools.
See the build documentation for more notes.
Dependencies
~6–32MB
~484K SLoC