tower-memlim

Enforces a limit on the underlying service when a machine's memory Threshold is met.

Load Shedding

By combining MemoryLimitLayer with tower's load_shed feature, incoming requests can be rejected once a certain memory Threshold is met. Ths can help to protect a system from running out of memory.

It also helps to maximize the usage of available resources while maintaining system stability. Compared to setting a limit that does not account for system resource variables, such as requests per second, relative resource bound limits like MinAvailableBytes do not require constant adjustment whenever system resources change. Hence memory based load shedding is a perfect match for memory based auto scaling strategies.

Exemplary scaling pattern:

• Auto-scaler provisions systems based on memory/cpu thresholds
• Load shedder rejects request upon threshold exceedance to prevent out of memory issues and to signal hard resource exhaustion
• Load balancer/upfront webserver detects exhausted system via rejected requests or failing health probes and redirects (or retries) traffic to healthy systems

Example

use tower::ServiceBuilder;
use tower_memlim::layer::MemoryLimitLayer;
use tower_memlim::error::BoxError;
use tower_memlim::memory::{Threshold, LinuxCgroupMemory};
use tower::service_fn;

// The friendliest service in town!
// Spreading joy, until the memory limit layer threshold is not exceeded.
async fn svc_handle(_req: &str) -> Result<&str, BoxError> {
    Ok("Nice to see you! (while memory lasts)")
}

let mut svc = ServiceBuilder::new()
    // Map the error to your needs
    .map_result(|result: Result<_, BoxError>| match result {
        Ok(resp) => Ok(resp),
        Err(err) => {
            if err.is::<tower::load_shed::error::Overloaded>() {
                // A web server may want to return a http status code instead
                Ok("Too many requests")
            } else {
                Err(err)
            }
        },
    })
    // Load shed and throw `Overloaded` error
    // when the next layer responds with `Poll::Ready(Err(_))`.
    // Without load shedder requests would be enqueued.
    .load_shed()
    // Upon memory exceeding, this layer responds with `Poll::Ready(Err(_))` 
    // to signal that the service is no longer able to service requests.
    // That allows other layers such as `load_shed` to react on it.
    .layer(MemoryLimitLayer::new(
        Threshold::MinAvailableBytes(11),
        LinuxCgroupMemory
    ))
    .service(service_fn(svc_handle));

Operating Systems

This crate provides support for a Linux memory stats provider, but any other struct implementing AvailableMemory can be used. When developing on an unsupported platform, consider disabling the layer using tower::util::option_layer.

AvailableMemory implementors:

• LinuxCgroupMemory

License

This project is licensed under the MIT license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in tower-memlim by you, shall be licensed as MIT, without any additional terms or conditions.