Edge Infer

EdgeInfer enables efficient edge intelligence by running small AI models, including embeddings and OnnxModels, on resource-constrained devices like Android, iOS, or MCUs for real-time decision-making.

Architecture:

Platform support (by Design):

• Android, iOS
• Linux, Windows, Mac,
• Raspberry Pi, MCU

Todos

• Inference wrapper
  • Onnx Runtime
  • Tokenizer
• UniFFI, is a toolkit for building cross-platform software components in Rust.
• GRPC server with tonic
• Multiple OS support:
  • Desktop: Windows, Mac, Linux (x86, x64)
  • Mobile: Android, iOS, Linux (ARM)
  • Embedded Linux (ARM).
• Flexible Configuration: Easily configurable via command-line parameters, including listening port, batch size, thread count, and others.

Usecases

• SearchEverywhere: Search for anything, anywhere, anytime.
  • Model: Embedding, like Sentence-Transformers MiniLM
  • Extra: Local Indexing
• Visualization
  • Model: Ultralytics YOLOv9
• AutoComplete
  • Model: Embedding with ??
• Summarization

Resources

Examples:

• Modern cross-platform telemetry: Glean

MCU

ToSpike

• ESP32: esp-rs
• Raspberry Pi Classic
  • built-onnxruntime-for-raspberrypi-linux
  • ONNX Runtime IoT Deployment on Raspberry Pi

Not working:

• Arduino M0 Pro, Flash: 256 KB, SRAM: 32Kb
  • Official: Arduino M0 Pro
  • Rust's cortex-m-quickstart
• Raspberry Pi Zero W, Flash: 512 MB, SRAM: 512 MB
  • Official: Raspberry Pi Zero W
  • Using Rust to Control a Raspberry Pi Zero W Rover
  • Not working reason: See in inference_rpi

License

This project is licensed under the MIT License, See LICENSE for the full license text.