Sipp: a local-first runtime for Hybrid AI Applications

Short summary

Sipp is a runtime that unifies local, gateway, and cloud AI inference endpoints behind a single interface, letting applications choose where models run without provider lock-in. It optimizes latency and throughput with intelligent KV cache reuse and scheduler design. Aims to make AI more accessible by bringing inference closer to users.

• •Unifies local, gateway, and cloud inference endpoints through a single SippClient interface
• •Smart KV cache management and request scheduling for optimal latency vs. throughput tradeoffs
• •Designed to make AI accessible locally without requiring cloud dependency or frontier model pricing

Generated with AI, which can make mistakes.