Shipping a Local LLM API with FastAPI and Ollama

Short summary

Engineer built a production text-to-SQL API using a fine-tuned 3B Qwen model via Ollama and FastAPI, achieving 90% accuracy while running on a $5 VPS with zero infrastructure costs. The post includes complete production code, deployment patterns, and a 5-layer SQL safety validation system. Demonstrates shipping sophisticated AI features without cloud API bills or expensive hardware.

• •3B fine-tuned Qwen model generates SQL from natural language with 90% accuracy on in-domain queries
• •Runs on $5 Hetzner VPS with CPU-only inference via Ollama—zero cloud API costs
• •Production-ready with 5-layer SQL safety validation and read-only execution sandbox

Generated with AI, which can make mistakes.