Benchmarking Gemini 2.5 Flash vs 3.1 Flash-Lite vs Gemma 4 with LLM judge (Claude Fable 5)

Short summary

A production reading app benchmarked Gemini 2.5 Flash, Gemini 3.1 Flash-Lite, and Gemma 4 26B across 12 real prompts using Melville's "Whiteness of the Whale" as an adversarial stimulus, judged by a human plus Claude. Flash-Lite was 40% cheaper and produced the sharpest observations, while Gemma 4's hidden reasoning tokens made it 4× more expensive than list price suggested. The key takeaway: always check usageMetadata.thoughtsTokenCount before comparing models on list price for metered API serving.

• •Gemini 3.1 Flash-Lite outperformed the incumbent at 40% lower output cost
• •Gemma 4's hidden reasoning tokens made it 4× more expensive than list price implied
• •Prompt contracts are model-agnostic — engines are swappable without changing product shape

Generated with AI, which can make mistakes.