Tokens Are Not the Unit

Short summary

Token pricing per million is misleading because you're billed for hidden reasoning tokens you never see or use. A model advertised as 30% cheaper was actually 10x more expensive once reasoning tokens were counted. The article also shows that models with near-identical accuracy scores can differ dramatically on trap tasks — one model picked the wrong tool 52% of the time when baited despite scoring 91% on accuracy.

• •Reasoning tokens are billed but invisible — a '30% cheaper' model was actually 10x more expensive in practice
• •Always compute dollars per useful output token, not advertised per-token rates
• •Models with similar accuracy can fail catastrophically on trap tasks: one 91%-accuracy model grabbed the wrong tool 52% of the time when baited

Generated with AI, which can make mistakes.