Back to feed
Dev.to
Dev.to
7/9/2026
The headline is about a bug where a scorer was dropping half its shot scores. Let me rewrite this concisely.

The headline is about a bug where a scorer was dropping half its shot scores. Let me rewrite this concisely.

Original: The Scorer Was Silently Dropping Half Its Shot Scores

Short summary

A vision-QA scorer in the Poindexter project was silently dropping ~42% of shot scores because qwen3-vl's think trace consumed a too-small 300-token budget before producing JSON output. The fix raised the token limit to 1024 and also addressed a hardcoded timeout override and missing module-registration checks that allowed silent production degradation. Two AST-based test guards were added to prevent future modules from silently failing at boot time.

  • Vision scorer dropped ~42% of scores due to insufficient token budget for qwen3-vl think trace
  • Fixed hardcoded timeout=300 that overrode a correctly-sized 900s client timeout
  • Added AST-based completeness tests to catch unregistered module imports and Dockerfile copy gaps

Generated with AI, which can make mistakes.

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more