Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

Short summary

Know2Guess is a 1,200-item, contamination-aware benchmark for measuring when large language models reliably answer vs. when they should abstain, treating contamination, prompt sensitivity, and refusal as distinct dimensions. Evaluates FLAN-T5, Qwen2.5-Instruct, and Llama-3-Instruct across five domains. Qwen2.5-3B achieves best reliability but reveals poor calibration and selective refusal—answering reliability remains unsolved.

• •Introduces Know2Guess: a 1,200-item multi-zone benchmark with explicit abstention expectations and contamination-risk metadata for rigorous LLM evaluation
• •Evaluates three major instruction-tuned models (FLAN-T5, Qwen2.5, Llama-3) under locked answer-or-abstain prompts across five domains
• •Qwen2.5-3B shows best reliability but poor calibration persists; models struggle with selective refusal and answer-expected zones

Generated with AI, which can make mistakes.