Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

Short summary

This study evaluates whether small language models follow instructions that conflict with their usual task behavior across MCQA, sentiment classification, and math QA tasks. Using an Instruction-Following Failure Rate (IFFR) metric, the authors find that small models stay competent but routinely ignore conflicting instructions, while larger models show a clearer gap between standard and non-standard instruction compliance. The key finding is that task competence and instruction following are distinct abilities, and standard accuracy metrics hide instruction-following failures.

• •Small models maintain task competence but routinely ignore conflicting non-standard instructions
• •Larger models show a clearer gap between standard accuracy and instruction-following compliance
• •Task competence and instruction following are distinct abilities; standard accuracy masks IFFR failures

Generated with AI, which can make mistakes.