Back to feed
arXiv cs.CL
arXiv cs.CL
6/23/2026
Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

Short summary

A new benchmark, PuMVR, exposes script-dependent bias in 10 state-of-the-art Vision-Language Models through 375 image-reasoning tasks in Punjabi's three active scripts. Accuracy gaps reach 16% between scripts, with Script Consistency Rates as low as 24.8%—bias persists even with visual input. The findings propose Script Consistency Rate (SCR) as a fairness metric and challenge current multilingual AI evaluation paradigms.

  • •PuMVR benchmark evaluates 10 VLMs across Punjabi's three scripts (Gurmukhi, Shahmukhi, Roman)
  • •Script-dependent bias persists with accuracy gaps up to 16% and Script Consistency Rates as low as 24.8%
  • •Proposes Script Consistency Rate (SCR) as a core metric for multilingual AI fairness evaluation

Generated with AI, which can make mistakes.

#research-breakthrough#ai-tools
Read full article at arXiv cs.CL

Is this a good recommendation for you?

Explore more

Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models
arXiv cs.CL
Are you speaking my languages? On spoken language adherence in multimodal LLMs
arXiv cs.CL
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
arXiv cs.CL
Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards
arXiv cs.CL