Back to feed
arXiv cs.LG
arXiv cs.LG
7/24/2026
Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement

Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement

Short summary

This paper frames multi-LLM aggregation as an uncertainty-aware trust estimation problem, adapting Cooke-style structured expert judgement from decision theory to weight LLMs by calibration quality. Context-aware calibration questions estimate each model's reliability, penalizing overconfident incorrect predictions. On MMLU and MMLU-Pro, Cooke weighting becomes critical under heterogeneity and contamination, achieving superior accuracy-reliability balance and robustness against unreliable or adversarial experts.

  • •Multi-LLM aggregation reformulated as uncertainty-aware trust estimation using Cooke-style log weighting
  • •Calibration questions penalize overconfident incorrect predictions and favor well-calibrated experts
  • •Cooke weighting proves critical under heterogeneous and contaminated expert panels on MMLU and MMLU-Pro

Generated with AI, which can make mistakes.

#research-breakthrough#ai-agents
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence
arXiv cs.CL
When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
arXiv cs.CL
Information Discernment in Large Language Models
arXiv CS.AI
Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs
arXiv CS.AI