Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations

Short summary

This paper explores whether contrastive decoding for hallucination mitigation extends to Mixture-of-Experts (MoE) architectures. The authors find that layer-wise differences don't exist in MoEs with shared experts, but higher layers show distinct expert activation patterns between factual and non-factual outputs. They propose EAACD, an expert-aware adaptive contrast decoding method that splits high-layer experts into reliability groups and contrasts their predictions, outperforming all baselines on four QA datasets.

• •Layer-wise contrastive decoding doesn't directly transfer to MoEs with shared experts
• •Higher MoE layers show distinct expert activation patterns for factual vs non-factual outputs
• •EAACD method outperforms baselines on four QA hallucination datasets

Generated with AI, which can make mistakes.