Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions

Short summary

This position paper argues that evaluating personal LLM agents requires a new protocol: replaying temporal interventions across persistent user-conditioned states and measuring cross-component failure propagation. The authors formalize four conditions for such evaluation and audit existing benchmarks, finding none satisfy all four. They propose a minimal benchmark design and candidate reporting metrics for user-conditioned adaptation in personal agents.

• •Four conditions proposed for personal-agent evaluation: temporal intervention, persistent state, cross-dimensional effects, user-conditioned variation
• •Audit of public benchmarks found none satisfying all four conditions
• •Minimal benchmark design and reporting metrics proposed for future personal-agent evaluation

Generated with AI, which can make mistakes.