Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

Short summary

SGPO improves LLM reasoning by teaching models strategic problem-solving rather than memorizing specific solution paths. The method extracts strategy patterns from strong models and uses adaptive weighting to guide weaker models' learning. Experiments show 2.2-point improvements on mathematical benchmarks with better generalization.

• •Replaces trajectory imitation with strategy distillation to improve generalization to novel problems
• •Uses forward-KL objective with adaptive instance-level weighting for selective guidance
• •Achieves 2.2-point improvement over baselines on Qwen2.5-7B-Instruct across four math benchmarks

Generated with AI, which can make mistakes.