Your Experts Have Partners. There's HOPE for Keeping Them Together
Summary
- Higher-Order Pruning of Experts (HOPE) is a second-order pruning method for Mixture-of-Experts (MoE) models that accounts for pairwise expert interactions.
- Unlike first-order methods that score experts independently, HOPE minimizes a tractable upper-bound of the pruning error by solving a quadratic program using an interaction matrix F.
- Evaluating across models like Qwen3.5 and GLM-4.5, HOPE achieves the best overall rank at aggressive pruning rates, showing significant gains on agentic coding benchmarks like SWE-Bench Pro.
- At a 50% pruning rate, HOPE matches unpruned model performance on SWE-Bench Pro while outperforming first-order baselines such as REAP.
Why it matters
HOPE reduces the severe memory footprint of large Mixture-of-Experts language models without retraining, preserving crucial cooperative expert groups and maintaining high performance even at aggressive 50% pruning rates.