Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning
Summary
- Researchers introduce Low-Rank Hierarchical Merging (LHM), a training-free framework for Long-to-Short model merging.
- LHM replaces linear interpolation with a low-rank Hessian approximation, a Layer Second-Order Sensitivity index, and Dual-Constraint Coefficient Mapping.
- Evaluated on six mathematical benchmarks using 1.5B and 7B models, LHM yields over 4% accuracy gains and up to 93.1% output length reduction.
Why it matters
LHM improves the efficiency and accuracy of model merging by combining long chain-of-thought reasoning with concise base model styles without requiring task-specific training data.