Policy Optimization With Multiple Verifiers via Pareto Rankings
Summary
- Researchers introduce Partial-Order Likelihood Policy Optimization (POLPO) to train large language models using multiple verifiers without scalar reward aggregation.
- POLPO utilizes Pareto dominance to identify responses that satisfy additional requirements without losing previously satisfied ones.
- The method maximizes the total probability of complete rankings consistent with partial orders, extending the listwise likelihood used in Direct Preference Optimization.
- Experiments on code generation and instruction following show that POLPO improves learning efficiency and increases complete requirement satisfaction rates compared to scalar-reward training.
Why it matters
By avoiding scalar reward aggregation from multiple verifiers, POLPO provides finer-grained supervision from partial successes, improving reinforcement learning efficiency for tasks like code generation.