Policy Optimization With Multiple Verifiers via Pareto Rankings

Impact 7 arXiv cs.AI

Summary

  • Researchers introduce Partial-Order Likelihood Policy Optimization (POLPO) to train large language models using multiple verifiers without scalar reward aggregation.
  • POLPO utilizes Pareto dominance to identify responses that satisfy additional requirements without losing previously satisfied ones.
  • The method maximizes the total probability of complete rankings consistent with partial orders, extending the listwise likelihood used in Direct Preference Optimization.
  • Experiments on code generation and instruction following show that POLPO improves learning efficiency and increases complete requirement satisfaction rates compared to scalar-reward training.

Why it matters

By avoiding scalar reward aggregation from multiple verifiers, POLPO provides finer-grained supervision from partial successes, improving reinforcement learning efficiency for tasks like code generation.

Who should care