Arxiv Preprint - A General Theoretical Paradigm to Understand Learning from Human Preferences

In this episode we discuss A General Theoretical Paradigm to Understand Learning from Human Preferences by Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos. The paper explores reinforcement learning from human preferences (RLHF) and proposes ΨPO, a new theoretical framework that directly utilizes pairwise preferences without relying on traditional approximations like pointwise rewards or reward model generalization. The authors thoroughly examine the potential shortcomings of existing methods like RLHF and DPO, which are incorporated under the umbrella of ΨPO. They also introduce an efficient optimization procedure for a special case of ΨPO, providing performance guarantees and showing its empirical advantages over DPO in various examples.