A General Theoretical Paradigm to Understand Learning from Human Preferences

Introduction

Learning from human preferences (Christiano et al., 2017) is a paradigm adopted in the natural language processing literature to better align pretrained (Radford et al., 2018; Ramachandran et al., 2016) and instruction-tuned (Wei et al., 2022) generative language models to human desiderata. It consists in first collecting large amounts of data where each datum is composed of a context, pairs of continuations of the context, also called generations, and a pairwise human preference that indicates which generation is the best. Then, a policy generating good generations given a context is learnt from the collected data. We frame the problem of learning from human preferences as an offline contextual bandit problem (Lu et al., 2010). The goal of this bandit problem is that given a context to choose an action (playing the role of the generation) which is most preferred by a human rater under the constraint that the resulting bandit policy should be close to some known reference policy. The constraint of staying close to a known reference policy can be satisfied e.g., by using KL regularisation (Geist et al., 2019) and its role is to avoid model drift (Lazaridou et al., 2020; Lu et al., 2020).

A prominent approach to tackle the problem of learning from human preferences is through reinforcement learning from human feedback (RLHF, Ouyang et al., 2022; Stiennon et al., 2020) in which first a reward model is trained in the form of a classifier of preferred and dispreferred actions. Then the bandit policy is trained through RL to maximize this learned reward model while minimizing the distance with the reference policy. Recently RLHF has been used successfully in solving the problem of aligning generative language models with human preferences (Ouyang et al., 2022). Furthermore recent works such as direct preference optimisation (DPO, Rafailov et al., 2023) and (SLiC-HF, Zhao et al., 2023) have shown that it is possible to optimize the bandit policy directly from human preferences without learning a reward model. They also have shown that on a selection of standard language tasks they are competitive with the state of the art RLHF while they are simpler to implement and require less resources.

Despite this practical success, little is known regarding theoretical foundations of these practical methods. Notable exceptions, that consider specific special cases, are (Wang et al., 2023; Chen et al., 2022) and prior work on preference-based (Busa-Fekete et al., 2014, 2013) and dueling bandits and RL (Novoseller et al., 2020; Pacchiano et al., 2023). However, these theoretical works focus on providing theoretical guarantees in terms of regret bounds in the standard bandit setting and they do not deal with the practical setting of RLHF, DPO and SLiC-HF.

In this work, our focus is on bridging the gap between theory and practice by introducing a simple and general theoretical representation of the practical algorithms for learning from human preferences. In particular, we show that it is possible to characterise the objective functions of RLHF and DPO as special cases of a more general objective exclusively expressed in terms of pairwise preferences. We call this objective $\Psi$-preference optimisation ($\Psi$PO) objective, where $\Psi$ is an arbitrary non-deceasing mapping. We then analyze this objective function in the special cases of RLHF and DPO and investigate its potential pitfalls. Our theoretical investigation of RLHF and DPO reveals that in principle they can be both vulnerable to overfitting. This is due to the fact that those methods rely on the strong assumption that pairwise preferences can be substituted with ELo-score (pointwise rewards) via a Bradley-Terry (BT) modelisation (Bradley and Terry, 1952). In particular, this assumption could be problematic when the (sampled) preferences are deterministic or nearly deterministic as it leads to over-fitting to the preference dataset at the expense of ignoring the KL-regularisation term (see Sec. 4.2). We then present a simple solution to avoid the problem of overfitting, namely by setting $\Psi$ to identity in the $\Psi$PO. This method is called Identity-PO (IPO) and by construction bypasses the BT modelisation assumption for preferences (see Sec. 5). Finally, we propose a practical solution, via a sampled loss function (see Sec. 5.2), to optimize this simplified version of $\Psi$PO empirically and, we compare its performance with DPO on simple bandit examples, providing empirical support for our theoretical findings (see Sec. 5.3 and Sec. 5.4).

Notations

For any two policy $\pi,\mu\in\Delta_{\mathcal{Y}}^{\mathcal{X}}$ and a context distribution $\rho$ we denote the total preference of policy $\pi$ to $\mu$ as

Background

The standard RLHF paradigm (Christiano et al., 2017; Stiennon et al., 2020) consists of two main stages: (i) learning the reward model; (ii) policy optimisation using the learned reward. Here we provide a recap of these stages.

Learning a reward model consists in training a binary classifier to discriminate between the preferred and dis-preferred actions using a logistic regression loss. For the classifier, a popular choice is Bradley-Terry model: for a given context $x$ and action $y$, we denote the pointwise reward, which can also be interpreted as an Elo score, of $y$ given $x$ by $r(x,y)$. The Bradley-Terry model represents the preference function $p(y\succ y^{\prime}|x)$ (classifier) as a sigmoid of the difference of rewards:

where $\sigma(\cdot)$ denotes the sigmoid function and plays the role of normalisation. Given the dataset $\mathcal{D}=(x_{i},y_{w,i}\succ y_{l,i})_{i=1}^{N}$ one can learn the reward function by optimizing the following logistic regression loss

Assuming that $p^{*}(y\succ y^{\prime}|x)$ conforms to the Bradley-Terry model, one can show that as the size of the dataset $\mathcal{D}$ grows, $p(y\succ y^{\prime}|x)$ becomes a more and more accurate estimate of true $p^{*}(y\succ y^{\prime}|x)$ and in the limit converges to $p^{*}(y\succ y^{\prime}|x)$.

1.2 Policy Optimisation with the Learned Reward

Using the reward (Elo-score) $r(x,y)$ the RLHF objective is simply to optimize for the policy $\pi\in\Delta^{\mathcal{X}}_{\mathcal{Y}}$ that maximizes the expected reward while minimizing the distance between $\pi$ and some reference policy $\pi_{\text{ref}}\in\Delta^{\mathcal{X}}_{\mathcal{Y}}$ through the following KL-regularized objective function:

in which the context $x$ is drawn from $\rho$ and the action $y$ is drawn from $\pi(.|x)$. The divergence $D_{\text{KL}}(\pi||\pi_{\text{ref}})$ is defined as follows:

The objective in Equation (3) is essentially optimized by PPO (Schulman et al., 2017) or similar approaches.

The combination of RLHF +PPO has been used with great success in practice (e.g., InsturctGPT and GPT-4 Ouyang et al., 2022; OpenAI, 2023).

2 Direct Preference Optimisation

An alternative approach to the RL paradigm described above is direct preference optimisation (DPO; Rafailov et al., 2023), which avoids the training of a reward model altogether. The loss that DPO optimises, given an empirical dataset $\mathcal{D}$, as a function of $\pi$, is given by

In its population form, the loss takes on the form

Rafailov et al. (2023) show that when (i) the Bradley-Terry model in Equation (1) perfectly fits the preference data and (ii) the optimal reward function $r$ is obtained from the loss in Equation (2), then the global optimisers of the RLHF objective in Equation (3) and the DPO objective in Equation (3.2) perfectly coincide. In fact, this correspondence is true more generally; see Proposition 4 in Appendix B.

A General Objective for Preference Optimisation

This objective balances the maximisation of a potentially non-linear function of preference probabilities with the KL regularisation term which encourages policies to be close to the reference $\pi_{\text{ref}}$. This is motivated by the form of Equation (3), and we will see in the next subsection that it strictly generalises both RLHF and DPO, when the BT model holds.

In the remaining, we omit the dependency on $x$ for the ease of notations. This is without losing generality and all the following results are true for all $x\in\texttt{Supp}(\rho)$.

We first connect DPO and RLHF with the $\Psi$-preference objective in Equation (6), under the special choice of $\Psi(q)=\log(q/(1-q))$. More precisely, the following proposition establishes this connection.

then the optimal policy for Equation (6), for the RLHF objective in Equation (3), and for the standard DPO objective in Equation (3.2) are identical.

Note that under the assumption that the Bradley-Terry model holds, we have

This is equal to the reward in Equation (3), up to an additive constant, and so it therefore follows that the optimal policy for Equation (6) and for optimizing the objective in Equation (3) are identical. Further, as shown by Rafailov et al. (2023), the optimal policy for the DPO objective in Equation (3.2) and the objective in Equation (3) are identical, which gives the statement of the proposition. ∎

Applying this proposition to the objective function of Equation (6), for which there exists an analytical solution, reveals that under the BT assumption the closed-form solution to DPO and RLHF can be written as

The derivations leading to Equation 7 is a well known result and is provided in App. A.1 for completeness.

2 Weak Regularisation and Overfitting

It is worth taking a step back, and asking what kinds of policies the above objective leads us to discover. This highly non-linear transformation of the preference probabilities means that small increases in preference probabilities already close to 1 are just as incentivized as larger increases in preference probabilities around $50\%$, which may be undesirable. The maximisation of logit-preferences, or Elo score in game-theoretic terminology, can also have counter-intuitive effects, even in transitive settings (Bertrand et al., 2023).

Consider the simple example where we have two actions $y$ and $y^{\prime}$ such that $p^{*}(y\succ y^{\prime})=1$, i.e., $y$ is always preferred to $y^{\prime}$. Then the Bradley-Terry model would require that $(r(y)-r(y^{\prime}))\rightarrow+\infty$ to satisfy (1). If we plug this into the optimal policy (7) then we would get that $\frac{\pi^{*}(y^{\prime})}{\pi^{*}(y)}=0$ (i.e., $\pi^{*}(y^{\prime})=0$) irrespective of what constant $\tau$ is used for the KL-regularisation. Thus the strength of the KL-regularisation becomes weaker and weaker the more deterministic the preferences.

The weakness of the KL-regularisation becomes even more pronounced in the finite data regime, where we only have access to a sample estimate of the preference $\hat{p}(y\succ y^{\prime})$. Even if the true preference is, e.g., $p^{*}(y\succ y^{\prime})=0.8$, empirically it can be very possible when we only have a few data points to estimate $\hat{p}(y\succ y^{\prime})=1$, in which case the empirical optimal policy would make $\pi(y^{\prime})=0$ for any $\tau$. This means that overfitting can be a substantial empirical issue, especially when the context and action spaces are extremely large as it is for large language models.

Why may standard RLHF be more robust to this problem in practice? While a purported advantage of DPO is that it avoids the need to fit a reward function, we observe that in practice when empirical preference probabilities are in the set $\{0,1\}$, the reward function ends up being underfit. The optimal rewards in the presence of $\{0,1\}$ preference probabilities are infinite, but these values are avoided, and indeed regularisation of the reward function has been observed to be an important aspect of RLHF training in practice (Christiano et al., 2017). This underfitting of the reward function is thus crucial in obtaining a final policy that is sufficiently regularised towards the reference policy $\pi_{\text{ref}}$, and DPO, in avoiding the training of the reward function, loses the regularisation of the policy that the underfitted reward function affords.

While standard empirical practices such as early-stopping can still be used as an additional form of regularisation to curtail this kind of overfitting, in the next section, we will introduce a modification of the $\Psi$PO objective such that the optimal empirical policy can be close to $\pi_{\text{ref}}$ even when preferences are deterministic.

IPO: ΨΨ\PsiPO with identity mapping

We have observed in the previous section that DPO is prone to overfitting, and this stems from a combination of the unboundedness of $\Psi$, together with not training an explicit reward function. Not training a reward function directly is a clear advantage of DPO, but we would like to avoid the problems of overfitting as well.

This analysis of DPO motivates choices of $\Psi$ which are bounded, ensuring that the KL regularisation in Equation 6 remains effective even in the regime of $\{0,1\}$-valued preferences, as it is often the case when working with empirical datasets. A particularly natural form of objective to consider is given by taking $\Psi$ to be the identity mapping in Equation (6), leading to direct regularized optimisation of total preferences:

The standard approach to optimize the objective function of Equation (8) is through RLHF with the choice of reward $r(y)=p^{*}(y\succ\mu)$. However both using RL and estimating the reward model $r(y)$ can be costly. Inspired by DPO one would like to devise an empirical solution for the optimisation problem of Equation (8) which can directly learn from the preference dataset. Thus it would be able to avoid RL and reward modeling altogether.

As with DPO, it will be beneficial to re-express Equation (8) as an offline learning objective. To derive such an expression, we begin by following the derivation of Rafailov et al. (2023), manipulating the analytic expression for the optimal policy into a system of root-finding problems. As in the previous section, we drop dependence on the context $x$ from our notation, as all arguments can be applied on a per-context basis.

For any $y,y^{\prime}\in\texttt{Supp}(\pi_{\text{ref}})$, we therefore have

The core idea now is to consider a policy $\pi$, define

Loss for IPO. We now depart from the approach to the analysis employed by Rafailov et al. (2023), to obtain a novel offline formulation of Equation (6), in the specific case of $\Psi$ as the identity function. In this case, Equation (12) reduces to

We begin by re-expressing these root-finding problems as a single optimisation problem $L(\pi)$:

One can easily show that for the choice of $\pi^{*}$ we have $L(\pi^{*})=0$. Thus $\pi^{*}$ is a global minimizer of $L(\pi)$. The following theorem establishes the uniqueness of this solution.

Assume that $\texttt{Supp}(\mu)=\texttt{Supp}(\pi_{\text{ref}})$ and define $\Pi$ to be the set of policies $\pi$ such that $\texttt{Supp}(\pi)=\texttt{Supp}(\mu)$. Then $\pi\mapsto L(\pi)$ has a unique local/global minimum in $\Pi$, which is $\pi^{*}$.

By assumption, $\pi^{*}\in\Pi$, and by definition $\forall\pi\in\Pi,L(\pi)\geq 0$ as $L(\pi)$ is an expectation of squared terms. Further, from Equation (11), it follows immediately that $L(\pi^{*})=0$, and so we deduce that $\pi^{*}$ is a global optimum for $L$. We now show that there are no other local/global minima for $L$ in $\Pi$.

The objective is quadratic as a function of the logits $s$. Further, by expanding the quadratic above, we see that the loss can be expressed as a sum of squares

plus linear and constant terms. This is therefore a positive-semidefinite quadratic, and hence is convex. We thus deduce that all local minimisers of the loss $\mathcal{L}(s)$ are global minimisers as well (Boyd and Vandenberghe, 2004, Chap. 4). We now notice since $\pi_{s}$ is a surjective continuous mapping from $s$ to $\pi$ one can easily show from the definition of local minimum that every local minimiser $\pi$ of $L$ corresponds to a set of local minimisers $\mathcal{S}_{\pi}$ of $\mathcal{L}$. Thus all local minimums of $L$ are also global minimums as well.

The strict convexity combined with the fact that $\pi^{*}$ is a global minima proves that $\pi^{*}$ is the unique global/local minima in $\Pi$ (Boyd and Vandenberghe, 2004, Chap. 4). ∎

2 Sampled Loss for IPO

In order to obtain the sampled loss for IPO we need to show that we can build an unbiased estimate of the right-hand side of the equation (13). To this end, we consider the Population IPO Loss:

where $I(y,y^{\prime})$ is drawn from a Bernoulli distribution with mean $p^{*}(y\succ y^{\prime})$, i.e., $I(y,y^{\prime})$ is $1$ if $y$ is preferred to $y^{\prime}$ (which happens with probability $p^{*}(y\succ y^{\prime})$), and otherwise. This straightforwardly yields a sample-based loss that can be used, by sampling a pair $(y,y^{\prime})$ from the preference dataset, and consulting the recorded preference to obtain a sample from $I(y,y^{\prime})$. The following proposition justifies the switch from Equation (13) to Equation (16), by demonstrating their equality.

The expressions in Equation (13) and Equation (16) are equal, up to an additive constant independent of $\pi$.

This equivalence is not completely trivial, since in general the conditional expectation

is not equal to the corresponding quantity appearing in Equation (13), namely

We instead need to exploit some symmetry between the distributions of $y$ and $y^{\prime}$, and use the fact that $h_{\pi}(y,y^{\prime})$ decomposes as an additive function of $y$ and $y^{\prime}$. To show this equality of losses, it is enough to focus on the “cross-terms” obtained when expanding the quadratics in Equations (13) and (16); that is, to show

Now, starting with the right-hand side, and using the shorthand $\pi_{y}=\log(\pi(y))$, $\pi^{\text{R}}_{y}=\log(\pi_{\text{ref}}(y))$, $p_{y}=p^{*}(y\succ\mu)$, and similarly for $y^{\prime}$, we have

We now discuss how to approximate the loss in Equation (16) with an empirical dataset. As in our earlier discussion, the empirical dataset $\mathcal{D}$ takes the form $(y_{w,i},y_{l,i})_{i=i}^{N}$. Note that each datapoint $(y_{w_{,}i},y_{l,i})$ contributes two terms to an empirical approximation of Equation (16), with $(y,y^{\prime},I(y,y^{\prime}))=(y_{w,i},y_{l,i},1)$, and also $(y,y^{\prime},I(y,y^{\prime}))=(y_{l,i},y_{w,i},0)$. This symmetry is important to exploit, and leads to a reduction in the variance of the loss. The overall empirical loss is therefore given by

This simplified form of the loss provides some valuable insights on the way in which IPO optimizes the policy $\pi$: IPO learns from preferences dataset simply by regressing the gap between log-likelihood ratios $\log(\pi(y_{w})/\pi(y_{l}))$ and $\log(\pi_{\text{ref}}(y_{w})/\pi_{\text{ref}}(y_{l}))$ to $\frac{\tau^{-1}}{2}$. So the weaker the regularisation becomes, the higher would be the log-likelihood ratio of $y_{w}$ to $y_{l}$. In other words IPO, unlike DPO, always regularizes its solution towards $\pi_{\text{ref}}$ by controlling the gap between the log-likelihood ratios $\log(\pi(y_{w})/\pi(y_{l}))$ and $\log(\pi_{\text{ref}}(y_{w})/\pi_{\text{ref}}(y_{l}))$, thus avoiding the over-fitting to the preference dataset. We summarize the sampled IPO in Algorithm 1:

3 Illustrative Examples

To illustrate the qualitative difference between our algorithm and DPO we will consider a few simple cases. For simplicity we assume there is no context $x$, i.e., we are in the bandit setting.

We first consider the simple case where we have 2 actions only, $y_{1}$ and $y_{2}$, and a deterministic preference between them: $p^{*}(y_{1}\succ y_{2})=1$. Suppose we start with a uniform $\pi_{\text{ref}}$ and $\mu$. We know from Section 4.2 that DPO will converge to the deterministic policy $\pi^{*}(y_{1})=1$, $\pi^{*}(y_{2})=0$ regardless of the value of $\tau$. Thus even when the regularisation coefficient $\tau$ is very large, this is very different from the uniform $\pi_{\text{ref}}$.

Now, let us derive the optimal policy for IPO. We have $p^{*}(y_{1}\succ\mu)=3/4$ and $p^{*}(y_{2}\succ\mu)=1/4$. Plugging this into equation (9) with $\Psi=I$ we get that $\pi^{*}(y_{1})=\frac{\exp(0.75\tau^{-1})}{\exp(0.75\tau^{-1})+\exp(0.25\tau^{-1})}=\sigma(0.5\tau^{-1})$, and $\pi^{*}(y_{2})=\sigma(-0.5\tau^{-1})$, where $\sigma$ is the sigmoid function. Hence we see that if we have large regularisation as $\tau\rightarrow+\infty$, then $\pi^{*}$ converges to the uniform policy $\pi_{\text{ref}}$, and on the flip side as $\tau\rightarrow+0$, then $\pi^{*}(y_{1})\rightarrow 1$ and $\pi^{*}(y_{2})\rightarrow 0$, which is the deterministic optimal policy. The regularisation parameter $\tau$ can now actually be used to control how close to $\pi_{\text{ref}}$ we are.

4 Sampled Preferences

So far we relied on the closed-form optimal policy from Eq. (9) to study DPO and IPO’s stability, but this equation is not applicable to more complex settings where we only have access to sampled preference instead of $p^{\star}$. We can still however find accurate approximations of the optimal policy by choosing a parametrisation $\pi_{\theta}$ and optimize $\theta$ with an empirical loss over a dataset and iterative gradient-based updates. We will use this approach to show two non-asymptotic examples where DPO over-fits the dataset of preferences and ignore $\pi_{\text{ref}}$: when one action $y$ wins against all others DPO pushes $\pi_{\theta}(y)$ to 1 regardless of $\tau$, and conversely when one action $y$ never wins against the others DPO pushes $\pi_{\theta}(y)$ to 0 again regardless of $\tau$. In the same scenarios, IPO does not converge to these degenerate solutions but instead remains close to $\pi_{\text{ref}}$ based on the strength of the regularisation $\tau$.

For the first example we sample each unique action pair once to collect a dataset $\mathcal{D}$ containing 3 observed preferences. Due to symmetries of pairwise preferences sampling only 3 preferences can results in only two outcomes (up to permutations of the actions):

where we focus on $\mathcal{D}_{1}$, which represent a total ordering, rather than $\mathcal{D}_{2}$, which represent a cycle. The outcome of the experiment is reported in Fig. 1 in which, we report the learning curves for varying values of $\tau$. We observe that DPO always converges to the deterministic policy for all values of $\tau$. In other word DPO completely ignores the reference policy, no matter how strong is the regularisation term, and converges to the action which is preferred in the dataset. On the other hand, IPO prevent the policy from becoming greedy when the regularisation is strong.

In the first example DPO converges to a deterministic policy because one action strictly dominates all others and the loss continues to push up its likelihood until it saturates. The opposite effect happens for the logical opposite condition, i.e., when one action does not have at least a victory in the dataset DPO will sets its probability to 0 regardless of $\tau$. While this is less disruptive than the first example (a single probability is perturbed whereas previously the whole policy was warped by an over-achieving action) it is also much more common in real-world data. In particular, whenever the action space is large but the dataset small, some actions will necessarily be sampled rarely or only once, making it likely to never observe a victory. Especially because we do not have data on their performance $\pi$ should stick close to $\pi_{\text{ref}}$ for safety, but DPO’s objective does not promote this.

In the final example the dataset consists of two observed preferences $\mathcal{D}_{3}=\{(y_{a},y_{b}),(y_{b},y_{a})\}$ and leave the pair $(y_{a},y_{c})$ completely unobserved. We compute solutions using Adam once again, and report the results in Fig. 2 for varying values of $\tau$. We observe again here that DPO ignores the prior $\pi_{\text{ref}}$ completely, no matter how strong we regularize the objective, whereas IPO gradually decreases the probability of unobserved action with $\tau$.

Conclusion and Future Work

We presented a unified objective, called $\Psi$PO, for learning from preferences. It unifies RLHF and DPO methods. In addition, we introduced a particular case of $\Psi$PO, called IPO, that allows to learn directly from preferences without a reward modelling stage and without relying on the Bradley-Terry modelisation assumption that assumes that pairwise preferences can be substituted with pointwise rewards. This is important because it allows to avoid the overfitting problem. This theoretical contribution is only useful in practice if an empirical sampled loss function can be derived. This is what we have done in Sec 5 where we show that IPO can be formulated as a root-finding problem from which an empirical sampled loss function can be derived. The IPO loss function is simple, easy to implement and theoretically justified. Finally, in Sec. 5.3 and Sec. 5.4, we provide illustrative examples where we highlight the instabilities of DPO when the preferences are fully-known as well as when they are sampled. Those minimal experiments are sufficient to prove that IPO is better suited to learn from sampled preferences than DPO. Future works should scale those experiments to more complex settings such as training language models on human preferences data.

References

APPENDICES

Appendix A Proofs

For completeness, we briefly recall the proof of existence and uniqueness of the argmaximum of the following regularized criterion that can also be found in the work of Rafailov et al. (2023):

Now, if we define the softmax probability $\delta^{*}\in\Delta_{\mathcal{S}}$ as:

then, under the previous definitions, we have the following result:

By definition of the KL, we now that \delta^{*}=\operatorname*{arg\,max}_{\delta\in\Delta_{\mathcal{S}}}\bigg{[}-\text{KL}(\delta\;||\;\delta^{*})\bigg{]} and as:

where \log\big{(}\sum_{s^{\prime}\in\mathcal{S}}\eta(s^{\prime})\exp(\tau^{-1}f(s^{\prime}))\big{)} is a constant (does not depend on $\delta$) and $\tau$ a positive multiplicative term, then $-\text{KL}(\delta\;||\;\delta^{*})$ and $\mathcal{L}_{\tau}(\delta)$ share the same argmaximum. This concludes the proof. ∎

A.2 Non-uniqueness when Supp​(π​(⋅))≠Supp​(μ)Supp𝜋⋅Supp𝜇\texttt{Supp}(\pi(\cdot))\neq\texttt{Supp}(\mu):

Notice that if we search for a solution where the support of $\pi$ is strictly larger than that of $\mu$ then there could be multiple solutions. Let us illustrate this case with a simple example. Consider a single state $x$ and 3 actions $y_{1},y_{2},y_{3}.$ The reference policy $\pi_{\text{ref}}$ is uniform over $\{y_{1},y_{2},y_{3}\}$ and the policy $\mu$ assigns a probability $1/2$ to both $y_{1}$ and $y_{2}$ and probability to $y_{3}$.

Thus the loss is L(\pi)=2\Big{(}\tau^{-1}\big{(}p^{*}(y_{1}\succ\mu)-p^{*}(y_{2}\succ\mu)\big{)}-\log\frac{\pi(y_{1})}{\pi(y_{2})}\Big{)}^{2}. We deduce that any policy $\pi=(p,q,1-p-q)$ such that $\frac{p}{q}=e^{\tau^{-1}(p^{*}(y_{1}\succ\mu)-p^{*}(y_{2}\succ\mu))}$ is a global minimum of $L(\pi)$.

In particular there are an infinity of solutions different from the optimal solution $\pi^{*}$. The problem comes from the fact that when the support of $\mu$ does not cover the whole action space there are not enough constraints to uniquely characterize $\pi^{*}$. Assuming that the supports of $\pi_{\text{ref}}$ and $\mu$ coincide enables us to recover uniqueness of the solution, as proven in Theorem 2.

Appendix B Additional results

In this section, we show the equivalence of DPO and RLHF, regardless of whether the preference model $p^{*}$ corresponds to a Bradley-Terry model. Note that the assumption of the existence of a minimizer is to exclude cases where the loss is minimized by taking the rewards of certain actions to $+/-\infty$.

Consider a preference model $p^{*}$ such that there exists a minimizer to the Bradley-Terry loss

Then, the optimal policy for the DPO objective in Equation (3.2) and for the RLHF objective in Equation (3) with reward model given as the minimizer to the Bradley-Terry loss above are identical, regardless of whether or not $p^{*}$ corresponds to a Bradley-Terry preference model.

Recall that the optimal policy $\pi^{*}_{r}$ for a given reward function $r$ for the objective in Equation (3) is given by $\pi^{*}_{r}(y|x)\propto\pi_{\text{ref}}(y|x)\exp(\tau^{-1}r(x,y))$. It therefore follows that

In words, the value of the Bradley-Terry reward objective for $r$ is the value of the DPO objective for $\pi^{*}_{r}$. We recall also that the map $r\mapsto\pi^{*}_{r}$ is surjective.

Now, suppose $r$ is optimal for the Bradley-Terry reward objective, meaning that $\pi^{*}_{r}$ is optimal for the RLHF objective. If $\pi^{*}_{r}$ is not optimal for the DPO objective, then there exists another policy $\pi^{\prime}$ that obtains a strictly lower value for the DPO loss. But then there exists a reward function $r^{\prime}$ such that $\pi^{\prime}=\pi^{*}_{r^{\prime}}$, such as $r^{\prime}(x,y)=\tau\log(\pi^{\prime}(y|x)/\pi_{\text{ref}}(y|x))$, and this $r^{\prime}$ therefore obtains a lower Bradley-Terry loss than $r$, a contradiction.

Similarly, if $\pi^{*}$ is optimal for the DPO objective, the corresponding reward function $r(x,y)=\tau\log(\pi^{*}(y|x)/\pi_{\text{ref}}(y|x))$ must be optimal for the Bradley-Terry reward loss. The corresponding optimizer for the RLHF objective is then given by $\pi(y|x)\propto\pi_{\text{ref}}(y|x)\exp(\tau^{-1}\tau\log(\pi^{*}(y|x)/\pi_{\text{ref}}(y|x)))=\pi^{*}(y|x)$, as required. ∎