Contrastive Preference Learning: Learning from Human Feedback without RL

Introduction

As large pretrained models have become increasingly performant, the problem of aligning them with human preferences have risen to the forefront of research. This alignment is especially difficult when larger datasets inevitably include suboptimal behaviors. Reinforcement learning from human feedback (RLHF) has emerged as a popular solution to this problem. Using human preferences, RLHF techniques discriminate between desirable and undesirable behaviors with the goal of refining a learned policy. This paradigm has shown promising results when applied to finetuning large language models (LLMs) (Ouyang et al., 2022), improving image generation models (Lee et al., 2023), and adapting robot policies (Christiano et al., 2017) – all from suboptimal data. For most RLHF algorithms, this process includes two phases. First, a reward model is trained from collected user preference data. And second, that reward model is optimized by an off-the-shelf reinforcement learning (RL) algorithm.

Unfortunately, this two-phase paradigm is founded on a flawed assumption. Algorithms that learn reward models from preference data require that human preferences are distributed according to the discounted sum of rewards or partial return of each behavior segment. However, recent work (Knox et al., 2022) calls this into question, positing that humans instead provide preferences based on the regret of each behavior under the optimal policy of the expert’s reward function. Intuitively, a human’s judgement is likely based on optimality, instead of which states and actions have higher quantity for reward. As a result, the correct quantity to learn from feedback might not be the reward, but instead the optimal advantage function or, in other words, the negated regret.

In their second phase, two-phase RLHF algorithms optimize the reward function learned from the first phase with RL. In practice, RL algorithms suffer from a suite of optimization challenges stemming from temporal credit assignment, such as the high-variance of policy gradients (Marbach & Tsitsiklis, 2003) or instability of approximate dynamic programming (Van Hasselt et al., 2018). Thus, past works limit their scope to circumvent these issues. For instance, RLHF techniques for LLMs assume a contextual bandit formulation (Ouyang et al., 2022), where the policy receives a single reward value in response to a given query to the user. While this reduces the need for long-horizon credit assignment, and consequently the high variance of policy gradients, in reality user interactions with LLMs are multi-step and sequential, violating the single-step bandit assumption. As another example, RLHF has been applied to low-dimensional state-based robotics problems (Christiano et al., 2017; Sikchi et al., 2023a), a setting where approximate dynamic programming excels, but not yet scaled to more realistic high-dimensional continuous control domains with image inputs. Broadly, RLHF methods not only incorrectly assume that the reward function alone drives human preferences, but also require mitigating the optimization challenges of RL by making restrictive assumptions about the sequential nature of problems or dimensionality.

In this work, we introduce a new family of RLHF methods that use a regret-based model of preferences, instead of the commonly accepted partial return model that only considers the sum of rewards. Unlike the partial return model, the regret-based model directly provides information about the optimal policy. A fortunate outcome of this is that it completely eliminates the need for RL, allowing us to solve RLHF problems in the general MDP framework with high-dimensional state and action spaces. Our key insight is to combine the regret-based preference framework with the principle of Maximum Entropy (MaxEnt), resulting in a bijection between advantage functions and policies. By exchanging optimization over advantages for optimization over policies, we are able to derive a purely supervised learning objective whose optimum is the optimal policy under the expert’s reward. We refer to our approach as Contrastive Preference Learning due to its resemblance with commonly accepted contrastive learning objectives.

CPL has three key benefits over prior work. First, CPL can scale as well as supervised learning because it uses only supervised objectives to match the optimal advantage without any policy gradients or dynamic programming. Second, CPL is fully off-policy, enabling effectively using any offline sub-optimal data source. Finally, CPL can be applied to arbitrary Markov Decision Processes (MDPs), allowing for learning from preference queries over sequential data. To our knowledge, no prior methods for RLHF simultaneously fulfill all three of these tenants. To demonstrate CPL’s adherence to the three aforementioned tenants, we show its effectiveness on sequential decision making problems with sub-optimal and high-dimensional off-policy data. Notably, we show that CPL can effectively use the same RLHF fine tuning procedure as dialog models to learn temporally extended manipulation policies in the MetaWorld Benchmark. Specifically, we pretrain policies using supervised learning from high-dimensional image observations, before fine tuning them with preferences. Without dynamic programming or policy gradients, CPL is able to match the performance of prior RL based methods. At the same time, it is $1.6\times$ faster and four times as parameter efficient. When using denser preference data, CPL is able to surpass the performance of RL baselines on 5 out of 6 tasks.

Preliminaries

We consider the general reinforcement learning from human feedback (RLHF) problem within a reward-free MDP ${\mathcal{M}}/r=({\mathcal{S}},{\mathcal{A}},p,\gamma)$ with state space ${\mathcal{S}}$, action space ${\mathcal{A}}$, transition dynamics $p(s_{t+1}|s_{t},a_{t})$, and discount factor $\gamma$. We assume all states are reachable by some policy. The goal of RLHF is to learn a policy $\pi(a|s)$ that maximizes an expert user’s reward function $r_{E}(s,a)$. However, since the reward function is not given in an MDP $/r$, it must be inferred from the expert’s preferences. Typically, a user preference orders two behavior segments. A length-$k$ segment is denoted $\sigma=(s_{1},a_{1},s_{2},a_{2},\dots,s_{k},a_{k})$. We use $\sigma^{+}\succ\sigma^{-}$ to indicate that segment $\sigma^{+}$ was preferred to $\sigma^{-}$ by the user without loss of generality and assume we are given a dataset $\mathcal{D}_{\text{pref}}=\{(\sigma^{+}_{i},\sigma^{-}_{i})\}_{i=1}^{n}$ of such preferences where $\sigma^{+}\succ\sigma^{-}$.

Maximum Entropy Reinforcement Learning. The aim of maximum-entropy reinforcement learning is to learn a policy $\pi$ that maximizes its causal entropy in addition to the cumulative discounted return, leading to the objective:

where $\alpha$ is a temperature parameter. Augmenting the reward function with an additional negated $\log\mu(a|s)$ term for reference distribution $\mu(a|s)$ yields the KL-constrained objective used in offline RL (Levine & Koltun, 2013; Garg et al., 2023) and prominent RLHF approaches for LLMs (Ziegler et al., 2019; Ouyang et al., 2022). Though we adopt the standard maximum entropy framework, our approach easily extends to the constrained setting. Under policy $\pi$ and reward function $r$, we denote the state-value function by $V^{\pi}_{r}(s)$ and state-action value function by $Q^{\pi}_{r}(s,a)$. The advantage function, $A^{\pi}_{r}(s,a)\triangleq Q^{\pi}_{r}(s,a)-V^{\pi}_{r}(s)$, measures how much worse taking action $a$ is than acting according to $\pi$. We use $\pi^{*}$ as short-hand for the solution to Eq. 1 with reward function $r_{E}$, and write its corresponding corresponding value functions as $V^{*}(s)$ and $Q^{*}(s,a)$ instead of $V_{r_{E}}^{\pi^{*}}$ and $Q_{r_{E}}^{\pi^{*}}$. We measure the optimality of behavior directly by using the advantage function of $\pi^{*}$, $A^{*}(s,a)$.

The Regret (or Advantage) Preference Model. Learning $\pi^{*}$ requires characterizing how preferences are generated according to a preference model $P_{E}\left[\sigma^{+}\succ\sigma^{-}\right]$, or the probability the expert prefers $\sigma^{+}$ to $\sigma^{-}$. Typically, the preference model is chosen to be the Boltzmann rational distribution over each segment’s discounted partial return, $\sum_{t=1}^{k}\gamma^{t}r_{E}(s_{t},a_{t})$, where $r_{E}$ is the expert’s hidden reward function. However, such models have been shown to be inconsistent with real human preferences (Knox et al., 2022). For instance, consider a sparse reward $r_{E}(s,a)=1\{s=g\}$. Two segments that do not reach the goal would have the same partial returns even if one moved towards the goal $g$ while the other moved away from it. This inconsistency is resolved by considering preferences to be distributed according to the Boltzmann rational distribution over the negated discounted regret under $r_{E}$, or $-\sum_{t=1}^{k}\gamma^{t}(V^{*}(s_{t})-Q^{*}(s_{t},a_{t}))$. In this framework, a user’s preference indicates that a segment has lower regret with respect to their intended optimal policy. Leveraging the equivalence of negated regret and the discounted sum of optimal advantages, we equivalently write the regret-based preference model as

where we use the shorthand “$+$” and “$-$” as indexing the states and actions of segments $\sigma^{+}$ and $\sigma^{-}$. In the next section, we use the regret preference model in combination with the principle of maximum causal entropy to derive CPL.

Contrastive Preference Learning

Though recent work has shown that human preferences are better modeled by the optimal advantage function or regret, most existing RLHF algorithms assume otherwise. By learning a reward function with a mistaken model of preference and then applying RL, traditional RLHF approaches incur a vast, unnecessary computational expense (Knox et al., 2023). Our aim is to derive simple and scalable RLHF algorithms that are purpose-built for the more accurate regret model of human preferences.

Modeling human preferences with regret is not new, but past work suffers from a number of shortcomings. Specifically, existing algorithms using the regret preference model are brittle, as they rely on estimating gradients with respect to a moving reward function, which thus far has only been approximated by computing successor features and assuming a correct linear or tabular representation of the expert reward function $r_{E}$ (Knox et al., 2022; 2023). Consequently, these algorithms appear unsuitable for complex scenarios beyond the simplistic grid world environments in which they have been tested.

The key idea of our approach is simple: we recognize that the advantage function, used in regret preference model, can easily be replaced with the log-probability of the policy when using the maximum entropy reinforcement learning framework. The benefit of this simple substitution is however immense. Using the log-probability of the policy circumvents the need to learn the advantage function or grapple with optimization challenges associated with RL-like algorithms. In sum, this enables us to not only embrace a more closely aligned regret preference model, but also to exclusively rely on supervised learning when learning from human feedback.

In this section, we first derive the CPL objective and show that it converges to the optimal policy for $r_{E}$ with unbounded data. Then, we draw connections between CPL and other supervised-learning approaches. Finally, we provide recipes for using CPL in practice. Our algorithms are the first examples of a new class of methods for sequential decision making problems which directly learn a policy from regret based preferences without RL, making them far more efficient.

Under the regret preference model, our preference dataset $\mathcal{D}_{\text{pref}}$ contains information about the optimal advantage function $A^{*}(s,a)$, which can intuitively be seen as a measure of how much worse a given action $a$ is than an action generated by the optimal policy at state $s$. Therefore, actions that maximize the optimal advantage are by definition an optimal actions and learning the optimal advantage function from preferences should intuitively allow us to extract the optimal policy.

Eliminating the need to learn advantage. In maximum entropy RL, Ziebart (2010) has shown that the following relationship between the optimal advantage function and optimal policy holds:

This means that in order for a learned advantage function to be optimal, it must be normalized, that is $\int_{\mathcal{A}}e^{A^{*}(s,a)/\alpha}da=1$. Enforcing this constraint is intractable, particularly in continuous spaces with large neural networks, making naïvely learning $A_{\theta}$ via maximum likelihood estimation difficult.

However, one might instead notice that the above equation establishes a bijection between the advantage function $A^{*}_{r}$ and the policy $\pi^{*}$, namely that the optimal advantage function is proportional to the optimal policy’s log-likelihood:

This means that instead of learning the optimal advantage function, we can directly learn the optimal policy. Given preferences are distributed according to the optimal advantage function for the expert reward function $r_{E}$, we can write the preference model in terms of the optimal policy $\pi^{*}$ by substituting Eq. 3 into Section A.6 as follows,

Thus, the maximum entropy framework has led to a model of human preferences that is solely in terms of the optimal policy $\pi^{*}$. Using this equivalent form of the advantage-based preference model, we can directly optimize a learned policy $\pi_{\theta}$ to match the preference model via maximum likelihood with the following convex objective:

Assuming sufficient representation power, at convergence $\pi_{\theta}$ will perfectly model the users preferences, and thus exactly recover $\pi^{*}$ under the advantage-based preference model given an unbounded amount of preference data. Specifically, in Appendix A, we prove the following Theorem:

Assume an unbounded number of preferences generated from a noisy rational regret-preference model with expert advantage function $A^{*}$. CPL recovers the optimal policy $\pi^{*}$ corresponding to reward $r_{E}$.

This proof relies on the bijection between optimal advantage functions and policies in maximum entropy RL and the fact that the regret preference model is identifiable (Knox et al., 2022), meaning the objective can achieve a loss of zero.

Benefits of directly learning the policy. Directly learning $\pi$ in this manner has several benefits, both practical and theoretical. Perhaps most obviously, directly learning the policy circumvents the need for learning any other functions, like a reward function or value function. This makes CPL extremely simple in comparison to prior work. When scaling to larger models, only learning the policy reduces both complexity and computational cost. Second, as pointed out by prior works (Christiano et al., 2017; Hejna & Sadigh, 2023), reward learning can be harmed by the invariance of Boltzmann rational preference models (Section A.6) to shifts; i.e., adding a constant to each exponent does not change $P[\sigma^{+}\succ\sigma^{-}]$. In CPL the distributional constraint of the policy ($\pi_{\theta}(a|s)\geq 0$ for all $a$ and $\int_{{\mathcal{A}}}\pi_{\theta}(a|s)da=1$) remedies this issue, since adding a constant makes $\int_{{\mathcal{A}}}\pi_{\theta}(a|s)da\neq 1$. This removes the need for any complicated normalization scheme. Finally, per previous arguments, the policy’s distributional constraint guarantees that $\int_{\mathcal{A}}e^{A_{\theta}(s,a)/\alpha}da=1$. Thus, it can be shown that CPL’s learned implicit advantage function is always the optimal advantage function for some reward function. We call this property, defined below, consistency and prove the following Proposition in Appendix A.

An advantage function $A(s,a)$ is consistent if there exists some reward function $r(s,a)$ for which $A$ is the optimal advantage, or $A(s,a)=A^{*}_{r}(s,a)$.

CPL learns a consistent advantage function.

The consequences of this are that no matter the amount of preference data used, CPL will always learn the optimal policy for some reward function, and adding additional preference data only improves the implicit estimate of $r_{E}$.

Connections to Contrastive Learning. When deriving CPL, we intentionally chose to denote preferred and unpreferred behavior segments by “+” and “-” to highlight the similarities between CPL and contrastive learning approaches. Though some two-phase RLHF approaches have drawn connections between their reward learning phase and contrastive learning (Kang et al., 2023), CPL directly uses a contrastive objective for policy learning. Specifically, Eq. 5 is an instantiation of the Noise Constrastive Estimation objective (Gutmann & Hyvärinen, 2010) where a segment’s score is its discounted sum of log-probabilities under the policy, the positive example being $\sigma^{+}$ and the negative $\sigma^{-}$. In the appendix we show that when applied to ranking data using a Plackett-Luce Model, CPL recovers the InfoNCE objective from Oord et al. (2018) where the negative examples are all the segments ranked below the positive segment. Effectively, CPL has fully exchanged the reinforcement learning objective for a supervised, representation learning objective while still converging to the optimal policy. As marked success has been achieved applying contrastive learning objectives to large-scale datasets and neural networks (Chen et al., 2020; He et al., 2020; Radford et al., 2021), we expect CPL to scale more performantly than RLHF methods that use traditional RL algorithms.

2 Practical Considerations

The Contrastive Preference Learning framework provides a general loss function for learning policies from advantage-based preferences, from which many algorithms can be derived. In this section, we detail practical considerations for one particular instantiation of the CPL framework which we found to work well in practice. In the appendix, we include several instantiations of CPL for different types of data and conservative regularizers.

where $\sigma_{i,t}^{+}$ denotes the $t$th timestep of the preferred segment from the $i$th comparison in $\mathcal{D}_{\text{pref}}$. We can reason about the set of all policies that yield the same CPL loss by assembling all comparison vectors into a matrix $X$, where the $i$th row of $X$ is the vector $x_{i}$ for the $i$th comparison in the dataset. Any changes to $\log\pi$ in the null space of $X$ have no effect on the logits of the logistic function, and consequently no effect on the loss. In practice, $|{\mathcal{S}}\times{\mathcal{A}}|>>n$, making the null space of $X$ often nontrivial such that there are multiple minimizers of the CPL loss, some of which potentially place a high probability on state-action pairs not in the dataset. In Section A.3 we provide constructions of $X$ where this is true. Next, we show how this problem can be resolved by incorporating regularization into the CPL objective.

Regularization. In finite settings, we want to choose the policy that minimizes the CPL loss function while placing higher likelihood on actions in the dataset. To accomplish this, we modify Eq. 5 with a conservative regularizer that assigns lower loss when the policy has higher likelihood on actions in $\mathcal{D}_{\text{pref}}$, keeping it in-distribution. Though there are many possible choices of regularizers, we use an asymmetric “bias” regularizer adapted from An et al. (2023) as it performed best in our experiments. Within our objective, the bias regularizer down-weights negative segments by $\lambda\in(0,1)$ as so:

If the policy places more weight on actions in the dataset, $\log\pi_{\theta}(a|s)$ will increase. In the standard Boltzmann model, increasing the log-probabilities of both the positive and negative segments by the same amount would have no effect on the loss. The bias, however, weighs the increased log-probabilities of the negative segments less, which ultimately decreases the loss. Thus, while a minimizer of the vanilla CPL loss function could place a high probability on unseen actions, Eq. 6 is minimized with a higher weight on in-distribution actions. This is formally captured by the following proposition, which shows that, for a fixed policy, $\mathcal{L}_{\text{CPL}{(\lambda)}}$ is lower when the policy places a higher likelihood on actions in the dataset versus other comparisons with the same CPL Loss.

Consider a comparison $\sigma^{+}\succ\sigma^{-}$ from $\mathcal{D}_{\text{pref}}$ and an arbitrary comparison $\sigma^{\prime+}\succ\sigma^{\prime-}$ such that $\mathcal{L}_{\text{CPL}}(\pi,\sigma^{+}\succ\sigma^{-})=\mathcal{L}_{\text{CPL}}(\pi,\sigma^{\prime+}\succ\sigma^{\prime-})$ for a fixed policy $\pi$. If $\sum_{\sigma^{+}}\gamma^{t}\log\pi(a_{t}^{+}|s_{t}^{+})>\sum_{\sigma^{\prime+}}\gamma^{t}\log\pi(a_{t}^{+}|s_{t}^{+})$, then $\mathcal{L}_{\text{CPL}(\lambda)}(\pi,\sigma^{+}\succ\sigma^{-})<\mathcal{L}_{\text{CPL}(\lambda)}(\pi,\sigma^{\prime+}\succ\sigma^{\prime-})$.

Essentially, this shows that the bias regularizer breaks ties in the CPL loss function by penalizing lower likelihoods. We prove this, along with a more general version, in Section A.4. In Appendix B we also consider CPL variants with other forms of conservative regularization.

Experiments

In this section, we address the following questions about CPL: First, is CPL effective at fine-tuning policies from regret-based preferences? Second, does CPL scale to high-dimensional control problems and larger networks? Finally, what ingredients of CPL are important for attaining high performance? Additional experiments and details are included in the appendix.

Preference Data. We evaluate CPL’s ability to learn policies for general MDPs from sub-optimal off-policy rollout data and preferences. In particular, we consider the training procedure commonly used for large foundation models: supervised learning, followed by fine-tuning with RLHF. To do this, we use six tasks from the simulated MetaWorld robotics benchmark (Yu et al., 2020). First, we train baseline policies until they approximately reach a 50% success rate. Then, we rollout 2500 episodes of length 250 for each suboptimal stochastic policy. We then form synthetic preference datasets $\mathcal{D}_{\text{pref}}$ of different sizes by sampling segments of length 64 uniformly from the rollout data. We estimate regret-based preference labels using the $Q$-function and policy of an oracle Soft Actor-Critic (SAC) (Haarnoja et al., 2018) model trained to 100% success on a combination of the suboptimal rollout and online data. In practice, we consider two main types of preference datasets: dense, where we label comparisons between every sampled segment (effectively ranking all segments), and sparse, where we label only one comparison per segment.

Baseline Methods. We consider three strong baselines. The first baseline is supervised fine-tuning (SFT), where a policy is first trained with BC on all segments in $\mathcal{D}_{\text{pref}}$, then further fine-tuned on only the preferred segments, i.e., all $\sigma^{+}$ in $\mathcal{D}_{\text{pref}}$. The second baseline is Preference IQL (P-IQL), which learns a reward function from $\mathcal{D}_{\text{pref}}$ assuming the partial return preference model, then subsequently learns a policy to maximize it with Implicit $Q$-Learning (Kostrikov et al., 2022), a state-of-the-art offline RL algorithm. Though P-IQL was first used with the partial return model, here it uses an approximation of $A^{*}_{r_{E}}$ as its reward function, which as we show in Appendix A’s Corollary 1 preserves the optimal policy. In fact, P-IQL should be even more performant with regret-based labels, since $A^{*}_{r_{E}}$ is a highly shaped potential-based reward function for $r_{E}$ Ng et al. (1999); Knox et al. (2023). Hejna & Sadigh (2023) found that a well-tuned implementation of P-IQL outperformed several recent state-of-the-art preference-based RL methods, so we use their implementation. Finally, to demonstrate CPL’s ability to extrapolate beyond the best performance found in the rollout data, we compare to %BC, where a policy is trained with behavior cloning on the top X% of rollouts according to the ground truth $r_{E}$.

How does CPL perform with state-based observations? Our main state-based results can be found in rows 1 and 3 of Table 1. When using sparser comparison data (row 3), CPL outperforms prior methods in 5 of 6 environments, often by a substantial margin of over P-IQL, particularly in Button Press, Bin Picking, and Sweep Into environments. When applied to datasets with more dense comparisons, CPL outperforms P-IQL even more (row 1), doing so substantially in all environments. Though the dense-comparison datasets have less state-action coverage, they have substantially more preference comparisons than the sparse comparison datasets. We posit that more comparisons per segment is more beneficial to CPL than to P-IQL because of its contrastive objective – more comparison-rich datasets are likely to have more informative positive-negative pairs that help shape the policy. We find that CPL consitently outperforms %BC, indicating the CPL is indeed exhibiting policy improvement beyond the best behaviors in the dataset.

How does CPL scale to high-dimensional observations? To test how CPL’s supervised objectives scale to high-dimensional continuous control problems, we render the MetaWorld datasets discussed above to $64\times 64$ images. We use the network architecture from DrQv2 (Yarats et al., 2022) and the same hyper-parameters as our state-based experiments. We additionally use random shift augmentations, which drastically improve the performance of RL from images (Laskin et al., 2020).

Our image-based results can be found in rows 2 and 4 of Table 1. Interestingly, we find that performance moderately increases for SFT but substantially for P-IQL. We posit that this is because data-augmentation, which is inapplicable in state, plays a key role in improving value representation for P-IQL. Despite this, when learning from denser preference data (row 2), CPL still outperforms P-IQL in 4 of 6 environments and ties on Sweep Into. When learning from sparser comparisons (row 4), CPL and P-IQL perform comparably on most tasks, even though CPL is drastically simpler than P-IQL. Again, the gap in performance between CPL and P-IQL is higher with denser comparison data, underscoring the importance of informative negatives.

These results are only more impressive considering CPL’s significant reduction in complexity. P-IQL must learn a reward function, a $Q$-function, a value function, and a policy. CPL avoids all of this, and only learns a policy, drastically reducing training time and parameter count. As we can see in Table 2, this means that CPL runs $1.62\times$ faster than P-IQL on images and has less than a quarter of the the parameters. As networks get larger and larger, the performance gain from using CPL would only increase.

2 What contributes to CPL’s performance?

As alluded to in previous sections, we find that the gap in performance between CPL and baselines is higher for datasets with denser comparisons. This is consistent with prior works in contrastive learning (Robinson et al., 2021). To study this effect, evaluate CPL’s performance as we increase the number of comparisons sampled per segment over a fixed dataset of 5000 segments. We show results of this for Drawer Open with state-based observations on the left of Fig. 2 and include the rest in Section C.3 in addition to dense data scaling. Overall, we find that CPL benefits from an increasing number of comparisons per segment in all tasks except Plate Slide. P-IQL is less affected, though sometimes performs worse with more comparisons, which we suspect is due to reward under-fitting. This highlights another drawback of P-IQL – due to its higher number of components, it has more hyperparameters and is consequently more sensitive to changes in the dataset. We tuned hyperparameters for all methods with 10K comparisons, then left them the same for scaling experiments.

Finally, we ablate both of CPL’s hyperparameters – the temperature value $\alpha$ and bias regularizer $\lambda$ – for Drawer Open on the right of Fig. 2. While CPL generally performs well with all values, we find that higher performance could have been attained with further hyper-parameter tuning, particularly for $\lambda$. In the Appendix B we ablate more design decisions, like the choice of conservative regularizer.

Related Work

Though RLHF has recently surged in popularity, learning policies from human preferences has been a long-studied problem, referred to as preference-based RL (PbRL).

PbRL methods typically start by learning a reward function, usually from pairwise comparisons, then use an RL algorithm for policy optimization (Fürnkranz et al., 2012). While Akrour et al. (2012; 2011); Wilson et al. (2012) were some of the first examples of PbRL, more recently several works have shown that, provided thousands of queries or sufficient pretraining, PbRL can train deep neural-network policies for control using comparisons (Christiano et al., 2017; Lee et al., 2021; Ibarz et al., 2018; Brown et al., 2020; Hejna & Sadigh, 2022; Shin & Brown, 2021) or rankings (Brown et al., 2019; Bıyık et al., 2019; Sikchi et al., 2023a). These approaches, however, are generally demonstrated only on low-dimensional state-based control because of the challenges RL faces when scaling to larger inputs and networks (Ota et al., 2021). In the past, removing RL has lead to effective algorithms for goal-conditioned RL from images (Hejna et al., ; Eysenbach et al., 2022). CPL does the same but for PbRL. Other works address the problem of selecting feedback (Sadigh et al., 2017; Biyik et al., 2020; Daniel et al., 2015), which we consider complementary because CPL can benefit from higher quality data elicitation.

To scale RLHF, recent approaches for refining LLMs have ignored the temporal component of RL, and instead treated text-generation as a contextual bandits problem (Ziegler et al., 2019). While this approach has proven effective at tasks like (Stiennon et al., 2020; Wu & Hu, 2018), instruction following (Ouyang et al., 2022; Nakano et al., 2021), and even image generation (Lee et al., 2023; Black et al., 2023), it fundamentally ignores the fact that interaction with users is often sequential, spanning multiple turns. Unlike these methods, CPL works with general MDPs. CPL’s unique ability to learn from sequence data with only supervised objectives makes it a prime candidate for scaling to more complex problems. In fact, Direct Preference Optimization (DPO) (Rafailov et al., 2023) recently demonstrated that a supervised objective similar to CPL works better than RL in the contextual bandits setting. We show in Appendix A that DPO can be derived as a special case of CPL in which segments are of length 1 and always start at the same state. This parallels Knox et al. (2023), who show that the common contextual bandit-approach is a special case of the naïve approach from Section 3.

To derive CPL’s objective, we leverage knowledge from works building on the principle of maximum entropy in control (Ziebart et al., 2008; Ziebart, 2010; Haarnoja et al., 2017). The resulting contrastive update directly learns the optimal policy with fully off-policy data. This is unlike many RL-based RLHF algorithms in both langauge (Ziegler et al., 2019) or control (Christiano et al., 2017) which require on policy rollouts and additional learned components that have been shown to increase variance (Hejna & Sadigh, 2023). Similar contrastive learning objectives have shown to be effective for temporal representation learning (Ma et al., 2023), even with preference data (Kang et al., 2023).

Discussion

In this work we introduce CPL, a novel framework for RLHF using the regret preference model. Theoretically, we proved that CPL always learns a consistent advantage function and converges to the optimal policy for the expert’s reward function. Practically, we showed that CPL’s supervised objective is able to outperform RL baselines when learning complex manipulation policies from dense preference data while being simpler and $1.6\times$ faster.

Limitations. CPL, like other RLHF approaches, assumes knowledge of the human rater’s temporal discounting (i.e., of the discount factor $\gamma$), which in practice would be difficult to communicate. As CPL’s loss function is computed over segments, it requires a substantial amount of GPU memory for large segment sizes. Finally, no model of human behavior is perfect.

Future Directions. Several exciting research directions remain. First is scaling CPL to larger datasets and architectures where we believe its benefits will be more pronounced. One potentially exciting application is LLMs, where CPL enables fine-tuning on multiple steps of turn-based dialogue. To our knowledge, no multi-step preferences dataset currently exists for LLMs. Second, our work only considers offline data generated by suboptimal policies. An online version of CPL could be developed that works with online human feedback, allowing policies to continually improve.

This work was supported by NSF Award 2006388, NSF Award 2218760, Ford, DARPA YFA, AFOSR YIP, NSF (IIS-1749204), AFOSR (FA9550-20-1-0077), ARO (78372-CS, W911NF-19-2-0333), ONR (N00014-21-1-2685) and the Center for AI Safety. JH is supported by a DoD NDSEG Fellowship. CF is a CIFAR Fellow in the Learning in Machines and Brains program. WK is supported by UT Austin’s Good Systems grand challenge. We would like to thank Archit Sharma for valuable discussions on the conservative regularizer used in CPL. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.

Contributions

JH led the project, contributing to all aspects including ideation, theory, experimentation, and writing. RR proposed linking advantages and likelihoods and contributed to early stage ideation. HS contributed to the theory, experiment design, and ran experiments. CF, SN, WBK, DS oversaw, advised, and provided feedback on the project.

References

Appendix A Theory

We first prove a lemma about the consistency of CPL as it is used when proving convergence.

Any function $A(s,a)$ that satisfies $\int_{\mathcal{A}}e^{A(s,a)/\alpha}da=1~{}\forall s\in{\mathcal{S}}$ is a consistent advantage function under some reward function $r$ in the MaxEntRL setting.

Idea. Given advantage $A(s,a)$, we want to show that there exists a reward function $r$ for which $A$ is the optimal advantage function.

Given $\int_{\mathcal{A}}e^{A(s,a)/\alpha}da=1$, consider the corresponding policy $\pi^{A}(a|s)=e^{A(s,a)/\alpha}$. Let the reward function be the advantage, or $r(s,a)=A(s,a)=\alpha\log\pi^{A}(a|s)$. We can determine the optimal policy $\pi^{*}$ for this reward according to Eq. 1:

Thus, the objective is point-wise maximized if and only if $\pi^{A}(\cdot|s)=\pi(\cdot|s)~{}~{}\forall~{}s\in{\mathcal{S}}$. Therefore, $\pi^{A}$ is the optimal policy for reward function $r(s,a)=A(s,a)$.Note that we assume that all states are reachable and therefore have support in $\rho^{t}_{\pi}(s)$ for any optimal MaxEnt policy. Under this reward function, $\pi^{*}=\pi^{A}=e^{A}$, which implies that $A$ is a consistent advantage function.

CPL learns a consistent advantage function.

Optimization via CPL fits a valid policy $\pi$ subject to $\int_{\mathcal{A}}\pi(a|s)da=1~{}\forall s\in{\mathcal{S}}$, with corresponding MaxEnt Advantage function $A(s,a)=\alpha\log\pi(a|s)$.

Thus, by the above Lemma CPL fits a consistent advantage function.

The reward function $r$ and the reward function defined as the optimal advantage function for $r$, $A^{*}_{r}$, have the same optimal MaxEnt policy.

This corollary can be seen by examining the proof of Lemma 1. According to the MaxEnt RL objective for reward $r$ the optimal policy is $\pi^{*}_{r}=e^{A^{*}_{r}/\alpha}$ (Ziebart, 2010). Therefore $A^{*}_{r}=\alpha\log\pi^{*}_{r}$. Repeating the steps of Lemma 1 by setting $r^{\prime}=A^{*}_{r}=\alpha\log\pi^{*}_{r}$, we get the following objective for the optimal policy $\pi^{*}_{r^{\prime}}$ with respect to $r^{\prime}$:

Since the final expression above is minimized only when $\pi=\pi^{*}_{r}$, then $\pi^{*}_{r^{\prime}}=\pi^{*}_{r}$. In other words, the reward function $r$ and reward function $r^{\prime}=A^{*}_{r}$ have the same optimal MaxEnt policy.

A.2 Proof of Convergence

Assume an unbounded number of preferences generated from a noisy rational regret-preference model with expert advantage function $A^{*}$. CPL recovers the optimal policy $\pi^{*}$.

Proof. Without loss of generality we let $\alpha=1$. For the purposes of this proof only, let $\sigma_{k}$ denote a segment of length $k$ where the state-actions in the segment are denoted by $\sigma_{k}=(s_{0},a_{0},s_{1},a_{1},...,s_{k-1},a_{k-1})$. Let $y$ be the label indicating whether the expert regret preference model prefers $\sigma_{k}^{1}$ to $\sigma_{k}^{0}$, i.e., $y\sim P_{A^{*}}\left[\sigma_{k}^{1}\succ\sigma_{k}^{0}\right]$. Let $\hat{A}=\log\hat{\pi}$ be the implicit estimate of $A^{*}$ learned by CPL. For brevity, we will use the shorthand $A(\sigma_{k})=\sum_{\sigma_{k}}\gamma^{t}A(s_{t},a_{t})$ to denote the discounted sum of advantages of a segment $\sigma$. Let $P(\sigma^{1}_{k},\sigma^{2}_{k})=\text{Bern}(\frac{e^{A^{*}(\sigma^{1})}}{e^{A^{*}(\sigma^{1})}+e^{A^{*}(\sigma^{0})}})$ and $Q(\sigma^{1}_{k},\sigma^{2}_{k})=\text{Bern}(\frac{e^{\hat{A}(\sigma^{1})}}{e^{\hat{A}(\sigma^{1})}+e^{\hat{A}(\sigma^{0})}})$ The cross-entropy CPL loss function can be re-written as follows:

The KL divergence is optimized only when the two distributions are exactly equal. Because the preference model is rational and we assume sufficient representation power and unbounded data, it is possible for the loss to converge to zero by pointwise matching KL-divergence for each two comparisons (See Knox et al. (2022) for more information specific to the identifiability of regret based preferences). Thus, under the assumption of unbounded data, for all possible segments $\sigma_{k}^{1},\sigma_{k}^{0}$ we must have that

Consider $\sigma_{k}^{p}=(s^{p}_{0},a^{p}_{0},s^{p}_{1},a^{p}_{1}...s^{p}_{k-1},a^{p}_{k-1})$ where $p\in\{0,1\}$. We will show that the above equality also holds for all sequences of length $k-1$. Consider the last action for the segment $\sigma_{k}^{1}$ denoted as $a^{1}_{k-1}$, then:

Now, we will use the consistency of CPL. Per Ziebart (2010), for the optimal value function $A^{*}$, $\sum_{a\in{\mathcal{A}}}e^{A^{*}(s,a)}=1,~{}\forall s$. Because CPL is consistent (Proposition 1), we also have that $\sum_{a\in{\mathcal{A}}}e^{\hat{A}(s,a)}=1,~{}\forall s$. We use this, in combination with the fact that all possible dynamically feasible segments of length $k-1$ are a subset of dynamically feasible segments of length $k$ to arrive at:

Applying the same argument again, this time for $\sigma_{k}^{0}$, we have

which is equivalent to $A^{*}(s,a)=\hat{A}(s,a)~{}~{}\forall s,a$.

A.3 Convexity of CPL with Finite Data

CPL is convex, but not strictly convex. Here we show that the CPL loss function is convex in $\log\pi$. Consider the logistic regression interpretation of CPL for finite data

where $x_{i}$ is the “comaprison” vector for the $i$th comparison in $\mathcal{D}_{\text{pref}}$. We can re-write this using matrix notation as:

The hessian of this objective (logistic regression) with respect to $\log\pi$ is $X^{\top}DX$, where $D$ is the diagonal matrix such that $D_{ii}=\textrm{logistic}(x_{i}\cdot\log\pi)(1-\textrm{logistic}(x_{i}\cdot\log\pi))$. As $X^{\top}DX$ is symmetric, it is guaranteed to be positive semi-definite making the objective function convex. The distributional constraint of CPL, that $\forall s\in{\mathcal{S}},\int_{\mathcal{A}}e^{\log\pi(a|s)}da=1$, is also convex as $e^{\log\pi}$ is convex in $\log\pi$. Thus, the overall objective is convex.

However, this does not imply strict convexity, or that there is a unique solution. $X^{\top}DX$ is only positive definite if it is full rank, which is unlikely to happen in practice as usually $|{\mathcal{S}}\times{\mathcal{A}}|>>n$. This means that the objective is likely not strictly convex in practice, as there can exist more than one minimizer of the objective, formally denoted $\hat{\pi}=\operatorname*{arg\,min}_{\pi}\mathcal{L}_{\text{CPL}}(\pi,\mathcal{D}_{\text{pref}})$. To prove that CPL is not always strictly convex, we construct another policy $\hat{\pi}^{\prime}$ such that $\mathcal{L}_{\text{CPL}}(\hat{\pi},\mathcal{D}_{\text{pref}})=\mathcal{L}_{\text{CPL}}(\hat{\pi}^{\prime},\mathcal{D}_{\text{pref}})$. First, we demonstrate this on a simple single-state MDP and then provide a general construction for arbitrary MDPs with discrete actions.

A simple example. Consider a single state MDP with three actions $a^{1},a^{2},a^{3}$ and expert reward function $r_{E}(s,a^{i})=r^{i}$ where $i$ indexes the actions. It can be shown that, due to the single state nature of this simple MDP, the optimal maximum entropy advantage function is $A^{*}(s,a^{i})=r^{i}$. Consider a preference dataset $\mathcal{D}_{\text{pref}}$ consisting only of comparisons between segments $(s,a^{1})$ and $(s,a^{2})$. According to the regret preference model, the expert labels these preferences according to $\text{Bern}\left(\exp r^{1}/(\exp r^{1}+\exp r^{2})\right)$ and thus we expect some labels in the preference matrix $X$ to conflict. The finite CPL loss becomes

where $c_{1}$ and $c_{2}$ are the number of comparisons where $a^{1}$ and $a^{2}$ were preferred respectively. By taking the gradient of this objective, it can be shown that the loss is optimized only when $\textrm{logistic}\left(\alpha\log\pi(a^{1}|s)-\alpha\log\pi(a^{2}|s)\right)=\frac{c_{1}}{c_{1}+c_{2}}$ or reducing, $\alpha\log\pi(a^{1}|s)-\alpha\log\pi(a^{2}|s)=\log\frac{c_{1}}{c_{2}}$. Intuitively, this makes sense, as the logits are optimized to produce the same ratio of preferences as found in the dataset. However, when we consider the unseen action $a^{3}$, to which we can assign arbitrary probability, the existence of multiple optimizers $\hat{\pi}$ becomes clear. For example, take $c_{1}=c_{2}$. By the conditions above its straightforward to see that $\hat{\pi}=[0.5,0.5,0.0]$ is an optimum of the CPL loss function. However, $\hat{\pi}=[0.1,0.1,0.8]$ achieves the same loss as its difference in log probabilities $\log\pi(a^{1}|s)-\log\pi(a^{2}|s)$ is the same. If $c_{2}=0$, or we have no conflicting preferences, $\hat{\pi}=$ and the implied $\hat{A}=\log\hat{\pi}$ is undefined, implying some of the reward values are infinite. This means we do not have enough data to accurately fit $\pi^{*}$. Next, we provide a construction for more general MDPs in the presence of OOD actions.

There exists a vector $u\in N(X)$ such that for state action $s,a$ contained in $X$, $u(s,a)\neq 0$. In other words, the null space is non-trival on the support of $\mathcal{D}_{\text{pref}}$.

For every state in the dataset where there is an action such that $u(s,a)\neq 0$, there exists at least one out-of-distribution (OOD) action $a_{\text{OOD}}$ not in the dataset. The indicator vector for $s,a_{\text{OOD}}$ is thus a basis vector for $N(X)$.

Let $\hat{\pi}$ be the minima of the CPL loss function. We will construct $\hat{\pi}^{\prime}$ as follows. Select a vector $u\in N(X)$ that is non-zero for at least one $s,a$ pair in $X$. As $u\in N(X)$, we have that $\mathcal{L}_{\text{CPL}}(\hat{\pi},\mathcal{D}_{\text{pref}})=\mathcal{L}_{\text{CPL}}(e^{\log\hat{\pi}+u},\mathcal{D}_{\text{pref}})$. However, $e^{\log\hat{\pi}+u}$ violates the policy constraint as it may not integrate to one. We can fix this problem by adding or removing probability mass from the OOD actions we have assumed exist at states where $u$ is non-zero. We do this by constructing another vector $v\in N(X)$ by choosing one $a_{\text{OOD}}$ at each state without loss of generality. By examining the total sum of probabilities of the modified policy,

we can normalize the sum using the indicator vectors for $s,a_{\text{OOD}}$, which are necessarily in the nullspace $N(X)$. Consider a vector $v$ such that at each state $s$, $v(s,a)=0$ except for at $a_{\text{OOD}}$, where $v(s,a_{\text{OOD}})=\log(1-\sum_{a\neq a_{\text{OOD}}}\hat{\pi}(a|s)e^{u(s,a)})-\log\hat{\pi}(a_{\text{OOD}}|s)$. Then,

As $v$ is formed from a linear combination of basis vectors of $N(X)$, $v\in N(X)$. Consequently, $\mathcal{L}_{\text{CPL}}(\hat{\pi},\mathcal{D}_{\text{pref}})=\mathcal{L}_{\text{CPL}}(e^{\log\hat{\pi}+u+v},\mathcal{D}_{\text{pref}})$ and by the above construction $\hat{\pi}^{\prime}=\hat{\pi}e^{u+v}$ is a valid policy. This completes the construction.

We have shown that an infinite number of policies can attain the same optima, just by shifting the amount of probability assigned to OOD actions. For some of these solutions, the entire mode of the policy is potentially out-of-distribution. In the offline setting, the pessimism principle dictates that we should discourage modes that are out-of-distribution. We fix this by introducing regularization.

A.4 Conservative Bias Regularization

CPL loss translates a relative weighting between preferences to a policy, but does not employ any mechanism to ensure the learned policy is close to the dataset. In the offline setting, this can be detrimental if the learned policy incorrectly extrapolates to out of distribution actions. A similar approach, under the name of pessimism or conservatism, is commonly seen in offline RL literature (Levine et al., 2020; Jin et al., 2021; Sikchi et al., 2023b). As expalined in Section 3.2, we want to learn policies that have a high-probability on the dataset. However, there are many datasets that potentially have the the same loss, as $\mathcal{L}_{\text{CPL}}$ depends only on the difference in probability for each preference comparison, or $\sum_{\sigma^{+}}\gamma^{t}\log\pi(a_{t}|s_{t})-\sum_{\sigma^{-}}\gamma^{t}\log\pi(a_{t}|s_{t})$, and thus constants added to the log probabilities of each segment cancel. However, we would prefer that a higher loss is given when the policy is assigns lower probability to actions in the dataset.

To remedy this, we introduced bias regularizer $\lambda\in(0,1)$ in Section 3, which leads to the modified preference loss:

Next, we prove that this loss discourages the policy from learning modes that are out-of-distribution, starting with the proposition from the main text.

Consider a comparison $\sigma^{+}\succ\sigma^{-}$ from $\mathcal{D}_{\text{pref}}$ and an arbitrary comparison $\sigma^{\prime+}\succ\sigma^{\prime-}$ such that $\mathcal{L}_{\text{CPL}}(\pi,\sigma^{+}\succ\sigma^{-})=\mathcal{L}_{\text{CPL}}(\pi,\sigma^{\prime+}\succ\sigma^{\prime-})$ for a fixed policy $\pi$. If $\sum_{\sigma^{+}}\gamma^{t}\log\pi(a_{t}^{+}|s_{t}^{+})>\sum_{\sigma^{\prime+}}\gamma^{t}\log\pi(a_{t}^{+}|s_{t}^{+})$, then $\mathcal{L}_{\text{CPL}(\lambda)}(\pi,\sigma^{+}\succ\sigma^{-})<\mathcal{L}_{\text{CPL}(\lambda)}(\pi,\sigma^{\prime+}\succ\sigma^{\prime-})$.

Succinctly, this proposition states that if preference comparisons each achieve the same loss, the less likely comparisons under the policy (in this case $\sigma^{\prime+}\succ\sigma^{\prime-}$), will have higher regularized CPL loss. Essentially, this shows that the regularized objective encourages the policy to have higher likelihood on the provided comparisons than any other potential comparison that exists.

Proof. By the stated assumptions, it must be that $\sum_{\sigma^{\prime+}}\gamma^{t}\log\pi(a_{t}|s_{t})+\delta=\sum_{\sigma^{+}}\gamma^{t}\log\pi(a_{t}|s_{t})$ for some $\delta>0$. As the two comparisons also have the same CPL Loss, their logits must be the same, or

Consequently, the same $\delta$ must hold for the negative segments, or $\sum_{\sigma^{\prime-}}\gamma^{t}\log\pi(a_{t}|s_{t})+\delta=\sum_{\sigma^{-}}\gamma^{t}\log\pi(a_{t}|s_{t})$. We can then examine the regularized CPL loss under each comparison. First, we evaluate the finite regularized loss for $\sigma^{+}\succ\sigma^{-}$, algebraically simplified for clarity:

We can then compare this to the regularized loss for $\sigma^{\prime+}\succ\sigma^{\prime-}$.

The key step in the above is substituting the relationship between the log probabilities of the comparisons. As $\delta>0$ and $0<\lambda<1$, it can easily be seen that the loss is lower for $\sigma^{+}\succ\sigma^{-}$, letting us conclude that

We can extend this proposition to the regularized CPL loss over entire datasets as follows:

For a fixed policy $\pi$, consider two preference datasets $\mathcal{D}_{n}=\{(\sigma^{+}_{i},\sigma^{-}_{i})\}_{i=1}^{n}$ and $\mathcal{D}^{\prime}_{n}=\{(\sigma^{\prime+}_{i},\sigma^{\prime-}_{i})\}_{i=1}^{n}$ such that $\forall m=1,2,...,n,\mathcal{L}_{\text{CPL}}(\mathcal{D}_{m},\pi)=\mathcal{L}_{\text{CPL}}(\mathcal{D}^{\prime}_{m},\pi)$. Then, if $\sum_{\sigma^{\prime+}_{i}}\gamma^{t}\log\pi(a_{t}|s_{t})\leq\sum_{\sigma^{+}_{i}}\gamma^{t}\log\pi(a_{t}|s_{t})$ for all $i$ and strictly for at least one $i$,

The proof of this amounts to first noticing that, because the preference losses are the same for every ordered subset, the losses for the $i$th datapoints in $\mathcal{D}_{n}^{\prime}$ and $\mathcal{D}_{n}$ must be the same. Then, we can repeatedly apply Proposition 2. Since the inequality is strict at at-least one datapoint, the regularized loss will be strictly lower.

We can construct datasets for which this is applicable. For example, consider a dataset $\mathcal{D}$ containing a total ordering over $n$ segments, $\sigma^{1}\succeq\sigma^{2}\succeq...\succeq\sigma^{n}$. The unregularized loss for this policy and dataset is $\mathcal{L}_{\text{CPL}}(\pi,\mathcal{D})$. We can construct another dataset $\mathcal{D}^{\prime}$ over a different set of totally ordered segments from anywhere in the state space $\sigma^{\prime 1}\succeq\sigma^{\prime 2}\succeq..\succeq\sigma^{\prime n}$ such that:

for all $i=1,2,...,n$ and some $\delta\geq 0$.

A.5 CPL for Rankings

We can derive a version of CPL for ranking data using a Plackett-Luce model (Plackett, 1975). We denote the chosen ranking as a permutation $\tau:[K]\to[K]$ where $K$ is the number of segments presented, $\sigma^{1},...,\sigma^{K}$. The Plackett-Luce model under regret based preferences is:

This model generalizes to Bradley-Terry (Bradley & Terry, 1952) when $K=2$. To learn the optimal policy, we maximize the log likelihood of the above and make the same substitution as CPL, $\alpha\log\pi^{*}(a|s)=A^{*}(s,a)$. This gives us the CPL loss function for rankings, which can be seen as a verison of the InfoNCE objective. Without loss of generality, we order the permutations $\tau$ such that $\sigma^{1}\succeq\sigma^{2}\succeq...\succeq\sigma^{K}$.

Except for the sum over $k$, this is the exact objective from Oord et al. (2018) where the scores are the discounted sum of log probabilities over the segments.

A.6 Direct Preference Optimization as special case of CPL

Reduction via Maximum Entropy Advantage. Note that by the Bellman equation,

DPO (Rafailov et al., 2023) assumes the contextual-bandits setting, thus the MDP terminates after a single step and there is no next state $s^{\prime}$. As we can see from the above, in this setting, $A^{*}(s,a)=r_{E}(s,a)-V^{*}(s)$. DPO also assumes that all preferences start from the same state $s$, and thus only actions $a^{+}$ and $a^{-}$ differ. This is consistent with RLHF on LLMs as humans score “responses” to fixed prompts.

which is the same preference model used in DPO. From here the same conservative derivation as DPO can be applied by noting that, for KL-constrained contextual bandits, $\pi^{*}(a|s)=\mu(a|s)e^{Q^{*}(s,a)-V^{*}(s)}=\mu(a|s)e^{r_{E}(s,a)-V^{*}(s)}$ for reference distribution $\mu$. Solving for $r_{E}$, we can perform a substitution just like in CPL to arrive at the DPO objective.

CPL under Constrained Regret Preferences. We can also consider a setting where users provide preferences constrained to a reference distribution $\mu$. This might arise in scenarios where users are only shown a fixed set of behaviors, and do not extrapolate far beyond them. Though we do not believe this premise has previously been considered, it leads to an interesting result.

Assume preferences to be distributed according to the $KL$ constrained advantage function. In this setting, $\pi^{*}(a|s)=\mu(a|s)e^{A^{*}(s,a)}$ and by substitution the CPL loss becomes

which is essentially a multi-step generalization of DPO which has not previously been considered. In the next section, we expand on this as a variant of CPL.

Appendix B Variants of CPL

In the main body of the paper, we presented the version of CPL which we found to consistenly attain good performance. In some of our experiments, we also considered two other variants of CPL. We detail these below.

BC-Regularized CPL. Instead of using our biased conservative regularization from An et al. (2023), we consider using a simple BC regularizer. This can be derived by considering the objective:

Relaxing the problem via Lagrangian duality with langrangian $\beta$, we arrive at a BC regularized version of CPL.

KL Constrained CPL. We can also consider the setting where preferences are assumed to be distributed according to the constrained advantage function. Though in practice we sample preferences according to the maximum entropy advantage function, we found this approach to still work in many settings. First, we learn the reference distribution $\mu$ using behavior cloning. Then, we use constrained CPL with bias regularization, making the final loss function:

CPL with Dense Preferences. When learning from “dense” preference data, it is possible to augment the batch to include more comparisons using the transitive property. Specifically, given a batch of $b$ segments, we compute all possible pairwise comparisons within a batch:

This provides as much contrastive signal per-batch as possible. We applied this technique to our CPL experiments with images, and found that it lead to a slight increase in performance for some tasks.

Appendix C Extended Results

In this section we provide our full experimental results:

Learning curves from state for CPL, baselines, and variants described in Appendix B.

Learning curves from images for CPL and baselines.

Scaling results for CPL and P-IQL with different sized dense datasets and fixed sparse datasets with a varying number of comparisons.

Results when varying the number of comparisons for a fixed dataset.

C.2 Image Learning Curves

C.3 Data Scaling

C.4 Additional Ablations

Appendix D Experiment Details.

Our datasets and code are publicly released at https://github.com/jhejna/cpl.

We use a modified version of the MetaWorld environments (Yu et al., 2020) in our experiments, which we found necessary to obtain good regret-based preference labels. MetaWorld was designed for Meta-RL, and thus by default hides the goal from the state spaces. Prior works like Lee et al. (2021), have randomized the goal but left it hidden, making the reward function stochastic. We randomize the goal, but make it observable to remove reward stochasticity. We additionally randomize the initial position of the arm, which is not done by default. This increases data coverage but also leads to more robust policies. Finally, in MetaWorld v2 the state by default includes object and proprioceptive history. We remove proprioceptive history to make the environment more Markovian.

D.2 Datasets and Preference Labeling

In Section 4 we provided details on how we generated our datasets. Though we tried to select suboptimal SAC checkpoints that achieves approximately a 50% success rate, there was some variance. In Table 3 we show the overall success rate of trajectories in the rollout dataset for each environment. We also apply gaussian noise of standard deviation 0.3 when collecting rollouts. Next, we provide further details on how we generated accurate regret-based labels.

First, we train an Oracle SAC policy to obtain $Q^{*}$ and $\pi^{*}$. To ensure low TD-error on the offline rollout dataset, we add all rollouts to the replay buffer of the SAC model before we start training. We then run SAC as usually, collecting online data, but with a sufficiently large replay buffer such that no data rollout data is overridden.

After training the policy, we estimate regret labels for entire segments at a time by writing the negated regret in terms of the value function and reward. We find that this lowers variance. Under deterministic dynamics, it can be shown that:

D.3 Evaluation

Evaluating CPL in comparison to other RL baselines can be difficult, as CPL uses only supervised learning, while P-IQL is RL based. Superivsed learning methods, like CPL can easily overfit the training data. On the other hand, off-policy RL methods like P-IQL converge to a fixed point and thus often take much longer to train before overfitting. While notable works in imitation learning (Mandlekar et al., 2021) have reported the average of the maximum performance of each seed, we find this to be a bit overly optimistic to randomness in evaluation episodes and assumes one can evaluate every checkpoint. on the other hand, offline RL works like (Kostrikov et al., 2022), report evaluation after a fixed amount of training, which can lead to overfitting for supervised approaches like CPL. We take a middle-of-the-road approach when reporting numbers in Table 1.

Every 5000 steps for state-based experiments and every 2500 steps for image-based experiments we run 25 evaluation episodes. We then average evaluation performance across eight neighboring checkpoints with a running average, totaling 200 evaluation episodes. We then average this value across seeds. Finally, we take the maximum point of the average. This exactly corresponds to the peak of the learning curves provided for all of our experiments. This evaluation procedure first averages performance over a number of checkpoints and episodes, and then averages over seeds. This maximum-of-the-average approach mitigates the over-optimism of average-of-the-maximum evaluation procedures like those used in Mandlekar et al. (2021). At the same time, it assumes that we can reasonably stop training before over-fitting begins.

D.4 Hyperparameters

Below we detail hyper-parameters for all methods. Note that we assumed all policies to be gaussian with fixed variance. Thus, we simply predicted actions using a standard MLP and computed the log probability $\log\pi(a|s)$ for CPL as $-||\pi(s)-a||_{2}^{2}$.