RLHF Workflow: From Reward Modeling to Online RLHF

Introduction

Reinforcement Learning from Human Feedback (RLHF) has become as a key technique for integrating human preference signals into machine learning methods, particularly in aligning Large Language Models (LLMs) with human values and preferences (Christiano et al.,, 2017; Ziegler et al.,, 2019). Notable examples include the revolutionary closed-source ChatGPT (OpenAI,, 2023), Claude (Anthropic,, 2023), and Gemini (Team et al.,, 2023), as well as the powerful open-source models like Zephyr (Tunstall et al.,, 2023), Starling (Zhu et al.,, 2023), and LLaMA-3 (Meta,, 2024). In particular, since the introduction of ChatGPT, RLHF has attracted significant interest in a diverse set of communities. However, compared to the supervised fine-tuning that is rather well studied with many great open-source projects like Open-Hermes (Teknium,, 2023) and Vicuna (Zheng et al.,, 2023), RLHF remains relatively under-explored within the open-source community.

To facilitate our discussion, we build upon the standard RLHF workflow (Ouyang et al.,, 2022; Bai et al., 2022b, ; Touvron et al.,, 2023). We characterize an LLM by a policy $\pi$, which takes a prompt $x\in\mathcal{X}$ and produces a response $a\in\mathcal{A}$ from the distribution $\pi(\cdot|x)$. We denote the initial model of RLHF as $\pi_{0}$, which is fine-tuned on some instruction-following data after the pre-training stage. We assume that we have a prompt set that is sampled from some unknown but fixed distribution $x\sim d_{0}$. The key component in RLHF is the Preference Oracle, which is mathematically defined as follows.

To further simplify the problem, it is commonly assumed that the preference signal can be modeled using the reward-based Bradley-Terry model, a well-known approach in preference learning (Bradley and Terry,, 1952; Ouyang et al.,, 2022; Bai et al., 2022a, ; Touvron et al.,, 2023).

There exists a ground-truth reward function $r^{*}$ and the preference model satisfies:

where $\sigma(z)=1/(1+\exp(-z))$ is the sigmoid function.

This modeling is a proxy of preference oracle and connects the learning objective of RLHF with reward maximization. In practice, since the BT model may not fully capture the complex human preference, we usually optimize the following KL-regularized target:

where $\eta>0$ is the KL penalty coefficient. This formulation is widely studied in practice (Ziegler et al.,, 2019; Wu et al.,, 2021; Ouyang et al.,, 2022; Rafailov et al.,, 2023; Liu et al., 2023a, ; Xiong et al.,, 2023) and admits the following intractable closed-form solution (Zhang,, 2023)

where Z(x)=\sum_{a^{\prime}\in\mathcal{A}}\pi_{0}(a^{\prime}|x)\exp\big{(}\frac{1}{\eta}r^{*}(x,a^{\prime})\big{)} is the normalization constant.

In the subsequent subsections, we first describe the existing approaches, and discuss their challenges, which should serve as the motivation for our project.

Broadly speaking, previous RLHF methods can be largely divided into two categories: (1) deep RL-based approach using Proximal Policy Optimization (PPO) (Schulman et al.,, 2017; Christiano et al.,, 2017; Ziegler et al.,, 2019) and (2) (offline) direct preference learning (e.g., DPO) approaches (Zhao et al.,, 2023; Rafailov et al.,, 2023; Azar et al.,, 2023; Tang et al.,, 2024).

This approach has been employed by ChatGPT (Ouyang et al.,, 2022) and Claude (Bai et al., 2022a, ) and has contributed to the alignment of LLaMA-2/3 (Touvron et al.,, 2023). However, it is known that even in the best case, tuning the DRL method to its best performance requires extensive efforts in hyper-parameter selection and code-level optimization (Choshen et al.,, 2019; Engstrom et al.,, 2020). This becomes even more challenging in the context of LLMs, as fine-tuning LLMs is computationally expensive. Additionally, the PPO algorithm requires loading multiple LLMs simultaneously, including the actor (policy), critic (value network), reward model, and reference model (for KL estimation), which places significant pressure on GPU memory, especially for resource-constrained open-source projects.

Direct preference learning algorithms are generally easier to tune and require fewer computational resources compared to DRL methods. Notably, in many LLM benchmarks’ leaderboards (Dubois et al.,, 2023; Zheng et al.,, 2023), powerful open-source models are primarily aligned using DPO. Considering these factors, in our project, we focus on direct preference learning algorithms while leaving the study of the DRL-based framework for future research.

While the vanilla offline direct preference learning algorithms are useful in some case studies, they also face certain challenges. Specifically, they are considered offline because they learn from an offline preference dataset collected through the following process:

Along the way of the RLHF training, the average density ratio $\frac{\pi(a|x)}{\pi_{0}(a|x)}>\exp(25)$ as reported in Figure 13 of Bai et al., 2022a . See similar results of rejection sampling fine-tuning (Dong et al.,, 2023) and DPO (Rafailov et al.,, 2023).

As a case study, consider using the behavior policy Gemma-7B-it to collect data for aligning Gemma-2B-it (Team et al.,, 2024) using 15k prompts from (Cui et al.,, 2023). The average KL divergence between Gemma-7B-it and Gemma-2B-it is calculated to be $456.4$.

Therefore, the distribution shift between policies is usually very large, and it is unlikely that we can learn the optimal policy solely from a pre-collected dataset.

2 Online Iterative RLHF

we first update the policy pair $(\pi_{t}^{1},\pi_{t}^{2})$ based on the historical data $\mathcal{D}$ collected so far;

update $\mathcal{D}\leftarrow\mathcal{D}\cup\mathcal{D}_{t}$.

The effectiveness of online data can be intuitively explained in the context of RLHF as follows. Initially, most of the samples from policy $\pi_{0}$ are in a low-reward regime and are used for the reward modeling. As a result, our reward model is accurate in evaluating low-reward responses. As training progresses, however, the policy improves and starts to generate responses with higher rewards. These high-reward responses may fall into the out-of-distribution (OOD) domain of the reward function, where the constructed reward model is no longer reliable. In contrast, using the intermediate policy to sample new responses, querying human feedback on these samples, and add them back to the training set can significantly mitigate the OOD problem, which traditional offline methods may not handle effectively. This approach enhances the reliability of the reward model in the high-reward regime, which in turn improves the policy’s performance. The intuition extends to the direct preference learning algorithms, where we essentially enlarge our preference dataset by continuously collecting new online data.

The great work of Huggingface (Tunstall et al.,, 2023) provides an open-source recipe to do direct preference learning from high-quality offline preference dataset, which can be viewed as an efficient way of distillation. In contrast, the online RLHF is still largely under-explored in the literature. Xiong et al., (2023) made the first step towards understanding the advantage of online exploration in RLHF from a theoretical perspective, and this technical report aims to provide a detailed recipe to verify the effectiveness of the proposed framework.

3 Human Feedback Approximation

Ideally, the online preference signal is sampled from a representative group of human labelers. However, human feedback is extremely expensive in practice, which the open-source community usually cannot afford. In the literature, there is a line of work showing that training a proxy preference model, and using the preference model to give proxy labels in a semi-supervised manner improve the model performance (Dong et al.,, 2023; Yuan et al.,, 2023; Liu et al., 2023a, ; Hoang Tran,, 2024). We conjecture that this is because the reward model (discriminator) usually generalizes better than the policy (generator).

In particular, Hoang Tran, (2024) shows that if the preference model (reward model) is trained on a diverse set of preference datasets, the Pair-RM (Jiang et al.,, 2023) with only 0.4B parameters can provide iterative preference learning with meaningful signals so that the resulting modelhttps://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO achieves an impressive AlpacaEval-2 length-control win rate of 26.4%. Motivated by this line of work, we first train a proxy preference (reward) model based on the diverse open-source preference datasets in Section 2 and then use the resulting model to provide preference signals for the subsequent iterative RLHF.

4 Related Work

RLHF and RLHF algorithms. The dominant RLHF framework used for the LLM alignment was first popularized in Christiano et al., (2017); Ziegler et al., (2019) and was further developed in Instruct-GPT (Ouyang et al.,, 2022), Claude (Bai et al., 2022a, ), and LLaMA-2 (Touvron et al.,, 2023). These works typically involve constructing a reward model based on the MLE of the Bradley-Terry model, and then using the PPO algorithm to optimize the reward signals with KL regularization. One notable exception is that the LLaMA-2 uses a mixture of rejection sampling fine-tuning (Dong et al.,, 2023; Wang et al.,, 2024) and PPO in their RLHF pipeline. We refer the interested readers to Bai et al., 2022a ; Touvron et al., (2023) for a detailed description. However, the use of PPO in RLHF has limitations. It is known to be unstable (Choshen et al.,, 2019), sensitive to implementation (Engstrom et al.,, 2020), and resource-intensive (Yuan et al.,, 2023). Despite some efforts to improve PPO in the context of RLHF (Li et al.,, 2023; Chan et al.,, 2024; Chang et al.,, 2024; Zhong et al.,, 2024), reproducing the successful results achieved with PPO is challenging for the open-source community due to these limitations as it requires extensive efforts and resources that the open-source communities usually cannot afford. In recognition of these issues of PPO, a line of work studies the (offline) direct preference learning algorithms, including Slic (Zhao et al.,, 2023), DPO (Rafailov et al.,, 2023), IPO (Azar et al.,, 2023), KTO (Ethayarajh et al.,, 2024), ARM (Pang et al.,, 2024), and GPO (Tang et al.,, 2024). These algorithms skip the reward modeling step, and optimize a designed loss target on the offline preference dataset directly (hence the name). It is widely observed that the direct preference learning algorithms are much more stable than the PPO, and achieve impressive performance evaluated by standard benchmarks (Tunstall et al.,, 2023; Dubois et al.,, 2023; Zheng et al.,, 2023).

RLHF benefits from online (iterative) learning. Roughly speaking, online iterative learning means that we will deploy the intermediate models and query human feedback for the responses of these models. Intuitively, this strategy can help to mitigate the OOD issue of the learned reward model (Gao et al.,, 2023), and its advantages have been reported in Ouyang et al., (2022); Touvron et al., (2023) for the PPO-based framework. Even when the additional feedback is derived from a proxy reward constructed from the same offline dataset (similar to semi-supervised learning), iterative rejection sampling fine-tuning (RAFT) (Dong et al.,, 2023) and DPO based on samples from the target distribution estimator have been shown to outperform the original offline counterparts (Pang et al.,, 2024; Liu et al., 2023a, ). Furthermore, recent works (Xiong et al.,, 2023; Xu et al.,, 2023; Hoang Tran,, 2024; Yuan et al., 2024b, ; Swamy et al.,, 2024; Chen et al., 2024b, ; Ye et al.,, 2024; Guo et al.,, 2024; Rosset et al.,, 2024; Tajwar et al.,, 2024; Calandriello et al.,, 2024; Wu et al.,, 2024) have demonstrated that online iterative variants of direct preference learning algorithms significantly outperform their offline counterparts. In particular, we refer the interested readers to Guo et al., (2024) for the extensive experimental results with different offline base algorithms.

Reward Modeling as Human Feedback Approximation

We present the details of preference model construction in this section, where we study both the reward modeling as MLE of the BT model and the general preference model. The training script and full recipe are provided in https://github.com/RLHFlow/RLHF-Reward-Modeling.

Following the Pair-RM (Jiang et al.,, 2023) and LLaMA-2 (Touvron et al.,, 2023), we use a mixture of open-source datasets as the training set. Here is a brief introduction to the datasets:

HH-RLHF (Bai et al., 2022a, ) is a pairwise preference dataset where each sample is accompanied by a conversation history and two alternative responses written by an early Claude model with 52B parameters. The preferences of the responses are annotated by humans.

SHP (Ethayarajh et al.,, 2022) is sourced from Reddit and includes examples from 18 subreddits, such as askacademia, askbaking, askengineers, and changemyview. Each example is a Reddit post with a question/instruction and a pair of top-level comments. One comment is preferred by more Reddit users than the other. All preferences and responses are provided by humans. Only samples with a score ratio $>$ 2 are used, and at most 5 pairs are taken for each prompt.

HelpSteer (Wang et al.,, 2023). This open-source dataset (Wang et al.,, 2023) contains prompts, responses, and five human-annotated attributes (helpfulness, correctness, coherence, complexity, and verbosity) ranging from 0 to 4. The prompts are generated using a mixture of template-generated and human-generated methods, while responses are generated by an in-house LLM. The authors generate up to 4 responses per prompt, and we can construct pairwise comparisons based on them.

PKU-SafeRLHF (Ji et al.,, 2024). This dataset (Ji et al.,, 2024) consists of 30k+ expert comparison data. Each sample includes two responses to a question and two preference signals for helpfulness and safety, respectively. The responses are generated by open-source chatbots, and the preference signals are merged through the results of 14 harm category multi-class classficiation.

UltraFeedback (Cui et al.,, 2023) consists of 64k prompts from diverse resources (including UltraChat, ShareGPT, Evol-Instruct, TruthfulQA, FalseQA, and FLAN) and the authors generate 4 responses per prompt using 4 different LLMs sampled from a diverse set of state-of-the-art open-source LLMs. The preference is from GPT-4 based on a fine-grained annotation instruction, which contains 4 different aspects, namely instruction-following, truthfulness, honesty and helpfulness. The dataset collection strategy of UltraFeedback has also influenced many subsequent works.

UltraInteract (Yuan et al., 2024a, ) is a preference dataset designed for complex reasoning tasks. The authors collect a preference tree for each instruction, with the instruction being the root and each action a node. A trajectory is a root-to-leaf path consisting of a sequence of actions. Paired correct and incorrect nodes or trajectories are used for preference learning.

Distilabel-Capybarahttps://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized is a preference dataset of multi-turn dialogues whose prompts are taken from Daniele and Suphavadeeprasit, (2023), where the responses are generated by open-source LLMs and preferences are generated by GPT-4.

Distilabel-Orcahttps://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs is collected similarly with Capybara but with the prompts from Lian et al., (2023).

The training of LLMs is highly data-dependent. To ensure high-quality training data, we conduct a filtering process on the open-source datasets we use. This process removes low-quality and meaningless samples. Additionally, conversations with empty rounds or incorrect labels (implied by the other features of the dataset) are eliminated. Furthermore, in datasets where absolute scores are available, pairwise comparisons with small margins are excluded as these preference signals tend to be noisy (Bansal et al.,, 2023). This process roughly deletes 10% of the data. We summarize the statistics of the open-source datasets that are used for the training in Table 1 and prepare them, as well as our data filtering script, on the huggingface, which is available at https://huggingface.co/collections/RLHFlow/standard-format-preference-dataset-662eec0252e194d5d40c252a.

2 Bradley-Terry Reward Model and Preference Model

Bradley-Terry model construction. We follow the previous works (Ouyang et al.,, 2022; Bai et al., 2022a, ) to initialize the reward model using the SFT modelFor preference/reward modeling, we use meta-llama/Meta-Llama-3-8B-Instruct since only this checkpoint is available in the early stage of this project. . We replace the last layer with a linear head to predict a scalar score suitable for preference learning. The reward model is trained using the negative log-likelihood loss function, enabling maximum likelihood estimation (MLE). This loss function is defined as:

instruction = [CONTEXT] {x} [RESPONSE A] {$a^{1}$} [RESPONSE B] {$a^{2}$}, and label = A.

3 Evaluation Result

We consider two versions of the training set:

Mix1: HH-RLHF + SHP + UltraFeedback + Summarization (Stiennon et al.,, 2020).

The Mix1 dataset is similar to the construction of UltraFeedback (Cui et al.,, 2023) with an additional summarization dataset. In comparison, the Mix2 consists of more reasoning preference pairs (math and code) and safety data. We also consider three different approaches to model the preference signals, including prompting in the LLM-as-a-judge manner (Zheng et al.,, 2023), reward modeling as the MLE of the BT reward model, and the preference model.

We evaluate the models using the RewardBench (Lambert et al.,, 2024), a benchmark designed to assess reward model capabilities in four categories: Chat, Chat-Hard, Safety, and Reasoning. The main evaluation results are in Table 2. It is evident that without explicit training, the prompting approach is inferior to both the BT model and the preference model across all metrics. Meanwhile, the preference model outperforms the BT model in reasoning tasks related to coding and math. We also notice that with more data, especially data specified in coding, math, and safety, the reward model trained by mix2 achieves higher accuracy in safety and reasoning compared with early versions of attempts. In particular, we use the Ultra-RM-13B as a reference model in Table 2, where we can observe that the extra data related to safety and reasoning (as well as a stronger base model) largely contribute to the superior performance of our reward model.

Length bias in reward modeling. It is known that the LLMs aligned by RLHF usually give longer responses (Xiong et al.,, 2023; Yuan et al., 2024b, ; Yuan et al., 2024b, ), where the length bias also exists in the reward models, likely influenced by the preference data used. To better understand this bias, we randomly sample 2K prompts from the prompt set and use the SFT model to generate 8 responses per prompt (see the details in Section 3). Then, we compute the lengths and rewards of the responses and plot the heatmaps of the Pearson correlation coefficient between them in Figure 5. Clearly, both of the two reward models are biased toward the longer responses to some degree. In comparison, UltraRM-13B demonstrates a stronger bias, as we observe that the mean Pearson correlation coefficient of the UltraRM-13B (left figure) is 0.19, while it is 0.06 for our BT reward trained on mix2 (right figure). This may partially result from the use of additional Capybara, OpenOrca, and UltraInteract, whose preferred responses are shorter than the rejected responses. We will return to the ablation study of the impacts of the reward models in Section 4.

We mention in passing that in the literature, it is also common to model the different types of signals separately, and strategically merge them in the subsequent policy optimization stage (Touvron et al.,, 2023; Wang et al.,, 2024), which we leave for future work.

Iterative Policy Optimization

We develop the main algorithms for the online iterative RLHF in this section, with both theoretical insights and implementation details. In particular, the algorithms will be designed in a direct preference learning style for stable and efficient training.

The base model used in this project is LLaMA-3-8B. To ensure the reproducibility and openness of the project, we perform SFT by ourselves to obtain the initial policy $\pi_{0}$. We collect a set of high-quality instruction datasets for SFT, such as ShareGPT, SlimOrca, MathInstruct, and Evol-Instruct (see the Appendix for a full list). The training is carried out for one epoch with a learning rate of $2\times 10^{-5}$. A cosine scheduler is employed, and the global batch size is set to 32 with a warm-up ratio of 0.03. To accelerate training, we follow Diao et al., (2023); Tunstall et al., (2023) to pack the samples and use a block size of 8192. We use the SFT-Trainer from TRL and the detailed recipe is available at https://github.com/RLHFlow/Online-RLHF.

2 Iterative Direct Preference Learning: Theoretical Insights and Algorithmic Principles

We present the main algorithmic framework in Algorithm 1 for the online iterative RLHF, and summarize the key principles and algorithmic ideas as follows.

Hybrid batch learning. We formulate a slightly more general framework to combine an initial offline dataset with online data collected during training, hence the name hybrid learning, similar to the recipe of Claude (Bai et al., 2022a, ) and LLaMA-2 (Touvron et al.,, 2023). Additionally, to mitigate the computational costs of training and deploying large LLMs, we use a large batch size m for sparse updates.

Non-symmetric structure to balance exploitation and exploration. The framework also features a non-symmetric structure as we divide the two agents into a main agent and an enhancer.

We have the following theoretical guarantees when strategic exploration methods are applied.

For any precision parameter $\epsilon>0$, with a batch size $m=\widetilde{O}(\frac{d_{e}}{\epsilon^{2}})$ and other suitable choices of hyper-parameters, then, with high probability, after at most $T=\widetilde{O}(d_{e})$ iterations, we can find a $t_{0}\in[T]$ so that

3 Practical Implementation Details

We now shift our focus from the theoretical insight to the practical implementation. We provide an illustration of our implementation in Fig. 6.

Exploration policy. The primary challenge lies in the choice of enhancer policy for exploration. Recall that our goal is to find an enhancer policy that maximizes the relative uncertainty to the main agent from the following policy subset:

Unfortunately, the uncertainty estimator does not have an analytical form except the linear case. But the main insight we can derive here is to maximize the policy difference with $\pi_{t}^{1}$, while maintaining a moderate KL divergence. This motivates us to use model variants of $\pi_{t}^{1}$. We discuss some popular heuristic implementations here.

Adjusting Temperature and Training Steps. In the project of Claude (Bai et al., 2022a, ), the authors choose to use the models with different training steps as $(\pi_{t}^{1},\pi_{t}^{2})$. For instance, if we run PPO for $2$ epoch in total, we may take $\pi_{t}^{1}$ as the model saved at the end of the first epoch and take $\pi_{t}^{2}$ as the one saved at the end of the second epoch. Additionally, the LLaMA-2 project (Touvron et al.,, 2023) adjusts the sampling temperature of $\pi_{t}^{1}$ to induce $\pi_{t}^{2}$. These modifications introduce diversity in the models and facilitate exploration.

Rejection Sampling is another popular ensemble-based exploration approach (Nakano et al.,, 2021; Dong et al.,, 2023; Gulcehre et al.,, 2023). In the context of LLMs, it is typically restricted to the best-of-$n$ sampling. Specifically, we sample $n$ independent responses by $\pi_{t}^{1}$ for each prompt, and then use a preference/reward function to rank the responses and take the one with the highest reward as the final output. In other words, we take $\pi_{t}^{2}$ as the best-of-n variant of $\pi_{t}^{1}$. In this way, the $\pi_{t}^{2}$ enlarges the margins between $\pi_{t}^{1}$ and provides exploration. Meanwhile, in this case, the KL divergence between the two policies is upper bounded by $\log n-\frac{n-1}{n}$ and is usually far better than this conservative estimation (Beirami et al.,, 2024).

Prompt set, and data generation. We collect prompts from UltraFeedback (Cui et al.,, 2023), HelpSteer (Wang et al.,, 2023), OpenOrca (Lian et al.,, 2023), UltraInteract (Yuan et al., 2024a, ), Capybara (Daniele and Suphavadeeprasit,, 2023) and DIBT-10Khttps://huggingface.co/datasets/DIBT/10k_prompts_ranked and prepare the full prompt set herehttps://huggingface.co/datasets/RLHFlow/prompt-collection-v0.1. In our experiments, we use a subset of 60K prompts and iterate for three iterations, so 20K prompts are used to generate 20K x 8 responses per iteration. To accelerate data generation, we use VLLM (Kwon et al.,, 2023) for inference. We set the max generation length as 2048, and use a sampling temperature of 1.0/0.7 without any top-k/top-p strategy. To offer a more intuitive comprehension of our prompts collection, we provide visualization plots in Figure 7, which are generated on Nomic Atlashttps://atlas.nomic.ai/ with the nomic-embed-text-v1.5 text embedding model (Nussbaum et al.,, 2024).

Evaluation of the Model

We evaluate the models by standard benchmarks, including AlpacaEval-2, MT-Bench, and Chat-Arena-Hard.

AlpacaEval-2 (Dubois et al.,, 2023): This benchmark focuses on single-turn conversations and consists of 805 test prompts covering various topics. The models are compared head-to-head with GPT-4-Preview (11/06) to compute the win rate. The same GPT-4 model is used as the judge. To mitigate the length bias of GPT-4, a length-control variant of the benchmark is also proposed.

MT-Bench (Zheng et al.,, 2023): This benchmark is a multi-turn benchmark and includes 160 test prompts from 8 different areas. The model should first answer an initial question, and then a pre-defined follow-up question. The model’s responses are then rated by the GPT-4 model with a scale from 1-10, and the final score is computed as the average score of two turns.

Chat-Arena-Hard (Tianle et al.,, 2024): This benchmark consists of 500 test prompts from the live data in Chatbot Arena, a crowd-sourced platform for LLM evaluations. The prompts evaluate the model’s ability in specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application. In addition to the agreement to human preference, compared with AlpacaEval-2 and MT-Bench, Chat-Arena-Hard further enjoys a clear separability among different models.

We also measure the ability of the resulting models using academic benchmark, including GSM-8K (Cobbe et al.,, 2021), MMLU (Hendrycks et al.,, 2020), HumanEval (Chen et al.,, 2021), TruthfulQA (Lin et al.,, 2021), ARC (Clark et al.,, 2018), and MBPP (Austin et al.,, 2021). These benchmarks evaluate the models’ ability in coding, reasoning, and general knowledge. In particular, it is known that RLHF alignment can introduce performance degeneration in reasoning, calibration (providing accurate confidence estimates), and truthfulness capabilities (generating accurate and factual responses), which is also referred to as the alignment tax in the literature (Ouyang et al.,, 2022; Bai et al., 2022a, ; OpenAI,, 2023). Therefore, evaluating our model on these benchmarks is crucial to understanding the impact of iterative RLHF on these specific aspects.

We summarize the benchmarks we use in this project in Table 3.

2 Main Results

Online iterative RLHF significantly improves conversation quality. We evaluate our model’s conversation abilities using AlpacaEval-2, MT-Bench, and Chat-Arena-Hard, (results in Table 4). Compared to other open-source models with less than 10B parameters, our model – SFR-Iterative-DPO-LLaMA-3-8B-R outperforms them on the conversation and instruction-following benchmarks with a significant margin. Notably, our model trained with iterative DPO consistently outperforms that of vanilla offline DPO (DPO baseline). This demonstrates the advantage of online iterative RLHF. Moreover, our model outperforms the Tulu-2-DPO-70B and GPT-3.5-turbo-1106, which are aligned by DPO or PPO and are much larger than our base model. These results show that the online iterative RLHF can effectively adjust the style of the model responses, thus improving the conversation quality.

Academic Task. As RLHF can impact a model’s reasoning and calibration abilities, typically in a negative way (Bai et al., 2022a, ; Ouyang et al.,, 2022; OpenAI,, 2023), we compare our model’s performance on academic benchmarks (Table 5) with the SFT checkpoint and other baselines. We don’t observe significant performance regression compared to the SFT baseline. Interestingly, our iteratively DPO-aligned model even outperforms the SFT model in GSM-8K, MMLU, TruthfulQA, and ARC benchmarks. We believe that these increased capacities of the model are injected in the pre-training stage and SFT stage, and iterative DPO helps it leverage them more effectively. This is because the 60K alignment data used in the iterative RLHF are orders of magnitude less than those used in the previous two stages.

The RLHF-aligned models based on some initial checkpoints and more epochs can approach or even outperform the state-of-the-art closed-source models like GPT-4 and Claude on benchmarks. However, we remark that we should be more careful in interpreting these results because the test sets of the benchmarks are finite and may not be representative enough to capture the complicated real-world scenarios. Moreover, the increased possibility of small models overfitting the benchmarks may lead to benchmark hacking, which means that the real capacity of a model with a high score is still limited. In particular, while it is possible to get even higher results on the benchmark (e.g., 44.84 in LC AlpacaEval-2 and 35.7 in Chat-Arena-hard, but the performance on academic benchmarks drops significantly), we presented our current model by human evaluation on some randomly chosen test prompts.

Ablation study on filtering data with length penalty. We observed that the aligned model’s response length was significantly longer than the SFT baseline (potentially due to reward model bias as shown in Figure 5). To address this, we conducted an ablation study by incorporating a length penalty into the reward function:

where $|a|$ is the number of characters of the response. We compare the model trained with this penalty to the vanilla version and report the results in Table 6. As expected, the length penalty effectively mitigated the length bias, leading to shorter responses. In particular, the model trained with length penalty achieves a superior length-control AlpacaEval-2 win rate, as well as better results on some academic benchmarks. This demonstrates the advantage of mitigating length bias and motivates us to study the verbosity issue in reward modeling further. Finally, we notice that the model trained with length penalty is worse in the Chat-Arena-Hard benchmark. This may suggest that we also need a length-control version for this benchmark to provide a more reasonable evaluation.

On the impact of reward model. We investigate the effects of the reward (preference) model used in the online iterative RLHF. Our model’s performance is compared to a model trained with UltraRM-13B, and the ablation study results are summarized in Table 6. We observe that the model trained with UltraRM-13B has longer responses than ours, which is consistent with its stronger bias, as shown in Figure 5. Considering the alignment tax, the accuracy on the academic benchmarks drops more than our models. One important reason is that the UltraRM-13B does not have a good reasoning ability (see Table 2), so it may not provide appropriate preference signals for reasoning-related conversions. For instance, the model may favor some responses with many comments in the coding task, which tend to be very helpful but are indeed useless when evaluated by humans. Notably, the model trained with UltraRM-13B achieves a higher Chat-Arena-Hard win rate than our concise version, which also supports the verbosity bias of the Arena-Hard benchmark. During the training process, we also observe that the model trained with UltraRM-13B achieves a lower training loss, which may suggest that the signals of UltraRM-13B are more consistent and easy to learn. In contrast, the convergence under our reward model is slower due to the complex preference signal.

End Note and Future Direction

In this technical report, we study the workflow of the online iterative RLHF, which leverages on-policy sampling and external preference signals from a proxy preference model trained on a diverse set of open-source preference datasets. The resulting model (referred to as SFR-Iterative-DPO-LLaMA-3-8B-R) demonstrates impressive performance on standard benchmarks, and the report provides detailed instructions for reproducing the results, including data, code, models, and hyper-parameter choices.

There are still many potential directions to explore. First, as we can see in Table 6, the iterative RLHF heavily relies on the quality of the preference signal. In this project, we use a proxy scalar reward model trained on a diverse set of open-source datasets to approximate human feedback. It would be interesting to see whether we can design a more effective strategy to model different types of preference signals, like a multi-head reward (Wang et al.,, 2024) and classification-based activation strategy (Touvron et al.,, 2023). Second, while the rejection sampling seems to be a good heuristic exploration strategy, it is still interesting to see whether we can design more effective ways for exploration. Finally, most of the models after RLHF tend to reply the prompts with much longer responses. Such a length bias is further amplified in the iterative RLHF framework. We presented a preliminary study on this issue by leveraging an additional length penalty in reward for data filtering. It would be interesting to see whether we can further mitigate this issue by additional algorithmic designs or post-training techniques.

We hope the results of this project can advance the direction of online iterative RLHF and contribute to the training of stronger and larger open-source LLMs.

Acknowledgements

The authors would like to thank the great open-source communities, including the Huggingface TRL team, the Huggingface H4 team, the Allen Institute AI RewardBench team, the Meta LLaMA team, and Axolotl team for sharing the models, codes, and training sets. We also thank Zihao Li for assistance in reward modeling.

References

Appendix A Authorship and Credit Attribution

All authors provided valuable contributions to this project, each bringing unique expertise and insights that were crucial for its success.

HD first demonstrated that iterative DPO algorithm can achieve state-of-the-art performance; wrote a development version code for SFT, iterative RLHF; contributed to the training of SFT model and BT-RM; conducted extensive experiments on training and hyper-parameter tuning of iterative RLHF; delivered the released BT reward model; provide preference dataset for final BT-RM and some initial versions of the prompt data; conducted the RM evaluation and GPT-based evaluation of the generative models; contributed to paper writing; contributed to the public version of iterative RLHF code.

WX wrote the codes for the Bradley Terry reward model and conducted most of the experiments for both reward and preference model training; delivered the released pairwise preference model; contributed to the preference dataset search and hyper-parameter tuning; initiated and organized the online iterative RLHF project; wrote the initial code for the online iterative DPO and prepared its public version on GitHub; contributed to the evaluation of the reward and preference models; assisted in the collection and the cleaning of the preference dataset; wrote the paper.

BP conducted most of the final SFT and RLHF experiments and delivered the released SFT and RLHF model; independently wrote a development version code for SFT and iterative RLHF; developed the SFT recipes (data, hyper-parameter, model selection); conducted extensive experiments on the training and hyper-parameter tuning of SFT, offline and iterative RLHF; conducted the GPT-based evaluation and all the academic benchmarks; contributed to paper writing.

HW initiated the training code of the pairwise preference model, and conducted experiments in the training of the Bradley Terry reward model and pairwise preference model; collected, filtered, and deduplicated the prompt set; contributed to the preference dataset collection and data cleaning; contributed to the evaluation and analysis of the reward and preference models; made substantial writing contributions to the reward modeling section and created illustrative figures for the algorithmic frameworks and dataset visualization.

HZ, YZ, NJ, DS, CX, TZ supported and advised the works of the junior authors, provided computational resources, and suggested experiments and writings.

Appendix B Additional Experimental Details

SFT Data List. We collect open-sourced instruction-finetuning data for our SFT model training. The following data is included: ShareGPT, Evol-Instruct, SlimOrca, MathInstruct, Magicoder-Evol-Instruct, GPT4-LLM, OrcaMath, GPTeacher, UltraInteract.

Offline Vanilla DPO. We use Nectar dataset for Offline DPO. We run 1 epoch with batch size 128, learning rate 5e-7, and cosine decay scheduler.

Additional Plots. We also visualized our performance as Figure 8.