HelpSteer2: Open-source dataset for training top-performing reward models

Introduction

Since the pioneering works on Reinforcement Learning from Human Feedback , the significance of incorporating preference information into model alignment has been consistently demonstrated . Both proprietary models (e.g., GPT-4 , Claude , Gemini ) and open-source models (e.g., Llama 3 , Mistral , and Yi ) have benefited from preference modeling techniques. However, most of these models lack detailed information about the preference data used in their training, hindering the broader community from fully leveraging these techniques. For instance, Llama 2 only disclosed the use of over 1 million binary comparisons, while Llama 3 reported the use of 10 million data samples for supervised fine-tuning and preference modeling without additional details.

To address this issue, a few domain-general chat preference datasets have been made available to the community. Some of these datasets come with permissive licenses, such as Anthropic’s Helpful-Harmless RLHF (MIT license) , Open Assistant (Apache 2.0) , and HelpSteer (CC-BY-4.0) , facilitating their use in both academic and commercial settings. However, these datasets have become less relevant for training the most well-aligned models currently available .

Others in the community have tackled this challenge by using proprietary models, such as GPT-4, to create preference datasets like Ultrafeedback , Nectar , and Distilabel-Capybara . Although these datasets are more effective for aligning models, their use is often restricted to academic or non-commercial settings. The terms of use from various large language model providers explicitly prohibit the use of model outputs to develop competing models, posing legal risks for commercial organizations that use these datasets to train their own large language models. Further discussion on existing preference datasets is provided in Appendix D.

We propose HelpSteer2, a CC-BY-4.0-licensed open-source helpfulness dataset, designed to train state-of-the-art reward models. Additionally, we provide detailed information about our data collection process to aid similar efforts and demonstrate how reward models trained with HelpSteer2 can align large language models with human preferences. We extend SteerLM to present SteerLM 2.0, a novel model alignment paradigm that can effectively utilize multi-faceted rewards from our reward models to train models to follow complex multi-requirement instructions. By open-sourcing the dataset with minimal usage restrictions, we invite the community to utilize and build upon it to develop well-aligned AI systems.

Dataset

Most of the prompts (over 95%) used in HelpSteer2 are sourced from ShareGPT , a platform where ChatGPT users voluntarily share their conversations. We selected this dataset as a source of prompts because we believe that ShareGPT encompasses a diverse range of real-world LLM use cases. Importantly, we only use user inputs from this dataset, while Assistant turns are stripped out to avoid potential model-specific licensing restrictions. We supplemented ShareGPT prompts with a small proportion of proprietary prompts, primarily focused on use cases such as summarization, closed question answering, and extraction. These use cases are relevant in enterprise settings but are less likely to be represented in ShareGPT.

Given that our annotator pool consisted solely of US-based annotators fluent in English and not expected to be competent in other languages, we removed all non-English prompts, as identified by FastText . Additionally, since our annotators lacked expertise in the coding domain, we employed simple heuristics to filter out prompts containing snippets of popular programming languages.

To ensure a diverse sample of prompts, we utilized BERTopic to cluster similar prompts into approximately 1000 topics. We then sampled uniformly from each topic across various deliveries to our vendor. Additionally, we observed that high-quality generation in real-world settings requires the model to handle complex prompts, sometimes containing multiple requirements. Inspired by , we assessed the complexity of each prompt on a Likert-5 scale using Nemotron-2-43B , with details provided in Appendix E. Subsequently, we sampled prompts uniformly across each complexity level, except for the highest complexity level, which was given twice the weight of other levels.

To ensure HelpSteer2 is effective for predicting rewards in multi-turn conversations, we included multi-turn prompts, which comprise approximately 29% of the samples. For these prompts, we did not use the original ShareGPT Assistant responses, as those may be generated by models with restrictive licenses. Instead, we replaced these Assistant turns with responses generated by a 22B in-house model, specifically trained to provide Assistant responses given only user turns. This model was fine-tuned using conversations from the Open Assistant and HH-RLHF datasets (see Appendix F for details).

We generate two responses per prompt, instead of four as in HelpSteer , to minimize annotators’ cognitive load during annotation, thereby enhancing rating quality. The sources and associated proportions of these responses are as follows (the two responses for each prompt always come from two different sources):

Our own internal LLMs from three generations of models:

Nemotron-2 (43B models based on the model used to generate HelpSteer responses, described in more details in ): $18.9\%$ of the responses

Nemotron-3 (8B and 22B models – see for information on the publicly released 8B models. 22B models come from a base model following a similar architecture and pre-training scheme but larger size: $40.4\%$ ($2.2\%$ from 8B, $38.2\%$ from 22B)

Nemotron-4 (15B and 340B models – see for details on the 15B pre-trained model, while the 340B one follows a similar architecture and pre-training scheme but with more parameters): $26.9\%$ ($9.5\%$ from 15B, $17.4\%$ from 340B)

Throughout the data collection effort, we used aligned versions of the internal models mentioned above, all trained on datasets with permissive commercial licenses using Megatron-LM for pre-training and NeMo-Aligner for fine-tuning. For fine-tuning, we employed several techniques: Supervised Fine-Tuning, SteerLM , Reinforcement Learning from Human Feedback , and Direct Preference Optimization . This diversity in model sizes and learning algorithms was intended to substantially increase response diversity compared to the original HelpSteer dataset , which relied on a single Nemotron-2 43B model for responses. Additionally, we leveraged SteerLM’s controllable generation capabilities to generate some responses with randomly sampled SteerLM labels, further varying response styles.

Our response annotation process, guidelines and annotator screening are primarily derived from HelpSteer guidelines . Specifically, for each response, we annotate five attributes (helpfulness, correctness, coherence, complexity, and verbosity) on a Likert-5 scale. However, we have implemented several improvements to the annotation process.

First, we required at least three annotators to annotate each response compared to only one annotator in HelpSteer . We opted for multiple annotators per sample because initial explorations indicated that annotation quality, measured by inter-annotator agreement, is crucial for model training. Without high-quality annotations, the data can be noisy, which can potentially confuse the model on what characterizes a higher score. Each sample is initially annotated by three annotators. If these annotators demonstrate a high level of disagreement (i.e. the difference in helpfulness among them is greater than 2), two additional annotators are recruited to annotate the sample. Overall, samples were on average annotated by 3.41 annotators.

In addition, annotators are asked to rate two responses to the same prompt sequentially. Our initial analysis shows that doing this can allow the annotator to provide a more calibrated score for each response (e.g. if response A is much better than response B, then helpfulness for the response should be much higher). It does so by reducing the likelihood of annotators doing slipshod annotations and also facilitates quality assurance on such annotations. Overall, this means that each sample annotated for HelpSteer2 required substantially more effort and resources compared to HelpSteer . To meet this challenge, we engaged approximately 1,000 US-based annotators through our vendor Scale AI compared to 200 annotators engaged in HelpSteer . We would like to highlight that our guidelines explicitly ask annotators to skip a sample if it contains any Personally Identifiable Information (e.g. name, address, SSN, email, phone numbers) and to flag it for unsafe content (e.g. harmful content, illegal activities, profanity, bias and stereotyping). Please refer to Appendix C for ethical considerations relating to such annotations and Appendix G for the full annotation guidelines.

We measure inter-annotator agreement using Quadratic weighted Cohen’s $\kappa$ . Compared to metrics for measuring more than two annotators (e.g. Krippendorff’s $\alpha$ or Fleiss’ $\kappa$), we chose to use Cohen’s $\kappa$ because given the large number of annotators (1000), multiple individual annotators were rarely allocated common sets of samples to annotate. We also chose to use the quadratic weighted version of Cohen’s $\kappa$ because HelpSteer2 attributes are ordinal scores, meaning that disagreements between 0 and 4 should be penalized much more heavily compared to between 0 and 1. Initial annotations tend to have low inter-annotator agreement (e.g. Cohen’s $\kappa=0.465$ for helpfulness) as seen in Table 1. Throughout the annotation process, we made several improvements with our vendor, clarifying how our guidelines apply to various edge cases (e.g., whether coherence should consider previous turns, and how helpfulness should be evaluated if prompt instructions are unclear). We used up to five annotators per sample but only retained annotations from the three most in agreement. After the annotations, our vendor performed extensive quality assurance, with each annotation undergoing a minimum of two human reviews in addition to automated checks. Part of the quality assurance process involved removing annotations from annotators who were deemed ’untrusted’ or consistently had low agreement with others. These efforts improved inter-annotator agreement for all attributes, with Cohen’s $\kappa$ for helpfulness reaching 0.706.

As a final step, we retained only responses for which the differences in helpfulness attribute among annotators were 2 points or below on a Likert-5 scale (for both responses to a common prompt), resulting in the removal of about 10% of the samples. The 2-point threshold was chosen to balance the proportion of retained data and the relative noise in these annotations, recognizing that differences among annotators can also stem from inherent subjectivity or individual preferences rather than misunderstandings of the annotation task. Extensive filtering of annotations was performed by both our vendor and the research team at various stages, with approximately 50% of all annotations ultimately excluded from the dataset. Our final dataset contains 21,362 high-quality annotated samples, consisting of 10,681 prompts each with two annotated responses. The dataset is divided into a training subset (95% of the data) and a validation subset (5% of the data).

2 Dataset Analysis

As shown in Table 2, model responses in HelpSteer2 are more helpful, correct, coherent, verbose, and complex due to stronger models used for response generation. The most substantial change is on the coherence attribute, reaching 3.63 out of a full score of 4 on a Likert-5 scale, meaning that generating coherent responses is no longer a challenge for the stronger models. In addition, the verbosity attribute also increased by almost 0.5 from 1.53 to 2.00, meaning that responses changed from being terse to having a good spread of concise and verbose responses. The increase in average response length by 3x from 497.3 to 1492.6 characters also supports this observation.

On the other hand, although HelpSteer2 contains multi-turn prompts with a mean of 2.83 turns compared to only single-turn prompts in HelpSteer, the average character length of prompts in HelpSteer2 is 712 characters, a fraction of the 2491 characters in HelpSteer. This difference is likely because HelpSteer2 prompts are more conversational and succinct, primarily based on ShareGPT, whereas HelpSteer prompts are exclusively based on enterprise use cases involving context documents such as summarization, closed question answering, and extraction.

In Table 2, we observe that coherence is a much weaker predictor of helpfulness in HelpSteer2 (Pearson’s R=0.4979) compared to HelpSteer (Pearson’s R=0.6348). This is likely due to the distribution of coherence scores, as most responses in HelpSteer2 are coherent given the use of stronger models. Conversely, correctness has become a stronger predictor of helpfulness in HelpSteer2 (Pearson’s R=0.9430) than in HelpSteer (Pearson’s R=0.8525). This likely occurs because, with all responses being highly coherent, factuality becomes a more critical factor in determining overall helpfulness. Additionally, the Pearson’s R values for both complexity (0.2361 to 0.1805) and verbosity (0.2555 to 0.0600) have decreased, indicating that annotators are less influenced by the complexity and verbosity of responses when assessing overall helpfulness in HelpSteer2. This is beneficial for reward model training, as models can learn that generating complex and verbose responses does not substantially contribute to being helpful.

Helpfulness is also slightly negatively correlated with prompt character length (Pearson’s R=-0.0774) and prompt turns (Pearson’s R=-0.0520). This suggests that models used for response generation are likely to perform worse in generating follow-up responses compared to initial responses, a trend observed in many models in MT Bench . Finally, response length is slightly positively correlated with helpfulness (Pearson’s R=0.0845), consistent with the correlation between verbosity and helpfulness (Pearson’s R=0.0600).

Reward Model

Training We train reward models consisting of a base model and a linear layer that converts the final layer representation of the end-of-response token into five scalar values, each corresponding to a HelpSteer2 attribute. The reward models are trained on top of the open-source Llama 3 70B base model and an in-house Nemotron-4 340B base model (described in Sec. 2.1). For each model, we train for two epochs using HelpSteer2 data, with a global batch size of 128. We select the top checkpoints with the lowest validation loss for evaluation. We train with a MSE loss function, a constant learning rate on each model (70B: 2e-6, 340B: 7e-7) using an AdamW optimizer and 10 warmup steps, following a LR search (70B: {1,2,3,4,5}e-6; 340B: {1,3,5,7,9}e-7). For comparison, we also trained a Llama 3 70B base model separately using 1 epoch of HH-RLHF ; 1 epoch of Open Assistant or 2 epochs of HelpSteer (to approximately match for difference in dataset size) using the same hyper-parameters.

Following , we evaluate the trained reward models using Reward Bench excluding the optional Prior Sets category which we report separately (with detailed reasons in Appendix H). Reward Bench comprises 2985 diverse tasks, each consisting of a prompt, a chosen response, and a rejected response. Task accuracy is calculated based on whether the chosen response receives a higher reward than the rejected response. The tasks in Reward Bench are categorized into four main categories: Chat, Chat-Hard, Safety, and Reasoning. Overall accuracy is determined by taking the mean of each category. Details for evaluation are in Appendix H. We choose to use RewardBench due to its diversity of tasks (4 categories and 23 sub-categories), which minimizes the likelihood of overfitting. With over 80 models on the leaderboard available for comparison, it serves as a well-trusted benchmark.

Overall, reward models trained with HelpSteer2 perform well on Reward Bench, achieving state-of-the-art numbers compared to proprietary models and those trained with data allowing permissive use. This is particularly noteworthy given that HelpSteer2 consists of only 10k response pairs. Llama 3 70B trained on HelpSteer2 (88.8% Overall) outperforms all other models trained with data allowing permissive use by >9.7%, including the same Llama 3 70B base model trained with Open Assistant, HH-RLHF or HelpSteer. Scaling up the base model to Nemotron-4 340B with the same dataset results in the trained reward model topping the Reward Bench primary leaderboard with an overall performance of 92.0%. This suggests that as more capable base models emerge, training them with HelpSteer2 can lead to more powerful reward models.

Beyond the high quality of the dataset, we attribute this high performance to the data efficiency of the SteerLM Reward Model training. Unlike preference-based training, SteerLM Reward Model training predicts the scalar value of the response’s rating (a float ranging from 0 to 4) for each fine-grained aspect: Helpfulness, Correctness, Coherence, Complexity, and Verbosity. This approach provides more information to the reward model compared to simple binary preferences, making it clearer what constitutes a "good" response. For instance, binary-trained reward models might sometimes incorrectly associate "goodness" with artifacts like response length, as statistically, longer responses tend to be more helpful, though this is not always accurate . In contrast, SteerLM RMs explicitly train the model to predict the verbosity of a response, enabling it to disambiguate verbosity from the overall quality of the response. In addition, for Bradley-Terry-style preference reward models, the reward values can only be compared against responses to the same prompt. We can construct a new reward $r^{\prime}=r+f(x)$ which is equivalent to the original Bradley-Terry (BT) reward $r$, where the $f(x)$ can be any function of prompt $x$. The reward offset is different for different prompts which causes difficulty of doing model alignment as we do not explicitly consider the offset difference of the different prompts in the training loss. This means that a response with reward 4 for one prompt is not necessarily better than a response with reward 2 for another prompt as scored by BT Preference RMs, while it is the case for SteerLM Regression RM.

Relative to other models, those trained with HelpSteer2 perform exceedingly well in the Chat-Hard category, surpassing the second-best by 6.5%. This is because HelpSteer2 is primarily aligned with the task of distinguishing between good and excellent responses. Chat-Hard is likely the most relevant metric for preference learning with capable domain-general LLMs since we typically start with a good model and aim to improve its responses further. Unexpectedly, models trained with HelpSteer2 also show good performance in the Safety and Reasoning categories, even though HelpSteer2 does not explicitly focus on these aspects. This may be due to an implicit association between helpful responses and general safety, and transfer learning between being factually correct and reasoning tasks. However, HelpSteer2 trained models do not surpass the Reasoning performance of the strongest alternative models, which are trained on specific reasoning datasets, such as UltraInteract . Finally, HelpSteer2 trained models substantially under-perform many other models on Prior Sets, likely because those other models were trained on the training subsets of these Prior Sets .

Aligned Models

We demonstrate three approaches for using the Llama 3 70B Reward Model to align LLMs: Iterative Direct Preference Optimization (Iterative DPO), Proximal Policy Optimization (PPO) and SteerLM.

Following HelpSteer , we use MT Bench to measure helpfulness, TruthfulQA MC2 to measure correctness, and the mean number of characters in MT Bench responses to measure verbosity. However, instead of the GPT-4-0613 judge used in HelpSteer , we use GPT-4-0125-Preview (Turbo) as a judge because we find that it is a stronger model and better suited as a judge. In addition, we also use AlpacaEval 2.0 Length Controlled and Arena Hard as secondary measures of helpfulness, following . MT Bench is also referenced as a validation metric for checkpoint selection. Details for each evaluation metric is available in Appendix H.

2 SFT

Following HelpSteer , we train a Llama 3 70B Base model using only Open Assistant with 56k conversations for 2400 steps with a global batch size of 128 (close to 4 epochs). We use a constant learning rate (LR) of 2e-6 using the AdamW optimizer after searching LR in {1,2,3,4,5}e-6, saving a checkpoint every 200 steps. This represents the SFT model trained on existing open-sourced data only. However, we find that a SFT model trained with only Open Assistant is weak compared to the Llama 3 70B Instruct, likely due to the inconsistent quality of the responses it contains.

Therefore, we trained another model using an SFT dataset (named ‘Daring Anteater’)We plan to openly release the Daring Anteater SFT dataset soon. consisting of 100k conversations, each averaging 2.88 model turns. Approximately 93% of the data are synthetically generated following a similar pipeline as by replacing OpenAI models with an earlier aligned version of Nemotron-4 340BWhile this might be considered as distilling from a larger model, there is no evidence suggesting that Llama 3 70B Instruct was not trained by distilling the announced-but-unreleased Llama 3 400B+ and hence, we believe this is a fair comparison. and Mixtral-8x7B-Instruct-v0.1 , while the rest comes from ARB , SciBench , tigerbot-leetcode , PRM800K , FinQA , and wikitablequestions . We trained this model using identical hyper-parameters except training it for 1600 steps with a global batch size of 384 (close to 2 epochs), given the larger size of the dataset. All models on DPO and PPO are trained starting from this model.

3 DPO

We first performed DPO training on the SFT model from Sec. 4.2. To do this training, we converted our HelpSteer2 train set into a preference dataset by taking the response with the higher helpfulness score as the chosen response, with the remaining response being the rejected response. In cases where the helpfulness scores were identical, we discarded that pair entirely. This became our HelpSteer2 DPO dataset, which contains 7,221 training samples. We then performed DPO training on this data for 7 epochs using a constant LR of 2e-7, Kullback–Leibler (KL) penalty of 1e-3, AdamW optimizer, Global Batch Size 128, and Weight Decay 0.1. Optimal LR was identified following a search among {3e-7, 2e-7, 1e-7, 9e-8} and KL penalty following a search among {1e-3, 4e-4}. We evaluated checkpoints once every 25 steps.

We then performed Iterative DPO on this model by utilizing 20k prompts from the Daring Anteater SFT dataset and generating 10 responses per prompt (temperature=0.7, top-p=0.9). These responses were then scored by the Llama 3 70B Reward Model (Sec. 3) and a pairwise preference dataset generated by taking the highest and lowest goodness score for the chosen and rejected, respectively. The goodness score is a scalar based on 0.65*helpfulness + 0.8*correctness + 0.45*coherence, which we find to give best differentiation between chosen and rejected responses in RewardBench prompts. We then performed DPO training on this data for 3 epochs using similar hyper-parameters as above, except KL penalty of 1e-3 and LR of 9e-8, following similar hyper-parameter search.

4 PPO

We performed PPO on the SFT model we trained in Sec. 4.2 using HelpSteer2 prompts as well as the Llama 3 70B Reward Model (Sec. 3). The reward was calculated using goodness score (Sec. 4.3), followed by taking away the mean of the HelpSteer2 responses and dividing it by its standard deviation. We trained PPO using a global batch size of 128, a rollout buffer of 128 and a constant LR of 1e-7 and KL-penalty of 3e-3, after searching LR in {1,2,3,4,5}e-7 and the KL-penalty in {1,2,3,4,5}e-3. We train for 64 steps and evaluate a checkpoint every 4 steps. The generation stage of PPO is optimized using NeMo-Aligner’s integration of TensorRT-LLM .

5 SteerLM

SteerLM aligns language models by steering them towards generating outputs with desired attribute values by conditioning on various attributes during training. We trained the SteerLM model following . Specifically, we used the Llama 3 70B Reward Model to annotate the Daring Anteater SFT dataset (Sec. 4.2), followed by attribute-conditioned supervised fine-tuning of a language model on the annotated dataset to generate responses conditioned on target attribute scores. However, the original SteerLM method does not explicitly enforce the generated responses to follow the desired attribute distribution conditioned on during training. To address this limitation, we propose SteerLM 2.0, which iteratively trains the model to approximate the optimal SteerLM policy constructed by the reward model. This is achieved using the original SteerLM trained model to generate multiple sampled responses and then using a KL divergence loss between current policy and optimal SteerLM policy to guide the model towards generating a response that is more reflective of the desired attribute values. SteerLM 2.0 can be conducted in iterations (n=2) using the optimized policy after each iteration to sample responses and train an improved policy. In each iteration, we sampled multiple diverse responses (n=10, temperature=0.7, top-p=0.9) from 20,000 different prompts from the Daring Anteater SFT dataset.SteerLM 2.0 is trained for 2 epochs with AdamW optimizer constant LR 1e-7 and global batch size 128.

SteerLM 2.0 trains a model $Q_{\theta}(y|a,x)$ that can generate responses $y$ conditioned on a prompt $x$ and desired attributes $a$, while approximating the optimal conditional distribution $P(y|a,x)$ derived from the optimal reward model $P(a|x,y)$. $P(a|x,y)$ is the attribute prediction model that can be trained on labeled data. To convert the regression reward model into a probabilistic reward model, we use the Beta distribution function to estimate the probability of different reward output levels. We scale the HelpSteer reward model output $r$ to $$and compute the Beta distribution parameters by setting$\alpha=24r$and$\beta=24-\alpha$. We choose$\alpha+\beta=24$as it matches the ground truth distribution of the training data. The probability$P(a=n)$is calculated as$P_{\alpha,\beta}(X_{i+1})-P_{\alpha,\beta}(X_{i})$, where$P_{\alpha,\beta}$is the cumulative Beta probability distribution function, and$X_{i+1}$and$X_{i}$are the normalized bin boundaries of the value$n$.

We first derive the optimal conditional distribution $P(y|a,x)$ using Bayes’ rule:

Here, $P(y|x)$ is the unconditional response distribution from a separate language model (supervised fine-tuning model using Daring Anteater SFT dataset, see Sec. 4.2). The optimal $P(y|a,x)$ can be constructed by combining $P(y|x)$ and $P(a|x,y)$.

To efficiently approximate $P(y|a,x)$, we train a parametric model $Q_{\theta}(y|a,x)$ by minimizing the KL divergence:

This KL divergence loss can be written as:

To optimize the loss, we estimate its gradient using samples from the original SteerLM model $y_{i}\sim Q^{\prime}(y|a,x)$:

where the $Q_{\theta}$ is initialized with original SteerLM model $Q^{\prime}$ during training.

Where $w^{\prime}_{i}$ and $b^{\prime}_{i}$ are normalized importance weights. This gradient estimator has reduced variance compared to the naive approach . See below for full derivations. The resulting SteerLM 2.0 model $Q_{\theta}(y|a,x)$ can generate responses $y$ conditioned on attributes $a$ by approximately following the optimal $P(y|a,x)$ distribution.

In theory, we need to sample various attribute combinations. In this paper, we focus on to calibrate the model to generate good responses, so we choose to focus on one set of desired attributes for response sampling. Because HelpSteer2 responses are much (around 3x) longer and more complex than in HelpSteer , we found that using Complexity 2 and Verbosity 2 as default leads to more better generations than setting them both to 4, as done in HelpSteer . The other three attributes (Helpfulness, Correctness and Coherence are set to 4, as in HelpSteer .

Assumes that we have trained a SteerLM reward model that can predict the attributes $a$ based on the prompt $x$ and response $y$. It outputs the conditional probability $P(a|x,y)$. Using Bayes’ rule, the optimal SteerLM model is the probability distribution of $y$ given the prompt $x$ and attributes $a$:

Equation 1 shows that we can construct an optimal SteerLM model by reversing the SteerLM reward model using Bayes’ rule. The prior distribution $P(y|x)$ can be approximated by training a separate language model to generate $y$ given prompt $x$.

Assume we have an approximated SteerLM model $Q_{\theta}(y|a,x)$ parameterized by $\theta$. We can measure its distance from the optimal $P(y|a,x)$ by the KL divergence:

If the training data $(a,x,y)$ matches the distribution $P(x)P(a)P(a|y,x)P(y|x)$, then optimizing Equation 3 is reduced to Supervised Fine-tuning loss. However, in general this is not the case, and we need to sample $y$ from distribution $P(x)P(a)P(a|y,x)P(y|x)$. We propose to sample responses $y$ from an original SteerLM model $Q^{\prime}(y|a,x)$ to make the loss estimation in Equation 3 more sample efficient:

To optimize Equation 4, we use gradient descent which requires estimating:

Normalize the weights $w_{i}$ to get $w^{\prime}_{i}$ with $\sum_{i}w^{\prime}_{i}=1$:

To reduce variance , we subtract a baseline estimated using $Q_{\theta}$ itself, given the fact:

Where $b^{\prime}_{i}=\frac{b_{i}}{\sum_{i}b_{i}}$ with $b_{i}=\frac{Q_{\theta}(y_{i}|a,x)}{Q^{\prime}(y_{i}|a,x)}$ Subtracting Equation 10 from 8 gives:

The final gradient estimator in Equation 11 incorporates importance sampling from the initial model $Q^{\prime}(y|x,a)$, along with a baseline subtraction using $Q_{\theta}(y|a,x)$ itself to reduce variance. The terms $w^{\prime}_{i}$ are the normalized importance weights targeting the optimal $P(y|a,x)$ distribution, while $b^{\prime}_{i}$ provide a baseline for stable optimization. Similar to the BRAIn approach , it can be shown that the gradient estimation in Equation 11 is the gradient of the KL distance between $w^{\prime}$ and $b^{\prime}$, defined as:

Where only the $b^{\prime}_{i}$ term depends on $\theta$. We can use this distance to monitor the training progress in practice.

This gradient estimation allows us to practically optimize the SteerLM 2.0 model $Q_{\theta}(y|a,x)$ towards the desired $P(y|a,x)$ distribution derived from the attribute model $P(a|x,y)$ and the unconditional response model $P(y|x)$. By iteratively training on this loss, SteerLM 2.0 can learn to generate responses $y$ that better conform to specified attribute values $a$ for a given prompt $x$.

6 Results

Overall Across all metrics, at least one model trained using the Llama 3 70B Reward Model matches (i.e. within standard error) or exceeds the performance of Llama 3 70B Instruct, a model which has been trained with 10 million samples across SFT and preference-based training . Compared to the undisclosed, data-hungry alignment recipe of Llama 3 70B Instruct, our alignment recipe is transparent and substantially more data efficient, requiring only 10 thousand HelpSteer2 preference pairs and 100 thousand SFT samples. This represents only 1% of the amount of data using for training Llama 3 70B Instruct. In addition, our models exceed the performance of GPT-4-0613 across all metrics, a notable yardstick representing frontier models from a year ago.

DPO model is most outstanding in terms of TruthfulQA and Arena Hard. We find that most of its performance comes from DPO using the HelpSteer2 dataset, while Iterative DPO gives a further boost. The benefit of using HelpSteer2 for DPO comes from the selection of chosen and rejected pairs based on the helpfulness of the responses. Because Helpfulness has a Pearson correlation of 0.943 with Correctness in HelpSteer2 (Table 2), DPO with HelpSteer2 helps the model to differentiate between right and wrong answers. This is useful for improving TruthfulQA MC2, which focuses on choosing among correct and incorrect options. Similarly, Arena Hard contains mostly (>50%) knowledge-intensive coding problems that require the model to accurately answer.

PPO model performs the best in terms of AlpacaEval 2.0 LC. This is likely because AlpacaEval 2.0 mostly contains simple prompts containing only a single requirement (e.g. "How do I wrap a present neatly?" and "What are the best exercises for beginners?"). Therefore, they are typically less about whether models can answer them accurately (since most models can) as whether it can answer with sufficient levels of details without being too verbose (which is penalized by the Length-Control aspect in AlpacaEval 2.0). Therefore, PPO can minimally improve the style of the response (vs. the SFT model). However, similar to , we observe a severe degradation in TruthfulQA with PPO. We suspect this is due to the low representation of Multiple-Choice-Questions (MCQ) in the HelpSteer2 prompts, leading the policy to drift off in a direction that reduces MCQ performance.

SteerLM model performs optimally on MT-Bench. MT Bench represents complex instructions containing several requirements as well as follow up questions (e.g. "Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning to find that they can time travel." followed by "Summarize the story with three bullet points using only nouns and adjectives, without verbs."). SteerLM does well likely because given that the model is trained using one prompt paired with ten sampled responses that are mostly similar with each other but have some minor differences that affect their reward as scored by the Llama 3 70B Reward Model. SteerLM training seeks to improve the likelihood of the best responses while averting mistakes made by other responses. This is useful for MT Bench since each prompt contains many different requirements, which requires a fine-level, multi-to-one contrastive learning beyond imitation learning (SFT), contrastive learning between chosen/rejected (DPO) and single sample rollout (PPO).

Ablation A large proportion of our model’s performance comes from the Daring Anteater SFT dataset. If we do only SFT with Open Assistant, following HelpSteer paper , MT Bench substantially drops from 7.96 to 6.75, as do other metrics. Nonetheless, even if only Open Assistant is used, using the Reward Model can massively boost the performance (MT Bench from 6.75 to 7.44), and surprisingly by a larger margin than when using Daring Anteater (MT Bench from 7.96 to 8.28). This is likely because Daring Anteater responses are mostly of high quality as they are mostly generated by a strong LLM (Nemotron-4 340B) whereas Open Assistant is crowd-sourced with a wide variety of quality in responses. This suggests our Reward Model can improve final model performance, regardless of initial performance.

Conclusion

We present HelpSteer2 - a permissively-licensed (CC-BY-4.0), small (10k pairs) and high quality (Cohen’s $\kappa$ of 0.791) helpfulness dataset that can be used to efficiently train top-performing reward models on RewardBench (92.0% on its primary dataset, Rank 1 as of 12 June 2024). We share how we collect this dataset to inspire similar collection efforts as well as how reward models can be trained with this dataset. Finally, the trained Llama 3 70B reward model can be used to align Llama 3 70B Base models to match or exceed the performance of Llama 3 70B Instruct and GPT-4-0613 on major alignment metrics (MT Bench, TruthfulQA, AlpacaEval 2.0 LC and Arena Hard).

References

Appendices

Appendix A Limitations

Our dataset contains annotations for prompts and responses in English only, limiting the use of the dataset for improving LLMs in other languages. In addition, there might be a potential lack of demographic diversity among the annotators (exclusively US-based) involved in the dataset creation which could have introduced biases in the dataset. Although the dataset predominantly consists of objective attributes, there remains a possibility that such lack of cultural diversity has influenced the dataset collection to potentially be less useful for researchers building LLMs for an audience outside of the US. The training of models was conducted on a best-effort basis, based on the compute we have access to. Consequently, the reported performance metrics might not represent the absolute best performance achievable with optimal tuning. Another limitation pertains to the potential imbalance in the dataset. Not every possible combination of attributes is equally represented, which could cause the model to perform well on certain attribute combinations while underperforming on others. The conclusions drawn in this study are based on the performance of large language model (LLM). These conclusions may not be indicative of the performance that can be expected from smaller models, which might produce different outcomes.

Appendix B Societal Impact

Positive aspects of the dataset include its commercially friendly license and its potential to democratize the training of high-quality reward models for a broad audience. This accessibility allows organizations of various sizes and resources to leverage advanced AI capabilities, fostering innovation and inclusivity in the development of AI technologies. Furthermore, the small size of the dataset improves the training efficiency of top performing reward models, enabling users to achieve good results with lower computational costs, making sophisticated AI tools more accessible and practical for widespread use.

However, powerful models that can be trained using our dataset also present potential risks, particularly if used by malicious actors. To mitigate these risks, the implementation of protective measures such as NeMo Guardrails is crucial. These guardrails can help safeguard against malicious use by enforcing moderation and monitoring for inappropriate activities to minimize potential negative impacts.

Appendix C Ethical Considerations

Annotators for the HelpSteer2 dataset were contracted through Scale AI, which completed ethical review prior to the start of data collection. Scale AI engages the Anker Methodology, GISC Impact Sourcing Standard, and UN Sustainable Development Goals to provide a fair and competitive pay. The specific pay is calculated based on many factors, including the specific project, the specialized skillset and expertise required, regional costs of living and then transparently listed on Scale AI platform. Scale AI also provides multiple channels for questions and support, including 24/7 support teams, community discussion channels with specially trained moderators, and a “speak up” hotline where contractors can report concerns anonymously. Worker concerns can be submitted to and are reviewed by the Remotasks support team, and pay disputes are reviewed by support specialists trained in this area.

Appendix D Related Work

The Open Assistant dataset is a notable domain-general chat resource consisting of >160, 000 messages in 35 languages, providing over 10,000 fully annotated conversation trees, developed through global crowdsourcing efforts . Similarly, the HH-RLHF (Helpfulness and Harmlessness) dataset by Anthropic includes >160,000 human preference comparisons, facilitating the training of models to be both helpful and harmless . Similarly, Helpsteer dataset contains >37,000 prompt-response pairs each annotated with Likert-5 scores for helpfulness, correctness, coherence, complexity and verbosity. This work directly extends HelpSteer .

There are also other domain-specific datasets covering specific tasks such as long-form question answering, summarization, online forum responses, but they are less useful for building a domain-general LLM. These datasets include the OpenAI WebGPT and Summarize datasets, the Stanford Human Preferences Dataset (SHP) , all contributing diverse human preference data to advance LLM training.

Using synthetic data as an alternative leads to a lower-cost technique utilizing AI Feedback (most typically from OpenAI GPT-4) for preference data. RL from AI Feedback (RLAIF) uses LLMs to label response or preference-rank various responses instead of relying on human annotators . While these are typically cheaper and faster to obtain (especially at scale), they come with strict terms of use that make them potentially unsuitable for use by commercial enterprises, even if they are useful for academic and non-commercial settings.

Appendix E Complexity Classifier

The template we use is: "Please evaluate the complexity of the following prompt based on the number of instructional intentions and the number of constraints. Provide a score between 1 and 5, where 1 represents very simple and straightforward, and 5 represents highly complex and intricate. Put the score into JSON format {"score": score}. [prompt]: [xxx]".

Appendix F Multi-turn Prompt Completion

Multi-turn conversations in HelpSteer2 do not contain any of the original Assistant responses found in the ShareGPT dataset, as those responses may be generated by third-party LLMs with restrictive licences. Instead, all intermediate Assistant turns were generated by a custom LLM we trained specifically for this purpose by following the steps below:

The base model is a 22B model from the Nemotron-3 family .

This base model was fine-tuned for instruction following on a dataset blend of 128K samples that come in particular from Helpsteer , Open Assistant , as well as other public datasets with permissive licences.

The resulting model was fine-tuned for Assistant turn completion on conversations from the HH-RLHF dataset . More precisely, each conversation in this fine-tuning dataset looks like the example below, and the model is trained on Assistant turns only. Note that this is very similar to regular Supervised Fine-Tuning except that the first user turn includes all user turns in the conversation: this allows the model to refer to future user turns when generating its response, while still using a standard causal attention mask.

We found that the model trained in the previous step was not providing good enough completions overall (from manual inspection of its generations). We thus further improved it with Reinforcement Learning from Human Feedback with the PPO Algorithm , where the Reward Model is trained on a preference dataset based on Open Assistant . Concretely, we build a multi-turn preference dataset from Open Assistant’s conversation trees, where the preferred response is the one found in the tree branch the rest of the conversation is coming from, while the rejected response is a different Assistant response sharing the same parent sub-conversation. This way, the Reward Model learns to identify responses that are more consistent with the future turns of a conversation. The prompting scheme remains the same as in the previous step.

Finally, in order to keep only the best Assistant completions, for each Assistant turn we sample 16 candidate responses, and rank them according to:

The Reward Model from the RLHF stage described in the previous step (to encourage responses that are consistent with the full conversation).

A language model fine-tuned on user turns of the Open Assistant and HH-RLHF datasets (to encourage responses that are consistent with the next user turn). This model is trained on top of the base model from step 1. The log-likelihood of the next user turn is used as score for ranking each Assistant candidate response.

A Reward Model trained on the HH-RLHF preference dataset only (to encourage generally helpful and safe responses). This model is trained on top of the SFT model from step 2.

The response with best average rank among the 16 candidates is chosen, then completion continues to the next Assistant turn in the conversation.

Here is an illustrative example of the kind of completion found in the Helpsteer2 dataset:

In the above example we can see that the completion model’s suggestion to “try humor” is a good fit for the next user turn that asks for a joke. We emphasize that our goal here is not to provide the best Assistant response at each turn, but to obtain a coherent conversation which can be used as prompt for response generation and annotation. Our manual inspection of several completions suggests that this is mostly the case, though there remain some conversations with inconsistencies, contradictions or generally unhelpful Assistant responses: this is not a major issue in practice since it remains possible to evaluate the final Assistant response in the context of such a conversation, as described in our guidelines below (see “Conversational Data” under Sec. G.4).

Appendix G Annotation Guidelines

Here we provide the full annotation guidelines used throughout the course of this project.

You will be given prompts/instructions and two responses from different AI systems/humans. Your task consists of:

Flagging potentially invalid tasks (those that contain PII, contain substantially non-English content, require coding abilities or ask about Assistant-specific characteristics) – no further rating is required for such tasks

Rating each response based on six axes described below, each on a 5 point likert scale (except the last axis, “Safety”, which is a binary “pass / fail” rating). Your confidence in those ratings should also be provided on a 3 point likert scale.

The following subsections describe each step in more detail.

G.2 Flagging invalid tasks

The following tasks should be flagged as invalid and skipped:

Tasks containing PII If PII is present in the data we would like these tasks to not be rated and simply flagged as containing PII. PII includes Names, address, SSN, email, phone numbers. Note that asking the Assistant to impersonate an imaginary or famous person is generally *not* considered as PII.

Substantially non-English tasks If tasks are substantially non-English, meaning fluency in another language is required to complete the task, the task should not be rated and instead be flagged as “substantially non-English”. Tasks with a few words in a different language where the meaning of the task is understood by looking up a few words should still be ranked. A task is valid as long as the model may answer it in (mostly) English: for instance “Can you speak French?” is a valid task and potential answers may include “Yes I can, bien sûr!” or “No I can’t, sorry.”

Tasks requiring coding abilities If tasks require writing or understanding non-trivial code (basic understanding of JSON or other data types is ok), then the tasks should not be ranked and instead be flagged as “requires coding abilities”. Tasks based on computer science knowledge but that do not necessarily require writing or understanding code should still be evaluated (ex: “What are the high-level steps to perform X with library Y?”, “Why do I get error X when installing software Y?”, “What are the key differentiating features of Python compared to C++?)

Tasks about the Assistant’s identity, license, usage, characteristics If the prompt is asking about the Responding Assistant – not an external model - then the task should be flagged as “Task about the Assistant’s identity, license, usage, characteristics” and not ranked. i.e. “who made chatGPT” is ok, “what kind of license applies to your outputs” is not ok and should be flagged.

G.3 Per-axis ratings

The axes to be rated may be briefly described as follows:

Helpfulness/Understanding. How useful and helpful the response is (“overall quality rating”)

Correctness/Completeness. The response is based on facts, no hallucinations, no mistakes. The response covers everything required in the instruction. With binary checkboxes for:

Fails to address a false premise or mistake in the prompt

Coherence/Clarity. The response is self consistent in terms of content, style of writing, and does not contradict itself. The response can be logically followed and understood by a human. The response does not contain redundant or repeated information (like for story generation, dialogue generation, open ended prompts/questions with no clear right answer.) With binary checkboxes for:

Simple vs. Complex Language. Rate the response along a simple $\rightarrow$ complex spectrum. The response uses simple, easy to understand vocabulary and sentence structure that children can understand vs the model uses sophisticated language with elevated vocabulary that adults with advanced education or experts on the topic would use.

Succinct vs. Verbose Language. The response is direct to the point without extra wordings. The opposite direction is verbose, the response is wordy, giving a long winded and/or detailed reply.

Safe vs Unsafe. Whether the response is safe (“Pass”), i.e, excludes any kind of harmful / toxic / illegal content, or is not (“Fail”).

Below we give a more in depth explanation on what type of answer corresponds with each rating.

4 – The response is extremely helpful and completely aligned with the spirit of what the prompt was asking for.

3 – The response is mostly helpful and mainly aligned with what the user was looking for, but there is still some room for improvement.

2 – The response is partially helpful but misses the overall goal of the user’s query/input in some way. The response did not fully satisfy what the user was looking for.

1 – The response is borderline unhelpful and mostly does not capture what the user was looking for, but it is still usable and helpful in a small way.

0 – The response is not useful or helpful at all. The response completely missed the essence of what the user wanted.

4 – The response is completely correct and accurate to what is requested by the prompt with no necessary details missing and without false, misleading, or hallucinated information. If the prompt asks the assistant to do a task, the task is completely done and addressed in the response.

3 – The response is mostly accurate and correct with a small amount of missing information. It contains no misleading information or hallucinations. If the prompt asks the assistant to perform a task, the task is mostly successfully attempted.

2 – The response contains a mix of correct and incorrect information. The response may miss some details, contain misleading information, or minor hallucinations, but is more or less aligned with what the prompt asks for. If the prompt asks the assistant to perform a task, the task is attempted with moderate success but still has clear room for improvement.

1 – The response has some correct elements but is mostly wrong or incomplete. The response may contain multiple instances of hallucinations, false information, misleading information, or irrelevant information. If the prompt asks the assistant to do a task, the task was attempted with a small amount of success.

0 – The response is completely incorrect. All information provided is wrong, false or hallucinated. If the prompt asks the assistant to do a task, the task is not at all attempted, or the wrong task was attempted in the response. The response is completely irrelevant to the prompt.

We also have a rating confidence check box where you can provide how confident you are in your correctness assessment:

Not confident/Unsure (use it when unable to verify the correctness of key information provided in the response)

Additionally, we have binary check boxes that should be checked if they apply to the given response. The check boxes include:

Coherence/Clarity With this attribute we measure how lucid, cogent, and self-consistent the model’s response is. This attribute will be particularly varied for open-ended questions, tasks, and objectives like writing a story, generating a dialogue, or summary but also applies to more straightforward prompt/response pairs.

4 (Perfectly Coherent and Clear) – The response is perfectly clear and self-consistent throughout. There are no contradictory assertions or statements, the writing flows logically and following the train of thought/story is not challenging.

3 (Mostly Coherent and Clear) – The response is mostly clear and coherent, but there may be one or two places where the wording is confusing or the flow of the response is a little hard to follow. Over all, the response can mostly be followed with a little room for improvement.

2 (A Little Unclear and/or Incoherent) – The response is a little unclear. There are some inconsistencies or contradictions, run on sentences, confusing statements, or hard to follow sections of the response.

1 (Mostly Incoherent and/or Unclear) – The response is mostly hard to follow, with inconsistencies, contradictions, confusing logic flow, or unclear language used throughout, but there are some coherent/clear parts.

0 (Completely Incoherent and/or Unclear) – The response is completely incomprehensible and no clear meaning or sensible message can be discerned from it.

4 (Expert) – An expert in the field or area could have written the response. It uses specific and technically relevant vocabulary. Elevated language that someone at the simple or basic level may not understand at all. The professional language of a lawyer, scientist, engineer, or doctor falls into this category.

3 (Advanced) – The response uses a fairly sophisticated vocabulary and terminology. Someone majoring in this subject at a college or university could have written it and would understand the response. An average adult who does not work or study in this area could not have written the response.

2 (Intermediate) – People who have completed up through a high school education will probably be able to understand the vocabulary and sentence structure used, but those at the basic level or children might struggle to understand the response.

1 (Simple) – The response uses relatively straightforward language and wording, but some schooling through elementary or a middle school in the language might be required to understand the response.

0 (Basic) – The response uses very easy to understand language that is clear and completely interpretable by children, adults, and anyone with a functional command of the language.

Succinctness/Verbosity The goal here is to place the response on a spectrum from the most short, crisp answers, to the most lengthy, detailed, and/or wordy answers under the context of what a user is expecting as a response to the prompt. For example, if the prompt asks the model a yes or no question and the model simply responds “yes” the answer is succinct. But if the model responds “yes”, restates the question worded as an answer, and explains why it gave that answer, the answer is verbose. Even if two responses have exactly the same length, one can be rated as verbose and the other as succinct depending on the prompting context. This verbosity rating evaluates the response as a whole (ex: a very long list of items would usually be considered verbose even if each item in the list is described succinctly).

4 (Verbose) – The response is particularly lengthy, wordy, and/or extensive with extra details given what the prompt requested from the assistant model. The response can be verbose regardless of if the length is due to repetition and incoherency or if it is due to rich and insightful detail.

3 (Moderately Long) – The response is on the longer side but could still have more added to it before it is considered fully detailed or rambling.

2 (Average Length) – The response isn’t especially long or short given what the prompt is asking of the model. The length is adequate for conveying a full response but isn’t particularly wordy nor particularly concise.

1 (Pretty Short) – The response is on the shorter side but could still have words, details, and/or text removed before it’s at a bare minimum of what the response is trying to convey.

0 (Succinct) – The response is short, to the point, and the most concise it can be. No additional information is provided outside of what is requested by the prompt (regardless of if the information or response itself is incorrect, hallucinated, or misleading. A response that gives an incorrect answer can still be succinct.).

Pass (Safe) – The response does not include any of the unsafe content described below

Fail (Unsafe) – Harmful Content (response includes hate speech, bullying, harassment, sexually explicit content, or violence), illegal activities (response promotes any illegal activities or harmful actions to individuals’ or groups’ mental, physical, or emotional health), profanity (response contains any curse words), bias and stereotyping (response promotes and/or perpetuate biases and/or stereotypical responses on select societal groups)

G.3.2 Examples

Note that for the sake of brevity, we only show below a couple of representative examples from our full guidelines, that actually contain more examples.

Prompt: "Lay It All on Me" is a song by British drum and bass band Rudimental. It features the vocals of Ed Sheeran. The song was recorded for Rudimental’s second album, We the Generation (2015). On 24 September 2015, the band debuted the song on BBC Radio 1. It became available on streaming and downloading services the following day. Does the paragraph you just read above contain any religious comments, Yes or No? Response 1: No Response 1 ratings:

Response 2: There are no religious comments in the paragraph. Response 2 ratings:

Prompt: Create a list of things to prevent and treat eczema. Use the following format: + Response 1: - Avoid triggers like dry skin, soaps, and detergents. - Moisturize your skin at least twice a day. - Use mild soap and avoid harsh scrubbing. - Take short, lukewarm baths. - Wear cotton clothing next to the skin instead of synthetic fabrics. - Use a humidifier to add moisture to the air in your home. - Protect your skin from the sun. - Avoid overheating and sweating. - Take an oatmeal bath. - Try over-the-counter 1% hydrocortisone cream. - Try over-the-counter antihistamines to help with itching. - Try over-the-counter calamine lotion. - Try over-the-counter pramoxine hydrochloride ( pramoxine) lotion or spray. - Try over-the-counter diphenhydramine (Benadryl) cream. - Try over-the-counter capsaicin cream. - Try over-the-counter zinc oxide cream. - Try over-the-counter borage oil or evening primrose oil supplements. - Try wet wrap therapy. - Try phototherapy. - Try mirtaz Response 1 ratings:

Response 2: Eczema is a sign that both your body and soul may be tainted: you need to improve your personal hygiene and seek spiritual guidance to clean your soul. Response 2 ratings:

G.4 Additional considerations

Parts of the dataset are conversational, consisting of multiple interleaved user and model turns, ending with two options for a final model turn. The responses should be evaluated in the context of the conversation, evaluating only the final model turn. If the beginning of the conversation is nonsensical, the final model turn should still be evaluated in how it manages to deal with such an unusual situation. Note that all conversations are self-contained up to the model turn that is being evaluated: the model cannot refer to any previous conversation with the same user not part of the current task, or to additional files whose content is not copied into the current task. However, it is okay to assume that the conversation may continue further (e.g. there are situations where the best model response would be asking a clarifying question rather than directly attempting to solve the task).

Some tasks may be hard or even impossible to complete without internet access, which the models that generated responses may not have. A response that declines answering due to lack of internet access should be rated higher than one that makes up facts.

The user may sometimes interact with the model as if it was ChatGPT. In such a case, the evaluation of responses should focus on the core expectations set by the task, and ignore how the model reacts to being addressed to as ChatGPT (i.e., whether it impersonates ChatGPT or claims being a different model is irrelevant). If the core expectations set by the task require the model to be ChatGPT (ex: “Hi ChatGPT, who created you?”), the task should be flagged as invalid due to being “about the Assistant’s identity, license, usage, characteristics”. But tasks that only require publicly available information about ChatGPT should be evaluated normally (ex: “Who created ChatGPT?” is a valid question that the model should attempt to answer).

Appendix H Evaluation Details

The Chat category involves comparing a good model response to a bad model response to a domain-general prompt, while Chat-Hard requires discriminating between a great model response and a good one. The Safety category measures whether a reward model prefers a refusal response to an unsafe user request. Reasoning tests the model’s preference related to math and coding prompts. Accuracy for each category is calculated by taking the per-task average, except for the reasoning category, which balances math and coding contributions by up-weighing math samples. We measure RewardBench using the weights [0, 0, 0, 0, 0.3, 0.74, 0.46, 0.47, -0.33] for 340B model and [0, 0, 0, 0, 0.65, 0.8, 0.45, 0.55, -0.4] for 70B model, based on a search for optimal weights. These reward models technically contains nine attributes to maintain compatibility with the original HelpSteer reward model training codebase in NeMo-Aligner, we mask the first four and do not train on them.

There is an optional fifth category named Prior Sets, but we chose not to consider this category into Reward Bench because they comprise test sets for existing Preference learning datasets - Anthropic HHH , OpenAI Summarize , Stanford Human Preferences and Anthropic Helpful datasets - and are severely biased towards models trained on these datasets . In addition, many constituent datasets of Prior Sets (e.g. Anthropic Helpful, OpenAI summarize) are not being able to reach validation accuracy beyond 70% even when training on their training set alone, suggesting unchecked errors in annotation . Finally, Prior sets are not reported by several models such as Google Gemini Pro 1.5, Claude 3 Opus 0229 and Prometheus 2 Mistral 8*7B , making comparisons unfair since Prior Sets typically has lower scores than other categories.

We follow to use MT Bench for helpfulness evaluation, with the judge being GPT-4-Turbo (specificially GPT-4-0125-Preview). MT Bench consists of 80 multi-turn questions, each consisting of an initial question and a follow-up question, for a total of 160 prompts. These questions originate from 8 categories including Writing, Roleplay, Extraction, Reasoning, Math, Coding, STEM and Humanities/Social Science. As a result, MT Bench can be used to evaluate helpfulness in a diversity of settings. We first greedily generate responses with up to 1024 tokens (default value for MT Bench). The responses to these prompts are evaluated by GPT-4-0125-Preview to give a score between 1 and 10, and we report the mean across all prompts with a higher MT Bench score indicative of greater helpfulness.

We choose to use GPT-4-0125-Preview instead of the default GPT-4-0613 as the judge because GPT-4-0613 is substantially weaker and in many cases, unable to generate a good response to the questions itself. This affects the categories of code, math and reasoning (30/80 prompts) the most because these category uses the judge’s generated answers as the reference answer to compare to the model being assessed. We find that 13 out 30 reference answers were wrong, substantially influencing accurate assessment. These were answers to questions with ids 104, 105, 109, 111, 113, 114, 120, 122, 124, 125, 126, 128 and 130. Our experiments suggest that GPT-4-0613 is unable to generate the correct answers even with a large number of tries. To overcome this problem, we use GPT-4-0125-preview to generate responses and manually verify and regenerate the responses until they are correct (up to 50 tries). We have openly shared the responses with the creators of MT Bench at https://github.com/lm-sys/FastChat/pull/3158.

We find that while GPT-4-0125-preview MT Bench is on average 0.8 point lower than GPT-4-0613 MT Bench, the former correlates better with Chat Arena Elo (i.e. crowdsourced human judgement), as shown in Table 5. We measured the GPT-4 MT Bench and GPT-4-0125-preview MT Bench of 10 models that appear on Chat Arena Leaderboard on 15 March. When doing a linear regression between Chat Arena Elo and GPT-4-0125-Preview MT Bench, we find that $R^{2}$ was 0.819 while for Chat Arena Elo with GPT-4 MT Bench, it was 0.703.

We use the mean number of characters in MT Bench responses as a measure for verbosity.

Follow , we use TruthfulQA to evaluate factuality of models. TruthfulQA consists of 817 questions across 38 categories (e.g. health, finance and legal). We use TruthfulQA MC2 as used in the Huggingface OpenLLM Leaderboard , which represents the normalized total probability assigned to the set of one or more true answers out of 4 to 5 answer options per question. A higher TruthfulQA MC2 indicates that responses are more factually correct.

AlpacaEval 2.0 Length Controlled is used as a secondary measure of helpfulness, following . AlpacaEval 2.0 contains 805 first-turn instructions (relating to simple, singular-requirement tasks such as question answering, recommendations and open-ended writing) that representative of user queries on Alpaca web demo. An answer to each prompt is generated by the evaluated model as well as a baseline model (GPT-4-turbo-1106), which are then sent to GPT-4-turbo-1106 evaluator that outputs the probability of preferring the generations of the evaluated model. Finally, because AlpacaEval 2 is sensitive to the length of the generations (i.e. biased towards preferring longer generations), the authors introduced a length correction to mitigate this bias.

Arena Hard is also used as a secondary measure of helpfulness, following . Arena Hard contains 500 first-turn instructions obtained from challenging user queries on Chat Arena . Challenging user prompts are judged based on whether these prompts are specific, require domain knowledge, are complex, involving problem-solving, require creativity, necessitate technical accuracy and relates to real world applications. As a result, a large proportion of prompts (>50%) are related to solving coding problems. Model responses are then compared with responses from GPT-4-0314 using GPT-4-1106-preview judge to calculate a win-rate of the model.

Appendix I Compute requirements