Training Language Models with Language Feedback at Scale

Introduction

Language Models (LMs) achieve strong performance across diverse NLP tasks, from summarization to question answering and dialog (Radford & Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020; Rae et al., 2021, inter alia). One of their key limitations, however, is that they generate text that violates human preferences, such as misinformation (Lin et al., 2021), offensive language (Gehman et al., 2020), and factually incorrect summaries (Stiennon et al., 2020). To alleviate such issues, existing methods train LMs to generate text that scores highly according to human preferences or a predictive model thereof (Ziegler et al., 2019; Stiennon et al., 2020; Nakano et al., 2021; Ouyang et al., 2022). These approaches learn from human feedback regarding which of two outputs is better. However, each comparison only conveys limited information about human preferences.

We propose an alternative approach that learns from language feedback, an information-rich and natural form of human feedback. We introduce Imitation learning from Language Feedback (ILF), a 3-step algorithm for learning from language feedback (Fig. 1). First, we generate multiple refinements of an LM-generated output, given the input, initial LM-generated output, and human-written feedback on the output. Second, we use an instruction-finetuned LM to choose the refinement that best incorporates the feedback. Third, we finetune the LM that generated the initial output on the chosen refinement given the input. In this way, we finetune an LM using language feedback; with the resulting model, we may then collect more feedback on its outputs and learn with the above refine-and-finetune approach. The algorithm’s pseudocode (Algorithm 1) and the corresponding graphical model are shown in Fig 2. ILF departs from prior work, which uses reinforcement learning (RL) (Ziegler et al., 2019; Stiennon et al., 2020, inter alia) or auxiliary losses (Stacey et al., 2021) and cannot be straightforwardly generalized to using free-form language feedback.

We analyze our approach both theoretically and empirically. We show that ILF can be viewed as Bayesian Inference, similar to RL with Human Feedback with KL penalties (Korbak et al., 2022). We then validate our algorithm on a carefully-controlled synthetic task of removing offensive words from a sentence with GPT-3-based models (Brown et al., 2020; Ouyang et al., 2022). We find that only the largest GPT-3-based models (175B parameters) accurately refine outputs. Using this insight, we use the largest GPT-3 models to test our algorithm on text summarization, following Stiennon et al. (2020). Our work extends our earlier unpublished results (Scheurer et al., 2022), showing that ILF improves LM-generated summaries monotonically with the amount of feedback provided, testing up to 5k samples. In all data regimes, ILF leads to comparable or better results to finetuning on human-written summaries, suggesting our approach is a strong alternative to supervised learning on human demonstrations. We also introduce an approach for learning from both language and comparison feedback by choosing the best-of-N samples from an ILF-trained model using a model trained with comparison feedback. The hybrid approach outperforms learning from each form of feedback alone, leading to summaries that human evaluators prefer over high-quality human reference summaries $\sim 50.8$% of the time. Our analysis shows that LM-generated refinements typically incorporate the feedback, especially when we use an LM to choose the refinement that best incorporates the feedback. In our concurrent paper (Chen et al., 2023), we show that ILF also achieves strong performance on code generation. Our results suggest that language feedback is a promising avenue for learning human preferences.

Methods

We now formulate the problem setting and describe our approach. We aim to generate improved outputs $x_{1}$ (e.g., high-quality summaries), according to human preferences, given language feedback $f$ on an initial model-generated output $x_{0}$, and a context $c$ (e.g., a source document). We tackle this problem by updating an LM $\pi_{\theta}$ based on evidence provided by language feedback.

Our goal is to sample a diverse set of high-quality outputs $x_{1}$ given a context $c$ (e.g., a summary of a document), where $c$ is drawn from the context distribution $p(c)$. We do so by fitting an autoregressive LM $\pi_{\theta}$ to approximate the ground-truth distribution $p_{c}^{*}(x_{1})$ which is proportional to the quality of $x_{1}$, measured by the reward function $R$. Fitting $\pi_{\theta}$ can be written down as minimizing the expected KL-divergence from the true distribution $p_{c}^{*}(x_{1})$ to $\pi_{\theta}$ over the context distribution $p(c)$:

Minimizing the objective in Eq. 1 equivalent to minimizing the cross-entropy loss (i.e., supervised learning):

It is intractable to compute this loss exactly for a number of reasons, including the exponential size of the space of $x_{1}$ as well as the intractability of computing the normalization constant of $p_{c}^{*}(x_{1})$. To avoid the first issue, we use Monte Carlo approximation sampling using a small set of samples drawn from $p_{c}^{*}$. Directly sampling from $p_{c}^{*}$ is however still intractable. We thus resort to using importance sampling with a proposal distribution $q_{c}(x_{1})$ that is simpler to sample:

To minimize the variance, we must design $q_{c}$ to be as close as possible to $p_{c}^{*}$. We achieve this goal by defining $q_{c}$ to incorporate human feedback that directly reflects the unknown reward function $R$, in the process of sampling. We do so by first drawing an initial output $x_{0}$ from a suboptimal LM $\pi_{\theta}$ given the context $c$. Second, we ask humans to rate $x_{0}$ and provide language feedback $f$ on the $(c,x_{0}),$ pair. Third, a refinement LM $\pi_{\psi}$ generates a refined output $x_{1}$ conditioned on $(c,x_{0},f)$. The proposal distribution, corresponding to this sampling procedure, can be written down as:

Let $x_{1}^{i},\dots,x_{1}^{N}$ be $N$ summaries sampled from $q_{c}(x_{1})$. Then, we can approximate the objective in Eq. 2 as:

where $\omega^{i}$ is the importance weight of the $i$-th sample from $q_{c}$. The importance weight $\omega^{i}$ is not computable as it is because we do not have access to $q_{c}$ other than being able to draw samples from it. We avoid this issue by assuming that $q_{c}(x_{1}^{i})$ is constant, implying that our samples are all equally good due to the high quality of human feedback. We then replace $R(x_{1}^{i}|c)$ in the definition of $p_{c}^{*}$ by $R(x_{1}^{i}|x_{0},f,c)$, as the quality is not dependent on the intermediate summary and feedback but can be more easily assessed with these quantities. This allows us to compute the unnormalized $p_{c}^{*}$, after which we use self-normalization to finally compute the above loss.

We implement $R$ by conditioning an instruction-finetuned LM on a binary question such as Does this new text $\left[x_{1}\right]$ incorporate the feedback $\left[f\right]$ provided on the initial text $\left[x_{0}\right]$? Answer Yes or No., where the label $y$ is either $y_{\text{good}}$ (“ Yes”) or $y_{\text{bad}}$ (“ No”). We use the probability of the positive answer $y_{\text{good}}$ as $R$, i.e. $R(x_{1}|x_{0},f,c)=\frac{p(y_{\text{good}}|\text{prompt})}{p(y_{\text{good}}|\text{prompt})+p(y_{\text{bad}}|\text{prompt})}$.

Finally, we use an extremely low temperature when computing $p_{c}^{*}$, i.e., $\beta\to\infty$. Due to self-normalization, this is equivalent to using only the best summary $x_{1}^{*}$ per context $c$ sampled from $q_{c}$ for computing the loss, resulting in the following, final objective:

Our objective of approximating the ground truth distribution $p_{c}^{*}(x_{1})$, which is proportional to the reward $R$ has clear connections to maximizing reward in RL. However, in RL, the goal is to find the best policy that maximizes the reward, whereas our algorithm results in a distribution of high-quality outputs $x_{1}$ given a document $c$, which allows us to draw a diverse set of outputs achieving a high reward. The broad diversity of high-quality outputs endows downstream users and systems with more control over which aspects they prefer and want to avoid. In App. A.1, we further provide an alternative derivation of ILF that follows variational inference and shows that ILF can also be understood as Bayesian Inference. This process involves updating an LM based on the evidence provided by language feedback. This different lense highlights the correspondence between ILF and RL with Human Feedback (Ziegler et al., 2019; Stiennon et al., 2020, inter alia), which was previously demonstrated to be equivalent to Bayesian inference (Korbak et al., 2022).

Can Language Models Use Feedback?

For our algorithm to work, LMs must be able to accurately incorporate feedback to generate refinements. Thus, we first validate the refinement step of our algorithm on a carefully-controlled synthetic task of removing specific offensive words from a given sentence. We examine how effectively various models incorporate feedback to determine what model to use for refining outputs.

We instruct an LM to refine an automatically-generated sentence with $\leq 10$ offensive words by removing $\leq 3$ specific words (see Appendix D for a detailed explanation and examples). In this experiment, we generate one output per sample with greedy decoding, i.e., we do not sample with best-of-$N$ . We evaluate how often the generated refinement exactly matches the target sentence, which we automatically generate. For our LMs, we use differently-sized GPT-3 models (Brown et al., 2020) and text-davinci-001, their instruction-finetuned (Feedback Made Easy or FeedME) counterparts (Ouyang et al., 2022; OpenAI, 2022b).Via the OpenAI API. We report all hyperparameters used in Appendix G. We report the mean and standard error for all results in our work.

Table 1 shows the results. We observe that only the largest GPT-3 and FeedME models (175B parameters) incorporate feedback in a non-negligible amount of time. Using this insight, we only use the 175B parameter models in the rest of our experiments. Specifically, we use FeedME, because it is an instruction-finetuned model.

Summarization from Language Feedback

Having established that large LMs can leverage language feedback, we now evaluate our algorithm on the real-world task of text summarization. In §4.1, we introduce a novel summarization dataset that we use to evaluate our algorithm, in §4.2, we explore different methods for ranking refinements and in §4.3, we use the best ranking method to learn from language feedback.

We evaluate the effectiveness of ILF on the task of text summarization using the TL;DR dataset (Völske et al., 2017), which consists of Reddit titles, posts, and their corresponding summaries. Stiennon et al. (2020) adapt this dataset and show that it is a more realistic task for evaluating summarization models compared to the commonly used CNN/DM dataset (Hermann et al., 2015). To ensure the quality of our dataset, we follow the same preprocessing steps as outlined in Stiennon et al. (2020) and extract a train dataset with 5000 samples, a development dataset with 200 samples, a validation dataset with 500 samples, and a test dataset with 698 samplesThe train and development datasets are taken from Stiennon et al. (2020)’s train dataset, and the validation and test set are taken from their test dataset.. We then hire experienced annotators through Surge AIhttps://surgehq.ai to create our language feedback dataset, which we open source along with our codeData: HuggingFace; Code: Github. For each sample, we first generate three summaries for each Reddit post using the instruction-finetuned model text-davinci-001 (FeedME) (Ouyang et al., 2022; OpenAI, 2022b). Two of these summaries are used for a binary comparison, in which annotators indicate their preference. The third summary serves as the initial output for which we solicit language feedback. This feedback should address the single most important shortcoming of the summary and can be related to coverage (how well the summary covers the important information in the post), accuracy (the factual accuracy of the summary), coherence (the coherence of the summary on its own), or other. We do not impose any restrictions on how the feedback should be written. In addition to providing feedback, annotators are also asked to write an ideal summary that is maximally 48 tokens long. The same crowd worker annotates all three tasks for a given sample. Overall the dataset collection and human evaluations cost 40K$. On selected samples of the binary comparison task, we achieve an author-annotator agreement of$81.0\%$and annotator-annotator agreement of$70.0\%$. The human summaries we collect are of excellent quality, as demonstrated in a human evaluation, where we compare our human-written summaries to the ones automatically extracted from Reddit (Völske et al., 2017) (also used as baselines in Stiennon et al. (2020); Scheurer et al. (2022)). We find that our human-written summaries are preferred$72.0\pm 3.2\%$ of the time, making them a much stronger baseline.

2 Comparing Refinement Ranking Methods

We condition FeedME on the initial summaries of our train dataset (generated with FeedME) and the human-written feedback and generate 5 refinements $x_{1}^{1},...,x_{1}^{5}$ using the instructions in App. J.1.

We chose a refinement with a scoring function $R$ that scores refinements for how effectively they incorporate feedback. For $R$ we use the instruction-finetuned LM FeedME and ask it whether a refinement is better than the initial summary (see §2 for more details). We then evaluate the probability that the refinement incorporates language feedback on the initial summary and is accordingly a high-quality summary, i.e., $p(y_{\text{good}}|\text{prompt})$. LMs are sensitive to the exact prompt used (Perez et al., 2021; Lu et al., 2021), so we write 5 different prompts (see App. J.2) and select the refinement with the highest average $p(y_{\text{good}}|\text{prompt})$ and call this method InstructRM Ensemble.

Previous work (Scheurer et al., 2022) use a contrastive pre-trained text-embedding function (Neelakantan et al., 2022) to embed the feedback $f$ and refinements $x_{1}^{1},...,x_{1}^{5}$ and select the refinement with the highest cosine similarity to the feedback. They use this scoring function because feedback would often describe what the ideal text should look like. This method is less general because it assumes that good refinements are semantically similar to the feedback, which is not necessarily the case for all tasks or forms of feedback.

We now evaluate the above ranking methods on the development dataset by calculating the fraction of times the refinement selected by a method is better than a randomly-selected refinement (“win rate”), according to a ranking given by human evaluators (see App. E for more details). The results, shown in Table 2, show that the embedding similarity selection does not outperform random selection, while most (4/5) InstructRM prompts do. While the embedding similarity worked well in previous work (Scheurer et al., 2022), it does not perform well on our dataset. We believe this is because the feedback we collect, written by many annotators, is much more diverse, while in Scheurer et al. (2022), the authors wrote the feedback themselves. InstructRM Ensemble has a win rate of $56.0\pm 3.0\%$ against random selection, demonstrating that an LM can evaluate its own output to some extent. Based on these results, we recommend using the InstructRM Ensemble approach, as it performs well and is less sensitive to the particular prompt. Throughout our paper, we use InstructRM Ensemble as our scoring function to select refinements and refer to our method of generating and selecting refinements as Refinement with Feedback + Best of N.

3 Comparing Feedback Learning Algorithms

In this section, we compare various algorithms for learning from language feedback, binary feedback, and normal supervised finetuning. We present an overview of each method and then provide the results of our evaluations.

For this evaluation, we use a single iteration of ILF to learn from language feedback. We finetune GPT3-175B (davinci) (Brown et al., 2020)FeedME cannot be finetuned via OpenAI’s API. to maximize the log-likelihood of the refinement given the input prompt (consisting of the Reddit title, and post), i.e., $\log p(x_{1}|\text{prompt})$, using the refinements generated with Refinement with Feedback + Best of N. For all our finetuning methods we add $\lambda\log p(\text{prompt})$ to the loss (Radford et al., 2018; OpenAI, 2022a), which maximizes the log-probability of the prompt. The prompt-loss weight $\lambda\in$ is chosen on our development dataset (see paragraph Finetuning on Human Summaries). The selected hyperparameters are detailed in App. G and the finetuning prompts in App. J.3.

Here we finetune GPT3-175B on the dataset of human-written summaries $x_{\text{human}}$, with the objective of maximizing the log-probability of human summaries given the input prompt (consisting of the Reddit title and post) with the additional loss term, i.e. $\log p(x_{\text{human}}|\text{prompt})+\lambda\log p(\text{prompt})$. To ensure the best performance of our finetuned models, we conduct thorough hyperparameter tuning on the human-written summary datasets of various sizes (100, 1K, 5K). The hyperparameters optimized include the number of training epochs, the prompt loss weight $\lambda$, and the learning rate multiplier, as detailed in the OpenAI documentation (OpenAI, 2022a). We use the perplexity of the predicted summaries on the development dataset to select the most effective hyperparameters. The selected hyperparameters are applied to all datasets, i.e., finetuning on refinements, initial summaries, and human-written summaries, with the same sample size. More details on hyperparameter tuning can be found in Appendix G.

We finetune GPT3-175B on the dataset of initial summaries (generated by FeedME). The objective is to maximize the log probability of the initial summary given the prompt (consisting of the Reddit title and post) with the additional loss term i.e. $\log p(x_{0}|\text{prompt})+\lambda\log p(\text{prompt})$. Details on hyperparameter tuning can be found in the paragraph Finetuning on Human Summaries and Appendix G.

We compare ILF against binary feedback as a baseline, the standard approach for learning from feedback. One way of learning from binary feedback is to train a reward model and use it to do best-of-$N$ sampling. We use best-of-N because it is often competitive with RL from human feedback (Nakano et al., 2021), a highly effective but more sophisticated approach (Stiennon et al., 2020; Ouyang et al., 2022). To train the RM, we finetune OPT-13B (OPT-RM) (Zhang et al., 2022) to classify whether a summary $x_{0}$ is high quality or not. To do so, we use the instruction Is the above an excellent summary of the given text? An excellent summary is coherent, accurate, concise, and detailed. Answer with Yes or No., where the label $y$ is either $y_{\text{good}}$ (“ Yes”) or $y_{\text{bad}}$ (“ No”). Given human labels on which of two summaries is preferred, we label the preferred summary with $y_{\text{good}}$ and the other summary with $y_{\text{bad}}$. We then finetune the LM to maximize $\log p(y|x_{0})+\lambda\log p(x_{0})$, where $\lambda\in$, chosen using the development dataset, and $y\in\{y_{\text{good}},y_{\text{bad}}\}$. Using the finetuned LM, we evaluate a given summary by computing $p(y_{\text{good}}|x_{0})$ and select the summary with the higher probability. We find that this approach leads to more accurate RMs than other RM training methods, such as the commonly used method from Stiennon et al. (2020); see Appendix F for comparisons and Appendix J.4 for the used prompts. We perform Bayesian hyperparameter optimization for OPT-RM and sweep over the learning rate, batch size, and prompt-loss weight $\lambda$, using classification accuracy on the development dataset as the selection criteria (see Appendix G for more details).

As a final step, we combine ILF and learning from binary feedback, by first finetuning GPT3-175B on the refinements as described in the paragraph finetuning on refinements (ILF). We then train the reward model, OPT-RM, and use it to perform best-of-$N$ sampling, as outlined in the paragraph on learning from binary feedback. At test time, we generate 64 summaries with our finetuned model and rank them based on their probability of being a high-quality summary, $p_{\text{norm}}(y_{\text{good}}|x_{0})$, using OPT-RM. The summary with the highest normalized probability is then selected.

3.2 Evaluation

We evaluate the effectiveness of our learning algorithm, by comparing it to human written reference summaries, several finetuning baselines, and OPT-RM on the task of text summarization using 100, 1K, and 5K train samples. Using a test dataset of 698 samples, we generate a summary for each method and evaluate them with human evaluators who rank them based on quality, using a standard ranking scheme that allows for ties between summaries (see App. G for more details). Based on the rankings, we calculate the fraction of times each method’s sampled summary outperforms the human-written reference summary, referred to as the “win rate”. We sample summaries up to 48 tokens in length (as in Stiennon et al. (2020)) using nucleus sampling (Holtzman et al., 2019) with $p=0.95$ and temperature $t=1.0$ (see App. G for further details on hyperparameters and postprocessing). We use best-of-64 sampling with summaries sampled from FeedME for learning from binary feedback.

3.3 Results

Our results, shown in Fig. 2, demonstrate that finetuning on refinements (ILF) outperforms all other finetuning methodsFinetuning on 100 refinements is tied with finetuning on 100 initial summaries.), including sampling from FeedME, with a win rate against human summaries of $31.3\pm 1.7\%$ (for finetuning on 5K samples), while the other methods achieve win rates of $27.3\pm 1.7\%$ (finetuning on initial summaries), $28.9\pm 1.7\%$ (finetuning on human summaries), and $22.5\pm 1.6\%$ (FeedME). It is surprising that ILF outperforms finetuning on human summarise across all sample sizes, despite human-written summaries generally being of higher quality (see Fig. 3, top). Further evaluation (see App. Fig. 7) shows that the model finetuned on 1K refinements (ILF) exhibits significantly lower loss when evaluated on the validation dataset of refinements compared to the model finetuned on human summaries when evaluated on the validation dataset of human summaries, suggesting that the model is more adept at approximating the distribution of refinements. Additionally, when evaluating GPT3-175B on the summaries of 1K samples from various train datasets, we observe significantly lower loss on the refinement dataset than on the dataset of human summaries (see Table. 6). Overall, these results demonstrate the effectiveness of our proposed ILF approach in accurately incorporating feedback and improving model performance, even outperforming finetuning on human summaries.

(Scheurer et al., 2022) found that ILF with 100 feedback samples outperformed FeedME, while here we find it underperforms FeedME with 100 feedback samples. Prior work uses author-written feedback that often conveys what the refinement should include, while our work includes more varied, crowdsourced feedback. As a result, we observe that embedding similarity does not properly rank refinements on our human feedback dataset (Table 2), and we believe the difference in feedback may be a significant source of differences in results in this section as well; see Appendix H.5 for more discussion.

Our results demonstrate that using OPT-RM for best-of-64 sampling on FeedME summaries outperforms all finetuning methods and sampling approaches across all sample sizes. The improved performance of OPT-RM best-of-64 FeedME comes at the cost of added inference time for best-of-$N$ sampling. Combining ILF and learning from binary feedback (ILF + OPT-RM (best-of-64)) achieves human-level summarization performance with a win rate of $50.8\pm 1.9\%$ using 5K samples for training. This suggests that both methods independently learn valuable information about human preferences that can be cumulative when used together. It should be noted that the result for ILF + OPT-RM (best-of-64) is obtained through a separate human evaluation with different comparison summaries (see App. Fig. 8), and was added to Fig. 2 for reference. In App. H.3, we present some initial, promising results for multiple iterations of ILF. These results suggest that the method is effective, but further experimentation is necessary to understand it better.

4 Does Language Feedback Improve Refinements?

The improvements from ILF suggest that the refinements used for finetuning are high-quality, so here we investigate whether language feedback is responsible for the high quality. To do so, we have human evaluators rank Refinement with Feedback + Best of N summaries against summaries from several other methods, similar to §4.2. We use the human ranking to compute a win rate between each method and the initial summary. We compare against Refinement with Feedback, which randomly chooses a refinement $\in{x_{1}^{1},\dots,x_{1}^{5}}$. This ablation helps to evaluate the importance of choosing a refinement with our scoring function $R$, i.e., InstructRM Ensemble. We also evaluate Refinement without Feedback, which instructs the LM to refine the initial summary but without feedback. This ablation helps to evaluate the importance of using language feedback. Lastly, we evaluate Human Summaries and Initial Summaries i.e., the initial summary $x_{0}$ generated by FeedME. We evaluate all methods on the validation dataset.

Fig. 3 (top) shows the win rates of summaries from various methods against initial summaries. Surprisingly, instructing a model to improve its output without feedback already leads to a significant improvement (win rate of $59.4\pm 2.1\%$ over the initial summaries). Refinements with Feedback achieve an improved win rate of $63.9\pm 2.0\%$, showing that language feedback is useful for improving refinement quality. Refinement with Feedback + Best of N achieves an even better win rate of $69.1\pm 1.9\%$, highlighting that Best-of-N with the InstructRM Ensemble further improves the refinements. Overall, language feedback is important for high-quality refinements, especially when using Best-of-N sampling.

5 Do Refinements Incorporate the Feedback?

To determine whether refinements are of higher quality due to incorporating feedback rather than improving the summary in other ways, we conduct a study on the validation dataset in which crowd workers evaluate how often the most important point of the feedback is incorporated in the refinements produced by various methods. As shown in Fig. 3, bottom, our method Refinement with Feedback + Best of N incorporates the most important point in the feedback most frequently ($57.4\pm 2.2\%$ often). Refinement with Feedback incorporates feedback $49.6\pm 2.2\%$ of the time, showing that Best-of-N sampling improves how often the feedback is incorporated. For reference, Refinement without Feedback fixes the most important point in the feedback $30.8\pm 2.1\%$ of the time, despite the model not receiving the language feedback. Human Summaries address the most important point in the feedback $74.0\pm 1.9\%$ of the time when writing the summary from scratch despite not receiving the feedback explicitly. Our results suggest that refinements are high-quality in part because they incorporate the most important point in the feedback.

6 Which Finetuning Dataset Changes Models Most?

Here, we aim to understand how the summaries used for finetuning influence how much the model changes after finetuning. Gao et al. (2022) find that models optimized with binary human feedback are more likely to learn undesirable behaviors when their output distribution deviates more from the initial, pretrained LM. It is unclear whether these findings apply to models trained with language feedback, but we take a preliminary step in this direction for understanding language feedback-trained models. In particular, we measure the (reverse) KL divergence (following Gao et al., 2022) between an ILF-finetuned model and the pretrained LM before ILF-training, $D_{\text{KL}}(\text{finetuned}|\text{GPT3-175B})$, by unconditionally sampling from the finetuned model and evaluating the log-likelihood of the generated text with GPT3-175B. We also report the forward KL divergence, $D_{\text{KL}}(\text{GPT3-175B}|\text{finetuned})$. For reference, we evaluate both of the above for models finetuned on the initial summaries and on human summaries.

Finetuning on refinements (ILF) shows the largest KL divergence (in both directions), followed by finetuning on human summaries, and then followed by finetuning on initial summaries; see App. Table 6 for the exact numbers. We find it surprising that finetuning on refinements results in higher KL divergences than finetuning on human summaries; we expected the refinements to be closer to the model’s initial output distribution, relative to human summaries, therefore causing the finetuned model to undergo less change. The larger KL divergence with ILF may be partly responsible for the larger gains in human evaluations observed in Fig. 2.

Related Work

Our work builds upon our previous report (Scheurer et al., 2022), which showed that large LMs can refine outputs with language feedback. There, we introduce the same three-step algorithm that ILF builds upon, with the key difference that here we use an LM, i.e., InstructRM Ensemble, to evaluate whether a refinement incorporates feedback, whereas in Scheurer et al. (2022) we use a contrastive pre-trained text-embedding function (Neelakantan et al., 2022). InstructRM Ensemble is more general than this Embedding Similarity since it does not assume semantic similarity of the refinements to the feedback. Another difference is that ILF is an iterative, refine-and-finetune algorithm, which can be understood as Bayesian Inference corresponding to RL with Human Feedback. In addition, here we conduct different and more extensive experiments than in Scheurer et al. (2022) and use human annotators. In particular, we show that ILF outperforms finetuning on human summaries and that combining ILF with learning from binary feedback achieves roughly human-level summarization performance. For a more detailed comparison to Scheurer et al. (2022) we refer to App. H.5.

Subsequent work to ours suggests several ways to improve upon our approach. Saunders et al. (2022) show that LMs themselves write high-quality feedback on LM outputs. Bai et al. (2022) then train a dialog assistant using ILF to learn from LM-written language feedback, eliminating the cost and effort of collecting human feedback. Liu et al. (2022); Schick et al. (2022) train LMs to refine outputs based on feedback (without finetuning on the refinements), an approach that improves results when incorporated into ILF, as shown in subsequent work to ours (Shi et al., 2022).

Other work aims to use language in other ways than we do. Some work investigates using explanations for gold labeled outputs to classification tasks, while our work addresses the more general text generation setting which classification tasks can be formulated as (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020). Explanations describe why a labeled output is correct, while feedback describes how to improve a candidate’s output. Prior work explores ways of using explanations to train text classification models, with mixed results (Camburu et al., 2018; Stacey et al., 2021; Pruthi et al., 2021; Wiegreffe et al., 2021; Hase & Bansal, 2021; Lampinen et al., 2022, inter alia). A few prior works also learn from language feedback for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022; Xu et al., 2022). Matiana et al. (2021) learn text embeddings of language feedback, where improvements could benefit the refinement-scoring step of our algorithm. Language has also been used for various purposes in RL settings as well, as discussed in App. B.

Several other works draw connections between Bayesian Inference and learning algorithms for LMs. Korbak et al. (2022) show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by a reward function. Dohan et al. (2022) further argues that the process of generating output through multiple rounds of interaction between prompted LMs and other agents (e.g. humans providing language feedback) can be seen as executing probabilistic programs.

Conclusion

In this work, we propose Imitation learning from Language Feedback (ILF), an iterative algorithm for training LMs to behave in line with human preferences, by learning from language feedback. We validate our approach on a carefully-controlled word-removal task, showing that only large LMs (175B parameters) accurately incorporate feedback. Using this insight, we then test our algorithm on the real-world task of text summarization. Combining ILF and learning from binary feedback brought a GPT-3 model to roughly human-level summarization ability. ILF on its own outperformed finetuning on human summaries, despite human summaries being of higher quality, suggesting that the model is better at approximating the distribution of refinements. Our work opens up many avenues for future work, from improving algorithms for learning from language to tackling settings where it is hard to learn from sparse or binary feedback.

Acknowledgements

We are grateful to Nat McAleese, Geoffrey Irving, Jeff Wu, Jan Leike, Cathy Yeh, William Saunders, Jonathan Ward, Sam Bowman, Daniel Ziegler, Seraphina Nix, Quintin Pope, Kay Kozaronek, Peter Hase, Asa Cooper Stickland, Jacob Pfau, David Lindner, Lennart Heim, Nitarshan Rajkumar, Kath Lumpante, Pablo Morena, Edwin Chen, Scott Heiner, and David Dohan for helpful conversations and feedback. Jérémy Scheurer and Jun Shern Chan thank Open Philanthropy for funding that enabled this research. Ethan Perez thanks the National Science Foundation and Open Philanthropy for fellowship support. Jon Ander Campos is supported by a doctoral grant from the Spanish MECD. Angelica Chen and Kyunghyun Cho are supported by the NYU Center for Data Science National Science Foundation (Award 1922658). KC was supported by 42dot, Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling), Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), and NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science We also thank OpenAI for providing access and credits to their models via the API Academic Access Program.

References

Appendix A Additional derivations

Our goal is to produce a high-quality output $x_{1}$ for a context $c\sim p(c)$ (e.g., a summary of a document). We use an LM $\pi_{\theta}$ to generate an output $x_{1}$, by conditioning on the context $c$, i.e., $x_{1}\sim p_{\theta}(x_{1}|c)$. We then introduce the predicate $\mathcal{I}$, a random variable such that $\mathcal{I}=1$ if the output is high quality according to human preferences. We denote this data-generating process, shown in Fig. 4 left, as:

We maximize the lower bound in Eq. 6, henceforth called $F(\theta,q)$, using an Expectation-Maximization (EM) procedure: alternating between maximizing $F$ w.r.t. the proposal distribution $q$ (E-step) and w.r.t. $\pi_{\theta}$ (M-step) We call this algorithm Imitation learning from Language Feedback.

Maximizing $F(\theta,q)$ w.r.t $q$ corresponds to refining the proposal distribution $q$ to assign higher likelihood to high-quality texts. This is achieved by embedding $x_{1}$ into a data-generating process involving humans, by introducing the initial output $x_{0}$, and human feedback $f$ (via sum rule):

Eq. 10 gives rise to the following sampling procedure (see also Fig. 4, right): First, an LM is conditioned on the context $c$ and generates an initial output $x_{0}$. Second, a human provides language feedback $f$ on the $(c,x_{0})$ pair. Third, the LM generates a refined text $x_{1}$ conditioned on $(c,x_{0},f)$. Finally, a binary variable $\mathcal{I}$ indicates whether $x_{1}$ is a high-quality text, given an initial output $x_{0}$, feedback $f$, and a context $c$. We model $p_{\theta}(\mathcal{I}=1|c,x_{0},f,x_{1})$ as a Boltzmann distribution:

which uses a reward function $R$ defined in terms of four variables: $c,x_{0},f,x_{1}$; $\beta$ is a temperature hyperparameter. This Boltzmann distribution makes quality easy to evaluate since it expresses it as a reward function $R$ of a previous output and human language feedback.

We now argue why the E-step results in a proposal distribution that is better than the original distribution $p_{\theta}(x_{1}|c)$, i.e., why samples from $q(x_{1}|c)$ tend to be of higher quality than samples from $p_{\theta}(x_{1}|c)$. First, we know that $x_{0}$ is already a reasonably good output (since $\pi_{\theta_{\text{old}}}\approx\pi_{\theta}$). We can assume that the feedback $f$ is informative and high-quality. Therefore $x_{1}\sim p_{\theta}(x_{1}|c,x_{0},f)$ is going to be of higher quality than $x_{0}\sim p_{\theta}(x_{0}|c)$ because it leverages useful information from the feedback. Furthermore, let us choose $R$ to assign higher values to refined texts $x_{1}$ that improve upon $x_{0}$ w.r.t to $f$ and $c$. Consequently, Eq. 11 assigns a higher likelihood to high-quality outputs $x_{1}$, allowing us to put additional weight on high-quality outputs and improving the proposal distribution $q$ further.

Maximizing $F(\theta,q)$ w.r.t. the policy $\pi_{\theta}$ is equivalent to supervised learning (minimizing cross-entropy loss) on a distribution defined by $q$. To see that, we drop all the terms from Eq. 7 that do not depend on $\theta$:

In ILF, we alternate between the E-step and M-step, using the pseudocode in Algorithm 1. In the M-step, we use the model from the previous iteration $\pi_{\theta_{\text{old}}}$ as both $p_{\theta}(x_{0}|c)$ and $p_{\theta}(x_{1}|c,x_{0},f)$. In practice, we implement $R$ by conditioning an instruction-finetuned LM on a binary question such as Does this new text incorporate the feedback provided? Answer Yes or No. where the label $y$ is either $y_{\text{good}}$ (“ Yes”) or $y_{\text{bad}}$ (“ No”). We use the probability of the positive answer $y_{\text{good}}$ given the prompt as a reward, i.e. $p(y_{\text{good}}|\text{prompt})=\frac{p(y_{\text{good}}|\text{prompt})}{p(y_{\text{good}}|\text{prompt})+p(y_{\text{bad}}|\text{prompt})}$. With these assumptions, $q$ takes the form:

We take advantage of this proposal distribution and perform the M-step, i.e., $\text{argmax}_{\theta}F(\theta,q)$ on optimized data. Finally, we approximate sampling from $q(x_{1}|c)$ by best-of-$N$ sampling. To obtain a sample $x_{1}\sim q$, we sample $N$ refinements $\{x_{1}^{1},\dots,x_{1}^{N}\}\sim\pi_{\theta_{\text{old}}}(x_{1}|c,x_{0},f)$, and compute

In summary, we show that ILF can be understood as Bayesian inference. This process involves updating an LM based on the evidence provided by language feedback. This lens highlights the correspondence between ILF and RL with Human Feedback (Ziegler et al., 2019; Stiennon et al., 2020, inter alia), which was previously demonstrated to be equivalent to Bayesian inference (Korbak et al., 2022).

Appendix B Additional Related Work on Language in RL Settings

Language has been widely used in RL for various purposes (see Luketina et al., 2019, for an overview), such as specifying tasks (“instruction following” , inter alia) driving exploration (Tam et al., 2022), inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia), and training a model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017), reward shaping (Goyal et al., 2019), or by providing descriptions of trajectories (Nguyen et al., 2021). In contrast, we use language to correct faulty behavior. Other work uses language feedback at test time to correct mistakes in a model’s behavior, e.g., image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021). In contrast, we use feedback to train models, and our approach does not require human intervention at test time.

Appendix C Dataset Collection and Analysis

To ensure the high quality of our human annotations, we employ experienced annotators sourced through the data-labeling company Surge AI. During an onboarding and evaluation process, we calculate author-annotator agreement on the binary comparison task and manually review the quality of the written feedback and ideal summaries to ensure their high quality. Then we select 31 qualified annotators for all annotation tasks, though they can choose which tasks to participate in and for how long. To further ensure the quality of our annotations, we provide detailed instructions, which we provide to the annotators, and update throughout the process to ensure continuous improvement (these instructions can be found in Appendix I). To measure the agreement rate between the annotators and the authors, we select a sample of 10 Reddit posts from the training dataset as a gold standard and have 17 annotators label them. When comparing the binary comparison annotations with our own ones, this results in an author-annotator agreement rate of $81.0\%$. We also calculate the average agreement rate between all the possible annotator combinations, yielding an annotator-annotator agreement of $70\%$. By utilizing these thorough processes and evaluations, we can ensure the accuracy and reliability of our human annotations.

The feedback we collect typically addresses the most critical shortcomings of the summaries. In $92.0\%$ of our train samples, the annotators’ feedback was complete and addressed all important shortcomings of the summary, as reported by the annotators. Across our train dataset, we observe that the majority of the feedback pertains to coverage ($77.0\%$), with smaller percentages relating to accuracy ($16.0\%$), coherence ($5.0\%$), and other categories ($2.0\%$). We also analyze the length of the various summaries and feedback, measured in the average number of tokens. Our human-written summaries have an average length of $41.0\pm 0.1$ tokens, the extracted human summaries from Reddit had an average length of $32.5\pm 0.1$ tokens, the initial summaries generated by FeedME have an average length of $29.3\pm 0.1$ tokens, and the feedback written by annotators on these initial summaries has an average length of $20.4\pm 0.2$ tokens.

In addition to these analyses, we also measure the time it takes annotators to complete various tasks (i.e., binary comparison, feedback writing, and ideal summary writing) on our development dataset. We ignore outliers and consider only samples with annotation times of at least 20 seconds and at most 420 seconds (7 minutes). Annotators take $61.5\pm 5.3$ seconds on average on the binary comparison task, $182.5\pm 6.3$ seconds on the feedback task, and $195.5\pm 6.1$ seconds on the ideal summary task. We plot the annotation times on the development dataset for the tasks of annotating binary comparisons, writing feedback, and writing ideal summaries as histograms in Fig. 5. The annotators are much faster at annotating binary comparisons than feedback or ideal summaries. Writing feedback takes less time than writing ideal summaries, which is expected, as critiquing a task is usually easier than solving it. These comprehensive evaluations demonstrate the high quality and thoroughness of our dataset and annotation processes.

Appendix D Targeted Word Removal Details

Below is an example of how we instruct or “prompt” an LM to remove specific, offensive words from a sentence.

“In this text, many toxic and offensive words are used: You are such a jerk, and a nice person, and an idiot. The ideal text should remove the word jerk, but otherwise be unchanged: You are”

Here, the target completion is “ such a nice person and an idiot.” More formally, we sample offensive sentences by using $k$ offensive words from a fixed set of 25 offensive words drawn uniformly at random (without replacement). Each offensive sentence also includes the words ”nice person” in addition to all the offensive words. For each $k\in\{1,\dots,10\}$, we sample 50 offensive sentences. The task is then to remove $l\in$ offensive words from a given sentence with $k\geq l$. Since we include the words ”nice person” in the offensive sentence, we can remove $l=k$ offensive words and still have a target sentence that intuitively makes sense.

Appendix E Details about Ranking Procedure

We use a standard ranking scheme where each of $K$ summaries is given a rank between 1 and $K$ (inclusive). Sometimes refinements are exact copies of the initial summaries or are very similar in terms of quality, which is why we allow for summaries to be tied. When calclating the win rate we assign $0.5$ wins for tied samples. We assign the rank $r^{\prime}$ to all summaries ranked in a tie, where $r^{\prime}=\frac{r+(r+n-1)}{2}$, $r$ is the rank of the tied elements, and $n$ is the number of ties at the rank. For example, we map a ranking of $(1,2,2,4,5)\rightarrow(1,2.5,2.5,4,5)$ and a ranking of $(1,2,3,3,3)\rightarrow(1,2,4,4,4)$.

Appendix F Reward Model

Here we describe the various RMs that we evaluate in more detail. We evaluate the final RM that we use, which produces a language output (e.g., “ Yes” or “ No”) and a standard reward model that produces a scalar output.

Akin to (Stiennon et al., 2020), we remove the last embedding layer of a language model and train it to output a scalar value. This scalar value predicts which summary, $x\in{\{x_{0}^{0},x_{0}^{1}\}}$, is better as judged by a human, given a context $c$. We use the OPT 13B LM, introduced in (Zhang et al., 2022), as the base model for our RM and finetune it on the human preference comparisons that we collected. It is worth noting that it is not possible to add linear layers on top of GPT-3 models provided via the API, which is why we use the OPT model.

In addition to the classic RM (Stiennon et al., 2020), we train an RM to output language tokens instead of a scalar value. To do so, we finetune an LM to classify whether a summary $x_{0}$ is high quality or not, by training it to predict a label $y\in\{y_{good},y_{bad}\}$. We then finetune the LM to maximize $\lambda\log p(x_{0})+\log p(y|x_{0})$, where $\lambda\in$, chosen using the development dataset. The complete loss can also be written as:

We evaluate all RMs on our validation dataset, and calculate the accuracy of predicting the preferred summary out of two, based on human preferences. Table 4 shows the complete results, and here we report on some of the RMs trained on 5K samples. The OPT model with the standard RM loss achieves an accuracy of $71.8\pm 2.0\%$ on the validation dataset. The results further show that both of our methods for training OPT with the LM loss outperform the standard RM loss, with OPT comparison achieving an accuracy of $72.6\pm 1.9\%$, and OPT-RM an accuracy of $73.4\pm 1.9\%$. We obtain similar results with finetuning GPT-3-175B, achieving an accuracy of $71.2\pm 2.0\%$ with the GPT3 Comparison, and an accuracy of $74.2\pm 2.0\%$ with GPT-3 Binary, which outperforms the OPT-RM.

Based on these results, we further evaluate the OPT Binary and GPT-3-175B Binary models on the development dataset that we use to evaluate the scoring functions in §4.2. We calculate the fraction of times the refinement selected by an RM is better than a randomly-selected refinement (“win rate”), according to a ranking given by human evaluators (see App. E for more details). The results can be found in Table 3. OPT-RM achieves a win rate of $63.3\pm 2.7\%$, and the GPT-3-175B Binary model achieved a win rate of $61.8\pm 2.9\%$. In this evaluation, OPT-RM outperforms GPT-3 Binary. When considering the results from both the validation and development datasets, both OPT-RM and GPT-3-Binary seem to perform similarly. Given that we have more control over the training process of OPT, the possibility of releasing the model, and the cost involved in training using OpenAI’s API, we select OPT-RM model as our reward model for comparison with ILF. In Figure 6, we show the validation accuracy of OPT-RM trained on 100, 1K, and 5K samples on a log-log plot. The figure shows scaling when increasing the dataset size.

We further evaluate results for finetuning OPT-RM on the dataset of Stiennon et al. (2020), and also evaluating their model with 1.3B parameters on our dataset. We observe that the binary preference distribution of the training dataset has a significant impact on the performance of the reward model. For example, OPT-RM trained on 5K samples of our own train dataset (i.e., our final reward model) achieves an accuracy of $61.9\pm 0.2\%$ on the test set from Stiennon et al. (2020) (not shown in Table 4). When this same model is trained on 90K samples from the train dataset of Stiennon et al. (2020), it achieves an accuracy of $69.3\pm 0.2\%$ on their test set (also not shown in Table 4). In contrast, this same model trained on 90K samples from their train dataset achieves an accuracy of only $68.6\pm 2.0\%$ on our validation dataset, which is significantly lower than the accuracy of $73.4\pm 1.9\%$ achieved by the model trained on 5K samples of our own train dataset. Similar patterns can be observed when comparing the OPT Binary model with 1.3B parameters trained on 5K samples of our own train dataset to the released 1.3B reward model trained by Stiennon et al. (2020) on approx. 64K samples of their own train dataset. The former model achieves an accuracy of $69.6\pm 2.0\%$ on our validation dataset, while the latter only achieves an accuracy of $63.8\pm 2.1\%$ (note, though, that the RMs are trained with different loss functions). These results highlight two important considerations: (1) preference distributions can vary significantly and have a strong effect on what a reward model learns, and (2) the sample efficiency of a reward model depends heavily on the train and test distributions. If the test distribution differs from the train distribution, reward models may be very sample inefficient and fail to accurately learn the true distribution, even when given significantly more samples.

Appendix G Hyper Parameters

For the targeted word removal experiments (§3), we use greedy decoding until 200 tokens or / n is generated. For all summarization experiments we sample up to 48 tokens (as in Stiennon et al., 2020) with nucleus sampling (Holtzman et al., 2019) with $p=0.95$ and temperature $t=1.0$. We strip non-alphanumeric characters (e.g., newlines) from the beginning of sampled summaries. We further remove empty white spaces in the generated summaries and remove all text that comes after a new line token / n. Due to the maximum token length, sampled summaries sometimes end with incomplete sentences. Thus, we remove ending sentences that do not end in “.”, “!”, or “?”. The described temperature and post-processing are applied to all summary generations, i.e., for generating initial summaries, refinements, and test summaries.

G.2 Finetuning on Summaries

We conduct independent hyperparameter optimization sweeps with three dataset sizes of human summaries of 100, 1K and 5K samples, and then use the same hyperparameters for finetuning on refinements (ILF) and finetuning on initial summaries. We choose to run the hyperparameter sweep on Human summaries since this will not give an unfair advantage to our algorithm that finetunes on refinements. For the sweep, we utilize the train dataset of human summaries (consisting of 100, 1K, and 5K samples) and evaluate on the development dataset. Unfortunately, the OpenAI API only provides validation loss and token accuracy for batches of the development dataset, making it impossible to evaluate the model on the full development dataset during training. As a result, we utilize the model API to evaluate on the full development dataset after finetuning and calculate the perplexity of the generated summaries as a performance measure.

To determine the optimal hyperparameters, we perform a sweep over a range of values for the following parameters: epochs $\{1,2,3,4\}$, prompt loss weight $\{0,0.01,0.05,0.1\}$, and learning rates $\{0.02,0.05,0.1,0.2\}$. We first sweep over epochs and select the best value, then perform a sweep using that value for the prompt loss weight, and so on. Our empirical observations indicate that the number of epochs has the greatest impact on perplexity, with training for more than one epoch resulting in overfitting. The selected hyperparameters can be found in Table 5.

During the finetuning phase for the Refinements and Initial Summaries datasets with 1K samples each, we made an error in our hyperparameter selection. Instead of using a prompt loss weight of $0.05$, we mistakenly used a value of 0, when finetuning on human summaries. While this error may have slightly impacted our results, the difference in perplexity between the two settings is minimal, with a value of $6.68$ for a prompt loss weight of $0.05$ and $6.71$ for a prompt loss weight of 0. Despite this mistake, our method still outperforms finetuning on human summaries for 1K samples, as well as finetuning on initial summaries using suboptimal hyperparameters.

G.3 Multiple Iterations of ILF

To evaluate multiple iterations of ILF, i.e., multiple iterations of refining-and-finetuning, we finetune GPT-3-175B on a refinement dataset with 200 and 300 samples. Thus we conduct a hyperparameter optimization on a train dataset of 200 and 300 refinements and evaluate on a development dataset of 200 refinements (instead of human summaries). To determine the optimal hyperparameters, we perform a sweep over a range of values for the following parameters: epochs $\{1,2,3,4\}$, prompt loss weight $\{0,0.01,0.05,0.1\}$, and learning rates $\{0.02,0.05,0.1,0.2\}$. We first sweep over epochs and select the best value, then perform a sweep using that value for the prompt loss weight, and so on. For finetuning on 200 refinements we select the following hyperparameters: $\text{epochs}=1$, $\text{prompt loss weight}=0.05$, $\text{learning rate multiplier}=0.1$. For finetuning on 300 refinements we select $\text{epochs}=1$, $\text{prompt loss weight}=0$, and $\text{learning rate multiplier}=0.2$.

G.4 Finetuning Reward Models

For finetuning the OPT Reward Model, we perform bayesian hyperparameter optimization for each of the three different types of reward models: Standard, Comparison and Classification (see section F). We sweep over the learning rate in the range of $[1\text{e}^{-5},1\text{e}^{-6}]$ and the batch size $\{32,64\}$ for all the models. For the reward models using the language loss, we also optimize the prompt-loss weight $\{0.0,0.01,0.05,0.1,0.5,1.0\}$. We run 10 iterations per model and evaluate all the sweeps with the 200 development examples. We use a linear learning rate scheduler and a weight decay of $0.1$ for all the runs. The optimal batch size is 32 for all the models. The best prompt loss weight is $0.01$ for both the Comparison and Classification RMs. As for the learning rate, we use $9.3\text{e}^{-6}$ for the Standard RM, $5.8\text{e}^{-6}$ for the Classification RM and $1\text{e}^{-6}$ for the Comparison RM. In the final finetuning, we select the best RM in the validation split over 10 epochs.

In order to finetune GPT-3-175B as an RM, we utilize the OpenAI API. We finetune two types of RMs: the Comparison RM, which learns to predict which of two summaries is superior, and the Classification RM, which predicts whether a given summary is of high quality or not. For cost considerations, we conduct hyperparameter tuning on a training dataset of 1K samples (instead of 5K) and evaluate on a development dataset of 200 samples. We use a dataset with 1K samples for cost reasons. We then apply the same hyperparameters when finetuning on 5K samples while implementing early stopping in terms of epochs. Due to the binary nature of the human preference annotations in the classification reward model, the effective train dataset size for this model is doubled to 2K samples.

In order to determine the optimal hyperparameters, we perform a sweep over a range of values for the number of epochs $\{1,2,3,4\}$ and the prompt loss weights $\{0,0.001,0.005,0.01,0.05,0.1,0.5\}$. The OpenAI API provides classification accuracy (for both the comparison and classification tasks) for the full development dataset after each epoch, allowing us to select the appropriate number of epochs and prompt loss weight. When finetuning on 5K samples, we utilize early stopping to prevent overfitting, using 1 epoch and a prompt loss weight of 0 for the comparison model and 4 epochs and a prompt loss weight of $0.001$ for the classification model. We use default values for all other hyperparameters, which may vary depending on the dataset size.

Appendix H Additional Results

In Table 6, we evaluate GPT-3-175B on various finetuning datasets used for finetuning: the refinements, the initial summaries, and the human summaries. We evaluate the log-likelihood of GPT-3-175B on the summaries of 1K samples from the various train datasets (i.e. initial summaries, refinements, and human summaries). Concretely, we pass the whole prompt to GPT-3-175B, including the Reddit post, but only evaluate the log-likelihood of the completion, i.e. the generated summary. We also measure the (reverse) KL divergence (following Gao et al., 2022) between an ILF-finetuned model and the pretrained LM before ILF-training, $D_{\text{KL}}(\text{finetuned}|\text{GPT-3-175B})$. We sample unconditionally (i.e. using a beginning of sentence token) from the finetuned models and evaluate the log-likelihood of the generated text with GPT-3-175B. We also report the forward KL divergence, $D_{\text{KL}}(\text{GPT-3-175B}|\text{finetuned})$. We discuss the results in §4.6.

H.2 Results: ILF + OPT-RM

In this section, we present the full results of our best-performing method ILF + OPT-RM and other additional methods (see §4.3.1 for a description of ILF + OPT-RM and §4.3.3 for a discussion of the results). We conduct the same evaluation as described in §4.3.2, i.e. in a human evaluation, annotators rank various test summaries based on quality. We then calculate the win rate against human written summaries, which we use as an evaluation metric. Importantly, all methods evaluated here are trained on datasets with 5K samples. Note that the methods compared here are not exactly the same as the methods compared in Fig. 2. Concretely, the test summaries generated by the methods finetuning on refinements (ILF), finetuning on human summaries, and OPT-RM best-of-64 FeedME are the same as in Fig. 2, for the test summaries generated by corresponding methods trained on 5K samples. Here, however, we don’t evaluate FeedME and finetuning on initial summaries. However, we evaluate ILF + OPT-RM (best-of-64), our best-performing model, which we also added to Fig. 2 for reference. We also evaluate a new method called Finetuned on Feedback + Refinements, which we describe below.

For finetuning on feedback + refinements, we us a title, post, and summary as input and the model is trained to predict the corresponding feedback and refinement. Our motivation for this approach is that generating feedback first may improve the quality of the resulting refinements, similar to the findings of previous work on self-prompting methods Saunders et al. (2022); Bai et al. (2022) and the Chain of Thought (CoT) prompting technique Wei et al. (2022). CoT has been shown to improve the performance of models across various tasks Wei et al. (2022) when allowing the model to reason before answering a question. For finetuning on feedback and refinements, we utilize the initial summaries that were used to gather human feedback, as well as the refinements generated by our method. We use the loss $\log p(x_{1},f|\text{prompt})+\lambda\log p(\text{prompt})$, i.e. we learn to predict the refinement and the feedback. We employ the same hyperparameters as in the finetuning on refinements algorithm (including the prompt loss weight). During testing, we require initial summaries, from which we generate feedback and refinements. As initial summaries, we use the test samples generated by FeedME (as evaluated in Figure 2). To ensure compatibility with the 48-token length restriction of the test summaries, we append the special end token / n ### to the end of the feedback and refinements during training. At test time, we set the maximum number of tokens to generate 300, and terminate generation when the stop-word / n ### appears. We then apply the same postprocessing procedure outlined in Appendix G.1 to shorten the refinements to 48 tokens. We refer to Appendix J.3 for the exact prompt templates we used.

We present all the results in Fig. 8. We find that finetuning on a set of 5K refinements achieves a win rate of $36.0\pm 1.8\%$, while ILF + OPT-RM (best-of-64) has a win rate of $50.8\pm 1.9\%$, achieving human-level summarization performance (see §4.3.3 for a more detailed discussion). OPT-rM best-of-64 FeedMe achieves a win rate of $45.1\pm 1.9\%$, finetuning on a set of 5K human-generated summaries achieves a win rate of $35.4\pm 1.8\%$, and finetuning on a combination of 5K feedback and refinements has a win rate of $26.1\pm 1.7\%$. It is worth noting that the performance of finetuning on feedback and refinements is lower than that of finetuning on refinements alone. We attribute this to the increased difficulty of generating both feedback and refinements and believe that this discrepancy may be due to limitations in our models, dataset size, or hyperparameters. Previous work has demonstrated the feasibility of training models to generate feedback Saunders et al. (2022); Bai et al. (2022), so we believe that further optimization and experimentation may improve the performance of this method. We further want to note that the results for finetuning on 5K refinements, 5K human summaries, and best-of-64 FeedME deviate from the results in Fig 2. This is because we compare different methods with each other, and human annotations generally contain some amount of noise (given that different people annotate the same samples).

H.3 Multiple Iterations of ILF

Our experiments suggest that ILF is an effective method for leveraging language feedback in the training of LMs. Here we explore ILF in its most general form by doing multiple iterations of refining-and-finetuning.

In this human evaluation, we compare the performance of the summaries generated by the baseline model ($M_{1}^{200}$) with those generated by two iterations of ILF ($M_{1,2}^{200}$) on our test set. Human evaluators are asked to indicate their preferred summary for each comparison, and the win rate of $M_{1,2}^{200}$ against $M_{1}^{200}$ is calculated and plotted in Fig. 9 (left)Note, we set the win rate manually to $50\%$ at 100 samples, since the baseline is equivalent to one iteration of ILF.. Our results show that two iterations of ILF outperform one iteration with a win rate of $53.2\pm 1.9\%$ indicating that applying multiple rounds of ILF can improve the data distribution. However, we also want to investigate whether multiple rounds of ILF lead to better models than directly finetuning on the same number of refinements from the first round from scratch. In other words, while our current baseline consists of further finetuning $M_{1}^{100}$ on an additional 100 samples, it is also possible to directly finetune GPT-3-175B on 200 refinements from the first iteration of ILF from scratch, i.e. $M_{scratch,1}^{200}$. We aim to determine the relative effectiveness of these two approaches in improving model performance on the text summarization task.

In this experiment, we aim to compare the performance of multiple rounds of ILF to directly finetuning on a comparable number of refinements from the first iteration of ILF. As a baseline, we finetune GPT-3-175B on 200 and 300 refinements from the first iteration of ILF and conduct hyperparameter tuning as described in the Appendix G. We then compare these baselines to two and three rounds of ILF. For the two-round ILF model, we use the previously described $M_{1,2}^{200}$. To obtain the three-round ILF model, we use $M_{1,2}^{200}$ to generate summaries for an additional 100 samples (on the same Reddit posts as for the baseline), gather human feedback, generate 5 refinements with GPT-3-175B using the feedback, and select the best refinement using InstructRM, resulting in $\mathcal{D}_{3}^{100}$. We then finetune $M_{1,2}^{200}$ on $\mathcal{D}_{3}^{100}$ to obtain the model $M_{1,2,3}^{300}$. It is important to note that while our baselines finetune GPT-3-175B from scratch on 200 and 300 refinements, the models $M_{1,2}^{200}$ and $M_{1,2,3}^{300}$ are obtained by continuously finetuning a model iteratively on additional refinements. This difference in approach may introduce a discrepancy in the results, as we use different hyperparameters, and the dataset size may affect the learning dynamics. To control for this potential difference, we also finetune GPT-3-175B from scratch on the refinements generated through various iterations of ILF. Specifically, as an alternative to $M_{1,2}^{200}$, we finetune GPT-3-175B from scratch on a concatenation of 100 refinements from the first round of ILF (i.e., $\mathcal{D}_{1}^{100}$) and 100 refinements from the second round of ILF (i.e., $\mathcal{D}_{2}^{100}$), and refer to the resulting model as $M_{scratch1,2}^{200}$. Similarly, as an alternative to $M_{1,2,3}^{300}$, we finetune GPT-3-175B from scratch on a concatenation of 100 refinements from the first round of ILF ($\mathcal{D}_{1}^{100}$), 100 refinements from the second round of ILF $\mathcal{D}_{2}^{100}$, and refinements from the third round of ILF (i.e. $\mathcal{D}_{3}^{100}$), and refer to the resulting model as $M_{scratch1,2,3}^{300}$. It is worth noting that the refinements from the second and third rounds of ILF (i.e. $\mathcal{D}_{2}^{100}$ and $\mathcal{D}_{3}^{100}$) are based on summaries generated using models that were continuously finetuned (i.e. $M_{1}^{100}$ and $M_{1,2}^{200}$). As such, the models $M_{scratch1,2}^{200}$ and $M_{scratch1,2,3}^{300}$ are not a direct application of ILF, but rather an approximation of the distribution induced by ILF. We refer to Table 7 for an overview of all models and train datasets.

Using a human evaluation, we compare the performance of the three methods on the test dataset: the baseline, ILF with continuous finetuning, and ILF approximated by finetuning from scratch. The results are shown in Fig. 9 (right). With this more realistic baseline, we find that directly applying ILF does not improve upon the baselines, with win rates of $49.4\pm 1.9\%$ and $50.9\pm 1.9\%$ for 200 and 300 samples, respectively. However, approximating ILF by finetuning from scratch on the distributions induced by ILF significantly improves upon the baseline for 300 samples, with a win rate of $55.6\pm 1.9\%$. The method is slightly worse than the baseline for 200 samples, with a win rate of $48.9\pm 1.9\%$. We currently hypothesize that continuous finetuning may lead to catastrophic forgetting, while finetuning from scratch may not have this problem. This could explain why $M_{scratch1,2,3}^{300}$ performs significantly better than $M_{1,2,3}^{300}$ for 300 samples. Specifically, $M_{1,2}^{200}$ may actually generate an improved distribution in the third iteration of ILF. However, when further finetuning $M_{1,2}^{200}$ on this improved distribution $\mathcal{D}_{2}^{100}$, the model may forget what it learned previously. On the other hand, the model $M_{scratch1,2,3}^{300}$ that learns from scratch on the concatenation of all datasets produced by ILF may actually benefit from the improved dataset distribution because it does not unlearn anything. It is, however, unclear why $M_{scratch1,2}^{200}$ does not benefit from the improved data distribution $\mathcal{D}_{2}^{100}$. It is also possible that the hyperparameters play a significant role in the final performance of the various models and that the dataset size has a strong influence on model performance (e.g., finetuning on more samples may be more stable than finetuning on fewer samples). In future work, we plan to conduct more elaborate experiments to answer these questions and better understand the effects of the dataset size and number of iterations on ILF. Specifically, we aim to run multiple iterations of ILF and use $M_{scratch1,2}^{200}$ as the model to generate summaries in the third round of ILF (instead of $M_{1,2}^{200}$). This would be a direct implementation of ILF, rather than an approximation of it, as we would be finetuning the same model with which we are also generating an improved distribution. We also hope to investigate the effect of the dataset size and number of iterations on ILF. Overall, our results suggest that ILF has the potential to improve the performance of natural language processing systems by continuously incorporating human feedback into the training of language models, but further research is needed to fully understand the best ways to leverage this approach.

H.4 Part-of-Speech Distribution for Finetuning Datasets

We evaluate the negative log-likelihood of GPT-3-175B on the three finetuning datasets, i.e. on initial summaries, refinements, and human summaries. We use the training dataset with 1K samples and calculate the negative log-likelihood over different Part-of-Speech tags. We use Stanza (Qi et al., 2020) as the PoS tagger for this experiment and then we separate the words into three groups: function words, content words, and others. The function words are words that have little lexical meaning: articles, pronouns, adpositions, conjunctions, auxiliary verbs, particles and interjections. On the other hand, content words are words that contain semantic information: nouns, adjectives, adverbs and lexical verbs. We keep numbers and symbols under the group others. With this analysis, we want to spot different patterns between model-generated (initial summaries and refinements) and human-written summaries. Note that a high negative log-likelihood implies a high loss. We present the results in Fig 10. Since the average loss is higher for human summaries, we normalize all the loss values by transforming them to have mean 0 and standard deviation 1. Overall, the word distribution is very similar for all three finetuning datasets. In terms of normalized mean loss, it is interesting how the content words have a bigger influence on the refinements dataset. We believe that this is related to our results in section 4.3.3, where we obtain the best results when finetuning on refinements.

H.5 Comparison to Results of Scheurer et al. (2022)

Here we relate our results to previous work by Scheurer et al. (2022). In Fig. 2 of Scheurer et al. (2022), they compare their method of finetuning on refinements against various baselines, such as finetuning on initial summaries, sampling from FeedME (called InstructGPT), and sampling from GPT-3-175B. They calculate the win rate of all methods against human written summaries (Völske et al., 2017) that are automatically extracted from Reddit. As shown in §4.1 and App.C, our human summaries are preferred $72.3\pm 3.2\%$ to the human summaries of Völske et al. (2017). This implies that the win rates in Scheurer et al. (2022) are much higher than in our case since we use a much stronger baseline.

We now present three differences between the results found in Scheurer et al. (2022) and the results found in our paper. Then we will provide various potential reasons that could explain the differences. First, when comparing the results (in relative terms) in Scheurer et al. (2022) Fig. 2 to our results in Fig. 2 where we finetune on 100 samples, we see differences in performance. Scheurer et al. (2022) reports that finetuning on refinements outperforms finetuning on initial summaries. And both methods outperform sampling from FeedME (i.e., InstructGPT). In our experiments finetuning on 100 refinements achieves a win rate of $19.6\pm 1.5\%$ against human summaries, finetuning on initial summaries a win rate of $19.6\pm 1.5\%$, and FeedME a win rate of $20.8\pm 1.5\%$. Thus both finetuned methods perform equally and are worse than sampling from FeedME.

Second, we compare the results of refining a summary with feedback. Note that Scheurer et al. (2022) uses an embedding-based scoring function to select refinements, whereas we use InstructRM. In Scheurer et al. (2022) Fig. 3 (left) Refine with Feedback + Best of N achieves a win rate of $67.0\pm 3.1\%$ against initial summaries (sampled from FeedME), Refine with Feedback achieves a win rate of $60.5\pm 3.0\%$, Refine without Feedback achieves $50.3\pm 2.6\%$ and Human Summaries have a win rate of $60.8\pm 3.4$. In our Fig. 3 (left) Refine with Feedback + Best-of-5 achieves a win rate of $69.1\pm 1.9\%$, Refine with Feedback achieves a win rate of $63.9\pm 2.0\%$, Refinement without Feedback achieves a win rate of $59.4\pm 2.0\%$ and Human Summaries a win rate of $83.2\pm 1.7\%$. The difference in the human summaries is expected, given that we use better human summaries. The Refinement without Feedback method achieves higher results in our work than in Scheurer et al. (2022).

Third, it is also noteworthy that using the embedding similarity as a scoring function worked well in Scheurer et al. (2022), while it does not work in our setting (see Table 2 and §4.2 for a discussion of the results). We believe this is because the feedback we collect is written by many annotators and is thus much more diverse, while in Scheurer et al. (2022), the authors themselves wrote the feedback.

Here we now list various differences in the setup of Scheurer et al. (2022) and our paper, which could all account for the different results.

Scheurer et al. (2022) use an embedding similarity as a scoring function, while we use InstructRM Ensemble. Looking at Tab. 2 and the corresponding discussion in §4.2, already shows that the methods are very different.

The human-written summaries are of much higher quality in our paper than in Scheurer et al. (2022) (see §4.1 and App. C)

In Scheurer et al. (2022), the annotation instructions specifically state that the feedback should mention how to improve a summary. In our work, we collect much more unrestricted and diverse feedback. This difference is also apparent in the fact that the embedding similarity does not work well as a scoring function in our setting.

In Scheurer et al. (2022), the authors themselves annotated the data, i.e., they wrote the feedback and evaluated the final summaries. In our case, we use independent evaluators who are trained on this task. Using 31 annotators overall also gives us a more diverse and less biased estimate of our methods. Also, doing human evaluations is inherently noisy and will never lead to the exact same results.

The evaluation in Scheurer et al. (2022) was done on a different dataset than in this work. Specifically, they used only 100 samples to evaluate their method, while we use a test set of 698 samples.

The hyperparameters in Scheurer et al. (2022) used for sampling and finetuning are different from the hyperparameters used in our work.

Overall, we use different prompts than Scheurer et al. (2022) (see App. J.3 and App. J.1)

Appendix I Annotator Instructions

Overall we completed many annotations to create datasets and evaluate our algorithm. The instructions were task-specific and also continuously updated. In the following, we provide the instructions we used to create our train dataset and the instructions we provided for evaluating the summary quality (of 6 summaries). We will not share more instructions for brevity but can provide them upon request.

You are given a Reddit Post, which you first need to read carefully. You then need to complete 5 subtasks which consist of comparing two summaries, writing feedback on a summary, classifying the type of feedback, indicating whether there is additional Feedback, and writing an ideal summary. When doing these tasks, please adhere to the guidelines below.

What makes for a good summary? Roughly speaking, a good summary is a short piece of text that has the essence of the original text. A good summary tries to accomplish the same purpose and conveys the same information as the original text. We would like you to consider these different dimensions of summaries:

Essence: Is the summary a good representation of the post? How well does the summary cover the important information in the post?

Clarity: Is the summary reader-friendly? Does it express ideas clearly?

Accuracy: Does the summary contain the same information as the post?

Purpose: Does the summary serve the same purpose as the original post?

Concise: Is the summary short and to the point?

Style: Is the summary written in the same style as the original post?

Generally speaking, we give higher weight to the dimensions at the top of the list. The evaluation can be complicated though, since none of the above dimensions are simple yes/no matters, and there aren’t hard and fast rules for trading off different dimensions. Use your best judgment and common sense to make these trade-offs. In case the subreddit, title, and Reddit post leave open some ambiguity about what happened, it is important to accurately reflect that in your annotations and not just interpret the text in a certain way. Always look at all the subreddit, title, and Reddit Post and use all information given to make your judgments (sometimes the title may contain crucial information that does not appear in the post but should nevertheless be used).

First, read the Subreddit category, title, and post carefully. A Subreddit is a forum dedicated to a specific topic on the website Reddit. Take your time with this step and re-read the parts that you might not have understood at first. Below is a detailed description of the task you will need to complete for each Reddit post.

Below is a detailed description of each task you will need to complete for each Reddit post:

Comparison Task: Given a pair of summaries, indicate which is better.

Details: Use the above description of what makes a good summary. It is alright to choose either summary if both summaries are identical copies of each other or if there is no distinguishing feature that makes one summary superior to the other. However, if there is a small detail that makes one summary better than the other, that is enough reason to select that summary.

Feedback Task: Write short and simple feedback on the given summary about the single, most important shortcoming of the summary. The feedback should NOT mention what category (Accuracy, Coverage, Coherence, other) the feedback belongs to, nor should it assume knowledge about the definitions of “Coverage”, “Accuracy”, or “Coherence” (see below). Otherwise, the feedback should be as short and simple as possible while still addressing the most important shortcoming of the summary.

Details: You can write the feedback in one or several sentences, but it should only address the single, most important shortcoming of the summary and be as short as possible. There are no other restrictions as to how you write the feedback and what exactly it addresses. If there are no shortcomings in the summary, the feedback can also mention a positive thing about the summary. Use the description of what makes a good summary to trade off the various dimensions that make for a good summary. Often the feedback will (but does not have to) address one of the following axes.

Coverage: For this axis, answer the question, “how well does the summary cover the important information in the post?” A summary has good coverage if it mentions the main information from the post that’s important to understand the situation described in the post. A summary has poor coverage if someone reading only the summary would miss several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g., to ask for advice).

Accuracy: For this axis, answer the question, “does the factual information in the summary accurately match the post?” A summary is accurate if it doesn’t say things that aren’t in the article, doesn’t mix up people, and is generally not misleading. If the summary says anything at all that is not mentioned in the post or contradicts something in the post, it is NOT accurate.

Coherence: For this axis, answer the question, “how coherent is the summary on its own?” A summary is coherent if, when read by itself, it’s easy to understand and free of English errors. A summary is not coherent if it’s difficult to understand what the summary is trying to say. Generally, it’s more important that the summary is understandable than being free of grammar errors.

Additional Rules: The feedback should NOT mention what category (Accuracy, Coverage, Coherence, other) the feedback belongs to, nor should it assume knowledge about the definitions of “Coverage”, “Accuracy”, “Coherence”, or “other” (as defined above). Example: One should NOT write ”This is missing in the area of coverage”, or ”This summary lacks in the category of accuracy, because …”. The feedback should be understandable to a person who has never read the definition of ”Coverage”, ”Accuracy”, and ”Coherence”. You are, however, ALLOWED to use those words if they make sense on their own, e.g., you CAN say, ”This summary does not cover the important parts of the text because”, or ”This summary is inaccurate as it states …”, or ”This is not a coherent summary because …”.

Feedback Type Task: If your feedback falls into the categories Accuracy-related, Coherence-related, or Coverage-related, mark it as such by checking the corresponding checkbox for the (single) category it is related to. If your feedback is not related to any of these three categories, then check the ”Other” checkbox.

More Feedback Task: Answer with Yes if there is additional Feedback about an important shortcoming of the summary that you would want to mention and No otherwise.

Ideal Summary Task: Ideal Summary Task: Write a short summary for the Reddit post that is ideal in your view.

Details: The ideal summary should be ideal in terms of all the criteria mentioned above, i.e., essence, clarity, accuracy, coverage, purpose, conciseness, coherence, and style. In other words, you should not be able to find an obvious critique of the ideal summary that you write. It is okay to reuse parts of previous summaries but only if those parts should be a part of an ideal summary. The ideal summary should maximally be 48 tokens long (otherwise, you can’t submit your annotation). Tokens are generated by taking your ideal summary and splitting up certain words into individual pieces (this is necessary to train our AI). The interface will show you how many tokens your ideal summary has already taken up.

I.2 Summary Quality Evaluation Instructions

You will be given a Subreddit category, a title, and a Reddit Post, which you first need to read carefully. Your task is then to compare 6 summaries and rank them according to quality.

What makes for a good summary? Roughly speaking, a good summary is a short piece of text that has the essence of the original text. A good summary tries to accomplish the same purpose and conveys the same information as the original text. We would like you to consider these different dimensions of summaries:

Essence: Is the summary a good representation of the post? How well does the summary cover the important information in the post?

Clarity: Is the summary reader-friendly? Does it express ideas clearly?

Accuracy: Does the summary contain the same information as the post?

Purpose: Does the summary serve the same purpose as the original post?

Concise: Is the summary short and to the point?

Style: Is the summary written in the same style as the original post?

Generally speaking, we give higher weight to the dimensions at the top of the list. The evaluation can be complicated though, since none of the above dimensions are simple yes/no matters, and there aren’t hard and fast rules for trading off different dimensions. Use your best judgment and common sense to make these trade-offs. In case the subreddit, title, and Reddit post leave open some ambiguity about what happened, it is important to accurately reflect that in your annotations and not just interpret the text in a certain way. Always look at all the subreddit, title, and Reddit Post and use all information given to make your judgments (sometimes the title may contain crucial information that does not appear in the post but should nevertheless be used).

First, read the Subreddit category, title, and post carefully. A Subreddit is a forum dedicated to a specific topic on the website Reddit. Take your time with this step and re-read the parts that you might not have understood at first. Below is a detailed description of the task you will need to complete for each Reddit post.

Comparison Task: Given 6 summaries, indicate which is better by ranking them according to quality. Rank 1 is considered the highest rank, and Rank 6 is considered the lowest rank. The summary with the best quality should be ranked highest, i.e., as Rank 1, and the summary with the worst quality should be ranked lowest, i.e. Rank 6. Use the above description of what makes a good summary. Ties between summaries are allowed, but only if summaries are exact copies of each other or if there is no distinguishing feature that makes one summary superior to the other. However, if there is a small detail that makes one summary better than the other, that is enough reason to rank that summary as better than the other summary. We use Standard Competition ranking (i.e., example rankings of 122456). In standard competition ranking, items that compare equally receive the same ranking number, and then a gap is left in the ranking numbers. The number of ranking numbers that are left out in this gap is one less than the number of items that are compared equally. Equivalently, each item’s ranking number is 1 plus the number of items ranked above it.

Appendix J Prompts

We report all prompt templates used to generate Intial Summaries, Refinement with Feedback, and Refinement without Feedback in Table 8.

J.2 InstructRM Prompts

We instructed one of the authors of this paper (who at the time had not been involved in the research project) to write 5 prompts that would achieve the goal of selecting high-quality summaries, i.e., refinements. The author did not have any domain knowledge or prior information on what kinds of prompts would work. The instructions provided to the author can be viewed here. We report all 5 prompt templates in Table LABEL:tab:instruct_rm_prompts.

J.3 Finetuning Prompts

In Table LABEL:tab:finetuning_prompts, we report the prompts we use for finetuning on summaries and finetuning on feedback + refinements. The completion for finetuning on summaries indicates that we can have completions generated from various sources, i.e., either initial summaries from FeedMe, refinements generated with our method, or ideal human written summaries. For finetuning feedback + refinements, we first generate the feedback and then the refinement.

J.4 Reward Model Prompts