\part{}

\begin{abstract}
  As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task.
  For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about---summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.  We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts \cite{volske2017tl} and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles \cite{hermann2015teaching}, producing summaries nearly as good as the human reference without any news-specific fine-tuning.\footnote{Samples from all of our models can be viewed \href{https://openaipublic.blob.core.windows.net/summarize-from-feedback/website/index.html}{on our website}.}
  We conduct extensive analyses to understand our human feedback dataset and fine-tuned models.\footnote{We provide inference code for our 1.3B models and baselines, as well as a model card,  \href{https://github.com/openai/summarize-from-feedback}{here}. We are not able to release our dataset of human preferences at this time.}
  We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans.
  We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.
\end{abstract}

\section{Introduction}
\label{sec:intro}

Large-scale language model pretraining has become increasingly prevalent for achieving high performance on a variety of natural language processing (NLP) tasks. When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations.

While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective---maximizing the likelihood of human-written text---and what we care about---generating high-quality outputs as determined by humans.
This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g.\@ making up facts \cite{hallucination}) and unimportant errors (e.g.\@ selecting the precise word from a set of synonyms); models are incentivized to place probability mass on all  human demonstrations, including those that are low-quality; and
distributional shift during sampling can degrade performance \cite{generalization, dagger}.
Quality can often be improved significantly by non-uniform sampling strategies such as beam search \cite{reddy1977speech}, but these can lead to repetition and other undesirable artifacts \cite{tradingoff, degeneration}.
Optimizing for quality may be a principled approach to overcoming these problems.

\begin{figure}
\vspace{-10mm}
    \centering
    \includegraphics[width=0.7\linewidth]{final_figs/tldr_main.pdf}
    \caption[Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.]{\label{fig:tldr_main} Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.\footnotemark
    Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in
    Appendix~\ref{apdx:length_control}; length differences explain about a third of the gap between feedback and supervised learning at 6.7B.
    \vspace{-3mm} }

\end{figure}
\footnotetext{Throughout the paper, error bars represent 1 standard error.}

Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community \cite{dorr2003hedge,chopra2016abstractive,rush2015neural,see2017get,ranzato2015sequence}, and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE \cite{lin2004automatic}, have received criticism for poor correlation with human judgments \cite{schluter2017limits,paulus2017deep,chaganty2018price,kryscinski2019neural}.

We follow the works of \cite{bohm2019better,ziegler2019fine}, who fine-tune language models from human feedback using reward learning \cite{leike2018scalable}. We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each `time step', and is updated using the PPO algorithm \cite{schulman2017proximal} based on the RM `reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. Our final dataset consists of 64,832 summary comparisons. We follow the works of \cite{radford2019language,brown2020language} and use large pretrained GPT-3 models with as many as 6.7 billion parameters.

\iffalse
\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-10mm}
    \centering
    \includegraphics[width=\linewidth]{final_figs/tldr_main.pdf}
    \caption[Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.]{\label{fig:tldr_main} Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.\footnotemark
    Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in
    Appendix~\ref{apdx:length_control}; length differences explain about a third of the gap between feedback and supervised learning at 6.7B.
    \vspace{-3mm} }

\end{wrapfigure}
\footnotetext{Throughout the paper, error bars represent 1 standard error.}
\fi

Our main contributions are three-fold.

\textbf{(1) We show that training with human feedback significantly outperforms very strong baselines on English summarization.} When applying our methods on a version of the Reddit TL;DR dataset \cite{volske2017tl}, we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure~\ref{fig:tldr_main}).

\textbf{(2) We show human feedback models generalize much better to new domains than supervised models.} Our Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any news-specific fine-tuning, almost matching the quality of the dataset’s reference summaries.
We perform several checks to ensure that these human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section~\ref{sec:data_quality}), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices~\ref{apdx:length_control} and \ref{sec:agreement_matrices}).

\textbf{(3) We conduct extensive empirical analyses of our policy and reward model.} We examine the impact of model and data size (Figure~\ref{fig:data_size}), study performance as we continue to optimize a given reward model (Section~\ref{sec:overopt}), and analyze reward model performance using synthetic and human-written perturbations of summaries (Section~\ref{sec:rm_evals}). We confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model directly results in better summaries than optimizing ROUGE according to humans (Section \ref{sec:automatic_metrics}).

\iffalse
\paragraph{Main result.} When applying our methods on a version of the Reddit TL;DR dataset \cite{volske2017tl}, we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure~\ref{fig:tldr_main}).
These Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any further fine-tuning, almost matching the quality of the dataset’s reference summaries.
We perform several checks to ensure that these strong human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section~\ref{sec:data_quality}), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices~\ref{apdx:length_control} and \ref{sec:agreement_matrices}).

\paragraph{Additional analysis.} We also examine the impact of model and data size (Figure~\ref{fig:data_size}), study performance as we continue to optimize a given reward model (Section~\ref{sec:overopt}), analyze reward model performance using synthetic and human-written perturbations of summaries (Section~\ref{sec:rm_evals}), and confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences (Section \ref{sec:automatic_metrics}).
\fi

The methods we present in this paper are motivated in part by longer-term concerns about the misalignment of AI systems with what humans want them to do. When misaligned summarization models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems become more powerful and are given increasingly important tasks, the mistakes they make will likely become more subtle and safety-critical, making this an important area for further research.

\section{Related work}
\label{sec:relatedwork}

Most directly related to our work is previous work using human feedback to train summarization models with RL \cite{bohm2019better,ziegler2019fine}. Bohm et al.\@ \cite{bohm2019better} learn a reward function from a dataset of human ratings of 2,500 CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE score.
Our work is most similar to \cite{ziegler2019fine}, who also train Transformer models \cite{vaswani2017attention} to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets.
Unlike us, they train in an online manner and find the model highly extractive.  They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to \cite{ziegler2019fine}, we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks and initializing our models from a policy fine-tuned on the reference summaries.

Human feedback has also been used as a reward to train models in other domains such as dialogue \cite{jaques2019way,yi2019towards,hancock2019learning}, translation  \cite{kreutzer2018can,bahdanau2016actor}, semantic parsing \cite{lawrence2018improving}, story generation \cite{zhou2020learning}, review generation \cite{cho2018towards}, and evidence extraction \cite{perez2019finding}.
Our reward modeling approach was developed in prior work on learning to rank \cite{liu2011learning}, which has been applied to ranking search results using either explicit feedback \cite{bartell1994automatic,fuhr1989optimum} or implicit feedback in the form of click-through data \cite{joachims2002optimizing,joachims2005accurately}.
In a related line of research, human feedback has been used to train agents in simulated environments \cite{christiano2017deep,ibarz2018reward}.
Finally, there is a rich literature on using RL to optimize automatic metrics for NLP tasks,
such as ROUGE for summarization \cite{ranzato2015sequence,wu2018learning,paulus2017deep,dong2018banditsum,gao2019reward}, BLEU for translation \cite{ranzato2015sequence,wu2016google,bahdanau2016actor,nguyen2017reinforcement}, and other domains \cite{tambwekar2018controllable,jaques2017tuning}.

\begin{figure}
    \centering
    \includegraphics[width=1.04\linewidth,trim=0 0 100 0, clip]{final_figs/human-feedback-reward-2.pdf}
    \caption{\label{fig:method}Diagram of our human feedback, reward model training, and policy training procedure. \vspace{-5mm}}
\end{figure}

\section{Method and experiment details}
\label{sec:expdetails}

\subsection{High-level methodology}

Our approach is similar to the one outlined in \cite{ziegler2019fine}, adapted to the batch setting. We start with an initial policy that is fine-tuned via supervised learning on the desired dataset (in our case, the Reddit TL;DR summarization dataset). The process (illustrated in Figure~\ref{fig:method}) then consists of three steps that can be repeated iteratively.

\textbf{Step 1: Collect samples from existing policies and send comparisons to humans.} For each Reddit post, we sample summaries from several sources including the current policy, initial policy, original reference summaries and various baselines. We send a batch of pairs of summaries to our human evaluators, who are tasked with selecting the best summary of a given Reddit post.

\textbf{Step 2: Learn a reward model from human comparisons.} Given a post and a candidate summary, we train a reward model to predict the log odds that this summary is the better one, as judged by our labelers.

\textbf{Step 3: Optimize a policy against the reward model.} We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, specifically with the PPO algorithm \cite{schulman2017proximal}.

We provide a more thorough description of our procedure, including details of the reward model and policy training and our quality control process, in the following sections.  In practice, rather than precisely iterating this sequence of three steps, we updated our data collection and training procedures over the course of the project while accumulating labels (see Appendix~\ref{sec:batch_breakdown} for details).

\subsection{Datasets and task}

\label{sec:dataset}
\paragraph{Datasets.} We use the TL;DR summarization dataset \cite{volske2017tl}, which contains \textasciitilde 3 million posts from \texttt{reddit.com} across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs).
We additionally filter this dataset (see Appendix~\ref{sec:dataset_details}) to ensure quality, including using a whitelist of subreddits that are understandable to the general population.  Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section~\ref{sec:main_results} and Appendix~\ref{apdx:length_control}).
Our final filtered dataset contains 123,169 posts, and we hold out \textasciitilde 5\% as a validation set.  For the remainder of this paper, we refer to this dataset simply as TL;DR.

We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because very strong performance can be attained on CNN/DM with simple extractive baselines.
We find in Section \ref{sec:cnndm} that our labelers prefer lead-3 over the CNN/DM reference summaries,\footnote{We manually check this result in Appendix~\ref{apdx:lead3_vs_ref} and find we generally agree with labeler ratings.} and that the supervised T5 model \cite{raffel2019exploring} with low-temperature sampling already surpasses the reference summary quality, while copying extensively from the article. On the other hand, simple extractive baselines perform poorly on TL;DR in our human evaluations (see Appendix~\ref{apdx:axis_evals}).
Instead of training on CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being trained to summarize Reddit posts.

\input{sample_tables/main}

\paragraph{Task.} We define our ground-truth task as producing a model that generates summaries fewer than 48 tokens long that are as good as possible, according to our judgments. We judge summary quality by how faithfully the summary conveys the original post to a reader who can only read the summary and not the post (see Appendix~\ref{apdx:labeler_instructions} for further discussion of criteria). Since we have limited capacity to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to ensure high agreement between labelers and us on the task, which we describe in the next section.

\subsection{Collecting human feedback}
\label{sec:collectingfeedback}

Previous work on fine-tuning language models from human feedback \cite{ziegler2019fine} reported ``a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated”, leading to model-generated summaries that were high-quality according to the labelers, but fairly low-quality according to the researchers.

Compared to \cite{ziegler2019fine}, we implement two changes to improve human data quality.
First, we transition entirely to the offline setting, where we alternate between sending large batches of comparison data\footnote{Our decision to collect comparisons rather than Likert scores is supported by recent work, e.g.\@ \cite{li2019acute}.} to our human labelers and re-training our models on the cumulative collected data. Second, we maintain a hands-on relationship with labelers:\footnote{We recruited labelers from a freelancing platform, Upwork, and two labeling services, Scale and Lionbridge.} we on-board them with detailed instructions, answer their questions in a shared chat room, and provide regular feedback on their performance. We train all labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher agreement over the course of the project.
See Appendix~\ref{sec:labeler_workflow} and \ref{apdx:labeler_instructions} for details.

As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77\% $\pm$ 2\% of the time, while researchers agree with each other 73\% $\pm$ 4\% of the time. We provide more analysis of our human data quality in Appendix~\ref{sec:data_quality}.

\subsection{Models}
\label{sec:models}
All of our models are Transformer decoders \cite{vaswani2017attention} in the style of GPT-3 \cite{radford2018improving,brown2020language}. We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters.

\paragraph{Pretrained models.} Similarly to \cite{dai2015semi,radford2018improving}, we start with models pretrained to autoregressively predict the next token in a large text corpus.   As in \cite{radford2019language,brown2020language}, we use these models as ‘zero-shot’ baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix~\ref{sec:training_details}, and on our zero-shot procedure in Appendix~\ref{sec:input_format}.

\paragraph{Supervised baselines.} We next fine-tune these models via supervised learning to predict summaries from our filtered TL;DR dataset (see Appendix~\ref{sec:training_details} for details). We use these supervised models to sample initial summaries for collecting comparisons, to initialize our policy and reward models, and as baselines for evaluation.
In our final human evaluations, we use T=0 to sample from all models, as we found it performed better than higher temperatures or nucleus sampling (see Appendix \ref{sec:hyperparameters}).

To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models \cite{zhang2019abstract} from mid-2019 (see Appendix~\ref{sec:rouge_graphs}).

\paragraph{Reward models.}
To train our reward models, we start from a supervised baseline, as described above, then add a randomly initialized linear head that outputs a scalar value. We train this model to predict which summary $y \in \left\{y_0, y_1\right\}$ is better as judged by a human, given a post $x$.  If the summary preferred by the human is $y_i$, we can write the RM loss as:
$$
\text{loss}(r_\theta) = E_{(x, y_0, y_1, i) \sim D} [\log ( \sigma( r_\theta(x, y_i) - r_{\theta}(x, y_{1-i} ) )) ]
$$
where $r_\theta(x,y)$ is the scalar output of the reward model for post $x$ and summary $y$ with parameters $\theta$, and $D$ is the dataset of human judgments.
At the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0.

\paragraph{Human feedback policies.}

We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm \cite{schulman2017proximal}, where each time step is a BPE token. We initialize our policy to be the model fine-tuned on Reddit TL;DR.  Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy $\pi^{\text{RL}}_\phi$ with parameters $\phi$ and this original supervised model $\pi^{\text{SFT}}$. The full reward $R$ can be written as:
$$
R(x, y) = r_\theta(x, y) - \beta \log [ \pi^{\text{RL}}_\phi (y|x) / \pi^{\text{SFT}} (y|x) ]
$$
This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn’t learn to produce outputs that are too different from those that the reward model has seen during training.

For the PPO value function, we use a Transformer with completely separate parameters from the policy. This prevents updates to the value function from partially destroying the pretrained policy early in training (see ablation in Appendix~\ref{sec:value_function_ablations}).
We initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy, and value function are the same size.

\section{Results}
\label{sec:results}

\subsection{Summarizing Reddit posts from human feedback} \label{sec:main_results}

\paragraph{Policies trained with human feedback are preferred to much larger supervised policies.}

Our main results evaluating our human feedback policies on TL;DR are shown in Figure~\ref{fig:tldr_main}. We measure policy quality as the percentage of summaries generated by that policy that humans prefer over the reference summaries in the dataset. Our policies trained with human feedback significantly outperform our supervised baselines on this metric, with our 1.3B human feedback model significantly outperforming a supervised model 10$\times$ its size (61\% versus 43\% raw preference score against reference summaries). Our 6.7B model in turn significantly outperforms our 1.3B model, suggesting that training with human feedback also benefits from scale. Additionally, both of our human feedback models are judged by humans to be superior to the human demonstrations used in the dataset.

\paragraph{Controlling for summary length.} When judging summary quality, summary length is a confounding factor. The target length of a summary is implicitly part of the summarization task; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Since our models learned to generate longer summaries, length could account for much of our quality improvements. We find that after controlling for length (Appendix~\ref{apdx:length_control}), the preference of our human feedback models vs.\@ reference summaries drops by \textasciitilde5\%; even so, our 6.7B model summaries are still preferred to the reference summaries \textasciitilde65\% of the time.

\paragraph{How do our policies improve over the baselines?}
\label{paragraph:axis_evals}

\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-5mm}
    \centering
    \includegraphics[width=\linewidth]{final_figs/tldr_axis.png}
    \caption{Evaluations of four axes of summary quality on the TL;DR dataset.}
    \label{fig:tldr_axis_evals}
    \vspace{-9mm}
\end{wrapfigure}
To better understand the quality of our models’ summaries compared to the reference summaries and those of our supervised baselines, we conduct an additional analysis where human labelers assess summary quality across four dimensions (or ``axes'') using a 7-point Likert scale \cite{likert1932technique}. Labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.

The results (Figure~\ref{fig:tldr_axis_evals}) indicate that our human feedback models outperform the supervised baselines across every dimension of quality, but particularly coverage.
Although our human labelers had a high bar for giving perfect overall scores, summaries from our 6.7B PPO model achieve a 7/7 overall score 45\% of the time (compared to 20\% and 23\% for the 6.7B supervised baseline and reference summaries, respectively).

\subsection{Transfer to summarizing news articles}
\label{sec:cnndm}

\begin{figure}
    \centering
    \begin{subfigure}{.49\textwidth}
      \centering
      \includegraphics[width=\linewidth]{final_figs/cnndm_main.pdf}
      \caption{}
      \label{fig:cnndm_main}
    \end{subfigure}

    \begin{subfigure}{.49\textwidth}
      \centering

      \includegraphics[width=\linewidth]{final_figs/lengthbinned_evals.png}
      \caption{}
      \label{fig:cnndm_lengthbin}
    \end{subfigure}
    \caption{Transfer results on CNN/DM. (a) Overall summary quality on CNN/DM as a function of model size. Full results across axes shown in Appendix \ref{apdx:axis_evals}. (b) Overall scores vs.\@ length for the 6.7B TL;DR supervised baseline, the 6.7B TL;DR human feedback model, and T5 fine-tuned on CNN/DM summaries. At similar summary lengths, our 6.7B TL;DR human feedback model nearly matches T5 despite never being trained to summarize news articles. \vspace{-5mm}}
    \label{fig:cnndm_results}
\end{figure}

Our human feedback models can also generate excellent summaries of CNN/DM news articles without any further training (Figure~\ref{fig:cnndm_results}).
Our human feedback models significantly outperform models trained via supervised learning on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model performs almost as well as a 6.7B model that was fine-tuned on the CNN/DM reference summaries, despite generating much shorter summaries.

Since our human feedback models transferred to CNN/DM  have little overlap in summary length distribution with models trained on CNN/DM, with about half as many tokens on average, they are difficult to compare directly. Thus our evaluations in Figure~\ref{fig:cnndm_results} use a 7-point Likert scale on four quality dimensions, as in Section~\ref{sec:main_results} (see Appendix~\ref{apdx:labeler_instructions} for labeler instructions).
In Figure~\ref{fig:cnndm_lengthbin} we show the average overall score at different summary lengths, which suggests our human feedback models would perform even better if they generated longer summaries.
Qualitatively, CNN/DM summaries from our human feedback models are consistently fluent and reasonable representations of the article; we show examples on  \href{https://openaipublic.blob.core.windows.net/summarize-from-feedback/website/index.html}{our website} and in Appendix~\ref{apdx:samples}.

\subsection{Understanding the reward model}
\label{sec:understanding_rm}

\paragraph{What happens as we optimize the reward model?}
\label{sec:overopt}

\iffalse
\begin{figure}
    \centering

    \caption{How reward modeling performance scales with data and model size. Doubling amount of training data gives a \textasciitilde 1\% increase in reward model validation accuracy, whereas doubling model size gives a \textasciitilde 2\% increase---a linear predictor of validation accuracy from log dataset size and log model size has coefficients of 1.1\% and 1.8\% respectively.
    \label{fig:data_size}
}
    \label{fig:data_abl}
\end{figure}
\fi

Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model isn’t a perfect representation of our labeler preferences, as it has limited capacity and only sees a small amount of comparison data from a relatively narrow distribution of summaries.
While we can hope our reward model generalizes to summaries unseen during training, it’s unclear how much one can optimize against the reward model until it starts giving useless evaluations.

To answer this question, we created a range of policies optimized against an earlier version of our reward model, with varying degrees of optimization strength, and asked labelers to compare samples from them to the reference summaries. Figure~\ref{fig:overopt} shows the results for  PPO at a range of KL penalty coefficients ($\beta$). Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note that this over-optimization also happens with ROUGE (see \cite{paulus2017deep} and Appendix \ref{apdx:bon_overopt}). Similar behavior has been observed in learned reward functions in the robotics domain \cite{cabi2019scaling}.

\begin{figure}
\centering
\begin{minipage}{.48\textwidth}
  \centering
  \includegraphics[width=\linewidth]{final_figs/newoveropt.pdf}
  \captionof{figure}{Preference scores versus degree of reward model optimization.

    Optimizing against the reward model initially improves summaries, but eventually overfits, giving worse summaries.
    This figure uses an earlier version of our reward model (see rm3 in Appendix~\ref{sec:batch_breakdown}).
    See Appendix \ref{apdx:overopt_samples} for samples from the KL 250 model. \vspace{-5mm}}
  \label{fig:overopt}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}{.47\textwidth}
  \centering
  \includegraphics[width=\linewidth]{final_figs/data_size_abl.pdf}
  \captionof{figure}{Reward model performance versus data size and model size. Doubling amount of training data leads to a \textasciitilde 1.1\% increase in reward model validation accuracy, whereas doubling the model size leads to a \textasciitilde 1.8\% increase. The 6.7B model trained on all data begins approaching the accuracy of a single human. \vspace{-5mm}}

  \label{fig:data_size}
\end{minipage}
\end{figure}

\paragraph{How does reward modeling scale with increasing model and data size?}
\label{sec:rm_scaling}

We conduct an ablation to determine how data quantity and model size affect reward modeling performance. We train 7 reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our dataset. We find that doubling the training data amount leads to a \textasciitilde 1.1\% increase in the reward model validation set accuracy, whereas doubling the model size leads to a \textasciitilde 1.8\% increase (Figure~\ref{fig:data_size}).

\paragraph{What has the reward model learned?}
\label{sec:rm_evals}

We probe our reward model by evaluating it on several validation sets. We show the full results in Appendix~\ref{sec:rm_validation}, and highlight them here.
We find that our reward models generalize to evaluating CNN/DM summaries (Appendix \ref{sec:agreement_matrices}), agreeing with labeler preferences 62.4\% and 66.5\% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B reward model nearly matches the inter-labeler agreement value of 66.9\%.

We also find that our reward models are sensitive to small but semantically important details in the summary. We construct an additional validation set by having labelers make minimal edits to summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4\% for 1.3B and 82.8\% for 6.7B) as a separate set of human evaluators (84.1\%). Further, when comparing the reference summaries to perturbed summaries where the participants' roles are reversed, our models reliably select the original summary (92.9\% of the time for 1.3B, 97.2\% for 6.7B).
However, our RMs are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary shorter only 62.6\% of the time (vs.\@ 76.4\% for humans).

\begin{figure}
    \centering
    \includegraphics[width=0.6\linewidth]{final_figs/bon_overopt.png}
    \caption{
Summary quality as a function of metric optimized and amount of optimization, using best-of-N rejection sampling.  We evaluate ROUGE, our main reward models, and an earlier iteration of the 1.3B model trained on approximately 75\% as much data (see Table \ref{tab:batch_data} for details).  Though the data is noisy, ROUGE appears to peak both sooner and at a substantially lower preference rate than all reward models.  Details in Appendix \ref{apdx:bon_overopt}.\vspace{-5mm}
}
    \label{fig:bon_overopt}
\end{figure}
\subsection{Analyzing automatic metrics for summarization}
\label{sec:automatic_metrics}

\paragraph{Evaluation.}

We study how well various automatic metrics act as predictors for human preferences, and compare them to our RMs. Specifically, we examine ROUGE, summary length, amount of copying from the post,\footnote{We measure copying by computing the longest common subsequence of bigrams with the original Reddit post or news article, and dividing by the number of bigrams in the summary.} and log probability under our baseline supervised models. We present a full matrix of agreement rates between these metrics in Appendix~\ref{sec:agreement_matrices}.

We find that our learned reward models consistently outperform other metrics, even on the CNN/DM dataset on which it was never trained.
We also find that ROUGE fails to track sample quality as our models improve. While ROUGE has \textasciitilde57\% agreement with labelers when comparing samples from our supervised baseline models, this drops to \textasciitilde50\% for samples from our human feedback model.

Similarly, log probability agreement with humans drops to below 50\% on comparisons between samples from our human feedback models, while our RMs still perform above chance (62\%). Scaling up the size of the supervised model does not reliably improve log probability's agreement with labelers.

\paragraph{Optimization.}

In Figure \ref{fig:bon_overopt}, we show that optimizing ROUGE using a simple optimization scheme doesn't consistently increase quality, as has been noted in \cite{paulus2017deep}.  Optimization against ROUGE peaks both sooner and at a substantially lower quality rate than optimization against our reward models.

\section{Discussion}

\paragraph{Limitations.}
One limitation of our work is the time and cost required to produce our final models. Notably, fine-tuning our 6.7B model with RL required approximately 320 GPU-days.
Our data collection procedure is also expensive compared to prior work ---
the training set took thousands of labeler hours
and required significant researcher time to ensure quality.
For this reason, we were unable to collect an equivalent amount of high-quality human demonstrations for training a better supervised baseline; our current baseline has seen strictly less human data than our RL models. We leave this ablation to future work.
Nevertheless, we believe
reward modeling is
more likely to scale to tasks where it is extremely skill-intensive or time-consuming to provide good demonstrations.

\paragraph{Future directions.}
The methods in this paper could be applied to any task where humans can compare samples, including dialogue, machine translation, question answering, speech synthesis,
and music generation. We expect this method to be particularly important for generating long samples, where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It may be possible to improve sample efficiency by training to predict feedback across many tasks~\cite{mccann2018natural}.

We are particularly interested in scaling human feedback to tasks where humans can’t easily evaluate the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML system is aligned with the human designer’s intentions. One approach is to train ML systems to help humans perform the evaluation task quickly and accurately \cite{christiano2018supervising}.

There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models \cite{unifying, fidler2017teaching, niu2018polite, welleck2019neural}. For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies.

\paragraph{Broader impacts.}

The techniques we explore in this paper are generic techniques that could be used in a wide variety of machine learning applications, for any task where it is feasible for humans to evaluate the quality of model outputs. Thus, the potential implications are quite broad.

Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences.
Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait \cite{covington2016deep}.  In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being.

In the long term, as machine learning systems become more capable it will likely become increasingly difficult to ensure that they are behaving safely: the mistakes they make might be more difficult to spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a news article is both easy to notice (one simply has to read the original article) and has fairly low consequences. On the other hand, imitating human driving may be substantially less safe than driving to optimize human preferences.  We believe that the techniques we explore in this paper are promising steps towards mitigating the risks from such capable systems, and better aligning them with what humans care about.

Unfortunately, our techniques also enable malicious actors to more easily train models that cause societal harm. For instance, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs, or to induce dependence of humans on the technology, or to generate large amounts of toxic or hurtful content intended to harm specific individuals. Avoiding these outcomes is a significant challenge for which there are few obvious solutions.

Large-scale models trained with human feedback could have significant impacts on many groups. Thus, it is important to be careful about how we define the `good' model behavior that human labelers will reinforce.
Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the ‘gold standard’; rather, individuals from groups impacted by the technology should be included in the process to define `good' behavior, and hired as labelers to reinforce this behavior in the model.

We chose to train on the Reddit TL;DR dataset because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they often contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content. For this reason, we recommend that the potential harms of our models be thoroughly studied before deploying them in user-facing applications.

Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.

\subsection*{Acknowledgements}

We'd like to thank Beth Barnes for help with labeler hiring and general encouragement; Geoffrey Irving for guidance on earlier iterations of the project and inspiring conversations; Ben Mann, Tom Brown, Nick Ryder, and Melanie Subbiah for training and evaluating our pretrained models; Chris Hesse, Eric Sigler, Benjamin Chess, Christopher Berner, Clemens Winter, Mateusz Litwin, and many others for supporting us through computing infrastructure improvements and maintenance; Scott Gray for writing fast GPU kernels; Arvind Neelakantan and Wojciech Kryscinski for discussions on how to present the work, experiment design, and what datasets to use; Shan Carter for help designing the main diagram; Douwe Kiela, Zach Lipton, and Alex Irpan for providing feedback on the paper; and Gretchen Krueger for co-writing the model card accompanying the paper.

Finally, we'd like to thank all of our contractors for providing the data that was essential for training the models in this paper: Emill Jayson Caypuno, Rachelle Froyalde, Cyra Denura, Alex Malek, Isik Agil, Reshmi Patel, William Yap, Natalie Silver, Erol Akbaba, Jennifer Brillo, Alexandra Uifalean, Morris Stuttard, Russell Bernandez, Tasmai Dave, Rachel Wallace, Jenny Fletcher, Jian Ouyang, Justin Dill, Maria Orzek, Megan Niffenegger, William Sells, Emily Mariner, Andrew Seely, Lychelle Ignacio, Jelena Ostojic, Nhan Tran, Purev Batdelgar, Valentina Kezic, Michelle Wilkerson, Kelly Guerrero, Heather Scott, Sarah Mulligan, Gabriel Ricafrente, Kara Bell, Gabriel Perez, and Alfred Lee.

\iffalse

- Where our approach fits in (technique that could be used for a wide variety of applications)

The good:
- better alignment with designer intentions
- short term --- can lead to better mitigation of
- long term --- uncertain, but can lead to a better outcome for powerful systems

The bad:
- potential for malicious actors to more explicitly target what they care about (bad things)
- for example, could fine-tune a system to be extra manipulative, or extra hurtful (trolls)
- automating summarization replacing jobs

Original jeff impact:
Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait \cite{covington2016deep}.  Improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human intent.  This could be used to align an algorithm with either benign or malicious human intent.  As an example of something potentially harmful to society, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs.

TODO longer term stuff?

TODO prose
\fi

\bibliographystyle{abbrv}

\part{}

\begin{abstract}
  As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task.
  For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about---summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.  We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts \cite{volske2017tl} and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles \cite{hermann2015teaching}, producing summaries nearly as good as the human reference without any news-specific fine-tuning.\footnote{Samples from all of our models can be viewed \href{https://openaipublic.blob.core.windows.net/summarize-from-feedback/website/index.html}{on our website}.}
  We conduct extensive analyses to understand our human feedback dataset and fine-tuned models.\footnote{We provide inference code for our 1.3B models and baselines, as well as a model card,  \href{https://github.com/openai/summarize-from-feedback}{here}. We are not able to release our dataset of human preferences at this time.}
  We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans.
  We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.
\end{abstract}

\section{Introduction}
\label{sec:intro}

Large-scale language model pretraining has become increasingly prevalent for achieving high performance on a variety of natural language processing (NLP) tasks. When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations.

While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective---maximizing the likelihood of human-written text---and what we care about---generating high-quality outputs as determined by humans.
This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g.\@ making up facts \cite{hallucination}) and unimportant errors (e.g.\@ selecting the precise word from a set of synonyms); models are incentivized to place probability mass on all  human demonstrations, including those that are low-quality; and
distributional shift during sampling can degrade performance \cite{generalization, dagger}.
Quality can often be improved significantly by non-uniform sampling strategies such as beam search \cite{reddy1977speech}, but these can lead to repetition and other undesirable artifacts \cite{tradingoff, degeneration}.
Optimizing for quality may be a principled approach to overcoming these problems.

\begin{figure}
\vspace{-10mm}
    \centering
    \includegraphics[width=0.7\linewidth]{final_figs/tldr_main.pdf}
    \caption[Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.]{\label{fig:tldr_main} Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.\footnotemark
    Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in
    Appendix~\ref{apdx:length_control}; length differences explain about a third of the gap between feedback and supervised learning at 6.7B.
    \vspace{-3mm} }

\end{figure}
\footnotetext{Throughout the paper, error bars represent 1 standard error.}

Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community \cite{dorr2003hedge,chopra2016abstractive,rush2015neural,see2017get,ranzato2015sequence}, and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE \cite{lin2004automatic}, have received criticism for poor correlation with human judgments \cite{schluter2017limits,paulus2017deep,chaganty2018price,kryscinski2019neural}.

We follow the works of \cite{bohm2019better,ziegler2019fine}, who fine-tune language models from human feedback using reward learning \cite{leike2018scalable}. We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each `time step', and is updated using the PPO algorithm \cite{schulman2017proximal} based on the RM `reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. Our final dataset consists of 64,832 summary comparisons. We follow the works of \cite{radford2019language,brown2020language} and use large pretrained GPT-3 models with as many as 6.7 billion parameters.

\iffalse
\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-10mm}
    \centering
    \includegraphics[width=\linewidth]{final_figs/tldr_main.pdf}
    \caption[Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.]{\label{fig:tldr_main} Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.\footnotemark
    Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in
    Appendix~\ref{apdx:length_control}; length differences explain about a third of the gap between feedback and supervised learning at 6.7B.
    \vspace{-3mm} }

\end{wrapfigure}
\footnotetext{Throughout the paper, error bars represent 1 standard error.}
\fi

Our main contributions are three-fold.

\textbf{(1) We show that training with human feedback significantly outperforms very strong baselines on English summarization.} When applying our methods on a version of the Reddit TL;DR dataset \cite{volske2017tl}, we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure~\ref{fig:tldr_main}).

\textbf{(2) We show human feedback models generalize much better to new domains than supervised models.} Our Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any news-specific fine-tuning, almost matching the quality of the dataset’s reference summaries.
We perform several checks to ensure that these human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section~\ref{sec:data_quality}), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices~\ref{apdx:length_control} and \ref{sec:agreement_matrices}).

\textbf{(3) We conduct extensive empirical analyses of our policy and reward model.} We examine the impact of model and data size (Figure~\ref{fig:data_size}), study performance as we continue to optimize a given reward model (Section~\ref{sec:overopt}), and analyze reward model performance using synthetic and human-written perturbations of summaries (Section~\ref{sec:rm_evals}). We confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model directly results in better summaries than optimizing ROUGE according to humans (Section \ref{sec:automatic_metrics}).

\iffalse
\paragraph{Main result.} When applying our methods on a version of the Reddit TL;DR dataset \cite{volske2017tl}, we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure~\ref{fig:tldr_main}).
These Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any further fine-tuning, almost matching the quality of the dataset’s reference summaries.
We perform several checks to ensure that these strong human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section~\ref{sec:data_quality}), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices~\ref{apdx:length_control} and \ref{sec:agreement_matrices}).

\paragraph{Additional analysis.} We also examine the impact of model and data size (Figure~\ref{fig:data_size}), study performance as we continue to optimize a given reward model (Section~\ref{sec:overopt}), analyze reward model performance using synthetic and human-written perturbations of summaries (Section~\ref{sec:rm_evals}), and confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences (Section \ref{sec:automatic_metrics}).
\fi

The methods we present in this paper are motivated in part by longer-term concerns about the misalignment of AI systems with what humans want them to do. When misaligned summarization models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems become more powerful and are given increasingly important tasks, the mistakes they make will likely become more subtle and safety-critical, making this an important area for further research.

\section{Related work}
\label{sec:relatedwork}

Most directly related to our work is previous work using human feedback to train summarization models with RL \cite{bohm2019better,ziegler2019fine}. Bohm et al.\@ \cite{bohm2019better} learn a reward function from a dataset of human ratings of 2,500 CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE score.
Our work is most similar to \cite{ziegler2019fine}, who also train Transformer models \cite{vaswani2017attention} to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets.
Unlike us, they train in an online manner and find the model highly extractive.  They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to \cite{ziegler2019fine}, we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks and initializing our models from a policy fine-tuned on the reference summaries.

Human feedback has also been used as a reward to train models in other domains such as dialogue \cite{jaques2019way,yi2019towards,hancock2019learning}, translation  \cite{kreutzer2018can,bahdanau2016actor}, semantic parsing \cite{lawrence2018improving}, story generation \cite{zhou2020learning}, review generation \cite{cho2018towards}, and evidence extraction \cite{perez2019finding}.
Our reward modeling approach was developed in prior work on learning to rank \cite{liu2011learning}, which has been applied to ranking search results using either explicit feedback \cite{bartell1994automatic,fuhr1989optimum} or implicit feedback in the form of click-through data \cite{joachims2002optimizing,joachims2005accurately}.
In a related line of research, human feedback has been used to train agents in simulated environments \cite{christiano2017deep,ibarz2018reward}.
Finally, there is a rich literature on using RL to optimize automatic metrics for NLP tasks,
such as ROUGE for summarization \cite{ranzato2015sequence,wu2018learning,paulus2017deep,dong2018banditsum,gao2019reward}, BLEU for translation \cite{ranzato2015sequence,wu2016google,bahdanau2016actor,nguyen2017reinforcement}, and other domains \cite{tambwekar2018controllable,jaques2017tuning}.

\begin{figure}
    \centering
    \includegraphics[width=1.04\linewidth,trim=0 0 100 0, clip]{final_figs/human-feedback-reward-2.pdf}
    \caption{\label{fig:method}Diagram of our human feedback, reward model training, and policy training procedure. \vspace{-5mm}}
\end{figure}

\section{Method and experiment details}
\label{sec:expdetails}

\subsection{High-level methodology}

Our approach is similar to the one outlined in \cite{ziegler2019fine}, adapted to the batch setting. We start with an initial policy that is fine-tuned via supervised learning on the desired dataset (in our case, the Reddit TL;DR summarization dataset). The process (illustrated in Figure~\ref{fig:method}) then consists of three steps that can be repeated iteratively.

\textbf{Step 1: Collect samples from existing policies and send comparisons to humans.} For each Reddit post, we sample summaries from several sources including the current policy, initial policy, original reference summaries and various baselines. We send a batch of pairs of summaries to our human evaluators, who are tasked with selecting the best summary of a given Reddit post.

\textbf{Step 2: Learn a reward model from human comparisons.} Given a post and a candidate summary, we train a reward model to predict the log odds that this summary is the better one, as judged by our labelers.

\textbf{Step 3: Optimize a policy against the reward model.} We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, specifically with the PPO algorithm \cite{schulman2017proximal}.

We provide a more thorough description of our procedure, including details of the reward model and policy training and our quality control process, in the following sections.  In practice, rather than precisely iterating this sequence of three steps, we updated our data collection and training procedures over the course of the project while accumulating labels (see Appendix~\ref{sec:batch_breakdown} for details).

\subsection{Datasets and task}

\label{sec:dataset}
\paragraph{Datasets.} We use the TL;DR summarization dataset \cite{volske2017tl}, which contains \textasciitilde 3 million posts from \texttt{reddit.com} across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs).
We additionally filter this dataset (see Appendix~\ref{sec:dataset_details}) to ensure quality, including using a whitelist of subreddits that are understandable to the general population.  Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section~\ref{sec:main_results} and Appendix~\ref{apdx:length_control}).
Our final filtered dataset contains 123,169 posts, and we hold out \textasciitilde 5\% as a validation set.  For the remainder of this paper, we refer to this dataset simply as TL;DR.

We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because very strong performance can be attained on CNN/DM with simple extractive baselines.
We find in Section \ref{sec:cnndm} that our labelers prefer lead-3 over the CNN/DM reference summaries,\footnote{We manually check this result in Appendix~\ref{apdx:lead3_vs_ref} and find we generally agree with labeler ratings.} and that the supervised T5 model \cite{raffel2019exploring} with low-temperature sampling already surpasses the reference summary quality, while copying extensively from the article. On the other hand, simple extractive baselines perform poorly on TL;DR in our human evaluations (see Appendix~\ref{apdx:axis_evals}).
Instead of training on CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being trained to summarize Reddit posts.

\input{sample_tables/main}

\paragraph{Task.} We define our ground-truth task as producing a model that generates summaries fewer than 48 tokens long that are as good as possible, according to our judgments. We judge summary quality by how faithfully the summary conveys the original post to a reader who can only read the summary and not the post (see Appendix~\ref{apdx:labeler_instructions} for further discussion of criteria). Since we have limited capacity to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to ensure high agreement between labelers and us on the task, which we describe in the next section.

\subsection{Collecting human feedback}
\label{sec:collectingfeedback}

Previous work on fine-tuning language models from human feedback \cite{ziegler2019fine} reported ``a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated”, leading to model-generated summaries that were high-quality according to the labelers, but fairly low-quality according to the researchers.

Compared to \cite{ziegler2019fine}, we implement two changes to improve human data quality.
First, we transition entirely to the offline setting, where we alternate between sending large batches of comparison data\footnote{Our decision to collect comparisons rather than Likert scores is supported by recent work, e.g.\@ \cite{li2019acute}.} to our human labelers and re-training our models on the cumulative collected data. Second, we maintain a hands-on relationship with labelers:\footnote{We recruited labelers from a freelancing platform, Upwork, and two labeling services, Scale and Lionbridge.} we on-board them with detailed instructions, answer their questions in a shared chat room, and provide regular feedback on their performance. We train all labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher agreement over the course of the project.
See Appendix~\ref{sec:labeler_workflow} and \ref{apdx:labeler_instructions} for details.

As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77\% $\pm$ 2\% of the time, while researchers agree with each other 73\% $\pm$ 4\% of the time. We provide more analysis of our human data quality in Appendix~\ref{sec:data_quality}.

\subsection{Models}
\label{sec:models}
All of our models are Transformer decoders \cite{vaswani2017attention} in the style of GPT-3 \cite{radford2018improving,brown2020language}. We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters.

\paragraph{Pretrained models.} Similarly to \cite{dai2015semi,radford2018improving}, we start with models pretrained to autoregressively predict the next token in a large text corpus.   As in \cite{radford2019language,brown2020language}, we use these models as ‘zero-shot’ baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix~\ref{sec:training_details}, and on our zero-shot procedure in Appendix~\ref{sec:input_format}.

\paragraph{Supervised baselines.} We next fine-tune these models via supervised learning to predict summaries from our filtered TL;DR dataset (see Appendix~\ref{sec:training_details} for details). We use these supervised models to sample initial summaries for collecting comparisons, to initialize our policy and reward models, and as baselines for evaluation.
In our final human evaluations, we use T=0 to sample from all models, as we found it performed better than higher temperatures or nucleus sampling (see Appendix \ref{sec:hyperparameters}).

To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models \cite{zhang2019abstract} from mid-2019 (see Appendix~\ref{sec:rouge_graphs}).

\paragraph{Reward models.}
To train our reward models, we start from a supervised baseline, as described above, then add a randomly initialized linear head that outputs a scalar value. We train this model to predict which summary $y \in \left\{y_0, y_1\right\}$ is better as judged by a human, given a post $x$.  If the summary preferred by the human is $y_i$, we can write the RM loss as:
$$
\text{loss}(r_\theta) = E_{(x, y_0, y_1, i) \sim D} [\log ( \sigma( r_\theta(x, y_i) - r_{\theta}(x, y_{1-i} ) )) ]
$$
where $r_\theta(x,y)$ is the scalar output of the reward model for post $x$ and summary $y$ with parameters $\theta$, and $D$ is the dataset of human judgments.
At the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0.

\paragraph{Human feedback policies.}

We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm \cite{schulman2017proximal}, where each time step is a BPE token. We initialize our policy to be the model fine-tuned on Reddit TL;DR.  Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy $\pi^{\text{RL}}_\phi$ with parameters $\phi$ and this original supervised model $\pi^{\text{SFT}}$. The full reward $R$ can be written as:
$$
R(x, y) = r_\theta(x, y) - \beta \log [ \pi^{\text{RL}}_\phi (y|x) / \pi^{\text{SFT}} (y|x) ]
$$
This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn’t learn to produce outputs that are too different from those that the reward model has seen during training.

For the PPO value function, we use a Transformer with completely separate parameters from the policy. This prevents updates to the value function from partially destroying the pretrained policy early in training (see ablation in Appendix~\ref{sec:value_function_ablations}).
We initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy, and value function are the same size.

\section{Results}
\label{sec:results}

\subsection{Summarizing Reddit posts from human feedback} \label{sec:main_results}

\paragraph{Policies trained with human feedback are preferred to much larger supervised policies.}

Our main results evaluating our human feedback policies on TL;DR are shown in Figure~\ref{fig:tldr_main}. We measure policy quality as the percentage of summaries generated by that policy that humans prefer over the reference summaries in the dataset. Our policies trained with human feedback significantly outperform our supervised baselines on this metric, with our 1.3B human feedback model significantly outperforming a supervised model 10$\times$ its size (61\% versus 43\% raw preference score against reference summaries). Our 6.7B model in turn significantly outperforms our 1.3B model, suggesting that training with human feedback also benefits from scale. Additionally, both of our human feedback models are judged by humans to be superior to the human demonstrations used in the dataset.

\paragraph{Controlling for summary length.} When judging summary quality, summary length is a confounding factor. The target length of a summary is implicitly part of the summarization task; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Since our models learned to generate longer summaries, length could account for much of our quality improvements. We find that after controlling for length (Appendix~\ref{apdx:length_control}), the preference of our human feedback models vs.\@ reference summaries drops by \textasciitilde5\%; even so, our 6.7B model summaries are still preferred to the reference summaries \textasciitilde65\% of the time.

\paragraph{How do our policies improve over the baselines?}
\label{paragraph:axis_evals}

\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-5mm}
    \centering
    \includegraphics[width=\linewidth]{final_figs/tldr_axis.png}
    \caption{Evaluations of four axes of summary quality on the TL;DR dataset.}
    \label{fig:tldr_axis_evals}
    \vspace{-9mm}
\end{wrapfigure}
To better understand the quality of our models’ summaries compared to the reference summaries and those of our supervised baselines, we conduct an additional analysis where human labelers assess summary quality across four dimensions (or ``axes'') using a 7-point Likert scale \cite{likert1932technique}. Labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.

The results (Figure~\ref{fig:tldr_axis_evals}) indicate that our human feedback models outperform the supervised baselines across every dimension of quality, but particularly coverage.
Although our human labelers had a high bar for giving perfect overall scores, summaries from our 6.7B PPO model achieve a 7/7 overall score 45\% of the time (compared to 20\% and 23\% for the 6.7B supervised baseline and reference summaries, respectively).

\subsection{Transfer to summarizing news articles}
\label{sec:cnndm}

\begin{figure}
    \centering
    \begin{subfigure}{.49\textwidth}
      \centering
      \includegraphics[width=\linewidth]{final_figs/cnndm_main.pdf}
      \caption{}
      \label{fig:cnndm_main}
    \end{subfigure}

    \begin{subfigure}{.49\textwidth}
      \centering

      \includegraphics[width=\linewidth]{final_figs/lengthbinned_evals.png}
      \caption{}
      \label{fig:cnndm_lengthbin}
    \end{subfigure}
    \caption{Transfer results on CNN/DM. (a) Overall summary quality on CNN/DM as a function of model size. Full results across axes shown in Appendix \ref{apdx:axis_evals}. (b) Overall scores vs.\@ length for the 6.7B TL;DR supervised baseline, the 6.7B TL;DR human feedback model, and T5 fine-tuned on CNN/DM summaries. At similar summary lengths, our 6.7B TL;DR human feedback model nearly matches T5 despite never being trained to summarize news articles. \vspace{-5mm}}
    \label{fig:cnndm_results}
\end{figure}

Our human feedback models can also generate excellent summaries of CNN/DM news articles without any further training (Figure~\ref{fig:cnndm_results}).
Our human feedback models significantly outperform models trained via supervised learning on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model performs almost as well as a 6.7B model that was fine-tuned on the CNN/DM reference summaries, despite generating much shorter summaries.

Since our human feedback models transferred to CNN/DM  have little overlap in summary length distribution with models trained on CNN/DM, with about half as many tokens on average, they are difficult to compare directly. Thus our evaluations in Figure~\ref{fig:cnndm_results} use a 7-point Likert scale on four quality dimensions, as in Section~\ref{sec:main_results} (see Appendix~\ref{apdx:labeler_instructions} for labeler instructions).
In Figure~\ref{fig:cnndm_lengthbin} we show the average overall score at different summary lengths, which suggests our human feedback models would perform even better if they generated longer summaries.
Qualitatively, CNN/DM summaries from our human feedback models are consistently fluent and reasonable representations of the article; we show examples on  \href{https://openaipublic.blob.core.windows.net/summarize-from-feedback/website/index.html}{our website} and in Appendix~\ref{apdx:samples}.

\subsection{Understanding the reward model}
\label{sec:understanding_rm}

\paragraph{What happens as we optimize the reward model?}
\label{sec:overopt}

\iffalse
\begin{figure}
    \centering

    \caption{How reward modeling performance scales with data and model size. Doubling amount of training data gives a \textasciitilde 1\% increase in reward model validation accuracy, whereas doubling model size gives a \textasciitilde 2\% increase---a linear predictor of validation accuracy from log dataset size and log model size has coefficients of 1.1\% and 1.8\% respectively.
    \label{fig:data_size}
}
    \label{fig:data_abl}
\end{figure}
\fi

Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model isn’t a perfect representation of our labeler preferences, as it has limited capacity and only sees a small amount of comparison data from a relatively narrow distribution of summaries.
While we can hope our reward model generalizes to summaries unseen during training, it’s unclear how much one can optimize against the reward model until it starts giving useless evaluations.

To answer this question, we created a range of policies optimized against an earlier version of our reward model, with varying degrees of optimization strength, and asked labelers to compare samples from them to the reference summaries. Figure~\ref{fig:overopt} shows the results for  PPO at a range of KL penalty coefficients ($\beta$). Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note that this over-optimization also happens with ROUGE (see \cite{paulus2017deep} and Appendix \ref{apdx:bon_overopt}). Similar behavior has been observed in learned reward functions in the robotics domain \cite{cabi2019scaling}.

\begin{figure}
\centering
\begin{minipage}{.48\textwidth}
  \centering
  \includegraphics[width=\linewidth]{final_figs/newoveropt.pdf}
  \captionof{figure}{Preference scores versus degree of reward model optimization.

    Optimizing against the reward model initially improves summaries, but eventually overfits, giving worse summaries.
    This figure uses an earlier version of our reward model (see rm3 in Appendix~\ref{sec:batch_breakdown}).
    See Appendix \ref{apdx:overopt_samples} for samples from the KL 250 model. \vspace{-5mm}}
  \label{fig:overopt}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}{.47\textwidth}
  \centering
  \includegraphics[width=\linewidth]{final_figs/data_size_abl.pdf}
  \captionof{figure}{Reward model performance versus data size and model size. Doubling amount of training data leads to a \textasciitilde 1.1\% increase in reward model validation accuracy, whereas doubling the model size leads to a \textasciitilde 1.8\% increase. The 6.7B model trained on all data begins approaching the accuracy of a single human. \vspace{-5mm}}

  \label{fig:data_size}
\end{minipage}
\end{figure}

\paragraph{How does reward modeling scale with increasing model and data size?}
\label{sec:rm_scaling}

We conduct an ablation to determine how data quantity and model size affect reward modeling performance. We train 7 reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our dataset. We find that doubling the training data amount leads to a \textasciitilde 1.1\% increase in the reward model validation set accuracy, whereas doubling the model size leads to a \textasciitilde 1.8\% increase (Figure~\ref{fig:data_size}).

\paragraph{What has the reward model learned?}
\label{sec:rm_evals}

We probe our reward model by evaluating it on several validation sets. We show the full results in Appendix~\ref{sec:rm_validation}, and highlight them here.
We find that our reward models generalize to evaluating CNN/DM summaries (Appendix \ref{sec:agreement_matrices}), agreeing with labeler preferences 62.4\% and 66.5\% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B reward model nearly matches the inter-labeler agreement value of 66.9\%.

We also find that our reward models are sensitive to small but semantically important details in the summary. We construct an additional validation set by having labelers make minimal edits to summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4\% for 1.3B and 82.8\% for 6.7B) as a separate set of human evaluators (84.1\%). Further, when comparing the reference summaries to perturbed summaries where the participants' roles are reversed, our models reliably select the original summary (92.9\% of the time for 1.3B, 97.2\% for 6.7B).
However, our RMs are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary shorter only 62.6\% of the time (vs.\@ 76.4\% for humans).

\begin{figure}
    \centering
    \includegraphics[width=0.6\linewidth]{final_figs/bon_overopt.png}
    \caption{
Summary quality as a function of metric optimized and amount of optimization, using best-of-N rejection sampling.  We evaluate ROUGE, our main reward models, and an earlier iteration of the 1.3B model trained on approximately 75\% as much data (see Table \ref{tab:batch_data} for details).  Though the data is noisy, ROUGE appears to peak both sooner and at a substantially lower preference rate than all reward models.  Details in Appendix \ref{apdx:bon_overopt}.\vspace{-5mm}
}
    \label{fig:bon_overopt}
\end{figure}
\subsection{Analyzing automatic metrics for summarization}
\label{sec:automatic_metrics}

\paragraph{Evaluation.}

We study how well various automatic metrics act as predictors for human preferences, and compare them to our RMs. Specifically, we examine ROUGE, summary length, amount of copying from the post,\footnote{We measure copying by computing the longest common subsequence of bigrams with the original Reddit post or news article, and dividing by the number of bigrams in the summary.} and log probability under our baseline supervised models. We present a full matrix of agreement rates between these metrics in Appendix~\ref{sec:agreement_matrices}.

We find that our learned reward models consistently outperform other metrics, even on the CNN/DM dataset on which it was never trained.
We also find that ROUGE fails to track sample quality as our models improve. While ROUGE has \textasciitilde57\% agreement with labelers when comparing samples from our supervised baseline models, this drops to \textasciitilde50\% for samples from our human feedback model.

Similarly, log probability agreement with humans drops to below 50\% on comparisons between samples from our human feedback models, while our RMs still perform above chance (62\%). Scaling up the size of the supervised model does not reliably improve log probability's agreement with labelers.

\paragraph{Optimization.}

In Figure \ref{fig:bon_overopt}, we show that optimizing ROUGE using a simple optimization scheme doesn't consistently increase quality, as has been noted in \cite{paulus2017deep}.  Optimization against ROUGE peaks both sooner and at a substantially lower quality rate than optimization against our reward models.

\section{Discussion}

\paragraph{Limitations.}
One limitation of our work is the time and cost required to produce our final models. Notably, fine-tuning our 6.7B model with RL required approximately 320 GPU-days.
Our data collection procedure is also expensive compared to prior work ---
the training set took thousands of labeler hours
and required significant researcher time to ensure quality.
For this reason, we were unable to collect an equivalent amount of high-quality human demonstrations for training a better supervised baseline; our current baseline has seen strictly less human data than our RL models. We leave this ablation to future work.
Nevertheless, we believe
reward modeling is
more likely to scale to tasks where it is extremely skill-intensive or time-consuming to provide good demonstrations.

\paragraph{Future directions.}
The methods in this paper could be applied to any task where humans can compare samples, including dialogue, machine translation, question answering, speech synthesis,
and music generation. We expect this method to be particularly important for generating long samples, where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It may be possible to improve sample efficiency by training to predict feedback across many tasks~\cite{mccann2018natural}.

We are particularly interested in scaling human feedback to tasks where humans can’t easily evaluate the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML system is aligned with the human designer’s intentions. One approach is to train ML systems to help humans perform the evaluation task quickly and accurately \cite{christiano2018supervising}.

There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models \cite{unifying, fidler2017teaching, niu2018polite, welleck2019neural}. For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies.

\paragraph{Broader impacts.}

The techniques we explore in this paper are generic techniques that could be used in a wide variety of machine learning applications, for any task where it is feasible for humans to evaluate the quality of model outputs. Thus, the potential implications are quite broad.

Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences.
Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait \cite{covington2016deep}.  In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being.

In the long term, as machine learning systems become more capable it will likely become increasingly difficult to ensure that they are behaving safely: the mistakes they make might be more difficult to spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a news article is both easy to notice (one simply has to read the original article) and has fairly low consequences. On the other hand, imitating human driving may be substantially less safe than driving to optimize human preferences.  We believe that the techniques we explore in this paper are promising steps towards mitigating the risks from such capable systems, and better aligning them with what humans care about.

Unfortunately, our techniques also enable malicious actors to more easily train models that cause societal harm. For instance, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs, or to induce dependence of humans on the technology, or to generate large amounts of toxic or hurtful content intended to harm specific individuals. Avoiding these outcomes is a significant challenge for which there are few obvious solutions.

Large-scale models trained with human feedback could have significant impacts on many groups. Thus, it is important to be careful about how we define the `good' model behavior that human labelers will reinforce.
Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the ‘gold standard’; rather, individuals from groups impacted by the technology should be included in the process to define `good' behavior, and hired as labelers to reinforce this behavior in the model.

We chose to train on the Reddit TL;DR dataset because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they often contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content. For this reason, we recommend that the potential harms of our models be thoroughly studied before deploying them in user-facing applications.

Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.

\subsection*{Acknowledgements}

We'd like to thank Beth Barnes for help with labeler hiring and general encouragement; Geoffrey Irving for guidance on earlier iterations of the project and inspiring conversations; Ben Mann, Tom Brown, Nick Ryder, and Melanie Subbiah for training and evaluating our pretrained models; Chris Hesse, Eric Sigler, Benjamin Chess, Christopher Berner, Clemens Winter, Mateusz Litwin, and many others for supporting us through computing infrastructure improvements and maintenance; Scott Gray for writing fast GPU kernels; Arvind Neelakantan and Wojciech Kryscinski for discussions on how to present the work, experiment design, and what datasets to use; Shan Carter for help designing the main diagram; Douwe Kiela, Zach Lipton, and Alex Irpan for providing feedback on the paper; and Gretchen Krueger for co-writing the model card accompanying the paper.

Finally, we'd like to thank all of our contractors for providing the data that was essential for training the models in this paper: Emill Jayson Caypuno, Rachelle Froyalde, Cyra Denura, Alex Malek, Isik Agil, Reshmi Patel, William Yap, Natalie Silver, Erol Akbaba, Jennifer Brillo, Alexandra Uifalean, Morris Stuttard, Russell Bernandez, Tasmai Dave, Rachel Wallace, Jenny Fletcher, Jian Ouyang, Justin Dill, Maria Orzek, Megan Niffenegger, William Sells, Emily Mariner, Andrew Seely, Lychelle Ignacio, Jelena Ostojic, Nhan Tran, Purev Batdelgar, Valentina Kezic, Michelle Wilkerson, Kelly Guerrero, Heather Scott, Sarah Mulligan, Gabriel Ricafrente, Kara Bell, Gabriel Perez, and Alfred Lee.

\iffalse

- Where our approach fits in (technique that could be used for a wide variety of applications)

The good:
- better alignment with designer intentions
- short term --- can lead to better mitigation of
- long term --- uncertain, but can lead to a better outcome for powerful systems

The bad:
- potential for malicious actors to more explicitly target what they care about (bad things)
- for example, could fine-tune a system to be extra manipulative, or extra hurtful (trolls)
- automating summarization replacing jobs

Original jeff impact:
Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait \cite{covington2016deep}.  Improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human intent.  This could be used to align an algorithm with either benign or malicious human intent.  As an example of something potentially harmful to society, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs.

TODO longer term stuff?

TODO prose
\fi

\bibliographystyle{abbrv}