Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative

Introduction

The increasingly popular pre-training paradigm (Dai & Le, 2015; Devlin et al., 2018; Gururangan et al., 2020) involves first training a generalist model on copious amounts of easy-to-obtain data, e.g. raw text data in NLP, and then using this model to initialize training on a wide swath of downstream tasks. Generalist models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020) have a strong appeal; a few institutions with significant resources incur the cost of training these large models whilst the rest of the research community enjoys a significant performance improvement at minimal computational overhead. However, the advantages of initializing a downstream task from a generalist model are not guaranteed. Previous work has shown that the benefits of pre-training depend heavily on the degree of domain overlap between the end-task data and the massive, heterogenous data on which the generalist model was trained (Beltagy et al., 2019; Gururangan et al., 2020).

Notably, Gururangan et al. (2020) have demonstrated the benefits of continued pre-training of generalist models using data that is similar to that of the end-task. Their approach is formalized into two classes: Domain Adaptive Pre-training (DAPT) and Task Adaptive Pretraining (TAPT) where further stages of pre-training of generalist models are conducted on domain- and task-specific data, respectively. DAPT and TAPT exploit the fact that we often know the end-task beforehand, and so we can make specific choices about our pre-training regimen to improve end-task performance.

However, in both pre-training for generalist models and continued pre-training, the training procedure itself does not explicitly incorporate the end-task objective function. Because of this, practitioners have to be careful with their choice of auxiliary tasks, the order in which they are trained on, and the early-stopping criteria for each pre-training stage so as to actually achieve good downstream end-task performance (Gururangan et al., 2020; Dery et al., 2021). In the absence of principled criteria to make these difficult design choices, it is common to instead resort to the computationally demanding heuristic of pre-training on as much data as possible for as long as possible.

In this paper, we raise the following question: “In settings where we have a particular end-task in mind, should we be pre-training at all?”. We define pre-training as any form of task-agnostic training that a model undergoes before it is finally fine-tuned on the end-task of interest. As a first milestone in addressing the larger question posed above, we explore the ubiquitous continued pre-training setting (Gururangan et al., 2020; Aghajanyan et al., 2021). Specifically, our paper questions the wisdom of having disjoint further pre-training then fine-tuning steps on a generalist model. In response, we advocate for an alternative approach in which we directly introduce the end-task objective of interest into the learning process. This results in a suite of end-task aware methods called TARTAN (end-Task AwaRe TrAiniNg). Our formulations incorporate both unsupervised auxiliary objectives traditionally used in NLP pre-training (such as masked language modeling as in Devlin et al. (2018)) and the end-task objective, followed by an optional fine-tuning step on the end-task. We motivate TARTAN experimentally in the continued pre-training setting and based on this, we make the following contributions to the literature on leveraging auxiliary tasks and data:

In lieu of standard end-task agnostic continued pre-training, we suggest introducing the end-task objective into the training process via multi-task learning (Caruana, 1997; Ruder, 2017). We call this procedure Multi-Tasking end-Task AwaRe TrAiniNg (MT-TARTAN) (Section 3.1). MT-TARTAN is a simple yet surprisingly effective alternative to task-agnostic pre-training. In Section 5, we demonstrate that MT-TARTAN significantly improves performance and data efficiency over Gururangan et al. (2020)’s results. It also obviates the need for fickle hyper-parameter tuning through direct optimization of validation performance.

To allow more fine-grained control of the end-task over the auxiliary tasks, in Section 3.2, we present an online meta-learning algorithm that learns adaptive multi-task weights with the aim of improving final end-task performance. Our META-learning end-Task AwaRe TrAiniNg (META-TARTAN) allows us to robustly modulate between multiple objectives and further improves performance over MT-TARTAN .

A naive implementation of META-TARTAN based on first-order meta-learning analysis results in a sub-optimal algorithm that ignores all tasks except the end-task. We trace this problem to the use of a single model training head for computing both the end-task training loss and meta-objective (end-task validation loss). To guard against this pathological solution, we introduce a separate model head for computing the meta-objective. In Section 3.3, we justify this simple-to-implement fix and validate its practical efficacy in Section 5.

Our results suggest that TARTAN may be an attractive alternative to the continued pre-training paradigm, and further research into the place of pre-training in end-task aware settings is warranted.

Formalizing Pre-training and Continued Pre-training

We can formalize the pre-training procedure as follows:

2 Continued Pre-training

Though TAPT and DAPT do not directly incorporate the end-task objective during training, it still indirectly informs both the choice of pre-training data and the order in which the pre-training tasks are trained on. Below, we explore stronger versions of this influence.

End-Task Aware Training (TARTAN)

MT-TARTAN allows us to prioritize performance on $T^{*}$ in several ways. First, we can weight the end-task higher than all the other auxiliary tasks. Also, during training, we can monitor $\mathcal{L}_{T^{*}}$ on the end-task validation set and early stop when it plateaus; even if the auxiliary tasks have not yet converged. This is not possible during standard pre-training because we do not train $T^{*}$ and so it performs at random before we actually start fine-tuning. Early stopping on $T^{*}$ can represent significant computational savings over end-task agnostic pre-training when the savings in data-efficiency supercede the extra overhead of end-task aware gradient descent steps.

2 End-task Aware Training via Meta-learning (META-TARTAN)

MT-TARTAN, DAPT and TAPT, all share the same drawback: they implicitly assume that the auxiliary tasks have static importance to the end-task over the lifetime of its training, either by being end-task agnostic (DAPT and TAPT) or by having static task weights (MT-TARTAN). With MT-TARTAN, an additional drawback noted by Wang et al. (2019); Yu et al. (2020) is that multi-tasking can negatively impact task performance compared to isolated training. These shortcomings motivate the formulation of an adaptive algorithm that can mitigate the negative influence of some tasks whilst responding to the changing relevance of auxiliary tasks over the lifetime of end-task training.

As they stand, the pre-training equation pair (Equations 1, 2) and the MT-TARTAN pair (Equations 2, 3) are decoupled. The inner-level variables of the pre-training phase do not depend on the outer-level variables of the fine-tuning phase. Thus the equation pairs are typically solved sequentially. We propose to tightly couple Equations 2 and 3 by formulating jointly learning $\mathbf{w}$ and $\theta_{0}$ as a bi-level optimization problem. A bi-level formulation allows us to leverage meta-learning (Schmidhuber, 1995) techniques to learn adaptive task weights which capture variable auxiliary task importance whilst mitigating the contribution of harmful tasks. We propose a meta-learning algorithm in the mold of Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) to learn task weights. As a bi-level problem, this can be formulated as :

We can take the gradient of the above first-order approximation w.r.t an individual weight $w_{i}$. This tells us how to update $w_{i}$ to improve the meta-objective.

Our analysis above is similar to that of Lin et al. (2019) with one key difference: we learn a weighting for the main task $w^{*}$ too. This ability to directly modulate $T^{*}$ allows us to capture the fact that at certain stages in training, auxiliary tasks may have greater impact on end-task generalization than the end-task’s own training data. This choice also allows us to control for over-fitting and the influence of bad (mislabelled or noisy) training data.

3 Introducing a separate classification head for meta-learning

Observe that from Equation 6, updates for $w\neq w^{*}$ involve gradients computed from different model heads $\phi^{i}$ and $\phi^{\prime}$ whilst for $w^{*}$, we are taking the dot product of gradients from the same end-task head $\phi^{\prime}$. As we will show empirically in Section 5.4, computing weight updates this way creates a strong bias towards the primary task, causing $w^{*}$ to rail towards 1 whilst the other weights dampen to 0, which may be sub-optimal in the long run.

Intuitively, this short-horizon (greedy) (Wu et al., 2018) behavior makes sense: the quickest way to make short-term progress (improve $\mathcal{L}^{val}_{T^{*}}(\theta_{t+1})$) is to descend solely on $T^{*}$. More formally, the greedy approach arises because we derive $\nabla_{w_{i}}\mathcal{L}^{val}_{T^{*}}(\theta_{t+1})$ in Equation 6 as a proxy for the gradient at $\theta^{*}$, the outer-loop end-point in Equation 4. Variations of this substitution are common in the meta-learning literature (Finn et al., 2017; Liu et al., 2018; Nichol et al., 2018) because it is computationally infeasible to train a model to convergence every time we wish to compute $\nabla_{w_{i}}\mathcal{L}^{val}_{T^{*}}(\theta^{*})$.

Equation 7 represents a simple-to-implement alternative to Equation 6. We provide a more detailed justification for Equation 7 in Appendix A.1. In Section 5.4, we empirically validate that the transition from Equation 6 to 7 improves performance whilst mitigating pathological solutions. Our approach of creating $\phi^{*}$ for approximating the meta-objective (down-stream validation performance) is inspired by Metz et al. (2018), who use a similar technique to construct a meta-objective for evaluating the quality of unsupervised representations.

Please see Algorithm 1 in Appendix A.3 for details about META-TARTAN.

Experimental Setup

SettingCode will be released at https://github.com/ldery/TARTAN Though our algorithms and methodology can be directly applied to both continued pre-training (Section 2.2) and pre-training from scratch (Section 2.1) of generalist models, we focus on the former scenario. This is because the continued pre-training setting is more common amongst everyday practitioners as it is less computationally demanding. It thus lends itself more easily to exploration under a realistic computational budget. In Appendix A.4, we show that end-task aware training from scratch is viable by studying a simple computer vision setting. Concurrent work by Yao et al. (2021) shows that from-scratch end-task aware training for NLP problems is viable.

Datasets Our experiments focus on two domains: computer science (CS) papers and biomedical (BIOMED) papers. We follow Gururangan et al. (2020) and build our CS and BIOMED domain data from the S2ORC dataset (Lo et al., 2019). We extract 1.49M full text articles to construct our CS corpus and 2.71M for our BIOMED corpus. Under both domains, our end-tasks are low-resource classification tasks. Using low-resource tasks allows us to explore a setting where pre-training can have a significant impact. Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018) and SCIERC (Luan et al., 2018). ACL-ARC is a 6-way citation intent classification task with 1688 labelled training examples. For SCIERC, the task is to classify the relations between entities in scientific articles. This task has 3219 labelled examples as training data. We choose CHEMPROT (Kringelum et al., 2016) as the classification task from the BIOMED domain. This task has 4169 labelled training examples and the goal is to classify chemical-protein interactions. More details of these datasets can be found in Table 2 of Gururangan et al. (2020). Gururangan et al. (2020) evaluate against all 3 tasks and their available code served as a basis on which we built MT-TARTAN and META-TARTAN.

Model Details We use a pre-trained RoBERTabase (Liu et al., 2019) as the shared model base and implement each task as a separate multi-layer perceptron (MLP) head on top of this pre-trained base. As in Devlin et al. (2018), we pass the [CLS] token embedding from RoBERTabase to the MLP for classification.

Training Details For DAPT and TAPT, we download the available pre-trained model bases provided by Gururangan et al. (2020). To train thier corresponding classification heads, we follow the experimental setup described in Appendix B of Gururangan et al. (2020).

As mentioned in section 3.3, we train as separate meta-classification head, $\phi^{*}$, to estimate the validation meta-gradients. To estimate $\phi^{*}$, we use batch sizes of $\{16,32\}$ samples from $T^{*}$’s train set. We regularize the meta-head with $l_{2}$ weight decay and set the decay constant to $0.1$. We use a learning rate $10^{-3}$ to learn the meta-head. We stop training $\phi^{*}$ after 10 gradient descent steps.

Results and Discussion

In this section, we will discuss the results of comparing our models against DAPT and TAPT baselines.Our results are slightly different from those presented in Table 5 of Gururangan et al. (2020) in terms of absolute values but the trends observed there still hold here. We attribute these differences to (1) minor implementation differences, and (2) averaging performance over ten seeds instead of five as used in the original paper in order to more strongly establish statistical significance. We observe slightly lower performance on ACL-ARC and SCIERC tasks due to these changes and higher performance on CHEMPROT. Broadly, we demonstrate the effectiveness of end-task awareness as improving both performance and data-efficiency.

Table 1 compares TAPT to its end-task aware variants. As in Gururangan et al. (2020), we observe that performing task adaptive pre-training improves upon just fine-tuning RoBERTa. However, note that introducing the end-task by multi-tasking with the TAPT MLM objective leads to a significant improvement in performance. This improvement is consistent across the 3 tasks we evaluate against. We find that both MT-TARTAN and META-TARTAN achieve similar results in this setting.

2 End-task awareness improves data-efficiency

Gururangan et al. (2020) train DAPT on large amounts of in-domain data to achieve results competitive with TAPT. They use 7.55 billion tokens for the BIOMED domain and 8.10 billion for the CS domain. This is on average over $10^{4}\times$ the size of the training data of our end-tasks of interest. The large amount of data required to train a competitive DAPT model represents a significant computational burden to the every-day practitioner. This begets the question: are such large amounts of auxiliary data necessary for achieving good downstream performance? To answer this, we train DAPT and its TARTAN version on variable amounts of data for both SCIERC and ACL-ARC tasks.

TARTAN is more data-efficient than DAPT+TAPT Table 2 compares DAPT and DAPT+TAPT (DAPT followed by TAPT) to *-TARTAN which multi-task DAPT, TAPT and the end-task. MT-TARTAN and META-TARTAN significantly outperform DAPT and DAPT+TAPT in 2 of the tasks whilst giving higher average performance in the ACL-ARC task. We thus conclude that end-task awareness allows us to get a greater performance boost out of the same amount of data.

Zhang et al. (2020) show that different tasks exhibit sigmoid-like curves in terms of how much pre-training data is required to achieve good results before performance levels off. We contextualize Tables 2 and 3 within said work and posit that the CHEMPROT task intrinsically requires much more data (compared to our other tasks) before performance begins to improve appreciably.

3 META-TARTAN more effectively utilizes out-of-distribution auxiliary data over MT-TARTAN

We have seen that leveraging TAPT (Table 1 and 2) leads MT-TARTAN and META-TARTAN to perform similarly. The advantage of learning adaptive weights becomes pronounced in the DAPT only setting. Whilst TAPT uses the end-task’s own training data for masked language modelling, DAPT uses heterogeneous domain data whose impact on the end-task performance is less clear. Notice from Table 4 that when required to rely solely on domain data for auxiliary tasking, META-TARTAN improves performance over MT-TARTAN. We attribute META-TARTAN’s improvement over MT-TARTAN to its ability to more flexibly adapt to incoming data of variable utility to the end-task.

4 Task weighting strategies discovered by meta-learning

To illustrate the importance of the separate classification head $\phi^{*}$ for computing the meta-signal for the task weights (described in Section 3.3), we run META-TARTAN experiments with ACL-ARC as the end-task and DAPT as the auxiliary task. We compare using either a separate ($\phi^{*}$) or the same ($\phi^{\prime}$) classification head for calculating the meta-gradient. Figure 3 plots the task weightings learned in each setting during training. We can clearly see that using a separate head counteracts the pathological solution of down-weighting all tasks that are not $T^{*}$ and as a result, improves performance: a delta of 1.7 F1 points in this case. The strategy discovered by META-TARTAN presents an interesting contrast to classical pre-training: whilst the initial phase of classical pre-training involves solely the auxiliary task, early in training, META-TARTAN up-weights the auxiliary task but does not fully zero out the end-task. Later in training, we see leveling off of weights instead of railing the end-task to 1 as in classical pre-training.

Next, we plot a similar graph for using both DAPT and TAPT across our three tasks in Figure 4. From the figure, it is apparent that META-TARTAN discovers similar task-weighting strategies across different end-tasks. This suggests that the MLM objective and META-TARTAN’s strategy for learning task weights are generic enough to induce similar behaviours across tasks. In general, DAPT is significantly up-weighted compared to the end-task and TAPT. Note that the TAPT + ACL-ARC task weights (Figure 4) has the same approximate trajectory as ACL-ARC task weight in Figure 3. It seems important to assign high weight to the task data (Figure 3) but not necessarily all of it needs to go to the actual task loss (Figure 4). We hypothesize that the diversity in the domain data counteracts overfitting to the end-task data and results in DAPT being up-weighted.

Related Work

Multi-task learning can be traced back to seminal work by Caruana (1995), Caruana (1997), and has since been the subject of a flourishing literature, recent surveys of which can be found in Ruder (2017) or Zhang & Yang (2021). In NLP, while initial work from Collobert & Weston (2008) already showed the benefits of multi-task learning, it has only recently become a central topic in the field, with the advent of multi-task benchmarks (Wang et al., 2018b; McCann et al., 2018).

Pre-training is where a machine learning model is first trained on a generic, data-rich task before being fine-tuned on an end-task. In NLP this practice dates back to the use of pre-trained word embeddings (Turian et al., 2010; Mikolov et al., 2013) and later pre-trained encoders (Kiros et al., 2015; Dai & Le, 2015). Peters et al. (2018) and Howard & Ruder (2018) heralded a renaissance of pre-training before BERT (Devlin et al., 2018) and its many offshoots (Liu et al., 2019; Yang et al., 2019; Lewis et al., 2019) cemented it as the de facto standard for modern NLP.

Meta-learning dates back to early work from Schmidhuber (1995); Thrun (1998). More relevant to our work is gradient-based meta-learning for solving bi-level optimization problems, first popularized by Finn et al. (2017) and followup work (Nichol et al., 2018; Rajeswaran et al., 2019) for few-shot learning. This method has transferred to a variety of applications such as architecture search (Liu et al., 2018) and model poisoning (Kurita et al., 2020).

Conclusion

We have advocated for a paradigm shift in the way we approach pre-training. We have motivated making pre-training more end-task aware when the end-task is known in advance. Our work introduced two novel end-task aware training algorithms: End-task Aware Training via Multi-tasking (MT-TARTAN) and End-task Aware Training via Meta-learning (META-TARTAN). In Section 5, we demonstrated the ability of our proposed algorithms to improve performance and data-efficiency over their end-task agnostic counterparts.

This work suggests several promising directions for future work. Instead of learning coarse task level weights, can further performance improvements be achieved via finer-grained example level weighting as in Wang et al. (2020)? Can meta-learning algorithms like META-TARTAN enable more effective utilization of previously discarded (Aroca-Ouellette & Rudzicz, 2020) pre-training auxiliary tasks like Next Sentence Prediction (NSP) (Devlin et al., 2018)? We hope this work spurs conversation around these questions and many more.

Acknowledgements

This work was supported in part by DSO National Laboratories, an ENS-CFM Data Science Chair, DARPA FA875017C0141, the National Science Foundation grants IIS1705121, IIS1838017, IIS2046613 and IIS-2112471, an Amazon Web Services Award, a Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., and a Block Center Grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.

Ethics Statement

Our work introduces new algorithms but leverages pre-existing datasets and models. Overall, this work inherits some of the risk of original work upon which it is implemented. Algorithms for continued training such as TAPT and DAPT necessitate per-task training of unsupervised objectives which result in corresponding green-house emissions due to energy consumption (Strubell et al., 2019). However, as shown in Sections 3 and 5, our new compute-efficient algorithms greatly increase the data efficiency of these algorithms, reducing these harms as well as the various harms associated with labor for data-collection (Jo & Gebru, 2020). Also, since our work is set in the context of pre-existing datasets and models (Section 4), we recognize that any ethical issues that have been revealed in these (such as bias (Bender et al., 2021) or privacy leakage (Carlini et al., 2021)) may also propagate to models trained using our work, and mitigation strategies such as Schick et al. (2021); Liang et al. (2021) may be necessary. Finally, there is a potential risk in META-TARTAN that leveraging a validation set for defining the meta-objective could amplifying bias that exists in this data split, although this is done indirectly through task weighting and hence we believe that this risk is small.

Reproducibility Statement

We pledge to release the source-code for this project to improve the ease of reproducibility of our results by the NLP and machine learning communities. In Section 4, we have specified details about datasets, training regimes and models to allow anyone who wishes to reproduce our results without our original source code to do so. Our discussion of the algorithmic and evaluation details can be found in Appendices A.1, A.3 and A.2. As we noted in 4, we build off of Gururangan et al. (2020)’s implementations which can be found at https://github.com/allenai/dont-stop-pretraining.

References

Appendix A Appendix

To arrive at Equation 7 we start with the closed form solution for $\nabla_{w_{i}}\mathcal{L}^{val}_{T^{*}}(\theta^{*})$ and then introduce approximations in order to produce Equation 7. First, note that :

To get $\nabla_{w_{i}}\theta^{*}(\mathbf{w})$ we invoke the Cauchy Implicit Function Theorem (IFT) as with Lorraine et al. (2020); Navon et al. (2020); Liao et al. (2018):

Computing $\nabla_{w_{i}}\mathcal{L}^{val}_{T^{*}}(\theta^{*})$ from Equation 9 is computationally unwieldy since we would not only have to optimize $\theta$ to convergence for every step of $w_{i}$ but we would also have to invert the Hessian of a typically large model. Our middle ground between Equations 9 and 6 (Equation 7) makes use of the following approximations:

We approximate the inverse Hessian with the identity. This approximation is not new; we follow previous work like Lorraine et al. (2020)(Table 3) who explore the use of this approximation because of computational efficiency.

We are assuming the contribution of terms with $i>0$ are negligible.

Bringing it all together, we get Equation 7, repeated here:

A.2 Calculating p-values from Permutation Test

We used the permutation test (Good, 2005; Dror et al., 2018) to test for statistical significance. For each test, we generate 10000 permutations to calculate significance level. This is sufficient to converge to a stable p-value without being a computational burden. We chose this over the common student t-test because :

We have only 10 runs per algorithm and permutation tests are more robust at low sample size

Permutation test is assumption free. Student t-tests assume that the samples are normally distributed

Permutation test is robust to variance in the samples, so even though error-bars can overlap, we still establish significant differences in the samples. Variance in our results is expected due to small dataset sizes of end-tasks.

A.3 Algorithm for META-TARTAN

A.4 Vision Experiments

We validate that the gains from end-task Aware Training are not siloed to only learning from text. We conduct an experiment comparing end-task aware training on images to its end-task agnostic variant.

We use the Cifar100 dataset (Krizhevsky et al., 2009). We use the Medium-Sized Mammals superclass (one of the 20 coarse labels) as our main task whilst the other 19 super classes are used as auxiliary data. Our primary task is thus a 5-way classification task of images different types of medium-sized mammals whilst whilst the remaining 95 classes are grouped into a single auxiliary task.

As can be seen from Table 5, being end-task aware improves over task agnostic pre-training. We find that, again, when our auxiliary task consist of solely domain data and no task data, META-TARTAN performs better than MT-TARTAN (as measured by averaged performance).

A.5 Full TAPT Table with Significance levels

We repeat Table 1 and provide details about levels of statistical signifance.

A.6 Full DAPT/DAPT+TAPT Table

We repeat Table 3 and provide details about levels of statistical signifance.

A.7 FAQ

What settings are TARTAN algorithms designed for? TARTAN algorithms specialize auxiliary objectives to a particular end-task. This comes at a risk of losing the generic representations afforded by generalist pre-trained models. Thus if a practitioner has a sufficiently important end-task where obtaining improved end-task performance is paramount over generic representations, then TARTAN is a viable option.

When do we get computational savings from META-TARTAN? MT-TARTAN does not add any extra overhead compared to pre-train then fine-tune approaches. META-TARTAN however, adds extra overhead per gradient descent step due to computing meta-gradients. However, as shown in Section 5 we are able to get several orders of magnitude improvement in data-efficiency from applying the method. In general, for the tasks we experimented with, we find that the savings in data-efficiency superseded the extra per-timestep meta-learning overhead.

When should we use META-TARTAN over MT-TARTAN?

In +TAPT settings (Tables 1, 3), we observe that META-TARTAN and MT-TARTAN perform similarly. We attribute this to the strength of TAPT-MLM objective. We were pleasantly surprised that the two methods performed comparatively in this setting but in hindsight, we appreciate the insight that went into designing TAPT-MLM as an objective which makes it a strong baseline. In other settings with less carefully designed auxiliary objectives and data (which can potentially be detrimental to the end-task) we expect META-TARTAN to perform better. Section 5.3 provides evidence of this.