On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models

Introduction

Ad click-through rate (CTR) prediction is a key component of online advertising systems that has a direct impact on revenue, and continues to be an area of active research (McMahan et al., 2013; He et al., 2014; Zhou et al., 2018; Ling et al., 2017). This paper presents a detailed case study to give the reader a ”tour of the factory floor” of a production CTR prediction system, describing challenges specific to this category of large industrial ML systems and highlighting techniques that have proven to work well in practice.

The production CTR prediction model consists of billions of weights, trains on more than one hundred billion examples, and is required to perform inference at well over one hundred thousand requests per second. The techniques described here balance accuracy improvements with training and serving costs, without adding undue complexity: the model is the target of sustained and substantial R&D and must allow for effectively building on top of what came before.

The recommender problem surfaces a result or set of results from a given corpus, for a given initial context. The initial context may be a user demographic, previously-viewed video, search query, or other. Search advertising specifically looks at matching a query $q$ with an ad $a$. CTR models for recommendation specifically aim to predict the probability $P(click|x)$, where the input $x$ is an ad-query pair $\langle a,q\rangle$, potentially adorned with additional factors affecting CTR, especially related to user interface: how ads will be positioned and rendered on a results page (Section 6).

Ads pricing and allocation problems create the per-example calibration requirement. Typically, predictions will flow through to an auction mechanism that incorporates bids to determine advertiser pricing. Auction pricing schemes (e.g, VCG (Varian and Harris, 2014)) rely on the relative value of various potential outcomes. This requires that predictions for all potential choices of $x$ be well calibrated with respect to each other. Additionally, unlike simple recommenders, ads systems frequently opt to show no ads. This requires estimating the value of individual ads relative to this ”null-set” of no ads, rather than simply maximizing for ad relevance.

Consider a query like ”yarn for sale”; estimated CTR for an ad from ”yarn-site-1.com” might be 15.3%. Estimated CTR for an ad from ”yarn-site-2.com” might be 10.4%. Though such estimates can be informed by the semantic relevance of the websites, the requirements for precision are more than what one should expect from general models of language. Additionally, click-through data is highly non-stationary: click prediction is fundamentally an online recommendation problem. An expectation of 15.3% is not static ground truth in the same sense as, for example, translation or image recommendation; it is definitively more subject to evolution over time.

2. Outline

For ads CTR predictors, minor improvements to model quality will often translate into improved user experience and overall ads system gains. This motivates continuous investments in model research and development. Theoretical and benchmark improvements from ML literature rarely transfer directly to problem-dependent settings of real-world applications. As such, model research must be primarily empirical and experimental. Consequently, a great deal of attention must be paid to the machine costs of model training experiments while evaluating new techniques. In Section 2 we first give a general overview of the model and training setup; Section 3 then discusses efficiency concerns and details several successfully deployed techniques. In Section 4, we survey applications of modern ML techniques targeted at improving measures of accuracy and geared explicitly toward very-large-scale models. Section 4.4 summarizes empirical results roughly characterizing the relative impact of these techniques.

Deep neural networks (DNNs) provide substantial improvements over previous methods in many applications, including large-scale industry settings. However, non-convex optimization reveals (and exacerbates) a critical problem of prediction: irreproducibility (Dusenberry et al., 2020; Shamir and Coviello, 2020a, b; Shamir et al., 2020; Snapp and Shamir, 2021; D’Amour et al., 2020). Training the same model twice (identical architecture, hyper-parameters, training data) may lead to metrics of the second model being very different from the first. We distinguish between model irreproducibility, strictly related to predictions on fixed data, and system irreproducibility, where a deployed irreproducible model affects important system metrics. Section 5 characterizes the problem and describes improvements to model irreproducibility.

An effective click prediction model must be able to generalize across different UI treatments, including: where an ad is shown on the page and any changes to the formatting of the ad (e.g., bolding specific text or adding an image). Section 6 describes a specific model factorization that improves UI generalization performance. Finally, Section 7 details a general-purpose technique for adding bias constraints to the model that has been applied to both improve generalization and system irreproducibility.

This paper makes the following contributions: 1) we discuss practical ML considerations from many perspectives including accuracy, efficiency and reproducibility, 2) we detail the real-world application of techniques that have improved efficiency and accuracy, in some cases describing adaptations specific to online learning, and 3) we describe how models can better generalize across UI treatments through model factorization and bias constraints.

Model and Training Overview

A major design choice is how to represent an ad-query pair $x$. The semantic information in the language of the query and the ad headlines is the most critical component. Usage of attention layers on top of raw text tokens may generate the most useful language embeddings in current literature (Vaswani et al., 2017), but we find better accuracy and efficiency trade-offs by combining variations of fully-connected DNNs with simple feature generation such as bi-grams and n-grams on sub-word units. The short nature of user queries and ad headlines is a contributing factor. Data is highly sparse for these features, with typically only a tiny fraction of non-zero feature values per example.

All features are treated as categorical and mapped to sparse embedding tables. Given an input $x$, we concatenate the embedding values for all features to form a vector $e$, the embedding input layer of our DNN. $E$ denotes a minibatch of embedding values $e$ across several examples.

Next, we formally describe a simplified version of the model’s fully-connected neural network architecture. Later sections will introduce variations to this architecture that improve accuracy, efficiency, or reproducibility. We feed $E$ into a fully-connected hidden layer $H_{1}\!=\!\sigma(EW_{1})$ that performs a linear transformation of $E$ using weights $W_{1}$ followed by non-linear activation $\sigma$. Hidden layers $H_{i}\!=\!\sigma(H_{i-1}W_{i})$ are stacked, with the output of the $k$th layer feeding into an output layer $\hat{y}\!=\!\text{sigmoid}(H_{k}W_{k+1})$ that generates the model’s prediction corresponding to a click estimate $\hat{y}$. Model weights are optimized following $\min_{W}\sum_{i}\mathcal{L}(y_{i},\hat{y}_{i})$. We found ReLUs to be a good choice for the activation function; Section 5 describes improvements using smoothed activation functions. The model is trained through supervised learning with the logistic loss of the observed click label $y$ with respect to $\hat{y}$. Sections 4 and 7 describe additional losses that have improved our model. Training uses synchronous minibatch SGD on Tensor Processing Units (TPUs) (Jouppi et al., 2020): at each training step $t$, compute gradients $G_{t}$ of the loss on a batch of examples (ranging up to millions of examples), and weights are optimized with an adaptive optimizer. We find that AdaGrad (McMahan and Streeter, 2010; Duchi et al., 2011) works well for optimizing both embedding weights and dense network weights. Moreover, In Section 4.2 discusses accuracy improvements from deploying a second-order optimizer: Distributed Shampoo (Anil et al., 2020) for training dense network weights, which to our knowledge, is the first known large-scale deployment in a production scale neural network training system.

Given the non-stationarity of data in ads optimization, we find that online learning methods perform best in practice (McMahan et al., 2013). Models train using a single sequential pass over logged examples in chronological order. Each model continues to process new query-ad examples as data arrives (Swaminathan and Joachims, 2015). For evaluation, we use models’ predictions on each example from before the example is trained on (i.e., progressive validation) (Blum et al., 1999). This setup has a number of practical advantages. Since all metrics are computed before an example is trained on, we have an immediate measure of generalization that reflects our deployment setup. Because we do not need to maintain a holdout validation set, we can effectively use all data for training, leading to higher confidence measurements. This setup allows the entire learning platform to be implemented as a single-pass streaming algorithm, facilitating the use of large datasets.

ML Efficiency

Our CTR prediction system provides predictions for all ads shown to users, scoring a large set of eligible ads for billions of queries per day and requiring support for inference at rates above 100,000 QPS. Any increase in compute used for inference directly translates into substantial additional deployment costs. Latency of inference is also critical for real-time CTR prediction and related auctions. As we evaluate improvements to our model, we carefully weigh any accuracy improvements against increases in inference cost.

Model training costs are likewise important to consider. For continuous research with a fixed computational budget, the most important axes for measuring costs are bandwidth (number of models that can be trained concurrently), latency (end-to-end evaluation time for a new model), and throughput (models that can be trained per unit time).

Where inference and training costs may differ, several ML techniques are available to make trade-offs. Distillation is particularly useful for controlling inference costs or amortizing training costs (see Section 4.1.2). Techniques related to adaptive network growth (Chen et al., 2015) can control training costs relative to a larger final model (with larger inference cost).

Efficient management of computational resources for ML training is implemented via maximizing model throughput, subject to constraints on minimum bandwidth and maximum training latency. We find that required bandwidth is most frequently governed by the number of researchers addressing a fixed task. For an impactful ads model, this may represent many dozens of engineers attempting incremental progress on a single modelling task. Allowable training latency is a function of researcher preference, varying from hours to weeks in practice. Varying parallelism (i.e., number of accelerator chips) in training controls development latency. As in many systems, lowered latency often comes at the expense of throughput. For example, using twice the number of chips speeds up training, but most often does so sub-linearly (training is less than twice as fast) because of parallelization overhead.

For any given ML advancement, immediate gains must be weighed against the long-term cost to future R&D. For instance, naively scaling up the size of a large DNN might provide immediate accuracy but add prohibitive cost to future training (Table 4.2 includes a comparison of techniques and includes one such naive scaling baseline).

We have found that there are many techniques and model architectures from literature that offer significant improvements in model accuracy, but fail the test of whether these improvements are worth the trade-offs (e.g., ensembling many models, or full stochastic variational Bayesian inference (Blundell et al., 2015)). We have also found that many accuracy-improving ML techniques can be recast as efficiency-improving via adjusting model parameters (especially total number of weights) in order to lower training costs. Thus, when we evaluate a technique, we are often interested in two tuning points: 1) what is the improvement in accuracy when training cost is neutral and 2) what is the training cost improvement if model capacity is lowered until accuracy is neutral. In our setting, some techniques are much better at improving training costs (e.g., distillation in Section 4.1.2) while others are better at improving accuracy. Figure 1 illustrates these two tuning axes.

We survey some successfully deployed efficiency techniques in the remainder of this section. Section 3.1 details the use of matrix factorization bottlenecks to approximate large matrix multiplication with reduced cost. Section 3.2 describes AutoML, an efficient RL-based architecture search that is used to identify model configurations that balance cost and accuracy. Section 3.3 discusses a set of effective sampling strategies to reduce data used for training without hurting accuracy.

2. AutoML for Efficiency

To develop an ads CTR prediction model architecture with optimal accuracy/cost trade-off, we typically have to tune the embedding widths of dozens of features and layer widths for each layer in the DNN. Assuming even just a small constant number of options for each such width, the combinatorial search space quickly reaches intractable scales. For industrial-scale models, it is not cost-effective to conduct traditional architecture search with multiple iterations (Zoph et al., 2018; Real et al., 2019). We have successfully adopted neural architecture search based on weight sharing (Bender et al., 2020) to efficiently explore network configurations (e.g., varying layer width, embedding dimension) to find versions of our model that provide neutral accuracy with decreased training and serving cost. As illustrated in Figure 2, this is achieved by three components: a weight-sharing network, an RL controller, and constraints.

The weight-sharing network builds a super-network containing all candidate architectures in the search space as sub-networks. In this way, we can train all candidate architectures simultaneously in a single iteration and select a specific architecture by activating part of the super-network with masking. This setup significantly reduces the number of exploration iterations from O(1000) to O(1).

The reinforcement learning controller maintains a sampling distribution, $\theta_{dist}$, over candidate networks. It samples a set of decisions ($d_{1},d_{2},...$) to activate a sub-network at each training step. We then do a forward pass for the activated sub-network to compute loss and cost. Based on that, we estimate the reward value $R(d_{1},d_{2},...)$ and conduct a policy gradient update using the REINFORCE algorithm (Williams, 1992) as follows:

where $\overline{R}$ denotes the moving average value of the reward and $\alpha_{0}$ is the learning rate for the reinforcement learning algorithm. Through the update at each training step, the sampling rate of better architectures will gradually increase and the sampling distribution will eventually converge to a promising architecture. We select the architecture with maximum likelihood at the end of the training. Constraints specify how to compute the cost of the activated sub-network, which can typically be done by estimating the number of floating-point operations or running a pre-built hardware-aware neural cost model. The reinforcement learning controller incorporates the provided cost estimate into the reward (e.g., $R=R_{\text{accuracy}}+\gamma\cdot|\text{cost}/\text{target}-1|$, where $\gamma<0$) (Bender et al., 2020) in order to force the sampling distribution to converge to a cost-constrained point. In order to search for architectures with lower training cost but neutral accuracy, in our system we set up multiple AutoML tasks with different constraint targets (e.g. 85%/90%/95% of the baseline cost) and selected the one with neutral accuracy and smallest training cost. A recent application of this architecture search to the model reduced time per training step by $16\%$ without reducing accuracy.

3. Data Sampling

Historical examples of clicks on search ads make up a large dataset that increases substantially every day. The diminishing returns of ever larger datasets dictate that it is not beneficial to retain all the data. The marginal value for improving model quality goes toward zero, and eventually does not justify any extra machine costs for training compute and data storage. Alongside using ML optimization techniques to improve ML efficiency, we also use data sampling to control training costs. Given that training is a single-pass over data in time-order, there are two ways to reduce the training dataset: 1) restricting the time range of data consumed; and 2) sampling the data within that range. Limiting training data to more recent periods is intuitive. As we extend our date range further back in time, the data becomes less relevant to future problems. Within any range, clicked examples are more infrequent and more important to our learning task; so we sample the non-clicked examples to achieve rough class balance. Since this is primarily for efficiency, exact class balance is unnecessary. A constant sampling rate (a constant class imbalance prior) can be used with a simple single-pass filter. To keep model predictions unbiased, importance weighting is used to up-weight negative examples by the inverse of the sampling rate. Two additional sampling strategies that have proved effective are as follows:

Sampling examples associated with a low logistic loss (typically examples with low estimated CTR and no click).

Sampling examples that are very unlikely to have been seen by the user based on their position on the page.

The thresholds for the conditions above are hand-tuned and chosen to maximize data reduction without hurting model accuracy. These strategies are implemented by applying a small, constant sampling rate to all examples meeting any of the conditions above. Pseudo-Random sampling determines whether examples should be kept and re-weighted or simply discarded. This ensures that all training models train on the same data. This scheme may be viewed as a practical version of (Fithian and Hastie, 2014) for large problem instances with expensive evaluation. Simple random sampling allows us to keep model estimates unbiased with simple constant importance re-weighting. It is important to avoid very small sampling rates in this scheme, the consequent large up-weighting can lead to model instability. Re-weighting is particularly important for maintaining calibration, since these sampling strategies are directly correlated to labels.

For sampling strategies that involve knowing the loss on an example, calculating that loss would require running inference on the training example, removing most of the performance gains. For this reason, we use a proxy value based on a prediction made by a ”teacher model”. In this two-pass approach. We first train once over all data to compute losses and associated sampling rates, and then once on the sub-sampled data. The first pass uses the same teacher model for distillation (Section 4.1.2) and is only done once. Iterative research can then be performed solely on the sub-sampled data. While these latter models will have different losses per example, the first pass loss-estimates still provide a good signal for the ‘difficulty’ of the training example and leads to good results in practice. Overall our combination of class re-balancing and loss-based sampling strategies reduces the data to ¡ 25% of the original dataset for any given period without significant loss in accuracy.

Accuracy

Next we detail a set of techniques aimed at improving the accuracy of the system. We discuss: additional losses that better align offline training-time metrics with important business metrics, the application of distillation to our online training setting, the adaptation of the Shampoo second-order optimizer to our model, and the use of Deep and & Cross networks.

Loss engineering plays an important role in our system. As the goal of our model is to predict whether an ad will be clicked, our model generally optimizes for logistic loss, often thought of as the cross-entropy between model predictions and the binary task (click/no-click) labels for each example. Using logistic loss allows model predictions to be unbiased so that the prediction can be interpreted directly as a calibrated probability. Binary predictions can be improved by introducing soft prediction through distillation methods (Hinton et al., 2015). Beyond estimating the CTR per ad, it is important that the set of candidate ads for a particular query is correctly ranked (such that ads with clicks have higher CTR than ads without clicks), thus incorporating proper ranking losses is also important. In this section, we discuss novel auxiliary losses and introduce multi-task and multi-objective methods for joint training with these losses

We found that Area under the ROC curve computed per query (PerQueryAUC) is a metric well correlated with business metrics quantifying the overall performance of a model. In addition to using PerQueryAUC during evaluation, we also use a relaxation of this metric, i.e., rank-loss, as a second training loss in our model. There are many rank losses in the learning-to-rank family (Pasumarthi et al., 2019; Burges, 2010). We find one effective approximation is Ranknet loss (Burges et al., 2005), which is a pairwise logistic loss:

where $s_{i},s_{j}$ are logit scores of two examples.

Rank losses should be trained jointly with logistic loss; there are several potential optimization setups. In one setup, we create a multi-objective optimization problem (Sculley, 2010):

where $\mathbf{s}$ are logit scores for examples, $\mathbf{y}_{\text{rank}}$ are ranking labels, $\mathbf{y}$ are the binary task labels, and $\alpha_{1}\in(0,1)$ is the rank-loss weight. Another solution is to use multi-task learning (Caruana, 1997; Ruder, 2017), where the model produces multiple different estimates $s$ for each loss.

where $W_{\text{shared}}$ are weights shared between the two losses, $W_{\text{logistic}}$ are for the logistic loss output, and $W_{\text{rank}}$ are for the rank-loss output. In this case, the ranking loss affects the ”main” prediction $\mathbf{s}_{\text{logistic}}$ as a ”regularizer” on $W_{\text{shared}}$.

As rank losses are not naturally calibrated predictors of click probabilities, the model’s predictions will be biased. A strong bias correction component is needed to ensure the model’s prediction is unbiased per example. More detail can be found in Section 7. Application of ranklosses to the model generated accuracy improvements of $-0.81\%$ with a slight increase in training cost of $1\%$.

1.2. Distillation.

Distillation adds an additional auxiliary loss requiring matching the predictions of a high-capacity teacher model, treating teacher predictions as soft labels (Hinton et al., 2015). In our model, we use a two-pass online distillation setup. On the first pass, a teacher model records its predictions progressively before training on examples. Student models consume the teacher’s predictions while training on the second pass. Thus, the cost of generating the predictions from the single teacher can be amortized across many students (without requiring the teacher to repeat inference to generate predictions). In addition to improving accuracy, distillation can also be used for reducing training data costs. Since the high-capacity teacher is trained once, it can be trained on a larger data set. Students benefit implicitly from the teachers prior knowledge of the larger training set, and so require training only smaller and more recent data. The addition of distillation to the model improved accuracy by $~{}0.41\%$ without increasing training costs (in the student).

1.3. Curriculums of Losses

In machine learning, curriculum learning (Bengio et al., 2009) typically involves a model learning easy tasks first and gradually switching to harder tasks. We found that training on all classes of losses in the beginning of training increased model instability (manifesting as outlier gradients which cause quality to diverge). Thus, we apply an approach similar to curriculum learning to ramp up losses, starting with the binary logistic loss and gradually ramping up distillation and rank losses over the course of training.

2. Second-order Optimization

Second-order optimization methods that use second derivatives or second-order statistics are known to have better convergence properties compared to first-order methods (Nocedal and Wright, 2006). Yet to our knowledge, second-order methods are rarely reported to be used in production ML systems for DNNs. Recent work on Distributed Shampoo (Anil et al., 2020; Gupta et al., 2018) has made second-order optimization feasible for our model by leveraging the heterogeneous compute offered by TPUs and host-CPUs, and by employing additional algorithmic efficiency improvements.

For our model, Distributed Shampoo provided much faster convergence with respect to training steps, and yielded better accuracy when compared to standard adaptive optimization techniques including AdaGrad (Duchi et al., 2011), Adam (Kingma and Ba, 2014), Yogi (Zaheer et al., 2018), and LAMB (You et al., 2019). While second-order methods are known to provide faster convergence compared to first-order methods in the literature - It often fails to provide competitive wall-clock time due to the computational overheads in the optimizer, especially on smaller scale benchmarks. For our model, second-order optimization method was an ideal candidate due to the large batch sizes used in training which amortizes the cost of costly update rule. Training time only increased by approximately 10%, and the improvements to model accuracy far outweighed the increase in training time. We next discuss salient implementation details specific to our model.

Learning Rate Grafting. One of the main challenges in online optimization is defining a learning rate schedule. In contrast to training on static datasets, the number of steps an online model will require is unknown and may be unbounded. Accordingly, popular learning rate schedules from literature depending on fixed time horizons, such as cosine decay or exponential decay, perform worse in contrast to the implicit data-dependent adaptive schedule from AdaGrad (Duchi et al., 2011). As observed in literature (Agarwal et al., 2020), we also find that AdaGrad’s implicit schedule works quite well in the online setting; especially after the $\epsilon$ parameter (the initial accumulator value) is tuned. Accordingly, we bootstrap the schedule for Distributed Shampoo via grafting the per-layer step size from AdaGrad. More precisely, we use the direction from Shampoo while using the magnitude of step size from AdaGrad at a per-layer granularity. An essential feature of this bootstrapping is that it allowed us to inherit hyper-parameters from previous AdaGrad tunings to search for a Pareto optimal configuration.

Momentum. Another effective implementation choice is the combination of Nesterov-styled momentum with the preconditioned gradient. Our analysis suggests that momentum added modest gains on top of Shampoo without increasing the computational overhead while marginally increasing the memory overhead. Computational overhead was addressed via the approximations described in (Sutskever et al., 2013).

Stability & Efficiency. Distributed Shampoo has higher computational complexity per step as it involves matrix multiplication of large matrices for preconditioning and statistics/preconditioner computation. We addressed these overheads with several techniques in our deployment. For example, the block-diagonalization suggested in (Anil et al., 2020) effectively reduced computational complexity while also allowing the implementation of parallel updates for each block in the data-parallel setting via weight-update sharding (Xu et al., 2020). This optimization reduced the overall step time. Moreover, optimizer overheads are independent of batch size; thus, our use of large batch sizes helped reduce overall computational overhead. Finally, we found that the condition number of statistics used for preconditioning can vary in range, reaching more than $10^{10}$. As numerical stability and robustness are of utmost importance in production, we use double precision numerics. To compute the preconditioners, we use the CPUs attached to the TPUs to run inverse-$p$th roots and exploit a faster algorithm; the coupled Newton iteration (Guo and Higham, 2006) for larger preconditioners as in Figure 3.

When integrated with the ad click prediction model, the optimizer improved our primary measure of accuracy, Area under the ROC curve computed per query (PerQueryAuc), by $0.44\%$. Accuracy improvements above 0.1% are considered significant. For comparison: a naive scaling of the deep network by 2x yields a PerQueryAUC improvement of $0.13\%$. See Table 4.2 for a summary of accuracy technique results.

In practice adding the Deep & Cross Network to the model yielded an accuracy improvement of $~{}0.18\%$ with a minimal increase in training cost of $3\%$.

4. Summary of Efficiency and Accuracy Results

Below we share measurements of the relative impact of the previously discussed efficiency and accuracy techniques as applied to the production model. The goal is to give a very rough sense of the impact of these techniques and their accuracy vs. efficiency tradeoffs. While precise measures of accuracy improvement on one particular model are not necessarily meaningful, we believe the coarse ranking of techniques and rough magnitude of results are interesting (and are consistent with our general experience).

The baseline 2x DNN size model doubles the number of hidden layers. Note, that sparse embedding lookups add to the overall training cost, thus doubling the number layers does not proportionally increase the cost.

Irreproducibility

PDs may be as high as $20\%$ for deep models. Perhaps surprisingly, standard methods such as fixed initialization, regularization, dropout, data augmentation, as well as new methods imposing constraints (Bhojanapalli et al., 2021; Shamir, 2018) either failed to improve PD or improved PD at the cost of accuracy degradation. Techniques like warm-starting model weights to values of previously trained models may not be preferable because they can anchor the model to a potentially bad solution space and do not help the development cycle for newer more reproducible models for which there is no anchor.

Other techniques have shown varying levels of success. Ensembles (Dietterich, 2000), specifically self-ensembles (Allen-Zhu and Li, 2020), where we average predictions of multiple model duplicates (each initialized differently), can reduce prediction variance and PD. However, maintaining ensembles in a production system with multiple components builds up substantial technical debt (Sculley et al., 2014). While some literature (Kondratyuk et al., 2020; Lobacheva et al., 2020; Wang et al., 2021a) describes accuracy advantages for ensembles, in our regime, ensembles degraded accuracy relative to equal-cost single networks. We believe this is because, unlike in the benchmark image models, examples in online CTR systems are visited once, and, more importantly, the learned model parameters are dominated by sparse embeddings. Relatedly, more sophisticated techniques based on ensembling and constraints can also improve PD (Anil et al., 2018; Shamir and Coviello, 2020a, b).

Techniques described above trade accuracy and complexity for better reproducibility, requiring either ensembles or constraints. Further study and experimentation revealed that the popular use of Rectified Linear Unit (ReLU) activations contributes to increased PD. ReLU’s gradient discontinuity at 0 induces a highly non-convex loss landscape. Smoother activations, on the other hand, reduce the amount of non-convexity, and can lead to more reproducible models (Shamir et al., 2020). Empirical evaluations of various smooth activations (Barron, 2017; Hendrycks and Gimpel, 2016; Ramachandran et al., 2017; Zheng et al., 2015) have shown not only better reproducibility compared to ReLU, but also slightly better accuracy. The best reproducibility-accuracy trade-offs in our system were attained by the simple Smooth reLU (SmeLU) activation proposed in (Shamir et al., 2020). The function form is:

In our system, $3$-component ensembles reduced PD from 17% to 12% and anti-distillation reduced PD further to 10% with no accuracy loss. SmeLU allowed launching a non-ensemble model with PD less than 10% that also improved accuracy by 0.1%. System reproducibility metrics also improved to acceptable levels compared to the unacceptable levels of ReLU single component models.

Generalizing Across UI Treatments

One of the major factors in CTR performance of an ad is its UI treatment, including positioning, placement relative to other results on the page, and specific renderings such as bolded text or inlined images. A complex auction must explore not just the set of results to show, but how they should be positioned relative to other content, and how they should be individually rendered (Cavallo et al., 2017). This exploration must take place efficiently over a combinatorially large space of possible treatments.

We solve this through model factorization, replacing estimated CTR with $\tau(Q\cdot U)$, composed of a transfer function $\tau$ where $Q$, $U$ are separable models that output vectorized representations of the Quality and the UI, respectively, and are combined using an inner-product. While $Q$, consisting of a large DNN and various feature embeddings, is a costly model, it needs to be evaluated only once per ad, irrespective of the number of UI treatments. In contrast, $U$, being a much lighter model, can be evaluated hundreds of times per ad. Moreover, due to the relatively small feature space of the UI model, outputs can be cached to absorb a significant portion of lookup costs (as seen in Figure 4).

Separately from model performance requirements, accounting for the influence of UI treatments on CTR is also a crucial factor for model quality. Auction dynamics deliberately create strong correlations between individual ads and specific UI treatments. Results that are lower on the page may have low CTR regardless of their relevance to the query. Failure to properly disentangle these correlations creates inaccuracy when generalizing over UI treatments (e.g., estimating CTR if the same ad was shown higher on the page). Pricing and eligibility decisions depend crucially on CTR estimates of sub-optimal UIs that are rarely occurring in the wild. For instance, our system shouldn’t show irrelevant ads, and so such scenarios will not be in the training corpus, and so estimates of their irrelevance (low CTR) will be out of distribution. But these estimates are needed to ensure the ads do not show. Even for relevant ads, there is a similar problem. Performance of ads that rarely show in first position may still be used to set the price of those ads that often do show in first position. This creates a specific generalization problem related to UI, addressed in Section 7.

Calibration is an important characteristic for large-scale ads recommendation. We define calibration bias as label minus prediction, and want this to be near zero per ad. A calibrated model allows us to use estimated CTR to determine the trade-off between showing and not showing an ad, and between showing one ad versus another; both calculations can be used in downstream tasks such as UI treatment selection, auction pricing, or understanding of ad viewability.

The related concept of credit attribution is similar to counterfactual reasoning (Bottou et al., 2013) or bias in implicit feedback (Joachims et al., 2017). It is a specific non-identifiability in model weights that can contribute to irreproducibility (Section 5). Consider an example to illustrate the UI effect (Section 6): assume that model A has seen many training examples with high-CTR ads in high positions, and (incorrectly) learned that ad position most influences CTR. Model B, defined similar to A, trains first on the few examples where high-CTR ads appear in low positions, and (correctly) learns that something else (e.g., ad relevancy to query) is causing high CTR. Both models produce the same estimated CTR for these ads but for different reasons, and when they are deployed, model A will likely show fewer ads because it will not consider otherwise useful ads in lower positions; these models will show system irreproducibility.

In our system, we use a novel, general-purpose technique called bias constraints to address both calibration and credit attribution. We add calibration bias constraints to our objective function, enforced on relevant slices of either the training set or a separate, labelled dataset. This allows us reduce non-identifiability by anchoring model loss to a desired part of the solution space (e.g., one that satisfies calibration) (Figure 5a). By extension, we reduce irreproducibility by anchoring a retrained model to the same solution.

Our technique is more lightweight than other methods used for large-scale, online training (counterfactual reasoning (Bottou et al., 2013), variations of inverse propensity scoring (Joachims et al., 2017; Lefortier et al., 2016)): in practice, there are fewer parameters to tune, and we simply add an additional term to our objective rather than changing the model structure. To address calibration, (Borisov et al., 2018) adjusts model predictions in a separate calibration step using isotonic regression, a non-parametric method. Our technique does calibration jointly with estimation, and is more similar to methods which consider efficient optimization of complex and augmented objectives (e.g., (Eban et al., 2017; Mann and McCallum, 2007)). Using additional constraints on the objective allows us to address a wide range of calibration and credit attribution issues.

Bias Constraints

We now optimize our original objective function with the constraint that $~{}\forall k\forall{i\in S_{k}},(y_{i}-\hat{y}_{i})=0$. Here, ${S_{k}}$ are subsets of the training set which we’d like to be calibrated (e.g., under-represented classes of data) or new training data that we may or may not optimize the original model weights over (e.g., out-of-distribution or off-policy data gathered from either randomized interventions or exploration scavenging (Wang et al., 2016; Joachims et al., 2017; Langford et al., 2008)). To aid optimization, we first transform this into an unconstrained optimization problem by introducing a dual variable $\lambda_{k,i}$ for each constraint and maximizing the Lagrangian relative to the dual variables. Next, instead of enforcing zero bias per example, we ask that the squared average bias across $S_{k}$ is zero. This reduces the number of dual variables to $\{\lambda_{k}\}$, and is equivalent to adding an L2 regularization on $\lambda_{k}$ with a constraint of zero average bias. For a constant $\alpha_{3}$ controlling regularization, and tuned via typical hyperparameter tuning techniques (e.g. grid search), our new optimization is: min_W max_λ_k ∑_i L(y_i, ^y_i) + ∑_k=1^K ∑_i ∈S_k (λ_k (y_i - ^y_i) - α32 λ_k^2) Any degraded accuracy or stability is mitigated by combinations of the following tunings, ordered by impact: ramping up the bias constraint term, reducing the learning rate on $\{\lambda_{k}\}$, increasing $\alpha_{3}$, or adding more or finer-grained constraints (breaking up $S_{k}$). We believe the first two can help normalize any differences between the magnitude of the dual variables and other weights, and the latter two help lessen the strength of the bias term if $S_{k}$ aren’t optimally selected.

2. Bias Constraints for General Calibration

If we plot calibration bias across buckets of interesting variables, such as estimated CTR or other system metrics, we expect a calibrated model to have uniform bias. However, for several axes of interest, our system shows higher bias at the ends of the range (Figure 5b). We apply bias constraints to this problem by defining $S_{k}$ to be examples in each bucket of, e.g., estimated CTR. Since we don’t use the dual variables during inference, we can include estimated CTR in our training objective. With bias constraints, bias across buckets of interest becomes much more uniform: variance is reduced by more than half. This can in turn improve accuracy of downstream consumers of estimated CTR.

3. Exploratory Data and Bias Constraints

We can also use bias constraints to solve credit attribution for UI treatments. We pick $S_{k}$ by focusing on classes of examples that represent uncommon UI presentations for competitive queries where the ads shown may be quite different. For example, $S_{1}$ might be examples where a high-CTR ad showed at the bottom of the page, $S_{2}$ examples where a high-CTR ad showed in the second-to-last position on the page, etc. Depending on how model training is implemented, it may be easier to define $S_{k}$ in terms of existing model features (e.g., for a binary feature ${f}$, we split one sum over $S_{k}$ into two sums). We choose $\{f\}$ to include features that generate partitions large enough to not impact convergence but small enough that we expect the bias per individual example will be driven to zero (e.g., if we think that query language impacts ad placement, we will include it in $\{f\}$). For the model in Table 3, we saw substantial bias improvements on several data subsets $S_{k}$ related to out-of-distribution ad placement and more reproducibility with minimal accuracy impact when adding bias constraints.

Viewing the bias constraints as anchoring loss rather than changing the loss landscape (Figure 5a), we find that the technique does not fix model irreproducibility but rather mitigates system irreproducibility: we were able to cut the number of components in the ensemble by half and achieve the same level of reproducibility.

Conclusion

We detailed a set of techniques for large-scale CTR prediction that have proven to be truly effective “in production”: balancing improvements to accuracy, training and deployment cost, system reproducibility and model complexity—along with describing approaches for generalizing across UI treatments. We hope that this brief visit to the factory floor will be of interest to ML practitioners of CTR prediction systems, recommender systems, online training systems, and more generally to those interested in large industrial settings.

References