Broken Neural Scaling Laws

Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger

Introduction

The amount of compute used for training, number of model parameters, and training dataset size of the most capable artificial neural networks keeps increasing and will probably keep rapidly increasing for the foreseeable future. However, no organization currently has direct access to these larger resources of the future; and it has been empirically verified many times that methods which perform best at smaller scales often are no longer the best performing methods at larger scales (e.g., one of such examples can be seen in Figure 2 (right) of Tolstikhin et al. (2021)). To work on, identify, and steer the methods that are most probable to stand the test-of-time as these larger resources come online, one needs a way to predict how all relevant performance evaluation metrics of artificial neural networks vary in all relevant settings as scale increases.

Neural scaling laws (Cortes et al., 1994; Hestness et al., 2017; Rosenfeld et al., 2019; Kaplan et al., 2020; Zhai et al., 2021; Abnar et al., 2021; Alabdulmohsin et al., 2022; Brown et al., 2020; Bahri et al., 2021) aim to predict the behavior of large-scale models from smaller, cheaper experiments, allowing to focus on the best-scaling architectures, algorithms, datasets, and so on. The upstream/in-distribution test loss typically (but not always!) falls off as a power law with increasing data, model size and compute. However, the downstream/out-of-distribution performance, and other evaluation metrics of interest (even upstream/in-distribution evaluation metrics) are often harder to predict, sometimes exhibiting inflection points (on a linear-linear plot) and non-monotonic behaviors. Discovering universal scaling laws that accurately model and extrapolate a wide range of potentially unexpected behaviors is clearly important not only for identifying that which scales best, but also for AI safety, as predicting the emergence of novel capabilities at scale could prove crucial to responsibly developing and deploying increasingly advanced AI systems. The functional forms of scaling laws evaluated in previous work are not up to this challenge.

One salient defect is that they can only represent monotonic functions. They thus fail to model the striking phenomena of double-descent (Nakkiran et al., 2021), where increased scale temporarily decreases test performance before ultimately leading to further improvements. Many also lack the expressive power to model inflection points (on a linear-linear plot), which can be observed empirically for many downstream tasks, and even some upstream tasks, such as our $N$ -digit arithmetic task, or the modular arithmetic task introduced by Power et al. (2022) in their work on “grokking”.

To address all the needs discussed in this paper thus far, we present broken neural scaling laws (BNSL) - a functional form that generalizes power laws (linear in log-log plot) to “smoothly broken” power laws, i.e. a smoothly connected piecewise (approximately) linear function in a log-log plot. An extensive empirical evaluation demonstrates that BNSL accurately model & extrapolate the scaling behaviors for various architectures & for each of various tasks within a large & diverse set of upstream & downstream tasks, in zero-shot, prompted, & fine-tuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, arithmetic, supervised learning, unsupervised/self-supervised learning, & reinforcement learning (single agent & multi-agent). When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Additionally, it captures well (i.e. accurately models & extrapolates) the non-monotonic transitions present in the scaling behavior of phenomena such as double descent & the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic.

The Functional Form of Broken Neural Scaling Laws

The general functional form of a broken neural scaling law (BNSL) is given as follows:

where $y$ represents a performance evaluation metric (e.g. prediction error, cross entropy, calibration error, AUROC, BLEU score percentage, F1 score, reward, Elo rating, or FID score) (downstream or upstream) and $x$ represents a quantity that is being scaled (e.g. number of model parameters, amount of compute used for training (or inference), training dataset size, model input size, number of training steps, or upstream performance). The remaining parameters $a,b,c_{0},c_{1}...c_{n},d_{1}...d_{n},f_{1}...f_{n}$ are unknown constants that must be estimated by fitting the above functional form to the $(x,y)$ data points. (In our experiments, SciPy curve-fitting library (Virtanen et al., 2020) was used.)

The constants in Equation 1 above are interpreted as follows. Constant $n$ represents the number of (smooth) “breaks” (i.e. transitions) between $n+1$ consecutive approximately linear (on a log-log plot) segments, for a total of $n+1$ approximately linear segments (on a log-log plot); when $n=0$ , equation 1 becomes $y=a+bx^{-c_{0}}$ . Constant $a$ represents the limit as to how far the value of $y$ (performance evaluation metric) can be reduced (or maximized) even if $x$ (the quantity being scaled) goes to infinity. Constant $b$ represents the offset of functional form on a log-log plot (analogous to the intercept $b$ in $y=mx+b$ on a linear-linear plot). Constant $c_{0}$ represents the slope of the first approximately linear region on a log-log plot. If there is a “slope $=0$ ” random guessing region (often there is not) at the beginning of the scaling behavior, $c_{0}=0$ and $b=(random\_guessing\_performance-a)$ . Constant $c_{i}$ represents the difference in slope of the $(i)$ th approximately linear region and the $(i+1)$ th approximately linear region on a log-log plot. Constant $d_{i}$ represents where on the x-axis the break between the $(i)$ th and the $(i+1)$ th approximately linear region (on a log-log plot) occurs. Constant $f_{i}$ represents the sharpness of the break between the $(i)$ th and the $(i+1)$ th approximately linear region on a log-log plot; smaller (nonnegative) values of $f_{i}$ yield a sharper break and intervals (before and after the $(i)$ th break) that are more linear on a log-log plot; larger values of $f_{i}$ yield a smoother (wider) break and intervals (before and after the $(i)$ th break) that are less linear on a log-log plot.

For mathematical analysis and explanation of why Equation 1 is smoothly piece-wise (approximately) linear function on a log-log plot, see Appendix A.1. For mathematical decomposition of Equation 1 into the power law segments it is composed of (e.g. as in Figure 1), see Appendix A.2.

Note that, a smoothly connected approximately piece-wise linear (in log-log plot) function such as BNSL can, with enough segments, fit well any smooth univariate function. However, it remained unclear whether BNSL would also extrapolate the scaling behaviors of deep neural networks well; yet as we demonstrate below, it extrapolates quite accurately. Additionally, we find that the number of breaks needed to accurately model an entire scaling behavior is often quite small.

Related Work

To the best of our knowledge, Cortes et al. (1994) was the first paper to model the scaling of multilayer neural network’s performance as a power law (also known as a scaling law) (plus a constant) of the form $y=ax^{b}+c$ in which $x$ refers to training dataset size and $y$ refers to test error; we refer to that functional form as M2. Hestness et al. (2017) showed that the functional form, M2, holds over many orders of magnitude. Rosenfeld et al. (2019) demonstrated that the same functional form, M2, applies when $x$ refers to model size (number of parameters). Kaplan et al. (2020) brought “neural” scaling laws to the mainstream & demonstrated that the same functional form, M2, applies when $x$ refers to the amount of compute used for training. Abnar et al. (2021) proposed to use the same functional form, M2, to relate downstream performance to upstream performance. Zhai et al. (2021) & Bansal et al. (2022) introduced the functional form $y=a(x+d)^{b}+c$ , (referred to by us as M3) where $d$ represents the scale at which the performance starts to improve beyond the random guess loss (a constant) & transitions to a power law scaling regime. Alabdulmohsin et al. (2022) proposed functional form $(y-\epsilon_{\infty})/((\epsilon_{0}-y)^{a})=bx^{c}$ , (referred to by us as M4) where $\epsilon_{\infty}$ is irreducible entropy of the data distribution and $\epsilon_{0}$ is random guess performance, for relating scale to performance & released a scaling laws benchmark dataset that we use in our experiments.

Hernandez et al. (2021) described a smoothly broken power law functional form (consisting of 5 constants after reducing redundant variables) in equation 6.1 of their paper, when relating scale and downstream performance. While this functional form can be summed with an additional constant representing unimprovable performance to obtain a functional form whose expressivity is equivalent to our BNSL with a single break, it is important to note that (i) Hernandez et al. (2021) describes this form only in the specific context, when exploring how fine-tuning combined with transfer learning scales as a function of the model size - thus, their functional form contains a break only with respect to number of model parameters but not with respect to other input quantities which we do explore such as dataset size, compute, number of training steps, model input size, & upstream performance; (ii) they mentioned this equation in passing and as a result did not try to fit or verify this functional form on any data; (iii) they arrived at it simply via combining the scaling law for transfer (that was the focus of their work) with a scaling law for pretraining data; (iv) they did not identify it as a smoothly broken power law, or note any qualitative advantages of this functional form; (v) they did not discuss the family of functional forms with multiple breaks.

Finally, we would like to mention that smoothly broken power law functional forms, equivalent to equation 1, are commonly used in the astrophysics literature (e.g. dam (2017)) as they happen to model well a variety of physical phenomena. This inspired us to investigate their applicability to a wide range of deep neural scaling phenomena as well.

Theoretical Limitations of Previously Proposed Scaling Laws

Our use of BNSLs is inspired by the observation that scaling is not always well predicted by a simple power law; nor are many of the modifications which have been applied in previous works sufficient to capture the qualitative properties of empirical scaling curves. Here we show mathematically two qualitative defects of these functional forms:

They are strictly monotonic (first-order derivative does not change its sign) and thus unable to fit double descent phenomena.

They cannot express inflection points (second-order derivative does not change its sign), which are frequently observed empirically. An exception to this is M4, proposed by Alabdulmohsin et al. (2022).

Note that some of these functional forms (e.g. M3) can exhibit inflection points on the log-log axes which are commonly used for plotting scaling data (as it was observed in several prior works). However, for inflection points on a linear-linear plot, the extra expressiveness of broken neural scaling laws appears to be necessary (and sufficient). Figure 4 and Figure 5 provide examples of BNSLs producing non-monotonic behavior and inflection points, respectively, establishing the capacity of this functional form to model and extrapolate these phenomena that occur in real scaling behaviors.

M1, M2, M3 functional forms cannot model non-monotonic behavior or inflection points: First, recall that expressions of the form $m^{n}$ can only take the value if $m=0$ . We now examine the expressions for the first and second derivatives of M1, M2, M3, provided in Table 1, and observe that they are all continuous and do not have roots over the relevant ranges of their variables, i.e. $x>0$ in general and $b<0$ in the case of M3 (we require $x>0$ because model size, dataset size, and compute are always non-negative). This implies that, for any valid settings of the parameters $a,b,c,d,x$ , these functional forms are monotonic (as the first derivative never changes sign), and that they lack inflection points (since an inflection point must have $f^{\prime\prime}(x)=0$ ).

M4 functional form cannot model non-monotonic behavior. The case of M4 is a bit different, since the relationship between $y$ and $x$ in this case is expressed as an inverse function, i.e.

However, non-monotonicity of $y$ as an inverse function $y=g^{-1}(x)$ is ruled out, since that would imply two different values of $x=g(y)$ can be obtained for the single value of $y$ – this is impossible, since $f(y)$ maps each $y$ deterministically to a single value of $x$ . As a result, M4 cannot express non-monotonic functions.

M4 functional form can model inflection points. It is easy to see that if $y=g^{-1}(x)$ had an inflection point, then $x=g(y)$ would have it as well. This is because an inflection point is defined as a point $x$ where $f(x)$ changes from concave to convex, which implies that $g(y)$ changes from convex to concave, since the inverse of a convex function is concave; the root(s) of $g^{\prime\prime}(y)$ are the point(s) at which this change occurs. Using Wolfram Alphahttps://www.wolframalpha.com/ and matplotlib (Hunter, 2007), we observe that M4 is able to express inflection points, e.g. $(a,b,c,\epsilon_{0},\epsilon_{\infty},x,y)=(1,1,-2,3/4,1/4,1/\sqrt{3},5/8$ ), or $(a,b,c,\epsilon_{0},\epsilon_{\infty},x,y)=(2,1,-3,2/3,1/3,(-5/6+\sqrt{3}/2)^{1/3},1/\sqrt{3})$ .

Empirical Results: Fits & Extrapolations of Functional Forms

We now show the fits & extrapolations of various functional forms. In all plots here, onward, & in the appendix, black points are points used for fitting a functional form, green (gray if color blind) points are held-out points used for evaluating extrapolation of functional form fit to the black points, & a red line is BNSL that has been fit to black points. 100% of the plots in this paper here, onward, & in the appendix contain green point(s) for evaluating extrapolation. See Section A.6 for details on fitting BNSL & determining the number of breaks. Except when stated otherwise, each plot contains the region(s) surrounding (or neighboring) a single break of a BNSL fit to black points that are smaller (along the x-axis) than the green points.

All the extrapolation evaluations reported in the tables (that have $\downarrow$ symbol in the top row) are reported in terms of root mean squared log error (RMSLE) ± root standard log error. See Appendix A.3 for definition of RMSLE and Appendix A.4 for definition of root standard log error.

Using the scaling laws benchmark of Alabdulmohsin et al. (2022), we evaluate how well various functional forms extrapolate performance on downstream vision tasks as upstream training dataset size increases. In this large-scale vision subset of the benchmark, the tasks that are evaluated are error rate on each of various few-shot downstream image classification (IC) tasks; the downstream tasks are: Birds 200 (Welinder et al., 2010), Caltech101 (Fei-Fei et al., 2004), CIFAR-100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009). The following architectures of various sizes are pretrained on subsets of JFT-300M (Sun et al., 2017): big-transfer residual neural networks (BiT) (Kolesnikov et al., 2020), MLP mixers (MiX) (Tolstikhin et al., 2021), and vision transformers (ViT) (Dosovitskiy et al., 2020). As can be seen in Tables 2 and 3, BNSL yields extrapolations with the lowest RMSLE (Root Mean Squared Logarithmic Error) for 69.44% of tasks of any of the functional forms, while the next best functional form performs the best on only 13.89% of the tasks.

To view plots of BNSL on each of these tasks, see figures 34, 35, 36, 40 in Appendix A.36. To view plots of M1, M2, M3, M4 on each of these tasks, see Appendix A.4 of Alabdulmohsin et al. (2022).

In Section A.9, we additionally show that BNSL yields accurate extrapolations of performance on large-scale downstream vision tasks when amount of compute used for (pre-)training is on the x-axis and compute is scaled in the manner that is Pareto optimal with respect to the performance evaluation metric on the y-axis (downstream accuracy in this case).

In Section A.11, we additionally show that BNSL yields accurate extrapolations of the scaling behavior of diffusion generative models of images.

In Section A.12, we additionally show that BNSL yields accurate extrapolations of the scaling behavior of generative models of video.

In Section A.24, we show that BNSL yields accurate extrapolations of robotics scaling behavior (out-of-distribution (OOD) generalization and in-distribution generalization).

In Section A.23, BNSL accurately extrapolates the scaling behavior of continual learning.

In Section A.19, BNSL accurately extrapolates the scaling behavior of adversarial robustness.

In Section A.30, BNSL accurately extrapolates the scaling behavior of fairness (and also ensembles).

In Section A.33, we show that BNSL accurately extrapolates the scaling behavior of the downstream performance of multimodal contrastive learning (e.g. non-generative unsupervised learning).

In Section A.13, we additionally show that BNSL yields accurate extrapolations of the scaling behavior when data is pruned Pareto optimally (such that each point along the x-axis uses the subset of the dataset that yields the best performance (y-axis value) for that dataset size (x-axis value)).

In Section A.14, we additionally show that BNSL yields accurate extrapolations when upstream performance is on the x-axis and downstream performance is on the y-axis.

In Section A.8 (and Figure 2), we additionally show that BNSL accurately extrapolates downstream performance to scales that are over an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting.

2 Language

Using the scaling laws benchmark of Alabdulmohsin et al. (2022), we evaluate how well various functional forms extrapolate performance on language tasks as the (pre-)training dataset size increases. In this large-scale language subset of the benchmark, the tasks that are evaluated are error rates on each of the various downstream tasks from the BIG-Bench (BB) (Srivastava et al., 2022) benchmark and upstream test cross-entropy of various models (e.g. Transformers & occasionally LSTMs) trained to do language modeling (LM) & neural machine translation (NMT). All LM & BB tasks use a decoder-only language model. As can be seen in Tables 2 & 4, BNSL yields extrapolations with the lowest RMSLE (Root Mean Squared Logarithmic Error) for 75% of tasks of any of the functional forms, while the next best functional form performs the best on only 10% of the tasks.

To view all plots of the BNSL on each of these tasks, see Figures 37, 39, 38 in Appendix A.36. To view plots of M1, M2, M3, and M4 on these tasks, see Figure 8 of Alabdulmohsin et al. (2022).

In Section A.15 (and also Figure 2), we additionally show that BNSL yields accurate extrapolations of performance (to scales that are over an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting) on large-scale downstream language tasks when number of model parameters is on the x-axis.

In Section A.18, we additionally show that BNSL accurately models and extrapolates the scaling behavior of sparse models (i.e. sparse, pruned models and sparsely gated mixture-of-expert models).

In Section A.16, BNSL accurately extrapolates the scaling behavior of retrieval-augmented models.

In Section A.17, BNSL accurately extrapolates the scaling behavior with input size (also know as input / context length) of the model on the x-axis.

In Section A.32, BNSL also accurately extrapolates the scaling behavior of recurrent neural nets.

In Section A.34, we additionally show that BNSL yields accurate extrapolations of performance on large-scale downstream audio (speech recognition) tasks.

In Section A.20, we show BNSL accurately models and extrapolates the scaling behavior with finetuning dataset size on the x-axis and the scaling behavior of computer programming / coding.

In Section A.21, we additionally show BNSL accurately models and extrapolates the scaling behavior of uncertainty estimation / calibration.

In Section A.28, BNSL accurately extrapolates the scaling behavior of tasks involving molecules.

In Section A.27, BNSL accurately extrapolates the scaling behavior of OOD detection.

In Section A.25, BNSL accurately extrapolates the scaling behavior of quantization.

In Section A.29, BNSL accurately extrapolates the scaling behavior of math word problems.

In Section A.26, BNSL accurately extrapolates the scaling behavior of distillation.

In Section A.35, BNSL accurately extrapolates the scaling behavior of downstream performance to scales that are over 100,000 times larger than the maximum (along x-axis) of the points used for fitting (and also when amount of compute used for (pre-)training is on the x-axis and compute is scaled in the manner that is Pareto optimal with respect to the downstream performance evaluation metric on the y-axis).

3 Reinforcement Learning

We show that BNSL accurately models and extrapolates the scaling behaviors of various multi-agent and single-agent reinforcement learning algorithms trained in various environments. In the top left plot and top middle plot and top right plot of Figure 3, BNSL accurately models and extrapolates the scaling behavior of the AlphaZero algorithm (Silver et al., 2017) trained to play the game Connect Four from Figure 4 and Figure 5 and Figure 3 respectively of Neumann & Gros (2022); the x-axes respectively are compute (FLOPs) used for training, training dataset size (states), and number of model parameters. In Figure 3 bottom left and bottom right respectively, BNSL accurately models and extrapolates the scaling behavior of the Phasic Policy Gradient (PPG) algorithm (Cobbe et al., 2021b) trained to play the Procgen (Cobbe et al., 2020) game called StarPilot and the scaling behavior of the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) trained to play the Procgen (Cobbe et al., 2020) game called Heist. In Section A.31, we additionally find BNSL accurately extrapolates the scaling behavior with amount of test-time (a.k.a. inference) compute (available to the agent) on the x-axis. In Section A.22, we additionally find BNSL accurately extrapolates the scaling behavior of a pretrained language model finetuned (i.e. aligned) via Reinforcement Learning from Human Feedback (RLHF) to be helpful from Figure 1 of Bai et al. (2022).

4 Non-Monotonic Scaling

We show that BNSL accurately models and extrapolates non-monotonic scaling behaviors such as those that are exhibited by Transformers (Vaswani et al. (2017)) during double descent (Nakkiran et al., 2021) in Figure 4. Various other functional forms are mathematically incapable of expressing non-monotonic behaviors (as shown in Section 4).

5 Inflection Points

We show that BNSL accurately models and extrapolates the scaling behavior of tasks that have an inflection point on a linear-linear plot such as the task of arithmetic (4-digit addition). Here we model and extrapolate the scaling behavior of a transformer model (Vaswani et al. (2017)) with respect to the training dataset size on the 4-digit addition task. Various other functional forms are mathematically incapable of expressing inflection points on a linear-linear plot (as shown in Section 4) and as a result, are mathematically incapable of expressing and modeling inflection points (on a linear-linear plot) that are present in the scaling behavior of 4-digit addition. In Figure 5 left, we show that BNSL expresses and accurately models the inflection point present in the scaling behavior of 4-digit addition and as a result accurately extrapolates the scaling behavior of 4-digit addition. For more details about the hyperparameters, see Appendix A.5. Also, in Section A.10 we additionally find that BNSL accurately models and extrapolates the scaling behavior with number of training steps on the x-axis. Also, in Section A.26 and the bottom left plot of Figure 16 of Section A.18 we additionally find that BNSL accurately models and extrapolates the scaling behavior of tasks that have an inflection point on a linear-linear plot when number of models parameters is on the x-axis.

The Limit of the Predictability of Scaling Behavior

We use BNSL to glean insights about the limit of the predictability of scaling behavior. Recent papers (Ganguli et al., 2022; Wei et al., 2022a; Srivastava et al., 2022) have advertised many tasks as having “emergent” “breakthrough” “phase transition/change/shift” scaling behaviors. We define all these quoted phrases as any scaling behaviors with greater than breaks (e.g. $\sim$ all the plots in this paper and the appendix due to the fact that 100% of plots in this paper and the appendix contain the region(s) surrounding (or neighboring) $\geq$ 1 break(s) of a BNSL fit to black points). The most extreme of these scaling behaviors are those in which it is simultaneously true that $|c_{i>0}|$ is large and $f_{i}$ is small (and nonnegative); one such example is the task of 4-digit addition (there are other such tasks in the appendix). In the previous section and in Figure 5 left, we successfully predicted (i.e. extrapolated) the scaling behavior of 4-digit addition (arithmetic). However, we are only able to accurately extrapolate the scaling behavior if given some points from training runs with a training dataset size of at least 720, and the break in which the scaling behavior of 4-digit addition transitions from one power law to another steeper power-law happens at around training dataset size of 415.

Ideally, one would like to be able to extrapolate the entire scaling behavior by fitting only points from before the break. In Figure 5 right, we use a noiseless simulation of the BNSL of 4-digit addition to show what would happen if one had infinitely many training runs / seeds to average out all the noisy deviation between runs such that one could recover (i.e. learn via a curve-fitting library such as SciPy (Virtanen et al., 2020)) the learned constants of the BNSL as well as possible. When using this noiseless simulation, we find that we are only able to accurately extrapolate the scaling behavior if given some points from training runs with a training dataset size of at least 415, which is very close to the break.

1) When the scaling behavior exhibits greater than 0 breaks that are sufficiently sharp, there is a limit as to how small the maximum (along the x-axis) of the points used for fitting can be if one wants to perfectly extrapolate the scaling behavior, even if one has infinitely many seeds / training runs.

2) If an additional break of sufficient sharpness happens at a scale that is sufficiently larger than the maximum (along the x-axis) of the points used for fitting, there does not (currently) exist a way to extrapolate the scaling behavior after that additional break.

3) If a break of sufficient sharpness happens at a scale sufficiently smaller than the maximum (along the x-axis) of the points used for fitting, points smaller (along the x-axis) than that break can often be useless for improving extrapolation. See Appendix A.7 for examples of scenarios in which points smaller (along the x-axis) than that break do improve extrapolation.

Discussion

We have presented a smoothly broken power law functional form that accurately models & extrapolates the scaling behaviors of artificial neural networks for various architectures & for each of various tasks from a very large & diverse set of upstream & downstream tasks. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution generalization, continual learning, transfer learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, “emergent” “phase transitions”, arithmetic, supervised learning, unsupervised/self-supervised learning, & reinforcement learning (single agent & multi-agent). The architectures in this set include ResNets, Transformers, MLPs, MLP-Mixers, RNNs, GNNs, U-Nets, Ensembles (& Non-Ensembles), MoE (& Non-MoE) models, & Sparse Pruned (& Non-Sparse Unpruned) models. When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Additionally, this functional form accurately models & extrapolates scaling behavior that other functional forms are incapable of expressing such as the non-monotonic transitions present in the scaling behavior of phenomena such as double descent & the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we used this functional form to glean insights about the limit of the predictability of scaling behavior.

We place relatively high probability on the claim that variants of smoothly broken power laws perhaps are the “true” functional form of the scaling behavior of many (all?) things that involve artificial neural networks. Due to the fact that BNSL is a variant of smoothly broken power laws, an ethical concern one might have about our work is that revealing BNSL might differentially (Hendrycks & Mazeika, 2022) improve A(G)I capabilities progress relative to A(G)I safety/alignment progress. A counter-argument is that BNSL will also allow the A(G)I safety/alignment field to extrapolate the scaling behaviors of its methods for aligning A(G)I systems and as a result will also accelerate alignment/safety progress. Existing scaling laws besides BNSL struggle especially to model downstream performance, e.g. on safety-relevant evaluations (especially evaluations (such as interpretability and controllability) that might exhibit non-monotonic scaling behavior in the larger scale systems of the future); we believe our work could differentially help in forecasting emergence of novel capabilities (such as reasoning (Wei et al., 2022b)) or behaviors (such as deception or dishonesty (Evans et al., 2021; Lin et al., 2021)), and thus help avoid unpleasant surprises.

A potential limitation of the current approach is the need to collect enough samples of the system’s performance (i.e. the (x,y) points required for estimating the scaling law’s parameters). A small number of samples sometimes may not be sufficient to accurately fit and extrapolate the BNSL functional form, and obtaining a large number of such samples can sometimes be costly. This has the ethical implication that entities with more compute to gather more points maybe will have considerably more accurate extrapolations of scaling behavior than entities with less compute. As a result, entities with less compute (e.g. academia) maybe will have less foresight than entities with more compute (e.g. Big Tech), which could maybe exacerbate the gap between entities with more compute (e.g. Big Tech) and entities with less compute (e.g. academia).

Acknowledgments

We are thankful for useful feedback and assistance from Kartik Ahuja, Ibrahim Alabdulmohsin, Ankesh Anand, Jacob Buckman, Guillaume Dumas, Leo Gao, Andy Jones, Neel Nanda, Behnam Neyshabur, Gabriel Prato, Stephen Roller, Michael Trazzi, Tony Wu and others.

Appendix A Appendix

Analysing Equation 1 reveals why BNSL is smoothly connected piecewise (approximately) linear function on a log-log plot. Considering $y$ as a function of $z:=\log(x)$ , applying logarithms to both sides and setting $a=0$ yields:

A.2 Decomposition of Broken Neural Scaling Law into power law segments that it is composed of

We now show a way to decompose a BNSL (Equation 1) with 3 breaks into the power law segments that it is composed of. This decomposition is what we used to produce power law segments 1-4 overlaid in Figure 1 and is usable when values of $f$ in Equation 1 are not too large and $a$ is subtracted out from right-hand side of Equation 1. This decomposition pattern is straight-forward to extend to $n$ breaks.

$segment_{2}=b*(d_{1})^{-(c_{0})}*(x\hskip 3.98337pt/d_{1})^{-(c_{1}+c_{0})}$

$segment_{3}=b*(d_{1})^{-(c_{0})}*(d_{2}/d_{1})^{-(c_{1}+c_{0})}*(x\hskip 3.98337pt/d_{2})^{-(c_{2}+c_{1}+c_{0})}$

$segment_{4}=b*(d_{1})^{-(c_{0})}*(d_{2}/d_{1})^{-(c_{1}+c_{0})}*(d_{3}/d_{2})^{-(c_{2}+c_{1}+c_{0})}*(x\hskip 3.98337pt/d_{3})^{-(c_{3}+c_{2}+c_{1}+c_{0})}$

A.3 Definition of Root Mean Squared Log Error

A.4 Definition of Root Standard Log Error

A.5 Experimental details of Sections 5.5 and A.10

We perform an extensive set of experiments to model and extrapolate the scaling behavior for the 4-digit arithmetic addition task with respect to the training dataset size. Our code is based on the minGPT implementation (Karpathy, 2020). We set the batch size equal to the training dataset size. We do not use dropout nor a learning rate decay / schedule / warmup here. Each experiment was run on a single V100 GPU and each run took less than 2 hours. For our experiments we train the transformer model using the following set of hyperparameters:

A.6 Experimental details of fitting BNSL and determining the number of breaks

We fit BNSL as follows: We first use scipy.optimize.brute to do a grid search of the values of the constants ( $a,b,c_{0},c_{1}...c_{n},d_{1}...d_{n},f_{1}...f_{n}$ ) of BNSL that best minimize the mean squared log error (MSLE) between the real data and the output of BNSL. We then use the values obtained from the grid search as the initialization of the non-linear least squares algorithm of scipy.optimize.curve_fit. We then use the non-linear least squares algorithm of scipy.optimize.curve_fit to minimize the mean squared log error (MSLE) between the real data and the output of BNSL.

The version of MSLE we use for such optimization is the following numerically stable variant:

With regards to determining the number of breaks $n$ in the BNSL, one way to go about doing so is to hold out the last few largest (along the x-axis) points used for fitting (not the green points used for evaluating extrapolation) as a validation set. The value of $n$ with lowest validation error when fitting on the remaining smaller (along the x-axis) points is then used. As mentioned in implication 3 of Section 6, there are many scenarios in which a break of sufficient sharpness happens at a scale sufficiently smaller than the maximum (along the x-axis) of the points used for fitting such that points smaller (along the x-axis) than that break can often be useless for improving extrapolation. For example, if a full scaling behavior contains a very sharp break and then a very smooth break, one can crop out the smaller (along the x-axis) points that contain the sharp break and fit a BNSL with a single break (i.e. with $n=1$ ) if one only cares about extrapolation accuracy. To determine where the crop point is, one can employ the same validation set strategy mentioned at the beginning of this paragraph, but selecting for crop point instead of number of breaks. One reason why someone may want to use the strategies mentioned throughout this paragraph is that sometimes the value(s) of $|c_{i>0}|$ can be small such that it is hard for humans to identify the break(s) via eyeballing.

There are examples in the appendix (e.g. one example is in Figure 24 of Appendix A.26) in which BNSL accurately models and accurately extrapolates scaling behaviors in which the points used for fitting contain only one point that is larger (along the x-axis) than the largest break. In such examples, it is impossible to accurately extrapolate the scaling behavior using only points after that break because a scaling behavior is unidentifiable if one can only use a single point for fitting. In such examples, BNSL accurately integrates information from before that break to obtain accurate extrapolation.

In Figure 6, we show that BNSL accurately extrapolates to scales that are over an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting. The upstream task is supervised pretraining of MLP mixers (MiX) (Tolstikhin et al., 2021) on subsets (i.e. the x-axis of plot) of JFT-300M (Sun et al., 2017). The downstream task is n-shot ImageNet classification (i.e. the y-axis of plot). The experimental data of this scaling behavior is obtained from Alabdulmohsin et al. (2022).

In Figure 7 via fitting BNSL, we additionally obtain accurate extrapolations of the scaling behavior of large-scale downstream vision tasks when compute (FLOPs) used for (pre-)training is on the x-axis and compute is scaled in the manner that is Pareto optimal with respect to the performance evaluation metric (downstream accuracy in this case). The y-axis is downstream test error rate on ImageNet. The upstream task is supervised pretraining of ViT (Dosovitskiy et al., 2020) models on subsets of JFT-300M (Sun et al., 2017). The experimental scaling data was obtained from Figure 2 of Zhai et al. (2021), and as a result in Table 6 we compare extrapolation of BNSL to the extrapolation of M3 (which was proposed in Zhai et al. (2021)); we find that BNSL yields extrapolations of scaling behavior that are more accurate (i.e. have lower RMSLE) on these tasks.

A.10 Extrapolation Results with Number of Training Steps on the x-axis

In Figure 8, we find BNSL accurately extrapolates the scaling behavior with number of training steps on the x-axis.

A.11 Extrapolation Results for Diffusion Generative Models of Images

In Fig. 9, BNSL accurately extrapolates the scaling behavior of Diffusion Generative Image Models.

A.12 Extrapolation Results for Generative Models of Video

In Figure 10, we show that BNSL accurately extrapolates the scaling behavior of generative models of video. Each frame is downsampled to a pixel resolution of 64x64; each frame is then tokenized via a pretrained 16x16 VQVAE (Van Den Oord et al., 2017) to obtain 256 tokens per frame. 16 consecutive frames are then input to an autoregressive transformer as a length 4096 (16x16x16) sequence. The training dataset (& testing dataset) is from 100 hours of videos scraped from the web.

A.13 Extrapolation Results when Data is Pruned Pareto Optimally

In Figure 11, we show that BNSL accurately extrapolates the scaling behavior when data is pruned Pareto optimally (such that each point along the x-axis uses the subset of the dataset that yields the best performance (y-axis value) for that dataset size (x-axis value)) from Figure 3D of Sorscher et al. (2022).

A.14 Extrapolation Results when Upstream Performance is on the x-axis

In Figure 12, we show that BNSL accurately extrapolates the scaling behavior when upstream performance is on the x-axis and downstream performance is on the y-axis. The upstream task is supervised pretraining of ViT (Dosovitskiy et al., 2020) on subsets of JFT-300M (Sun et al., 2017). The downstream task is 20-shot ImageNet classification. The experimental data of this scaling behavior is obtained from Figure 5 of Abnar et al. (2021).

We find in general for each of every modality that the variance between seeds is higher when number of model parameters is on x-axis (as opposed to e.g. training dataset size on the x-axis). Table H.1 of the GPT-3 arXiv paper (Brown et al., 2020) release includes results for 8 numbers of model parameters. In Figure 13, we include examples of when 8 numbers of model parameters (7 for fitting, and largest held-out to evaluate extrapolation) are sufficient for obtaining accurate downstream extrapolation from BNSL due to variance between seeds being low enough. For many other downstream tasks with number of model parameters on the x-axis, the variance between seeds is much higher such that a number considerably larger than 7 points along the curve is needed to obtain an accurate extrapolation.

A.16 Extrapolation Results for Retrieval-Augmented Models

In Figure 14, we find BNSL accurately extrapolates the scaling behavior (even downstream) of models augmented with a mechanism to retrieve data from a very large collection of data. In right plot of Figure 14, x-axis notably is Size (in Number of Tokens) of the Retrieval Dataset.

A.17 Extrapolation Results with Input Length (of the Model) on the x-axis

In Figure 15, we find BNSL accurately extrapolates the scaling behavior (even downstream) with input size (also known as context length) (of the model) on the x-axis.

A.18 Extrapolation Results for Sparse Models

In Figure 16, we find BNSL accurately extrapolates the scaling behavior of sparse models (i.e. sparse, pruned models (bottom 2 plots) & sparsely gated mixture-of-expert models (top 2 plots)).

A.19 Extrapolation Results for Adversarial Robustness

In Figure 17, we find BNSL accurately extrapolates the scaling behavior of adversarial robustness. The adversarial test set is constructed via a projected gradient descent (PGD) attacker (Madry et al., 2018) of 20 iterations. During training, adversarial examples for training are constructed by PGD attacker of 30 iterations.

A.20 Extrapolation Results with Finetuning Dataset Size on the X-axis (and also for Computer Programming / Coding)

In Figure 18, we find BNSL accurately models and extrapolates the scaling behavior with finetuning dataset size on the x-axis (i.e. model that is pretrained on a large amount of mostly english text from the internet and then finetuned on a large amount of python code).

A.21 Extrapolation Results for Uncertainty Estimation / Calibration

In Figure 19, we find BNSL accurately extrapolates the scaling behavior of downstream uncertainty estimation / calibration on BIG-Bench (Srivastava et al., 2022).

A.22 Extrapolation Results for AI Alignment via RLHF

In Figure 20, we find BNSL accurately extrapolates the scaling behavior of a pretrained language model finetuned (i.e. aligned) via Reinforcement Learning from Human Feedback (RLHF) to be helpful from Figure 1 of Bai et al. (2022). The y-axis is Elo score based on crowdworker preferences. The x-axis is the number of model parameters that the language model contains.

A.23 Extrapolation Results for Continual Learning (i.e. Catastrophic Forgetting)

In Figure 21, we find that BNSL accurately extrapolates the scaling behavior of continual learning (i.e. catastrophic forgetting).

A.24 Extrapolation Results for Robotics (Out-of-Distribution Generalization and In-distribution Generalization)

In Figure 22, we find BNSL accurately extrapolates the scaling behavior of a transporter (Zeng et al., 2021) model trained via imitation learning to do robotic control tasks. Plots with “unseen-colors” in the plot title evaluate on a test set that contains colors that are unseen (i.e. out-of-distribution) with respect to the training set. Plots with “seen-colors” in the plot title evaluate on a test set that contains colors that are seen (i.e. in-distribution) with respect to the training set.

A.25 Extrapolation Results for Quantization

In Figure 23, we find BNSL accurately extrapolates the scaling behavior (even downstream) of quantized models.

A.26 Extrapolation Results for Distillation

In Figure 24, we find BNSL accurately extrapolates the scaling behavior of distillation (even to scales over an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting).

A.27 Extrapolation Results for Out-of-Distribution Detection

In Figure 25, we find BNSL accurately extrapolates the scaling behavior of Out-of-Distribution Detection performance (even to scales over an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting).

A.28 Extrapolation Results for Molecules

In Figure 26, we find BNSL accurately extrapolates the scaling behavior of Neural Equivariant Interatomic Potentials (NequIP) graph neural networks (Batzner et al., 2022) trained via minimizing the weighted sum of energy and a force loss terms in order to predict the forces of molecules.

A.29 Extrapolation Results for Math Word Problems

In Figure 27, we find BNSL accurately extrapolates the scaling behavior of large language models finetuned to solve math word problems.

A.30 Extrapolation Results for Fairness (and also Ensembles)

In Figure 28, we find BNSL accurately extrapolates the scaling behavior of fairness (and also ensembles).

A.31 Extrapolation Results with Amount of Test-Time (a.k.a. Inference) Compute on the X-axis

In Figure 29, we find BNSL accurately extrapolates the scaling behavior when the amount of test-time (a.k.a. inference) compute (available to the model) is on the x-axis.

A.32 Extrapolation Results for Recurrent Neural Networks

In Figure 30, we find BNSL accurately extrapolates the scaling behavior of recurrent neural networks (even downstream).

In Figure 31, we show that BNSL accurately extrapolates the scaling behavior of the Downstream Performance of Multimodal Contrastive Learning (e.g. Non-Generative Unsupervised Learning).

A.34 Extrapolation Results for Downstream Performance on Audio Tasks

In Figure 32, we show that BNSL accurately extrapolates the scaling behavior of the Downstream Performance on Audio Tasks.

In Figure 33, we show that BNSL accurately extrapolates the scaling behavior of Downstream Performance (e.g. on Language/Coding Tasks) to scales that are greater than 100,000 (i.e. greater than 5 orders of magnitude) times larger than the maximum (along the x-axis (e.g. amount of compute used for training)) of the points used for fitting.