Compute Trends Across Three Eras of Machine Learning

Introduction

Predicting progress in the field of Machine Learning (ML) is hard but of significant relevance for actors in industry, policy, and society. How much better will Computer Vision be in a decade? Will machines ever write better fiction than us? What jobs will we be able to automate?

Answering these questions is hard because they depend on many factors. However, one factor that influences all of them has been astonishingly regular over time—compute.

Various researchers have highlighted the relationship between AI capabilities and the scaling of ML models (Kaplan et al. (2020); Sutton (2019); Z. Li et al. (2020); Jones (2021); Rosenfeld et al. (2019); Hestness et al. (2017)). Therefore, compute can be seen as a quantifiable proxy for the progress of ML research.

This paper is a detailed investigation into the compute demand of milestone ML models over time. We make the following contributions:

We curate a dataset of 123 milestone Machine Learning systems, annotated with the compute it took to train them.

We tentatively frame the trends in compute in terms of three distinct eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. We offer estimates of the doubling times during each of these eras.

We extensively check our results in a series of appendices, discussing alternate interpretations of the data, and differences with previous work.

Our dataset, figures, and an interactive visualization are publicly available. If you use our dataset, please cite us as: Parameter, Compute and Data Trends in Machine Learning by Jaime Sevilla, Pablo Villalobos, Juan Felipe Cerón, Matthew Burtell, Lennart Heim, Amogh B. Nanjajjar, Anson Ho, Tamay Besiroglu and Marius Hobbhahn; 2021.

Related work

Amodei & Hernandez (2018) introduced two methods for estimating training compute in AI and Compute, and analyzed a trend based on 15 ML systems. They found that scaling in ML training compute followed a 3.4 month doubling time between 2012 and 2018.

In a later addendum, Sastry et al. (2019) supplemented their analysis with 10 papers from the pre-2012 era. They found a doubling rate in training compute of about 2 years between 1959 and 2012.

Lyzhov (2021) expanded upon Amodei & Hernandez (2018)’s dataset with seven subsequently released ML models and argued that growth stalled after the publication of AI and Compute (Amodei & Hernandez (2018)). In particular, the author found that the most compute-intensive model of 2020 (GPT-3) only required 1.5$\times$ more compute for training than the most compute-intensive model of 2017 (AlphaGo Zero).

In a similar effort, Sevilla et al. (2021) investigated trends in trainable parameter counts. They found an 18 to 24 month doubling time in all application domains from 2000 to 2021. For language models, they found that a discontinuity occurred between 2016 and 2018, where the doubling time for parameters sped up to 4 to 8 months.

Thompson et al. (2020) studied the increasing reliance of Deep Learning on computational power. They concluded that progress was becoming increasingly infeasible as compute requirements grew faster than progress in computing hardware.

In a recent report, Lohn & Musser (2022) investigated the limits of the compute trend by extrapolating the training costs into the future (based on a 3.4 month doubling time from Amodei & Hernandez (2018)) and exploring potential limitations. The authors concluded that the current rate of increase is unsustainable due to cost, hardware availability, and engineering difficulties, and that a slow-down may have already begun.

There have been other initiatives to collect data on important ML models. Akronomicon is a publicly available leaderboard of large-scale ML models (Akronomicon (2022)). Computer Progress (2022) has been collecting information on model performance and training compute of ML models on some common benchmarks. AI Tracker (2022) is collecting information on the capabilities of modern ML models. We are collaborating with each of these three projects and (with their permission) we incorporate part of their work in our dataset.

Furthermore, Desislavov et al. (2021) investigate inference compute in Computer Vision and Natural Language Processing systems. We use their results to inform some of our estimates.

Compared to prior work, our data collection is more comprehensive. Our dataset contains three times more ML models than previous ones and includes data up to 2022. We also offer novel interpretations of previous data, which we believe have important implications for understanding progress in ML.

Trends

We explain the data we curated in terms of three distinct eras and three distinct trends. In short, there was an era of slow growth before Deep Learning took off. Around 2010, the trend sped up and has not slowed down since then. Separately, in 2015 to 2016 a new trend of large-scale models emerged, growing at a similar rate, but exceeding the previous one by two orders of magnitude (OOMs hereafter). See Figure 1 and Table 2 for a summary.

First we will discuss the transition to Deep Learning circa 2010-2012. Then we will discuss the emergence of large-scale models circa 2015-2016.

We performed some alternative analyses to examine our conclusions from additional perspectives. In Appendix B we discuss trends in record-setting models. In Appendix C we discuss trends in different ML domains.

Consistent with the results from Amodei & Hernandez (2018), we find two very different trend regimes before and after the advent of Deep Learning. Before then, the amount of compute required to train ML systems doubled once every 17 to 29 months. Subsequently, the overall trend speeds up and doubles every 4 to 9 months.

The trend in the Pre Deep Learning Era roughly matches Moore’s law, according to which transistor density doubles roughly every two years (Moore (1965)) – often simplified to computational performance doubling every two years.

It is not clear when the Deep Learning Era starts We discuss the start of the Deep Learning Era in more detail in Appendix D. — there are no noticeable discontinuities in the transition from the Pre Deep Learning to the Deep Learning era. Moreover, our results barely change if we place the start of the Deep Learning era in 2010 or in 2012, see Table 3.

2 Trends in the Large-Scale era

Our data suggests that around 2015-2016 a new trend of large-scale models emerged, see Figure 3. This new trend began with AlphaGo in late 2015 and continues up to the present day. These large-scale models were trained by large corporations, whose larger training budgets presumably enabled them to break the previous trend.

Note that we made an intuitive decision in deciding which systems belong to this new large-scale trend. We justified it post hoc as the systems that exceed a certain $Z$-value threshold with respect to nearby models, see Appendix A for details on our method. See Appendix F for discussion on what makes large-scale models categorically different. There is room for alternative interpretations of the data.

Separately, the trend of regular-scale models continued unperturbed. This trend before and after 2016 is continuous and has the same slope, doubling every 5 to 6 months, see Table 4. Among other reasons, this reinforces our belief that the trend of large-scale models is a separate one.

The trend of increasing compute in large-scale models is apparently slower, doubling every 9 to 10 months. Since we have limited data on these models, the apparent slow-down might be the result of noise. In Appendix G we discuss some possible causes for this potential slowdown. In Appendix B we also show that the trend is equally fast before and after September 2015 if we look only at record-setting models.

Our results contrast with Amodei & Hernandez (2018), who find a much faster doubling period of $3.4$ months between 2012 and 2018, and with Lyzhov (2021), who finds a much longer doubling period of >2 years between 2018 and 2020. We make sense of these discrepancies by noting that their analyses have limited data samples and assume a single trend 5 , while ours studies large-scale and regular-scale models separately. Since the large-scale trend only recently emerged, previous analyses could not differentiate these two distinct trends. We discuss this in more depth in Appendix E.

Conclusion

In this article, we have studied trends in compute by curating a dataset of training compute with more than 100 milestone ML systems and used this data to analyze how the trend has grown over time.

Our findings seem consistent with previous work, though they indicate a more moderate scaling of training compute. In particular, we identify an 18-month doubling time between 1952 and 2010, a 6-month doubling time between 2010 and 2022, and a new trend of large-scale models between late 2015 and 2022, which started 2 to 3 orders of magnitude over the previous trend and displays a 10-month doubling time.

To summarize: in the Pre Deep Learning Era compute grew slowly. Around 2010, the trend accelerated as we transitioned into the Deep Learning Era. In late 2015, companies started releasing large-scale models that surpassed the trend, e.g. AlphaGo – marking the beginning of the Large-Scale Era. Framing the trends in terms of these three eras helps us explain the discontinuities we observed in the data, though we are not confident in the distinction between large-scale and regular-scale models.

We hope our work will help others better understand how much recent progress in ML has been driven by increases in scale and improve our forecasts for the development of advanced ML systems.

Moreover, the growing trend in training compute highlights the strategic importance of hardware infrastructure and engineers. Cutting-edge research in ML has become synonymous with access to large compute budgets or computing clusters, and expertise to leverage them.

One aspect we have not covered in this article is another key quantifiable resource used to train Machine Learning models — data. We will be looking at trends in dataset size and their relationship to trends in compute in future work.

Acknowledgments

We thank Alex Lyzhov, Girish Sastry, Danny Hernandez, Haydn Belfield, Jack Clark, Markus Anderljung, Alexis Carlier, Noemi Dreksler, Ben Garfinkel, Anton Korinek, Toby Shevlane, Stella Biderman, and Robert Trager. Jaime Sevilla is funded by the Open Philanthropy Project. Lennart Heim conducted part of this work during a summer fellowship in 2021 at the Stanford Existential Risks Initiative (SERI).

References

Appendix A Methods

All models in our dataset are mainly chosen from papers that meet a series of necessary criteria (has an explicit learning component, showcases experimental results, and advances the state-of-the-art) and at least one notability criterion ($>$1000 citations, historical importance, important SotA advance, or deployed in a notable context). For new models (from 2020 onward) it is harder to assess these criteria, so we fall back to a subjective selection. We refer to models meeting our selection criteria as milestone models.

We curated this collection of systems from various sources, including literature reviews, Papers With Code https://paperswithcode.com/, historical accounts, previous datasets, most cited publications of top conferences, and suggestions from individuals.

This dataset is biased in a number of important ways, and it is likely to contain mistakes. Beware of jumping to strong conclusions from our data. We discuss the limitations of our investigation in Appendix H.

When the training compute is not shared in the paper, we follow the techniques in AI and Compute’s appendix to estimate the training compute of our models (Amodei & Hernandez (2018)). These include estimating the total training compute from the forward pass compute, and from GPU time. A detailed description of our guidelines to estimate training compute is available online (Sevilla et al. (2022)). Our reasoning for each estimate is annotated in the respective cells of the main dataset.

ML systems are often trained multiple times to choose better hyperparameters (e.g. number of layers or training rate). However, this information is often not reported in papers. Our dataset only annotates the compute used for the final training run.

The regressions and doubling rates are derived from log-linear fits to the training compute. Where confidence intervals are indicated, those are derived from a bootstrap with $B=1000$ samples. To account for the uncertainty of our estimates, we randomly adjust each estimate by randomly multiplying it by a number between $\frac{1}{2}$ and 2. We use a factor of 2 for the range as this matches the range of empirical differences we found when using two different methods to estimate the training compute for a few papers (Sevilla et al. (2022)). The concrete distribution we sample the random adjustment from is log uniform between $\frac{1}{2}$ and 2. We use the notation [quantile 0.025; median; quantile 0.975] to indicate 95% confidence intervals.

Throughout the article, we have excluded low-compute outliers from the dataset. To do so, we compute the log training compute $Z$-score of each model with respect to other models whose publication date is within 1.5 years. We exclude models whose $Z$-score is 2 standard deviations below the mean. By default we only filter outliers with low compute, since we are actively interested in studying high compute models that are pushing the boundaries of ML. This criteria results in the exclusion of 5 models out of 123 between 1952 and 2022. The models excluded this way are often from relatively novel domains, such as poker, Hanabi, and hide and seek.

Later we used a similar methodology to automatically select papers with exceedingly high compute, choosing papers that exceed the $Z>0.76$ threshold after 2016. In both cases, we first decided by visual inspection which papers to mark as outliers and then chose the thresholds accordingly to automatically select them.

Appendix B Analyzing record-setting models

We describe models which set a record in compute demand—therefore outcompeting all previously released models—as record-setting models.

In general, we caution against regressing on record-setting compute budgets, as these trends are likely to be dominated by outliers. They are more representative of expensive efforts to push the SotA, rather than of the trend pushing training compute upwards.

However, it is still informative. We find that our conclusions from the post are still supported by the record-setting models. We see a slow-growing era from 1957 to 2010, and a fast-growing era from 2010 to 2015. Around September 2015, we observed a discontinuity.

The biggest difference between our main results and the trend in record-setting models is found in 2015-2022. If we include AlphaGo Zero and AlphaGo Master in the dataset, the trend is noticeably slower, with a one-year doubling time. However, if we exclude them our results agree with our main analysis: a trend with a doubling time similar to 2010-2015.

Appendix C Trends in different domains

Different domains rely on different architectures and we can naively expect them to be subject to different scaling laws. Therefore, we have decided to study different domains separately. In this section, we investigate trends in vision, language, games, and other Other incorporates domains with fewer than 10 systems each in our dataset. This includes drawing, speech, driving, robotics, recommender systems, multimodal systems, and other domains. domains.

The trends in vision and language seem fairly consistent over time and grow following the same doubling pattern as the overall dataset.

However, we could not find a consistent trend in games. This could be either because data is quite sparse in the games domain, because different games are essentially different domains subject to different scaling laws or because the field has not been systematically pushing forward in games to the same extent as in other domains. The games domain includes a mix of large-scale models by large corporations and more modest contributions.

Finally, the other domains seem to follow the same trend as the overall dataset when grouped together.

Appendix D When did the Deep Learning Era start?

The Deep Learning revolution is often noted to have started in 2012 with the creation of AlexNet (e.g., Alom et al. (2018)). Our guess is that this is up for reasonable debate, but we think that the year 2010 probably best fits the available evidence about when the era started. This section explains our reasoning.

AlexNet has some key features usually associated with Deep Learning: it is a large model, it was trained on GPUs, and outperformed traditional approaches. However, there are models before AlexNet that arguably have some or all of these features:

Model size/depth. Models including neural networks at least as large as AlexNet have existed since the early 2000s (notably, Viola & Jones (2001), Raina et al. (2009), and Mikolov et al. (2010)). In addition, neural networks roughly as deep as AlexNet (in terms of the number of hidden layers) have existed since 2010 (notably, D.C. Cireşan et al. (2010) and D. Cireşan, Meier & Schmidhuber (2012) both implemented up to 9 hidden layers vs. AlexNet’s 8).

GPU-based training. The insight of using GPUs to train ML models has been around for at least 7 years prior to AlexNet, and the utility for large-scale ML models is spelled out as early as 2005 (Steinkraus et al. (2005)). CNNs were trained on GPUs at least as early as 2006 (Chellapilla et al. (2006)), as were some other models that were large at the time (such as the 100M parameter Deep Belief Network by Raina et al. (2009) and the neural networks of D.C. Cireşan et al. (2011)). Other types of ML models, such as support-vector machines (SVMs) were previously also trained on GPUs (Catanzaro et al. (2008)).

Performance. AlexNet significantly outperformed prior techniques in ImageNet. However, drastic improvements over previous results are not rare in the field of ML, not even amongst large ML models that predate AlexNet. D.C. Cireşan et al. (2010); D.C. Cireşan et al. (2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task. Similarly, D. Cireşan, Meier & Schmidhuber (2012)’s deep CNNs (which again predates AlexNet) also beat all competitors who were using traditional techniques on a traffic sign recognition competition and improved on the state-of-the-art on several common image classification benchmarks.

In addition, there is evidence that somewhere between 2009 and early 2012 the field of speech recognition realized that Deep Learning would be capable of achieving major breakthroughs on standard tasks within the domain (and interestingly, this occurred before the September 2012 ImageNet competition that AlexNet won). In particular, Deng, Yu and Hinton’s 2009 workshop titled Deep Learning for Speech Recognition and Related Applications suggest that “deep architectures with efficient learning algorithms” would be needed to overcome challenges in the subfield (Deng et al. (2009)). There is evidence that between 2009 and early-2012 this became the dominant view in the subfield. For example, G. Hinton et al. (2012) presents the “shared view” of which at the time were the top 4 Speech Recognition labs. Their view was broadly that Deep Learning-based models would enable major advances in the field. This further supports the view that the switch to Deep Learning-based methods in the field of ML predates AlexNet and occurred somewhere between 2009 and 2012.

Taking this into account, we think that 2010 is the starting date most consistent with the evidence. This is because (a) the use of GPUs to train large ML models was already common at the time, (b) there were at least a few Deep Neural Networks that achieve highly competitive levels of performance (notably Mikolov et al. (2010); D.C. Cireşan et al. (2010); D.C. Cireşan et al. (2011)), and (c) this timeline is consistent with the adoption of Deep Learning within the field of Speech Recognition.

In this article, we have therefore opted to use 2010 as a default date for the start of the Deep Learning Era, though as noted in Section 3.1 our results do not change when we use the more common starting point of 2012.

Appendix E Comparison to Amodei & Hernandez (2018)’s analysis

Amodei & Hernandez (2018)’s analysis shows a 3.4 month doubling from 2012 to 2018. Our analysis suggests a 5.7 month doubling time from 2012 to 2022 (Table 3). In this section, we investigate this difference.

Our analysis differs in three points (number of samples, extended time period, and the identification of a distinct large-scale trend). Of these, either the time period or the separation of the large-scale models is enough to explain the difference between our results. To show this, we investigate the same period as in the Amodei & Hernandez (2018) dataset. The period starts with AlexNet in September 2012 and ends with AlphaZero in December 2018.

As discussed, our work suggests that between 2015 and 2017 a new trend emerged — the Large-Scale Era. We discuss two scenarios: (1) assuming our distinction into two trends and (2) assuming there is a single trend (similar to Amodei & Hernandez (2018)’s analysis).

We can interpret these results in two ways:

There is a single trend, which showed a 4 month doubling time between September 2012 and December 2017. Afterwards, the trend slowed down to a 5 month doubling time.

A new trend of large-scale models split off the main trend in late 2015. If we separate the large-scale models, we can see that the regular-scale trend had a similar doubling time before and after 2017. Amodei & Hernandez (2018)’s result is different from ours because they are mixing together the regular-scale and large-scale trends.

In the first interpretation, our result is different from Amodei & Hernandez (2018) because we are grouping together the pre-2017 and post-2017 trends into a single analysis.

In the second interpretation, our result is different because we are analyzing the trend in large-scale and regular-scale models differently.

We currently favor the second explanation. This is because (1) the large-scale trend story seems to better predict developments after 2017, while Lyzhov (2021) found that the single-trend story does not extend past 2017, and (2) we think that the models in the large-scale trend are explained by a drastic departure in funding.

Appendix F Are large-scale models a different category?

We hypothesized that some projects that use extraordinarily large amounts of compute are a different category of flagship models, e.g. AlphaGo/Zero or GPT-3. From 2016 onwards, companies were willing to spend significantly more compute—and therefore money—than previous trends would have predicted. AlphaGo Zero in 2017 (Silver, Schrittwieser et al. (2017)) is estimated to have cost $35M (H (2020)) and AlphaStar (Vinyals et al. (2019)) following in 2019 with an estimated cost of$12M (K. Wang (2020)). GPT-3 (T.B. Brown et al. (2020)), a recent SotA NLP model, has been estimated to have cost around $4.6M to train (C. Li (2020)). We do not know the exact spending of the relevant companies and these should be treated as rough estimates.

It is notable that AlphaGo Zero and AlphaStar have both gathered significant media attention For example the documentary on AlphaGo (Kohs et al. (2017)) or AlphaStar competing in public competitions (OpenAI (2019)). which might justify the extreme costs. On the other hand, GPT-3 is now monetized to potentially make up for its significant costs (OpenAI (2021)).

However, without inside knowledge, it is hard to evaluate whether these were just continuations of a trend or categorically different projects: Were the expected economic returns of some models significantly bigger? Was AlphaGo a unique project given this milestone? We are planning to investigate this in more detail in the future.

Another question: where should we draw the line for large-scale models? There is a reasonable case for including NASv3, Libratus, Megatron-LM, T5-3B, OpenAI Five, Turing NLG, iGPT-XL, GShard (dense), Switch, DALL-E, Pangu-$\alpha$, ProtT5-XXL and HyperClova on either side of the division. For example, Figure 8 depicts an alternate reasonable choice of Large-Scale models.

In Table 9 we show the effects of choosing different $Z$-value thresholds to separate the Large-Scale models. See Appendix A for a description of how the large-scale models are selected based on the $Z$-value threshold. The differences are small.

Appendix G Causes of the possible slowdown of training budget in large-scale models between 2016 and 2022

As discussed in Section 3.2, the trend of increasing compute in large-scale models between 2016 and 2022 is slower (10 month doubling time) than the overall trend (6 month doubling time).

Some hypotheses that might explain the slowdown include:

The 2020-22 global chip shortage. This occurred as a result of strong demand for computer and electronic equipment to enable working from home (Attinasi et al. (2021); S. Wu et al. (2021)), supply shocks caused by severe weather disruptions See for instance Patel (2021) and Barrett (2021). and trade frictions caused by the ongoing trade war between the US and China. The shortage has led to blockages in automotive manufacturing (Ajmera & Ramakrishnan (2021)). GPU prices have been higher than usual. For example, a survey of German firms revealed that in Germany and Austria, GPUs are selling for up to 3$\times$ the manufacturer’s suggested retail price (MSRP) in 2021 (3D Center (2022)). It has been reported that NVIDIA has been struggling to provide some of its latest and top-performing chips, such as the A100 (Shilov (2020)). There is also anecdotal evidence (Woodie (2021)) that the chip shortage is affecting AI training runs. Woodie (2021) quotes an analyst saying: “A lot of GPU users are complaining that it’s hard for them to get the GPU time. […] They put a job in a queue and it takes a while for it to ramp. Previously they would just say there are [some] GPUs and they were just sitting there. Now they don’t always have GPUs available, so it takes a while for them to get in the queue and get their jobs running.”

Challenges with building the required High-Performance Computing (HPC) infrastructure. The hardware constraints involved in massive training runs (including memory limitations and communication bandwidths) force users to segment massive models into groups of layers, which are then trained in parallel (Hazelwood et al. (2018); Huang et al. (2019); Athlur et al. (2021)). Designing and implementing algorithms that do this efficiently can be extremely hard, and often requires dedicated engineering teams. For an account of some of these challenges, see Huang et al. (2019) and Athlur et al. (2021), A further challenge with massive training runs, given how neural networks are trained, is that mistakes can often not be corrected after the training, which means that getting the infrastructure right the first time is very important. Relatedly, T.B. Brown et al. (2020) document a mistake in the training run which they found only after training and were therefore unable to correct: “We initially tried to address the issue of contamination by proactively searching for and attempting to remove any overlap between our training data and the development and test sets of all benchmarks studied in this paper. Unfortunately, a bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t feasible to retrain the model.” We suspect that the cultivating of relevant expertise and the designing, testing, and deploying of HPC infrastructure for training massive Deep Learning models has created challenges unique to the Large-Scale Era.

Budget caps. The monetary costs of training the most compute-intensive ML models can be relatively large. For example, Sharir et al. (2020) estimates that Google’s T5 project—which is by no means the biggest training run to date—might have cost a total of $10M in cloud computing costs. Maintaining a constant growth rate in the budgets dedicated to training runs might, therefore, be challenging at the massive-scale. For example, OpenAI received$500M worth of cloud computing credits from Microsoft (Orme (2022)). Assuming that this is their entire budget for computing resources, then this would set a hard cap on the scale of training runs they could run. If they intend this budget to fund many different experiments, the total available budgets might be of the same order of magnitude as the largest training runs to date.

Undisclosed large models. Most compute intense models stem from corporate AI labs which might not publish their results publicly.

Appendix H Limitations

Over the course of this project, we have identified several sources of uncertainty and potential weaknesses with the analysis. In this appendix we discuss these and how we have accounted for them, or why we believe that they do not pose a major problem to our conclusions.

Uncertainties in compute calculations How much would the compute values change given the uncertainties of the inputs (e.g. utilization rate, FLOP/s)? We expect most of the compute estimates to be accurate within a factor of about two based on some comparisons we did between different estimation methods (Sevilla et al. (2022)). To account for this we introduce some noise when bootstrapping – see Appendix A for more details.

Non-sampling errors What if there are many incorrect calculations? Ideally, our calculations should be easily verifiable; they are included as annotations for the cell in which the compute estimate is contained. In practice, we have seen few corrections suggested since making our dataset public, and we expect this to be a significant source of error.

Small sample size What would be the consequences of a larger sample size (e.g., $n=1000$)? We expect our results to be different in a few subtle ways if we had a larger dataset. If we increased the number of models in our dataset, we would sample in greater proportion from less-cited papers than currently – which tend to involve lower-cost experiments with smaller compute budgets. If we uniformly increase the number of less-cited papers in our dataset across each era, this should affect just the intercept of our trend-lines, without affecting the slope, thereby leaving the doubling period unchanged.

However, it is easier to find recent less-cited papers than those from many decades ago (as we expect the latter to be less consistently archived). If this was the case, we expect that if we increased the number of models in our dataset, we would increase the number of lower-budget experiments in the recent past without a commensurate increase in the number of lower-budget experiments from the more distant past. This would cause the estimated doubling time in the Pre Deep Learning Era to be slightly longer.

More recently, the largest-scale models (often more highly cited) seem to have a longer doubling-time than all other models. If we were to increase the size of our dataset (which would involve sampling relatively more from smaller experiments), this would reduce the intercept and shorten the average doubling time over more recent eras.

Selection bias What happens if we modify the (fairly subjective) notability criteria? Overall, we are biased towards AI systems that are:

Found in academic publications: Less data is available about closed-source commercial systems. Additionally, some papers omit important information for determining training compute, such as the total training time.

Written in English: This should not be too large of a problem, since the vast majority of published scientific research is in English, and this is almost certainly the case for notable ML models.

Models that are subjectively ‘notable’: These are more likely to be models that are large and recent. The inclusion of a higher proportion of these models makes our estimate of doubling times higher.