“emergent abilities appear due the researcher's choice of metric rather than due to fundamental changes in model behavior with scale”

This is one of the most important methodological critiques in NLP in the last five years. If 'emergence' is an artifact of discontinuous metrics, then the claim that scaling produces qualitatively new capabilities is a measurement illusion — models are just getting uniformly better, and we're quantizing that improvement into apparent phase transitions. The implication for safety research is uncomfortable: discontinuous capability jumps may be much rarer than believed.

“nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes”

The metric dependency here is empirical but the explanation is underspecified. Why do researchers keep choosing nonlinear metrics? Partly habit (accuracy is intuitive), partly task structure (some tasks really do have pass/fail thresholds), partly publication incentive (discontinuous results are more newsworthy). The paper diagnoses the problem but doesn't address the harder question: are there tasks where emergence is real even under continuous metrics?

“sharp and unpredictable changes might be induced by the researcher's choice of measurement, even though the model family's per-token error rate changes smoothly”

The gap between per-token error rate (what the model actually optimizes) and task-level accuracy (what we measure) is a known source of measurement confounds. But this paper quantifies how large that gap can be — the same model family can appear to have emergent abilities or not, depending purely on whether you measure token-level or task-level performance. This undermines a decade of 'capabilities evaluations' that didn't control for metric choice.

“metric choice can be used to induce emergent abilities in a novel domain (vision) in diverse architectures and tasks”

The generalization to vision models is the paper's strongest evidence. If metric-induced emergence only appeared in language, one could argue it's a language-specific phenomenon. That it replicates in vision, with different architectures and tasks, strongly suggests a general measurement pathology — not something specific to transformers or NLP benchmarks. The implication is that any capability claim grounded in threshold metrics should be treated with suspicion.