PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
Emergent Capabilities and the Competence Threshold: Phase Transitions in Language Model Behavior
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
The scaling behavior of language models is neither smooth nor uniform across task types. While aggregate loss metrics decrease predictably with scale, downstream task performance exhibits discontinuous jumps, inversions, and threshold effects that aggregate measures obscure. We propose the concept of a competence threshold: a scale boundary for a given task class below which architectural scaffolding (chain-of-thought prompting, retrieval augmentation, tool use, multi-agent decomposition) cannot compensate for fundamental model limitations, and above which scaffolding becomes the dominant capability amplifier. We formalize this notion, review empirical evidence from the emergent abilities debate, inverse scaling phenomena, and scaffolding interaction studies, and argue that identifying competence thresholds for specific task distributions enables principled model selection and system design. We present a measurement protocol for empirically locating these thresholds and discuss implications for resource allocation in production systems. The central claim is that system builders should select models based on competence threshold proximity for their target tasks, not by aggregate benchmark ranking.
Language models, scaling laws, emergent capabilities, competence thresholds, phase transitions, scaffolding, chain-of-thought, model selection, system design.
The dominant paradigm for evaluating language model progress relies on aggregate benchmarks: MMLU [1], BIG-Bench [2], and similar multi-task suites that collapse diverse capabilities into a single leaderboard ranking. While these benchmarks serve a useful cataloguing function, they mask a structural property of language model scaling that has significant practical consequences: for specific task classes, model capabilities do not improve smoothly with scale. Instead, there exist discrete scale boundaries where qualitative shifts in behavior occur.
This observation is not new. Wei et al. [3] documented what they termed emergent abilities, capabilities absent in smaller models that appear at larger scales. Schaeffer et al. [4] challenged this framing, arguing that apparent emergence is an artifact of metric choice rather than a genuine discontinuity in underlying model behavior. The subsequent debate has been productive but has not fully resolved the question of what the observed scaling patterns mean for practitioners who must select and deploy models for specific applications.
We propose a reframing that is useful for system builders regardless of how one resolves the emergence debate. We define the competence threshold for a task class T as the minimum model scale at which architectural scaffolding can reliably amplify performance on T. Below this threshold, no amount of scaffolding design compensates for the model's inability to perform the underlying computation. Above it, scaffolding becomes the primary lever for improving task performance, often yielding gains that dwarf what additional parameter scaling alone would provide.
This framing has direct practical implications. If a system builder knows that their target task distribution has a competence threshold at approximately 13B parameters, they can select the smallest model above that threshold and invest the remaining compute budget in scaffolding, rather than purchasing the largest available model. Conversely, if no model in their budget range exceeds the threshold, they should invest in data curation, fine-tuning, or task reformulation rather than scaffolding.
The paper proceeds as follows. Section II reviews the background on scaling laws and the emergent abilities debate. Section III provides a formal definition of the competence threshold. Section IV reviews empirical evidence for threshold behavior. Section V analyzes the interaction between model scale and scaffolding. Sections VI and VII discuss implications for model selection and system design, respectively. Section VIII presents a measurement protocol. Section IX surveys related work. Section X discusses limitations.
Kaplan et al. [5] established that the cross-entropy test loss of Transformer language models follows a power-law relationship with model size N, dataset size D, and training compute C. Specifically, the loss L scales as:
L(N) ~ N^(-alpha_N) L(D) ~ D^(-alpha_D) L(C) ~ C^(-alpha_C)
where alpha_N, alpha_D, and alpha_C are empirically fitted exponents. These scaling laws hold across several orders of magnitude and have proven remarkably predictive for loss, which measures average next-token prediction quality.
Hoffmann et al. [6] refined these findings with the Chinchilla analysis, demonstrating that many previously trained models were compute-suboptimal: they were too large for their training data. The Chinchilla scaling laws suggest that model size and training tokens should scale roughly equally, a result that has influenced subsequent training decisions across the field.
Muennighoff et al. [7] extended scaling law analysis to data-constrained regimes, showing that repeated data can partially substitute for novel data up to a point, but with rapidly diminishing returns after approximately four epochs. This work is relevant to our analysis because it establishes that scaling laws have boundary conditions: they predict behavior within a regime but do not generalize cleanly across regime boundaries.
The critical limitation of scaling laws for our purposes is that they predict loss, not capability. Loss is an average over the token distribution. A model can have excellent average loss while failing catastrophically on specific task classes. The relationship between loss and downstream task performance is task-dependent, non-linear, and in some cases non-monotonic.
Wei et al. [3] introduced the concept of emergent abilities in large language models, defining an ability as emergent if it is not present in smaller models but appears at larger scales. Their analysis of BIG-Bench [2] and other evaluation suites identified numerous tasks where performance jumped from near-random to well-above-chance over relatively narrow scale ranges. Examples included multi-step arithmetic, word unscrambling, and certain logical reasoning tasks. The claimed emergence was visible when using exact-match or multiple-choice accuracy metrics.
Schaeffer et al. [4] challenged this interpretation directly. Their central argument was that emergent abilities are a mirage produced by the researcher's choice of metric. When using metrics that are more continuous (such as token-level log-likelihood rather than exact-match accuracy), the apparent phase transitions disappear and performance improves smoothly with scale. They demonstrated this in three ways: by showing that metric choice explains the patterns on InstructGPT/GPT-3 tasks, by meta-analyzing BIG-Bench results, and by constructing seemingly emergent abilities in vision tasks through metric manipulation.
The resolution of this debate has significant consequences for how we understand model scaling. If Schaeffer et al. are entirely correct, then there are no true capability discontinuities and the appearance of emergence is a measurement artifact. If Wei et al. are correct, there exist genuine phase transitions in model behavior. Our position is that both camps capture part of the truth, and that the competence threshold framework accommodates both observations.
The assumption that larger models are uniformly better on all tasks has been challenged by several lines of evidence. McKenzie et al. [8] ran the Inverse Scaling Prize, identifying 11 tasks where model performance degraded with scale. Examples included tasks exploiting training data biases, tasks requiring resistance to majority voting when the majority is wrong, and tasks requiring negation understanding.
Wei et al. [9] subsequently showed that six of the eleven inverse-scaling tasks exhibited U-shaped scaling: performance first decreased with scale, then recovered at the largest model sizes. This suggests that inverse scaling may be a transient phenomenon that larger models eventually overcome, but the existence of the trough demonstrates that scaling trajectories are task-specific and can be non-monotonic.
Wu et al. [10] analyzed these patterns at a finer granularity, grouping questions by difficulty level and observing that hard questions exhibit U-shaped scaling while easy questions show steady improvement. Their Slice-and-Sandwich pipeline attempts to predict emergence thresholds from smaller-scale behavior, an approach conceptually related to our competence threshold framework, though we focus on scaffolding interaction rather than raw performance prediction.
These findings collectively establish that aggregate scaling narratives are insufficient. The scaling trajectory of any specific capability depends on the task structure, the metric, and the interaction between the model's internal representations and the task's computational demands.
We now provide a formal definition of the competence threshold and related concepts.
Definition 1 (Task Class). A task class T is a set of tasks sharing a common computational structure, such that a model capable of performing the core computation of T can, with appropriate prompting or scaffolding, solve individual instances of T.
Definition 2 (Scaffolding). A scaffolding S is an architectural wrapper around a language model M that structures the model's inputs and outputs to improve task performance. Examples include chain-of-thought prompting [11], retrieval augmentation [12], tool use [13], self-consistency decoding, and multi-agent decomposition.
Definition 3 (Scaffolding Amplification Factor). For a model M, a scaffolding S, and a task class T, the scaffolding amplification factor is:
A(M, S, T) = Perf(M with S, T) / Perf(M alone, T)
where Perf denotes task performance measured on an appropriate metric.
Definition 4 (Competence Threshold). A model M crosses the competence threshold for task class T when there exists a scaffolding S such that:
A(M, S, T) > 1 + epsilon
for some specified epsilon > 0. That is, M crosses the competence threshold for T when scaffolding can reliably amplify its performance on T by more than a trivial margin.
Below the competence threshold, we have:
For all scaffolding S: A(M, S, T) <= 1 + epsilon
This means that no scaffolding design can extract meaningful performance gains on T from M. The model lacks the underlying computational capacity that the scaffolding would amplify.
Above the competence threshold, there exists at least one scaffolding S that produces substantial gains, and the system builder's optimization problem shifts from model selection to scaffolding design.
Proposition 1 (Monotonicity). If model M' is a strict scaling superset of M (same architecture, more parameters, same or more training data), and M is above the competence threshold for T, then M' is also above the competence threshold for T. This holds because any computation M can perform, M' can also perform.
Proposition 2 (Non-Universality). The competence threshold for task class T1 may differ from the competence threshold for task class T2. There is no single scale point at which a model crosses all competence thresholds simultaneously.
Proposition 3 (Scaffolding Dependence). The competence threshold may depend on the class of scaffolding considered. A model might be below the threshold for chain-of-thought scaffolding but above it for retrieval-augmented scaffolding on the same task class, because the two scaffolding types demand different capabilities from the base model.
The most well-documented threshold effect involves chain-of-thought (CoT) prompting. Wei et al. [11] showed that CoT prompting improves performance on arithmetic, commonsense, and symbolic reasoning tasks, but only for models above approximately 100B parameters. Below this scale, CoT provides no benefit and can degrade performance. This is a clear instance of a competence threshold: smaller models lack the capacity to benefit from the scaffolding of decomposed reasoning steps.
This pattern has been replicated across multiple model families. The threshold is not identical for all reasoning tasks: simple two-step arithmetic has a lower threshold than multi-step word problems, which have a lower threshold than complex logical puzzles. The competence threshold is thus not a property of the model alone but of the (model, task class) pair.
Schick et al. [13] demonstrated that language models can learn to use external tools (calculators, search engines, calendars, translation systems) in a self-supervised manner. Their Toolformer model showed that tool use capability emerges at sufficient scale. Below the threshold, models cannot reliably decide when to call a tool, what arguments to pass, or how to integrate tool outputs into their generation. Above it, tool use scaffolding provides substantial gains, often making a smaller model competitive with a much larger model that lacks tool access.
This is a particularly clean example of the competence threshold because the scaffolding (API calls) is well-defined and the failure mode below the threshold is clear: the model cannot perform the meta-reasoning required to decide that an external computation would be helpful, formulate the computation correctly, and interpret the result.
Lewis et al. [12] introduced retrieval-augmented generation (RAG), combining parametric memory with non-parametric retrieval from a document index. RAG provides clear benefits for knowledge-intensive tasks, but the magnitude of the benefit depends on the base model's capacity to integrate retrieved context. Smaller models may be overwhelmed by retrieved passages, failing to distinguish relevant from irrelevant information or to synthesize multiple retrieved documents. Larger models show substantially greater gains from retrieval, suggesting a competence threshold for the integration task.
Wang et al. [14] demonstrated that GPT-4, when embedded in their Voyager architecture with automatic curriculum generation and a skill library, could perform open-ended exploration and skill acquisition in Minecraft. Critically, this agent architecture was tested across model scales and showed dramatic differences: GPT-4 enabled sophisticated agent behavior, while smaller models produced incoherent exploration. The gap between GPT-3.5 and GPT-4 in this setting was not a smooth improvement but a qualitative shift in the type of behavior the agent could sustain.
Xi et al. [15] survey the broader landscape of LLM-based agents and note that planning, tool use, and self-reflection capabilities all exhibit scale-dependent availability. Agent scaffolding architectures that work well with frontier models fail or produce unreliable behavior with smaller models, consistent with our competence threshold framework.
Srivastava et al. [2] evaluated models across 204 tasks in BIG-Bench and found that tasks exhibiting "breakthrough" behavior at a critical scale often involved multiple reasoning steps or brittle metrics, while tasks showing gradual improvement typically involved large knowledge or memorization components. This distinction maps onto our framework: knowledge tasks have low competence thresholds (scaffolding like retrieval augmentation can compensate early), while multi-step reasoning tasks have higher thresholds (the model must internalize the decomposition capacity before scaffolding can help).
Bubeck et al. [16] documented a related observation in their analysis of GPT-4, noting qualitative capabilities that were absent in prior model generations. Their analysis, while focused on a single model, provides detailed case studies of tasks where the transition from "cannot do" to "can do with scaffolding" is sharp.
The competence threshold framework reveals a structure in how model scale and scaffolding interact. We identify three regimes.
Regime 1: Sub-Threshold. In this regime, the model has not crossed the competence threshold for the target task class. No scaffolding design produces meaningful gains. The model either lacks the internal representations to perform the computation, or it cannot reliably execute the meta-cognitive operations that scaffolding requires (self-decomposition, self-evaluation, tool selection). In this regime, the only productive investments are larger models, more training data, or task-specific fine-tuning.
Regime 2: Threshold Crossing. The model has just crossed the competence threshold. Scaffolding now produces gains, but the gains are fragile: they depend on careful prompt engineering, the specific scaffolding design, and the task distribution. In this regime, system builders must invest significant effort in scaffolding optimization, and the gains are sensitive to distribution shift.
Regime 3: Post-Threshold Saturation. The model is well above the competence threshold. Scaffolding produces large, robust gains, and the marginal return on additional model scale is low relative to the marginal return on scaffolding investment. In this regime, a well-scaffolded smaller model can outperform a poorly scaffolded larger model.
The practical implication is that the optimal allocation of resources between model scale and scaffolding design depends on which regime the system occupies. This is a function of both the model and the task class, and it can change as models improve or as task requirements shift.
We can state this as a design principle:
Principle 1 (Architecture-Over-Scale). When a model is in Regime 3 (well above the competence threshold for the target task class), additional compute should be allocated to scaffolding design rather than model scaling. When a model is in Regime 1 (below the threshold), compute should be allocated to model scaling or fine-tuning, not scaffolding.
This principle has direct cost implications. A 7B parameter model with sophisticated scaffolding can outperform a 70B parameter model on tasks where the 7B model is above the competence threshold. The cost difference between serving these two models is substantial (roughly 10x in inference compute), making threshold identification economically valuable.
Current model selection practice relies heavily on aggregate benchmarks. A system builder choosing between Llama 2 7B, 13B, and 70B [17] will typically consult MMLU [1] scores, HumanEval pass rates, and similar aggregate metrics. This approach has two failure modes.
First, aggregate metrics can recommend a larger model when a smaller model with appropriate scaffolding would perform equally well on the target tasks. If the 13B model has crossed the competence threshold for the target task class, the additional 57B parameters of the 70B model may produce only marginal gains that do not justify the 5x cost increase.
Second, aggregate metrics can recommend a smaller model that has not crossed the competence threshold for the target tasks, leading to system failure that no amount of scaffolding can fix. A model with excellent MMLU scores may still be below the competence threshold for a specific multi-step planning task that the system builder needs.
We propose that model selection should proceed through the following steps:
1. Define the target task distribution precisely, decomposed into task classes. 2. For each task class, estimate the competence threshold by testing scaffolding amplification across available model scales. 3. Select the smallest model that is above the competence threshold for all required task classes. 4. Allocate remaining compute budget to scaffolding optimization.
This procedure requires more upfront evaluation effort than benchmark consultation, but it produces better resource allocation for production systems.
The competence threshold framework has implications for how production systems should be architected.
When a system must handle a diverse task distribution, different tasks may have different competence thresholds. A cost-effective architecture routes tasks to the smallest model that exceeds the competence threshold for that task class. Easy tasks (low threshold) go to small models; hard tasks (high threshold) go to larger models. This is a generalization of existing cascading approaches, but with a principled criterion for the cascade boundaries.
Organizations often invest in a single scaffolding architecture and apply it uniformly. The competence threshold framework suggests that scaffolding investment should be targeted: invest in scaffolding for task classes where the model is well above the threshold (Regime 3), and invest in model scaling or fine-tuning for task classes where the model is below or near the threshold (Regimes 1 and 2).
When a new model is released, the competence threshold framework provides a principled way to evaluate whether upgrading is warranted. If the current model is already above the competence threshold for all target task classes, upgrading to a larger model will produce diminishing returns relative to scaffolding improvements. If the current model is below the threshold for critical task classes, upgrading should be prioritized.
Recent work on multi-agent systems [15] has shown that decomposing complex tasks across multiple specialized agents can yield substantial gains. However, each agent in such a system must cross the competence threshold for its assigned subtask. If any agent in the pipeline is below its threshold, the entire pipeline fails. This means that multi-agent architectures amplify the importance of threshold identification: every link in the chain must be above threshold.
We propose a practical protocol for identifying competence thresholds for specific task distributions. The protocol requires a set of models spanning a range of scales (ideally from the same family to control for architectural and training differences) and a representative sample of tasks from the target distribution.
Step 1: Task Decomposition. Partition the target task distribution into task classes T1, T2, ..., Tn based on the computational structure required. Group tasks that require the same type of reasoning or capability. This step may require domain expertise and empirical testing.
Step 2: Baseline Measurement. For each model Mi and task class Tj, measure baseline performance Perf(Mi, Tj) without scaffolding. Fit a scaling curve to identify whether performance is improving, plateaued, or degrading with scale.
Step 3: Scaffolding Measurement. For each model Mi and task class Tj, measure performance with a representative set of scaffolding approaches {S1, S2, ..., Sk}. Compute the amplification factor A(Mi, Sj, Tj) for each combination.
Step 4: Threshold Identification. For each task class Tj, identify the smallest model Mi such that max_j A(Mi, Sj, Tj) > 1 + epsilon. This model's scale is the estimated competence threshold for Tj. The choice of epsilon should reflect the minimum practically meaningful improvement; we suggest epsilon = 0.1 (10% relative improvement) as a starting point.
Step 5: Validation. Validate the identified thresholds by testing on held-out tasks from the same distribution. Thresholds should be robust across task instances within the class.
Step 6: Scaffolding Sensitivity. For models above the threshold, measure the variance of amplification factors across scaffolding designs and task instances. High variance indicates Regime 2 (fragile threshold crossing); low variance indicates Regime 3 (stable post-threshold behavior).
The entire protocol requires evaluating O(|models| x |task classes| x |scaffolding types|) configurations, which is feasible for typical production settings with 3-5 candidate models, 5-10 task classes, and 2-4 scaffolding architectures. The evaluation cost is dominated by the model inference required, which can be managed by using smaller task samples for initial screening and full samples for validation.
This protocol is related to the Slice-and-Sandwich approach of Wu et al. [10], which attempts to predict emergence thresholds from fine-grained difficulty analysis. Our approach differs in that it directly measures scaffolding amplification rather than trying to predict thresholds from raw scaling curves, which we believe is more reliable for practical decision-making.
Our work builds on several research threads that we synthesize under the competence threshold framework.
The scaling laws literature [5, 6, 7] provides the foundation for understanding how model properties change with scale. We extend this from loss prediction to capability prediction by incorporating scaffolding interaction.
The emergent abilities debate [3, 4, 9, 10] surfaces the empirical phenomena we seek to explain. We take no strong position on whether the underlying capability transitions are continuous or discrete, as the competence threshold framework is useful in either case. What matters for system builders is whether scaffolding can amplify performance, not the philosophical status of the transition.
The chain-of-thought literature [11] provides the canonical example of a scaffolding that only works above a threshold scale, motivating our framework. Brown et al. [18] showed that few-shot in-context learning itself exhibits scale-dependent behavior, with larger models showing qualitatively different adaptation capabilities.
The tool use and agent literature [13, 14, 15] extends scaffolding beyond prompting to include external computation and environment interaction. These approaches have the most dramatic threshold effects because they require meta-cognitive capabilities (knowing when to use a tool, formulating API calls) that smaller models lack.
Rafailov et al. [19] showed that alignment techniques like DPO can shift a model's effective capability on preference-sensitive tasks. This is relevant because alignment fine-tuning can move a model across a competence threshold for certain task classes, effectively changing the threshold landscape without changing the base model's scale.
The agent-based systems literature [14, 15] demonstrates that complex scaffolding architectures can dramatically amplify model capabilities, but only when the base model is sufficiently capable. This is precisely the competence threshold pattern: the scaffolding is the dominant capability lever, but only above the threshold.
We identify several limitations of our framework.
First, the competence threshold is defined relative to a scaffolding class, and different scaffolding approaches may yield different thresholds for the same task class. Our framework does not provide a task-intrinsic threshold independent of scaffolding; this is by design, as we argue the scaffolding interaction is what makes the threshold useful for practitioners, but it means that threshold estimates are scaffolding-dependent.
Second, the protocol we propose requires access to multiple model scales. Practitioners with access to only one model (e.g., a single API provider) cannot directly measure thresholds. In such cases, published threshold measurements from similar task distributions can serve as rough guides, but transfer across task distributions is unvalidated.
Third, our framework assumes that task classes can be cleanly decomposed. In practice, production tasks often mix multiple computational requirements (reasoning, knowledge retrieval, format compliance), and the relevant competence threshold may be determined by the hardest subtask. We do not provide a formal method for handling mixed-composition tasks.
Fourth, we have not validated the framework with a large-scale controlled experiment. The evidence we cite is drawn from published results across different model families, training procedures, and evaluation protocols. A definitive test would require evaluating scaffolding amplification across a controlled scale series (e.g., the Llama 2 family at 7B, 13B, and 70B) on a curated task distribution with standardized scaffolding. We plan such an evaluation in future work.
Fifth, the competence threshold may shift with improvements in training data, alignment techniques, and architecture design. A model that is below the threshold today may cross it with better instruction tuning or preference optimization, even at the same parameter count [19]. The threshold is thus a property of the (model, training, task) triple, not the model alone.
Finally, our framework focuses on capability thresholds and does not address safety thresholds: scale boundaries where models develop problematic behaviors (sycophancy, hallucination patterns, jailbreakability) that may also exhibit threshold dynamics. Extending the framework to safety-relevant task classes is an important direction for future work.
We have proposed the competence threshold as a framework for understanding when and how architectural scaffolding amplifies language model capabilities. The central contributions are: (1) a formal definition of the competence threshold as the minimum model scale at which scaffolding can reliably improve performance on a task class; (2) an identification of three operational regimes (sub-threshold, threshold crossing, and post-threshold saturation) with distinct resource allocation implications; (3) a measurement protocol for empirically locating thresholds for specific task distributions; and (4) design principles for model selection and system architecture that follow from the framework.
The practical value of this framework is in replacing the current heuristic of "bigger is better, pick the model with the highest benchmark score" with a principled analysis of which model is large enough for the target tasks and where the remaining compute budget should be allocated. In an era where frontier model training costs hundreds of millions of dollars and inference costs dominate production budgets, the difference between a well-chosen threshold and an over-provisioned model is substantial.
We do not claim that the competence threshold framework resolves the emergent abilities debate. What we claim is that regardless of whether capability transitions are mathematically continuous or discontinuous, the practical question for system builders is binary: does scaffolding work for this model on this task, or does it not? The competence threshold captures this binary structure and provides a principled basis for the engineering decisions that follow.
REFERENCES
[1] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, "Measuring massive multitask language understanding," in Proc. Int. Conf. Learn. Representations (ICLR), 2021. arXiv:2009.03300.
[2] A. Srivastava et al., "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models," Trans. Mach. Learn. Res., 2022. arXiv:2206.04615.
[3] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, "Emergent abilities of large language models," Trans. Mach. Learn. Res., 2022. arXiv:2206.07682.
[4] R. Schaeffer, B. Miranda, and S. Koyejo, "Are emergent abilities of large language models a mirage?" in Proc. Neural Inf. Process. Syst. (NeurIPS), 2023. arXiv:2304.15004.
[5] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv:2001.08361, 2020.
[6] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, "Training compute-optimal large language models," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2022. arXiv:2203.15556.
[7] N. Muennighoff, A. M. Rush, B. Barak, T. Le Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and C. Raffel, "Scaling data-constrained language models," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2023. arXiv:2305.16264.
[8] I. R. McKenzie, A. Lyzhov, M. Pieler, A. Parrish, A. Mueller, A. Prabhu, E. McLean, A. Kirtland, A. Ross, A. Gritsevskiy, D. Cuenca, D. Kiela, D. Pyatkin, D. Sadeh, E. Sclar, F. Ladhak, F. Filandrianos, G. Thomas, H. S. Shandilya, I. Gemp, J. Chen, J. Hu, J. Qi, K. Tirumala, K. Reinecke, L. Elsen, M. Muennighoff, N. Garg, P. Nakkiran, R. Thakker, S. R. Bowman, S. Koyejo, and Y. Goldberg, "Inverse scaling: When bigger isn't better," Trans. Mach. Learn. Res., 2023. arXiv:2306.09479.
[9] J. Wei, Y. Tay, and Q. V. Le, "Inverse scaling can become U-shaped," arXiv preprint arXiv:2211.02011, 2022.
[10] T. Y. Wu, M. Qian, C. Guestrin, and J. Lu, "U-shaped and inverted-U scaling behind emergent abilities of large language models," arXiv preprint arXiv:2410.01692, 2024.
[11] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2022. arXiv:2201.11903.
[12] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, S. Riedel, and D. Kiela, "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2020. arXiv:2005.11401.
[13] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language models can teach themselves to use tools," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2023. arXiv:2302.04761.
[14] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, "Voyager: An open-ended embodied agent with large language models," arXiv preprint arXiv:2305.16291, 2023.
[15] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui, "The rise and potential of large language model based agents: A survey," arXiv preprint arXiv:2309.07864, 2023.
[16] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, "Sparks of artificial general intelligence: Early experiments with GPT-4," arXiv preprint arXiv:2303.12712, 2023.
[17] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023.
[18] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2020. arXiv:2005.14165.
[19] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, "Direct preference optimization: Your language model is secretly a reward model," in Proc. Neural Inf. Process. Syst. (NeurIPS), 2023. arXiv:2305.18290.