PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
DISTILLATION AS ARCHITECTURE TRANSFER: WHAT SMALL MODELS LEARN AND WHAT THEY CANNOT
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
Knowledge distillation has become the dominant paradigm for compressing large language models into smaller, deployable systems. We examine what distillation actually transfers from teacher to student and, with equal attention, what it does not. We argue that distillation operates primarily as a transfer of surface behavior: output distributions, reasoning chain text, format compliance, and task-specific heuristics. What remains untransferred is the latent computation that produced those behaviors in the teacher, including the internal search process underlying chain-of-thought reasoning, the capacity for novel compositional generalization, and the depth of representations encoded in intermediate layers. We formalize this as the distillation ceiling, a bound determined by the student's architectural capacity and the representational gap between teacher and student. We propose that for tasks requiring genuine reasoning depth rather than pattern matching, scaffolding around a smaller model may be more effective than distilling deeper capability into it. We present a measurement protocol for distinguishing learned behavior from mimicked behavior in distilled models and discuss implications for practical deployment decisions.
Knowledge distillation, language model compression, chain-of-thought reasoning, model architecture, scaffolding, small language models, reasoning transfer.
The deployment of large language models (LLMs) in production systems faces a persistent tension between capability and cost. Models with hundreds of billions of parameters demonstrate impressive reasoning, instruction following, and generation quality, but their computational demands make them expensive to serve at scale. Knowledge distillation, the process of training a smaller student model to reproduce the behavior of a larger teacher, has emerged as the primary strategy for resolving this tension [1].
The appeal of distillation is straightforward: if a 70-billion-parameter model can solve a task, perhaps a 7-billion-parameter model can learn to solve it too, at a fraction of the inference cost. Recent work has produced distilled models that match or approach teacher performance on standard benchmarks [5][6][16]. These results have led to widespread adoption of distillation pipelines in both research and industry.
We argue that this adoption has outpaced our understanding of what distillation actually accomplishes. When a small model is trained on the outputs of a large model, it learns to produce similar outputs. But producing similar outputs is not the same as performing the same computation. A student model that generates a correct chain-of-thought after distillation may be pattern-matching against learned textual templates rather than executing the multi-step reasoning that the teacher performed to produce those templates.
This distinction has practical consequences. If distillation transfers surface behavior but not underlying computation, then distilled models will fail systematically on tasks that require the kind of reasoning the teacher performed but the student only learned to imitate. Understanding where this boundary lies is essential for making informed deployment decisions.
In this paper, we make three contributions. First, we categorize what distillation transfers (Section III) and what it does not (Section IV), drawing on empirical evidence from the literature. Second, we formalize the distillation ceiling as a function of model capacity and representation gap (Section V). Third, we argue for scaffolding as a complement to distillation for tasks that exceed the student's intrinsic capacity (Section VI) and propose a measurement protocol for characterizing what a distilled model actually learned (Section VII).
The foundational formulation of knowledge distillation for neural networks is due to Hinton, Vinyals, and Dean [1]. The key insight is that a teacher model's softmax output distribution contains richer information than hard labels alone. The "soft targets" encode relative similarity between classes, providing a form of dark knowledge that guides the student toward a smoother loss surface. The training objective combines the standard cross-entropy loss with a Kullback-Leibler divergence term between the student's softened outputs and the teacher's softened outputs, controlled by a temperature hyperparameter.
This framework was developed for classification tasks and assumes that the teacher and student operate over the same output space. For language models, the output space is the full vocabulary at each generation step, making the distillation objective a sequence-level extension of the original formulation.
In the language model setting, task-specific distillation trains a student model on the teacher's outputs for a particular downstream task. The Alpaca model [7] exemplifies this approach: a 7-billion-parameter LLaMA model was fine-tuned on 52,000 instruction-following demonstrations generated by text-davinci-003, at a training cost reported to be under $600. The Vicuna model [17] followed a similar recipe, fine-tuning on user-shared conversations from ChatGPT.
These approaches transfer the teacher's instruction-following behavior to the student through supervised learning on (input, output) pairs. The student learns to produce outputs that resemble the teacher's, but the mechanism by which it does so is unconstrained. The student may develop internal representations that differ substantially from the teacher's, so long as the surface-level outputs match.
A limitation of standard distillation is that the student is trained on the teacher's outputs, which may not reflect the states the student itself would visit during generation. On-policy distillation addresses this by having the student generate outputs and then receiving the teacher's feedback on those outputs [2]. This approach, formalized by Agarwal et al. [2] and shown to be effective at ICLR 2024, allows the student to learn from its own mistakes rather than only from the teacher's successes.
On-policy distillation improves sample efficiency for the student, but it does not resolve the fundamental limitation we identify: the teacher provides corrective signal at the output level, not at the level of internal computation. The student still learns what to output, not how the teacher would have computed it internally.
Chain-of-thought (CoT) prompting, introduced by Wei et al. [11], demonstrated that large models can solve complex reasoning tasks when prompted to generate intermediate reasoning steps. This raised a natural question: can a small model learn to reason by being trained on a large model's chain-of-thought traces?
Several approaches have explored this. Wang et al. [3] proposed SCOTT, which uses contrastive decoding to elicit consistent rationales from the teacher and trains the student with counterfactual reasoning to ensure faithfulness. Li et al. [4] showed that symbolic chain-of-thought distillation (SCoTD) enables models with 125 million to 1.3 billion parameters to benefit from chain-of-thought prompting, with human judgments rating student-generated chains as comparable to teacher chains. Hsieh et al. [14] demonstrated that distilling step-by-step allows a 770-million-parameter T5 model to outperform a 540-billion parameter PaLM model on certain benchmarks by using extracted rationales as additional supervision.
These results are striking, but they measure success in terms of output accuracy and surface-level chain quality. Whether the student model actually performs multi-step reasoning, or instead learns to generate plausible-sounding reasoning text that correlates with correct answers, remains an open question.
The most recent wave of reasoning distillation comes from large reasoning models such as DeepSeek-R1 [16]. The DeepSeek-R1 system demonstrates that reinforcement learning can incentivize reasoning capabilities in LLMs without supervised reasoning traces, and that these capabilities can then be distilled into smaller models. The distilled variants (1.5B to 70B parameters) show improved performance on mathematical and reasoning benchmarks.
The success of reasoning distillation at the benchmark level does not resolve the question of mechanism. When a distilled 7B model solves a math problem it could not solve before distillation, has it learned to reason, or has it learned a mapping from problem surface features to solution templates? We argue that the answer is mostly the latter, and that the distinction matters.
We identify four categories of knowledge that distillation reliably transfers from teacher to student.
The most basic form of transfer is stylistic and structural pattern matching. When a student model is trained on a teacher's outputs, it learns the distributional properties of the teacher's text: sentence structure, vocabulary choice, register, and discourse organization. This is the easiest form of knowledge to transfer because it requires only that the student model learn surface-level statistical regularities.
The success of models like Alpaca [7] and Vicuna [17] in producing high-quality conversational text confirms this. These models learned to mimic the teacher's communication style with high fidelity, despite being an order of magnitude smaller.
When a student model is trained on chain-of-thought outputs from a teacher, it learns to produce text that resembles reasoning chains. The student generates steps that follow the same structure as the teacher's chains: identifying relevant quantities, performing intermediate calculations, and arriving at a final answer.
We call these surface reasoning chains because the student reproduces the textual form of reasoning without necessarily performing the underlying computation. The distinction is analogous to a student who memorizes worked examples in a textbook: they can reproduce the solution steps for problems similar to those they have seen, but they may not be able to solve novel problems that require the same reasoning principle applied in a new context.
Empirical evidence supports this characterization. Li et al. [4] found that distilled models produce chains judged as comparable to the teacher's by human evaluators, yet these models still underperform the teacher on out-of-distribution tasks. The chains look right, but the reasoning they represent is shallower than it appears.
Distillation effectively transfers task-specific heuristics: learned shortcuts that correlate with correct outputs for a given task distribution. A distilled model trained on mathematical reasoning traces may learn heuristics such as "when the problem mentions rate and time, multiply them" or "if the question asks for a percentage, the answer is likely between 0 and 100."
These heuristics are valuable within the training distribution. They allow the student to solve many problems correctly by pattern matching against known problem types. The limitation is that heuristics are brittle: they degrade when the problem distribution shifts, when novel combinations of concepts appear, or when the correct solution requires reasoning that contradicts the learned heuristic.
A practical and often undervalued form of transfer is format compliance. Large models learn to follow complex output formats: structured JSON, specific answer templates, multi-section responses, and domain-specific conventions. Distilling this behavior into smaller models is reliable because format compliance is primarily a surface behavior that can be learned from examples.
This form of transfer has high deployment value. A smaller model that reliably produces correctly formatted outputs can be integrated into production pipelines more easily than a larger model that sometimes deviates from the expected format, even if the larger model has superior reasoning capabilities.
We identify three categories of knowledge that distillation does not reliably transfer, and argue that these represent the most significant limitations of the approach.
When a large language model solves a complex reasoning problem, it performs computation across many layers and many forward passes. Each token generation step involves the full forward pass of the model, and the intermediate representations encode information that guides subsequent generation. The depth of this computation is a function of the model's parameter count, layer count, and the width of its hidden representations.
A smaller model has fewer parameters, fewer layers, and narrower hidden representations. This means it has less computational capacity per forward pass. When distillation trains the smaller model to produce the same outputs as the larger model, it is asking the smaller model to achieve the same result with less computation. For tasks that require the full computational depth of the teacher, the student simply does not have the capacity to perform the same operations.
This is not a data problem or a training problem. It is an architectural constraint. No amount of training data, whether from distillation or otherwise, can give a model more parameters or more layers than it has. The scaling laws established by Kaplan et al. [8] and refined by Hoffmann et al. [9] predict smooth improvements in loss with increasing model size, but they also imply that smaller models have a lower performance ceiling for any fixed task.
Chain-of-thought distillation transfers the text of reasoning chains, but not the internal process that generated them. When a large model produces a chain-of-thought, each step in the chain is the result of computation performed by the model's layers. The model may be performing internal search, evaluating candidate intermediate steps, and selecting the most promising path. This internal process is not visible in the output text.
A distilled model trained on these chains learns to produce text that follows the same structure, but it has no direct access to the teacher's internal search process. The student must discover its own internal mechanism for producing each step, and with fewer parameters and layers, this mechanism will typically be simpler. The student may learn to generate the correct next step by pattern matching against similar steps in the training data, rather than by performing the reasoning that the teacher performed.
This distinction has been partially explored in the interpretability literature. Work on mechanistic interpretability suggests that the computations performed by transformer layers are not easily decomposed into discrete reasoning steps, and that the relationship between chain-of-thought text and internal computation is complex and often non-monotonic [18]. If the teacher's internal reasoning does not cleanly map onto the chain-of-thought text it produces, then training a student on that text provides only an indirect and lossy signal about the teacher's actual computation.
The most significant limitation of distillation is its inability to transfer compositional generalization: the capacity to combine known concepts in novel ways to solve problems not seen during training.
A large model that has learned to reason about rates, distances, and time independently can often compose these concepts to solve a problem that involves all three, even if it has not seen that specific combination. This compositional ability arises from the model's capacity to perform multi-step internal computation, combining representations of individual concepts into a coherent solution.
A distilled model that has learned surface reasoning chains for individual concept types may not be able to compose them. The student has learned templates for each concept, but composing templates requires a form of reasoning that goes beyond template matching. If the student has not learned the underlying representations that make composition possible, it will fail on novel combinations even when it can solve problems involving each component concept in isolation.
This is consistent with the observations of Wei et al. [10] on emergent abilities: certain capabilities appear only at sufficient model scale. If these emergent capabilities depend on model capacity rather than training data, then distillation cannot transfer them to a model below the capacity threshold.
We formalize the limitations of distillation as the distillation ceiling, a concept that captures the maximum performance a student model can achieve through distillation from a given teacher.
Definition 1 (Distillation Ceiling). Let T be a teacher model with parameters theta_T and S be a student model with parameters theta_S, where |theta_S| < |theta_T|. Let D_train be the distillation dataset generated by T, and let L(S, D) be the loss of model S on dataset D. The distillation ceiling DC(S, T) is the infimum of L(S, D_train) over all possible distillation procedures, as the amount of distillation data approaches infinity.
In practice, the distillation ceiling is determined by two factors:
(1) Model Capacity. The student's parameter count, layer depth, and hidden dimension impose a hard ceiling on the complexity of functions it can represent. For any task whose solution requires a function more complex than the student can represent, the student cannot match the teacher regardless of the distillation procedure.
(2) Representation Gap. The teacher and student may represent the same function differently due to their architectural differences. Even if the student can represent a function that achieves the same outputs as the teacher, the distillation process may not find it because the optimization geometry differs between architectures.
We state this as a proposition:
Proposition 1. For any student model S with finite capacity C_S and teacher model T with capacity C_T > C_S, there exists a family of tasks F such that for all tasks f in F, the distillation ceiling DC(S, T) is strictly greater than zero (i.e., perfect transfer is impossible), and the complexity of tasks in F increases monotonically with the ratio C_T / C_S.
This proposition follows from standard results in approximation theory. A model with fewer parameters cannot represent all functions representable by a larger model. The set of tasks that require the larger model's capacity defines the set of tasks for which distillation has a nonzero ceiling.
The practical implication is that the distillation ceiling is not a single number but a function of the task distribution. For simple tasks that require only surface pattern matching, the ceiling may be at or near the teacher's performance. For tasks requiring deep compositional reasoning, the ceiling may be far below it.
The scaling laws literature [8][9] provides empirical evidence for this. Performance improves predictably with model size, and the improvement is roughly log-linear. This means that each factor-of-two reduction in model size incurs a predictable performance cost, and this cost is concentrated on the tasks that require the most capacity. Distillation can recover some of this cost for tasks within the student's capacity, but it cannot eliminate it for tasks beyond it.
A related observation comes from the LIMA result [6], which showed that alignment (instruction following) requires relatively little training data because the knowledge is already present in the pretrained model. This suggests that the knowledge a distilled model needs is largely acquired during pretraining, and distillation primarily teaches the student how to surface that knowledge in the right format. For tasks where the required knowledge is not present in the pretrained student, distillation has limited value.
If distillation has a ceiling determined by model capacity, then approaches that augment the student model's capacity without increasing its parameter count may be more effective for tasks beyond the ceiling. We use the term scaffolding to describe external mechanisms that compensate for what the student model cannot do on its own.
Scaffolding takes several forms:
(1) Tool Use. Providing the student model with access to external tools (calculators, search engines, code interpreters) effectively increases its computational capacity for specific task types. A model that cannot perform multi-digit arithmetic can still solve arithmetic problems if it can call a calculator.
(2) Retrieval Augmentation. Giving the student model access to external knowledge bases compensates for the knowledge it lacks due to its smaller parameter count. The model does not need to encode all knowledge in its parameters if it can retrieve relevant information at inference time.
(3) Multi-Step Decomposition. Breaking complex tasks into simpler subtasks, each within the student's capacity, allows the system as a whole to solve problems that no individual step requires deep