PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
Neural Knowledge Crystallization: How Facts Emerge Across Transformer Depth
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
Factual knowledge in large language models is not uniformly distributed across transformer layers. We argue that factual associations undergo a multi-stage process analogous to crystallization: early layers compute contextual and relational representations, middle layers compose these into thematic and factual groupings, and late layers project specific facts into vocabulary space for output. We formalize this "crystallization" view, drawing on recent work in mechanistic interpretability, knowledge probing, causal tracing, and the logit lens. We present evidence from knowledge neuron analyses, activation patching experiments, and layer-wise decoding to support a three-phase model of how facts are stored, composed, and expressed. We discuss direct implications for model editing (targeting specific layers for surgical fact updates), retrieval augmentation (understanding when retrieved information integrates with parametric knowledge), and interpretability (reading intermediate representations). We also provide a formal framework with definitions of knowledge states across depth and conditions under which crystallization occurs.
Modern large language models (LLMs) store and retrieve vast quantities of factual knowledge encoded in their parameters. When a model answers "The capital of France is Paris," it must activate the relevant factual association, compose it with the syntactic structure of the sentence, and project the answer token into vocabulary space. This process occurs across dozens of transformer layers, and the representations at each depth play distinct roles.
Recent work in mechanistic interpretability has begun to illuminate the internal computations of transformers. Studies on knowledge neurons [6], causal tracing [1], the logit lens [8], and transformer circuits [5] have each contributed partial views of how facts are stored and retrieved. However, a unified picture of the layer-wise progression from distributed contextual representations to specific factual outputs has been lacking.
In this paper, we propose the "neural knowledge crystallization" framework. We use this metaphor deliberately: just as a crystalline solid forms from a disordered solution through nucleation and growth, factual knowledge in transformers condenses from broadly distributed early-layer representations into precisely targeted late-layer activations that project specific tokens. The metaphor captures several key observations: (1) early representations are diffuse and multi-purpose, (2) middle layers exhibit increasing factual specificity, and (3) late layers make sharp projections into vocabulary space.
Our contributions are as follows:
1. We formalize the crystallization metaphor with definitions of knowledge states at each layer depth, drawing on the residual stream framework [5]. 2. We synthesize evidence from knowledge probing, causal tracing, and the logit lens to support a three-phase model. 3. We discuss implications for model editing, showing that layer targeting matters for surgical fact updates. 4. We discuss implications for retrieval augmentation, identifying the layer range where retrieved information integrates with parametric knowledge. 5. We provide a formal framework with crystallization conditions and propositions.
The remainder of this paper is organized as follows. Section II reviews background on mechanistic interpretability. Section III presents the crystallization metaphor. Section IV reviews evidence from knowledge probing. Section V provides the layer-by-layer view. Sections VI and VII discuss implications for model editing and retrieval augmentation, respectively. Section VIII presents the formal framework. Section IX reviews related work. Section X discusses limitations. Section XI concludes.
Elhage et al. [5] introduced a mathematical framework for analyzing transformer circuits. A central insight is that the residual stream serves as the primary communication channel between layers. Each transformer layer reads from the residual stream via its attention and MLP sublayers, and writes back to it additively. Formally, if x_l denotes the residual stream at layer l, then:
x_{l+1} = x_l + Attn_l(x_l) + MLP_l(x_l)
This additive structure means that information written by an early layer persists through all subsequent layers unless overwritten. The residual stream framework has been foundational for understanding how transformers compose computations across depth [5].
Dai et al. [6] identified "knowledge neurons" in pretrained transformers: specific neurons in MLP layers whose activation correlates with the expression of particular factual associations. By suppressing or amplifying these neurons, they demonstrated causal influence on the model's factual outputs. Their analysis of BERT and RoBERTa found that knowledge neurons tend to cluster in middle-to-late layers, with each neuron encoding a specific subject-relation-object triple. This work established that factual knowledge is localized, at least partially, within specific network components.
Meng et al. [1] developed causal tracing (also called causal intervention analysis) to identify which components of a transformer are causally responsible for factual recall. In their work on GPT-style models, they corrupted subject tokens, restored activations at specific layers, and measured the recovery of correct factual predictions. They found that middle MLP layers (roughly layers 15-25 in a 48-layer model) carry the most causally relevant information for factual recall. This technique has since been applied to multiple model families and consistently reveals layer-specific causal contributions [1][3].
The logit lens [8] is a technique that projects intermediate layer representations through the model's unembedding matrix to observe what token the model would predict at each layer depth. This provides a direct window into how predictions evolve across layers. Typically, early layers produce diffuse or incorrect predictions, middle layers shift toward the correct token, and late layers sharpen the distribution. Belrose et al. [9] extended this with the "tuned lens," which trains affine transformations at each layer to better decode intermediate representations. The tuned lens revealed that models often "know" the correct answer several layers before the final output, but the information must still be refined and projected.
Geva et al. [2] demonstrated that transformer feed-forward (MLP) layers operate as key-value memories. The first linear layer (W1) acts as keys that match against the input representation, and the second linear layer (W3) acts as values that write information back to the residual stream. This interpretation provides a mechanistic account of how MLP layers store and retrieve knowledge: specific rows of W1 correspond to input patterns, and the corresponding rows of W3 output semantically meaningful vectors in the residual stream. Geva et al. [4] later showed that these value vectors promote specific concepts in vocabulary space, providing a direct link between MLP computations and factual output.
We propose that factual knowledge in transformers undergoes a process we call "neural knowledge crystallization." The term is chosen to highlight three structural parallels with physical crystallization.
In physical crystallization, a solution contains dissolved molecules in a disordered state. As conditions change (temperature, concentration), nucleation sites form, and molecules arrange into ordered crystalline structures. The process is not instantaneous; it proceeds through identifiable phases.
In neural knowledge crystallization, the input representation at early layers contains a mixture of contextual, syntactic, and semantic information in a distributed form. As the representation passes through transformer layers, factual associations condense into increasingly specific patterns, ultimately projecting precise tokens at the output. We formalize this into three phases.
Definition 1 (Dispersed State). At layer l, a representation h_l is in a dispersed state with respect to fact f if the information needed to express f is spread across multiple dimensions of h_l and cannot be decoded by a simple linear probe to the vocabulary.
Definition 2 (Nucleation). A representation h_l nucleates with respect to fact f at layer l if there exists a linear projection of h_l that assigns non-trivial probability mass to the correct answer for f, but the distribution remains entangled with other facts or contextual information.
Definition 3 (Crystallization). A representation h_l is crystallized with respect to fact f at layer l if (a) a linear projection of h_l assigns high probability to the correct answer for f, (b) this assignment is robust to perturbations of non-essential context, and (c) competing factual associations have been suppressed.
These definitions capture the progression from diffuse to specific. The crystallization metaphor is not merely illustrative; it highlights that the process has identifiable phases, that each phase corresponds to specific layer ranges, and that the transition between phases is not always monotonic (some facts crystallize earlier or later depending on their complexity and the model's training distribution).
We propose the following informal thesis:
Thesis (Crystallization). For a given factual query, the transformer's computation across layers proceeds through three phases: (1) contextual encoding (early layers), where syntactic and entity-level information is extracted; (2) thematic composition (middle layers), where relational and factual groupings are assembled; and (3) vocabulary projection (late layers), where specific factual tokens are promoted and competitors are suppressed.
In the following sections, we review evidence supporting each phase.
Dai et al. [6] conducted systematic experiments on BERT and RoBERTa, probing individual neurons for their correlation with specific facts. They identified neurons whose activation was both necessary and sufficient for correct factual expression. Several patterns emerged. First, knowledge neurons were concentrated in middle-to-late layers (layers 6-12 in a 12-layer model). Second, individual neurons showed specificity: suppressing a single neuron could flip the model's output from "Paris" to an incorrect city for "The capital of France is ___." Third, knowledge neurons for related facts (e.g., capitals of European countries) tended to co-locate in nearby layers, suggesting a compositional structure.
These findings align with the crystallization view. If knowledge were uniformly distributed, no single neuron would show strong causal influence. The localization of knowledge neurons suggests that middle layers perform a kind of nucleation, where distributed representations condense into specific neuronal activations that carry factual content.
Meng et al. [1] applied causal tracing to GPT-J (6B parameters, 28 layers) and GPT-Neo (2.7B, 32 layers). Their protocol corrupts the subject token embedding, then restores activations at individual layers and measures the model's ability to recall the correct fact. Key findings include:
1. Subject token embeddings (layer 0) carry essential information, but alone are insufficient. 2. Middle MLP layers (layers 15-25 in GPT-J) show the strongest causal effect when restored. 3. Late attention layers (layers 25-28) show strong causal effect, consistent with their role in directing the output. 4. Early layers (1-10) show relatively weak causal contribution to factual recall.
This pattern maps cleanly onto the crystallization phases. Early layers extract contextual information (necessary but not sufficient), middle layers nucleate the factual association, and late attention layers project it toward the output. The causal dominance of middle MLP layers is precisely what we would expect if factual knowledge crystallizes through MLP key-value memories [2].
The logit lens [8] provides a complementary view by decoding intermediate representations. When applied to factual queries such as "The Eiffel Tower is located in ___," several consistent observations arise:
1. Layers 0-5: The decoded distribution is heavily influenced by the surface form of the prompt. The top tokens are often generic (articles, prepositions) or reflect syntactic expectations rather than factual content. 2. Layers 6-15: The distribution begins to shift. Tokens semantically related to the correct answer gain probability, but often alongside plausible alternatives. For "Paris," tokens like "London," "Berlin," and "France" may appear. 3. Layers 16-25: The correct answer ("Paris") rises sharply, and alternatives are suppressed. The distribution narrows. 4. Layers 26-30 (for a 30-layer model): The correct answer is firmly at the top with high probability.
The tuned lens [9] confirmed and sharpened these observations. By training layer-specific affine transformations, Belrose et al. showed that the "true" prediction at each layer is often better than the raw logit lens suggests, meaning that the model encodes the correct answer earlier than naive projection would indicate. However, the qualitative progression remains: diffuse early, specific late.
Geva et al. [4] analyzed how MLP layers promote concepts in vocabulary space. They showed that MLP output vectors have a direct interpretation in the vocabulary: each MLP output promotes specific tokens and suppresses others. In their analysis, middle-to-late MLP layers were responsible for promoting the final factual token, while earlier MLP layers promoted semantically related but less specific tokens.
This finding provides mechanistic support for the crystallization view. MLP layers do not simply store facts as static key-value pairs; they actively promote and suppress tokens at each layer. The progression from broad semantic promotion (early-middle layers) to specific factual promotion (late layers) is the computational realization of the crystallization process.
We now synthesize the evidence above into a layer-by-layer account of factual knowledge processing.
In the first third of a transformer, the primary computations are syntactic and contextual. Clark et al. [11] analyzed attention patterns in BERT and found that early layers attend heavily to syntactic dependencies: subject-verb, determiner-noun, and preposition-object relationships. Vig et al. [10] confirmed this with a broader set of linguistic phenomena.
For factual recall, early layers perform two essential functions. First, they extract the subject entity and its immediate context. When the model processes "The Eiffel Tower is located in ___," early layers must identify "Eiffel Tower" as the subject and "is located in" as the relation cue. Second, early layers perform routing: they direct the representation toward the appropriate middle-layer circuits that handle geographical facts, as opposed to biographical or temporal facts.
We characterize early-layer representations as being in a dispersed state (Definition 1). The information needed for the factual answer is present in a weak, distributed sense (the embedding of "Eiffel Tower" contains cues about Paris), but it cannot be decoded by a simple linear projection. The representation is entangled with syntactic, positional, and contextual information.
The middle layers are where factual knowledge nucleates and begins to crystallize. This is the layer range where:
1. MLP layers activate knowledge neurons [6]. 2. Causal tracing shows maximum factual contribution [1]. 3. The logit lens shows the correct answer rising in probability [8][9]. 4. MLP outputs promote semantically relevant tokens [4].
We interpret this as the composition phase. Middle-layer MLPs assemble distributed cues into coherent factual associations. The key-value memory model of Geva et al. [2] explains the mechanism: keys in the MLP match patterns like "subject: Eiffel Tower, relation: location," and the corresponding values output vectors that promote "Paris" in the residual stream.
This phase corresponds to nucleation (Definition 2). The representation can be linearly decoded to produce the correct answer, but the answer may still be entangled with alternatives. The crystallization is not yet complete; competing facts (other cities, other landmarks) retain some probability mass.
Importantly, the middle layers are also where relational and thematic knowledge composes. A model answering "The currency of the country where the Eiffel Tower is located is ___" must compose two facts: (Eiffel Tower, location, France) and (France, currency, Euro). This composition appears to occur in the middle layers, where circuits for individual facts chain together [5][7].
In the final third of the model, the computation shifts from factual composition to output projection and disambiguation. Late layers perform three functions:
1. Vocabulary projection: The residual stream representation is projected through the unembedding matrix to produce the final token probabilities. Late attention layers often attend to the position where the answer should be placed [1]. 2. Disambiguation: Competing factual associations are suppressed. If the middle layers promoted both "Paris" and "France" for "The Eiffel Tower is located in ___," late layers suppress "France" (which is correct for a different query) and amplify "Paris." 3. Calibration: The output probability distribution is calibrated. The model adjusts the confidence of its prediction based on the accumulated evidence across layers.
This phase corresponds to crystallization (Definition 3). The representation projects a high-confidence, specific answer. Competing alternatives are suppressed. The factual output is robust to surface-level perturbations of the prompt (e.g., "The famous tower in Paris is called the ___" activates the same factual circuit with different surface form but the same core knowledge).
The crystallization framework has direct implications for model editing, the task of surgically updating a model's knowledge without retraining.
If factual knowledge is concentrated in specific layers (the nucleation phase), then editing should target those layers rather than applying uniform modifications. Meng et al. [3] demonstrated this with ROME (Rank-One Model Editing), which modifies a single MLP weight matrix at a specific layer to update a factual association. Their method outperformed approaches that distribute edits across multiple layers, precisely because it targets the layer where the fact is most localized.
The crystallization view predicts that the optimal editing layer lies in the middle range (the nucleation zone). Editing too early (dispersed state) would require modifying many dimensions and would risk affecting unrelated representations. Editing too late (crystallized state) would require overcoming the accumulated downstream computations that have already committed to the original fact.
Yao et al. [14] surveyed model editing methods and found consistent evidence that middle-layer edits are most effective for factual updates. Methods like MEMIT [3], which edit multiple middle MLP layers simultaneously, achieve better results than single-layer or uniformly distributed edits.
The crystallization framework also predicts the types of side effects that model edits produce. If a fact is nucleated in a specific set of neurons that also participate in related facts, then editing one fact may affect related facts. For example, editing "The capital of France is Paris" to "The capital of France is Lyon" might also affect "The capital of the country containing the Eiffel Tower is ___" because both facts share middle-layer circuits.
This prediction aligns with empirical findings. Meng et al. [1][3] showed that fact edits sometimes cause cascading errors in related facts. The degree of cascading depends on the overlap of the affected neurons and circuits in the nucleation zone.
Based on the crystallization framework, we suggest the following guidelines for model editing:
1. Identify the nucleation layer range using causal tracing [1] or activation patching. 2. Target edits to MLP layers within the nucleation zone, as these carry the most localized factual information [6]. 3. Verify that the edit does not disrupt related facts by testing on paraphrases and related queries. 4. If a single-layer edit is insufficient, use multi-layer edits within the nucleation zone (as in MEMIT [3]) rather than spreading edits across unrelated layers.
Retrieval-augmented generation (RAG) systems combine parametric knowledge (stored in model weights) with retrieved knowledge (provided in context). The crystallization framework raises a natural question: at which layer does retrieved information integrate with parametric knowledge?
When a model receives retrieved context alongside a query, the context tokens are processed through the same transformer layers as the query. Early layers extract contextual and syntactic information from the retrieved text. Middle layers compose this with the model's parametric knowledge. The key question is whether retrieved information integrates at the nucleation phase (middle layers) or at the projection phase (late layers).
We propose that retrieved information integrates primarily at the nucleation phase, for the following reason. If retrieved information reached only the late projection layers, it would need to override already-crystallized parametric facts. This would require the model to "undo" its own factual outputs, which is computationally expensive and unreliable. Instead, if retrieved information enters the computation during the nucleation phase, it can shape the factual association before crystallization occurs.
This is consistent with observations by Geva et al. [2] that MLP key-value memories are context-sensitive: the keys match against the full input, including retrieved context. A retrieved passage that directly states "The Eiffel Tower is located in Paris" provides a strong activation for the relevant MLP keys, reinforcing the parametric knowledge.
When retrieved information conflicts with parametric knowledge (e.g., the model believes "Paris" but the retrieved text says "Lyon"), the crystallization framework predicts a layer-dependent resolution. At the nucleation phase, both the parametric and retrieved activations compete. If the retrieved activation is strong enough (e.g., the passage is long, authoritative, and directly relevant), it can shift the nucleation toward the retrieved fact. If the parametric activation is stronger, the model defaults to its stored knowledge.
This competition occurs in the middle layers, which explains why RAG systems sometimes fail to override parametric knowledge: the parametric fact has already nucleated before the retrieved information gains sufficient influence. Enhancing the early-layer processing of retrieved context (e.g., through longer context windows or specialized attention patterns) may improve the integration of retrieved information.
The crystallization view suggests several design principles for RAG systems:
1. Place retrieved context before the query in the prompt, so that early layers can extract relevant information before the factual nucleation begins. 2. Use passage-level retrieval rather than token-level retrieval, as passage context provides richer activations for the middle-layer nucleation process. 3. When retrieved information must override parametric knowledge, provide strong signals (multiple passages, explicit statements) to shift the nucleation phase. 4. Consider layer-specific interventions (e.g., amplifying retrieved-context activations at middle layers) to improve integration.
We now provide a formal framework for the crystallization process, building on the residual stream model [5].
Let L be the total number of transformer layers. Let h_l in R^d denote the residual stream state at layer l, where d is the model dimension. Let V be the vocabulary and W_U in R^{|V| x d} be the unembedding matrix.
Definition 4 (Layer-wise Prediction). The layer-wise prediction at layer l is:
p_l = softmax(W_U h_l)
This is the logit lens [8] projection.
Definition 5 (Knowledge State). A model is in knowledge state K_l with respect to fact f = (s, r, o) at layer l if:
K_l(f) = p_l(o) - max_{o' != o} p_l(o')
where o is the correct object token and o' ranges over competitors. K_l(f) > 0 means the correct answer is ranked first.
Definition 6 (Crystallization Threshold). Fact f crystallizes at layer l* if:
l* = min{l : K_l(f) >= tau for all l' >= l}
where tau > 0 is a threshold parameter. This is the first layer after which the correct answer maintains a consistent lead.
Proposition 1 (Monotonic Crystallization). For most factual queries in well-trained models, the knowledge state K_l(f) is approximately monotonically increasing in l after the nucleation layer. That is, once the correct answer begins to lead, its lead generally grows.
This proposition is supported by logit lens observations [8][9], though exceptions exist for ambiguous or multi-hop facts.
Proposition 2 (MLP Contribution to Crystallization). Let delta_l = MLP_l(h_l) be the contribution of the MLP at layer l. For facts f that crystallize at layer l*, the contribution ||delta_{l*}|| is larger (relative to other layers) than for facts that crystallize at other layers. That is, the crystallization layer shows elevated MLP activity.
This follows from the causal tracing results of Meng et al. [1], who found that middle MLP layers carry the dominant causal signal for factual recall.
Proposition 3 (Early Dispersal). For well-trained models, the knowledge state K_l(f) for typical factual queries satisfies K_l(f) < 0 for l in [0, alpha L], where alpha is approximately 0.2-0.3. That is, in the first quarter of the model, the correct answer is not yet ranked first.
This is a weaker claim than saying "the model doesn't know the answer." The tuned lens [9] shows that affine transformations can sometimes recover the correct answer from early layers. We claim only that the standard (linear) projection does not identify the correct answer in early layers.
Definition 7 (Crystallization Span). The crystallization span of fact f is the interval [l_nuc(f), l_crys(f)], where l_nuc(f) is the nucleation layer (first layer where K_l(f) > 0) and l_crys(f) is the crystallization layer (first layer where K_l(f) >= tau stably).
We observe empirically that the crystallization span varies by fact type:
1. Simple associative facts (e.g., capital cities) tend to have narrow spans in the middle layers. 2. Compositional facts (e.g., "currency of the country where X is located") have wider spans, as they require multiple nucleation stages. 3. Rare or long-tail facts have spans shifted toward later layers, consistent with less training signal.
Definition 8 (Editing Fidelity). An edit to layer l is said to have fidelity epsilon for fact f if the post-edit knowledge state satisfies:
|K_l^edited(f) - K_l^original(f')| < epsilon
where f' is the desired new fact. High fidelity means the edit successfully shifts the model's knowledge from f to f' without disrupting other computations.
The field of mechanistic interpretability seeks to reverse-engineer the computations performed by neural networks. Elhage et al. [5] established the transformer circuits framework, which decomposes transformers into independently analyzable circuits. Neel Nanda [13] argued for the importance of interpretable bases and progress measures in understanding model computations. Bricken et al. [16] applied dictionary learning (sparse autoencoders) to decompose transformer activations into interpretable features, finding monosemantic neurons that correspond to specific concepts. Bills et al. [17] used language models themselves to generate natural language explanations of individual neurons.
Geva et al. [2] showed that MLP layers function as key-value memories, providing a concrete mechanism for knowledge storage. Geva et al. [4] further demonstrated that MLP outputs promote specific concepts in vocabulary space. Hernandez et al. [15] showed that language models implement simple vector arithmetic for certain factual associations (e.g., "Paris" - "France" + "Italy" = "Rome"). Meng et al. [1][3] localized and edited factual associations, providing both analytical tools and practical editing methods.
Clark et al. [11] analyzed attention patterns across BERT layers, finding early-layer syntactic specialization. Vig et al. [10] conducted a comprehensive analysis of linguistic phenomena across layers. The logit lens [8] and tuned lens [9] provided direct tools for layer-wise prediction analysis. Li et al. [18] introduced inference-time intervention, steering model outputs by modifying activations at specific layers. Turner et al. [19] proposed activation addition, modifying model behavior by adding carefully chosen vectors to intermediate activations.
The model editing literature has grown rapidly. Meng et al. [1] introduced causal tracing and localized factual associations. Meng et al. [3] developed MEMIT for multi-layer editing. Yao et al. [14] provided a comprehensive survey of editing methods and their trade-offs. Hase et al. [20] studied how model edits affect downstream predictions, finding that edits are more reliable when they target the correct layer range.
We identify several limitations of the crystallization framework.
First, the metaphor should not be taken too literally. Physical crystallization is a thermodynamic process with well-defined free energy landscapes. Neural knowledge processing is a learned computation without such formal guarantees. The metaphor captures the qualitative progression from diffuse to specific, but the quantitative details depend on model architecture, training data, and hyperparameters.
Second, our framework is based primarily on evidence from autoregressive transformers (GPT-style models). Encoder-only models (BERT, RoBERTa) and encoder-decoder models (T5, BART) may exhibit different layer-wise dynamics. The bidirectional attention in encoder models may allow earlier factual integration, which would compress or shift the crystallization phases.
Third, the logit lens [8] and tuned lens [9] are imperfect tools for reading intermediate representations. The unembedding matrix is optimized for the final layer, and applying it to earlier layers may misrepresent the model's internal state. While the tuned lens [9] mitigates this by training layer-specific transformations, it introduces additional parameters that may overfit.
Fourth, our formal framework is descriptive rather than predictive. We define knowledge states and crystallization thresholds, but we do not provide a theory for predicting at which layer a given fact will crystallize. Such a theory would require understanding the relationship between training data, fact frequency, and layer allocation, which remains an open problem.
Fifth, the interactions between attention and MLP sublayers within a single transformer layer are complex. Our framework treats layers as atomic units, but in practice, the attention and MLP sublayers perform distinct computations that may not align cleanly with the crystallization phases. Some facts may nucleate in an MLP layer and then be modified by the subsequent attention layer before reaching the next MLP.
Sixth, we have focused on factual knowledge of the form (subject, relation, object). Other types of knowledge, such as procedural knowledge (how to perform multi-step reasoning), linguistic knowledge (grammar rules), and commonsense knowledge (physical intuitions), may follow different layer-wise dynamics. Extending the crystallization framework to these knowledge types is an area for future work.
We have proposed the neural knowledge crystallization framework for understanding how factual knowledge emerges across transformer depth. The framework identifies three phases: contextual encoding in early layers, factual nucleation in middle layers, and vocabulary projection in late layers. We supported this framework with evidence from knowledge neuron analysis [6], causal tracing [1], the logit lens [8], the tuned lens [9], and MLP vocabulary promotion studies [2][4].
The framework has practical implications. For model editing, it suggests targeting the nucleation zone (middle layers) for surgical fact updates, as demonstrated by ROME [1] and MEMIT [3]. For retrieval augmentation, it suggests that retrieved information integrates at the nucleation phase, and that conflicts between retrieved and parametric knowledge are resolved in middle layers. For interpretability, it provides a layer-wise map of what the model "knows" at each depth.
We presented a formal framework with definitions of knowledge states, crystallization thresholds, and crystallization spans. While the framework is primarily descriptive, it provides a structured vocabulary for discussing layer-wise knowledge processing and generates testable predictions about the effects of model edits and retrieval interventions.
Understanding how facts are stored and retrieved in transformers is not only an interpretability challenge; it is a prerequisite for building models that can be reliably updated, augmented, and trusted. The crystallization framework is one step toward that understanding.
REFERENCES
[1] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, "Locating and editing factual associations in GPT," in Advances in Neural Information Processing Systems, vol. 35, 2022. arXiv:2112.03544.
[2] M. Geva, R. Schuster, J. Berant, and O. Levy, "Transformer feed-forward layers are key-value memories," in Proceedings of EMNLP, 2021. arXiv:2012.14913.
[3] K. Meng, A. S. Sharma, A. J. Andonian, Y. Belinkov, and D. Bau, "Mass-editing memory in a transformer," in Proceedings of ICLR, 2023. arXiv:2210.07229.
[4] M. Geva, A. Caciularia, J. Berant, and O. Levy, "Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space," in Proceedings of EMNLP, 2022. arXiv:2203.14680.
[5] N. Elhage, N. Nanda, C. Olsson, et al., "A mathematical framework for transformer circuits," Transformer Circuits Thread, Anthropic, 2021.
[6] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, "Knowledge neurons in pretrained transformers," in Proceedings of ACL, 2022. arXiv:2104.08696.
[7] N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt, "Progress measures for grokking via mechanistic interpretability," in Proceedings of ICLR, 2023. arXiv:2301.05217.
[8] nostalgebraist, "Interpreting GPT: the logit lens," AI Alignment Forum, 2020.
[9] N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt, "Eliciting latent predictions from transformers with the tuned lens," arXiv preprint arXiv:2303.08112, 2023.
[10] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber, "Investigating BERT's knowledge of language: Five analysis methods with NPIs," in Proceedings of ACL, 2020. arXiv:1909.02597.
[11] K. C. Clark, U. Khandelwal, O. Levy, and C. D. Manning, "What does BERT look at? An analysis of BERT's attention," in Proceedings of the ACL Workshop BlackboxNLP, 2019. arXiv:1906.04341.
[12] N. Nanda, "Mechanistic interpretability, variables, and the importance of interpretable bases," Transformer Circuits Thread, Anthropic, 2022.
[13] N. Nanda, "How to think about interpretability," Transformer Circuits Thread, Anthropic, 2022.
[14] Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, and N. Zhang, "Editing large language models: Problems, methods, and opportunities," in Proceedings of EMNLP, 2023. arXiv:2305.13172.
[15] E. Hernandez, B. Z. Li, and J. Andreas, "Language models implement simple word2vec-style vector arithmetic," arXiv preprint arXiv:2305.16264, 2023.
[16] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al., "Towards monosemanticity: Decomposing language models with dictionary learning," Transformer Circuits Thread, Anthropic, 2023.
[17] S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders, "Language models can explain neurons in language models," Technical Report, OpenAI, 2023.
[18] K. Li, O. Patel, F. Vi'egas, H. Pfister, and M. Wattenberg, "Inference-time intervention: Eliciting truthful answers from a language model," in Advances in Neural Information Processing Systems, vol. 36, 2023. arXiv:2306.03341.
[19] A. Turner, L. Thiergart, G. Leech, D. Udell, J. Vazquez, U. Mini, and M. MacDiarmid, "Activation addition: Steering language models without optimization," arXiv preprint arXiv:2308.10248, 2023.
[20] P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun, "Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing in language models," in Advances in Neural Information Processing Systems, vol. 36, 2023. arXiv:2301.04213.
[21] H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey, "Sparse autoencoders find highly interpretable features in language models," arXiv preprint arXiv:2309.08600, 2023.
[22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback," in Advances in Neural Information Processing Systems, vol. 35, 2022. arXiv:2203.02155.