PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
Context Windows Are Not Memory: The Structural Limits of Attention and the Case for Persistent State
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
The rapid scaling of context windows in large language models (LLMs), from 512 tokens in early Transformers to over one million tokens in contemporary systems, has led to a widespread but incorrect assumption: that longer contexts solve the memory problem for AI agents. We argue that this represents a category error. Context windows provide high-bandwidth, ephemeral access to information within a single forward pass, but they lack five properties that any system deserving the name "memory" must exhibit: (1) persistence across sessions, (2) principled revision and deletion, (3) typed schema for structured state, (4) append-only audit trails, and (5) content-addressed retrieval. We formalize the distinction between working memory and durable memory, show that the "lost in the middle" phenomenon is a symptom of treating availability as recall, and present design principles for agent architectures that compose context windows with external persistent state. Our analysis indicates that treating context length as a substitute for memory architecture produces agents with no durable identity, no principled state management, and no repair mechanism when information degrades.
AI agents, attention mechanisms, persistent state, retrieval-augmented generation
The history of context window scaling in large language models is a history of impressive engineering. The original Transformer [1] operated over 512 tokens. Within five years, models routinely handled 8,192 tokens (GPT-3 [2]), then 32,768 (GPT-4 [3]), then 128,000 (Claude 2.1), then 200,000 (Claude 3), and now over one million tokens (Gemini 1.5 Pro [4]). Each expansion has been greeted with claims that the memory problem is solved or nearly solved.
We contend that these claims rest on a fundamental confusion. A context window is not a memory system. It is a bandwidth mechanism. It controls how much information can be attended to in a single forward pass, not how information persists, evolves, or is retrieved across time. The difference is not one of degree but of kind.
This paper makes three contributions. First, we provide a formal comparison of what context windows provide versus what memory systems require, identifying five structural properties that context windows cannot satisfy. Second, we revisit the "lost in the middle" phenomenon [5] and show that it is not merely a positional bias but a deeper indictment of the assumption that presence in the context window equals availability for recall. Third, we present a framework for agent architecture that treats context windows and persistent stores as complementary systems serving different computational roles.
The practical stakes are significant. Agents built on the premise that longer contexts eliminate the need for memory architecture exhibit predictable failure modes: they cannot maintain identity across sessions, cannot reason about what they have forgotten, cannot correct prior beliefs, and cannot provide auditable records of their state transitions. These are not edge cases. They are the central requirements for any system that claims to remember.
The context window of a Transformer-based language model is determined by the positional encoding scheme, the memory cost of the attention mechanism, and the training data distribution. The original Transformer [1] used sinusoidal positional encodings and operated on sequences of up to 512 tokens. The introduction of learned positional embeddings in GPT-2 extended this to 1,024 tokens.
The scaling trajectory accelerated with several technical advances. Rotary Position Embedding (RoPE) [6] provided a mechanism for extrapolating to sequence lengths beyond those seen during training. FlashAttention [7] reduced the memory cost of exact attention from O(n^2) in terms of memory writes to O(n) by tiling the computation to exploit GPU memory hierarchies. These advances enabled the training of models with 8,192-token contexts (GPT-3 [2]), then progressively longer windows.
By 2024, context lengths had reached territory that would have seemed implausible five years earlier. GPT-4 Turbo operated over 128,000 tokens [3]. Claude 3 supported 200,000 tokens. Gemini 1.5 Pro [4] demonstrated effective performance at one million tokens, with experimental results at ten million tokens.
Several architectural innovations have enabled the extension of context windows beyond the naive quadratic limit. Ring Attention [8] distributes the computation of attention across multiple devices arranged in a ring topology, allowing the effective context size to scale linearly with the number of devices. The key-value blocks are passed between devices while each device computes attention over its local query block and the arriving key-value blocks, overlapping communication with computation.
Infini-attention [9] takes a different approach by incorporating a compressive memory into the attention mechanism itself. Rather than attending over the full sequence, the model maintains a summary of past segments in a linear attention memory, combining local masked attention with global linear attention in a single Transformer block. This yields bounded memory usage even as the input length grows without bound.
An alternative to scaling the context window is to augment the model with external retrieval. Retrieval-Augmented Generation (RAG) [10] prepends retrieved documents to the model's input before generation. The Longformer [11] combines local sliding-window attention with global attention tokens, achieving linear complexity in sequence length. RETRO [12] integrates retrieval directly into the Transformer architecture by cross-attending to retrieved chunks during generation.
These approaches share a common structure: they extend the effective context by pulling in information from outside the current window. They do not, however, address the fundamental limitation we identify. Whether the information arrives via a 200,000-token window, a one-million-token window, or a retrieval system, it is still ephemeral. It exists for one forward pass and then vanishes.
The recognition that context windows are insufficient for persistent agents has spawned a line of work on explicit memory architectures. MemGPT [13] introduces a virtual context management system inspired by operating system memory hierarchies, with an inner context serving as main memory and an external storage system serving as disk. The agent manages data movement between these tiers using function calls.
Mem0 [14] proposes a persistent memory layer for conversational agents that maintains structured memories across sessions, with a graph-based variant that captures relational structure between memory entries. These systems represent the early stages of a shift from scaling context to architecting memory, but the distinction between the two has not been formally articulated.
We now formalize the distinction. Let C denote a context window of size n, defined as a sequence of tokens (t_1, t_2, ..., t_n). Let M denote a memory system, defined as a persistent, addressable store of structured entries.
Definition 1 (Context Availability). A token t_i is context-available for a model at inference time if and only if t_i is a member of the current context window C.
Definition 2 (Memory Accessibility). An entry e is memory-accessible if and only if e is stored in M and can be retrieved given an appropriate query, regardless of when e was written.
The difference is temporal. Context availability is bounded by the current inference pass. Memory accessibility is bounded only by the lifetime of the store. A token that falls out of the context window is not merely less salient; it ceases to exist as far as the model's computation is concerned.
Proposition 1. For any context window C of finite size n, there exists a sequence of inputs of length greater than n such that information from the first n - k tokens is no longer available to the model's attention mechanism, for any k < n.
This is trivially true but has profound implications. Every context window, no matter how large, is a sliding aperture over a stream of tokens. Information that passes through the aperture does not degrade gracefully; it disappears entirely. A memory system, by contrast, provides access to entries written at arbitrary points in the past, subject to the system's retrieval mechanism.
Table I summarizes the structural comparison:
Property Context Window Memory System ------------------------------------------------------- Temporal persistence Session-bound Cross-session Revision mechanism None (overwrite) Explicit CRUD Schema structure Flat token seq. Typed entries Audit capability None Append-only log Retrieval method Positional Content-addressed
Each row represents a property that is essential for a system to qualify as memory in any meaningful sense, and each row represents a property that context windows structurally lack. We examine each in turn.
A context window exists only within a single inference call or, at most, a single session bounded by the model's maximum input length. When a conversation ends and a new one begins, the context window is re-initialized. All information that was present in the previous window must be re-introduced, either by the user, by a system prompt, or by an external mechanism.
This is not a limitation that can be solved by making the window longer. A one-million-token context window that resets between sessions has exactly the same persistence as a 512-token context window that resets between sessions: zero. The scaling of n does not change the temporal boundary of the window; it changes only the volume of information that can be held within that boundary.
For an agent to maintain identity, preferences, learned facts, or task state across sessions, it requires a store that outlives the context window. This store must be addressable, queryable, and independently maintained from the model's inference loop.
In a context window, information is represented as tokens at fixed positions. There is no mechanism to revise a token in place, to mark a previously presented fact as superseded, or to delete information that is no longer accurate. When the context is reconstructed for a new session, the system must decide what to include, but this decision is made by the scaffolding, not by the model's attention mechanism.
A memory system must support at minimum four operations: create (add a new entry), read (retrieve an entry), update (revise an existing entry), and delete (remove an entry). These operations require addressable entries with stable identities. Tokens in a context window have positions, not identities. A token at position k in one session has no relationship to the token at position k in a different session.
The absence of revision mechanisms means that context-window-based agents cannot retract statements, correct misunderstandings, or update beliefs in a way that affects future behavior. Each new session starts from whatever is placed in the system prompt, not from a curated and evolving understanding.
A context window is a flat sequence of tokens. It has no notion of types, no distinction between a fact, a preference, a goal, a constraint, or a memory of a past interaction. All information is homogeneous at the representational level: it is text.
This is adequate for a single inference pass, where the model's attention mechanism can, in principle, distinguish between different kinds of information based on their content and position. It is inadequate for a persistent memory system, where entries must be classified, queried by type, and managed according to their category.
A preference ("the user prefers concise responses") should be stored differently from a fact ("the user's name is Alice") and differently from an episodic memory ("in the last session, we discussed the deployment timeline"). Each type has different persistence requirements, different revision policies, and different retrieval patterns. A flat token sequence cannot encode these distinctions.
In safety-critical and regulated domains, it is essential to maintain a record of what an agent knew, when it knew it, and how its knowledge changed over time. This requires an append-only log of memory operations, where each write is timestamped, attributed, and immutable once recorded.
A context window provides no such log. When the context is reconstructed for a new session, the previous context is not preserved anywhere in the system. There is no record of what was attended to in previous sessions, what information was available, or what decisions were made on the basis of that information.
This absence is especially problematic for agents that operate in domains requiring accountability: medical, legal, financial, or governmental applications. Without an audit trail, it is impossible to reconstruct the reasoning chain that led to a particular output, and it is impossible to determine whether the agent had access to information that should have changed its behavior.
In a context window, information is accessed by position. The model's attention mechanism computes relevance scores between query tokens and all tokens in the window, but this computation is bounded by the window and is sensitive to positional bias (as we discuss in Section V).
A memory system supports content-addressed retrieval: given a query, the system returns entries that are semantically related to the query, regardless of when or where they were stored. This is the fundamental operation that distinguishes memory from mere presence. A book on a shelf is present, but it is not remembered. A book that can be found by asking "what did the author say about X?" is retrieved by content.
Content-addressed retrieval requires an indexing structure (dense vectors, sparse indices, or hybrid systems) that is maintained independently of the model's context window. The context window can serve as the display surface for retrieved content, but it cannot serve as the retrieval mechanism itself, because it cannot index content that is not currently loaded.
Liu et al. [5] demonstrated that language models exhibit a U-shaped performance curve when relevant information is placed at different positions within a long context. Performance is highest when the relevant information appears at the beginning (primacy bias) or end (recency bias) of the context, and degrades significantly when it appears in the middle.
The standard interpretation of this result is that models have a positional attention bias. This is correct but incomplete. The deeper implication is that presence in a context window does not guarantee availability for recall.
It is tempting to model the context window as a set: a token is either in the set or not, and if it is in the set, it is available. This model is wrong. The effective availability of a token depends on its position, on the attention pattern of the specific model, and on the other tokens competing for attention.
Consider two tokens, t_i and t_j, both present in the context window. If t_i is at position 10 and t_j is at position n - 10, and the model exhibits the U-shaped attention bias documented by Liu et al. [5], then t_j will receive substantially more attention than t_i, even though both are "present."
This means that the effective capacity of a context window is not n tokens but some function f(n) that depends on position, model architecture, and attention pattern. The function f(n) is not known a priori and varies across models and tasks.
The lost-in-the-middle phenomenon reveals a conflation between two distinct concepts. Salience refers to how much attention a token attracts, given the model's attention pattern. Availability refers to whether the token can, in principle, be used to produce a correct output.
In a well-functioning memory system, these are decoupled. A memory entry that was stored long ago may not be salient (it is not frequently accessed), but it is fully available (it can be retrieved given the right query). In a context window, salience and availability are tightly coupled. A token in the middle of the context is both less salient and effectively less available, because the model's attention mechanism fails to attend to it with sufficient weight.
This analysis suggests that scaling context windows does not yield proportional gains in effective information capacity. A window of size 2n does not provide twice the usable information of a window of size n, because the attention bias that degrades middle-position information operates at all scales. The result is diminishing returns: each doubling of context length provides less marginal utility than the last.
Hsieh et al. [15] corroborate this observation with their RULER benchmark, showing that many long-context models fail to effectively utilize their claimed context lengths on tasks requiring retrieval from the middle of the input. The gap between nominal context length and effective usable context length widens as contexts grow longer.
We propose a formal separation between two memory systems that agents require, drawing on established distinctions in cognitive science.
Definition 3 (Working Memory). Working memory is a high-bandwidth, volatile store that provides the model with access to information during a single inference pass. Its capacity is bounded by the context window size n. Its temporal extent is bounded by a single session.
Definition 4 (Durable Memory). Durable memory is a persistent, structured, queryable store that maintains information across sessions. Its capacity is bounded by available storage. Its temporal extent is bounded only by the system's operational lifetime.
The properties of each system are as follows:
Working Memory requires: - High bandwidth (all tokens accessible in parallel via attention) - Positional encoding (order matters for sequential reasoning) - Transience (the window is rebuilt each session)
Durable Memory requires: - Persistence (entries survive across sessions) - Addressability (entries have stable identities) - Modifiability (entries can be revised or deleted) - Typed structure (entries can be classified by kind) - Queryability (entries can be retrieved by content, not position)
These two systems are not competing alternatives. They serve different computational roles and must be composed, not conflated. The context window is the scratchpad. The durable memory is the notebook. A system that uses only a scratchpad forgets everything when the session ends. A system that uses only a notebook cannot perform the rapid, parallel reasoning that attention over a context window enables.
The MemGPT architecture [13] makes a similar observation, drawing an analogy to virtual memory in operating systems. The inner context is analogous to physical memory (fast, limited), and the external store is analogous to disk (slower, larger). We extend this analogy by noting that the properties required of the "disk" layer are substantially more complex than those of a simple key-value store. The external memory must support structured queries, typed entries, revision histories, and content-addressed retrieval, none of which are provided by the context window mechanism.
The formal separation between working memory and durable memory has direct consequences for how agents should be designed.
The context window should be used for information that is: - Needed for the current reasoning chain (immediate relevance) - Ephemeral (no value beyond the current session) - High-bandwidth (must be attended to in parallel)
Examples include the current conversation, documents being analyzed in the current session, and intermediate reasoning steps. The context window excels at these tasks because attention provides O(1) access to any token in the window, enabling the rapid cross-referencing that complex reasoning requires.