Where to Spend Your Context Window

Correction (August 2026): Two descriptions of the Psyche merged benchmark in this post were wrong about what goes into it. It was described as synthesized from instruments plus an interview "rather than Opus corpus inference," when the merged profile in fact includes that corpus inference as one of three weighted channels (self-report 0.47, corpus inference 0.27, interview 0.27). Calling it "an independent accuracy test" was the same error in milder form. Both sentences now name all three channels and say that the instrument channel carries the largest weight. The instrument count has also been repaired: 39 is the full battery, while 16 were administered at this profile's standard tier. The pre-correction wording is kept in the site's private revision archive; superseded text is not republished.

Correction (October 2026): The description of this post said that "planning context is the dominant factor in output quality, with output length as a secondary driver", and its results section gave the ranking "long (1.7) < filtered (2.4) < 1M (3.9) < old (11.4)". The old and 1M scores in that ranking are each narrative's first Opus evaluation, while the other two are means of four runs. Scored as the mean of four Opus evaluation runs throughout, the conditions read old 8.4, 1M 2.3, filtered 2.4 and long 1.7, and every figure derived from them is restated on that basis. The three conditions built on planning artifacts from the full archive still outperform the old pipeline, by 6.0 to 6.7 points. They also changed the planning architecture, and the runs may differ in Opus version, so the gap is no longer credited to planning context alone. On the same basis three claims are withdrawn: that output length is a secondary factor, that fuller writing context helps the generator, and that a regenerated narrative scored identically to the original. The post named the model as "Opus 4.6", but the run records name only a floating alias, so the post now claims no single version. The sample was called "a single 8-month messaging arc". That span is the author's own description of the archive, not a figure from the experiment's reports, and the post now attributes it to him. Other corrections: the length ablation's target (about 24,000 words, where the post gave 28,000), the separate results for SMS and academic writing, the scope of the generator equivalence and of the calibration result, and a disclosure that the measured subject is the author. The merged profile's Openness value is removed, because the experiment records withhold merged profile values, and so is a correlation coefficient (r = 0.966) that no experiment record carries. The August note above no longer carries its sentence that restated "planning-context dominance" and estimated that its correction "moves the reported deltas by about 2.2 points on average", a calculation not found in any record.

The assumption that giving a language model more context during generation improves output quality is widespread enough to function as an axiom in most pipeline designs. The reasoning is straightforward: more information available at the point of generation means better informed generation. Retrieval augmented generation and the steady expansion of context windows, from a few thousand tokens to 128,000 and then to a million, both rest on the same premise, which is that the bottleneck is how much the model can see while producing its output.

An ablation study on a narrative generation pipeline suggests that this premise is incomplete in a specific and measurable way. The largest difference in output quality lay between the conditions that planned from the full archive and the one that did not. Averaged over four evaluation runs each, the condition that planned from limited context scored 6.0 to 6.7 points worse than each of the three that used planning artifacts derived from the full archive, and those three differed from one another by 0.7 points at most. That comparison cannot separate planning context from the change in planning architecture that came with it. The smaller differences among the three do not distinguish output length from writing context, and regenerating one of the narratives from identical inputs moved its score by more than any gap between them.

The Pipeline and the Problem

The experimental context is a narrative generation pipeline that converts a text message archive into first person literary memoir. The personality being measured is the author's own (N = 1): the archive is one messaging arc between the author and one other person, who is not identified here, and both benchmarks are profiles of the author. One is an Opus corpus reference profile, Opus's own Big Five inference from the author's writing corpus, which measures internal model consistency. The other is a Psyche merged profile, synthesized from the 16 validated instruments administered at the standard tier, an interview and the corpus inference, so that it is grounded partly in instruments.

The original pipeline operated within a 100,000 to 150,000 token context window. For each chapter of the memoir, an LLM generated a research brief: a curated list of 5 to 15 messages along with a description of the chapter's emotional arc. A separate writing agent produced the chapter from that brief plus approximately 500 filtered messages. The model never saw the full message archive at once. It saw summaries of summaries, each layer of abstraction compressing the source signal and potentially amplifying distortions in the process.

The full archive is 19,867 messages at roughly 534,000 tokens, and the author describes it as an 8 month life arc. When the context window expanded to a million tokens, the whole archive fit in a single context, and the pipeline was rebuilt around that fact. A discovery agent read all messages and produced emotional phase maps, canonical facts and character notes; an outline agent designed the chapter structure with the whole archive in view; and each chapter writer received all messages plus the discovery artifacts, the outline and all prior chapters. The key architectural change was that every agent in the pipeline could see everything. Claude Opus wrote the narratives of both pipelines and scored all of them, but its version was not held constant in any recorded way. The evaluation run records name only the command line's floating opus alias, which the experiment report's April 2026 addendum says resolved to Opus 4.5, 4.6 and 4.7 at different points, so this post claims no single version.

The personality metric is mean absolute delta (mean |Δ|): the average, over the five Big Five domains, of the distance between the score inferred from the narrative and the score in the Opus corpus reference profile, where lower is better. Each narrative was scored four times by the Opus evaluator (the filtered narrative has a fifth, earlier evaluation), and the text gives every condition's score as the mean of four runs. On that basis the old pipeline scored 8.4 and the 1M pipeline 2.3, a reduction of 72 percent. About 70 percent of that reduction is in Neuroticism: averaged over its four runs, the old pipeline read Neuroticism at 74.9 against the reference's 51.1, where the 1M pipeline read it at 53.8. Agreeableness and Conscientiousness account for most of the rest (about 15 and 13 percent), while Extraversion and Openness changed little, which fits the hypothesis that Neuroticism is the domain most sensitive to which messages the model sees. The first of the old pipeline's four evaluations, whose score the results table carries, also put Agreeableness 19.3 points above the reference, a reading the replication record treats as an outlier.

The Ablation Design

Ablation is a method borrowed into machine learning from neuroscience and early cognitive science, where it meant the surgical removal of a brain region to learn which function depended on that region. The method removes or disables one component at a time and measures whether performance changes: a drop means the component was contributing, and no change means it was redundant or compensated for elsewhere.

Reviews of the first results argued that the 1M pipeline's improvement over the old pipeline confounded several variables at once. The 1M pipeline used a different planning architecture, with discovery agents reading all messages where the old pipeline had curated briefs, and it produced shorter output (20,000 words against 24,000). The same model family also evaluated both pipelines, so Opus might have been scoring its own writing favorably. Two ablations and two validation analyses were designed in response to those reviews.

The context restriction ablation asked whether restricting the chapter writer to approximately 200 messages per chapter, which simulates the old pipeline's restricted access, would reproduce the old pipeline's personality inflation. It kept the 1M pipeline's outline and discovery artifacts, which had been created with full archive visibility, and gave each chapter writer a random subsample of 200 messages from its time period, drawn with a fixed seed for reproducibility. If context during writing were the driver, this version should have scored closer to the old pipeline than to the 1M pipeline.

The length ablation asked whether the improvement was partly an artifact of shorter text having less contradictory surface area. It kept the 1M pipeline architecture but instructed the chapter writers to produce longer chapters, 1,800 to 2,200 words each, targeting about 24,000 words in total (the old pipeline's length); the run produced about 28,000. Each chapter writer also received every message from its chapter's time range instead of the full archive, so this condition changed writing context as well as length. If brevity were the confound, the longer version should have scored worse than the 1M pipeline.

Confidence intervals addressed measurement noise. The Opus evaluation was run four times on each narrative, and those repeat evaluations of the same generated text bound the evaluator's scoring variance but not the variance of generation itself, which a later test measured by regenerating the 1M narrative from identical inputs.

An evaluation by a second model, GPT-5.4 run through Codex, addressed the concern that Opus was scoring its own writing. It had two parts: a factorial evaluation of the author's corpus (5 source registers × 2 models × 3 runs = 30 chunked runs, plus 9 Opus runs that read a whole source in a single context, 39 in total) to quantify systematic evaluator biases, and a 2×2 generation confound experiment to test whether the generating model affects the personality signal.

The Results

The table below is kept exactly as first published. Its rows do not share a run basis, and the text below it explains the difference and restates all four conditions on one basis.

*The filtered condition produced 32,000 words despite no explicit instruction to write more, so restricted writing context may prompt more verbose narration; it also means that output length differs between the filtered and 1M conditions.

Every narrative was evaluated four times, as the Runs column says, but the old and 1M rows show the first of those evaluations (11.4 and 3.9) beside intervals centred on those first values, and the old row's interval appears in no experiment record; the filtered and long rows show the mean of four. Recomputed from the unrounded runs on one basis, the mean of the four Opus evaluation runs recorded for each narrative, with a normal approximation of 1.96 standard errors (the rule the replication record used), the conditions read old 8.4 [6.4, 10.4], 1M 2.3 [1.3, 3.4], filtered 2.4 [2.1, 2.8] and long 1.7 [1.4, 1.9]. The filtered interval deserves some caution, because the first two filtered runs carry identical scores in every domain and the original single evaluation of that narrative (1.8) is not among its four runs; the long record's first run, by contrast, is the original single evaluation of the long narrative.

On that basis the ranking under Opus evaluation is long (1.7), then 1M (2.3) and filtered (2.4), then old (8.4). The old pipeline scores 6.0 to 6.7 points worse than each of the three new conditions, which share planning artifacts derived from the 1M pipeline, and that gap is the clearest finding. Among the new conditions the 1M interval runs from 1.3 to 3.4 and contains the other two. Under the normal rule long and filtered separate narrowly, but with the t distribution that four runs call for, which the experiment's own interval code uses, their intervals overlap as well, while the old pipeline's interval still lies above all three.

The intervals for Neuroticism, the domain with the largest single improvement, did not overlap at 95% confidence: old pipeline N = [72.4, 77.4] against 1M pipeline N = [52.5, 55.2], with a reference profile of 51.1. The Agreeableness and Conscientiousness intervals overlapped, so this sample of four runs does not separate the two conditions in those domains.

The factorial corpus evaluation found systematic biases between the two evaluators that are consistent across all five source registers (personal SMS, academic writing, casual messaging, AI conversations and a stratified mix). GPT-5.4 scores higher than Opus on every domain except Openness (E +15.3, C +19.2, A +10.6, N +8.3), and Openness is the only domain where both models agree (mean bias -2.8), consistent with O being the strongest textual signal. The Conscientiousness inflation does the most damage to comparisons between the models, since GPT applies +17.3 to +22.8 points of it across source registers (mean +19.2), which dominates aggregate metrics and flattens the differences between conditions. Both evaluators agree that the old pipeline shows the largest Neuroticism inflation and that every condition derived from the 1M pipeline reduces it, while the finer rankings of long, filtered and 1M remain specific to each evaluator.

The factorial design also tested the evaluator itself. Opus inferred personality worse when it read a whole source in a single context than when it read the same source in chunks, on all three sources tested (chunked against single context, scored against the merged profile: academic 5.8 against 15.8, mixed 8.3 against 12.0, SMS 8.3 against 9.1 mean |Δ|). The later tests found different mechanisms in different registers. For personal SMS, shuffling the message order reproduced the same bias, which points to the volume of text as its cause, and an instruction to "weight mundane passages as heavily as emotionally intense passages" removed it, returning every domain to within about one point of the chunked baseline. For academic writing, the same instruction applied to the whole corpus in a single context did not lower the error, and it did not for the mixed register either. A later test found shuffling and the instruction about equal on the academic papers (5.3 and 5.7 against the Opus reference) and read that as a trajectory of increasing rigor that the evaluator builds from chronologically ordered papers. Both of its runs read the papers in chunks, though, and so scored close to an ordinary chunked reading (about 5.8), which leaves the instruction's benefit established for personal SMS alone.

The 2×2 generation confound experiment crossed the two models as generators and as evaluators, with GPT-5.4 writing a parallel narrative from the same pipeline inputs (outlines, chapter briefs, voice profiles and canonical facts). With 2 or 3 runs per cell, the evaluator effect (12.4 points of mean |Δ| between conditions that share a generator) was nearly 7 times the generator effect (1.8 points between conditions that share an evaluator). Under Opus evaluation the two generators' aggregate error against the merged profile differed by about 0.1 points, and a formal equivalence test places the mean generator effect inside a ±3 point region of practical equivalence, so aggregate profile error was practically equivalent across generators. The profiles still differ in one domain, since the two cells' Agreeableness intervals under Opus evaluation do not overlap.

Against the Psyche merged profile, which blends self-report instruments, the interview and corpus inference with the instrument channel carrying the largest weight, the main finding holds on the same basis of four runs: old 9.7, 1M 7.8, filtered 7.0 and long 8.1. The order of the new conditions changes against this benchmark, with filtered ahead of the 1M pipeline and long behind it, which is one more reason to treat their finer differences as open. Every condition reads Openness about 11 to 13 points above the merged profile, likely a structural artifact of narrative generation (literary memoir is inherently text about ideas), and the Neuroticism correction from the old pipeline to the 1M pipeline remains the clearest signal against this benchmark too.

What the Ablation Reveals

The first single-run results told a clean story: the filtered and long narratives both scored 1.8 against 3.9 for the 1M pipeline, as if planning were everything and writing context irrelevant or even harmful. Replication changed that story twice. Four runs for each ablation condition put filtered at 2.4 and long at 1.7, which refuted the idea that restricting writing context helps. Scoring the old and 1M narratives on the same basis changes the story again, because the 1M narrative's own mean over four runs is 2.3: most of the gap that once separated the 1M pipeline from both ablations was the 1M narrative's first evaluation.

What all three new conditions share, and the old pipeline lacks, is outline and discovery artifacts from an agent that could see all 19,867 messages. The mechanism the study proposes is compounding abstraction in the old pipeline's briefs, which an LLM wrote from limited context: a person who occasionally expressed anxiety can become "a person characterized by anxiety" in a brief, and then a chapter whose dominant emotional register is anxiety. The 1M pipeline collapsed those layers by giving the planner everything, so that on this account the planner could calibrate emotional peaks against the full behavioral baseline.

In the reviews of the first results, GPT-5.4 and Opus argued that the improvement over the old pipeline could come from the change in planning architecture, or possibly from a change in Opus version, rather than from the larger context window itself. The author doubted that argument, because the pipeline had been changed in order to use the million token window. He reasoned that without those changes the larger window would have made no difference, and that, since he understood the same Opus model to have run both pipelines, the window was the one thing that changed and the redesign followed from it. These runs cannot settle the disagreement. No condition kept the discovery architecture while restricting what the planners could see, and the experiment's records do not establish which Opus version ran either pipeline, so they can neither confirm nor rule out his premise about the model.

Output length seemed to separate the 1M pipeline from both ablations while the 1M narrative was scored by its first evaluation alone, since that narrative ran to about 20,000 words against about 32,000 for filtered and 28,000 for long. On one basis, though, the filtered narrative, the longest of the three, scored no better than the 1M narrative (2.4 against 2.3), so these runs do not show that longer narratives score better. The long narrative scored best (1.7), but its chapter writers saw only the messages in their chapter's time range, so the length ablation changed writing context and length together and cannot isolate either. The old pipeline, at 24,000 words, still scored far worse than the 1M pipeline at 20,000 (8.4 against 2.3), so output length alone does not look like the explanation for the large gap, though these runs cannot rule it out.

Writing context gives a similarly mixed answer. Restricting each chapter writer to 200 messages left the score where the 1M pipeline's was (2.4 against 2.3), so restricted writing context gave no measurable gain and the hypothesis that full writing context adds noise finds no support. The filtered condition did score worse than the long condition (2.4 against 1.7), whose writers saw every message in their chapter's time range and produced fewer words. That is the only comparison in which fuller writing context comes out ahead, and because the two conditions also differ in length and in how their messages were chosen, it does not show how much writing context is worth.

The regeneration test puts a scale on all of these differences. A second 1M narrative, generated from identical inputs and scored over four Opus runs, came out at 3.9 [3.3, 4.4], against 2.3 for the original narrative's mean over four runs, a gap more than twice the largest difference among the three new conditions (0.7). On this post's basis, regeneration moved the five domain scores by 3.2 points on average (the record's own figure of 4.9 compares the regenerated narrative with the original's first evaluation), more than three times the evaluator's scatter between repeat readings, which that record puts at about 0.9 points. Differences of a point or less between single generations of the new conditions are therefore within what regenerating one of them can produce.

Generalization: Where Context Matters Most

The specific domain (personality signal preservation in narrative generation) is narrow, and whether its architectural observations transfer is a question this study cannot answer. The structural question does apply to any multistep LLM pipeline with distinct planning and generation phases, whether for document generation, code production, research synthesis or agentic task execution: where in the pipeline does context quality matter most?

The standard answer is to maximize context at the point of generation. Retrieval augmented generation retrieves at generation time, and long context models are marketed on what they can hold while generating. This study points toward a different answer without settling it: every variant built on a plan created from full context outperformed the pipeline that planned from limited context, while the variants differed from one another by 0.7 points at most. The comparison with the old pipeline cannot separate planning context from the accompanying change in planning architecture. Whether the generator also benefits from full context, and whether longer output helps, this study leaves open.

This framing maps onto a distinction familiar in other engineering domains between design time information and execution time information: an architect who has surveyed the entire site before drawing plans produces better buildings than one who surveys each room during construction. The analogy has limits, since an LLM writer with full context can calibrate locally in ways a construction worker following blueprints cannot, and this study could not measure what that calibration is worth. It also assumes that the plan is complete before generation begins, and pipelines with feedback loops between planning and generation may distribute the value of context differently.

The domain has specific properties (personality signal as a measurable outcome against a partly instrument-grounded reference profile, and a clear boundary between the planning and writing phases) that may not transfer to domains where that boundary is less clear.

Limitations

The sample is a single messaging arc of 19,867 messages between two people, and the measured subject is the author, so N = 1. Every narrative compared here was generated from that one archive, which the author describes as an 8 month life arc. The gap between the old and new pipelines may not hold for relationships with different emotional textures, different messaging densities or different temporal structures.

The evaluation metric (Big Five personality inference from generated text) is specific to this pipeline, and whether these patterns hold for other measures of generation quality, such as factual accuracy or narrative coherence, remains an open question. Where a longer narrative did score better, the difference may partly reflect the evaluator having more text to work with rather than the narrative carrying better personality signal.

The old pipeline and the 1M pipeline differ in more than context window size. The old pipeline used a fundamentally different planning architecture (sequential brief generation where the 1M pipeline used holistic discovery), which means the ablations isolate context restriction during writing but not during planning. A cleaner test would restrict planning context to 100,000 tokens while keeping the discovery agent architecture, which would separate the effect of context size from the effect of architectural differences in the planning phase.

The measurement has weak points of its own. Each condition was evaluated four times on one generated narrative, so the intervals bound the evaluator's variance and not the variance of generation, which the regeneration test showed to be the larger of the two. The Opus release behind each evaluation run was not recorded, only the alias that called it. GPT-5.4's own evaluations of the four narratives confirmed the Neuroticism finding but not the finer rankings of long, filtered and 1M, which remain observations about Opus.

What Holds and What Remains Open

The old pipeline's Neuroticism inflation, and its reduction under planning artifacts derived from the full archive, appear for both evaluators and against both benchmarks. The generator test shows that the inflation reading does not depend on which model wrote the narrative, with Neuroticism at 75.7 for the Opus narrative and 75.1 for the GPT-5.4 narrative under the Opus evaluator.

The concern that Opus recognized its own writing rather than genuine signal is substantially addressed for aggregate error, where the generator effect is negligible under equivalence testing. It is bounded rather than eliminated, since systematic biases may be shared across frontier models, and the two generators' profiles still differ in Agreeableness.

A temporal analysis of the message archive bears on the mechanism without settling it. Evaluating chronological slices of the archive separately moves the inferred profile: the last quarter reads Extraversion 6.4 points and Agreeableness 4.9 points above the whole archive, and the first quarter reads the highest Neuroticism of the slices. Those movements are much smaller than the old pipeline's inflation of Neuroticism by about 24 points against the reference. That is consistent with the reading that partial coverage of the archive does not by itself produce the old pipeline's distortion, and that the sequential brief architecture, compressing and amplifying emotional framing at each layer of abstraction, does. Calibrating one evaluator against the other has a similar status. Additive correction of GPT-5.4's scores by a fixed bias per domain reduced the average error, to a mean RMSE of 3.6 points on the register left out of each fit, but the residuals depend on domain and register: that error ranges from 1.4 on the mixed register to 6.7 on personal SMS, and single domains stay off by as much as 10.8 points, so some disagreements between the evaluators are structural rather than a matter of simple calibration.

The generalization to multistep LLM pipelines remains a hypothesis drawn from one domain. These runs leave open whether the generator in such a pipeline also needs full context, and how much of the gain belongs to planning context and how much to the change in planning architecture or to possible model-version effects. For anyone building a pipeline with distinct planning and generation phases, what this study observed is that every pipeline built on a plan created from full context outperformed the one that was not, by 6.0 to 6.7 points, while those pipelines differed from one another by 0.7 points at most.