Ashita Orbis

Context Contamination: How a Coding Harness De-Blinded a Blind Evaluation

Ashita Orbis|August 1, 2026|8 min read

Audit conducted: 2026-06-09 · Published: 2026-08-01 · Status: confirmed empirically; forward fix deployed; no retroactive re-runs

Correction (August 2026): One severity rating in the blast-radius table was over-rated against this audit's own reasoning. psycheeval v0.2/v0.3 Opus judging was graded High, while the structurally identical benchmark row — a contaminated judge scoring other people's profiles — was graded Low–moderate, "a bias vector, not an answer key." psycheeval judges only synthetic personas — pure-synthetic seeds plus fictionalized public-figure-inspired ones the seed bank labels "not the real person" — none of which is the subject, and it embeds the subject's profile in no prompt or dataset. There is no ground truth about the subject in these runs for the leaked scores to anchor to. The row now reads Low–moderate, and the implications section says why. The pre-correction wording is preserved in this document's public version history. Nothing else in the audit changes: the contamination happened, the mechanism is as described, and the narrative-scoring row remains High.

This is the source document behind the context-contamination caveat in Convergent Personality Assessment. That paper's Section 5 summarizes the finding in a paragraph and annotates the results it touches. This is the audit itself: what was checked, what the canary returned, which run classes were affected and how badly, and what the forward protocol actually enforces.

It is published because the failure mode is not specific to this project. Any evaluation that shells out to an agentic CLI and calls the result a blind model call inherits it, and the detector is four lines of shell.

The one-sentence version

Every evaluative claude -p invocation made by the project before 2026-06-09 ran non-blind: the Claude Code harness silently injected the subject's psychometric profile — including ground-truth trait scores — into the model's context by loading the operator's global memory file. Evaluations designed as blind personality inference therefore had the answer key in context while scoring.

The codex exec (GPT) backend and the OpenAI-SDK paths were never affected, so an uncontaminated comparison arm exists in the historical data.

Mechanism

  1. claude -p is not a raw model call. By default it boots the full Claude Code harness, which loads the global user memory file into context — the same as an interactive session.
  2. Since at least 2026-02-27 (the earliest snapshot in the config git history; the import predates the repo), that global memory file has contained a "User Profile" section importing the subject's full behavioral specification, which after 2026-03-09 includes explicit trait scores, score discrepancies, and interpretive findings.
  3. None of the project's CLI call sites passed an isolation flag:
    • the corpus analysis script — --system-prompt only, which replaces the system prompt and does not suppress memory injection
    • the narrative analysis script — same pattern
    • the psycheeval harness — a completely bare claude -p --model opus
    • the benchmark runner — bare invocation for both candidate generation and judging

Point 3 is the part worth transferring. --system-prompt looks like isolation and is not. Replacing the system prompt says nothing about what else the harness appends, and the memory file arrives through a different door.

Empirical verification

The finding was not inferred from documentation; it was verified with a canary:

  • A canary call through the exact pre-fix invocation pattern (project working directory, --system-prompt set) answered YES to "do you have the subject's trait scores in context?" and quoted the profile's interpersonal-pattern passage and the Agreeableness self-report/corpus discrepancy figures verbatim.
  • The same canary through claude -p --safe-mode from a neutral working directory answered NO — residual context was the operator email and the current date, neither of which carries trait information.
  • The negative control is retained as a permanent test: the canary must FAIL through a non-isolated call and PASS through the isolated path. Both were confirmed, and a canary that stops failing on the non-isolated path means the detector itself has broken.

Working directory matters independently of the flag: project-local memory files load too.

Blast radius

| Run class | Date(s) | Contamination | Severity | |---|---|---|---| | Canonical corpus analysis (provenance claude-opus-via-claude-cli) | 2026-03-04 | Profile-derived preference bullets; the full spec import was likely dangling (the behavioral spec was first generated 2026-03-09) | Mild — trait-suggestive characterizations, no scores | | Narrative scoring runs, incl. the 24-run Opus batch and the 1M-context experiment evaluations | post-2026-03-09 | Full profile incl. exact ground-truth Big Five scores | High — an evaluator scoring narratives against ground truth could see the ground truth | | psycheeval v0.2/v0.3 Opus judging | 2026-05/06 | Full profile of the subject, while judging synthetic personas, none of them the subject | Low–moderate — a bias vector, not an answer key (corrected August 2026; see the note above) | | Benchmark Opus judge + Opus candidate generation (mostly synthetic profiles) | 2026-03 | Full profile of the subject, while judging/generating for other profiles | Low–moderate — a bias vector, not an answer key | | Behavioral-spec regeneration, narrative report generation | various | Profile in context | None — non-blind by design; the profile is their explicit input | | All codex exec runs; all OpenAI-SDK calls | all | None — these paths load no operator memory | Clean |

The bottom two rows are the reason the audit is a caveat rather than a retraction: one family of runs was never contaminated, and it is the family the paper leans on where blindness is load-bearing.

Implications for existing conclusions

  1. The 1M-context narrative experiment. The headline calibration finding — the new pipeline's Neuroticism estimate landing within ~4 points of ground truth against the old pipeline's ~24-point miss — is confounded by anchoring, because the evaluator could see that ground-truth value while scoring. A critical review of 2026-03-16 identified four confounds (architecture, model version, word count, evaluator–generator overlap) and missed this fifth. The clean GPT evaluation arm in the same experiment partially survives; any future use of the Opus numbers must carry the caveat.
  2. psycheeval v0.2/v0.3. Opus-judge results are non-blind; codex-judge results are clean. Cross-judge agreement statistics computed across the two arms mix a contaminated and a clean judge, which is a worse problem than either arm alone. What the Opus judge could see, however, was not an answer key: psycheeval scores only synthetic personas — pure-synthetic seeds and fictionalized public-figure-inspired ones, none of them the subject — and no psycheeval prompt or dataset embeds the subject's profile, so there is no ground truth about the subject in these runs for a leaked profile to anchor to. That makes it the same structural case as the benchmark row below it, and the severity was corrected to match in August 2026.
  3. Canonical corpus scores. Mildly affected: inferred with profile-derived preference text (not scores) in context. Treat the published LLM-method scores as carrying a small unquantified anchoring risk.
  4. The research paper. The methods section's blindness claim needed a correction note wherever it covers CLI-backed runs. That note shipped as the paper's v6 revision.

Remediation decision

No retroactive re-runs. A planned re-derivation under verified isolation supersedes them — it re-derives corpus scores with multiple runs for error bars, which is strictly better than re-running the contaminated design. This audit plus call-site comments serve as the permanent record, and the historical results remain archived untouched under the project's archive-don't-delete convention.

The alternative — quietly re-running and replacing the numbers — would have destroyed the evidence that the contamination happened at all.

Forward protocol

  • A new isolation module provides call_claude_isolated() (--safe-mode plus a neutral working directory), a verify_isolation() canary, and an isolation_provenance() fragment stamped into output JSON. (--bare is not usable here: it refuses subscription OAuth.)
  • Every CLI-backed evaluation call site was patched to the isolated invocation: corpus analysis, narrative analysis, the psycheeval harness, and both judge call sites in the benchmark runner.
  • Two call sites are intentionally not isolated — the narrative generator and the behavioral-spec regenerator — because the profile is their explicit input. Isolation is not a blanket good; it is correct only where blindness is claimed.
  • Every isolated batch must be preceded by one verify_isolation() call, and every output JSON from an isolated run must embed isolation_provenance(). Provenance in the artifact is what makes the claim auditable a year later, when nobody remembers which invocation pattern produced which file.

What transfers

Three things generalize past this project:

  1. An agentic CLI is not a model API. If your evaluation harness shells out to one, the harness's context-assembly behavior is part of your method section whether you wrote it down or not.
  2. Test blindness, don't assert it. A canary that asks the model what it can see costs one call per batch and is the only thing that would have caught this. The design reviews did not catch it; four confounds were enumerated by a critical review that missed the fifth because it was reasoning about the experiment rather than about the plumbing.
  3. Keep an uncontaminated arm. The only reason this audit narrows the damage instead of voiding the dataset is that a second vendor's path never loaded the operator's memory. Multi-provider evaluation is usually justified by measuring family preference; here it paid off as contamination insurance.