Research

Benchmarking Structured Conversation Extraction Across Three Evaluation Layers: A Comparison of Eight Models with a Public Replication

Ashita Orbis|June 8, 2026|21 min read|working-paper

Evaluating structured extraction from conversational data with a single quality metric conceals the failure modes that matter most to systems built on top of the extraction. This paper presents a benchmark that evaluates extraction quality across three layers: field comparison against a reference extraction, evaluation by a panel of three judge models reading the raw conversation, and propagation of extraction differences into downstream consumers such as drift detection and loop tracking. Eight frontier models extracted a seventeen field schema from thirty chunks of private SMS conversation, with Claude Opus 4.6 serving as the reference. The extraction layer discriminates strongly (composites from 0.496 to 0.623 against a reference ceiling of 0.996), the judge layer discriminates weakly (0.869 to 0.942), and correlation between judge composites and downstream composites is near zero (mean Pearson r of 0.134), a result that initially appears to undermine the methodology but instead reflects an aggregation artifact: across the full cross correlation matrix, dimension pairs designed to correspond average Spearman rho 0.510 while the fifteen mismatched pairs average 0.051, although eight models leave all such estimates imprecise. One judge dimension resists validation entirely, and the investigation explains why: emotional field extraction saturates structurally, with every model producing specific mood labels, grounded turning points, and equivalent label diversity, leaving no structural signal for a downstream proxy to exploit. A reduced replication on the public Persuasion for Good corpus (reference plus three challengers, thirty chunks) reproduces the discrimination pattern and the top model ordering, with the middle ordering proving sensitive to corpus register and to the treatment of schema validation failures. Judge family preference for same family models measures 0.019 to 0.021 on a unit scale, constraining though not eliminating the circularity concern inherent in model referenced evaluation.

The Etymology Tax: Etymological Register Effects on LLM Multi-Step Reasoning

Ashita Orbis|March 6, 2026|42 min read|working-paper

This study tests whether the etymological register of input text affects large language model performance on multi-step reasoning tasks. Using 250 murder mystery narratives from the MuSR benchmark, eight models were evaluated across six conditions in a partially crossed design varying vocabulary register (baseline, Germanic, Latinate) and clarification, with explicit clarification applied only to the baseline register and full register translations introducing implicit clarification. Both Germanic and Latinate register translations significantly reduced accuracy relative to baseline English (Germanic: -2.5%, p=0.019; Latinate: -3.7%, p=0.003), while explicit clarification of ambiguous narratives produced no significant effect (+0.5%, p=0.67). A stochastic validation experiment using 80 target problems with three additional translation variants per problem revealed a critical asymmetry: accuracy reductions caused by register translation were robust across wording variants (2% flip rate from harmful to helpful), whereas accuracy improvements were fragile (20% flip rate from helpful to harmful). Full register translations exhibited significantly higher variability across translation variants than minimal translations (ANOVA F=2.97, p=0.038), with Latinate full translations showing the highest fragility (40% of problems classified as fragile) compared to Germanic minimal translations (0% fragile). Linguistic analysis identified pronoun count (d=1.39), sentence count (d=1.10), and causal reasoning markers (d=1.07) as significant predictors of translation sensitivity. The findings suggest that vocabulary register constitutes an uncontrolled variable in LLM benchmarking, that models are calibrated to natural English word distributions rather than to any particular register, and that the robustness asymmetry between harmful and helpful translation effects has implications for the reliability of prompt engineering strategies that depend on vocabulary choice.

Convergent AI-Mediated Personality Assessment: Psychometric Profiling and Narrative Inference from Digital Communication Data

Ashita Orbis|March 3, 2026|96 min read|working-paper

This paper examines the convergence between two AI-mediated approaches to personality assessment applied to a single participant: explicit psychometric measurement through seventeen validated instruments triangulated across three inference methods, and implicit personality inference through literary narrative generation from a 267MB text messaging archive. The explicit approach (Psyche) produced a ten-dimension persona model with dimensional scores and confidence intervals. The implicit approach (narrative pipeline) produced approximately 78,000 words of first-person literary memoir grounded in 679 cryptographic source citations with a 99.85% validation rate. Cross-method comparison across ten personality dimensions yielded five consistent, three evolved, two partially consistent, and zero divergent ratings, where 'evolved' denotes documented developmental change rather than measurement disagreement. Eleven analytical contributions emerge from the data: systematic self-enhancement patterns in per-method Big Five divergence, facet-level reliability differences between 4-item and 10-item NEO measurement, temporal behavioral markers operationalized as personality indicators, citation density gradients as measures of inferential distance from evidence, cross-arc attachment consistency as evidence for trait stability from behavioral rather than self-report data, quantitative signal preservation analysis showing that narrative generation preserves extreme personality traits (Openness, Extraversion) while amplifying Neuroticism through literary genre effects, communication medium effects demonstrating that the same person's inferred personality varies by up to 37 points depending on the text register analyzed, temporal personality trajectory analysis revealing a Neuroticism peak that aligns with the narrative pipeline's independently inferred emotional arc, the operationalization of a negative finding (lexical frequency analysis removed from personality synthesis and repositioned as comparative corpus characterization after demonstrating insensitivity to personality variation), and a differential personality experiment demonstrating that narrative perspective determines inferred personality profile, in which interlocutor first-person narratives produce profiles differing by 10.7-12.3 mean absolute points from the subject's baseline while third-person narratives about the subject stay within 3.7-5.6 points, ruling out model-invariant output as the primary explanation for signal preservation, and a cross-model evaluator analysis demonstrating that the choice of evaluator model shifts inferred profiles roughly seven times more than the choice of generator model (12.4 vs 1.8 points), with evaluator biases systematic enough to be partially calibrated. The findings suggest that digital communication data contains more psychological signal at the corpus level than individual messages suggest, that extraction method determines accessibility of this signal, and that the combination of psychometric snapshots with narrative trajectories captures dimensions neither approach reaches independently.