Benchmarking Structured Conversation Extraction Across Three Evaluation Layers: A Comparison of Eight Models with a Public Replication

Abstract

Evaluating structured extraction from conversational data with a single quality metric conceals the failure modes that matter most to systems built on top of the extraction. This paper presents a benchmark that evaluates extraction quality across three layers: field comparison against a reference extraction, evaluation by a panel of three judge models reading the raw conversation, and propagation of extraction differences into downstream consumers such as drift detection and loop tracking. Eight frontier models extracted a seventeen field schema from thirty chunks of private SMS conversation, with Claude Opus 4.6 serving as the reference. The extraction layer discriminates strongly (composites from 0.496 to 0.623 against a reference ceiling of 0.996), the judge layer discriminates weakly (0.869 to 0.942), and correlation between judge composites and downstream composites is near zero (mean Pearson r of 0.134), a result that initially appears to undermine the methodology but instead reflects an aggregation artifact: across the full cross correlation matrix, dimension pairs designed to correspond average Spearman rho 0.510 while the fifteen mismatched pairs average 0.051, although eight models leave all such estimates imprecise. One judge dimension resists validation entirely, and the investigation explains why: emotional field extraction saturates structurally, with every model producing specific mood labels, grounded turning points, and equivalent label diversity, leaving no structural signal for a downstream proxy to exploit. A reduced replication on the public Persuasion for Good corpus (reference plus three challengers, thirty chunks) reproduces the discrimination pattern and the top model ordering, with the middle ordering proving sensitive to corpus register and to the treatment of schema validation failures. Judge family preference for same family models measures 0.019 to 0.021 on a unit scale, constraining though not eliminating the circularity concern inherent in model referenced evaluation.

1. Introduction

1.1 The Evaluation Problem in Structured Extraction

Structured extraction from conversation is the task of converting informal dialogue into typed records: claims with polarity and attribution, action items with ownership, decisions with evidence chains, emotional states with intensity and grounding. Any frontier language model will produce output for this task that appears reasonable on inspection, which is precisely the difficulty, because surface plausibility provides no way to answer the operational question that motivates evaluation in the first place, namely whether substituting a cheaper model for an expensive one breaks the systems that consume the extraction. The standard answer, computing F1 against a gold standard, collapses a many dimensional comparison into one number and treats the reference as ground truth even when the reference is itself the output of a model, and it weights all errors equally even though downstream consumers are differentially sensitive: a polarity inversion on a single claim is invisible to fuzzy text matching yet catastrophic to a drift detector whose entire signal is the direction of claims over time.

This paper describes a benchmark built around the premise that extraction quality is not one property but at least three, and reports results from applying it to eight frontier models. The first layer measures fidelity to a reference extraction at the level of individual fields, with metrics matched to field type. The second layer removes the reference from the loop by asking judge models from three vendor families to evaluate each extraction against the raw conversation text. The third layer runs the full downstream pipeline (drift detection, loop tracking, arc construction, report generation) on each model's extractions and scores the resulting artifacts and per chunk tasks, in the final design directly against the conversation text rather than against reference artifacts. The layers disagree with one another in instructive ways, and a substantial portion of this paper concerns what the disagreements reveal about which measurements can be validated and which cannot.

1.2 Three Claims

The benchmark supports three claims that single metric evaluation cannot express. First, extraction errors can concentrate rather than distribute, clustering in the fields that particular consumers depend on, which is the failure mode the layered design exists to detect and which the divergence between layer rankings in Section 3.1 is consistent with, though concentration is demonstrated here at the level of layer disagreement rather than traced field by field. Second, judge evaluation and volume sensitive pipeline metrics measure different constructs, precision and recall respectively, such that correlating an aggregate of one with an aggregate of the other produces noise even when matched pairs of dimensions correlate well. Third, the methodology survives transfer to a public corpus, which matters because the primary corpus is private and a benchmark whose evidence cannot be examined invites justified suspicion.

1.3 Research Questions

Four questions organize the results. RQ1 asks whether the benchmark discriminates among models, since a benchmark on which all systems score identically measures nothing. RQ2 asks whether the judge layer is internally coherent (do judges agree with each other?) and externally valid (do judge scores predict downstream behavior?). RQ3 asks which judge dimensions can be validated against downstream proxies and which cannot, and why. RQ4 asks whether the discrimination pattern observed on the private corpus reproduces on a public one.

2. Method

2.1 Corpus and Chunking

The primary corpus consists of two long running dyadic SMS conversations, designated Conversation A and Conversation B, drawn from the author's personal message archive. Messages were normalized to a common record format (speaker, text, timestamp, thread identifier) and segmented into chunks by session day with a token ceiling, producing conversation units large enough to contain negotiation arcs and emotional trajectories but small enough for single call extraction. Thirty chunks constitute the benchmark set. The corpus is private and is not distributed; Section 2.7 describes the public replication that compensates for this, and Section 4.3 discusses the limitation directly.

2.2 Extraction Schema

The schema contains seventeen independently scored fields. Ten fields derive from the first schema generation: topics, entities, decisions, action items, open questions, claims (scored on text match, polarity, and subject attribution as separate components), links, warnings, evidence message identifiers, and calibrated confidence scores. Seven fields were added through a three phase screening pipeline that screened nineteen of twenty five candidate fields on consistency, information gain, discrimination, and redundancy across repeated extraction runs (six established first generation fields were exempt from screening): summary, question and answer pairs, emotional tone, emotional arc, conversation phases, implicit assumptions, and negotiation patterns. Every extraction call presents the same JSON schema and system prompt, and outputs are validated against the schema before storage.

2.3 Models

Eight models were benchmarked: Claude Opus 4.6 (the reference, evaluated against itself as a ceiling check), Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.2 at low reasoning effort, GPT-5.4 at xhigh reasoning effort, GPT-5.3-Codex at high reasoning effort, Gemini 3 Flash Preview, and Gemini 3.1 Pro Preview. Each model extracted all thirty chunks under the identical schema and prompt, through provider native structured output mechanisms where available.

2.4 Layer 1: Field Comparison Against the Reference

Text fields use token set ratio matching with a 0.60 threshold, chosen empirically as the point below which false matches proliferate and above which legitimate paraphrases are rejected. Claims receive compound scoring: sixty percent text F1, twenty five percent polarity accuracy on matched claims, fifteen percent subject consistency, a weighting that reflects the downstream sensitivity of drift detection to polarity. Entities use lenient name matching (0.70 threshold) with exact type matching. Confidence calibration uses Expected Calibration Error (Naeini, Cooper, and Hauskrecht, 2015) across ten bins, matching the experimental setting of the original paper, and converted by this benchmark to a score on which higher is better. The seven newer fields use scorers designed per field; the emotional arc scorer, for example, compares opening mood, closing mood, trajectory, and turning points as four equally weighted components. Field scores combine into a weighted composite in which claims carry the largest weight (0.15) and the newer fields carry weights between 0.04 and 0.08 derived from their screening scores.

2.5 Layer 2: Judge Panel

Three judge models, one from each vendor family represented among the challengers (Claude Opus 4.6, GPT-5.4 at xhigh effort, Gemini 3.1 Pro), evaluated every extraction against the raw conversation text without knowledge of which model produced it. Judges score three dimensions from 0 to 100 (content accuracy, completeness, and a hallucination check with an associated issue taxonomy) and, in the second judge protocol, five finer dimensions: claim grounding, question closure fidelity, entity canonicalization, emotional signal fidelity, and negotiation pattern accuracy. The deliberate inclusion of three families allows family preference, the tendency of a judge to reward outputs from its own lineage, to be measured rather than assumed (Zheng et al., 2023, document the general concern). All judges scored all thirty chunks for all eight models.

2.6 Layer 3: Downstream Propagation

The downstream layer originally compared pipeline artifacts produced from challenger extractions against artifacts produced from reference extractions, a design that left several metrics degenerate: report overlap scored 0.000 for every model, escalation agreement scored 1.000 for every model, and arc agreement scored 0.500 for every model, so that sixty percent of the downstream tasks discriminated nothing. The reworked design evaluates without reference artifacts, scoring six tasks per chunk (entity coverage, question and answer completeness, emotional fidelity, topic and claim coverage, drift quality, loop quality) directly against the conversation text, and adds artifact level scores for drift, loops, arcs, and a negotiation richness measure designed to approximate precision through selectivity, specificity, and calibration proxies rather than through volume. Under the reworked design every task discriminates, the score spread widens from 0.09 to 0.133, and the three scoring modes (multi reference, single reference, and reference free) agree on model ranking with Kendall tau between 0.714 and 1.000, indicating that the reference free simplification largely preserves the ordering, with tau 0.714 still permitting individual pairwise inversions.

2.7 Public Replication Design

The replication uses Persuasion for Good (Wang et al., 2019), a corpus of dyadic donation persuasion dialogues collected through crowdworking and available through ConvoKit (Chang et al., 2020), selected after a coverage probe established that its conversations exercise thirteen of the fifteen content bearing schema fields (fifteen rather than seventeen because evidence identifiers and confidence scores attach to other fields rather than carrying content of their own), including every field that distinguishes this schema from generic extraction (negotiation patterns, implicit assumptions, emotional arcs, claims with polarity); the two exceptions are links, which spoken style dialogue does not produce, and open questions, which the declarative register of donation dialogues rarely generates. Thirty conversations of at least twelve utterances were selected by deterministic stride across the 1,016 eligible conversations to avoid ordering bias. The reference model is pinned to the same Claude Opus 4.6 version as the private benchmark, a detail that matters because the unqualified model alias had begun resolving to a newer Opus generation by the time of the replication, and an unpinned run would have silently changed the reference. Three challengers represent the strong, middle, and cheap tiers: Claude Sonnet 4.6, Claude Haiku 4.5, and Gemini 3 Flash Preview. GPT-5.2, the original middle challenger, was retired from the provider's consumer access tier between the private benchmark and the replication and returned an unsupported model error on every call, so Claude Haiku 4.5, whose private corpus composite (0.558) sits nearest the retired model's (0.554), was substituted; the retirement itself is reported as a finding about the reproducibility half life of commercial model benchmarks.

3. Results

3.1 RQ1: Discrimination

The extraction layer separates the eight models cleanly. Against the reference ceiling of 0.996 (Opus 4.6 scored against its own stored extractions, with the residual arising almost entirely from the confidence calibration scorer, which is not an identity metric and scores 0.896 even on identical extractions, rather than from any field disagreement), Claude Sonnet 4.6 leads the challengers at 0.623, followed by Claude Haiku 4.5 at 0.558, GPT-5.2 at 0.554, GPT-5.4 xhigh at 0.531, GPT-5.3-Codex at 0.521, Gemini 3.1 Pro at 0.498, and Gemini 3 Flash at 0.496. The gap between the reference and the best challenger (0.373) exceeds the gap between the best and worst challengers (0.127), and per field results show the discrimination is not uniform: question and answer pairs compress into a narrow band (0.791 to 0.902) while negotiation patterns spread from 0.000 to 0.479, which is consistent with the interpretation that constrained output spaces are easy for every model and genuinely inferential fields are hard for most.

The judge layer discriminates far less. Judge composites for the same eight models span 0.869 to 0.942, a spread of 0.073 against the extraction layer's 0.500, and the per judge spreads (0.067 for the Gemini judge, 0.073 for the Opus judge, 0.085 for the GPT judge) are consistent across families. Judges reading raw conversations evidently find most frontier extractions acceptable most of the time, and the discrimination the benchmark achieves comes principally from the reference comparison and downstream layers.

The reworked downstream layer discriminates and, notably, reorders. Under reference free scoring, GPT-5.4 xhigh leads (0.479), followed by GPT-5.2 (0.474), GPT-5.3-Codex (0.468), Opus 4.6 (0.467), Sonnet 4.6 (0.427), Haiku 4.5 (0.394), Gemini 3.1 Pro (0.360), and Gemini 3 Flash (0.346). The extraction layer ranks Sonnet first among challengers while the downstream layer ranks the OpenAI models first, and the reference itself falls to fourth, which together indicate that proximity to the reference and quality of downstream artifacts are related but distinct properties, the central premise of the layered design.

3.2 RQ2: Judge Coherence and the Aggregation Artifact

Inter judge agreement is moderate. Spearman correlations on composite rankings across the eight models are 0.714 for the GPT and Opus pair, 0.667 for the Gemini and Opus pair, and 0.524 for the Gemini and GPT pair, averaging 0.635. Family preference is small: each judge scores models from its own family 0.019 to 0.021 higher on the unit scale than models from other families, an effect roughly a quarter to a third of the judge layer's own discrimination spread, which constrains the concern that a same family judge invalidates the protocol, though Section 4.1 returns to what this measurement does and does not establish.

External validity at the aggregate level fails. The Pearson correlation between each judge's composite and the downstream artifact composite across models is 0.287 for the GPT judge, 0.211 for the Opus judge, and negative 0.094 for the Gemini judge, averaging 0.134. Read naively, this number indicates that judge opinion is unrelated to downstream behavior, which would be fatal to a methodology whose third layer exists to capture consequences. The full cross correlation matrix reverses the reading. Crossing the five judge dimensions with the four artifact metrics yields twenty cells, of which five are matched in the sense that the judge dimension was designed to predict that artifact metric: the five matched pairs average Spearman rho 0.510 while the fifteen mismatched pairs average 0.051, and the contrast widens to 0.613 against 0.051 when the saturated emotional pair, the one matched exception that Section 3.3 explains, is set aside. Individually, claim grounding against drift quality reaches rho 0.738 (p equal to 0.04), question closure fidelity against loop quality reaches 0.667 (p equal to 0.07), negotiation pattern accuracy against the negotiation richness measure reaches 0.595 (p equal to 0.12), and entity canonicalization against arc breadth reaches 0.452 (p equal to 0.26); with eight models only the first clears the conventional significance threshold, so the evidential weight rests on the consistency of the matched against mismatched contrast rather than on any single cell, and the matrix contains honest irregularities, including one off diagonal cell (claim grounding against arc breadth, rho 0.833) that exceeds every matched pair, which is consistent with claim grounding functioning as a general quality signal rather than a drift specific one. The composite to composite correlation is low because it averages the matched signal with mismatched cells that the design gives no reason to expect to correlate, and aggregate correlation understates the validity of a multidimensional instrument whenever the dimensions measure different things, which is the instrument's reason for existing.

The negotiation case carries an additional methodological lesson. The original pipeline metric for negotiation, pattern density, correlated negatively with judge scores (the judges measure precision, and models that label casual banter as negotiation are penalized rather than rewarded for volume), so the validated 0.595 correlation was achieved only after the downstream measure was redesigned around precision proxies: topic specificity, selectivity with an optimum near 2.5 patterns per chunk, tactic diversity with a penalty at the ceiling, and confidence calibration. The general principle is that judge dimensions measure precision while naive pipeline metrics measure recall, and correlating across that mismatch produces signed noise. The redesign was undertaken after the negative correlation was observed, without a holdout, and the validated 0.595 must therefore be read with the post hoc caveat that metric fitting of this kind approaches the circularity this paper elsewhere declines; the emotional case in Section 3.3, where the same redesign was attempted and abandoned when no signal existed to fit, is the check that the negotiation redesign tracked a real mechanism rather than manufactured agreement.

3.3 RQ3: The Limit of Downstream Validation

The remaining weak pair, emotional signal fidelity against arc breadth at rho 0.095, was investigated with the same redesign intent and the investigation returned a negative result that is more informative than the redesign would have been. Eight candidate structural signals were computed per model from the emotional fields (tone density, emotion label specificity, mean and dispersion of confidence, intensity diversity, turning point grounding, arc trajectory variability, mood vocabulary diversity) and correlated against the judge dimension across twelve enrichment runs. Four of the eight signals are constant across every model: all models produce specific rather than generic emotion labels, all use the full intensity range, all ground every turning point in evidence identifiers, and all deploy the same mood vocabulary breadth. The non constant signals correlate at noise levels (the strongest, mean confidence, reaches rho 0.203 with p equal to 0.527). The judge dimension itself spans only 72.8 to 83.2 across models, the narrowest range of any judge dimension.

The conclusion is that emotional field extraction saturates structurally at the frontier: the schema's emotional fields are easy enough that every current model produces structurally equivalent output, differences the judges perceive are not expressed in any structural property a proxy can measure, and no downstream validation of this dimension is possible by these means. The benchmark reports emotional signal fidelity as an unvalidated judge dimension rather than engineering a proxy to manufacture agreement, on the reasoning that a correlation produced by fitting the measure to the judge would be circular in exactly the way Section 4.1 cautions against.

3.4 RQ4: Public Replication

On Persuasion for Good, scored against the pinned Opus 4.6 reference at the extraction layer, Sonnet 4.6 leads at 0.663, Gemini 3 Flash scores 0.633 over its twenty seven completed chunks, and Haiku 4.5 scores 0.597, against private corpus values of 0.623, 0.496, and 0.558 for the same models. Three features of this table support the replication claim and two complicate it. The benchmark discriminates on public data with a challenger spread of 0.066; the strongest challenger is the same on both corpora; and every model scores higher on the public corpus, consistent with its shorter and cleaner dialogues. The complications are that the middle ordering inverts (Haiku above Flash on private data, below it on public data), and that the inversion itself depends on the treatment of failures: Gemini 3 Flash failed schema validation on three of thirty chunks in both attempts, emitting enumeration values outside the schema (the string "small talk" for the defined token "small_talk", and an invented mood label), and scoring those failures as zero lowers its public composite to 0.570, below Haiku. Near tied models reordering across corpus registers is the expected behavior of a discriminating instrument near its resolution limit, but the failure sensitivity is a substantive finding: a model's effective extraction quality includes its reliability at producing schema valid output at all, and a composite that silently drops failed chunks overstates the quality of the least reliable model. Both treatments are therefore reported.

The replication also surfaced a reproducibility hazard external to the methodology. GPT-5.2, benchmarked three months before the replication, had been retired from consumer tier access by replication time, making one eighth of the original benchmark unreproducible at any price through the channel used to produce it. Commercial model benchmarks decay not only through model drift but through model disappearance, which strengthens the case that benchmark methodology, rather than any particular leaderboard, is the durable contribution.

4. Discussion

4.1 Circularity, Measured

Using Opus 4.6 as reference, judge, and contestant invites the objection that the benchmark measures similarity to one vendor's house style rather than quality. The objection cannot be dismissed, but it can be bounded, and it is properly two objections, each bounded by a different measurement. The judge circularity objection, that an Opus judge favors Opus lineage output, is bounded by the family preference measurement: 0.019 to 0.021 on the unit scale across three judge families, small relative to every reported effect. The reference circularity objection, that layer one merely rewards similarity to one vendor's house style, is bounded by the downstream layer, which involves no reference and no judge yet ranks the OpenAI models above the reference model itself, an ordering a benchmark that merely rewarded proximity to Opus could not produce. The cross mode agreement of the downstream rankings (Kendall tau from 0.714 to 1.000 between reference based and reference free scoring) adds a robustness check that the orderings are not artifacts of any single reference choice. What the measurements do not establish is the absolute quality of the reference: a field where the reference extracts poorly poisons layer one for every challenger equally, and only human annotation, planned for fifteen to twenty chunks, can calibrate that residual.

4.2 What the Saturation Finding Implies

The emotional saturation result generalizes beyond this benchmark, with a scope worth stating precisely: what saturates is structural discriminability, not necessarily the capability itself, since the judges still perceive a quality range (72.8 to 83.2) that no structural property expresses. Validation of a judge dimension through downstream proxies requires that models differ structurally on the dimension, so as frontier models converge structurally on a capability, that dimension becomes unvalidatable by between model correlation even while remaining measurable in an absolute sense, and evaluation effort should migrate toward the dimensions where models still differ (here, negotiation patterns and claims rather than emotional tone). Saturation in this sense is a moving boundary, and a benchmark maintained over model generations should expect dimensions to cross it.

4.3 Limitations

The primary corpus is private, and although the methodology, the schema, the scoring code, and the public replication are all examinable, the headline private corpus numbers are not independently verifiable, a limitation the replication narrows but does not remove. The benchmark set is thirty chunks, large enough to separate composites that differ by several hundredths but without confidence intervals, and significance testing between adjacent models remains future work. The reference is a model rather than human annotation, with the consequences described in Section 4.1. The judge panel contains three families but one judge per family, confounding family with individual judge identity. The emotional signal investigation used corpus level aggregates, and although the constant signals are constant by saturation rather than by aggregation alone, a per chunk variance design might recover discrimination the aggregate design cannot. Finally, one challenger was retired mid study and one challenger substitution was made on composite proximity grounds, both documented in Section 2.7, and both illustrating that the population of commercial models is itself unstable in ways that bound any leaderboard's shelf life.

4.4 Closing

A single number cannot say whether an extraction pipeline survives a model swap, because survival depends on which errors occur, where they concentrate, and what consumes them. The layered design reported here is one way to make those dependencies measurable, and its most useful outputs were not the leaderboard but the disagreements between layers: the discovery that judges measure precision while pipelines measure recall, the demonstration that aggregate correlations can understate multidimensional instruments, and the identification of a dimension whose structural variation has quietly saturated. The errors that score well are still not the errors that break the pipeline, but it is now possible to say which is which.

References

Chang, J. P., Chiam, C., Fu, L., Wang, A., Zhang, J., and Danescu-Niculescu-Mizil, C. (2020). ConvoKit: A toolkit for the analysis of conversations. Proceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue, 57-60.

Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2901-2907.

Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., and Yu, Z. (2019). Persuasion for good: Towards a personalized persuasive dialogue system for social good. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5635-5649.

Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36, 46595-46623.