Inter-Rater Reliability
Inter-rater reliability is not one statistic but a family of estimands for asking whether independent judges produce interchangeable labels or scores under a specified measurement design. Cohen’s kappa, Fleiss’s kappa, Krippendorff’s alpha, intraclass correlation coefficients, and generalizability theory differ in what they treat as chance, what scale type they assume, whether they tolerate missing data, and whether they estimate exact agreement, rank association, or variance partitioning. For LLM-as-judge systems such as PsycheEval, the defensible standard is not a single universal coefficient but a predeclared reliability battery matched to categorical red flags, scalar rubric dimensions, pairwise preferences, and judge-bias diagnostics.
Coverage note: verified through May 19, 2026.
Why inter-rater reliability is a design problem, not a coefficient lookup
The core reliability question is: would the result be materially the same if the item were judged by another competent rater from the intended judge population? That question sounds simple, but the statistic changes depending on the answer to several design choices:
| Design question | Why it matters |
|---|---|
| Are ratings categorical, ordinal, interval-like, or continuous? | Nominal red-flag labels, 1–5 Likert scores, 0–10 scalar rubric scores, and continuous measurements require different loss functions. |
| Are there two raters, a fixed panel, or a sample from a larger judge population? | Cohen’s kappa assumes two raters; Fleiss’s kappa generalizes to multiple raters; ICC variants differ depending on whether raters are fixed or random. |
| Are all items rated by all raters? | Many agreement coefficients assume complete rectangular data; Krippendorff’s alpha was designed to handle missingness and variable numbers of coders more naturally. |
| Is the goal exact agreement, consistency, rank-order agreement, or decision dependability? | Pearson/Spearman can be high even when raters systematically disagree in level; ICC and alpha are closer to reliability; generalizability theory asks where variance comes from. |
| Is the label prevalence skewed? | Kappa can be low even when raw agreement is high under rare-label conditions, creating the classic Kappa Paradox. |
| Are raters exchangeable with future raters? | A fixed expert panel, a pool of crowd workers, and a set of LLM judges are different measurement universes. |
Cohen introduced kappa as a chance-corrected coefficient for nominal-scale agreement between two raters in 1960; Fleiss later generalized kappa-like agreement to multiple raters; Shrout and Fleiss framed intraclass correlations as a family of reliability coefficients for ratings by judges; Krippendorff developed alpha as a broader disagreement-based coefficient for content analysis and other coding settings. These are not interchangeable names for the same thing; they answer different measurement questions. OUP Academic+5Sage Journals+5Ovid+5
Summary table: what to use when
| Statistic | Primary use | Scale type | Raters/design | Handles missing ratings? | Main strength | Main failure mode |
|---|---|---|---|---|---|---|
| Percent agreement | Descriptive agreement | Nominal or ordinal | Any, usually simple panels | Yes, if computed pairwise | Transparent and intuitive | No chance correction; inflated by common labels |
| Cohen’s Kappa | Two-rater categorical agreement | Nominal; weighted form for ordinal | Exactly two raters | Not naturally | Familiar, simple, chance-corrected | Sensitive to prevalence and marginal imbalance |
| Weighted Cohen’s kappa | Two-rater ordered-category agreement | Ordinal | Exactly two raters | Not naturally | Penalizes near-misses less than far misses | Weight choice can dominate interpretation |
| Fleiss’s Kappa | Multiple-rater categorical agreement | Nominal | Usually fixed number of ratings per item; raters may vary | Limited | Extension of kappa logic to panels | Still inherits prevalence/marginal problems |
| Krippendorff’s Alpha | General agreement coefficient | Nominal, ordinal, interval, ratio via distance functions | Two or more raters; variable coders per item | Yes | Flexible; suitable for messy annotation designs | Less familiar; results depend on distance metric |
| Intraclass Correlation Coefficient | Continuous or interval-like ratings | Interval/continuous | Several variants: one-way/two-way, fixed/random, single/average | Usually requires modeled design | Estimates variance attributable to target rather than noise | Easy to report the wrong ICC form |
| Pearson correlation | Linear association between score vectors | Interval/continuous | Usually two raters or aggregate vectors | Pairwise | Useful diagnostic of calibration slope | Not agreement; ignores constant offsets |
| Spearman correlation | Rank association | Ordinal/continuous ranks | Usually two raters or aggregate vectors | Pairwise | Useful for ordinal ordering | Not agreement; ignores monotone distortions |
| Generalizability Theory | Reliability under complex designs | Any modeled measurement design | Raters, items, occasions, prompts, models as facets | Yes, with model design | Partitions variance by source | Requires larger designs and explicit modeling |
The most common error is to select a statistic by habit rather than by measurement design. Kappa is not “the” agreement statistic; ICC is not a generic substitute for kappa; Pearson and Spearman are not reliability coefficients unless the intended construct is only association or rank order.
Cohen’s kappa: two raters, categorical outcomes
Cohen’s Kappa is appropriate when two raters independently assign items to mutually exclusive categories and the categories are nominal. Its basic form is:
κ=Po−Pe1−Pe\kappa = \frac{P_o - P_e}{1 - P_e}κ=1−PePo−Pe where PoP_oPo is observed agreement and PeP_ePe is expected agreement under chance agreement implied by the two raters’ marginal category distributions. Cohen’s original paper, “A Coefficient of Agreement for Nominal Scales,” appeared in Educational and Psychological Measurement in 1960 and was explicitly about nominal-scale agreement. Sage Journals
For a two-rater binary label, the contingency table is:
| Rater B positive | Rater B negative | |
|---|---|---|
| Rater A positive | aaa | bbb |
| Rater A negative | ccc | ddd |
Po=a+da+b+c+dP_o = \frac{a+d}{a+b+c+d}Po=a+b+c+da+d Pe=(a+bNa+cN)+(c+dNb+dN)P_e = \left(\frac{a+b}{N}\frac{a+c}{N}\right) + \left(\frac{c+d}{N}\frac{b+d}{N}\right)Pe=(Na+bNa+c)+(Nc+dNb+d) Kappa is attractive because it subtracts a model of chance agreement. It is also dangerous because the “chance” term is not a neutral background constant; it is computed from the observed marginal distributions. This means kappa is partly a statistic about agreement and partly a statistic about base rates and rater bias.
When Cohen’s kappa is appropriate
Use Cohen’s kappa when:
| Condition | Example |
|---|---|
| There are exactly two raters | Two clinicians label each case “present” or “absent.” |
| Categories are nominal | “hallucination,” “refusal,” “privacy violation,” “no issue.” |
| Both raters classify the same items | Every output receives both labels. |
| Marginal distributions are meaningful to inspect | The frequency of each category is itself part of the measurement story. |
Cohen’s kappa is not ideal for scalar 0–10 rubric scores, multi-rater panels, sparse rare-event labels, or annotation designs with missing raters. Weighted kappa can be used for ordered categories, but the weight matrix must be justified rather than chosen because it produces a pleasing number.
Fleiss’s kappa: multiple raters for categorical labels
Fleiss’s Kappa generalizes kappa-like chance-corrected categorical agreement to settings with more than two raters. Fleiss’s 1971 paper is commonly cited as extending kappa beyond a fixed pair of raters to multiple raters; the original setting was nominal-scale agreement where a set of subjects receive ratings from several judges, and the statistic estimates agreement beyond chance across the panel. Ovid+2ResearchGate+2
Fleiss’s kappa is useful when each item receives the same number of categorical ratings, especially when raters are sampled from a larger population or when different items may be rated by different subsets of raters. It is less natural when the design is highly incomplete, when categories are ordered, or when disagreement distances matter. In those cases, Krippendorff’s alpha is usually the cleaner generalization.
A common mistake is to report Fleiss’s kappa as if it says that every rater agrees with every other rater equally. It does not. It summarizes panel-level agreement, and it can hide individual judge pathologies such as one judge overusing a rare category or one LLM judge showing systematic position bias.
Krippendorff’s alpha: a disagreement-first generalization
Krippendorff’s Alpha is best understood as a general coefficient of disagreement rather than as a simple “multi-rater kappa.” Its canonical form is:
α=1−DoDe\alpha = 1 - \frac{D_o}{D_e}α=1−DeDo where DoD_oDo is observed disagreement and DeD_eDe is expected disagreement under a suitable chance model. The key feature is that disagreement can be defined using a distance function appropriate to the measurement level: nominal, ordinal, interval, ratio, or other structured distances. Krippendorff’s early interval-data reliability work appeared in 1970, and his 2004 paper in Human Communication Research revisited reliability in content analysis, including misconceptions and recommendations for reliability coefficients. ERIC
Alpha is often preferable when:
| Design feature | Why alpha helps |
|---|---|
| More than two raters | Alpha naturally handles panels. |
| Missing ratings | Items need not be rated by every coder. |
| Different numbers of raters per item | Alpha can use available pairable information. |
| Ordinal or interval ratings | Disagreement can be weighted by distance. |
| Annotation projects rather than fixed psychometric panels | Messy coding designs are common in content analysis and NLP annotation. |
Hayes and Krippendorff argue that alpha should be treated as a general reliability coefficient for content analysis because it supports multiple observers, missing data, and different measurement levels; they contrast this with coefficients such as Fleiss’s kappa that are restricted to nominal data in their basic form. asc.upenn.edu
The alpha-versus-kappa debate is partly technical and partly sociological. Kappa is more familiar in medicine, psychology, NLP, and evaluation papers; alpha is often more faithful to the actual annotation design. Familiarity is not a measurement argument. But alpha is not magic either: it still depends on the chosen distance function, the sampling design, and the assumption that the observed codings represent the intended universe of coding behavior.
ICC: reliability for continuous and interval-like ratings
The Intraclass Correlation Coefficient family is the usual tool for continuous or interval-like ratings, such as 0–10 rubric dimensions, quality scores, or severity ratings. Shrout and Fleiss’s 1979 paper framed intraclass correlations as reliability coefficients for nnn targets rated by kkk judges and emphasized that the correct ICC form depends on the statistical model and the intended application. PubMed
The conceptual ICC is:
ICC≈variance due to true target differencesvariance due to target differences+measurement error varianceICC \approx \frac{\text{variance due to true target differences}} {\text{variance due to target differences} + \text{measurement error variance}}ICC≈variance due to target differences+measurement error variancevariance due to true target differences But the exact denominator depends on the design. Rater variance, rater-by-target interaction, residual error, and whether scores are averaged across raters all matter.
The key ICC choices
| Choice | Options | Consequence |
|---|---|---|
| Rater design | One-way, two-way random, two-way mixed | Determines whether raters are treated as random samples from a judge population or as the only raters of interest. |
| Agreement target | Absolute agreement vs consistency | Absolute agreement penalizes systematic offsets; consistency may ignore them. |
| Unit of reporting | Single-measure vs average-measure ICC | Reliability of one judge is lower than reliability of the mean of several judges. |
| Inference target | Current panel vs future interchangeable judges | Changes whether judge effects belong in the error term. |
A practical interpretation:
| ICC form family | Use when | Example |
|---|---|---|
| One-way random | Each target is rated by a random subset of raters, and raters are not crossed with all targets | Crowd workers rate different subsets of outputs. |
| Two-way random | The same raters judge all targets, and you want to generalize to similar future raters | A benchmark estimates reliability of an LLM-judge population, not just the three judges used. |
| Two-way mixed | The same raters judge all targets, and those raters are the fixed panel of interest | A report concerns GPT-5.4, GPT-5.5, and Claude Opus as named judges, not as samples. |
| Absolute agreement | You care whether judges give the same numeric score | Two judges scoring “safety = 8” vs “safety = 6” is disagreement. |
| Consistency | You care whether judges rank outputs similarly despite offsets | One judge is stricter but preserves ordering. |
| Average-measure ICC | The deployed score is the mean of multiple judges | PsycheEval reports the mean of three judge scores. |
| Single-measure ICC | A single judge score will be used operationally | One model judge grades each item. |
A paper that reports only “ICC = 0.71” without saying which ICC was computed is under-specified. The same data can yield materially different ICCs depending on whether the model treats raters as fixed or random, whether it asks for consistency or absolute agreement, and whether it estimates reliability of a single judge or an average judge panel.
Agreement is not correlation
Pearson and Spearman correlations are useful diagnostics, but they are not substitutes for agreement coefficients.
Suppose two judges score ten outputs on a 0–10 scale. Judge B always gives exactly two points more than Judge A. Pearson correlation can be 1.0, and Spearman correlation can be 1.0, because the judges preserve a perfect linear and rank relationship. But they do not agree in absolute score. If the benchmark threshold says “scores below 4 fail,” the offset can change decisions.
That distinction matters for LLM-as-Judge Evaluation. Pearson and Spearman answer: do judges move together? ICC and alpha answer closer questions: are the ratings interchangeable under the measurement design? For scalar LLM judge scores, correlations are best reported as supplementary calibration diagnostics, not as the primary reliability statistic.
The kappa paradox: high agreement, low kappa
The Kappa Paradox is the best-known failure mode of kappa. Feinstein and Cicchetti’s 1990 paper “High agreement but low kappa” identified paradoxes in which high observed agreement can produce a low kappa, especially when marginal distributions are imbalanced; their companion paper proposed separating positive and negative agreement to diagnose the problem. PubMed
A simple binary example shows the problem:
| Rater B positive | Rater B negative | |
|---|---|---|
| Rater A positive | 1 | 4 |
| Rater A negative | 4 | 91 |
Observed agreement:
Po=1+91100=0.92P_o = \frac{1+91}{100} = 0.92Po=1001+91=0.92 Expected agreement from marginals:
Pe=(0.05×0.05)+(0.95×0.95)=0.905P_e = (0.05 \times 0.05) + (0.95 \times 0.95) = 0.905Pe=(0.05×0.05)+(0.95×0.95)=0.905 Kappa:
κ=0.92−0.9051−0.905≈0.158\kappa = \frac{0.92 - 0.905}{1 - 0.905} \approx 0.158κ=1−0.9050.92−0.905≈0.158 The raters agree on 92% of cases, but kappa is only about 0.16 because almost everything is negative. The statistic is saying: “given the skewed marginals, much of this agreement is unsurprising.” That may be mathematically defensible, but operationally it can be misleading if the rare positive cases are what matter.
Prevalence and bias problems
Kappa’s two central pathologies are:
| Problem | Mechanism | Practical symptom |
|---|---|---|
| Prevalence problem | One category dominates the data | High raw agreement, low kappa |
| Bias/marginal problem | Raters use categories at different rates | Kappa changes because of rater marginal distributions, not just item-level disagreement |
For rare-event annotation, such as red-flag labels in safety evaluation, this matters enormously. A label that appears in 1% of examples can produce unstable or low kappa even when raters agree on the 99% absence class. Conversely, a high kappa on a balanced artificial set may not transfer to a realistic population where the label is rare.
A responsible kappa report should include:
| Required diagnostic | Why it matters |
|---|---|
| Raw agreement | Shows the literal agreement rate. |
| Category prevalence | Reveals rare-label regimes. |
| Rater marginals | Shows whether raters use categories differently. |
| Positive agreement | Measures agreement on presence of the condition. |
| Negative agreement | Measures agreement on absence of the condition. |
| Confidence intervals or bootstrap intervals | Avoids overinterpreting small samples. |
| Per-label results for multilabel tasks | A single aggregate can hide rare-label collapse. |
For LLM evaluation, the prevalence problem is not a footnote. Many of the most important safety, deception, sycophancy, manipulation, or “red flag” labels are rare by construction. A reliability coefficient that behaves badly under skewed prevalence can make a good screening system look unreliable or make a trivial “always negative” system look deceptively stable if only raw agreement is reported.
Landis and Koch thresholds: useful folklore, weak measurement theory
Landis and Koch’s 1977 Biometrics paper is the source most often associated with the familiar kappa interpretation scale: below 0 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.00 almost perfect. The paper addressed observer agreement for categorical data, but the thresholds have since been copied far beyond their original context. jstor.org
| Kappa range | Conventional Landis–Koch label |
|---|---|
| < 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
The active critique is straightforward: these labels are not laws of measurement. A kappa of 0.35 may be acceptable for early exploratory coding of ambiguous phenomena, but unacceptable for clinical diagnosis, safety-critical classification, or a benchmark used to rank frontier models. Conversely, a kappa of 0.75 can be uninformative if the categories are badly designed, the prevalence is artificial, or the raters share the same systematic blind spot.
Krippendorff’s reliability writing is especially hostile to treating coefficient labels as substitutes for design reasoning: a coefficient must be interpreted relative to the consequences of error, the measurement level, the sampling universe, and the intended use of the coded data. OUP Academic
The better practice is to report both the coefficient and the design context:
“Red-flag label kappa was 0.32, with 94% raw agreement, 3% positive prevalence, low positive agreement, and high negative agreement. This is adequate for exploratory clustering but not adequate for adjudication without a third judge or human calibration.”
That sentence is more informative than “kappa was fair.”
Alpha versus kappa: the real tradeoff
The alpha-versus-kappa debate is often framed as a contest between two coefficients. The deeper issue is whether the statistic matches the annotation design.
| Dimension | Kappa family | Krippendorff’s alpha |
|---|---|---|
| Familiarity | Very familiar in psychology, medicine, NLP, and benchmark papers | Familiar in content analysis and annotation methodology, less common in ML papers |
| Two-rater nominal labels | Cohen’s kappa is simple and standard | Alpha works, but may feel heavier |
| Multiple raters | Fleiss’s kappa works under suitable categorical designs | Alpha works naturally |
| Missing data | Awkward | Native strength |
| Ordinal/interval ratings | Requires weighted variants or other statistics | Built around distance functions |
| Rare-label behavior | Kappa paradox remains | Alpha can still be affected by prevalence, but the distance/disagreement framing is clearer |
| Interpretability | Widely recognized thresholds, though overused | Requires more methodological explanation |
| Best use | Clean categorical designs, comparability with prior literature | Messy annotation, mixed scales, missingness, multi-rater content analysis |
A defensible compromise is to report alpha as the primary statistic when the design is multi-rater, incomplete, or ordinal/interval, and report kappa-family coefficients as familiar comparators when the data are categorical and readers expect them. This avoids the false choice between methodological fit and literature comparability.
Inter-rater reliability versus generalizability theory
Generalizability Theory is the natural extension when a single coefficient is too small for the design. Cronbach, Rajaratnam, and Gleser’s 1963 work framed generalizability theory as a liberalization of reliability theory: instead of treating measurement error as one undifferentiated residual, it partitions variance across facets such as raters, items, occasions, and their interactions. Sage Journals
In classical inter-rater reliability, one might ask:
Do judges agree?\text{Do judges agree?}Do judges agree? In generalizability theory, one asks:
How much variance comes from targets, raters, items, occasions, prompts, judge models, and their interactions?\text{How much variance comes from targets, raters, items, occasions, prompts, judge models, and their interactions?}How much variance comes from targets, raters, items, occasions, prompts, judge models, and their interactions? A simplified G-theory model for LLM evaluation might look like:
Youtput,judge,dimension,occasion=μ+output+judge+dimension+occasion+output×judge+output×dimension+judge×dimension+ϵY_{output, judge, dimension, occasion}
\mu
- output
- judge
- dimension
- occasion
- output \times judge
- output \times dimension
- judge \times dimension
- \epsilonYoutput,judge,dimension,occasion=μ+output+judge+dimension+occasion+output×judge+output×dimension+judge×dimension+ϵ For LLM-as-Judge Evaluation, this is often a better mental model than a single kappa or ICC. The observed score may vary because of:
| Facet | Example in LLM evaluation |
|---|---|
| Output | The model response being judged is genuinely better or worse. |
| Judge | GPT-5.5 is stricter than Claude Opus on emotional overreach. |
| Dimension | “Helpfulness” is more reliably scored than “therapeutic boundary safety.” |
| Scenario | Judges agree on obvious cases and diverge on ambiguous ones. |
| Prompt | The judge rubric changes the score distribution. |
| Position | Pairwise A/B order changes preferences. |
| Occasion | Same judge model changes behavior across sampling or version updates. |
| Author model | Some answer-generating models create distinctive artifacts. |
Generalizability theory separates two activities: a G-study, which estimates variance components, and a D-study, which asks how reliable a future measurement design would be if you changed the number of raters, items, occasions, or dimensions. This matters for benchmark design because the practical question is rarely “what was reliability in this dataset?” It is usually “how many judges, items, and counterbalanced comparisons do we need for a dependable benchmark?”
PsycheEval as a case study
PsycheEval is a useful testbed because it mixes categorical red-flag labels, scalar rubric judgments, pairwise preferences, multiple judge models, personas, scenarios, and position-bias controls. In the pilot, PsycheEval used multiple author and judge models, generated outputs across conditions and scenarios, scored them with scalar rubric dimensions, applied a red-flag taxonomy, and included pairwise preference judgments. The pilot explicitly reported modest inter-judge kappa on red-flag labels: about 0.32 between GPT-5.4 and GPT-5.5, 0.33 between GPT-5.4 and Opus, and 0.29 between GPT-5.5 and Opus. It also noted that scalar pooling rested on an unmeasured assumption and that future versions should add ICC or Krippendorff’s alpha. Ashita Orbis
PsycheEval v0.2 then added stronger controls, including AB/BA counterbalancing for pairwise judgments and an anchored rubric that widened score spread. Its own analysis found that some apparent pairwise wins collapsed under AB/BA controls, while a reduced set of findings survived; it also found judge-specific position effects, including a strong slot-B bias for one judge model in that run. Ashita Orbis
That makes PsycheEval a compact example of why inter-rater reliability cannot be summarized by one number.
Red-flag labels
For categorical red flags, the natural statistics are:
| PsycheEval object | Recommended statistic | Required companion diagnostics |
|---|---|---|
| Any red flag vs no red flag | Cohen/Fleiss kappa or nominal alpha | Raw agreement, prevalence, positive/negative agreement |
| Specific red-flag labels | Per-label kappa/alpha | Label frequency, rater marginals, undefined-label warnings |
| Multilabel red-flag sets | Label-wise agreement plus set-overlap metrics | Jaccard/F1-style descriptive overlap, not as replacement for reliability |
| Rare severe flags | Positive agreement and adjudication rate | Kappa alone is unsafe under low prevalence |
The pilot’s kappa values around 0.29–0.33 fall into the conventional “fair” range, but that label should not be allowed to do the interpretive work. For exploratory research, those values may be enough to show that the phenomenon is noisy but not random. For a benchmark that claims to rank systems on psychological safety, they are too low unless paired with adjudication, human calibration, or a reliability-aware scoring rule.
Scalar rubric dimensions
For scalar dimensions such as empathy, helpfulness, boundary clarity, or emotional overreach, kappa is the wrong primary statistic unless the scores are discretized into categories. Better options are:
| Data | Primary reliability statistic | Supplementary diagnostics |
|---|---|---|
| 0–10 scalar scores from the same fixed judge models | Two-way mixed ICC, single or average depending on scoring rule | Pearson, Spearman, judge mean/variance shifts |
| 0–10 scalar scores intended to generalize to future judge models | Two-way random ICC | Judge-family variance, confidence intervals |
| Incomplete scalar ratings | Krippendorff interval alpha | Missingness pattern, per-dimension sample sizes |
| Ordinal rubric buckets | Krippendorff ordinal alpha or weighted kappa | Weight matrix sensitivity |
PsycheEval should report scalar reliability per dimension, not only as an aggregate. A high overall ICC can hide that judges agree on surface helpfulness but diverge on therapeutic boundary violations. Per-dimension reliability is also necessary for deciding whether to merge dimensions, revise rubrics, or remove unstable constructs.
Pearson and Spearman correlations between judges remain useful, but only as secondary diagnostics. Pearson detects whether two judges move linearly together; Spearman detects whether they preserve rank order. Neither proves that judges assign interchangeable scores.
Pairwise preferences
Pairwise preference data are often treated as easier than scalar data, but they have their own reliability traps. Zheng et al.’s MT-Bench work made LLM-as-judge evaluation prominent while also documenting biases such as position, verbosity, and self-enhancement bias; later work on position bias found that the bias is not merely random noise and can vary by judge and task. arXiv
For pairwise PsycheEval-style comparisons, report:
| Pairwise object | Reliability or bias statistic |
|---|---|
| A/B/tie categorical choice | Cohen/Fleiss kappa or nominal alpha across judges |
| Win rate | Counterbalanced AB/BA win rate |
| Position sensitivity | Difference between AB and BA outcomes |
| Judge-specific bias | Per-judge slot preference |
| Verbosity sensitivity | Length-controlled or length-stratified win rates |
| Stability of ranking | Bootstrap intervals over prompts/scenarios |
The v0.2 PsycheEval result that some claims collapsed under AB/BA counterbalancing is exactly the sort of finding a reliability section should surface rather than hide. A benchmark whose conclusions change when answer order changes is measuring a mixture of model quality and judge-position artifacts.
LLM-as-judge reliability: the open standardization problem
The open question is: which statistic should be standard for LLM-as-judge evaluation? The honest answer is that no single statistic should be standard across all LLM-as-judge tasks. The standard should be a reporting protocol.
Recent LLM-as-judge research reinforces this. MT-Bench showed that strong LLM judges can approximate human preferences in some settings but also exhibit biases such as position and verbosity effects. JUDGE-BENCH evaluated LLM judges across many datasets and found substantial variation across models and tasks, recommending validation against human judgments before use. Information-retrieval evaluation guidelines likewise warn against circularity, leakage, and unvalidated LLM metrics, recommending validation against human or user judgments and complementary metrics. arXiv+2ACL Anthology+2
A defensible LLM-as-judge reliability standard would look like this:
| Evaluation output | Primary reliability report | Required bias/validity report |
|---|---|---|
| Binary/categorical safety labels | Krippendorff nominal alpha; Cohen/Fleiss kappa as comparator | Raw agreement, prevalence, positive/negative agreement, rater marginals |
| Ordinal rubric buckets | Krippendorff ordinal alpha or weighted kappa | Weight sensitivity, category histograms |
| 0–10 scalar dimensions | Predeclared ICC form plus interval alpha if missingness exists | Pearson/Spearman, judge offsets, confidence intervals |
| Pairwise preferences | Counterbalanced win rates plus alpha/kappa on A/B/tie choices | Position bias, verbosity bias, judge-family effects |
| Aggregate benchmark rankings | Bootstrap uncertainty plus G-theory variance components | Human calibration, leakage/circularity audit |
| High-stakes claims | Human adjudication or expert calibration sample | Error analysis by category and scenario |
The standard should also require authors to say what their raters are. “Three LLM judges” is not a sampling theory. Are these judges fixed instruments? Are they meant to represent frontier LLMs? Are they proxies for expert humans? Are they calibrated against a human panel? Each answer implies a different reliability model.
Reporting checklist
A good inter-rater reliability section should include the following.
| Report item | Minimum acceptable detail |
|---|---|
| Rating unit | What exactly is being rated: output, sentence, scenario, pairwise comparison, dimension, or label? |
| Rating scale | Nominal, ordinal, interval-like, continuous, multilabel, pairwise. |
| Rater set | Number of raters, identity/type of raters, fixed vs random interpretation. |
| Missingness | How many ratings are missing and why. |
| Primary statistic | Kappa, alpha, ICC, or G-theory model, justified by design. |
| Statistic variant | For ICC: model, agreement/consistency, single/average. For alpha: distance metric. For kappa: weighted/unweighted. |
| Raw agreement | Always include for categorical labels. |
| Marginals/prevalence | Required for kappa-family statistics. |
| Confidence intervals | Bootstrap or analytic intervals where appropriate. |
| Per-category or per-dimension results | Avoid hiding unstable categories inside aggregates. |
| Bias diagnostics | Position, verbosity, rater strictness, model-family bias. |
| Validity evidence | Agreement with humans, experts, outcomes, or external criteria. |
A reliability coefficient without these details is not reproducible as a measurement claim. It is at best a descriptive statistic and at worst a false certificate of objectivity.
Practical recommendations
1. Do not use kappa as a generic synonym for reliability
Use kappa for categorical agreement, especially when comparing with prior two-rater literature. Do not use it for scalar ratings unless the scores have been intentionally discretized and the discretization is justified.
2. Always report prevalence with kappa
A kappa without marginals is underinterpretable. For rare labels, report positive and negative agreement separately. This is especially important for safety red flags, clinical labels, content moderation categories, and other low-base-rate phenomena.
3. Prefer alpha for messy multi-rater annotation
When the design includes missing ratings, variable numbers of raters per item, ordinal scales, interval-like scores, or multiple coders, Krippendorff’s alpha is often the better primary statistic. Report the distance function.
4. Use ICC for scalar judge scores, but specify the exact form
For 0–10 ratings, ICC is usually the right family. But “ICC” alone is not a result. State whether the model is one-way or two-way, random or mixed, absolute agreement or consistency, single-measure or average-measure.
5. Treat Pearson and Spearman as diagnostics, not reliability
They are useful for detecting whether judges preserve ordering or linear association. They do not establish agreement.
6. Use generalizability theory when the benchmark has many facets
If raters, prompts, dimensions, scenarios, author models, position, and occasions all matter, a single reliability coefficient is not enough. Estimate variance components and ask which facets dominate measurement error.
7. For LLM judges, separate reliability from validity
Reliable LLM judges can reliably share the same blind spot. Validity requires calibration against humans, experts, outcomes, or task-specific ground truth. Reliability is necessary for measurement; it is not sufficient for truth.
References
[Cohen 1960] Jacob Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement 20(1):37–46. Sage Journals
[Fleiss 1971] Joseph L. Fleiss, “Measuring Nominal Scale Agreement Among Many Raters,” Psychological Bulletin 76:378–382. Ovid
[Shrout & Fleiss 1979] Patrick E. Shrout and Joseph L. Fleiss, “Intraclass Correlations: Uses in Assessing Rater Reliability,” Psychological Bulletin 86(2):420–428. PubMed
[Krippendorff 1970] Klaus Krippendorff, “Estimating the Reliability, Systematic Error and Random Error of Interval Data,” Educational and Psychological Measurement 30(1):61–70. ERIC
[Krippendorff 2004] Klaus Krippendorff, “Reliability in Content Analysis: Some Common Misconceptions and Recommendations,” Human Communication Research 30(3):411–433. OUP Academic
[Hayes & Krippendorff 2007] Andrew F. Hayes and Klaus Krippendorff, “Answering the Call for a Standard Reliability Measure for Coding Data,” Communication Methods and Measures. asc.upenn.edu
[Feinstein & Cicchetti 1990a] Alvan R. Feinstein and Domenic V. Cicchetti, “High Agreement but Low Kappa: I. The Problems of Two Paradoxes,” Journal of Clinical Epidemiology. PubMed
[Feinstein & Cicchetti 1990b] Alvan R. Feinstein and Domenic V. Cicchetti, “High Agreement but Low Kappa: II. Resolving the Paradoxes,” Journal of Clinical Epidemiology. PubMed
[Landis & Koch 1977] J. Richard Landis and Gary G. Koch, “The Measurement of Observer Agreement for Categorical Data,” Biometrics. jstor.org
[Cronbach, Rajaratnam & Gleser 1963] Lee J. Cronbach, N. Rajaratnam, and Goldine C. Gleser, “Theory of Generalizability: A Liberalization of Reliability Theory,” British Journal of Statistical Psychology 16:137–163. Sage Journals
[Zheng et al. 2023] Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv
[Shi et al. 2024/2025] Work on LLM judge position bias across benchmarks and judge models. arXiv
[Bavaresco et al. 2025] JUDGE-BENCH, a benchmark study of LLM judges across NLP evaluation tasks. ACL Anthology
[Dietz et al. 2025] Guidelines for using large language models as judges in information retrieval evaluation. Computer Science
[PsycheEval Pilot] PsycheEval pilot report on psychologically rich LLM evaluation, including red-flag kappa and scalar-rubric caveats. Ashita Orbis
[PsycheEval v0.2] PsycheEval v0.2 report with AB/BA counterbalancing, position-bias findings, and revised claims. Ashita Orbis
Companion entries
Core theory: Inter-Rater Reliability, Measurement Theory, Psychometrics, Generalizability Theory, Variance Components, Reliability vs Validity
Agreement statistics: Cohen’s Kappa, Fleiss’s Kappa, Weighted Kappa, Krippendorff’s Alpha, Intraclass Correlation Coefficient, Percent Agreement
Failure modes and critiques: Kappa Paradox, Prevalence Bias, Rater Marginal Imbalance, Landis-Koch Thresholds, Agreement Is Not Correlation, Benchmark Validity
LLM evaluation practice: LLM-as-Judge Evaluation, PsycheEval, Pairwise Evaluation, AB/BA Counterbalancing, Judge Bias, Position Bias, Verbosity Bias, Human Calibration
Implementation and reporting: Reliability Reporting Checklist, Bootstrap Confidence Intervals, Per-Dimension Agreement, Red-Flag Taxonomy, Scalar Rubric Design, Variance Component Design