Abstract. Cluster bootstrap is a practical fix for a recurring failure mode in LLM Evaluation: treating thousands of judgment rows as independent when they are actually repeated measurements over shared prompts, personas, scenarios, authors, outputs, and judges. The method resamples whole dependency units rather than individual records, producing confidence intervals that better reflect the number of independent experimental units actually observed. Its value is not that it solves every evaluation confound, but that it makes false precision harder to hide.

Coverage note: verified through May 19, 2026.

Cluster Bootstrap for Hierarchical Evaluation Data

The methodological problem

Modern LLM evaluation datasets often look large at the row level and small at the independent-unit level. A pairwise eval table may contain 20,000 judgment records, but those records can be generated from a much smaller design: 80 scenarios, 12 personas, 5 author models, 3 judge models, and repeated A/B comparisons over the same outputs. The row count is real as logging volume, but it is not automatically real as Effective Sample Size.

The statistical issue is familiar from clustered survey sampling, panel econometrics, education trials, medical repeated-measures designs, and other settings where observations are grouped. Cameron and Miller’s practitioner guide frames the problem as data “grouped into clusters,” with errors independent across clusters but correlated within clusters; it warns that default standard errors can overstate precision, and that failing to control within-cluster correlation can yield misleadingly small standard errors, narrow confidence intervals, large test statistics, and low p-values. Colin Cameron's Home Page+1

In LLM evals, clustering appears in several forms:

Eval structure	Example dependency	Why row-level independence fails
Multiple judges rate the same output	GPT-5.4, GPT-5.5, and Opus all score one answer	Their judgments are not three fresh samples of user preference; they share the same response, prompt, and context.
Multiple condition pairs reuse the same output	C3 output compared against C4, C5, and C5_CONTRACT	The same output quality drives several pairwise rows.
Multiple outputs share an author model	Many responses generated by the same base model	Author-model quirks create correlated successes and failures.
Multiple scenarios share a persona	A synthetic persona appears across several tasks	Persona-specific affordances or failure modes can persist across scenarios.
Multiple scenarios share a family	“authority disagreement” or “moral uncertainty” families recur	Task-family difficulty affects multiple rows.
Pairwise judge prompts share position/order	Candidate A or B placement is systematic	LLM-as-Judge Bias can turn display order into a pseudo-treatment effect.

The common bad pattern is to compute a Bernoulli or normal confidence interval as if each judgment row were an independent draw. For a binary win/loss statistic, this might mean reporting a Wilson interval for a win rate over all pairwise decisions. Wilson intervals are useful for independent binomial observations, but a repeated-measures eval table is not a bag of independent Bernoulli trials when rows share outputs, scenarios, authors, or judges. PsycheEval v0.1 made exactly this caveat: its pairwise channel used Wilson confidence intervals, but the author noted that the intervals assumed independence across pair records, while the same persona × scenario × author cell appeared in multiple condition-pair rows, inflating effective n; the post explicitly anticipated that cluster bootstrap intervals by persona × scenario would likely widen the pairwise intervals. Ashita Orbis

The core statistical object is not “how many rows are in the CSV?” It is “what units were independently sampled, randomized, or intended to support generalization?” If the design samples 48 scenarios and then generates many judgments per scenario, the scenario count matters. If it samples 12 personas and runs many conditions per persona, the persona count matters. If every candidate output is scored by the same three judge models, the judge ensemble matters, and judge-specific bias may be a fixed feature of the measurement process rather than independent noise.

Why naive confidence intervals become too narrow

Suppose an eval reports a pairwise win rate:

p^=1n∑i=1nyi,\hat{p}=\frac{1}{n}\sum_{i=1}^n y_i,p^​=n1​i=1∑n​yi​, where yi=1y_i = 1yi​=1 if condition A beats condition B in judgment row iii. A naive binomial or normal interval treats the variance as roughly:

Var⁡(p^)≈p(1−p)n.\operatorname{Var}(\hat{p}) \approx \frac{p(1-p)}{n}.Var(p^​)≈np(1−p)​. That formula uses nnn as the number of independent observations. If 3,000 rows come from 300 underlying persona × scenario × author cells, and rows within each cell are correlated, the denominator is too optimistic. The missing term is covariance within groups:

Var⁡(p^)=1n2(∑iVar⁡(yi)+2∑i<jCov⁡(yi,yj)).\operatorname{Var}(\hat{p}) = \frac{1}{n^2} \left( \sum_i \operatorname{Var}(y_i) + 2\sum_{i<j}\operatorname{Cov}(y_i,y_j) \right).Var(p^​)=n21​(i∑​Var(yi​)+2i<j∑​Cov(yi​,yj​)). The naive interval implicitly drops the covariance terms. Positive within-cluster correlation makes the variance larger than the independent-row formula says. This is why a small intra-cluster correlation can have a large effect when each cluster contributes many rows.

Survey methodology describes the same phenomenon with a Design Effect: the ratio of a statistic’s variance under a complex design to its variance under simple random sampling. Effective sample size is then neff=n/deffn_\text{eff}=n/\text{deff}neff​=n/deff, the size of a simple random sample that would give the same variance. For clustered designs, a standard design-effect approximation is 1+ρ(m−1)1+\rho(m-1)1+ρ(m−1), where ρ\rhoρ is within-cluster homogeneity and mmm is a cluster-size term; the practical lesson is that clustering typically increases design effect and decreases effective sample size. CRAN

In LLM evals, this means a nominally large table can be statistically thin. A 6,000-row table with 20 repeated rows per scenario and strong scenario-level correlation can behave more like a few hundred independent observations. Worse, if the repeated rows are produced by the same judge model family, the table may be precise about that judge family’s behavior while pretending to be precise about human preference.

The error is not merely philosophical. It changes conclusions. Cluster-aware intervals often turn “significant” marginal differences into “not detected,” especially when the point estimate is close to 50%, when cluster-level heterogeneity is high, or when the design reuses the same outputs across many pairwise rows.

The cluster-bootstrap solution

The bootstrap, in its standard form, estimates uncertainty by repeatedly resampling the observed data and recomputing the statistic. Davison and Hinkley’s Bootstrap Methods and their Application is a canonical reference for bootstrap standard errors, confidence intervals, and significance tests across parametric, semiparametric, and nonparametric settings. Cambridge University Press & Assessment

The cluster bootstrap changes the resampling unit. Instead of resampling individual rows, it resamples whole clusters. If the relevant independent unit is persona × scenario × author, then a bootstrap replicate samples those cells with replacement and includes all judgment rows inside each selected cell. This preserves the within-cell dependency structure: the same outputs, judges, ties, position effects, and repeated comparisons remain bundled.

A basic cluster bootstrap for a pairwise win rate looks like this:

1. Define the statistic T(D)T(D)T(D): for example, the condition-A win rate, a difference in win rates, an AB/BA-controlled win rate, or a regression coefficient.

2. Define the cluster unit ccc: for example, scenario, persona × scenario, author × scenario, or persona × scenario × author.

3. Let the dataset be partitioned into GGG clusters: D={D1,…,DG}D = {D_1,\ldots,D_G}D={D1​,…,DG​}.

4. For bootstrap replicate b=1,…,Bb=1,\ldots,Bb=1,…,B, sample GGG clusters with replacement from {1,…,G}{1,\ldots,G}{1,…,G}.

5. Construct D∗(b)D^{*(b)}D∗(b) by concatenating all rows from the selected clusters. If a cluster is selected twice, its full row block appears twice.

6. Recompute T(D∗(b))T(D^{*(b)})T(D∗(b)).

7. Use the empirical distribution of T(D∗(1)),…,T(D∗(B))T(D^{(1)}),\ldots,T(D^{(B)})T(D∗(1)),…,T(D∗(B)) to compute a percentile interval, bootstrap-t interval, BCa interval, or other bootstrap interval.

The econometric version is especially clear. Cameron, Gelbach, and Miller’s cluster-bootstrap work emphasizes resampling entire clusters, and their model assumes independence across clusters while allowing correlation within clusters. Their paper investigates cluster bootstrap-t procedures for few-cluster inference, noting that standard asymptotic tests can over-reject with roughly 5–30 clusters and that cluster bootstrap-t procedures can improve rejection rates in their Monte Carlo settings. NBER The 2008 published version appears in The Review of Economics and Statistics as “Bootstrap-Based Improvements for Inference with Clustered Errors.” IDEAS/RePEc

For simple descriptive LLM eval statistics, the pairs cluster bootstrap is often the most transparent: sample clusters, keep their rows intact, recompute the headline number. For regression-style eval analysis, especially with covariates or treatment indicators, bootstrap-t or wild cluster bootstrap variants may be preferable. Cameron, Gelbach, and Miller caution that bootstrap improvements depend on the resampling method and data discreteness; they identify pairs cluster bootstrap as the standard way to preserve within-cluster features, while also finding wild cluster bootstrap-t useful in some few-cluster settings. NBER

Field and Welsh’s “Bootstrapping clustered data” provides a statistics-side reference specifically on clustered data. They describe mixed conclusions in the literature and report that consistency of bootstrap variance estimates depends on the assumed model; in their abstract, the cluster bootstrap gives consistent variance estimates under both a transformation model and a random-effect model. The Australian National University

The methodological slogan is simple: resample the unit that was independently sampled, not the row that was cheaply logged.

Choosing the cluster level

The hard part is not running the bootstrap. The hard part is choosing the cluster. That choice should follow the estimand: what population or process is the evaluation trying to generalize over?

Hypothesis being tested	Plausible cluster unit	Why	What goes wrong if clustered too low
“Condition C4 beats C1 across these personas and scenarios.”	persona × scenario × author	Repeated judge rows and pair rows share one generated output cell.	Treats multiple judges or pairwise comparisons over the same output as independent.
“This result generalizes across scenarios.”	scenario, or scenario family if families were sampled	Scenario difficulty is a major source of heterogeneity.	A few easy or hard scenarios dominate precision.
“This result generalizes across personas.”	persona	Persona-specific behavior may drive many scenario outcomes.	Repeated scenarios for one persona inflate n.
“The author model effect is stable across prompts.”	author × prompt, or author × scenario	Author-level quirks are reused across outputs.	The same model’s style or failure mode is counted repeatedly.
“Judge ensemble agrees on the effect.”	judge, or judge-stratified reporting	Judge models can have systematic bias.	Pooled CI hides judge disagreement.
“Pairwise preference survives order effects.”	matched AB/BA pair cell	Original and swapped judgments are paired observations.	Position/order noise is treated as new evidence.
“Effect survives crossed persona and scenario variation.”	multi-way method or conservative highest-level cluster	Persona and scenario are crossed, not nested.	One-way clustering misses correlation along the other axis.

The “highest grouped factor” heuristic is conservative but not always correct. If the eval uses four personas, clustering only at persona leaves four clusters, which is too few for stable bootstrap inference. If the eval uses 200 persona × scenario cells, clustering at that interaction may be statistically stable but may not support a claim about generalizing to new personas. The correct response is not to pick whichever level gives a nicer interval. It is to state the estimand and report sensitivity to plausible clustering choices.

Crossed designs require special care. In many LLM evals, scenario, persona, author, and judge are crossed rather than nested. A persona appears in multiple scenarios; a scenario may be answered by multiple authors; each output may be judged by multiple judges. Standard one-way cluster bootstrap does not automatically handle all dependency axes. Econometric Multi-Way Clustering methods address nonnested two-way or multiway clustering by extending the cluster-robust variance estimator; Cameron, Gelbach, and Miller’s multiway-clustering paper proposes a variance estimator for OLS and nonlinear estimators when clustering is two-way or multiway and nonnested. EconPapers

For LLM eval practice, there are three defensible options:

Design situation	Recommended approach
One dominant dependency axis	Cluster bootstrap on that axis.
Nested design, e.g. records within outputs within scenario	Cluster at the highest level needed for the claim, or use nested/hierarchical bootstrap if lower-level variation is also substantively important.
Crossed design, e.g. persona × scenario × judge	Use multi-way cluster-robust inference, a crossed random-effects model, or report multiple one-way cluster bootstraps as sensitivity analysis.

The cluster choice is a Researcher Degrees of Freedom problem. It affects the result. That does not make cluster bootstrap invalid; it means the cluster rule should be declared before looking at the headline significance pattern, and robustness across plausible clusterings should be reported when the result is close.

Evidence that clustering matters

The strongest evidence base is not from LLM evaluation. It comes from the broader cluster-robust-inference literature.

Bertrand, Duflo, and Mullainathan’s difference-in-differences critique is a classic demonstration of false precision under dependence. In placebo-law simulations using state-level wage data, they report that standard errors were severely biased and that with about 20 years of data, difference-in-differences estimation found a placebo “effect” significant at the 5% level for up to 45% of placebo laws. papers.ssrn.com This is the same failure mode as an LLM eval that finds too many “wins” because repeated rows share scenario, persona, or judge structure.

Cameron and Miller’s guide generalizes the point: in clustered data, default standard errors can overstate precision; cluster-robust methods are appropriate when the number of clusters is large; complications include few clusters, multi-way clustering, and uncertainty about which level to cluster. Colin Cameron's Home Page+1 Their concluding section names the two leading practical difficulties as defining the clusters and dealing with few clusters. Colin Cameron's Home Page

Cameron, Gelbach, and Miller’s bootstrap paper adds a small-cluster inference result. Standard cluster-robust asymptotics assume many clusters; with few clusters, asymptotic tests can over-reject. Their cluster bootstrap-t procedures resample entire clusters and are designed to improve inference in those settings. NBER

The direct LLM-eval evidence is thinner but already suggestive. PsycheEval v0.1 is valuable because it explicitly names the mistake. The pilot had 648 assistant outputs, 1,944 scalar judgments, and 3,456 pairwise judgments, generated from a design with personas, scenarios, author models, and judge models. Ashita Orbis It then warns that the pairwise Wilson intervals assume independent pair records even though persona × scenario × author cells appear in multiple condition-pair rows, inflating effective n. Ashita Orbis

PsycheEval v0.2 is valuable because it operationalizes the fix. The post reports 2,000-resample cluster-bootstrap confidence intervals for the AB/BA-controlled win-rate table, with clustering unit persona × scenario × author. Ashita Orbis It is also methodologically useful because it shows that cluster bootstrap is necessary but not sufficient: two expected C5_CONTRACT headlines collapsed after counterbalanced AB/BA judging, not merely after widening intervals. The v0.2 post says the original pairwise table put the lower-numbered condition almost always in slot A, while C5_CONTRACT was in slot B for 100% of the pairwise records in which it appeared. Ashita Orbis After AB/BA correction, C3 vs C5_CONTRACT had controlled lo_win 0.501 with bootstrap CI [0.436, 0.566], and C4 vs C5_CONTRACT had controlled lo_win 0.482 with bootstrap CI [0.413, 0.548], both reported as collapsed. Ashita Orbis

That example should not be oversold. The public v0.2 post does not prove that cluster bootstrap alone caused the retraction; the dominant issue was position bias, and AB/BA counterbalancing did the causal audit. The lesson is sharper: cluster bootstrap addresses dependence in uncertainty estimation, while AB/BA addresses order bias in measurement. Both are needed in pairwise LLM-as-judge studies.

The position-bias concern is independently supported by LLM-as-judge literature. Zheng et al.’s MT-Bench and Chatbot Arena paper examines position, verbosity, and self-enhancement biases in LLM-as-judge setups. arXiv Shi et al.’s “Judging the Judges” study reports position bias across pairwise and list-wise settings, involving 15 LLM judges, 22 tasks, about 40 solution-generating models, and more than 150,000 evaluation instances; it finds that position bias is not random chance and varies across judges and tasks. arXiv PsycheEval v0.2’s per-judge breakdown is consistent with that broader result: GPT-5.4 was nearly position-neutral, GPT-5.5-xhigh showed strong slot-B preference, and Opus showed moderate slot-B preference in the tested corpus. Ashita Orbis

PsycheEval v0.2 as a methodological case study

PsycheEval v0.2 should be read less as a definitive substantive study and more as a useful Evaluation Audit case. The v0.2 design added a C5_CONTRACT condition, C1_padded and C4_shuffled controls, an anchored 0–10 rubric, harder scenarios, and tri-model authoring across GPT-5.4, GPT-5.5-xhigh, and Opus 4.7. Ashita Orbis The public post says the expected headline “C5_CONTRACT > C3 and C5_CONTRACT > C4” did not survive counterbalanced AB/BA judging. Ashita Orbis

The report’s methodological contribution is the combination of:

• cluster-bootstrap CIs for the corrected pairwise table;

• AB/BA swap rejudging for position-bias correction;

• per-judge bias decomposition;

• explicit retraction of headline findings that failed the audit.

The cluster-bootstrap implementation detail is clear: the AB/BA table uses 2,000-resample cluster bootstrap on the controlled lo_win rate, clustering at persona × scenario × author. Ashita Orbis This directly addresses the v0.1 caveat that Wilson pairwise intervals assumed independence and inflated effective n. Ashita Orbis

The Wilson-vs-cluster-bootstrap contrast is conceptually important even where the public post does not publish a complete side-by-side Wilson table for every v0.2 pair. Wilson intervals answer, “If these rows were independent Bernoulli trials, what interval would we report?” Cluster bootstrap asks, “If these persona × scenario × author cells are the repeatable sampling units, what interval do we get when preserving their internal row structure?” The gap between the two is an overconfidence diagnostic. A large gap says the row-level table is carrying repeated-measures dependence that the naive interval ignores.

PsycheEval v0.2 also shows a limit of pooled cluster bootstrap. The post explicitly says the cluster bootstrap pooled across judges and hid disagreement: the “C5_CONTRACT > C3” aggregate was inflated by two position-biased judges while contradicted by the position-neutral judge. Ashita Orbis This is an important warning for LLM evals that pool judge models. A cluster bootstrap over persona × scenario × author does not automatically protect against judge-family bias. If judges are a source of systematic measurement error, report judge-stratified results or include judge as a clustering/modeling dimension.

Comparison to alternatives

Cluster bootstrap is one tool in a broader family of methods for hierarchical data. It is attractive for LLM evals because many headline metrics are custom statistics rather than simple regression coefficients: win rates, tie-adjusted win rates, red-flag rates, rubric deltas, judge-agreement summaries, AB/BA-controlled rates, and composite safety metrics. Bootstrap can wrap around such statistics without requiring a closed-form variance derivation.

Method	What it does	Strengths	Weaknesses	Good fit for LLM eval?
Naive Wilson/normal CI	Treats rows as independent Bernoulli or normal observations	Simple, familiar, useful as a baseline	Overstates effective n under clustering	Only as a transparency baseline, not final inference for hierarchical evals
Record-level bootstrap	Resamples individual rows	Easy for arbitrary statistics	Same independence mistake as naive CI	Usually wrong when records share outputs/scenarios/judges
Cluster bootstrap	Resamples whole clusters	Model-light; preserves within-cluster dependence; works for custom metrics	Needs enough clusters; cluster choice affects result	Strong default for descriptive hierarchical eval metrics
Cluster-Robust Standard Errors	Regression sandwich variance allowing within-cluster correlation	Handles covariates; familiar in econometrics	Needs many clusters; harder for arbitrary non-regression metrics	Good for regression-style eval analysis
Mixed-Effects Models	Adds random intercepts/slopes for clusters	Estimates variance components; supports partial pooling; efficient if model right	Model-dependent; misspecification can mislead	Good for structured hypothesis testing and variance decomposition
Bayesian Hierarchical Models	Full probabilistic model with priors and partial pooling	Handles sparse cells; gives posterior uncertainty; flexible crossed effects	Requires priors, computation, model checking; can hide sensitivity	Excellent when the model is part of the scientific claim
Randomization/permutation tests	Uses assignment mechanism to build null distribution	Strong when randomization is known	Limited when eval design was observational or post-hoc	Useful for controlled prompt/condition experiments
Multi-way cluster methods	Accounts for nonnested cluster dimensions	Matches crossed designs	More complex; fewer off-the-shelf tools for custom metrics	Important when persona, scenario, author, and judge are crossed

Mixed-effects models are the most natural model-based alternative. The lme4 paper describes linear mixed-effects models as formulas containing both fixed- and random-effects terms, with maximum likelihood or REML estimation. jstatsoft.org In an LLM eval, a mixed model might estimate:

logit P(y=1)=β0+β1⋅condition+uscenario+upersona+uauthor+ujudge.\text{logit},P(y=1) = \beta_0 + \beta_1 \cdot \text{condition}

• u_\text{scenario}
• u_\text{persona}
• u_\text{author}
• u_\text{judge}.logitP(y=1)=β0​+β1​⋅condition+uscenario​+upersona​+uauthor​+ujudge​. This can be more informative than a cluster bootstrap because it estimates where variance comes from. But it assumes the chosen random-effects structure is adequate. Cameron and Miller note the same tradeoff for random effects and hierarchical models in regression: they can give valid and efficient inference if the within-cluster correlation model is correctly specified, but those desirable properties rely on a strong correct-specification assumption. Colin Cameron's Home Page

Bayesian hierarchical models extend this approach by modeling partial pooling directly. The Stan User’s Guide describes hierarchical logistic regression as an intermediate between complete pooling and no pooling, where the degree of pooling is determined by the data and a prior on the amount of pooling. Stan For small LLM evals, Bayesian modeling can be more honest than frequentist large-cluster approximations because it forces explicit priors. But it can also launder subjectivity if priors, likelihood, and posterior checks are not reported.

Cluster-robust standard errors are most appropriate when the estimand is a regression coefficient. Cameron and Miller emphasize that cluster-robust standard errors do not require specifying a within-cluster correlation model, but they require the number of clusters, not merely observations, to go to infinity. Colin Cameron's Home Page This matters for LLM evals with only a handful of personas or judge models. A table with 30,000 rows and 6 personas is not a large-cluster design for persona-level inference.

Design choices for LLM evaluation

Define the estimand before the interval

The cluster level follows the estimand. A study might ask any of the following:

• Does condition C4 beat C1 on this fixed benchmark instance set?

• Does C4 generalize across a population of scenarios?

• Does C4 generalize across personas?

• Does C4 generalize across author models?

• Does the judge ensemble detect a preference, or do individual judges agree?

• Does the result survive pairwise position reversal?

These are different claims. They need different uncertainty estimators.

A fixed-benchmark claim may cluster by benchmark item or output cell. A scenario-generalization claim should treat scenarios as sampled units. A persona-generalization claim must account for persona-level variation. A judge-generalization claim is usually weak unless there are many judges; with three judges, the honest move is judge-stratified reporting, not pretending that three is enough for stable judge-cluster inference.

Preserve paired structure

Pairwise evals often have matched comparisons. If the same persona × scenario × author cell is judged in both original AB and swapped BA order, the bootstrap should resample the matched cell and keep its AB/BA records together. Breaking the pairing loses the design’s strongest control.

For AB/BA-controlled metrics, define the statistic at the matched-cell level where possible:

p^controlled=12(p^AB+p^BA),\hat{p}\text{controlled} = \frac{1}{2}(\hat{p}\text{AB}+\hat{p}_\text{BA}),p^​controlled​=21​(p^​AB​+p^​BA​), or a more explicit estimator that classifies stable condition wins, stable slot wins, ties, and flips. Then bootstrap the matched units.

Treat judge models as measurement instruments, not just rows

LLM-as-judge results are not neutral observations. They are outputs of measurement instruments. PsycheEval v0.2’s result that two judges had strong slot-B bias while one was position-neutral is a concrete example. Ashita Orbis A pooled cluster-bootstrap CI can be precise about the average of biased instruments while obscuring disagreement among instruments.

A good LLM eval report should include:

Report item	Why it matters
Per-judge point estimates	Detects judge-family disagreement.
Per-judge AB/BA bias	Separates condition preference from slot preference.
Pooled estimate	Summarizes the chosen judge ensemble.
Cluster-bootstrap CI	Corrects row-level false precision.
Naive CI	Shows the overconfidence gap.
Cluster count	Tells readers whether the bootstrap has enough independent units.
Cluster definition	Makes the estimand auditable.

Handle ties explicitly

Many pairwise LLM evals include ties. A win-rate statistic can handle ties by excluding them, counting ties as 0.5, or modeling decisive-vote probability separately. Each choice changes the estimand. The bootstrap should recompute the same tie rule inside every replicate, and the report should state it.

Use multi-way sensitivity for crossed factors

If personas, scenarios, authors, and judges are crossed, a single interaction cluster such as persona × scenario × author can be reasonable for output-cell dependence, but it may not cover judge dependence. Conversely, clustering by judge with three judges is unstable. The practical compromise is to report:

1. the primary cluster-bootstrap CI for the main output-cell dependency;

2. judge-stratified point estimates and intervals where possible;

3. scenario-family and persona sensitivity tables;

4. a multi-way cluster-robust or hierarchical model when regression structure is appropriate.

Relationship to power analysis

Cluster bootstrap is not only an analysis tool. It is a power-analysis tool for Evaluation Power Analysis.

A naive power analysis might say: “We need 2,000 pairwise judgments to detect a 5 percentage-point win-rate difference.” That is incomplete. If those 2,000 judgments come from 100 clusters with 20 repeated records each, the power depends on between-cluster variance and within-cluster correlation, not just row count.

A cluster-aware power workflow is:

1. Run a pilot with the intended design.

2. Estimate cluster-level variability for the intended statistic.

3. Simulate new studies by resampling clusters, not rows.

4. Vary the number of personas, scenarios, authors, and judges separately.

5. Compute the probability that the planned interval excludes the null or that the planned decision rule fires.

6. Report minimum detectable effect under cluster-aware assumptions.

The design-effect formula gives the intuition: if clustering increases variance by a factor of 4, then 4,000 rows may have the precision of 1,000 independent rows. CRAN Adding more judge calls per output can reduce measurement noise, but it does not create new scenarios, personas, or author outputs. AB/BA swap calls are often essential for bias correction, but they also do not double the independent sample size; they improve measurement of the same units.

This matters for budget allocation. If cluster-level heterogeneity dominates, spend on more scenarios or personas. If judge noise dominates, spend on more judges or repeated judging. If position bias dominates, spend on AB/BA counterbalancing. If output randomness dominates, sample multiple completions per author-condition cell.

Active critiques and failure modes

Too few clusters

Cluster bootstrap needs enough clusters to approximate the sampling distribution. There is no universal threshold. Cameron and Miller state that “few” has no clear-cut definition and may range from less than 20 to less than 50 clusters in balanced cases. Colin Cameron's Home Page With very few clusters, percentile cluster bootstrap intervals can be unstable or misleading. Cameron and Miller’s simulations also show that some pairs cluster percentile-t bootstraps can fail with very few clusters, while wild cluster bootstrap variants can perform better in some settings. Colin Cameron's Home Page

For LLM evals, this means “clustered by persona” is not enough if there are four personas. In that case, report persona-stratified results and avoid strong population claims about personas. Do not use bootstrap machinery to create a false aura of rigor.

Cluster choice is a researcher degree of freedom

Different cluster levels can produce different intervals. A result may “survive” clustering by persona × scenario × author but fail clustering by scenario family. The choice should be justified by the claim, not by the desired conclusion.

Best practice is to predeclare:

• primary estimand;

• primary cluster unit;

• fallback if clusters are too few;

• sensitivity cluster levels;

• interval method;

• bootstrap replicate count;

• tie handling;

• judge pooling rule.

Cluster bootstrap does not fix bias

Cluster bootstrap estimates sampling variability under the observed design. It does not fix systematic judge bias, length confounds, prompt leakage, data contamination, nonrandom scenario selection, or bad rubric design. PsycheEval v0.2 is a clean warning: cluster bootstrap was present, but pooled judge bias still hid the structural disagreement until AB/BA and per-judge analysis exposed it. Ashita Orbis

Cluster bootstrap can preserve bad structure

Because the cluster bootstrap preserves within-cluster structure, it also preserves artifacts inside clusters. If every C5_CONTRACT comparison puts C5_CONTRACT in slot B, resampling clusters cannot repair that. It will honestly estimate uncertainty around a biased measurement process. Design fixes—randomization, counterbalancing, blinding, judge calibration—must come before inferential fixes.

Crossed dependence is harder than nested dependence

Many tutorials explain cluster bootstrap using one-way clusters, but LLM evals often have crossed structure. Persona and scenario are not always nested; judge and author are usually crossed. A one-way cluster can be a useful approximation, but it should not be represented as a universal correction for every dependency axis.

The target population may be undefined

Cluster bootstrap treats observed clusters as a sample from some population. But many evals hand-curate scenarios, personas, or prompts. If the benchmark is fixed, then inference over a hypothetical scenario population may be inappropriate. In that case, the interval may describe robustness over the constructed benchmark rather than population uncertainty. The report should say which.

Practitioner heuristics

The following heuristics are intentionally conservative:

Heuristic	Rationale
Start with the estimand, not the CSV schema.	The cluster unit is determined by the claim.
When in doubt, cluster at the highest grouped factor that supports the claim.	Avoid counting repeated lower-level rows as independent evidence.
Report both naive and cluster-bootstrap CIs.	The gap makes effective-sample-size inflation visible.
Always report number of clusters.	A cluster-bootstrap CI without cluster count is uninterpretable.
Do not rely on cluster bootstrap with very few clusters.	Use stratified descriptive reporting or model-based analysis with explicit caveats.
For pairwise LLM judging, use AB/BA by default.	Position bias is documented and can reverse headline conclusions.
Keep matched AB/BA records together in the bootstrap.	Preserve the experimental control structure.
Report judge-stratified results.	Pooled judge CIs can hide judge-family bias.
Use multi-way sensitivity for crossed designs.	One-way clustering may miss major dependency axes.
Treat “significance” near the boundary as fragile.	If a result depends on a cluster choice, call it exploratory.

A minimal reporting template:

Metric	Point estimate	Naive CI	Cluster-bootstrap CI	Cluster unit	Clusters	Decision
C4 vs C1 win rate	0.62	[0.58, 0.66]	[0.51, 0.72]	persona × scenario × author	96	suggestive, cluster-wide
C5_CONTRACT vs C3 controlled lo_win	0.50	[0.45, 0.55]	[0.44, 0.57]	persona × scenario × author	96	no detected preference
C5_CONTRACT vs C5 controlled lo_win	0.32 for lower-numbered condition	[0.27, 0.38]	[0.27, 0.38]	persona × scenario × author	96	C5_CONTRACT favored

The exact numbers in the last two rows mirror the style of PsycheEval v0.2’s AB/BA-controlled table, where C3 vs C5_CONTRACT collapsed around 0.501 and C5 vs C5_CONTRACT survived with controlled lo_win 0.323, implying C5_CONTRACT wins 67.7%. Ashita Orbis

What cluster bootstrap contributes to LLM evaluation

Cluster bootstrap changes the epistemic posture of eval reporting. It pushes reports away from “we have 10,000 judgments” and toward “we have 120 independent scenario-persona-author cells, measured by three biased instruments under a counterbalanced design.” That is a better scientific sentence.

Its contribution is especially strong for:

• pairwise preference evaluations;

• LLM-as-judge pipelines;

• synthetic persona studies;

• prompt-condition ablations;

• red-team scenario families;

• rubric scoring with repeated judges;

• benchmark subsets with shared sources or templates;

• studies where outputs are reused across multiple comparisons.

It is less central when the benchmark items are genuinely independent, when each row is a separately randomized unit, or when the goal is purely descriptive over a fixed finite table. Even there, row-level intervals should be labeled as such.

The open question is whether cluster bootstrap becomes default in LLM-eval reporting. It should, but probably not alone. The better default is a bundle:

1. row-level descriptive statistics;

2. naive Wilson or normal CI for transparency;

3. cluster-bootstrap CI for the primary dependency structure;

4. judge-stratified estimates;

5. AB/BA counterbalancing for pairwise judging;

6. sensitivity over plausible cluster levels;

7. explicit power analysis using cluster-level resampling.

That bundle makes LLM evals less flashy and more credible. It will also retract more headlines. PsycheEval v0.2 is instructive because the methodology worked by destroying a planned claim: two expected C5_CONTRACT wins collapsed after AB/BA correction, and the author framed the retraction as the successful part of the pipeline. Ashita Orbis

References and source notes

[Davison & Hinkley 1997 — Bootstrap Methods and their Application]. Canonical monograph on bootstrap standard errors, confidence intervals, significance tests, and applied bootstrap methodology. Cambridge University Press & Assessment

[Field & Welsh 2007 — “Bootstrapping clustered data”]. Statistics paper focused specifically on clustered data bootstraps, including theory and model-dependence of consistency. The Australian National University

[Cameron, Gelbach & Miller 2008 — “Bootstrap-Based Improvements for Inference with Clustered Errors”]. Econometric reference on cluster bootstrap-t procedures for clustered errors and few-cluster inference; NBER working-paper version and publication metadata both consulted. NBER+2NBER+2

[Cameron & Miller 2015 — “A Practitioner’s Guide to Cluster-Robust Inference”]. Practitioner reference for cluster-robust standard errors, cluster choice, few clusters, multi-way clustering, and the consequences of ignoring within-cluster correlation. Colin Cameron's Home Page+2Colin Cameron's Home Page+2

[Bertrand, Duflo & Mullainathan 2004 — “How Much Should We Trust Differences-in-Differences Estimates?”]. Classic false-positive demonstration under serial correlation and clustered dependence in difference-in-differences designs. papers.ssrn.com

[Cameron, Gelbach & Miller 2011 — “Robust Inference With Multiway Clustering”]. Reference for nonnested two-way and multiway cluster-robust inference. EconPapers

[Bates et al. 2015 — “Fitting Linear Mixed-Effects Models Using lme4”]. Reference implementation and explanation for mixed-effects models with fixed and random effects. jstatsoft.org

[Stan User’s Guide — Hierarchical Logistic Regression]. Reference for Bayesian hierarchical logistic regression and partial pooling. Stan

[Zheng et al. 2023 — “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”]. LLM-as-judge paper discussing position, verbosity, and self-enhancement biases. arXiv

[Shi et al. 2024/2025 — “Judging the Judges”]. Systematic study of position bias across LLM judges, tasks, and candidate models. arXiv

[PsycheEval v0.1 — pilot post]. LLM-eval example that used Wilson intervals while explicitly caveating independence violations and anticipating cluster-bootstrap widening. Ashita Orbis

[PsycheEval v0.2 — cluster-bootstrap and AB/BA audit]. LLM-eval example reporting 2,000-resample cluster-bootstrap CIs clustered by persona × scenario × author, while showing that AB/BA position-bias correction collapsed two planned pairwise headlines. Ashita Orbis+3Ashita Orbis+3Ashita Orbis+3

Companion entries

Core theory: Bootstrap Inference, Cluster-Robust Standard Errors, Design Effects and Effective Sample Size, Hierarchical Models, Mixed-Effects Models, Bayesian Hierarchical Models, Multi-Way Clustering

LLM evaluation: LLM Evaluation, LLM-as-Judge Bias, Pairwise Evaluation, AB/BA Counterbalanced Judging, Wilson Confidence Intervals, PsycheEval, Evaluation Audit

Practice: Evaluation Power Analysis, Reporting Standards for LLM Evals, Judge-Stratified Evaluation, Scenario-Family Sensitivity Analysis, Prompt-Condition Ablations

Counterarguments and failure modes: Few-Cluster Inference, Researcher Degrees of Freedom, Construct Validity in Synthetic Evaluations, Position Bias, False Precision, Benchmark Overfitting