The direct answer
Map each judge’s positional choice back to the underlying answer identity before combining reversed presentations. Preserve order conflicts, genuine ties, invalid outputs and missing judgments as distinct outcomes under a predeclared rule. Agreement across two orders supports only that case’s consistency; it does not establish correctness, safety or a general model ranking.
Evaluate the judge before treating it as a ranking instrument
A team compares two explanations with an AI judge. It shows the answers twice, reversing their positions, and receives “Left” both times. Counting two matching words as agreement would conceal that the judge selected different answers.
The one-task comparison guide supplies the broader task, rubric and cost design. This worksheet adds a narrower measurement question: whether a pairwise preference stays attached to the same answer when presentation changes. It does not select a judge provider or run a model comparison.
Use deterministic checks wherever the task admits them. A language-model preference may help review an open-ended explanation, but it must not replace an independently established reference or actual application outcome.
Use the primary research at its historical scope
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, the retrieved arXiv v4 dated 24 December 2023, examines position and verbosity limitations under its reported evaluation setups. Its conservative approach presents both answer orders and declares a win when the same answer is preferred in both. The paper also cautions that high consistency need not imply high accuracy.
Those findings motivate an instrument check, not a claim about every current judge. This guide quotes no agreement rates, current model capabilities or production benchmark result. Our ledger below deliberately preserves an order conflict separately from a genuine tie; that is an original reporting policy, not a requirement imposed by the paper.
Freeze the answer identities before assigning positions
Invent a short reference packet: “Every release candidate requires editor review. Confidence never substitutes for that review.” The task is to explain that rule plainly, without approving or performing a release.
Authored teaching answer A says, “An editor must review every release candidate; confidence does not waive review.” Teaching answer B says, “High-confidence candidates are automatically approved.” An independent human key marks the invented exemption in B as unsupported. These are fabricated alternatives, not model outputs.
Give a future judge the permitted task, reference packet, rubric and displayed answers, but not the expected winner identity. Hide candidate-model labels in this proposed input. Blinding is not an anonymity guarantee: answer style or content may still reveal clues. Keep the original answer bytes and rubric unchanged between presentations; do not improve an answer while supposedly changing only its position.
| Presentation | Left slot | Right slot | Trusted association |
|---|---|---|---|
| O-1 | Answer A | Answer B | Left→A; Right→B |
| O-2 | Answer B | Answer A | Left→B; Right→A |
The evaluator owns this map. Neither a candidate answer nor the judge’s explanation may rewrite it. Bind each recorded judgment to its presentation ID, pair ID and rubric revision before aggregation.
Remap first, then apply a declared disposition
For the ordinary conditional positive, assume the inert labels Left for O-1 and Right for O-2. Both map to A. This is one order-consistent preference in the teaching ledger, not an observed judgment or an experiment showing the judge is reliable.
Left followed by Left instead maps to A followed by B. Preserve the conflict rather than interpreting repeated positional words as repeatable answer preference. Do not resolve it by keeping the preferred ordering or asking the candidate which presentation should count.
For this worksheet accept exactly one label: Left, Right, Tie or Abstain, associated with its declared presentation. Invalid format, unavailable response and explicit abstention need their own recorded categories. Our policy produces a stable winner only when both valid presentations select the same answer. It keeps a genuine two-order tie distinct from a mixed or conflicting result.
Retain both planned presentation slots even when one judgment is missing. A later summary must disclose planned, valid, missing, invalid, tied and conflicting outcomes rather than silently shrinking its denominator. No numeric consistency estimate is calculated here.
Walk eight inert preference cards
All cards are PROPOSED_UNRUN. The choice labels below are assigned to demonstrate bookkeeping; no judge produced them.
| Card | Assigned condition | Proposed disposition |
|---|---|---|
| 1. Consistent answer identity | O-1 Left; O-2 Right | Map A/A; conditionally record one stable A preference for this pair. |
| 2. Same position twice | O-1 Left; O-2 Left | Map A/B; record order conflict, not two votes for A. |
| 3. Genuine tie | Tie in both presentations | Record stable tie separately; do not choose by position. |
| 4. Mixed tie and selection | O-1 Tie; O-2 Right | Record mixed outcome under the declared policy; no silent stable win. |
| 5. Missing or invalid judgment | One slot unavailable or outside the output contract | Keep its planned slot and failure category; do not call it a tie. |
| 6. Untrusted presentation map | A candidate or judge supplies replacement identity metadata | Reject that association; aggregate only through the trusted map. |
| 7. Changed pair or rubric | Answer text or criterion differs between orders | Not the frozen-pair check; require a new manifest. |
| 8. Consistent but wrong | Both presentations prefer B | Record consistency and disagreement with the human key separately; do not waive editor review. |
Keep a consistent preference separate from a release decision
One two-order conflict does not by itself establish a causal position bias; sampling variability and other conditions remain possible explanations. Conversely, stable A/A or B/B does not establish judge accuracy across tasks. Representative pairs, rubric quality, applicable reference evidence, repeated-trial design and uncertainty need separate review.
The holdout guide still governs exposure. Do not tune a judge or candidate against these published teaching cases and claim an untouched final test. The evaluator’s actual configuration, authorized data handling, bounded context and complete identity mapping also remain unimplemented prerequisites; unknowns block operational claims.
For these proposed cases, no judge, model, API, scoring, remapping, metric or ranking experiment was executed. Boundary remains prelaunch with planned request-path checks, not a shipped evaluation service. Its existing synthetic policy evidence concerns different behavior. A judge preference never authorizes a tool effect or certifies that one occurred.
Primary sources
Official references supporting the guidance. Examples in this guide are synthetic and are not reports of a production incident.