arXivPaperNeeds Review

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.

arXiv · Sep 3, 2026

Open Source Open Markdown Open JSON

Source Summary

Across **52,988 attempts**, same-window rankings reached Spearman **0.400** against a required 0.90; byte-identical next-day replays reached **0.78** against 0.99. Label mapping, gaps below the noise floor, and changing rankings for identical inputs explained the failures.

Practical Implication

If an LLM judge gates agent releases or training data, test it as a measurement instrument first. Repeat identical requests, estimate the noise floor, retain execution records, and run a small pilot; the authors say roughly **2% of call volume** would have revealed both unreachable gates.

Agent-Ready Context

Across **52,988 attempts**, same-window rankings reached Spearman **0.400** against a required 0.90; byte-identical next-day replays reached **0.78** against 0.99. Label mapping, gaps below the noise floor, and changing rankings for identical inputs explained the failures.

If an LLM judge gates agent releases or training data, test it as a measurement instrument first. Repeat identical requests, estimate the noise floor, retain execution records, and run a small pilot; the authors say roughly **2% of call volume** would have revealed both unreachable gates.

Changing metrics, sampling, waiting, or switching among four tested providers did not repair reliability on the tested grids. The finding concerns externally observed behavior on shared endpoints; self-hosting helped only while the server was quiet, so it is not a general guarantee.

Connected Context · Feed7 Judgment

This upgrades benchmark-noise concerns into a preregistered measurement failure: identical inputs on shared endpoints were not stable enough for the specified ranking and replay gates. It makes repeatability testing, noise-floor estimation, retained execution records, and a small pilot prerequisites before an LLM judge controls releases or training data, while avoiding a general claim that self-hosting fixes the problem.

Phantom Gains: Auditing Self-Improvement Against a Measured NullBoth require an empirical null before interpreting evaluation changes: Phantom Gains measures baseline transition noise, while this study measures judge repeatability and ranking noise under identical requests.Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasThe mechanistic study identifies systematic judge bias; this study adds a separate operational failure mode in which black-box judgments and rankings vary across identical calls.What Do Compliance Detectors Read? An Audit of Activation Probes and Guard ModelsCounterfactual policy swaps test whether a compliance detector follows the governing rule, but this Signal shows that such validity tests should be preceded by repeatability checks on identical inputs.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioLatchBio shows that brittle graders can reject valid scientific paths; this study adds that even unchanged inputs may receive unstable judgments, further strengthening the case for human validation before evaluator deployment.

Context Map

benchmarkresearch#agent-evals#benchmark-integrity#agent-reliability

Uncertainty

Changing metrics, sampling, waiting, or switching among four tested providers did not repair reliability on the tested grids. The finding concerns externally observed behavior on shared endpoints; self-hosting helped only while the server was quiet, so it is not a general guarantee.