arXivPaperNeeds Review

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

Mechanistic tests suggest two open LLM judges route local errors early and form ratings late, giving eval builders concrete layers and components to probe for judge failures.

arXiv · Sep 1, 2026

Open Source Open Markdown Open JSON

Source Summary

Across **eight perturbation attacks**, Themis and Prometheus showed a two-stage judging process: attention below layer 15 compared local errors and routed evidence to the last input position, while later MLPs integrated it into a rating. Decisions crystallized at **layer 26 for Themis** and **layer 25 for Prometheus**.

Practical Implication

If an LLM judge drives agent evals or training, inspect intermediate behavior rather than trusting its scalar score alone. Controlled corruptions, causal tracing, vocabulary projection, and component knockouts can reveal whether the judge notices an error and where that signal becomes a verdict.

Agent-Ready Context

Across **eight perturbation attacks**, Themis and Prometheus showed a two-stage judging process: attention below layer 15 compared local errors and routed evidence to the last input position, while later MLPs integrated it into a rating. Decisions crystallized at **layer 26 for Themis** and **layer 25 for Prometheus**.

If an LLM judge drives agent evals or training, inspect intermediate behavior rather than trusting its scalar score alone. Controlled corruptions, causal tracing, vocabulary projection, and component knockouts can reveal whether the judge notices an error and where that signal becomes a verdict.

The experiments cover summarization quality in two evaluator models, with one same-scale base-model control. The reported mechanisms therefore should not be assumed to transfer unchanged to other judge architectures, tasks, or rubric formats.

Connected Context · Feed7 Judgment

This adds a mechanistic audit layer to prior warnings about LLM judges: a scalar failure may reflect missed local evidence, failed routing, or faulty late-stage integration, and those stages can be separated experimentally. The result strengthens the case for inspecting judge internals and controlled corruptions, but narrows that evidence to two summarization evaluators rather than establishing a universal judging circuit.

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasBoth locate judge failures in internal representations; this Signal adds a staged account of how error evidence is routed and converted into ratings, complementing the candidate’s steerable bias directions.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioThe candidate shows that deterministic graders can reject valid scientific work; this Signal offers causal tracing and component knockouts as ways to diagnose whether such a judge noticed relevant evidence before issuing the wrong verdict.Phantom Gains: Auditing Self-Improvement Against a Measured NullThey address different prerequisites for trustworthy evaluation: this Signal audits how a judge forms a verdict, while the candidate requires a measured statistical null before score changes are interpreted as improvement.Eval awareness in Claude Opus 4.6’s BrowseComp performanceThe candidate exposes leakage from the evaluated agent into benchmark answers, whereas this Signal examines failure inside the evaluator; together they separate compromised test conditions from defective score formation.

Context Map

benchmarkresearch#agent-evals#benchmark-integrity

Uncertainty

The experiments cover summarization quality in two evaluator models, with one same-scale base-model control. The reported mechanisms therefore should not be assumed to transfer unchanged to other judge architectures, tasks, or rubric formats.