arXivPaperNeeds Review

Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.

arXiv · Aug 26, 2026

Open Source Open Markdown Open JSON

Source Summary

Trace Integrity evaluates whether a data agent's computation is executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. On BIRD Mini-Dev, three SQL modes reached only **20–24% answer accuracy** and **39–43% trace-integrity pass rates**.

Practical Implication

For production data agents, retain a structured execution contract linking user intent, schema fields, operator plans, assumptions, queries, verification, and the final answer. Evaluate both output accuracy and whether the recorded computation can actually support that output.

Agent-Ready Context

Trace Integrity evaluates whether a data agent's computation is executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. On BIRD Mini-Dev, three SQL modes reached only **20–24% answer accuracy** and **39–43% trace-integrity pass rates**.

For production data agents, retain a structured execution contract linking user intent, schema fields, operator plans, assumptions, queries, verification, and the final answer. Evaluate both output accuracy and whether the recorded computation can actually support that output.

The reported **CAIT rates were 45.8–59.1%**, showing that answer-only grading can accept unsupported results. This is an empirical demonstration on one benchmark, so the criterion still needs validation across broader data systems and workloads.

Connected Context · Feed7 Judgment

Trace Integrity turns the gap between a correct answer and a supportable computation into an explicit evaluation contract for data agents. Its low joint answer-and-integrity rates provide empirical evidence that answer-only grading can accept unsupported work, while narrowing the conclusion to SQL modes on BIRD Mini-Dev until the criteria are validated across broader systems and legitimate alternative workflows.

The Low Frequency Trap: Video Language Models Fail at Simple Event BookkeepingThe video bookkeeping results independently show that aggregate answer accuracy can improve without faithful event recovery, reinforcing the need to verify intermediate evidence against executable ground truth.Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta SoftwareTrace Integrity supplies a structured implementation of queryable trajectories and final-state inspection for data work, linking plans, schema fields, queries, verification, and answers.Fisher-R1: Training LLM Agents for Reliable Hypothesis TestingFisher-R1 similarly evaluates whether the selected statistical method and inference are valid rather than merely whether code runs, extending supportability checks from SQL traces to hypothesis testing.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioLatchBio adds a boundary condition: strict trace contracts can reject valid scientific work when multiple analysis paths exist, so operator-faithfulness criteria may require domain-expert validation.

Context Map

benchmarkdata#agent-evals#agent-reliability#benchmark-integrity

Uncertainty

The reported **CAIT rates were 45.8–59.1%**, showing that answer-only grading can accept unsupported results. This is an empirical demonstration on one benchmark, so the criterion still needs validation across broader data systems and workloads.