CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
CordisBench finds that models lose reliability as harness lifecycle interactions grow, while deterministic execution semantics can replace costly reasoning on controlled cases.
CordisBench contains **1,200 questions** about dependency propagation, teardown order, final state, and executable reconfiguration in dynamic agent harnesses. It tests three efficiency-oriented models across **2 to 32 relevant interactions** with deterministic task-specific scoring.
Do not let an agent reason unaided about complex plugin cleanup when the lifecycle can be encoded and executed. Use explicit dependency state, deterministic teardown checks, and a reference semantics as validation; reserve higher reasoning effort for cases that cannot be mechanically resolved.
CordisBench contains **1,200 questions** about dependency propagation, teardown order, final state, and executable reconfiguration in dynamic agent harnesses. It tests three efficiency-oriented models across **2 to 32 relevant interactions** with deterministic task-specific scoring. Do not let an agent reason unaided about complex plugin cleanup when the lifecycle can be encoded and executed. Use explicit dependency state, deterministic teardown checks, and a reference semantics as validation; reserve higher reasoning effort for cases that cannot be mechanically resolved. Extra inference improved some models, but cost rose sharply: GPT-5.6 Luna used **nearly 3,000 reasoning tokens per question** at medium effort on the 16-interaction subset. The finite semantics matched Cordis on all **528 executable questions**, but that finding is limited to these controlled instances.
This introduces lifecycle reasoning as a distinct harness-evaluation target and shows when mechanical semantics should replace costly unaided inference. It sharpens architecture-aware eval guidance into dependency propagation, teardown, final-state, and reconfiguration checks, while the token-cost result limits the value of simply increasing reasoning effort. The reference-semantics result remains confined to controlled executable cases.