arXivPaperNeeds Review

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Change2Task converts merged PR history into executable coding-agent tasks on current repository states. It could expand continuous eval sets while reducing stale environments and repeated setup.

arXiv · Jul 30, 2026

Open Source Open Markdown Open JSON

Source Summary

Change2Task turns merged pull requests into verified tasks on modern repository revisions using **Patch Reversal, Code Mapping, or Agent Reconstruction**. Across **1,130 eligible changes**, it constructed verified tasks at a 79.6% rate.

Practical Implication

Use repository history as renewable eval data, but preserve the full lifecycle: healthy base, reconstructed task state, and restored state. The five tested families include bug fixes, features, tests, API migrations, and security repairs.

Agent-Ready Context

Change2Task turns merged pull requests into verified tasks on modern repository revisions using **Patch Reversal, Code Mapping, or Agent Reconstruction**. Across **1,130 eligible changes**, it constructed verified tasks at a 79.6% rate.

Use repository history as renewable eval data, but preserve the full lifecycle: healthy base, reconstructed task state, and restored state. The five tested families include bug fixes, features, tests, API migrations, and security repairs.

It recovered **29.2% more verified tasks** than a PR-based baseline and reached up to 98.0% matched outcome agreement. These are construction results from the evaluated repositories, not proof that generated tasks represent every real maintenance workload.

Connected Context · Feed7 Judgment

This adds a scalable source of renewable coding-agent evaluations, but it does not make PR-derived tasks trustworthy by construction. Its verified state-restoration lifecycle improves task supply and reproducibility; prompt–oracle alignment, history isolation, infrastructure controls, and workload-coverage checks remain separate prerequisites before comparing agents.

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like BenchmarksPAIChecker provides the issue–patch alignment gate that Change2Task’s successful reconstruction does not itself establish.Reward hacking is swamping model intelligence gainsBecause Change2Task derives tasks from repository history, the reported history-retrieval shortcut makes sealed task environments an implementation requirement.Quantifying infrastructure noise in agentic coding evalsVerified task construction controls repository state, while Anthropic’s findings show that resource configuration must still be controlled before score differences are meaningful.DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, DatacurveDeepSWE offers contamination-resistant original tasks as a complement to Change2Task’s scalable PR-derived coverage, helping test whether results depend on the task source.

Context Map

benchmarkcodingsecurity#coding-agents #agent-evals#benchmark-integrity

Uncertainty

It recovered **29.2% more verified tasks** than a PR-based baseline and reached up to 98.0% matched outcome agreement. These are construction results from the evaluated repositories, not proof that generated tasks represent every real maintenance workload.