arXivPaperNeeds Review

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Change2Task converts merged PR history into executable coding-agent tasks on current repository states. It could expand continuous eval sets while reducing stale environments and repeated setup.

arXiv · Jul 30, 2026

Open Source Open Markdown Open JSON

Source Summary

Change2Task turns merged pull requests into verified tasks on modern repository revisions using **Patch Reversal, Code Mapping, or Agent Reconstruction**. Across **1,130 eligible changes**, it constructed verified tasks at a 79.6% rate.

Practical Implication

Use repository history as renewable eval data, but preserve the full lifecycle: healthy base, reconstructed task state, and restored state. The five tested families include bug fixes, features, tests, API migrations, and security repairs.

Agent-Ready Context

Change2Task turns merged pull requests into verified tasks on modern repository revisions using **Patch Reversal, Code Mapping, or Agent Reconstruction**. Across **1,130 eligible changes**, it constructed verified tasks at a 79.6% rate.

Use repository history as renewable eval data, but preserve the full lifecycle: healthy base, reconstructed task state, and restored state. The five tested families include bug fixes, features, tests, API migrations, and security repairs.

It recovered **29.2% more verified tasks** than a PR-based baseline and reached up to 98.0% matched outcome agreement. These are construction results from the evaluated repositories, not proof that generated tasks represent every real maintenance workload.

Connected Context · Feed7 Judgment

Change2Task expands renewable coding-agent evaluation beyond directly replaying old PRs by reconstructing verified tasks on newer repository states across several maintenance families. Against prior benchmark concerns, its construction success improves supply, not validity by itself: task alignment, history leakage, environment noise, and workload coverage still require separate controls.

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like BenchmarksPAIChecker identifies issue–patch–test mismatch in PR-derived tasks, so Change2Task’s reconstructed lifecycle should still be checked for semantic alignment before scores are trusted.Reward hacking is swamping model intelligence gainsBecause Change2Task derives tasks from repository history, sealed evaluation must prevent agents from retrieving the originating fix instead of solving the reconstructed task.DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, DatacurveDeepSWE’s original tasks reduce contamination, whereas Change2Task favors scalable reuse of historical changes; together they expose a tradeoff between renewable task volume and independence from public fixes.Quantifying infrastructure noise in agentic coding evalsChange2Task verifies task-state construction, but infrastructure-noise evidence shows that comparable agent scores also require controlled runtime resources after tasks are built.

Context Map

benchmarkcodingsecurity#coding-agents #agent-evals#benchmark-integrity

Uncertainty

It recovered **29.2% more verified tasks** than a PR-based baseline and reached up to 98.0% matched outcome agreement. These are construction results from the evaluated repositories, not proof that generated tasks represent every real maintenance workload.