Jerry Li

Research

Verifying the Second Agent

Controlled Experiments on Recursive Delegation in Coding Agents

Jerry Li, sole author. Accepted at two NeurIPS 2026 workshops in Sydney: Who Verifies the Agents? Toward Reliable Agent Development, and Managing Agents that Manage Agents.

The question

A coding agent can do a task itself, or spawn other agents, hand them pieces, and merge what comes back. Does a tree of agents that costs several times as much deliver more correct work?

The experiment

Two campaigns on DeepSWE: repository repairs from public GitHub projects in five languages, 97 of 113 tasks gradable. A fix counts only if an external verifier passes every failing test and breaks none. Every comparison is between arms on the same task, with one model (gpt-5.6-luna), fixed prompts, tools and per-agent turn budget, and a 6,000,000-token cap that never bound. Run order is randomised; intervals are bootstrapped over repositories.

The depth campaign varies the tree: one agent, then balanced trees of 3, 7 and 15. The return campaign fixes it at seven and changes only what workers hand back.

Fig. 1Trees cost 3 to 12 times more, with no resolved gain
ArmTree only the root delivers codeCost relative to one agent1×5×10×Return paired change in verified success−20−10+10 pp0lossone agentgainno gainresolved1 agentno delegation1× tokens1× wall clockmean 252,704 billable tokens · 1× costreference3 agentsdepth 1194 patches handed up40.2% refused2.73×1.42×mean 621,968 billable tokens · 2.43× cost−11.54 pp[−20.93, −2.05]78 tasksexcludes zero7 agentsdepth 2delegatesreturns a patch516 patches handed up36.0% refused5.98×3.43×mean 1,427,748 billable tokens · 5.64× cost−4.00 pp[−15.09, +8.00]50 tasksincludes zero15 agentsdepth 31,316 patches handed up31.9% refused12.50×5.08×mean 2,603,307 billable tokens · 11.39× cost−6.03 pp[−16.96, +5.56]58 tasksincludes zeroPer verified fix, on the same 58 tasks$0.58 one agent → $6.92 15 agentstaskpatchrefused
TreeCost relative to one agent1×5×10×1 agent no delegation1× tokens1× wall clock3 agents depth 1194 patches up, 40.2% refused2.73×1.42×7 agents depth 2516 patches up, 36.0% refused5.98×3.43×15 agents depth 31,316 patches up, 31.9% refused12.50×5.08×taskpatchrefused
One row per arm of the depth campaign: one agent, then balanced trees of 3, 7 and 15. Bars are tokens relative to one agent; the thin line is wall-clock time. Dots are the paired change in verified success with 95% intervals, each on its own paired tasks. No tree resolves a gain; the 3-agent tree resolves a loss.

1. Trees cost more, with no resolved gain

Trees of 7 and 15 agents spent 5.98 and 12.50 times the tokens of one agent. Verified success did not follow: −4.00 points (95% interval −15.09 to +8.00) at 7 agents and −6.03 (−16.96 to +5.56) at 15, and the 3-agent tree was resolvably worse at −11.54 (−20.93 to −2.05). A verified fix cost $0.58 with one agent and $6.92 with 15.

2. The tree repeats its own work

Explicit coordination is under a fifth of a percent of the tokens, so the cost is not overhead. Sibling agents touch the same files more often than the task alone predicts, by 9.8 points (5.0 to 15.1) at 7 agents, and 149 of 165 patches a parent refused, 90.3%, collide line for line when replayed.

Fig. 2The tree repeats its own work
Three panels at increasing granularity. (a) Compute: of the tokens beyond the single-agent baseline, the 7-node tree spent 58.7% in workers with at least one applied patch, 41.2% in workers whose every patch was refused, and 0.17% on explicit orchestration; the 15-node tree spent 68.2% in workers with at least one applied patch, 31.6% in workers whose every patch was refused, and 0.17% on explicit orchestration. (b) Files: sibling agents touch the same files more often than a task-conditioned null predicts, by +9.8 percentage points at 7 nodes (95% interval +5.0 to +15.1) and +7.4 percentage points at 15 nodes (95% interval +3.5 to +11.7). (c) Lines: of 165 refused patches that could be replayed with a three-way merge, 149 (90.3%) collide at the line level and 16 are disjoint.7 nodes: 58.7% retained-class, 41.2% unapplied15 nodes: 68.2% retained-class, 31.6% unapplied7 nodes: +9.8 pp over the null [+5.0, +15.1]15 nodes: +7.4 pp over the null [+3.5, +11.7]149 of 165 collide at line level (90.3%)16 of 165 are disjointThree panels at increasing granularity. (a) Compute: of the tokens beyond the single-agent baseline, the 7-node tree spent 58.7% in workers with at least one applied patch, 41.2% in workers whose every patch was refused, and 0.17% on explicit orchestration; the 15-node tree spent 68.2% in workers with at least one applied patch, 31.6% in workers whose every patch was refused, and 0.17% on explicit orchestration. (b) Files: sibling agents touch the same files more often than a task-conditioned null predicts, by +9.8 percentage points at 7 nodes (95% interval +5.0 to +15.1) and +7.4 percentage points at 15 nodes (95% interval +3.5 to +11.7). (c) Lines: of 165 refused patches that could be replayed with a three-way merge, 149 (90.3%) collide at the line level and 16 are disjoint.7 nodes: 58.7% retained-class, 41.2% unapplied15 nodes: 68.2% retained-class, 31.6% unapplied7 nodes: +9.8 pp over the null [+5.0, +15.1]15 nodes: +7.4 pp over the null [+3.5, +11.7]149 of 165 collide at line level (90.3%)16 of 165 are disjoint
Three measurements at increasing granularity, not a causal chain. Retained-class in (a) counts all of a worker's tokens if any of its patches was applied, so it is an upper bound. Each cell in (c) is one refused patch, replayed with a three-way merge.

If merging is the bottleneck, one writer should fix it. In the return campaign workers handed back written findings and only the root wrote code. That removed every shared write and moved verified success by +1.25 points (−9.76 to +12.82). The design had 80% power at a 16.1-point effect, so this is unresolved, not proof of no effect.

Fig. 3One writer, same tree
Two identical seven-node trees. In arm B, shared write, every worker returns a patch, so all seven nodes can change the delivered code. In arm C, findings return, workers return written findings and only the root writes code. Cost per attempt: A, single agent, $0.17; B, shared write, $0.97; C, findings return, $1.03. Cost difference C minus B: +0.051 dollars per attempt, 95% interval −0.005 to +0.110. Primary contrast, verified success of C minus B: +1.25 percentage points, 95% interval −9.76 to +12.82, p = 0.925. A bracket marks the 16.1 point effect the design had 80% power to detect. The interval crosses zero, so the effect is unresolved. 40 tasks over 38 repositories, two attempts each.A, one agent: $0.17 per attempt [0.15, 0.20]B: $0.97 [0.86, 1.08], −8.8 pp vs AC: $1.03, −7.50 pp vs A [−15.85, +1.16]C − B cost: +$0.051 [−0.005, +0.110]C − B: +1.25 pp [−9.76, +12.82], p = 0.925Two identical seven-node trees. In arm B, shared write, every worker returns a patch, so all seven nodes can change the delivered code. In arm C, findings return, workers return written findings and only the root writes code. Cost per attempt: A, single agent, $0.17; B, shared write, $0.97; C, findings return, $1.03. Cost difference C minus B: +0.051 dollars per attempt, 95% interval −0.005 to +0.110. Primary contrast, verified success of C minus B: +1.25 percentage points, 95% interval −9.76 to +12.82, p = 0.925. A bracket marks the 16.1 point effect the design had 80% power to detect. The interval crosses zero, so the effect is unresolved. 40 tasks over 38 repositories, two attempts each.A, one agent: $0.17 per attempt [0.15, 0.20]B: $0.97 [0.86, 1.08], −8.8 pp vs AC: $1.03, −7.50 pp vs A [−15.85, +1.16]C − B cost: +$0.051 [−0.005, +0.110]C − B: +1.25 pp [−9.76, +12.82], p = 0.925
Squares are patches, dots are written findings. The bracket marks the 16.1 pp effect the design had 80% power to detect. The interval crosses zero, so the effect is unresolved.

3. No tested router resolved an advantage

Delegating only where it helps would rescue the average. Routers trained on 23 features known before a run and tested on unseen repositories did not: none beat the best constant choice with an interval excluding zero (best +0.037, −0.036 to +0.125), and one fitted router chose the single agent for all 39 tasks.

Fig. 4Could a router pick the right tasks?
Static pre-task routers, tested on unseen repositories: the verified solve rate of the tasks each policy would choose. Depth campaign arms, 41 tasks, 37 repositories, with each policy's difference from the best constant and the 95% interval of that difference: D threshold 30.5%, +3.7 points, −3.6 to +12.5; best constant 26.8%, the reference; ridge treatment learner 26.8%, 0.0 points, −11.1 to +11.5; ridge outcome learner 24.4%, −2.4 points, −9.8 to +5.1; difficulty only 24.4%, −2.4 points, −7.3 to 0.0; always 7-node shared write 23.2%, −3.7 points, −16.7 to +10.3; boosted treatment learner 13.4%, −13.4 points, −23.2 to −5.0. No policy's advantage over the constant has an interval excluding zero; the one interval that excludes zero is the boosted treatment learner's, on the losing side. Return campaign arms, 39 tasks, 37 repositories, with the 95% interval of each value: oracle benchmark 38.5%, 25.6 to 52.6 (not runnable); ridge treatment learner 28.2%, 15.4 to 43.2; best constant 28.2%, 15.0 to 42.5; always single agent 28.2%, 15.5 to 42.1; D threshold 20.5%, 10.3 to 32.1; always findings return 20.5%, 11.5 to 30.8; always shared write 19.2%, 9.8 to 30.0. On the return arms the fitted ridge router chose the single agent on all 39 tasks, so it equals the constant exactly.D threshold: +3.7 pp [−3.6, +12.5]best constant: 26.8%, the referenceridge treatment: 0.0 pp [−11.1, +11.5]ridge outcome: −2.4 pp [−9.8, +5.1]difficulty only: −2.4 pp [−7.3, 0.0]always 7-node: −3.7 pp [−16.7, +10.3]boosted treatment: −13.4 pp [−23.2, −5.0]oracle: 38.5% [25.6, 52.6], not runnableridge treatment: 28.2% [15.4, 43.2]best constant: 28.2% [15.0, 42.5]always single agent: 28.2% [15.5, 42.1]D threshold: 20.5% [10.3, 32.1]findings return: 20.5% [11.5, 30.8]shared write: 19.2% [9.8, 30.0]ridge router: 39 of 39 to one agentStatic pre-task routers, tested on unseen repositories: the verified solve rate of the tasks each policy would choose. Depth campaign arms, 41 tasks, 37 repositories, with each policy's difference from the best constant and the 95% interval of that difference: D threshold 30.5%, +3.7 points, −3.6 to +12.5; best constant 26.8%, the reference; ridge treatment learner 26.8%, 0.0 points, −11.1 to +11.5; ridge outcome learner 24.4%, −2.4 points, −9.8 to +5.1; difficulty only 24.4%, −2.4 points, −7.3 to 0.0; always 7-node shared write 23.2%, −3.7 points, −16.7 to +10.3; boosted treatment learner 13.4%, −13.4 points, −23.2 to −5.0. No policy's advantage over the constant has an interval excluding zero; the one interval that excludes zero is the boosted treatment learner's, on the losing side. Return campaign arms, 39 tasks, 37 repositories, with the 95% interval of each value: oracle benchmark 38.5%, 25.6 to 52.6 (not runnable); ridge treatment learner 28.2%, 15.4 to 43.2; best constant 28.2%, 15.0 to 42.5; always single agent 28.2%, 15.5 to 42.1; D threshold 20.5%, 10.3 to 32.1; always findings return 20.5%, 11.5 to 30.8; always shared write 19.2%, 9.8 to 30.0. On the return arms the fitted ridge router chose the single agent on all 39 tasks, so it equals the constant exactly.D threshold: +3.7 pp [−3.6, +12.5]best constant: 26.8%, the referenceridge treatment: 0.0 pp [−11.1, +11.5]ridge outcome: −2.4 pp [−9.8, +5.1]difficulty only: −2.4 pp [−7.3, 0.0]always 7-node: −3.7 pp [−16.7, +10.3]boosted treatment: −13.4 pp [−23.2, −5.0]oracle: 38.5% [25.6, 52.6], not runnableridge treatment: 28.2% [15.4, 43.2]best constant: 28.2% [15.0, 42.5]always single agent: 28.2% [15.5, 42.1]D threshold: 20.5% [10.3, 32.1]findings return: 20.5% [11.5, 30.8]shared write: 19.2% [9.8, 30.0]ridge router: 39 of 39 to one agent
Stems run from the best constant policy to each router's verified solve rate on unseen repositories; depth-arm whiskers are 95% intervals of the difference. Return-arm intervals are on the values; pick a row to read them. The oracle is not runnable: a null with identical arms reproduces 81% of its gap.

Scope and limits

For this model, harness, benchmark and budget, one agent is the defensible default. The cost multiple is precise; the correctness return is bounded but unresolved, and a larger study could still find a modest effect either way. Nothing here speaks to other models, flat teams or other ways to merge. A harness that watches for overlap once workers start could stop paying for branches that find nothing new. I did not build that controller.

Cite

The camera-ready version is not public yet. For the manuscript, email me.

@inproceedings{li2026second,
  title     = {Verifying the Second Agent: Controlled Experiments on Recursive Delegation in Coding Agents},
  author    = {Li, Jerry},
  booktitle = {NeurIPS 2026 Workshop: Who Verifies the Agents? Toward Reliable Agent Development},
  year      = {2026},
  note      = {Also accepted at the NeurIPS 2026 Workshop: Managing Agents that Manage Agents}
}

Back to research