Can LLMs build the systems that build capability?
A plain-English deep dive into a benchmark that evaluates the agent harness itself: the loop, tools, context, memory, recovery and verification around a model.
Based on “HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?” by Yuhao Wu et al. · arXiv:2609.01437v1
The model isn’t the whole agent.
HarnessDev asks whether an LLM can create and improve the software layer that turns its outputs into reliable actions.
Evaluate the scaffold
Instead of only scoring an answer, freeze the harness and reuse it across downstream tasks.
Capability can live outside weights
Tool policy, context handling, retries, state and verification can change outcomes even when the model stays identical.
Improvement is not the same as generalisation
Visible feedback gains often shrink on held-out tasks, and transfer across runtime models is fragile.
Practical takeaway: treat your harness like a product and an experimental artefact. Version it, test it, observe it, and measure cost as well as success.
Plain English
What a harness is, why ordinary agent benchmarks hide it, and where this becomes useful.
A harness is the operating system around the model.
It decides what the model sees, which tools it can call, when it should retry, how it carries state, and what counts as done.
Most benchmarks start after the hard systems work is done.
They give every model the same task, reward and execution scaffold. That is useful for controlled comparisons, but it treats the harness as background configuration.
Fixed harness
- Tools already wired
- Context policy selected
- Retries and stopping rules chosen
- Verifier supplied
System builder
- Starts from a weak but runnable seed
- Builds a complete execution system
- Freezes the artifact
- Runs it on unseen tasks
The shift is subtle but important: the evaluated artefact is persistent runnable infrastructure, not a one-off answer.
Two stages: create it, then live with it.
HarnessDev separates the ability to build a harness from the ability to maintain one as real execution feedback arrives.
Weak seed → runnable H
A creator LLM receives a task specification and 1–3 development cases. It adds the missing control layer, then the harness is frozen.
H₀ → H₁ → … → Hₙ
The creator revises its own harness from downstream feedback, with every official version frozen, tested and audited.
Six control modules turn a model into an agent.
Select a module to see the job it performs. The paper’s interface makes these functions inspectable and testable.
When the harness becomes the product.
These are adapted use cases, not benchmark results. The pattern is useful wherever the same model must act repeatedly under changing tools, data and constraints.
Coding agent
Use verification gates, patch tracking and rollback to prevent “looks done” from becoming “is done”.
Research analyst
Make source retrieval, deduplication, citation evidence and stopping rules explicit and auditable.
Lab notebook agent
Persist experiment state, record provenance and recover cleanly after a long-running tool failure.
ML experiment agent
Separate orchestration from the executor so you can compare cost, reproducibility and final artefacts.
Customer workflow
Keep a stable contract while adapting tools, permissions and compliance rules to each deployment.
Agent platform
Turn traces and failures into targeted harness changes, then check whether they transfer beyond the feedback set.
Deep dive
The protocol, the evidence, the failure modes and the uncomfortable portability question.
Separate the builder from the runner.
The central experimental discipline is role separation. The creator edits the harness; the executor uses it only after it is frozen; the evaluator scores the resulting task artefact.
Works in a mutable development workspace. Reads cases, edits code and responds to visible feedback.
Solves downstream tasks using the frozen harness. In Self-Eval, LE = LC; in Unified-Eval, it is fixed.
Reads the authoritative domain artefact: repository diff, submission, prose, or cited answer.
Why this matters: if the same model builds and runs the harness, a score reflects model capability, harness design and their compatibility together.
Start with enough floor to measure design.
The shared seed is runnable and auditable, but deliberately not intelligent. It parses inputs, exposes passive primitives and writes audit files. It does not solve the task.
Improve the harness without mistaking noise for progress.
Evolution exposes downstream execution feedback, but holds out a disjoint set for post-freeze generalisation. The creator gets up to ten official evaluation pairs after H₀.
Evolution is a software search process under partial observability: the agent must diagnose which structural change caused a real failure, then verify the fix end to end.
Capability and efficiency are separate axes.
A harness can improve task success by spending far more tokens, or consume less while silently failing. HarnessDev reports both.
Capability
Native downstream task performance on held-out benchmarks: success, accuracy, medal rate or rubric score depending on the domain.
Efficiency
Executor-model tokens consumed during deployment, reported as total and mean per task. Creator tokens are excluded.
Models can build runnable harnesses, but quality depends on the domain.
Use the toggle to compare Self-Eval with a fixed Gemini executor. The average is the unweighted mean of SWE-Pro, Terminal-Bench, EQ-Bench3 and BrowseComp.
Opus 4.8 leads Self-Eval at 67.8, but remains below the selected human reference at 86.2.
More code did not mean a better harness.
The Code artifacts added 17,111 net lines in total, yet focused changes and verification sometimes beat larger rewrites.
net lines added across 18 Code harnesses.
Gemini added the fewest lines: 1,006. It still achieved the best Terminal-Bench score among creators: 68.8.
A runnable harness isn’t automatically portable.
Changing only the executor model can change the ranking. The harness may have co-adapted to the model that created or originally ran it.
Opus Code: 69.3 → 33.0
SWE-Pro score under Self-Eval versus fixed Gemini. The paper links the collapse to a hard-coded 120-step limit around the original executor.
Qwen BrowseComp: +17.6
Qwen’s harness improved under fixed Gemini on BrowseComp, suggesting its original executor was constraining performance.
Compatibility is a first-class property: prompts, tool protocol, budgets, stopping rules and message formats all matter.
Visible gains were real. Held-out gains were smaller.
Self-runtime creators all improved on the visible feedback pair. The held-out improvement was more modest, and fixed-Gemini transfer was weaker.
Best self-runtime held-out improvement: Opus 4.8 at +4.44 points. Mean gain across the five self-runtime declarations: +3.11.
The hard part is diagnosis, not just editing.
The benchmark’s most useful lessons are about what a self-improving harness must notice before it changes itself.
Optimise the visible score, lose the future
Across 64 comparable switches, feedback and held-out scores moved in the same direction only 34 times: 53.1%.
Declared code may never run
Of 169 new functions or classes in Evolution, 113 were reachable from the entry point, 31 only through dead code and 25 had no caller.
Counting tests is not enough
Self-test count correlated weakly with downstream score: Spearman ρ only 0.13–0.26. Revision calls reached 0.57.
The best visible version may be lucky
Only 2 of 9 declared versions were held-out optimal. Repeated runs could vary by about ±4.75 pair-score points.
How to implement
A practical, model-agnostic recipe adapted from the benchmark contract.
Build the smallest harness you can actually measure.
Use this as an implementation template. It is adapted from HarnessDev’s interface and evaluation protocol, not a copy-paste production framework.
Make the control plane explicit.
The code shape below keeps the harness modules separable. That makes failures easier to trace and changes easier to audit.
# adapted interface sketch
class Harness:
run(task) -> Result
step(state, observation) -> Action
tools.call(name, **params)
context.build(task, history, state)
state.save(checkpoint)
lifecycle.onFailure(error)
evaluation.evaluate(result, criteria)
# always emit auditable artefacts
result.json
trajectory.jsonl
response.md
runtime logsA one-week pilot for a real workflow.
Pick one narrow task family. The goal is not to build a general agent; it is to learn whether a versioned harness creates durable gains.
Don’t ship an improvement you can’t explain.
Before accepting a harness revision, require evidence across behaviour, cost, reachability and transfer.
Name the failure
- Which trace shows it?
- Which module owns it?
- What should change?
Prove the path
- Does the new code trigger?
- Does the old case still pass?
- Does the verifier catch false success?
Check transfer
- Does cost move with score?
- Does it work on held-out tasks?
- Does another executor survive?
Rule of thumb: an edit is evidence-backed only when you can connect failure → mechanism → end-to-end re-test → generalisation result.
The harness is another place capability accumulates.
Source paper: arXiv:2609.01437v1 · Project page: self-developing-agents.github.io
Deck prepared from the attached paper, “HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?”, dated September 2, 2026.