Self-Harness
Harnesses that improve themselves.
The idea is simple: let an agent study its own execution failures, propose small changes to the scaffolding around it, and keep only edits that survive regression tests.
An agent can improve its operating system without changing its brain.
Self-Harness keeps the model weights fixed. It evolves the surrounding protocol: prompts, tools, runtime controls, verification and recovery.
Mine
Run tasks. Use verifier-grounded traces to cluster recurring failure mechanisms.
Propose
Ask the same model for diverse, minimal edits tied to those mechanisms.
Validate
Regression-test each candidate on held-in and held-out tasks. Promote only non-regressive gains.
Agent performance is more than model capability.
The harness is the layer that decides what the model sees, what it can do, when it checks its work and how it recovers.
Two harnesses can produce different outcomes because execution protocol changes behaviour over long horizons.
Who gets to improve the harness?
Self-Harness moves improvement inside the evaluated agent, while keeping the model and evaluator fixed.
Human engineering
Experts inspect failures and manually revise the scaffold.
Meta-Harness
A stronger external agent guides improvement of a weaker target.
Self-Harness
The evaluated agent improves the very harness it runs through.
Evidence becomes a harness state transition.
Every round starts with a current harness hₜ and ends with either a tested update hₜ₊₁ or no change.
Run the fixed model. Collect traces and verifier outcomes. Cluster recurring failure signatures.
Generate several diverse but minimal edits, each tied to one mechanism and one editable surface.
Regression-test held-in and held-out tasks. Reject trade-offs; merge compatible winners.
Fixed throughout: model weights, evaluator, task protocol, non-editable runtime settings. Allowed to change: the surrounding harness.
Don’t cluster symptoms. Cluster mechanisms.
A timeout, missing file or assertion failure is only the verifier’s surface-level verdict.
The evaluation system attributes the relevant agent behaviour and groups failures only when the verifier cause, causal status and reusable mechanism agree.
task · messages · tool calls · output · verifier outcome
“timeout”
“bad state”
cause · causal status · mechanism
Recurring patterns, representative tasks, shared symptoms, verifier evidence and an actionable mechanism.
The proposer explores, but it doesn’t get a blank cheque.
The same fixed model is invoked in a proposer role with a bounded context.
Distinct branchesSeveral candidates should test different mechanisms or surfaces.
Minimal editsChange only what is needed; preserve unrelated behaviour.
Audit recordEach proposal names its target, surface, expected effect and risks.
Editable surfaces
bootstrap, execution, verification
availability, adapters, schemas
loops, budgets, middleware
memory, recovery, persistence
The proposal stage is deliberately narrower than “redesign the agent”.
Promotion is conservative by design.
A candidate must improve at least one split without degrading the other.
Δin ≥ 0 and Δho ≥ 0
and max(Δin, Δho) > 0
When evaluation is stochastic, the paper repeats candidate evaluation and applies the rule to aggregate pass counts.
Better on held-in, no worse held-out.
Or better on held-out, no worse held-in. Compatible winners can be merged.
Trade-off, invalid surface or failed evaluation.
Rejected candidates stay logged. The active harness does not change.
Why it matters: the gate is the paper’s defence against overfitting the proposer to the failures it has seen.
Three models. Three agentic environments. One controlled comparison.
The model backend, tool set, budget, benchmark environment and evaluator stay fixed within each comparison.
Terminal-Bench 2.0
64-case fixed subset of containerised terminal tasks: artefacts, commands, verification and recovery.
SWE-bench Verified
100 cases: repository inspection, patch implementation and repository tests.
AppWorld
180 examples: application APIs and state-based grading across workflows.
Models: MiniMax M2.5 · Qwen3.5-35B-A3B · GLM-5 | Splits: held-in evidence + held-out regression gate | Metric: Pass (%) across two repeated attempts unless otherwise specified.
Every final harness improved both splits.
Across all nine model–benchmark combinations, the promoted harness beat or matched the starting harness on held-in and held-out performance.
The gains show up on held-out tasks too.
Use the table as a calibration point: the largest improvements are concentrated in weaker starting harness/model combinations.
| Benchmark · model | Initial in | Initial out | Initial overall | Final in | Final out | Final overall |
|---|---|---|---|---|---|---|
| Terminal-Bench 2.0 | ||||||
| MiniMax M2.5 | 43.0 | 40.5 | 42.2 | 50.0 | 61.9 | 53.9 |
| Qwen3.5-35B-A3B | 15.1 | 23.8 | 18.0 | 36.0 | 38.1 | 36.7 |
| GLM-5 | 47.7 | 42.9 | 46.1 | 57.0 | 57.1 | 57.0 |
| SWE-bench Verified | ||||||
| MiniMax M2.5 | 51.5 | 34.8 | 46.0 | 58.2 | 40.9 | 52.5 |
| Qwen3.5-35B-A3B | 20.1 | 18.2 | 19.5 | 42.5 | 39.4 | 41.5 |
| GLM-5 | 53.7 | 48.5 | 52.0 | 58.2 | 50.0 | 55.5 |
| AppWorld | ||||||
| MiniMax M2.5 | 51.7 | 45.6 | 48.6 | 62.8 | 55.0 | 58.9 |
| Qwen3.5-35B-A3B | 25.0 | 20.0 | 22.5 | 60.0 | 44.4 | 52.2 |
| GLM-5 | 47.8 | 41.1 | 44.4 | 92.2 | 77.8 | 85.0 |
The retained edits are workflow interventions.
The paper reports targeted changes, not simply longer prompts.
Create earlier. Handle schemas. Stop loops.
Edits target missing required artefacts, structured tool content and stalled tool-use loops.
Precheck. Retry with discipline. Recover.
Edits target dependencies, repeated failed commands, exploration cycles and artefact recovery.
Persist state. Move to implementation.
Edits target environment persistence and the transition from exploration to testing.
Interpretation: the same base model family can benefit from different harness mechanisms because its failure patterns differ in context.
Held-out regression is the difference between a patch and a reusable mechanism.
The proposer sees failure evidence from held-in tasks. Held-out tasks are kept behind the promotion gate.
Held-in split
Trajectories, verifier outcomes and clustered failure evidence motivate the candidate.
Held-out split
Regression gate checks whether the edit transfers without degrading unseen cases.
The paper says accepted edits may remain benchmark-specific. Higher-stakes changes need stronger gates than pass-rate non-regression.
Build the smallest safe self-improvement loop.
You don’t need a frontier benchmark to borrow the pattern. You do need observable failures, declared edit surfaces and a promotion gate.
Start with an evaluation substrate you can trust.
Before self-improvement, make agent behaviour measurable and replayable.
Task setFreeze representative tasks and a separate regression split.
VerifierReturn structured pass/fail reasons, not only a score.
Trace schemaStore task, messages, tool calls, outputs, state and verifier evidence.
RepeatabilityFix non-editable budgets, tools, environments and evaluator versions.
{
"task_id": "...",
"harness_version": "h_003",
"trace": { "messages": [], "tools": [] },
"output": "...",
"verifier": {
"status": "fail",
"cause": "missing_artifact",
"evidence": "..."
}
}Make the harness editable, bounded and reversible.
Put configuration points behind a versioned interface. The proposer edits declarations, not arbitrary production infrastructure.
bootstrap, execution, verification, recovery
loop breakers, message budgets, middleware
sources, adapters, schemas, persistence
diffable, testable, rollback-ready
Promotion boundary: candidate edits are applied in an isolated branch, evaluated, then merged only after the gate passes. Keep every proposal and decision in an append-only lineage.
Turn traces into an evidence bundle.
Ask an analysis pass to find repeated, actionable mechanisms across failed runs.
Good pattern: “The agent repeatedly deletes or never creates the verifier-required artefact after a tool error.”
Weak pattern: “The agent timed out.”
Select failed records with verifier evidence.
Separate terminal cause from agent-side behaviour.
Group only when cause + causal status + mechanism agree.
Order by support and likely actionability.
Generate candidates in parallel. Promote them like code.
A proposal is a hypothesis about behaviour. Evaluation decides whether it becomes part of the harness.
Target: missing artefact
Surface: bootstrap instruction
Target: tool loop
Surface: runtime middleware
Target: bad recovery
Surface: error policy
Evaluate each branch
Run current and candidate harnesses on the same held-in and held-out tasks. Record split-wise deltas, repeats, changed surfaces and decision.
Merge only compatible winners
If all candidates fail, keep hₜ. If several pass and don’t conflict, merge into hₜ₊₁. Log rejected branches for future analysis.
One implementation shape.
Keep the orchestration boring. The intelligence belongs in the evidence, proposal and verifier interfaces.
Important: this is a practical adaptation of the paper’s algorithm, not a drop-in library.
for round in rounds: baseline = evaluate(model, harness, tasks, verifier) evidence = mine_failures(baseline.failures) proposals = propose_parallel( model, harness, evidence, editable_surfaces, width=K) accepted = [] for proposal in proposals: candidate = apply(harness, proposal.edit) result = evaluate(model, candidate, tasks, verifier) if non_regressive(result, baseline): accepted.append(proposal) else: log_rejection(proposal, result) harness = merge(harness, accepted) save_lineage(harness, accepted)
Where this pattern is useful.
Choose environments with repeatable tasks, meaningful verifiers and a harness you actually control.
Repository repair and local tooling
Detect loops, missing patches, skipped tests or broken recovery. Promote narrow verification and runtime changes.
Example: a coding harness that learns to run the targeted test before declaring success.
Evidence and provenance workflows
Detect unsupported claims, missing citations or incomplete extraction. Promote provenance checks and source-of-truth retrieval.
Example: an agent that learns to attach a source note before finalising a literature brief.
Tool-driven product operations
Detect stale state, repeated failed actions or partial writes. Promote confirmation, recovery and bounded retry policies.
Example: an agent that learns to re-read state after a write and stop after a fixed retry budget.
Not a fit yet: open-ended creative work without reliable verifiers, high-stakes changes without strong approval gates, or environments where traces contain sensitive data you can’t safely process.
Self-improvement is only as good as its boundary conditions.
The paper’s acceptance rule is useful, but it is not a complete safety case.
Benchmark overfitting
Accepted edits may encode benchmark-specific failure patterns. Add cross-domain, temporal or adversarial regression tasks.
Verifier weakness
A weak or gameable verifier can reward the wrong behaviour. Validate the evaluator independently.
Trace contamination
Proposer context can leak held-out information or sensitive data. Enforce split isolation and redaction.
Reversible promotion
Version every harness, keep rejected candidates, require approvals for high-impact surfaces and make rollback cheap.
Stronger gates
For consequential workflows, add invariants, human review, canary runs, policy checks and failure budgets.
Auditability
Store the evidence, proposal rationale, diff, evaluation repeats and accept/reject decision for every transition.
Build agents that can learn from their own execution, with receipts.
The useful unit of progress is not a clever prompt. It’s a reversible, evidence-backed harness transition.
Observe behaviour
Record traces and verifier-grounded failure mechanisms.
Change the scaffold
Propose diverse, minimal edits to declared surfaces.
Earn promotion
Regress on what the proposer didn’t see. Keep the diff, evidence and decision.