Research paper deep dive · September 2026
Repo-To-Skill:
turning repositories into operating knowledge
A visual guide to DisCo, the AREX-Skill Library, and the idea that autonomous research agents need more than a model and a harness.
Jianlyu Chen et al.arXiv:2609.02749v148 pages
K
Plain-English summary first, then the machinery and a practical implementation path01
01
Section break
The missing layer between knowing and doing
What the paper adds to the usual model-plus-harness picture, why it matters, and where it can be useful.
Section 1 · Beginner’s view02
TL;DR
A capable model can explain a method. A good harness can run a loop. Operational knowledge is what helps the agent make the method work in a real environment.
MODEL
Reasoning and execution
The backbone understands, plans, writes code, and reacts to observations.
HARNESS
Procedure and control
The harness provides orchestration, memory, verification, and iterative refinement.
SKILLS
Domain operating context
Skills encode what to choose, when it applies, how to use it, and how to recover.
Source: paper, pp. 2–403
TL;DR
Skills replace repeated rediscovery with reusable procedures
WITHOUT SKILLS
Trial and error spends the budget
Problem
Unfamiliar package, method, or environment.
Loop
Guess a setup, run it, inspect the failure, guess again.
Cost
Useful discoveries remain trapped in the current run.
VS
WITH SKILLS
Operating context narrows the search
Entry
Route to a relevant capability and its evidence.
Procedure
Follow known setup, checks, failure modes, and recovery actions.
Reuse
Carry validated know-how into later tasks.
Source: paper, pp. 2–404
TL;DR
Where the idea becomes useful
The pattern fits any workflow where agents repeatedly touch specialised tools, procedures, or failure modes.
01
ML engineering
Distil data preparation, fine-tuning, evaluation, serving, and package-specific troubleshooting from a repository.
02
Research reproduction
Turn prior papers and implementation evidence into module-level skills that help reproduce a method without replaying a fixed solution.
03
Enterprise agents
Create verified operating playbooks for internal APIs, data workflows, governance checks, or hardware control surfaces.
Useful mental model: a skill is a compact, inspectable unit of know-how, not a hidden weight update.
Source: paper, pp. 2–6; use cases are applications of the paper’s framework05
02
Section break
How DisCo turns source material into a skill graph
The paper’s central mechanism: scope the capability, ground it in evidence, package it for an agent, then verify it.
Section 2 · Distillation and consumption06
The distillation loop
Four stages separate a skill from a summary
01 / SCOPE
Choose the capability
Understand the source or task, then decide what is worth exposing.
02 / GROUND
Collect evidence
Select admissible material, map it to skill boundaries, and record gaps or conflicts.
03 / CONSTRUCT
Build the graph
Write agent-facing procedures, usage conditions, references, scripts, and links.
04 / VERIFY
Test and refine
Run checks or task trials, repair unsupported claims, and preserve unresolved gaps.
The paper’s key distinction is verification. A source can support a claim without proving that an agent can execute it.
Source: paper, pp. 5–6, Eq. 807
The skill format
Three layers keep the context compact and actionable
name: package-workflow
when_to_use: "When the task needs X"
procedure:
1. Inspect the input shape
2. Choose the supported interface
3. Run the smoke check
failure_modes:
- version mismatch
- wrong data layout
see_also: references/ · scripts/
SKILL.md · knowledge interfaceWhat the skill is for, when it applies, the standard procedure, key checks, and links onward.
references/ · knowledge substrateDeeper API, algorithm, configuration, and implementation material loaded only when needed.
scripts/ · execution interfaceExecutable wrappers with defined inputs and outputs that the agent invokes rather than reimplements.
Source: paper, pp. 5–6, Eq. 608
Two anchors
Distil ahead of time, or distil for the task in front of you
Task-agnostic
The anchor is a source such as a repository or paper. The process asks: what capabilities does this source make possible? The result becomes reusable infrastructure for future tasks.
Examples: package workflows, model APIs, paper-derived modules.
Task-oriented
The anchor is a concrete problem. The process asks: what does this task demand, and where are the agent’s gaps? The result is produced on demand and can remain reusable.
Examples: a competition, a compiler benchmark, an open-ended algorithmic problem.
The anchor changes. Verification does not.
Source: paper, pp. 6–709
The AREX-Skill Library
A router lets the agent hold a large library without loading it all
5,353
verified repository skills
RequestWhat does the task need?
AreaComputer vision, MLOps, robotics...
FamilyFind the capability boundary
Skill graphOpen only the useful branch
A repository may appear in multiple families. The taxonomy is a navigation aid, not a claim that the ecosystem is neatly disjoint.
Source: paper, pp. 7–9, Figure 310
One agent, two modes
Creator mode writes the library. Researcher mode spends only what the task needs.
CREATOR MODE
Distil once, amortise later
Scope a source or task, gather evidence, construct a candidate graph, verify it, then record the checks and remaining gaps.
Cost is paid at construction time.
RESEARCHER MODE
Load selectively, execute normally
Start from the router, open the relevant entry skill, follow links as needed, and let the existing harness run the task.
Cost is paid for the branch the task actually opens.
Architectural boundary: a skill graph adds operating context. It isn’t a new control loop, planner, or model fine-tune.
Source: paper, pp. 6–711
03
Section break
What the paper measured
Four benchmark settings, a fixed GPT-5.5 Codex setup, and a controlled question: what changes when the agent gets distilled skills?
Section 3 · Controlled evaluation12
Experimental design
The controlled variable was access to skills
HELD FIXED
✓GPT-5.5 backbone
✓Codex research harness
✓Downstream execution budgets
✓Benchmark task suites and graders
VARIED
+Whether the agent received DisCo-distilled skills
+Construction happened before the downstream run
+Skills were frozen during evaluation
This isolates the claim that externalised operational knowledge can improve performance without changing the backbone or the harness.
Source: paper, pp. 9–1013
Main results
Skills improved all four headline benchmark results
without skillswith AREX-SkillBars are normalised within each metric for visual comparison.
The headline percentages are relative improvements reported by the paper. They compare matched conditions, but the benchmark metrics themselves are not interchangeable.
Source: paper, pp. 1, 10–13; Tables 1–414
Reading the results
The largest practical value appears where unguided exploration stalls
72.89%
MLE-bench Any-Medal
Up from 31.11%. The relative lift was especially large on high-difficulty tasks, where the no-skill score rose from 13.33% to 62.22%.
39.59%
PaperBench replication
Up from 29.45%. Skills improved 18 of 20 papers, but two tasks regressed, suggesting retrieval precision still matters.
77.14
FrontierCS score
Up from 70.63. The largest lift came on the 47 tasks below 50, where the mean rose from 19.43 to 45.99.
90.76%
PassNet correctness
Up from 81.35%. Failed samples fell from 14 to 5, while AS Score increased from 1.343 to 1.5313.
Source: paper, pp. 10–1415
Caveats and design implications
Skills help, but retrieval and verification remain product problems
Retrieval can distract
Two PaperBench tasks scored lower with skills. A poor match can pull the agent away from a strategy it would have found unaided.
Design response: add confidence, fallback, and “don’t route” conditions.
Rules can be too absolute
The PassNet analysis found a warning that blocked a valid exception. The revision changed it from a ban into a defeasible prior.
Design response: state when a rule can be overridden by evidence.
A good skill carries a procedure and the conditions under which that procedure should be trusted.
Source: paper, pp. 11–14, 2316
04
Section break
How to implement a repo-to-skill pipeline
A practical, smaller-scale version you can use for one repository before thinking about a library.
Section 4 · Build your first verified skill17
Implementation guide
Start with one repository and one operational workflow
Pick a narrow anchorChoose a versioned repository and a workflow such as fine-tuning, evaluation, serving, or troubleshooting.
Map the evidence boundaryInclude source roots, documentation, examples, tests, scripts, and configuration. Exclude caches, build outputs, vendored dependencies, and unrelated internals.
Scope capabilitiesList what an agent must decide, invoke, check, and recover from. Don’t turn the whole repository into one undifferentiated skill.
Write the entry skillMake SKILL.md router-like: when to use it, minimum setup, the standard procedure, known failure modes, and links to deeper material.
Verify with small casesRun smoke tests, assertion-backed examples, CLI checks, or tiny fixtures before accepting claims. Record unresolved gaps.
Use it in a separate taskLet the agent choose the relevant branch during a real task. Compare with an unguided baseline and refine only when evidence supports the change.
Source: paper, pp. 7, 18–21; adapted into a practical guide18
Implementation guide
A useful first repository skill is a router, not a dump
Suggested folder
repo-skill/
├── SKILL.md
├── references/
│ ├── api.md
│ ├── configs.md
│ └── failure-modes.md
└── scripts/
├── smoke_test.py
└── inspect_inputs.py
Acceptance criteria
✓An agent can tell when the skill applies.
✓The main workflow is explicit enough to follow.
✓Important claims point to evidence.
✓Failure modes have checks or recovery actions.
✓Scripts have stable inputs and outputs.
✓Unresolved gaps are visible rather than hidden.
Keep the first version small. A single, well-verified workflow is more valuable than a sprawling skill that claims to cover every feature.
Source: paper, pp. 5–6, Appendix A.119
Implementation guide
Make verification failure-driven and local
REFINE ONLY WHAT FAILED
Run
Execute a small case with the candidate skill.
Observe
Inspect logs, outputs, checks, and the agent’s chosen path.
Diagnose
Find the earliest implicated skill, link, reference, or setup step.
Repair
Change the smallest local unit, then rerun the relevant check.
RECORD THE RESULT
+Evidence used
+Checks performed
+Observed failure or success
+What remains uncertain
+Whether the revision reproduced the intended change
This is the part that turns “AI-generated documentation” into an operational knowledge asset.
Source: paper, pp. 6, 20–2320
Transferable pattern
For an enterprise agent, the repository can be any operational system
The paper evaluates ML repositories, but its architecture maps cleanly to other domains where the hard part is reliable execution.
DATA WORKFLOW
Source → skill
Data contracts, schema checks, lineage, privacy gates, and recovery paths.
HARDWARE
Manuals → skill
Setup procedures, safe operating limits, diagnostics, and service escalation.
GOVERNANCE
Policy → skill
When a rule applies, what evidence is required, and what happens when the case is ambiguous.
The transferable asset is the verified procedure with its boundaries.
Source: paper’s framework; applications are interpretive extensions21
Takeaways
Operational knowledge is the part agents repeatedly need, but rarely retain
Repo-To-Skill proposes a practical answer: distil source material into compact, routed, verified skill graphs that compatible agents can load when the work demands them.
Start small: choose one repository, one workflow, one acceptance test, and one documented recovery path.
SOURCES
Paper citation
Chen et al., “Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills”, 2 September 2026.
End of deck · use ← →, Space, Home, End, or Overview22