The problem
Exploration is expensive
In long-horizon discovery, a poor search strategy can waste many agent calls before its weaknesses become visible.
Research explainer · 2026
Dream-RSI
Recursive self-improvement through evolving worlds.
A framework that lets an AI agent improve how it explores by replaying its own past discovery histories before spending more compute in the real search.
Beginner → builder20 slidesSelf-contained HTML
Online explorecollect new discovery traces
Replay simulatorreuse observed outcomes
Dream policytest and refine orchestration
↘
↗
→
Source: Zheng et al., “Dream-RSI: Recursive Self-Improvement through Evolving Worlds”, arXiv:2609.14858v1, abstract and Figure 1, pp. 1–2. Paper link
01 · Plain-English TL;DR
The one-minute version
Let the agent practise its search strategy before it searches again.
Dream-RSI turns old discovery runs into replayable worlds. A controller can try thousands of alternative ways to branch, refine, parallelise, and stop, using outcomes that have already been recorded.
The idea
History becomes a simulator
Recorded branches and their outcomes can be replayed without rerunning the coding agent or evaluator.
The payoff
Improve orchestration cheaply
Use low-cost offline replay to choose a better exploration policy, then redeploy it online.
Plain-English analogy: it’s like learning a better route through a city by replaying a GPS log, rather than driving every possible route again.
Source: Paper pp. 1–4. The analogy above is an explanatory interpretation of the paper’s replay-simulator framing.
02 · Why it matters
A meta-level bottleneck
The agent may be capable. Its search controller may still be wasteful.
Dream-RSI targets the layer that decides where discovery effort goes. It doesn’t change the underlying coding agent, evaluator, or execution interface.
Long horizon
late
Feedback about a strategy often arrives only after many proposal–evaluation cycles.
Huge policy space
wide
There are many choices: open new branches, deepen existing ones, batch workers, or stop.
Delayed signal
costly
Testing one new meta-policy online can require another expensive rollout.
Potential adaptations
Koinaku: decide which memory branches to consolidate or revisit.
FlavourMind: prioritise which hypotheses deserve another modelling cycle.
AI R&D: choose which experiment families to parallelise.
These are plausible adaptations, not tasks evaluated in the paper.
Where it helps most
Problems with an executable evaluator, repeated search cycles, branching work, and enough historical traces to replay.
Source: Paper pp. 1–4. Use-case adaptations are recommendations derived from the paper’s mechanism, not reported results.
Section I
The mechanism
From static history to an evolving world the controller can explore.
03 · The core insight
History is more than context
A completed discovery tree can act like a replay simulator.
Common pattern
Summarise the past
Give the next agent a set of previous attempts, lessons, or directions to read before making its next move.
Risk
Over-constrain exploration
The history becomes a strong semantic nudge. In the paper’s ConvDiv analysis, adding explicit directional guidance underperformed unguided exploration.
Highlighted nodes are revealed by one replay trajectory. No new agent call is needed.
The shift
Replay decisions, not just text
An alternative policy can choose a different subset of branches, a different order, different parallel groupings, and an earlier or later stopping point.
The replay is grounded in realised outcomes, so it can compare controllers against the same frozen history.
Source: Paper pp. 2–5 and Figure 2. The visual tree is a simplified representation of the paper’s discovery-tree model.
04 · The loop
Online → offline → online
Three stages close the self-improvement loop.
1
Online explore
The current policy guides a fixed discovery agent. New attempts and evaluator outcomes are recorded into a tree.
2
Construct replay simulator
Historical trees become frozen worlds. Replay reveals recorded children without generating new candidates.
3
Dream policy improvement
Candidate policy versions are evaluated on the replay worlds. The best version is redeployed online.
Stage 1 detail: this is the only phase that invokes the underlying discovery agent and evaluator to create new evidence.
Important boundary: the paper changes the exploration-policy code. The discovery model, evaluator, and execution interface remain fixed.
Source: Paper pp. 1–2 and 4–6, Figure 1. “Dream” means offline replay-based evaluation, not an ungrounded free-form simulation.
05 · Decision interface
What the policy controls
At each round, choose a legal batch of frontier nodes.
One possible batch: A1 + B1. They are independent frontiers and can use two workers.
Branching
Open a new root
Explore a direction that hasn’t been tried in the current tree.
Refinement
Continue a leaf
Push further along an existing branch using its saved workspace.
Parallelism
Batch independent work
Schedule up to W legal nodes in one decision round.
Stopping
End with an empty batch
Stop when the evidence says more probing is unlikely to pay.
Source: Paper pp. 4–6 and Appendix B. The selected batch is illustrative; the paper defines legal actions as subsets of roots and leaves up to worker capacity.
06 · Replay objective
Quality, cost, parallelism
The score rewards useful outcomes without pretending compute is free.
quality − β₁ × attempts + β₂ × parallelism
In the paper, quality is the best score reached in the replay trajectory. Cost penalises the number of revealed attempts. Parallelism rewards getting more useful work done per decision round.
The equation above is a readable rendering of Equation (1), not a replacement for its exact notation.
Illustrative toy score8.68
Why this matters: a controller that reaches a good result only by probing everything may be less useful than one that reaches a similar result with fewer, better-batched attempts.
Source: Paper pp. 5–6, Equation (1). The interactive score uses illustrative fixed values for quality, attempts, and rounds to make the trade-off tangible.
07 · Why replay is cheaper
Off-policy evaluation
One costly online run can support many cheap policy comparisons.
Online
Agent callnew
Evaluatornew
Policy testslow
A selected batch creates new candidates and new measured outcomes.
Replay
Agent callnone
Evaluatornone
Policy testfast
The policy follows recorded paths and reads stored outcomes.
relative effort shown conceptuallyno new discovery execution
What replay cannot do: it can’t reveal a branch that was never explored in the historical tree. Online exploration still supplies the coverage that future dreaming depends on.
Source: Paper pp. 2–5. The cost comparison is a conceptual rendering of the paper’s claim that replay avoids rerunning the discovery agent and evaluator.
Section II
The evidence
What the paper tested, where it helped, and what remains unproven.
08 · Experimental map
Eight tasks · three domains
The same meta-idea travels across different discovery settings.
01
Algorithm engineering
Lasso regularisation-path discovery, evaluated on 17 synthetic instances plus six held-out datasets.
correctnessruntime03
Mathematical optimisation
Sum–Difference, Autocorrelation Inequalities, and Circle Packing.
qualityconstraints04
GPU kernel engineering
VGG16, LayerNorm, ConvDiv, and ConvMax from KernelBench.
correctnesshardwareControlled baseline
Recursive Fixed Exploration uses the same discovery agent, evaluator, initialisation, and resource constraints, but keeps the exploration policy static.
Compute measure
The paper quantifies discovery cost using the cumulative number of discovery-agent calls.
Source: Paper pp. 6–10. “Eight tasks” is the paper’s count across one Lasso task, three mathematical tasks, and four GPU kernel tasks.
09 · Result spotlight
Algorithm engineering
On Lasso path discovery, Dream-RSI used less discovery compute and reached lower average runtime.
Gemini-3.1 Pro six held-out datasets
317
550
2,931.0 ms
3,587.1 ms
Gemini-3.7 Flash six held-out datasets
1,879
3,200
2,350.6 ms
2,516.7 ms
Compared with SimpleTES
SimpleTES is reported with a 51,200-generation budget. Dream-RSI reached lower average runtime with roughly two orders of magnitude fewer discovery-agent calls in this comparison.
Read carefully
These are benchmark results under the paper’s setup. They don’t establish that Dream-RSI will improve every agent workflow or every evaluator.
Source: Paper pp. 7–8 and Figure 3. Metrics are transcribed from Table 1 in the paper; lower runtime is better.
10 · Result spotlight
Mathematical and GPU discovery
Across other domains, the gains show up as either better quality or fewer generations.
Mathematical optimisation
1.145427
2.635983
1.456375
The paper reports a best or competitive result depending on task. SimpleTES remains stronger on Autocorrelation, but uses 51,200 generations.
GPU kernel engineering
2.43× fewer generations
1.79× fewer generations
2.09× higher score
1.44× higher score
For VGG16 and LayerNorm, final performance was comparable. For ConvDiv and ConvMax, performance was higher at similar budgets.
Source: Paper pp. 9–10, Table 1 and Figure 4. “Higher” and “lower” depend on each task’s objective, as specified in the paper.
11 · Further analysis
The interesting part
Interactive history beat high-level advice.
History as guidance
“Try this direction next.”
Explicit semantic guidance was injected into prompts for both fixed exploration and Dream-RSI. In the ConvDiv analysis, both guided variants underperformed their unguided counterparts under equivalent budgets.
Replayhigher
Guidancelower
Policy behaviour
Spend less, then spend more
On ConvDiv, the learned policy initially reduced evaluated attempts as performance improved, then increased exploration when progress plateaued and performance rose again.
110 → 50
attempts in an early shift described by the paper
plateau
trigger for renewed exploration effort
Interpretation: replay preserves diversity by letting a policy interact with the branching record, rather than collapsing the record into one prescriptive narrative.
Source: Paper pp. 11–12, Figure 5 and Figure 6. The final interpretation is labelled as an inference from those analyses.
12 · Limits and guardrails
What Dream-RSI doesn’t prove
A replay world is useful, but it isn’t the whole environment.
Method boundary
Replay is bounded by coverage
Unexplored branches cannot be discovered offline. You still need online exploration to expand the simulator pool.
inference from replay designEvaluation boundary
Replay improvement is not live improvement
The paper’s no-worse property applies to average replay score on the fixed historical set, not automatically to the next online rollout.
paper pp. 5–6Quality boundary
The evaluator remains the judge
If the evaluator is noisy, narrow, or misaligned with the real objective, the controller can become better at optimising the wrong signal.
general systems riskGeneralisability boundary
The evidence is benchmarked
The reported results cover eight discovery tasks. They don’t establish performance in production workflows, safety-critical domains, or changing environments.
evidence scopeSource: Paper pp. 4–10. The distinction between replay guarantees and live outcomes follows the paper’s fixed-history selection procedure. The evaluator and deployment risks are general implementation cautions.
Section III
Build it
A practical path from an ordinary discovery loop to replay-based policy improvement.
13 · Implementation guide
A staged build plan
Start with the evaluator and the trace format. Add dreaming after the loop is observable.
Repository companion
Use the official Dream-RSI repo as your source map
Start with the paper, then use the repository for the project README, paper assets, citation metadata, release status, and links to the project page. The README currently says the full codebase and reproduction scripts are still being prepared, so treat it as a companion reference until those artefacts are released.
Open github.com/zhengkid/Dream-RSI ↗
Open the project walkthrough ↗
Source: Implementation sequence adapted from the paper’s online rollout, replay, objective, policy-improvement, and redeployment stages, pp. 4–6 and Appendix B. Repository status and links: official Dream-RSI GitHub README, accessed 16 September 2026. This is a practical recommendation, not a verbatim implementation prescription.
14 · Minimal architecture
Keep the first version small
You need five pieces before you need a sophisticated controller.
1 · Discovery agent
Produces a candidate from a saved workspace and available context.
2 · Evaluator
Returns a score plus diagnostics, validity, and failure information.
3 · Trace store
Persists parent, child, workspace snapshot, candidate, score, errors, and cost.
4 · Policy interface
Reads only revealed observations and returns a legal batch of frontier nodes.
5 · Replay runner
Reveals stored children deterministically and records the replay trajectory.
6 · Selector
Scores policy versions across historical worlds and retains the best evaluated version.
Design rule: keep the discovery agent and evaluator stable while you iterate on the orchestration layer. That makes the effect of policy changes easier to measure.
Source: Paper pp. 4–6. The six-part decomposition is a practical implementation mapping of the paper’s components.
15 · Policy skeleton
Prefix-only decisions
Make every decision explainable from what the policy has actually seen.
Controller contract
def solve(question, budget=None): question.reset() result = SimResult() closed = set() while not_done(question, budget): observed = question.observed() update_closed(closed, observed) batch = select_batch(observed, question, closed) if not batch: break question.probe_batch(batch) return finalise(question, result)
Do not leak the future
- Use revealed observations, legal actions, structural metadata, and measured history.
- Don’t use unrevealed scores, hidden trace data, or hard-coded winning cell IDs.
- Keep batches legal, distinct, and within worker capacity.
- Classify failures before closing a branch. A repairable implementation error isn’t proof that the direction is bad.
Source: Paper Appendix B.2, pp. 19–22. Code is a compact illustrative skeleton inspired by the paper’s required loop and constraints, not a drop-in implementation.
16 · Validation plan
How to know it’s working
Compare discovery quality and discovery cost, not just the final winner.
Success criteria
At comparable quality, fewer discovery-agent calls. Or, at comparable cost, better downstream quality. Also check that batching is real and that failure recovery doesn’t hide wasted work.
Red flags
Replay score rises while live score falls; the policy overfits one tree; branch diversity collapses; the evaluator’s signal changes; or the controller exploits bookkeeping artefacts.
Source: Paper pp. 6–12 and Appendix B. Validation recommendations extend the paper’s controlled comparisons and prefix-only constraints.
17 · Takeaways
The durable idea
Don’t throw away the search tree after the result is found.
01
History can be executable. A discovery trace becomes more valuable when a policy can traverse it and receive grounded feedback.
02
Meta-exploration is a first-class optimisation target. Better branching, batching, and stopping can improve what the underlying agent discovers.
03
Replay is a complement to online search. It lowers the cost of policy evaluation, but it can’t replace new exploration or protect you from a weak evaluator.
04
Keep the loop measurable. Trace decisions, outcomes, cost, and failure types so every policy revision has an evidence trail.
Source: Paper pp. 1–12. Takeaways synthesise the paper’s mechanism, experiments, analysis, and conclusion.
18 · Sources and open questions
Read, test, extend
The paper is a strong mechanism paper. The next question is where it earns its keep in your workflow.
Open questions
How much history is enough? How should replay worlds be refreshed when the environment changes? How should policy improvement handle distribution shift? What safeguards are needed when evaluator errors are costly?
Citation
Zheng, Tong, et al. “Dream-RSI: Recursive Self-Improvement through Evolving Worlds.” arXiv:2609.14858v1, 14 September 2026. Google, University of Maryland, Google DeepMind, and University of Virginia.
Deck note: Claims are grounded in the supplied PDF. Where a slide makes an adaptation or implementation recommendation, it is labelled as an inference or recommendation.