Research Presentation • Meta Superintelligence Labs

Agent Learning via Early Experience

Bridging the gap between Imitation Learning and Reinforcement Learning through reward-free, self-generated environment interactions.

Core Contributors: Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang
Institutions: Meta Superintelligence Labs, FAIR at Meta, The Ohio State University
Published: October 2025
The Challenge

Why Current Agent Training Paradigms Struggle

Autonomous language agents need to learn from experience, but existing paradigms face fundamental bottlenecks.

1. Imitation Learning (SFT) Bottlenecks

  • Static & Non-Scalable Data: Expert human demonstrations are expensive and hard to scale.
  • Distribution Shift: Agents do not observe outcomes of their own actions. Minor errors compound rapidly when off-demonstration trajectory.
  • Passive Execution: Agents memorize fixed trajectories without understanding underlying environment dynamics.

2. Reinforcement Learning (RL) Bottlenecks

  • Sparse/Missing Rewards: Real-world tasks (e.g., complex web navigation) lack verifiable dense reward signals.
  • Inefficient Exploration: Multi-turn tool usage requires long-horizon rollouts making credit assignment unstable.
  • High Compute Overhead: Cold-start RL training for LLMs is notoriously prone to divergence.
Paradigm Shift

The "Early Experience" Middle Ground

Using the agent's proposed non-expert actions and resulting future states as direct supervision—without requiring external rewards.

Era 1: Human Data (SFT)
✕ Scalable Data
✓ Reward-Free

Trained on static expert demonstrations. Fails on unobserved states.

Early Experience (Ours)
✓ Scalable Data
✓ Reward-Free

Agent proposes actions, collects future states, and converts environment feedback into direct self-supervision.

Era 2: Experience (RL)
✓ Scalable Data
✕ Requires Verifiable Reward

Optimizes cumulative rewards through trial & error. Constrained by simulator availability.

Methodology

Two Strategies for Supervision from State Transitions

Transforming exploratory rollouts into effective learning objectives.

1. Implicit World Modeling (IWM)

Concept: Internalizes environment dynamics by learning to predict future textual states resulting from non-expert actions.

L_IWM = - Σ log P_θ ( s_{i}^{j} | s_i, a_{i}^{j} )
  • Two-stage pipeline: First train on state prediction, then fine-tune on expert actions.
  • Grounds policy in state changes, error messages, and DOM alterations.

2. Self-Reflection (SR)

Concept: Compares expert actions against proposed alternatives to synthesize fine-grained rationale monologues.

L_SR = - Σ log P_θ ( c_{i}^{j}, a_i | s_i )
  • Generates explanations (c_i^j) detailing why non-expert outcomes fail constraints.
  • Jointly predicts reasoning path and optimal action during training.
Interactive Breakdown

Data Construction & Reflection Walkthrough

Select a method tab to inspect how early experience constructs learning data.

Environment: WebShop (E-commerce task: "Blue wireless headphones under $130")

State s_1: [Product Page: PowerLocus Headphones, Price $24.99, Colors: Black, Blue, Red] Proposed Non-Expert Action a_1': "click < Prev" Observed State s_1': "Navigated back to Search Results page. Displays list of listings." --> Training Objective: P_θ("Navigated back to Search..." | State s_1, "click < Prev")

*The policy internalizes that clicking "Prev" loses the current product selection state without needing explicit rewards.

Self-Reflection Monologue Generation Prompt & Target:

[Situation]: User wants blue wireless headphones under $130. [Expert Action]: click "non-ears blue" [Alternative Action]: click "red shirt option" -> Result: Selected red option ($30) [Generated Reflection Monologue c_i]: "The goal requires blue color and price < $130. While the alternative action selects a valid product, it chooses 'red' which violates the color constraint. The expert action 'non-ears blue' satisfies both style and price bounds directly."
Experimental Results

Benchmark Performance Across 8 Environments

Consistent improvements over Supervised Fine-Tuning across model families (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B).

Environment Domain Benchmark SFT Baseline Ours (IWM) Ours (Self-Reflection) Absolute Gain
Embodied Sim ALFWorld 80.5% 85.9% 85.2% +5.4%
Scientific Sim Science World 54.7% 57.0% 68.0% +13.3%
Planning TravelPlanner 17.2% 25.0% 32.2% +15.0%
Tool-Use API BFCLv3 6.7% 20.0% 20.0% +13.3%
Customer Support Tau-Bench 35.9% 40.8% 41.7% +5.8%
Web Navigation WebShop 47.3% 58.6% 58.2% +11.3%
Downstream Synergy

A Practical Bridge to Reinforcement Learning

Initializing post-hoc RL (GRPO) with Early Experience checkpoints yields substantially higher post-RL performance ceilings.

WebShop Post-RL

SFT → GRPO: 80.5%

IWM → GRPO: 91.4%

Higher convergence ceiling when environment rewards become available.

ALFWorld Post-RL

SFT → GRPO: 93.8%

SR → GRPO: 98.5%

Pre-grounding prevents policy collapse during early RL exploration steps.

SearchQA Post-RL

SFT → GRPO: 47.1%

SR → GRPO: 51.0%

Reduces hallucinated search query generation during online RL steps.

Conclusion

Key Takeaways & Future Directions

Early Experience transforms unrewarded agent interaction into high-value self-supervision.

Summary of Contributions

  • Formalized Paradigm: Established Early Experience as a scalable bridge between static SFT and reward-dependent RL.
  • Effective Strategies: Introduced Implicit World Modeling and Self-Reflection without requiring standalone simulators or reward models.
  • Data Efficiency: Achieved equal or superior performance using only 50% or less of human demonstration data.

Future Horizons

  • Long-Horizon Credit Assignment: Extending reward-free reflection to multi-step execution traces.
  • Cross-Environment Transfer: Pre-training foundational world-modeling priors across diverse web and OS environments.
  • Organic Deployment: Continuously updating agent policies from production interaction traces.
1 / 8