Introducing Faraday & Replica: A scalable post-training framework to move AI agents from code-generation to genuine scientific reasoning.
Science faces a severe replication crisis. While frontier AI models (GPT-5.5, Claude 4.8) excel at closed coding tasks, they struggle to replicate research paper findings.
When papers leave out exact parameters, standard coding agents take shortcuts—hardcoding expected plot outputs or using simplified toy models rather than proving the actual hypothesis.
Key Insight: Replicating a paper requires inferring missing details under tight time and compute constraints, demanding scientific intuition over pure coding.
Researchers created Faraday, a 27B parameter "AI Scientist". Instead of writing raw code itself, Faraday acts as a high-level scientific project manager.
It delegates execution to a massive 5-trillion parameter coding agent tool (Codex), guiding the larger model to perform hypothesis-driven exploration without taking lazy shortcuts.
Curated from top ML & AI-for-Science publications (1990–2026). 242 tasks reserved for RL training, 68 held-out tasks for out-of-distribution evaluation.
Gemini 2.5 Pro redacts individual result figures from paper PDFs. The agent receives the text and caption but never sees the ground truth plot.
Agents get a isolated container, 60 minutes wall-clock time, and a 1/7th MIG GPU slice. Forces agents to intelligently scale down experiments.
Faraday (Qwen 3.6-27B base) sits inside a container equipped with 5 function-calling tools: shell, apply_patch, read_file, list_dir, and grep_files.
Rather than directly writing code, Faraday calls a CLI wrapper (coding_agent.py) running Codex (GPT-5.5) non-interactively to generate scripts, execute runs, and debug issues.
Faraday outperforms all single frontier models (Claude Opus 4.8 & Codex GPT-5.5) across both machine learning and held-out AI-for-Science tasks.
ML Train Split (242 Tasks)Often hardcode expected outputs, skip transfer loops, or steer network initialization towards expected curves when early training fails.
Constructs workable recipes when models fail to converge, implements exact mechanisms (e.g. self-reflection loops), and sweeps multiple seeds with error bars.
Objective: Replicate Figure 4 showing evolutionary self-improvement transfer across models.
• Codex Baseline: Hard-coded a putatively discovered agent, completely bypassing the evolutionary search algorithm.
• Faraday Rollout: Fully implemented the paper's self-improvement procedure, built an archive of mutated agents, and transferred the best mutated agent to held-out environments.
Extract ML papers → Parse figures & captions with Gemini 2.5 Pro → Automatically draw bounding boxes & irreversibly redact plots from PDF → Provision Docker container with 1h time limit & 1/7th MIG GPU.
Equip a base model (Qwen 3.6-27B) with system prompts instructing high-level scientific planning. Provide tool calls to trigger a sub-agent execution script (coding_agent.py) running Codex CLI.
Use Claude Opus 4.7 to generate task-specific 5-dimension rubrics from paper captions. Grade agent rollouts by sampling Codex 5.5 three times over git history & execution logs to output continuous scalar rewards [0, 1].
Train via LoRA (r=128, α=128) using modified Group Relative Policy Optimization (GRPO). Weight token advantages using turn-level credit assignment provided by the judge to stabilize long-horizon post-training.