A benchmark for evaluating AI agents on long-horizon, economically valuable, real-world professional tasks with verifiable outcomes.
Based on O*NET / SOC 2018 federal occupational taxonomy — all 55 subdomains covered. Numbers show task instances per domain.
Tasks must reflect real professional practice using the software domain experts actually use. A structural engineer uses SolidWorks or Rhino — not AutoCAD — to convert a 2D blueprint to a 3D model.
Tasks must be end-to-end deliverables that take an expert substantial time — not a single UI action.
✗ "Apply a color filter in DaVinci" — too narrow.
✓ "Move a running cheetah into another race video" — tracking, rotoscoping, compositing, color matching.
Outputs must admit deterministic checking or an unambiguous rubric tied to observable artifacts.
✗ "Design an RPG game with monsters" — no checkable target.
✓ "Reproduce mota.exe in RPGMaker XP" — map geometry, character attributes are verifiable.
Only ~10% (150 tasks) are public to prevent contamination. 1,017 remain private, 323 in QC. Tasks rotate continuously.
Three lifecycle methods orchestrate every run:
Generalist Computer-Use Agent — combines visual perception, code execution, tool use, and long-horizon planning in a single action loop.
Examples: Claude Code, Codex, OpenClaw
| Agent Configuration | Near-Term Pass% | Full-Spectrum Pass% | Last Exam Pass% | Overall Pass% |
|---|---|---|---|---|
| 1Codex (GPT-5.5) | 38.1% | 22.7% | 0.0% | 24.0% |
| 2ALE-Claw (GPT-5.5) | 32.8% | 23.6% | 2.6% | 23.0% |
| 3Claude Code (Fable 5) | 34.3% | 20.9% | 0.0% | 22.0% |
| 4Cursor (GPT-5.5) | 32.1% | 20.0% | 2.6% | 20.7% |
| 5Cursor (Opus 4.7) | 29.9% | 20.0% | 2.6% | 20.4% |
| 6Claude Code (Opus 4.8) | 26.9% | 10.9% | 0.0% | 15.8% |
| 7Claude Code (Opus 4.7) | 20.9% | 12.7% | 0.0% | 13.2% |
| 8Gemini CLI (Gemini 3.1 Pro) | 26.9% | 12.7% | 0.0% | 15.8% |
| 9Grok CLI (Grok 4.3) | 9.0% | 7.3% | 0.0% | 6.6% |
~75% of failures stem from knowledge/understanding gaps, not execution limitations.
34% of tasks require graphical software as the primary tool, yet GUI usage stays low — agents substitute domain tools with ad-hoc Bash scripts.
The choice of foundation model accounts for roughly 3× the performance spread compared to the choice of agent harness — among well-engineered systems.
Higher API cost, more tokens, and longer runs do not reliably translate to better performance. Resource efficiency varies widely across configurations.
| Benchmark | Task Form | Size | Domain Coverage | Horizon | Task Source | Verification |
|---|---|---|---|---|---|---|
| MMLU | Knowledge QA | ~16K | 26 / 55 | Seconds | Mined (exams) | Exact match |
| GPQA | Knowledge QA | ~500 | 8 / 55 | Minutes | Domain experts | Exact match |
| SWE-bench | Code patch | ~2K | 5 / 55 | Minutes–Hours | Mined (GitHub) | Unit tests |
| OSWorld | GUI Operation | ~400 | 5 / 55 | Minutes | Curators | State checks |
| Terminal-Bench | CLI Operation | ~100 | 6 / 55 | Minutes–Hours | Curators | State checks |
| GDPval | Project deliverable | ~200 | 16 / 55 | Hours–Days | Industry experts | Human (expert) |
| ALE (ours) | Project deliverable | ~1.5K | 55 / 55 | Hours–Weeks | Industry experts | Auto (deterministic) |
105 Linux-only task instances that can be attempted by CLI-only agents (ForgeCode, Hermes, Terminus) without GUI desktop access.
A natural comparison to Terminal-Bench (~100 terminal-centric tasks), but substantially harder and requiring longer agent sessions.
Codex + GPT-5.5 achieves 82% on Terminal-Bench but:
AI systems regularly ace academic benchmarks but score below 1% on professional workflows that generate real economic value. The problem is the benchmarks themselves.
1,490 expert-sourced tasks across all 55 O*NET subdomains, with fully automated deterministic verification. A living benchmark that grows continuously.
~75% of failures are domain knowledge and approach problems, not execution. Agents default to generic scripts instead of professional domain tools.
The Last Exam tier is nearly unsolved across all frontier agents. If AI can pass this exam, benchmark progress may begin to register as real economic transformation.