A paper in plain English
Are omni models really omni?
The Modality Maturity Index asks a deceptively simple question: when a user needs an image, an audio clip, a video, or a document, does the model actually return it?
Patel, Hupkes & Strader / Meta Superintelligence Labs
01 / TL;DR
Being able to understand media is not the same as using the right media.
Most multimodal tests ask a model to look at an image, video, or document and answer in text. MMI tests the missing half: can the model choose and deliver the output format the user actually needs?
The punchline
Across five evaluated models, the best Modality Presence Score was only 34.9. Models almost never returned the wrong modality. They mostly returned too little, especially outside text.
893self-contained prompts
5input and output modalities
34.9best MPS in the main evaluation
70.8%judge-human agreement in rubric validation
02 / The blind spot
Most multimodal benchmarks stop at text.
They may feed in an image, video, audio clip, or document. But the answer is usually text. That means a model can look multimodal while behaving like a text-only assistant at the point of delivery.
THE USUAL TEST
InputImage, audio, video or document
Rich media enters the model.
OutputText
The model describes or answers.
Hidden assumption: text is always enough.
THE MMI QUESTION
“What format would make this answer useful to the person asking?”
MMI makes the model select from five possible output modalities, including combinations of up to three.
03 / The benchmark
Five modalities. One user goal.
Each prompt is written so the most useful answer requires a clear set of output formats. The formats are not always named explicitly. The model has to infer them.
T
Text
explanations, lists, instructions
I
Image
edits, annotations, visual references
A
Audio
spoken guidance, pronunciation
V
Video
tutorials, simulations, demonstrations
D
Document
PDF, DOCX, XLSX, PPTX, CSV, TXT
1-3input modalities per prompt
1-3output modalities per prompt
100%prompts manually reviewed by at least two authors
04 / Prompt design
One request can carry several jobs.
MMI prompts are ordinary-sounding user requests. The benchmark maps the underlying jobs to the modalities that should carry them.
Example / visual + textual + document output
“I would like to recreate this look. Can you list all the products I need and write out a very detailed instruction in a .txt file?”
INPUT / VIDEOOUTPUT / TEXTOUTPUT / DOCUMENT
Example / multiple output modalities
“Can you show me how to make soft chocolate chip cookies? A list of ingredients and a tutorial video would be nice.”
OUTPUT / TEXTOUTPUT / VIDEO
475one input / one output
264multiple inputs / one output
76one input / multiple outputs
78multiple inputs / multiple outputs
05 / Dataset construction
Start simple. Then add modality combinations.
The authors built the benchmark in stages, learning where prompts became ambiguous before scaling up.
01
Pilot 50 prompts
Text input, one desired output modality. Use manual review to make the “right format” clear without spelling it out.
02
Scale to 500 single-input / single-output prompts
Cover varied tasks and modalities while keeping the annotation problem manageable.
03
Add interleaved inputs, then interleaved outputs
Introduce combinations of modalities in the input, then ask for combinations in the output.
04
Review every final prompt twice
At least two authors checked clarity, correctness, diversity and whether the gold output modalities made sense. Disagreements were discussed.
7,165 rubric criteria1,099 prompt-modality groupsmedian 6 criteria per groupall prompts in English
06 / Scoring
Separate “missing” from “wrong”.
MMI uses two complementary scores. The first grades answer quality. The second checks whether the model returned the required formats at all.
MMI Value = average across prompts of
mean (score for each expected modality)
If a required modality is missing, its score is 0.
mean (score for each expected modality)
If a required modality is missing, its score is 0.
The MMI Value is quality-aware. An LLM judge grades human-written criteria for each expected modality. Every modality gets equal weight within a prompt.
MPS = mean per-prompt F1
F1 = 2 × precision × recall
precision + recall
F1 = 2 × precision × recall
precision + recall
The Modality Presence Score ignores content quality. It asks only: did the response contain the correct set of modalities?
Why lead with MPS? The evaluated models returned too few non-text assets for MMI Value to be a meaningful main comparison.
07 / Try it yourself
See why presence comes first.
Set the expected output formats, choose what the model actually returned, then adjust answer quality. The two scores tell different stories.
Expected output modalities
Returned by the model
Quality of returned expected modalities
Live benchmark readout
0
Modality Presence Score / MPS
Choose an expected modality to begin.
precision0
recall0
quality-aware MMI0
08 / Main evaluation
High precision. Low recall.
Models rarely returned an output modality that was not wanted. The problem was omission: they often failed to return the modalities the prompt required.
MPS / per-prompt mean F1
GPT-5.434.9
Gemini 3.1 Flash27.2
Gemini 3.1 Pro25.4
Llama 4 Maverick16.3
Claude Opus 4.615.6
Signal: every model scored near 100 on precision.
Table 4 / all figures are percentages
| Model | MPS | Recall | Pass | Input fail |
|---|---|---|---|---|
| GPT-5.4 | 34.9 | 33.8 | 31.0 | 30.1 |
| Gemini 3.1 Flash | 27.2 | 25.4 | 20.3 | 8.2 |
| Gemini 3.1 Pro | 25.4 | 23.6 | 18.6 | 8.0 |
| Llama 4 Maverick | 16.3 | 15.1 | 12.0 | 36.5 |
| Claude Opus 4.6 | 15.6 | 14.5 | 11.3 | 37.0 |
09 / Interpretation
The failure is mostly “not enough”, not “wrong”.
01 / OMISSION
Models default to text.
Even when the user asks for a visual, spoken, moving, or downloadable artefact, text remains the safest fallback.
02 / AUDIO
Audio was the hardest.
No evaluated model generated even a link to an audio snippet for prompts requiring audio output.
03 / INPUTS
Input support still matters.
Input failure rates ranged from 8.0% for Gemini 3.1 Pro to 37.0% for Claude Opus 4.6.
Important nuance
These results describe bare provider APIs with default parameters, no system prompt, and no tools. They are not a complete ranking of the assistants people use in products.
10 / Measurement detail
What counts as “returned”?
MMI uses a lenient default because a user may be satisfied by a working link, even if the model did not render the asset itself.
01
Native asset
The API returns actual image, audio, video or document data.
strict + lenient
02
Recognised URL
A link points to an asset of the requested modality, such as a video host.
lenient
03
Fallback judge
Gemini 3 Flash parses the response when direct detection finds nothing.
lenient
WHY THIS CHOICE MATTERS
A working link can still be a useful answer.
But lenient detection can blur the difference between a model that generated an asset and one that retrieved or linked to one. The paper also reports strict native precision separately.
11 / Rubric validation
Can an LLM judge multimodal quality?
The authors ran a separate tool-scaffolded experiment to create enough image, audio and video assets to exercise the rubrics. It was for judge validation, not model ranking.
JUDGE / HUMAN AGREEMENT
70.8%overall agreement
0.41Scott's pi
1,499judged triples
71%modalities gradeable
Text 70.9%Image 66.5%Audio 73.6%Video 74.4%
WHAT THE RUBRIC DOES
A
One modality at a time
The judge receives only the payload for the modality being graded.
B
Atomic criteria
Each criterion asserts one concrete, objectively checkable property.
C
Blind-friendly
The criterion must be decidable from that asset alone, without relying on other outputs.
Read the result carefully
Agreement is useful, but not perfect. The paper flags judge mistakes, subjective prompts, rubric-writing difficulty and possible self-preference because Gemini judged many Gemini-family assets.
12 / So what?
MMI is a reality check for “omni”.
THE PRODUCT LESSON
Capability claims should be tested at the moment of delivery.
A model may ingest many modalities, yet still fail the user because it answers in the wrong form or omits a required artefact.
LIMITATIONS TO CARRY FORWARD
1
MMI Value is under-tested
Too few assets were returned in the main run for quality scoring to be a meaningful headline.
2
APIs are not products
Tools, orchestration and system prompts could change the observed behaviour substantially.
3
Correctness can be subjective
Open-ended requests create ambiguity about both content and acceptable output modality.
4
English only, for now
The authors identify multilingual benchmark data as future work.
13 / How to implement it
Build an MMI-style eval for your own agent.
Use the paper as a blueprint, then adapt the modalities, tools and rubrics to the workflows your product actually promises.
List the input and output modalities your agent claims to support. Include what counts as a native asset, a retrieved link, a generated file, or a tool-mediated result.
Start with ordinary user language. Do not name the output format when the point is to test whether the agent can infer it. Include single-modality and multi-modality tasks.
For each prompt, record the minimum set of modalities that makes the answer useful. Have at least two reviewers resolve ambiguity before evaluation.
Write criteria that are binary-decidable from one output modality. Include delivery and integrity checks where relevant, then concrete content criteria. Keep each criterion single-purpose.
Capture native payloads, parse links, and log fallback detections separately. Report both strict and lenient presence so product teams can see what the user received versus what the model generated.
Use MPS, precision, recall, pass rate and input failure rate to diagnose coverage. Only headline MMI Value once enough outputs are present to grade answer quality reliably.
14 / Practical loop
Make the benchmark useful to a product team.
RECOMMENDED REPORTING LOOP
ARun the same prompt set across model + harness versions
Keep the user request fixed while changing tools, routing or prompts.
BTrack presence and quality separately
Do not hide missing assets inside one blended score.
CSlice by modality and input type
Find whether the bottleneck is ingestion, tool choice, generation or content quality.
DONE CONDITION
You can answer four questions.
Product principle
Evaluate the complete agent experience, but preserve a model-level diagnostic underneath it.
15 / Takeaways
True multimodality is a delivery problem.
01
Test the output choice
Understanding an image is only half of a multimodal interaction.
02
Separate absence from error
MPS tells you whether the right formats appeared. MMI Value tells you how good they were.
03
Benchmark the whole harness
Tools and orchestration can change what users actually receive.
If an agent can only explain the video it should have made, it is not yet omni in the way the user experiences it.
16 / Sources
Read the paper. Run the benchmark.
This deck is a visual explainer of the attached paper. Figures and numerical claims are transcribed from the paper, with wording simplified for readability.
01Patel, Rohit; Hupkes, Dieuwke; Strader, Sloan. Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models. arXiv v1, August 26, 2026. arXiv
02Code and evaluation harness: github.com/facebookresearch/modality-maturity-index
03Dataset and prompt viewer: huggingface.co/datasets/facebook/mmi
KEY IDEA TO KEEP
A model is only as multimodal as the useful outputs it can deliver.