Narrative video understanding requires more than recognizing visible events: models must extract evidence from dialogue, bind it to speakers, and use it to infer narrative meaning. Existing video benchmarks evaluate aspects of this problem, including video question answering, audio-visual reasoning, and temporal grounding, but rarely test whether these abilities form a coherent dialogue-to-plot evidence chain within the same clips.
We introduce CineMME, a human-verified benchmark designed to diagnose dialogue-grounded plot understanding, supported by approximately 1,600 expert person-hours of annotation. CineMME contains 975 curated short narrative video clips and two linked tracks: CineMME-Grounding, with 13,554 timestamped speech segments and 33,083 speaker face boxes for temporal dialogue localization, transcription, and active-speaker grounding; and CineMME-Reasoning, with 1,600 multiple-choice and 316 open-ended questions spanning perception, audio, narrative, social, and scene-presentation understanding.
Our main evaluation compares 20 MLLMs on reasoning and 9 omni-modal models on grounding. The results reveal a pronounced who-what-when decoupling: Gemini-3 Pro localizes dialogue in time (68.34% tIoU) and transcribes it with a WER of 28.99%, yet achieves only 7.74% vIoU for active-speaker grounding while achieving 67.66% MCQ accuracy on questions with extractable answers. Modality ablations further show that vision is not uniformly beneficial: it improves performance on perception-heavy questions but can hurt dialogue-centric reasoning. CineMME provides a compact diagnostic testbed for evaluating how multimodal video models connect dialogue, speakers, visual evidence, and narrative meaning.
To add a model, run the evaluation scripts in the code repository and open an issue with your predictions and the scores they produce.
Gemini-3 Pro localizes dialogue in time and transcribes much of it, yet localizes the speaking face in only 7.74% vIoU, the best score of any model. In the main table, the best open-weight model (Qwen3-Omni) reaches 3.18% vIoU.
Change in Gemini-3 Pro MCQ accuracy (points) when video frames are added to the audio track. Perception rises from 67.54% to 79.39%; Audio drops from 74.43% to 68.21%.
MCQ accuracy. Qwen3-Omni is the best open-weight model. Six graduate volunteers scored 77.64% on a 161-question subset (95% CI 70.60–83.39%).
| Qwen3-Omni input | tIoU ↑ | vIoU ↑ | WER ↓ |
|---|---|---|---|
| A+V, 16 frames | 54.66 | 3.82 | 72.20 |
| A+V, 64 frames | 50.48 | 3.67 | 75.92 |
| A+V, 256 frames | 50.40 | 3.01 | 71.52 |
| A+V, 1 FPS (main table) | 49.25 | 3.18 | 75.79 |
| Audio only | 63.58 | 1.92 | 36.77 |
Going from 16 to 256 frames per video keeps MCQ accuracy between 37.31% and 39.40% and vIoU between 3.01% and 3.82%. Audio-only input gives the best timing and transcription.
Short speech segments demand precise timing, and 86.0% of speaker face boxes cover less than 15% of the frame, which makes speaker grounding hard in multi-person scenes. Questions are spread across five categories, and 76.1% of them exceed 20 words.
| Benchmark | Videos | Q&A | Anno. | Modality | MCQ | Open-ended | Spatial grounding | Temporal grounding | Speech |
|---|---|---|---|---|---|---|---|---|---|
| Visual benchmarks | |||||||||
| Video-MME | 900 | 2.7K | M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| MLVU | 1,730 | 3.1K | A+M | V | ✓ | ✓ | ✗ | ✗ | ✗ |
| Omni benchmarks | |||||||||
| LongVALE | 8.4K | – | A | V+A | ✗ | ✗ | ✗ | ✓ | ✓ |
| AV-Odyssey | – | 4.6K | A+M | I+A | ✓ | ✗ | ✗ | ✗ | ✗ |
| Plot benchmarks | |||||||||
| CinePile | 9,396 | 304K | A+M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| Movie101/v2 | – | – | M | V | ✗ | ✓ | ✗ | ✗ | ✗ |
| Grounding benchmarks | |||||||||
| TimeLens-Bench | 4,279 | 6.5K | M | V | ✗ | ✗ | ✗ | ✓ | ✗ |
| SVAG-Bench | 688 | 19.6K | A+M | V | ✗ | ✗ | ✓ | ✓ | ✗ |
| CineMME (ours) | 975 | 1.9K | A+M | V+A | ✓ | ✓ | ✓ | ✓ | ✓ |
Anno.: A automatic, M manual. Modality: V video, A audio, I image. ✓ supported, ✗ not supported, – not reported.
@inproceedings{liu2026cinemme,
title = {{CineMME}: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding},
author = {Liu, Jin and Du, Dawei and Liu, Yexiang and Zhu, Sijie and Chen, Fan and Duan, Junxian and Huang, Huaibo and He, Ran and Wen, Longyin and Chen, Zhenfang},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Track on Evaluations and Datasets},
year = {2026},
note = {arXiv:2611.15161}
}