Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

Weihao Bo^1,2,*, Shan Zhang^3,†, Yanpeng Sun^4,*, Jie Liu^2,5, Yongke Yao^2,6, Jinhao Du⁷, Wei He², Kai Zou⁸, Zechao Li^1,†, Jingdong Wang^2,‡

¹Nanjing University of Science and Technology ²Baidu Inc ³AIML, Adelaide University ⁴SUTD ⁵Southeast University ⁶East China Normal University ⁷University of Oxford ⁸NetMind.ai

^*Equal Contribution ^†Corresponding Author ^‡Project Lead

arXiv Code Dataset

This video gives an overview of Diagram-MMU: the diagram domains it covers, the tasks and evaluation settings it designs, and the main experimental findings.

Video · Abstract · Benchmark · Main Results · Findings · BibTeX

Diagram-MMU covers six scientific diagram types including charts, planar geometry, 3D shapes, graphs, chemistry, and circuit diagrams — for the first time covering chemistry and circuits for diagram-to-code tasks. We adopt TikZ code, which is commonly used in paper writing in LaTeX and integrates directly into Overleaf and Prism. The final 3,744 unique diagrams paired with 18,305 evaluation samples are cross-validated by 13 graduate students.

Abstract

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks.

The Diagram-MMU Benchmark

Three Tasks for Vibe Writing Workspaces

Diagram-MMU evaluates MLLMs on three tasks that benefit scientific writing:

Diagram-to-Code Parsing (D2C-P): parse the input diagram into editable TikZ code so it can be included in a manuscript.
Diagram-to-Code Editing (D2C-E): generate the TikZ code whose rendered diagram meets a modification requirement, across four editing dimensions — text, color, scope, and layout.
Diagram Question Answering (DQA): answer descriptive and reasoning questions (including hypothetical what-if questions) about the diagram, grounded in domain-specific semantics and laws.

Task overview of Diagram-MMU: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.

Data Collection & Curation

TikZ sources are collected from official package handbooks (PGFPlots, CircuiTikZ, TKZ-Euclide, ChemFig, TikZ-Network) and community resources (texample.net, TeX Stack Exchange, GitHub tikz_favorites), yielding 6,849 candidate code pieces. After simplification, compile-and-render verification, de-duplication, and MLLM-assisted domain classification with manual checks, the final benchmark contains 3,744 unique diagrams paired with 18,305 evaluation samples, cross-validated by 13 graduate students.

Data collection and agentic task generation pipeline — Data collection and the agentic pipeline for D2C-E and DQA task generation, with agent-check and human-check stages.

Domain	D2C-P #Q	TikZ Package	DQA #Q	Domain Knowledge
Charts	959	`pgfplots`	1,834	statistics, trend/extremum analysis
Planar Geom.	598	`tikz/tkz-euclide`	1,118	Euclidean notation/formulas
3D Shapes	236	`pgfplots`	455	solid geometry, cross-sections
Graph	1,356	`tikz`	2,679	degree, connectivity, paths, tree
Chemistry	187	`chemfig`	366	bond, atom, functional groups
Circuit	403	`circuitikz`	694	topology, electrical/power laws

D2C-E metadata breakdown — Metadata breakdown of D2C-E editing dimensions across text, color, scope, and layout.

Multi-Level Evaluation Metrics

Object-level: a Semantic Object Model (SOM) is extracted from compiled TikZ, and four F1 scores (type, text, color, bbox) measure whether the code draws the correct basic objects; for editing, metrics are split into preserve-only and edit-only parts.
Code-level: CrystalBLEU measures TikZ syntactic correctness.
Image-level: SSIM, CLIP Score, LPIPS, and FID measure visual appearance similarity of rendered outputs.
Answer accuracy: DQA is graded by an LLM judge with binary scores.

16 Controllable Evaluation Settings: How to Think & How to Act

Diagram-MMU is the first benchmark to provide 16 controllable evaluation settings evaluating both foundational and agentic capability of current MLLMs: 3 for foundational ability and 13 for agentic ability. The four levels of agentic ability are defined as: (1) Context utilization: whether models can leverage task-relevant information from context; (2) Tool use: knowing when to invoke tools, what to query, and how to incorporate the results; (3) State management: whether models can build on prior outputs incrementally toward the final goal; (4) Planning: deciding which ability is needed and how to combine them for task solving. Importantly, for tool use we build our TikZ search tool as an MCP server, enabling MLLMs to selectively access relevant references, reducing the noise of web search and the context rot caused by loading full PDF manuals.

ID	Setting	Abbreviated Prompt	Capability
S1	Diagram-to-code parsing	Direct coding	Foundational
S2	+ Objects	Use the provided perception data for coding	Agentic · Context
S3	+ TikZ search tool	Use search tool for TikZ syntax and examples when necessary	Agentic · Tool use
S4	+ Objects (model-gen.)	First perceive basic objects in diagram, then generate code	Agentic · State
S5	+ S2 & S3	Plan where to call tool in synergy with perception data	Agentic · Planning
Diagram-to-Code Editing (S6–S11) and Diagram Question Answering (S12–S16) follow analogous settings.

Main Results

Main results on foundational capability (pass@1, averaged over 6 diagram types). All averages F1_avg, CrystalBLEU, and SC (higher is better). D2C-E object/code metrics are split into preserve (p) / edit (e). Bold is best in class; underline is second best.

Model	Diagram-to-Code Parsing			Diagram-to-Code Editing			DQA Acc.
Model	All	F1_avg	CBLEU	All	F1_avg (p/e)	CBLEU (p/e)	DQA Acc.
Closed-Source Multimodal Large Language Models
Gemini-3.1 Pro	48.38	49.94	32.88	51.22	60.60 / 40.84	42.02 / 2.98	86.29
Gemini-3.0 Pro	53.28	54.64	32.76	53.60	65.12 / 40.24	44.77 / 2.34	86.46
Gemini-3.0 Flash	50.97	51.19	30.67	49.01	59.29 / 37.24	42.21 / 2.17	84.07
GPT-5.2	51.88	51.35	28.81	46.15	55.59 / 31.15	38.75 / 1.15	81.67
Claude-4.6 Opus	52.41	52.23	31.27	51.18	62.56 / 31.95	43.59 / 1.42	68.75
Seed-2.0 Pro	50.89	47.57	32.56	43.73	53.03 / 30.14	37.49 / 1.41	75.90
Open-Source Multimodal Large Language Models
Qwen3.5-397B-A17B	52.72	51.38	32.92	50.82	62.75 / 36.35	43.66 / 1.91	83.42
Qwen3-VL-235B-A22B	55.98	55.11	37.92	52.25	63.79 / 33.38	43.05 / 1.85	63.60
Kimi-K2.5	56.97	57.48	36.13	52.13	63.12 / 36.52	42.11 / 2.02	79.74
Qwen3-VL-8B	48.26	44.66	33.31	19.00	21.26 / 9.95	16.29 / 0.51	47.12
InternVL3-38B	41.13	31.47	30.91	38.85	44.72 / 22.69	33.13 / 1.29	56.39
TikZero+ 10B	25.30	15.43	17.19	–	–	–	–

Key Findings

Foundational vs. Agentic Performance

Models can reason well over diagrams but struggle to code them. DQA accuracy is up to 86%, while D2C-P object-level F1 ranges 31–57%, revealing perception and coding limitations.
Models that fail on diagram-to-code parsing also struggle with editing (D2C-E) and answering tasks (DQA). The object F1 gap between the D2C-P success/failure sets reaches 20–40 points across most models.
Gemini-3.0 Pro achieves the most balanced profile across the three tasks.
Agency helps editing more than parsing, likely because textual editing instructions guide tool and context use.
Most models degrade from excessive retrieval or poorly targeted queries in the TikZ search process, while Claude-4.6 Opus has the strongest tool use ability.
Planning is the weakest agentic capability, especially on DQA (−0.3 to −8.8 accuracy degradation); only Claude-4.6 Opus consistently improves across all three tasks.

Fine-Grained Object Perception

Breakdown of F1 scores across perception dimensions — Breakdown of F1 scores (avg, type, bbox, color, text). Spatial grounding (bbox) is consistently the weakest perception dimension — F1_bbox drops sharply in D2C-P, especially for GPT-5.2, Claude-4.6 Opus, and Seed-2.0 Pro.

Pass@k evaluation across three tasks — Pass@k evaluation: performance improves with k, gains peak from pass@1 to pass@2 and saturate beyond k=4.

Performance split by D2C-P success or failure — D2C-E/DQA performance on diagrams split by D2C-P output (successful vs. failed rendering): models that fail D2C-P also score lower on D2C-E and DQA.

Agentic Capability Takeaways

        Context utilization: most models can use provided diagram objects to ground local edits (F1 gains of 4.1–10.6 on preserve), but struggle to integrate them into syntax or domain-knowledge reasoning.
      

        Tool use: only Claude-4.6 Opus consistently benefits from the MCP-based TikZ search tool; Gemini-3.1 Pro suffers from excessive retrieval without a stopping criterion, and Qwen3-VL-8B fails at issuing targeted queries.
      

        State management: most models fail to manage intermediate code states across steps; only Claude-4.6 Opus maintains coherence from coding to editing and reasoning (+3.8 on DQA).
      

        Planning: the weakest agentic capability — no model reliably composes multiple abilities as task complexity grows from D2C-P to DQA, where all six evaluated models degrade under optional code generation.
      

BibTeX

@article{bo2026diagrammmu,
  title   = {Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams},
  author  = {Bo, Weihao and Zhang, Shan and Sun, Yanpeng and Liu, Jie and
             Yao, Yongke and Du, Jinhao and He, Wei and Zou, Kai and
             Li, Zechao and Wang, Jingdong},
  year    = {2026}
}