PaperlayerView exact revision on arXiv

Source-linked AI summary

Exemplar2VQA: A Scalable Exemplar-Driven Visual Question Answering Generation Framework via Multi-Agent Coding

Jiayu Ying, Qijian Tian, Ruijie Xu, Xinnan Zhu, Daoguo Dong, Jiachen Xu, Xin Tan

arXiv:2609.37655v1cs.CV

TL;DR

Spatial QA data is scarce, manual annotation is costly, and direct LLM synthesis struggles with spatial computation. Exemplar2VQA uses exemplar-driven multi-agent coding and deterministic geometric utilities to generate synthetic QA in simulated environments. Indoor-only synthetic training improves benchmark performance and transfers to outdoor and mixed-scene settings.

  • Problem

    Large-scale, high-quality 3D spatial QA data is scarce, while manual annotation is costly and direct LLM synthesis struggles with geometric computation.

    Source: §1 · pp. 1–2

  • Method

    Exemplar2VQA uses multi-agent coding, geometric utilities, and simulated-environment feedback to scale query exemplars into synthetic spatial QA datasets.

    Source: §1 · p. 2

  • Results

    Exemplar2VQA-generated synthetic indoor training improves benchmark performance and transfers to outdoor and mixed-scene evaluations without outdoor training samples.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework offers a scalable, cost-effective paradigm for acquiring high-fidelity spatial data to advance Embodied AI.

    Source: §1 · p. 2

    The current framework is optimized for static, object-centric configurations, relies on explicit simulator metadata, and its success rate is bounded by the coding capabilities of its foundational LLM.

    Source: §A.7 · p. 23

Abstract

from arXiv · show

Advancing spatial intelligence in Multimodal Large Language Models (MLLMs) is bottlenecked by the scarcity of complex, scalable 3D question-answer (QA) data. While manual annotation is labor-intensive, directly utilizing LLMs to synthesize these QA pairs often fails due to their inherent deficiencies in spatial and geometric computation. We introduce Exemplar2VQA, a scalable exemplar-driven visual question answering generation framework that rapidly synthesizes large-scale spatial QA pairs in simulated environments via multi-agent coding. By equipping collaborative agents with a meticulously designed library of geometric utilities, Exemplar2VQA bypasses LLMs' spatial reasoning flaws through deterministic code execution. Crucially, the framework exhibits remarkable versatility: taking diverse static object-centric spatial query templates as exemplars, it seamlessly and autonomously scales them into massive, high-fidelity synthetic datasets. Fine-tuning Qwen2.5-VL (3B/7B) exclusively on Exemplar2VQA-generated synthetic indoor data yields significant performance improvements across various diverse benchmarks. Furthermore, its effectiveness is not limited to in-domain indoor datasets but also robustly extends to outdoor and mixed-scene benchmarks. These results establish Exemplar2VQA as a scalable and powerful paradigm for bridging the sim-to-real gap in Embodied AI. Our code is at https://github.com/yingjiayu12/Exemplar2VQA

1 Introduction

Exemplar2VQA addresses costly, unscalable 3D annotation and the spatial computation failures of direct LLM synthesis with exemplar-driven, code-based generation. Models fine-tuned on its synthetic indoor data improve across indoor, outdoor, and mixed-scene evaluations.

Motivation

  • Spatial QA synthesis faces costly, unscalable manual annotation, spatial hallucinations from direct LLM prompting, and rigid automated pipelines.The text links LLM synthesis failures to limited deterministic geometric computation.Source: §1 · pp. 1–2

Results

  • Exemplar2VQA-fine-tuned models improve on indoor benchmarks, transfer to outdoor and mixed-scene environments without outdoor training samples, and outperform human-annotated-data baselines.The reported evaluation uses Qwen2.5-VL 3B and 7B models fine-tuned exclusively on synthetic indoor data.Source: §1 · p. 2

Contribution

  • The framework uses collaborative agents, geometric utilities, and deterministic mathematical execution to generate spatial QA instead of relying on LLM spatial computation.Physics-constrained simulator feedback also supports the generation process.Source: §1 · p. 2
  • Diverse static object-centric query exemplars can be autonomously scaled into massive, high-quality synthetic datasets without manual intervention.Source: §1 · p. 2

2 Related Work

Related work spans 3D and video spatial benchmarks, synthetic multimodal data generation, and physics-enabled simulators. The paper positions Exemplar2VQA against human-curated or rigidly generated datasets and prompting-based synthesis that produces spatial hallucinations.

Spatial Understanding in Vision-Language Models

  • Spatial benchmarks have shifted toward 3D and video, while existing spatial datasets rely on human-curated annotations or rigid rule-based generation.The cited examples include UniQA-3D, ViewSpatial-Bench [24], Multi-SpatialMLLM [25], and MMSI-Bench [27].Source: §2 · pp. 2–3

Automated Synthetic Data Generation

  • Prompting-based synthetic data generation scales multimodal instruction data, but direct prompting for precise 3D relationships frequently produces spatial hallucinations [31] [32] [33].The passage contrasts this with Exemplar2VQA's different paradigm.Source: §2 · p. 3

Embodied Environments and Sim-to-Real Transfer

  • Physics-enabled simulators provide controllable, scalable environments and scene metadata for constructing ground-truth spatial relationships [8] [9] [35] [36] [37].Source: §2 · p. 3

3 Method

Exemplar2VQA maps simulated environments and spatial-query exemplars to image, question, and answer datasets. Its dual-track agent pipeline combines simulator navigation, metadata-based QA generation, and deterministic geometric computation.

3.1 Problem Formulation and Exemplar2VQA Overview

  • Given a simulated environment and query exemplar, Exemplar2VQA generates a dataset of images, instantiated questions, and ground-truth answers.The images follow user-specified camera viewpoints; the mapping is written as D = ΦE2V(S, E).Source: §3.1 · p. 3

3.2 Dual-Track Multi-Agent Collaborative Coding

  • The camera track turns natural-language instructions into simulator navigation and multi-view observations, while the QA track uses scene metadata to instantiate questions and compute answers.The camera pipeline uses Architect, Coder, Reviewer, and Refiner agents; simulator feedback supports iterative code refinement.Source: Figure 1 · p. 4, Figure 2 · p. 4, §3.2 · pp. 4–5
  • The QA track abstracts an exemplar into a parameterized schema, generates Python code using geometric utilities, and iteratively debugs execution faults before instantiating QA pairs.The compiled script iterates over scene-metadata permutations to produce the dataset.Source: §3.2 · p. 5

3.3 Geometric API Orchestration

  • A deterministic geometric-utility library and built-in filters offload spatial calculations from the LLM and discard ambiguous layouts before QA generation.The code-generation agent orchestrates prerequisites such as projecting 3D metadata into 2D footprints before testing spatial relations.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate Exemplar2VQA-generated data on primary and zero-shot generalization benchmarks, and assess agent design, direct LLM annotation, and synthesis efficiency. The reported results show gains over base and spatial-data-finetuned models, with trade-offs on some tasks.

4.1 Experimental Settings

  • Evaluation covers MMSI-Bench and VSI-Bench for primary performance, plus SpaCE-10 and ViewSpatial-Bench for zero-shot transfer; the latter tests unseen indoor queries and mixed indoor/outdoor scenes.The zero-shot evaluations use Qwen2.5-VL-7B fine-tuned solely on MMSI-Bench synthetic data.Source: §4.1 · pp. 6–7

4.2 Performance on Primary Spatial Benchmarks

  • 28.2% overall on MMSI-Bench surpasses larger models; 42.9% overall on VSI-Bench outperforms proprietary and 72B open-source models and spatial-data-finetuned baselines.The results are reported for Qwen2.5-VL-7B on MMSI-Bench and Qwen2.5-VL-3B on VSI-Bench, respectively.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 Performance on Zero-Shot Generalization Benchmarks

  • On SpaCE-10, Exemplar2VQA improves overall performance by 8.9% over the base model and leads other spatial-data-finetuned baselines, with gains in 7 of 8 categories.Scene Quantification drops, which the authors attribute to its functional-zone grouping requirement versus the pipeline’s instance-level bounding-box geometry.Source: §4.3 · pp. 8–9, Table 3 · p. 8
  • On ViewSpatial-Bench, indoor-only synthetic training yields an 8.5% overall boost, including a 23.5% gain in person-perspective Object View Orientation.The benchmark includes mixed indoor and outdoor scenes.Source: Table 4 · p. 8, §4.3 · p. 9

4.4 Ablation Study on Multi-Agent Architectures and Generator Capacity

  • On 100 seed examples, a single-agent configuration achieved 34.0% QA-generation and 60.0% camera-navigation success; progressively separating debugging from code patching raised peak rates to 92.0% and 84.0%, respectively.Success means executed code produces mathematically and semantically correct final outputs.Source: §4.4 · p. 9, Table 5 · p. 9
  • Replacing Qwen3-Coder-30B with Qwen3.5-9B drops QA-generation success from 92.0% to 48.0% and camera-trajectory success from 84.0% to 56.0%.The comparison uses the same four-agent pipeline.Source: §4.4 · pp. 9–10

4.5 Comparison with Direct LLM Annotation

  • Gemini3.5-Flash annotation of 10K VSI-Bench examples underperforms code-based generation and degrades average performance relative to the base model.The authors identify cross-view consistency, de-duplication, and stable context tracking as requirements not guaranteed by direct LLM/VLM labeling.Source: Table 6 · p. 9, §4.5 · p. 10

4.6 Remarks on Efficiency

  • Across two GPUs, code synthesis averages 3 minutes for camera trajectories and 2 minutes for QA scripts; subsequent scene execution scales linearly without further LLM intervention.Source: §4.6 · p. 10

5 Conclusion

Exemplar2VQA uses geometric utilities and simulator feedback to generate spatial QA datasets, and its models improve on spatial benchmarks and generalize from synthetic indoor data to unseen outdoor environments.

5 Conclusion

  • Exemplar2VQA-trained models gain substantially over base models on MMSI-Bench and VSI-Bench, and outperform GPT-4o specifically on VSI-Bench.The conclusion reports these benchmark comparisons without giving numerical values.Source: §5 · p. 10
  • Zero-shot evaluations on ViewSpatial-Bench and SpaCE-10 show generalization to complex, unseen outdoor environments from spatial priors learned exclusively on synthetic indoor data.Source: §5 · p. 10
  • Exemplar2VQA scales query exemplars into high-fidelity spatial QA datasets using geometric utilities and physics-constrained simulator feedback.The framework uses deterministic mathematical execution to mitigate MLLMs’ spatial reasoning limitations.Source: §5 · p. 10

A.1.1 Training Hyperparameters

The appendix describes the shared training setup and separate hyperparameters for Qwen2.5-VL-3B on VSI-Bench and Qwen2.5-VL-7B on MMSI-Bench.

A.1.1 Training Hyperparameters

  • Both models used the swift distributed framework, DeepSpeed ZeRO-2, and NEFTune with noise alpha 5.Inference ran on one NVIDIA H200 GPU; fine-tuning used four H200 GPUs, and data generation used two RTX A6000 GPUs.Source: §A.1.1 · p. 15
  • Qwen2.5-VL-3B on VSI-Bench used 3 epochs, a 1e-5 learning rate, and a global effective batch size of 32.It used a cosine scheduler, 0.05 warmup ratio, 0.01 weight decay, per-device batch size 4, and 2 gradient accumulation steps.Source: §A.1.1 · p. 15
  • Qwen2.5-VL-7B on MMSI-Bench used LoRA with rank 16, alpha 32, 3 epochs, and a 1e-4 learning rate.Training used bf16, global effective batch size 32, and maximum image resolution 1,003,520 pixels.Source: §A.1.1 · p. 15

A.1.2 Data Formatting and Training Prompts

Training data formatting was adapted to each benchmark’s input modality and output requirements, with prompts designed to support predictable, evaluation-friendly answers.

A.1.2 Data Formatting and Training Prompts

  • The training datasets were formatted according to each task’s input modality, and explicit formatting guides models toward deterministic, evaluation-friendly outputs.The human prompt includes a task instruction, answer instruction, question field, and answer field.Source: §A.1.2 · pp. 15–16
  • For VSI-Bench, multi-view images were converted into continuous rotation videos, and training used a single video sequence with answers in XML-style tags.The prompt asks the model to analyze object arrangements in the video and choose one answer.Source: §A.1.2 · p. 15
  • For MMSI-Bench, training used one <image> token per input frame and prompted analysis of spatial arrangements and motion across images.The model was instructed to output only the selected answer letter.Source: §A.1.2 · pp. 15–16

A.1.3 Evaluation Prompts

Evaluation prompts mirror the training output constraints while adapting image or video modality tokens to each benchmark.

A.1.3 Evaluation Prompts

  • Evaluation prompts mirror training’s output-format constraints and adapt only the modality tokens to the benchmark’s video or image input.Source: §A.1.3 · p. 16
  • The VSI-Bench evaluation prompt presents a video and asks for one answer selected from the listed options.Source: §A.1.3 · p. 16
  • The MMSI-Bench evaluation prompt presents a sequence of images and asks for one answer selected from the listed options.Source: §A.1.3 · p. 16

A.2 Infinite Generation via ProcTHOR and Holodeck

Although experiments capped generated datasets at 6K and 10K pairs, Exemplar2VQA is designed to scale with available simulated environments. ProcTHOR supports broad scene expansion, while Holodeck enables prompt-driven custom scenes.

A.2 Infinite Generation via ProcTHOR and Holodeck

  • Exemplar2VQA’s generation capacity is limited by the diversity and quantity of available 3D simulated environments, rather than a fixed generation capacity.Experiments capped datasets at 6K and 10K pairs to conserve computational resources.Source: §A.2 · pp. 16–17
  • ProcTHOR provides tens of thousands of procedurally generated interactive houses and floorplans that Exemplar2VQA can process without manual curation.The framework extracts varied object configurations, room layouts, and occlusions to maintain domain diversity and reduce geometric repetition.Source: §A.2 · p. 17
  • Holodeck generates new AI2-THOR rooms from natural-language definitions and outputs their scene metadata.This supports targeted domain adaptation and specific spatial scenarios.Source: §A.2 · p. 17
  • Combining ProcTHOR’s broad expansion with Holodeck’s customized scene creation provides a continuously expanding source of high-fidelity spatial configurations.The framework describes this combination as a virtually unlimited sandbox.Source: §A.2 · p. 17

A.3 Extended Experiments: Template Adaptation and QA Visualizations

Extended experiments address template adaptation and OOD generalization, while repeated MMSI-Bench runs examine the stability of reported gains across random seeds.

A.3 Extended Experiments: Template Adaptation and QA Visualizations

  • The extended experiments are presented as evidence of Exemplar2VQA’s versatility and OOD generalization.The framework is described as not being confined to a predefined set of tasks.Source: §A.3 · p. 17
  • Exemplar2VQA adapts static object-centric spatial query templates from diverse downstream benchmarks.The section frames this as a way to extend beyond predefined tasks.Source: §A.3 · p. 17

A.3.3 Stability Across Random Seeds

  • MMSI-Bench fine-tuning was rerun with three random seeds using the same training data to examine result stability.Source: §A.3.3 · p. 19
  • The repeated MMSI-Bench runs suggest that the reported gains are stable across seeds.Source: §A.3.3 · p. 19
  • The magnitude of MMSI-Bench improvement varies across sub-categories.Source: §A.3.3 · p. 19
  • The stability assessment keeps the training data constant while changing the random seed.Source: §A.3.3 · p. 19

A.3.1 Evaluation Results on OSR-Bench

This section reports benchmark results for Exemplar2VQA, including OSR-Bench performance under a view-transfer gap and gains on VSI-Bench and SpaCE-10. It also notes the absence of error bars and describes the data-generation setup and reproducibility information.

A.3.1 Evaluation Results on OSR-Bench

  • The evaluation fine-tuned Qwen2.5-VL-7B with LoRA on approximately 6K synthesized QA pairs following OSR-Bench templates for object counting, relative distance, and relative direction.OSR-Bench contains over 153,000 diverse QA pairs; training used four NVIDIA H200 GPUs and the same hyperparameters as the MMSI-Bench evaluation.Source: §A.3.1 · p. 17
  • 36.5% overall average outperformed Qwen2.5-VL-72B (33.5%) and LLaVA-v1.5-13B (31.6%), while leading Object Count (59.0%) and Relative Distance (41.1%).This was an 8.2% absolute overall accuracy boost over base Qwen2.5-VL-7B (28.3% to 36.5%), including increases of +12.7% in Object Count and +10.0% in Relative Distance.Source: §A.3.1 · p. 18
  • Training used four discrete sparse surround-view images per scene, not OSR-Bench’s continuous equirectangular 360° panoramas; the authors deliberately omitted panorama rendering to test QA generation’s utility and geometric accuracy.Although the pipeline can stitch and project continuous panoramas, the authors state that the gains under this zero-shot view-transfer setting suffice to validate their premise.Source: §A.3.1 · p. 18

2. Limitations

  • The authors say the work’s code will be open-sourced upon publication.The checklist answer affirms open access to code and data with instructions for reproducing the main experiments.Source: §2 · p. 29
  • The authors say training and test details and computational resources are specified in Section 4 and Appendix A.1.The checklist answers affirm that the paper provides details needed to understand the results and reproduce the experiments.Source: §2 · pp. 29–30
  • +7.6% on VSI-Bench and +8.9% on SpaCE-10 are reported as absolute gains over baselines.The authors state that they did not run multiple independent fine-tuning trials to calculate error bars, citing the computational cost of repeated 3B and 7B MLLM fine-tuning.Source: §2 · p. 29
  • The paper discusses the work’s broader impact and societal implications in Appendix A.4.The checklist answer indicates that the paper addresses both potential positive and negative societal impacts.Source: §2 · pp. 30–31
  • The multi-agent framework uses Qwen3-Coder-30B-A3B-Instruct in Architect, Coder, Reviewer, and Refiner roles to generate code and synthesize spatial QA datasets.The paper says this LLM usage is described in Section 3.Source: §2 · p. 33

Related papers

Loading 2609.37655v1…