PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPath: Adaptive Coordination of Understanding and Generation for Unified Multimodal Reasoning

Hayes Bai, Yinyi Luo, Wenwen Wang, Qingsong Wen, Jindong Wang

arXiv:2605.11400v1cs.MM

TL;DR

UMM research lacks effective inference-time coordination between understanding and generation, especially when different inputs favor different paths. UniPath trains a path-conditioned executor and lightweight planner for adaptive path selection, improving performance over fixed strategies while retaining interpretable intermediate behaviors. Its main remaining limitation is that deployable path selection still trails oracle routing and generalizes imperfectly across domains.

  • Problem

    Different inputs favor different coordination paths, but existing methods either omit explicit inference-time coordination or impose fixed patterns.

  • Method

    UniPath represents solving as path selection and execution, trains a role-aligned path-conditioned executor, and uses a lightweight planner for input-dependent routing.

  • Results

    UniPath improves accuracy over fixed coordination strategies while providing lower token cost and interpretable reasoning traces.

  • Takeaways & Limitations

    Coordination policy can be treated as a first-class component of unified multimodal reasoning rather than a single universal protocol.

  • Takeaways & Limitations

    The deployable planner remains substantially below oracle routing, and robust cross-domain path-selection generalization remains challenging.

Abstract

from arXiv · show

Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture. However, it remains underexplored how to effectively coordinate these two capabilities for more effective and efficient reasoning. Existing coordination approaches either perform coupling during training, without explicit inference-time coordination, or impose a fixed coordination pattern for all inputs. In this work, we show that multimodal tasks exhibit substantial coordination-path diversity: different inputs favor different coordination paths. This suggests that exploiting such diversity is key to improving performance. We propose UniPath, a framework for adaptively modeling and exploiting coordination-path diversity. Instead of enforcing a single coordination pattern, we represent task solving as the selection and execution of a path, ranging from direct answering to textual inference, visual-thought construction, and hypothesis-based exploration. We construct role-aligned trajectories to train a path-conditioned executor and introduce a lightweight planner mechanism to enable input-dependent path selection. Experiments show that leveraging coordination-path diversity improves performance over fixed coordination strategies while providing interpretable intermediate behaviors. The code is available at:https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/unipath.

1 Introduction

UMMs need adaptive coordination because inputs benefit from different reasoning paths, while fixed or insufficiently planned coordination can waste computation and miss useful intermediate reasoning. UniPath addresses this with a planner-executor system that selects and executes input-dependent paths.

  • UMMs combine understanding and generation, motivating coordination that uses both capabilities effectively.
  • Suitable reasoning paths can reduce unnecessary output tokens, while fixed long paths waste computation and may increase errors.
  • Different subjects and instances favor different paths, and many inputs are solved correctly by only a subset of paths.
  • Practical adaptive coordination requires path categorization, role-aligned training data, and a planner that generalizes to instance-level selection.
  • UniPath trains a path-conditioned executor on role-aligned trajectories and uses a planner to select among five representative coordination paths.
  • The system targets improved accuracy, lower token cost, and interpretable reasoning traces through per-input path selection.

2 Related Work

Prior work coordinates understanding and generation through training alignment, intermediate representations, or fixed inference protocols. UniPath instead treats coordination as adaptive path selection followed by execution of the selected role sequence.

  • Unified multimodal models integrate understanding and generation within a single architecture using unified token, diffusion, or flow-based designs.
  • Training-based coordination improves perception-synthesis consistency but does not specify how capabilities should coordinate during inference.
  • Intermediate multimodal reasoning methods expose coordination, but fixed structures do not distinguish which functional roles different inputs require.
  • UniPath selects a coordination path first and then executes its role sequence, replacing a universal protocol with adaptive exploitation.

3 Methodology

UniPath formulates multimodal reasoning as input-dependent coordination-path selection and execution rather than a fixed sequence of understanding and generation steps. Its planner-executor system uses role-aligned trajectories, staged executor training, and calibrated path selection to support diverse coordination strategies.

  • Problem Formulation: UniPath represents an input as x = (q, I) and introduces a coordination path p that organizes intermediate states before producing the final output.The path-conditioned executor receives the original input and selected path, then generates the corresponding trace and answer.
  • Coordination Categorization: The framework defines five functional roles—understanding, reasoning, construction, hypothesis, and answer—to describe how multimodal capabilities contribute during inference.These roles abstract recurring operations in visual question answering, multimodal chain-of-thought, and interleaved understanding-generation systems.
  • Coordination Path Space: UniPath uses five representative paths: direct answering, explicit understanding, textual reasoning, visual-thought construction, and hypothesis exploration.Each path is centered on a core role with surrounding steps needed for execution, keeping the path space compact while covering qualitatively different strategies.
  • Planner-Executor Framework: The planner selects a path conditioned on the input, while the UMM executor follows that path and returns intermediate states and the final output.This planner-executor interface avoids assuming one fixed coordination pattern for every input.
  • Executor Training: Role-aligned trajectories pair each input with a path label and arrange tagged segments in the path’s role order for executor training.The executor uses a staged four-stage LoRA curriculum, role-weighted language modeling, and aligned visual-thought supervision for construction or hypothesis paths.
  • Planner Training and Selection: The planner is trained as a multi-label predictor because multiple paths can solve the same input, then uses calibrated scores to choose one path at inference time.Weighted binary cross-entropy preserves multi-path supervision, while query-form calibration adjusts planner scores using surface cues correlated with coordination needs.

4 Experiments

UniPath is evaluated across understanding, generation, consistency, planner behavior, and token-accuracy tradeoffs. The experiments show gains over the BAGEL backbone, dataset-dependent path selection, and improved accuracy with fewer output tokens.

  • Main results: UniPath improves understanding accuracy over BAGEL across all evaluated datasets, with gains of +4.3% on MMMU, +4.4% on MMBench-EN, and +7.7% on MMStar.The improvement on MathVista is smaller at +0.8%.
  • Main results: UniPath improves BAGEL from 78.81 to 80.00 on GenEval and from 0.3989 to 0.4100 on WISE.These generation results use the trained executor without planner routing and report relative gains of +1.5% and +2.8%, respectively.
  • Understanding-generation consistency: The executor improves UnifiedBench overall performance from 0.8346 to 0.8380, supporting the effect of aligned visual-thought supervision on understanding-generation consistency.UnifiedBench evaluates whether image information survives a reconstruction loop.
  • Planner analysis: The planner selects different paths across benchmarks: MMMU is mostly assigned to pC, MMBench-EN and MMStar to pA, while MathVista has a more balanced distribution.The distributions indicate that the planner does not apply a fixed preference for deeper reasoning.
  • Token-accuracy tradeoff: Across five understanding benchmarks, UniPath uses substantially fewer output tokens while matching or improving accuracy on most benchmarks.The reported comparison omits AD-Loop because per-example outputs needed for token accounting are unavailable.

5 Conclusion and Limitations

UniPath treats coordination policy as a first-class component of unified multimodal reasoning, using a planner to select among paths for each input. Different examples favor different paths, but robust path selection remains an unresolved limitation.

  • A query-form calibrated planner selects a coordination path for each input, while the executor follows that path through a unified interface.
  • Different examples favor different paths, and oracle selection remains far above any fixed path, indicating strong complementarity among coordination patterns.
  • The deployable planner still leaves a large gap to oracle routing, and learning a planner that generalizes robustly across domains remains challenging.
  • Subject domains favor different paths, while the oracle row remains substantially higher than any fixed path on MMMU.
  • Path complementarity also appears at the individual-question level through sparse, non-identical correctness patterns across examples.

B Training Data Construction

UniPath constructs path-aligned trajectories from understanding and generation sources, covering representative coordination paths with role-specific intermediate operations.

  • Path-aligned trajectories are constructed from both understanding and generation sources according to the defined coordination space.
  • Perception-only pU trajectories use VQAv2, while pR combines ScienceQA understanding data with LAION-Aesthetics-High-Resolution-GoT generation data.
  • ScienceQA trajectories combine questions with answer choices, map answer indices to option text, and use lecture and solution fields as textual reasoning.
  • pC examples require intermediate visual-thought construction using CoMT and CoF-SFT understanding data and FLUX-Reason-6M generation data.

C Benchmark and Metric Details

The evaluation covers multimodal understanding, image generation, and understanding-generation consistency, with metrics defined for interpreting the reported results.

  • Metric details: Understanding and GenEval scores are accuracies in percent, WISE reports WiScore, and Avg. is the mean over five understanding benchmarks when all scores are available.
  • Understanding benchmarks: The benchmark suite includes MMMU, MMBench-EN, MMBench-CN, MathVista, and MMStar for multimodal understanding.
  • Generation benchmarks: GenEval measures text-to-image object binding and compositional accuracy, while WISE evaluates broader text-to-image alignment across multiple domains.
  • Understanding-generation consistency: UnifiedBench evaluates image-information preservation across an image-to-text-to-image reconstruction loop using CLIP, DINOv2, DINOv3, and LongCLIP similarities.

D Aligned Visual Thought Analysis

Aligned visual thought keeps intermediate reasoning readable while using image-derived supervision, outperforming explicit latent or image feedback in routed MMMU analysis and reducing runtime.

  • Experimental setup: The analysis compares readable visual-thought traces with latent and image feedback under fixed pC/pH execution and routed MMMU settings.
  • Design rationale: Full image feedback is expensive and can introduce synthesis errors, whereas pure visual-latent feedback can disrupt semantic continuity in text reasoning contexts.
  • Accuracy comparison: Replacing aligned visual thoughts with generated latents drops routed MMMU accuracy by 4.44 points, while generated-image feedback drops it by 3.44 points.
  • Runtime comparison: Aligned visual thought reduces per-sample cost by 27.3–30.3% versus latent feedback and 24.4–28.6% versus image feedback.
  • Planner details: The planner produces one score for each path in P = {pA, pU, pR, pC, pH} from a path-aware feature vector before query-form calibration selects the route.
  • Planner training: Planner supervision is multi-label because more than one path can solve the same input, using weighted binary cross-entropy over five path labels.

E.2 Planner Training Transfer

Planner utility transfers to routed accuracy, with lower-utility variants often collapsing onto narrow path preferences. Feature analyses indicate that domain structure is stronger globally than path separation, while query-form buckets reveal more localized path structure.

  • Planner transfer: Lower-utility planners often collapse toward narrow path preferences, such as routing 856/900 examples to pR or 690/900 to pU.These distributions accompany the transfer analysis from held-out planner validation utility to routed MMMU accuracy.
  • Planner transfer: The final planner, pl1, preserves a broader routing pattern than lower-utility planner variants.
  • Feature-space analysis: Global planner features cluster more strongly by dataset or domain than by oracle-correct path labels.For multi-path examples, one correct path label was sampled for visualization, leaving labels heavily mixed in the global embedding.
  • Feature-space analysis: Within representative query-form buckets, planner-selected successful paths are more homogeneous and expose clearer local structure than the global path view.The local views use examples correctly solved by the routed model and color each point by a planner-selected successful path.
  • Execution audit: The executor usually follows requested role sections and produces parsable final answers, shifting the larger remaining bottleneck toward selecting the right path.Format checks cover structures such as Understanding, Reasoning, Visual, Hypothesis, and Answer.

F.3 Different Backbone Experiments

Experiments with Harmon-1.5B indicate that complementary coordination paths persist beyond the main BAGEL backbone, although deployable routing captures only part of the oracle potential. The appendix also details the staged executor setup used to train path-conditioned behavior.

  • Backbone transfer: 67.78% MMMU accuracy from the five-path oracle exceeds raw Harmon by 33.45 points, revealing substantial complementary behavior with Harmon-1.5B.Single-path performance remains close to raw Harmon performance, while the oracle selects correctly whenever any path-conditioned execution is correct.
  • Backbone transfer: The routed Harmon system improves over raw Harmon on four of five understanding datasets, but gains are much smaller than the oracle gap.This indicates that path usefulness transfers across backbones more readily than reliable path selection.
  • Generation evaluation: Generation appendix tables expand the main-text results with category-level GenEval and WISE scores.These categories allow the overall generation numbers to be traced to their underlying subcategories.
  • Executor training: The executor is trained through staged LoRA updates over four role-aligned data splits, combining text, visual-alignment, and image-latent objectives when enabled.Stages progress from text-only understanding through aligned visual thoughts and final image-generation objectives, passing the best checkpoint forward.

G.2 Executor Training Diagnostics

Executor diagnostics distinguish surface format compliance from meaningful path execution. Staged LoRA is selected because it best balances stage-level learning, format preservation, checkpoint size, and downstream path behavior.

  • Training strategy: Staged LoRA provides the best balance between stage-level learning, format preservation, checkpoint size, and downstream path behavior.It is therefore used for the executor in the main experiments.
  • Training diagnostics: Raw BAGEL often preserves requested output structure, but low answer accuracy and weak intermediate or final losses indicate limited meaningful path execution.Format accuracy alone is therefore insufficient evidence that the intended reasoning or visual-grounding operations are being executed.
  • Training diagnostics: Partial-SFT and multitasking can maintain perfect formatting while improving task accuracy only weakly.The comparison separates superficial template following from useful path execution.
  • Path execution: Execution-time wrappers specify role order and output format after the planner selects a path, rather than replacing the learned path selector.Examples include Understanding -> Reasoning -> Answer and Understanding -> Reasoning -> Visual -> Reasoning -> Answer.

I Additional Qualitative Examples

Qualitative examples show that different questions benefit from different coordination paths, from direct perception to textual reasoning, visual construction, or hypothesis exploration. The examples also expose the remaining gap between oracle path complementarity and reliable planner selection.

  • Qualitative path examples: In a potato-root question, direct answering succeeds because the query already states the decisive visual information and further decomposition risks distractor errors.Only pA is correct among the listed predictions for this example.
  • Limitations and broader impact: The current limitation is path selection: the deployable planner remains far behind oracle path selection, leaving substantial potential in the path space unused.Closing this gap requires more accurate instance-level selection and greater robustness across domains, but path-outcome supervision is costly.
  • Limitations and broader impact: Query-form buckets calibrated on auxiliary data implicitly assume reasonable alignment between calibration and target test distributions.Performance may vary across domains, languages, or user groups whose inputs differ from that calibration distribution.
  • Limitations and broader impact: The system may support more selective computation and failure analysis, but incorrect path selection can produce confident-looking outputs with inappropriate reasoning processes.The broader-impact discussion also notes that multimodal generation can support deceptive uses without safeguards.
Loading 2605.11400v1…