Omni-Modal LLMs Know More Than They Say

1 INTRODUCTION

• Omni-LLMs encode sensory inputs through modality-specific pathways and project representations into a shared space for language-model response generation.Source: §1 · pp. 1–2
• Non-textual information can be correctly perceived yet fail to reach answer generation, and the disconnect intensifies as input modalities increase.The paper describes this as a knows-but-cannot-say problem and locates the bottleneck in read-out rather than encoding.Source: §1 · p. 2
• Token-level Modality Alignment (TMA) measures how individual modality representations interact with language generation, complementing global and local alignment measures.The introduction reports that higher-TMA modality tokens show greater downstream-performance sensitivity, but not knowledge-extraction sensitivity, especially as modalities increase.Source: §1 · p. 2
• OMNI-CATALYST self-generates descriptions for weaker modalities, using them as semantic intermediates to steer Omni-LLMs toward answers.The descriptions complement rather than replace modality inputs and place their information in the language-generation space.Source: §1 · p. 2
• Across diverse Omni-LLMs and benchmarks, OMNI-CATALYST improves omni-modal reasoning by 7.29% and increases TMA and attention during knowledge extraction, with enhanced modality balance.Source: §1 · p. 2

Omni-LLMs

• Prior omni-modal work spans unified architectures, training, and inference-time strategies, while this paper focuses on why encoded non-textual information can remain underused during generation.Source: §2 · p. 3

Knowing without saying

• Studies across language and vision-language models report information that is internally decodable or attended to but absent from predictions, motivating the question of whether Omni-LLMs show the same gap.Source: §2 · p. 3

Modality imbalance

• Representational alignment and behavioral modality use are distinct: reducing representational gaps alone does not necessarily improve downstream performance.Prior work reports visual preference despite audio, image, and text being equally available, and reliance on static visual cues over audio-visual alignment.Source: §2 · p. 3

3.1 MODALITY REPRESENTATIONAL MISALIGNMENT

• TMA addresses limitations of aggregate alignment measures by preserving token-level alignment distributions within and across samples.It compares last-decoder-layer states at modality-specific placeholders with states at generated response-token positions.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 TMA REFLECTING LIMITED ACCESSIBILITY TO RESPONSE GENERATION

• TMA measures how closely input representations align with representations underlying the generated response, rather than whether they contain relevant information.Generation-time attention is related but distinct: it measures how much the decoder attends to an input representation.Source: §3.2 · p. 4
• Text representations receive higher attention and exhibit higher TMA overall than non-text representations; within samples, higher TMA only modestly tracks attention, which remains low for video and audio.The TMA distributions overlap, so attention and TMA capture related but distinct aspects of accessibility.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
• Knowledge performance remains high across masked TMA levels, while answering performance is lower and degrades further when higher-TMA tokens are masked.This contrast suggests information can remain recoverable even when its representations are weakly aligned with generation.Source: §3.2 · p. 5, Figure 4 · p. 5
• Adding input modalities raises both mean and variance in TMA, indicating stronger average alignment alongside greater differences in token accessibility.Thus, new modalities do not uniformly translate into information expressed during generation.Source: §3.2 · pp. 5–6, Figure 4 · p. 5

4.1 SELF-GENERATING MODALITY DESCRIPTIONS

• Self-generated descriptions complement, rather than replace, non-textual representations, providing a more language-compatible pathway for modality information to reach response generation.The training-free two-pass formulation generates descriptions from each non-textual input, then answers using both descriptions and original representations.Source: §4.1 · p. 6
• A single-pass variant generates descriptions and the final response jointly from the same modality inputs and query, avoiding a separate description-generation pass.The two variants differ in whether descriptions are generated separately or jointly with the final response.Source: §4.1 · p. 6

4.2 REFINING MODALITY DESCRIPTIONS

• The refinement framework trains both downstream use of self-generated descriptions and generation of descriptions from raw inputs.When ground-truth descriptions are available, they supervise generation; otherwise, self-generated descriptions can serve as targets for self-training or refinement.Source: §4.2 · pp. 6–7
• Tables 1 and 2 compare OMNI-CATALYST with training-free methods and Self-CoT, including results across audio types.The Table 1 caption specifies comparisons without an external reasoning model or adaptive agentic framework.Source: Table 1 · p. 7, Table 2 · p. 7

5.1 GENERALIZATION ACROSS TASKS

• +5–7 improvements on OmniBench and WorldSense, compared with CoM’s 0–3% gains, show OMNI-CATALYST’s advantage on tasks requiring visual and audio information.It also consistently outperforms competitive specialists and training-free methods across benchmarks.Source: §5.1 · p. 7
• OMNI-CATALYST significantly improves performance across fMRI, IMU, and point-cloud modalities, especially for weaker modalities such as fMRI and IMU.The authors describe text representation of modality-specific information as a general mechanism that does not require external reasoning models or complex inference configurations.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 APPLICABILITY TO DIFFERENT TRAINING METHODS

• +2–3% over the untrained baseline follows internalizing OMNI-CATALYST in model weights; applying it at inference adds another 5.1–6.1% on average across training approaches.Perception-only training gains +2.8%, and the combined variants reach 42.4–43.6%; applying STaR to the perceive-only-trained model adds no further performance improvement.Source: §5.2 · p. 8

6.1 HOW DOES OMNI-CATALYST BRIDGE THE DIAGNOSED MODALITY–GENERATION GAP?

• Reasoning performance correlates positively with modality-preference balance, according to Figure 9.Source: Figure 9 · p. 9
• The knowledge-extraction step shifts TMA and attention toward the weaker modality, and the filtered attention visualization shows clearer response alignment.Figure 7 describes this as enabling the model to “hear” better and showing more balanced “seeing” and “hearing.”Source: Figure 7 · p. 8, §6.1 · p. 8, Figure 8 · p. 9

6.2 DOES BRIDGING THE GAP IMPROVE REASONING AND MODALITY BALANCE?

• Audio-description variants shift modality preference toward balance by +10% and improve accuracy by up to 8% for SA, while the baseline has 35% accuracy.Ground-truth audio descriptions yield a 13% performance gain; disparity-mitigation methods MG and I0T leave performance and preference largely unchanged.Source: §6.2 · p. 9
• Masking original non-textual modalities and retaining only descriptions degrades performance, while Self-CoT yields only +3% performance and +1% preference-balance gains.The authors conclude that OMNI-CATALYST’s gains are not simply due to additional inference.Source: §6.2 · p. 9

7 CONCLUSION

• Non-textual representations are less aligned with response tokens and receive less attention, while self-generated descriptions increase alignment and make input information more available during generation.Audio descriptions yield particularly consistent gains in performance and modality balance across models and benchmarks.Source: §7 · p. 9
• Experiments cover only modalities supported by current Omni-LLMs, although the authors say TMA and OMNI-CATALYST can extend to emerging modalities.Examples named include tactile, visual odometry, and olfactory modalities.Source: §7 · p. 9
• The root causes of modality imbalance remain unresolved; proposed sources include LLM-centric architecture, multimodal pretraining data, and heterogeneous-representation alignment difficulty.Source: §7 · p. 10

A AI USE STATEMENT

• The authors used AI tools for manuscript and code assistance, literature and bibliography checks, proof-writing support, and reconstructing an evaluation dataset; they verified the outputs and take responsibility for the final work.They state that the research ideas, methods, experiments, analyses, and conclusions were developed and verified without AI assistance.Source: A AI USE STATEMENT · p. 19

B ETHICS STATEMENT

• The authors expect no direct negative societal impacts beyond those of the underlying pretrained models; they say their method neither introduces nor fully removes those models’ biases.Source: B ETHICS STATEMENT · p. 19

• The authors report providing model, dataset, evaluation, implementation, prompt, hyperparameter, and training details, alongside code and configuration files for reproducing the main results.For trained variants, they also report training data, objectives, and optimization settings; code and other necessary resources are to be released upon publication.Source: C REPRODUCIBILITY STATEMENT · p. 19

• Single-modality evaluation covers image, video, audio, IMU, fMRI, and point-cloud tasks, using public datasets including MS COCO, MSVD QA, Clotho, Ego4D, NSD, and Objaverse.Source: D IMPLEMENTATION DETAILS · p. 20
• The reconstructed single-modality dataset maps source annotations to abstract categories, selects one correct option and three distractors, and excludes examples without sufficiently distinct distractors.The mapping uses keyword-based scoring over pooled captions or QA text, and the category need not appear verbatim in the source.Source: D IMPLEMENTATION DETAILS · p. 20, Table S1 · p. 20
• Evaluation reports reasoning accuracy, perception and hallucination accuracy, normalized modality-preference entropy, and standard automatic audio-captioning metrics.Higher entropy indicates a more distributed preference across modalities.Source: D IMPLEMENTATION DETAILS · pp. 20–21
• The evaluated Omni-LLMs combine modality-specific encoders with language models; most use Qwen-family backbones, while OneLLM and Gemma use other backbones.The supplementary evaluation adds Qwen3-Omni-30B-A3B-Instruct, OmniVinci, and MiniCPM-o-2 6.Source: D IMPLEMENTATION DETAILS · p. 21, Table S2 · p. 21
• Trained variants use LoRA and AdamW, while STaR-based training runs four outer iterations and uses self-generated validation accuracy for early stopping.The setup includes rank-16 LoRA, α = 32, model-specific learning rates, and training on two A100-SXM-80GB GPUs.Source: D IMPLEMENTATION DETAILS · p. 21
• Training variants use self-generated modality descriptions as supervision, with a binary answer-correctness reward guiding learning and each outer iteration initialized from the original checkpoint.STaR filters successful descriptions, R3V samples and ranks descriptions, and SD-Zero uses reward to guide a self-reviser and distills its token-level distribution.Source: D IMPLEMENTATION DETAILS · pp. 21–22
• Training-free OMNI-CATALYST improves over baseline across prompt variants, with comparable gains across models and limited sensitivity to prompt wording.The default training prompt asks the model to describe what it sees and hears.Source: Table S3 · p. 22, D IMPLEMENTATION DETAILS · p. 22, Figure S2 · p. 23
• Complementing original modalities with descriptions generally outperforms replacing them; query-relevant descriptions perform best on the modality-conditioned benchmark, while oracle audio descriptions yield the largest OmniBench gains.The comparisons hold across both tested description placements.Source: Table S4 · p. 22, D IMPLEMENTATION DETAILS · p. 22, Figure S1 · p. 23

• Text tokens generally have higher TMA and attention than non-text tokens, while higher-TMA non-text tokens tend to receive more generation-time attention, with modest association and low absolute attention.TMA measures alignment with response representations; attention measures allocation during decoding.Source: E VISUALIZATION DETAILS · p. 24
• A positive TMA–attention association follows only under stated representation and projection conditions, so the two measures are not equivalent.Source: E VISUALIZATION DETAILS · p. 24
• As the TMA threshold becomes more selective, retained attention-map regions stay concentrated on prompt-relevant content rather than spreading uniformly across the input.The visualizations project token-level attention to input positions and retain tokens above a within-sample threshold.Source: E VISUALIZATION DETAILS · p. 25
• The highest-TMA token’s modality distribution is more balanced for knowledge evaluation (H = 0.82 bits) than reasoning (H = 0.07 bits) across 120 pooled samples per task.The authors interpret this pattern as consistent with reasoning relying on localized evidence while perceptual knowledge may be distributed across modalities and tokens.Source: E VISUALIZATION DETAILS · p. 26
• The added-modality analysis finds the strongest modality-to-generation alignment effect when the additional modality is semantically relevant, while high TMA for irrelevant input can reflect generic preference rather than grounding.The analysis averages each example’s maximum TMA across available modalities and models.Source: E VISUALIZATION DETAILS · pp. 27–28
• Masking high-TMA tokens causes larger downstream-performance degradation while perceptual knowledge remains comparatively stable, supporting TMA as a measure of generation-pathway accessibility.Source: E VISUALIZATION DETAILS · p. 29
• Response-resampling robustness is assessed with ICC(2,1) for continuous TMA and Fleiss’ κ for stability of low-, mid-, and high-TMA tier assignments.The ICC is described as a two-way random-effects, single-measure, absolute-agreement measure.Source: E VISUALIZATION DETAILS · p. 29

Robustness and representation analyses

• TMA is consistent across independently sampled responses, and audio-description representations lie closer to audio and answer representations than to image representations in Qwen-7B and Gemma-4B.In trained OMNI-CATALYST, the knowledge-extraction stage emphasizes audio evidence and the refined answering stage shows more balanced visual and auditory attention.Source: F ADDITIONAL RESULTS · pp. 29, 33, Table S5 · p. 29, Figure S14 · p. 33

Modality gap

• Existing mitigation methods do not consistently reduce centroid distance, local 1-CKNNA, and 1-TMA together; lower 1-CKNNA is associated with incremental performance gains.Randomly inserted text can reduce centroid distance and 1-TMA despite being semantically unrelated to the response.Source: F ADDITIONAL RESULTS · pp. 29–30, Figure S12 · p. 30, Figure S13 · p. 30

Modality preference

• OMNI-CATALYST generally yields more distributed modality preferences than prior mitigation methods, reflected in higher normalized preference entropy.The reported entropy is normalized so that 1 denotes perfectly balanced preference.Source: F ADDITIONAL RESULTS · p. 30, Table S6 · p. 31

Additional Omni-LLMs

• +11.5 percentage points on average for OmniVinci; OMNI-CATALYST also improves performance on additional models, including the larger Qwen3-Omni-30B-A3B-Instruct.The gain for the larger Qwen model is smaller than for smaller models.Source: F ADDITIONAL RESULTS · p. 30, Table S7 · p. 31

Training ablation and thinker mode

• Training-free OMNI-CATALYST improves trained models, while adding modality-description generation during training provides further gains.The method also improves Gemma in both thinker and non-thinker modes, complementing rather than depending on its thinking mode.Source: F ADDITIONAL RESULTS · pp. 30, 32, Table S8 · p. 31, Table S12 · p. 33

Modality-description configurations

• Single-pass training generally improves inference-time descriptions; across two-turn configurations, audio descriptions give the most consistent gains, while visual or all-modality descriptions can yield smaller or negative gains.The audio-description pattern holds across models trained with audio, visual, or all-modality descriptions.Source: F ADDITIONAL RESULTS · p. 31, Table S9 · p. 32

Transfer and external descriptions

• Perceive + STaR improves audio-captioning performance for Gemma and Qwen over both baseline and perceive-only training; external descriptions raise average performance from 34.2 to 39.8.The external-description comparison concerns average performance; the two-call scheme's final performance does not significantly differ across external description models.Source: F ADDITIONAL RESULTS · pp. 31–32, Table S10 · p. 32, Table S11 · p. 32

Qualitative results

• Successful examples align high-TMA tokens with answer-relevant regions or speech, but beyond-direct-perception questions can spread those tokens onto weakly relevant evidence.These failures illustrate the limitation of treating a single maximally aligned token as sufficient when answers require integrating multiple regions or modalities.Source: F ADDITIONAL RESULTS · pp. 34–35, Figure S16 · p. 35, Figure S17 · p. 36