PaperlayerView exact revision on arXiv

Source-linked AI summary

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou, Wen Wu, Chao Zhang

arXiv:2610.17856v2cs.LGcs.AI

TL;DR

Existing multimodal time-series benchmarks often pair signals with text or derived visualizations, leaving open whether MLLMs can combine temporal data with real-world visual evidence. MMTA benchmarks that integration across diverse tasks, and TimeOmni-v adds dedicated time-series processing; the evaluation finds shared representation and alignment difficulties while reporting gains in long-video classification and inference efficiency.

  • Problem

    Many benchmarks pair time series with text or visualizations derived from the signal, leaving systematic evaluation of integration with real-world visual evidence open; temporal patterns can also depend on task, domain, and event context.

    Source: §1 · pp. 1–2

  • Method

    MMTA benchmarks tri-modal tasks across nine domains and two layouts, while TimeOmni-v adds a dedicated time-series encoder, temporal alignment, modality-aware token layout, and a forecasting head.

    Source: §1 · p. 2

  • Results

    Evaluated MLLMs share a failure mode from long textualized time series and weak visual-temporal alignment, while TimeOmni-v improves long-video classification and inference efficiency.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MMTA provides an evaluation substrate for studying temporal grounding, modality reliance, efficient time-series representation, and reasoning under missing or weakly aligned evidence.

    Source: §1 · p. 2

    Baselines are zero-shot while TimeOmni-v is fine-tuned per dataset, and transfer across backbones and domains remains untested.

    Source: F Limitations · p. 36

Abstract

from arXiv · show

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.

1 Introduction

MMTA addresses the lack of benchmarks testing whether multimodal language models integrate time series with real-world visual evidence, introducing a tri-modal benchmark across nine domains and two layouts. Its evaluation identifies long textualized time series and weak visual-temporal alignment as shared bottlenecks, motivating TimeOmni-v.

1 Introduction

  • Existing benchmarks do not systematically test integration of visual, textual, and temporal evidence, although real-world decisions can require all three together.Numerically similar temporal patterns can have different meanings depending on task, domain, and event context.Source: §1 · pp. 1–2
  • MMTA spans nine real-world domains and video-plus-time-series or image-plus-time-series layouts, with a 3,524-sample test split covering classification and forecasting.Its tasks vary in which modality is decisive, supporting, or nearly redundant.Source: §1 · pp. 2–3
  • Evaluations of eight representative MLLMs expose a shared failure mode: textualized time series create excessive token sequences and lack a mechanism for aligning temporal samples with co-occurring visual evidence.The paper reports that these issues are especially consequential for long-video settings.Source: §1 · pp. 2–3
  • MMTA standardizes heterogeneous sources with domain-specific prompts, provenance and license documentation, and success-weighted metrics that penalize incorrect answers and failed generations.The benchmark is intended to support study of temporal grounding, modality reliance, efficient time-series representation, and reasoning under missing or weakly aligned evidence.Source: §1 · pp. 2–3
  • TimeOmni-v improves long-video classification and inference efficiency, and supports fine-tuning on long-series domains where textualized time series runs out of memory.The extension adds a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head.Source: §1 · p. 3

2 Related Work

Prior multimodal time-series benchmarks commonly pair temporal signals with text, rendered plots, or images that are reduced to captions. MMTA differs by using raw scene evidence across domains and task types, while TimeOmni-v integrates time series into an omni-modal model as a first-class modality.

Multimodal time-series datasets and benchmarks

  • Earlier benchmarks pair time series with free-form text [2] [3] [4] [5] [9] [10] [11] [12], render the signal itself as an image [13] [14] [15] [16], or use real images that are often caption-reduced and concentrated on recommendation [6].MMTA instead aligns raw scene video or images with text and time series across nine domains and supports classification and forecasting.Source: §2 · p. 3

Time-series-aware language models

  • Time-series-aware text LLMs lack a co-occurring visual stream [17] [18] [19], while omni-modal LLMs lack a time-series encoder and serialize long signals as digit strings [21] [22].TimeOmni-v integrates time series into an omni-modal LLM alongside vision and audio.Source: §2 · p. 3

3 The MMTA Benchmark

MMTA aligns heterogeneous real-world sources into tri-modal tasks across nine domains and two layouts, with substantial variation in modality content and sequence lengths. Its data-quality procedures include human checks of constructed components and mechanical audits of video samples.

3 The MMTA Benchmark

  • MMTA combines native vision, text, and time-series data from openly available sources across nine domains, using video-plus-series and image-plus-series layouts.Its 3,524-sample test split contains 2,924 classification and 600 forecasting samples; domains vary widely in series length, channel count, video duration, and image count.Source: §3 · p. 3

3.1 Video+TS Domains

  • AgiBotWorld robotics instances pair full-task egocentric clips and 10-channel arm and end-effector traces with task instructions to classify anomaly causes.The four labels are paused, meaningless operation, fall, or other.Source: §3.1 · p. 4
  • CoVLA braking segments pair dashcam clips with five synchronized vehicle channels for brake-reason classification, while CUHK-X pairs skeleton-rendered action clips with five six-channel wearable IMU streams across 40 actions.CoVLA labels are rule-derived and partially overlap with the released time-series inputs.Source: §3.1 · p. 4
  • Future Factories combines synchronized assembly-line cameras, 36 PLC channels, and cycle reports for defect classification; HoloAssist pairs 20–80-second egocentric segments and nine HoloLens channels with ordered actions for correctness assessment.Future Factories defects are created by removing components from modular rocket assemblies.Source: §3.1 · p. 4

3.2 Image+TS Domains

  • MIMIC healthcare instances pair chest radiographs, six vital signs, and aligned radiology reports for mortality and discharge-readiness prediction.The two tasks use different clinical anchors and observation windows, including the first 48 hours of data for mortality prediction.Source: §3.2 · p. 4
  • PixelRec forecasts multi-day item interactions from daily counts, cover images, and metadata text, while FinMultiTime forecasts 12-day closing prices using price history, candlestick charts, and pre-horizon news.FinMultiTime's chart and series share closing prices; the news is selected strictly from before the forecast horizon.Source: §3.2 · pp. 4–5
  • Terra forecasts 20 daily values for seven climate variables from historical series, 4–7 remote-sensing and geophysical images, and LLM-generated geographic descriptions.Samples represent global one-degree latitude–longitude grid cells.Source: §3.2 · p. 5

3.3 Data Quality

  • Human checks of 100 samples per constructed component found 96% accuracy for CoVLA labels and 93% for Terra descriptions; mechanical audits covered all 25,057 video-plus-series samples.The audit identified four HoloAssist samples that did not fully cover the target window and 1.42% of released frames outside it.Source: §3.3 · p. 5

4 TimeOmni-v for Multimodal Temporal Analysis

TimeOmni-v integrates time series through an encoder branch with temporal alignment and modality-aware placement, and uses a dedicated head for forecasting while retaining the standard LM head for classification.

4 TimeOmni-v for Multimodal Temporal Analysis

  • TimeOmni-v combines a Chronos-2 time-series encoder, temporal positional structure, modality-aware token placement, and a lightweight forecast head for multimodal temporal tasks.The framework is built on Qwen2.5-Omni and compresses raw time series into a compact token sequence.Source: §4 · p. 5

4.1 Time-series modality integration

  • The Chronos-2 branch converts channel-independent temporal patches into LLM-dimensional tokens, which are aligned and arranged relative to video or still-image streams.For video, time-series tokens are interleaved with frames and aligned to the video timeline; with still images, they form a contiguous block.Source: §4.1 · pp. 5–6, Figure 2 · p. 6

4.2 Forecast head for numerical prediction

  • The forecast head maps LLM hidden states to multi-step numerical predictions in one forward pass; classification uses the standard LM head.The forecast head is used only for forecasting tasks, so the same model supports both formats.Source: §4.2 · p. 6

5 Experimental Setup

The experiments evaluate eight MLLM baselines and TimeOmni-v under shared protocols, using success-weighted metrics that account for both task errors and invalid or failed generations.

5.1 Evaluated Models

  • The evaluation includes eight open-source and proprietary MLLMs, plus TimeOmni-v initialized from Qwen2.5-Omni-7B and Chronos-2 and fine-tuned with LoRA and task-specific components.Official checkpoints or APIs use default inference settings unless stated otherwise.Source: §5.1 · pp. 6–7
  • Controlled comparisons use a fine-tuned TS-as-text baseline, feature-based probes and fusion models, and a port of TimeOmni-v’s integration to InternVL3.5-8B.The TS-as-text baseline shares TimeOmni-v’s backbone, LoRA configuration, and per-dataset budget; the port is tested on four datasets.Source: §5.1 · p. 7

5.2 Tasks and Metrics

  • Both headline metrics incorporate success rate so failed or unparsable generations count alongside ordinary task errors.Context overflow and CUDA out-of-memory errors are recorded as inference failures for open-source baselines.Source: §5.2 · p. 7
  • Classification uses swF1 = Macro-F1 × SuccessRate, combining balanced per-class performance with the fraction of parsable answers.Unparsable outputs are treated as misclassifications.Source: §5.2 · p. 7
  • Forecasting uses swMAPE = MAPE/SuccessRate for multi-step predictions over a fixed horizon, penalizing both large errors and failed generations.The base metric is MAPE on the forecast values.Source: §5.2 · p. 8
  • Appendix D.1 reports parsed-subset base metrics, success rates, and failure causes, while modality ablations and controlled baselines use relative Macro-F1 or MAE changes.Failure categories include CUDA OOM, other inference failures, and unparsable outputs.Source: §5.2 · p. 8

6 Results and Discussion

TimeOmni-v leads MMTA classification, particularly on long-video, long-series domains, while forecasting results remain mid-pack. Its time-series encoder sharply reduces token use and improves efficiency; controlled comparisons show that modality contributions vary by dataset, long-series text baselines can be infeasible, and the integration transfers to another backbone.

6.1 Main Results

  • 0.601 Avg-swF1 versus 0.442 for Gemini-3-flash gives TimeOmni-v the highest classification average; it leads on the long-video AgiBot, CoVLA, and HoloAssist domains, but is mid-pack on Avg-swMAPE (90.8).On the long subset, every baseline stays below 0.6 swF1; TimeOmni-v has the highest swF1 on all three domains. On AgiBot, even 30B-class baselines remain below 0.13 swF1 or run out of memory.Source: §6.1 · p. 8

6.2 Inference Efficiency on the Worst-Case TS Domain

  • 232.8× fewer median time-series tokens on AgiBot (722 versus 167,041) accompanies 23.0× lower latency and 3.1× lower peak VRAM; end-to-end speedups shrink on short-series domains.Token compression is dataset-independent, but speedup depends on the time series’ share of the full prompt; video tokens, system prompt, and decoding overhead are unchanged.Source: §6.2 · p. 9
  • 18 of 131 AgiBot samples (14%) exceed Qwen2.5-Omni-7B’s 140 GB H200 activation budget, while TimeOmni-v completes all 131.The passage attributes this reliability gap to long serialized time series, which are expensive and infeasible on a non-trivial fraction of samples.Source: §6.2 · p. 9

6.3 Ablations and Controlled Comparisons

  • No reduced-modality subset improves on the full model by more than 0.7%, and each loses on at least two datasets where defined; the dominant modality differs across datasets.Dropping text costs nothing on vision-led CoVLA, and dropping time series costs nothing on text-led MIMIC; every subset loses on AgiBot, and all subsets raise forecasting MAE by 19.7–81.4%. Figure 3 reports these relative changes against the full tri-modal model.Source: §6.3 · p. 9, Figure 3 · p. 9
  • On MIMIC, the matched TS-as-text baseline is within −0.4% of TimeOmni-v and leads by +11.0% on CoVLA, but its prompts run out of memory on AgiBot and HoloAssist.On forecasting, the TS-as-text baseline has 7.8–55.0% higher MAE, a gap that also includes the change from digit decoding to the forecast head.Source: §6.3 · pp. 9–10, Figure 4 · p. 10
  • Non-LLM probes and late fusion exceed TimeOmni-v by 7.9–8.9% on AgiBot but trail it by 1.7–19.6% on CoVLA and MIMIC; their forecasting MAE rises 45.1% on PixelRec and over an order of magnitude on SP500 and Terra.The AgiBot result indicates its labels are learnable without an LLM, while zero-shot MLLMs perform far below a linear probe on the same inputs.Source: §6.3 · p. 10, Figure 4 · p. 10
  • On four tested datasets, the InternVL3.5-8B port parses every sample and stays within 12% of the Qwen2.5-Omni-7B version; it leads on CoVLA and trails on AgiBot and MIMIC.Its AgiBot swF1 of 0.746 exceeds every zero-shot baseline by more than 0.6.Source: §6.3 · p. 10, Figure 4 · p. 10

7 Conclusion

This section presents MMTA as a tri-modal benchmark and TimeOmni-v as an approach to integrating time series into multimodal LLMs, alongside appendix descriptions of task constructions across five domains. These tasks pair time series with visual and textual context for classification and forecasting.

7 Conclusion

  • TimeOmni-v raises Avg-swF1 over the strongest baseline and supports long-sequence fine-tuning where its text-series backbone runs out of memory.The evaluation finds that every single- and two-modality variant trails the tri-modal model on part of the benchmark; textualized time series can cause out-of-memory failures and unparsable outputs on long sequences, with low accuracy even when samples are answered.Source: §7 · p. 10

A Real-World Multimodal Temporal Datasets in MMTA

  • MMTA covers nine domains in video+TS and image+TS layouts, with a 3,524-sample test split for classification and forecasting.The appendix describes dataset sources, modality-specific construction pipelines, and representative evaluation prompts under a unified schema.Source: A Real-World Multimodal Temporal… · p. 14

A.1 Robotics: AgiBotWorld

  • AgiBotWorld contributes synchronized robot video, 10-channel state and action trajectories, and task text for manipulation-anomaly analysis.Its MMTA instances use task-frame clips with aligned state traces and instructions.Source: §A.1 · p. 14

A.2 Autonomous Driving: CoVLA

  • CoVLA contributes cropped dashcam braking segments, five synchronized driving signals, and scenario text for three-way brake-reason classification.Brake and gas signals locate the braking interval but are not model inputs.Source: §A.2 · p. 16

A.3 Human Action: CUHK-X

  • CUHK-X pairs skeleton-rendered action clips with five wearable IMU streams for classification across 40 action categories.The MMTA subset aligns the pose-based visual stream and sensor records within each action trial.Source: §A.3 · p. 18

A.4 Manufacturing: Future Factories

  • Future Factories pairs surveillance footage, 36 PLC channels, and cycle reports to classify production cycles with missing-part defects.The task covers four labels, from correct cycles to combinations of missing components.Source: §A.4 · p. 19

A.5 AR Assistance: HoloAssist

  • HoloAssist pairs egocentric action-segment video and ordered fine-grained actions with synchronized gaze, head-pose, and gyroscope streams for correctness judgment.The binary task distinguishes segments whose contained actions are all correct from those with at least one incorrect action.Source: §A.5 · p. 21
  • AR assistance asks models to judge whether a coarse-grained action segment contains an incorrect fine-grained action, using its video and synchronized sensor data.The prompt frames the task as a choice between all fine-grained actions being correct and at least one being incorrect.Source: §A.5 · p. 22

A.6 Healthcare: MIMIC

  • Healthcare evaluates mortality and discharge-readiness prediction by combining vital-sign histories with chest X-rays and aligned radiology reports.Each task has 150 samples; mortality uses the first 48 hours after admission, while discharge readiness predicts discharge within the next 48 hours.Source: §A.6 · p. 23

A.7 Recommendation: PixelRec

  • Recommendation forecasts each item’s next six days of interaction counts from historical daily counts, metadata text, and its cover image.Instances aggregate user–item interactions into daily item-level series, retaining a long consecutive segment and reserving the final days as forecast targets.Source: §A.7 · p. 24

A.8 Finance: FinMultiTime

  • Finance forecasts 12 trading-day closing prices from half-year price histories, aligned candlestick charts, and recent news published before the forecast horizon.The MMTA subset uses assets from the S&P 500 universe.Source: §A.8 · pp. 25–26

A.9 Climate: Terra

  • Climate forecasting predicts the next 20 daily values for all variables using a grid cell’s historical series, aligned imagery, and location text.Each instance uses a 120-day window, with 100 days as history and 20 days as the forecast horizon.Source: §A.9 · p. 27

A.10 Dataset Licenses and Terms of Use

MMTA’s dataset reuse follows each source’s governing access and license terms, while TimeOmni-v code is released separately under a permissive license.

A.10 Dataset Licenses and Terms of Use

  • MMTA reuse follows the most restrictive applicable source term, and MIMIC remains behind PhysioNet credentialed access.Raw data from CoVLA, AgiBotWorld, CUHK-X, and PixelRec are not redistributed; construction scripts operate on users’ own upstream copies.Source: §A.10 · p. 28
  • For sources with share-alike clauses, the corresponding MMTA subset uses the same license.The stated sources are Future Factories and AgiBotWorld.Source: §A.10 · p. 28
  • TimeOmni-v code is released under a permissive license independently of the data.Source: §A.10 · p. 29

B Modality Coverage and Data Quality

This appendix details sample-level text coverage and data-quality checks for MMTA. Sample-level text is variable, sample-specific contextual information rather than fixed task instructions or label definitions.

B Modality Coverage and Data Quality

B.1 Sample-Level Text

  • Text counts as sample-level evidence only when it varies across samples and conveys scene, event, or background information.Seven domains have such text in every sample; CoVLA and CUHK-X contain only fixed task descriptions, variable or sensor definitions, and label spaces, making them video+TS tasks for sample-level evidence.Source: §B.1 · p. 29

B.2 Label Derivation

The section documents how labels and modality alignment are constructed and audited, and explains where source overlap or incomplete time-series cues constrain interpretation. These details distinguish pipeline-derived labels from upstream labels and identify domain-specific boundaries on complementarity.

B.2 Label Derivation

  • Only CoVLA’s brake-reason label is pipeline-generated; the other three relabeled domains retain upstream semantics through deterministic, traceable rules.AgiBotWorld normalizes error causes, Future Factories maps experimental conditions, HoloAssist aggregates action correctness, and CoVLA derives braking labels from interval and signal rules.Source: §B.2 · pp. 29–30

B.3 Human Audit of the Constructed Components

  • The CoVLA and Terra constructed components were audited on 100 random samples each by two annotators, with disagreements adjudicated.CoVLA errors involve fine-grained driving semantics, while Terra errors include contradictory ocean-cell text and negative elevation interpreted as land altitude; identified items are scheduled for correction.Source: §B.3 · pp. 30–31, Table 6 · p. 30

B.4 Cross-Modal Alignment Audit

  • The video+TS audit checked all 25,057 samples; four coverage failures were HoloAssist annotations extending 3–4 frames beyond the source video.The data are aligned at the annotated-segment level, with boundary context shaped by video encoding; image+TS domains are joined using deterministic domain-specific keys.Source: §B.4 · p. 31, Table 7 · p. 31

B.5 CoVLA Labels and the Input Time Series

  • CoVLA labels and released time series overlap only partially, and the TS-only TimeOmni-v variant trails the full model by 16.4% on CoVLA.The released channels omit brakePressed and turn signals, while only lead_a enters the class-A rule; the local label trigger spans two frames but the released series covers 6–600-step braking intervals.Source: §B.5 · p. 31

B.6 Domains with Shared Sources

  • Shared-source modalities can still carry distinct information: FinMultiTime’s chart adds open, high, and low prices, Terra’s text and images add static or visual context, and MIMIC’s vital signs are a complementarity boundary case.On MIMIC, removing vital signs does not lower training-time Macro-F1; its image and report provide distinct information, while the vital-sign ablation is not strong evidence of complementarity.Source: §B.6 · pp. 31–32

C TimeOmni-v Training Details

TimeOmni-v training uses a common recipe across MMTA datasets, varying only the learning rate and epoch count by dataset.

C.1 Initialization and trainable modules

  • Training fine-tunes the Chronos-2 encoder, TS-to-LLM adapter, new TS-token embeddings, and applicable forecast head, while applying rank-8 LoRA to the Omni-LLM.The vision and audio towers remain frozen; the backbones start from Qwen2.5-Omni-7B and Chronos-2.Source: §C.1 · p. 32

C.2 Optimization

  • AdamW optimization uses a 10% linear warmup followed by cosine decay, gradient clipping at 1.0, bf16, gradient checkpointing, and DeepSpeed ZeRO stage 2.Experiments run on NVIDIA H200 140 GB GPUs, with one to eight GPUs per dataset.Source: §C.2 · p. 32

C.3 Per-dataset hyperparameters

  • Only the learning rate and number of epochs vary across datasets; the remaining training settings are shared.Table 8 summarizes the dataset-specific hyperparameters.Source: §C.3 · p. 32, Table 8 · p. 33
  • Learning rates start at 1 × 10−5 and are halved until a pilot run remains finite throughout its first epoch.Forecasting variants tend to use smaller rates because the new forecast head amplifies early gradients into LoRA-side parameters.Source: §C.3 · p. 32
  • Epoch counts are set to keep each dataset above 400 optimizer steps, extending schedules as needed for small training sets.Large datasets use 10 epochs, while some small datasets use 30–100.Source: §C.3 · p. 33

D.1 Decomposed Metrics and Failure Taxonomy

The appendix reports paired comparisons, failure patterns, and test-time modality removal analyses for TimeOmni-v and baseline models.

D.1 Decomposed Metrics and Failure Taxonomy

  • Open-source Qwen models encounter CUDA out-of-memory failures on long-TS domains, InternVL3.5 outputs are often unparsable there, and TimeOmni-v has no test-set failures.Proprietary models rarely fail, but their AgiBot Macro-F1 scores are 0.118 and 0.126; the failure tables decompose metric, success rate, and failure causes.Source: §D.1 · pp. 33–34, Table 11 · p. 34

D.2 Paired Bootstrap Tests

  • TimeOmni-v leads significantly on CoVLA, HoloAssist, and AgiBot; the strongest baseline leads on CUHK-X, PixelRec, and Terra, while other comparisons remain tied or unresolved.Future Factories and MIMIC intervals span zero; SP500 is a tie.Source: Table 12 · p. 34, §D.2 · p. 34

D.3 Inference-Time Modality Removal

  • On MIMIC, removing time series at test time raises swF1 from 0.602 to 0.615; zero-shot Qwen2.5-Omni-7B sometimes scores higher after a modality is removed.The appendix distinguishes reliance of a trained model from whether a task requires a modality.Source: §D.3 · p. 34
  • Removing vision lowers TimeOmni-v Avg-swF1 from 0.601 to 0.349, while removing time series lowers it to 0.406.Reliance varies by domain: CUHK-X relies on vision, Future Factories on time series, and HoloAssist changes little under either removal.Source: §D.3 · p. 34, Figure 5 · p. 35

E Broader Impacts

The broader-impacts discussion covers benefits and risks tied to MMTA and TimeOmni-v’s supported use cases.

Positive impacts

  • MMTA can surface model failure modes in decision-related domains and support apples-to-apples comparisons across model families.The discussion focuses on use cases represented by MMTA’s nine domains.Source: E Broader Impacts · p. 35
  • TimeOmni-v’s time-series compression lowers activation-memory and inference-latency barriers, making the recipe more accessible to academic and clinical groups with limited hardware budgets.Source: E Broader Impacts · p. 35

Negative impacts and mitigations

  • The authors identify surveillance, inherited ICU demographic bias, and unvalidated live-trading signals as risks, with access controls and non-release of sensitive-domain checkpoints among the mitigations.MIMIC-derived prognosis tasks are not validated for clinical decision-making; FinMultiTime results are not production-ready trading signals.Source: E Broader Impacts · p. 35

F Limitations

The limitations concern benchmark modality complementarity and the scope of the evaluation comparisons.

F Limitations

  • MMTA does not guarantee non-redundant evidence from all three modalities: CoVLA and CUHK-X lack sample-level text, while some other domains combine modalities from a shared source.Source: F Limitations · p. 36
  • Per-sample checks that each task needs at least two modalities were not conducted because reliable judgments require domain experts.Human validation covers only the two self-constructed components.Source: F Limitations · p. 36
  • Table 1 compares per-dataset fine-tuned TimeOmni-v against zero-shot baselines, and long-series TS-as-text fine-tuning could not be run on AgiBotWorld or HoloAssist.Forecasting comparisons also differ in output method, while success-weighted metrics combine semantic errors with parsing and hardware failures.Source: F Limitations · p. 36
  • Transfer across backbones and domains remains untested because the integration was ported to one other backbone on four datasets and domains were trained separately.Source: F Limitations · p. 36

Related papers

Loading 2610.17856v2…