Merlin:Empowering Multimodal LLMs with Foresight Minds

• MLLMs struggle to predict object movements and interactions from current or multiple-frame observations, unlike human foresight reasoning.Source: §1 · p. 2
• Trajectory is proposed as a structured learning objective that links temporal contexts between past observations and future behavior.Source: §1 · p. 3
• FPT causally models subject trajectories from initial observations using coordinated multitask pre-training, while FIT uses trajectories to support future-event reasoning.Source: §1 · p. 3
• Merlin supports spatial coordinates or tracklets from single images or multiple frames and reasons inductively about potential future events.Source: §1 · p. 3
• The paper constructs a future-reasoning benchmark to evaluate Merlin’s logical reasoning and future prediction abilities.Source: §1 · p. 3
• Merlin shows strong future reasoning and broader visual understanding across VQA, comprehensive understanding, and hallucination scenarios.Source: §1 · p. 4

• Large language models progressed from early systems such as BERT, GPT-2, and T5 to GPT-3’s 175-billion-parameter model and strong zero-shot performance.Source: §2.1 · p. 4
• MLLMs extend language-model capabilities through cross-modal transfer, mapping multiple modalities into a shared language-semantic space for autoregressive decoding.Source: §2.2 · p. 4
• Merlin’s pipeline combines an image encoder, a large language model, and a modality-align projector for multi-image inputs and trajectory-plus-future-reasoning outputs.Source: Figure 3 · p. 5

3.1 Architecture

• Merlin integrates visual tokens from images and videos into a language sequence using an image encoder, decoder-only LLM, and modality-alignment block.Source: §3.1 · p. 5

3.2 Foresight Pre-Training

• Foresight Pre-Training predicts a concerned subject’s full trajectory across video frames from its first-frame observation.Source: §3.2 · p. 6
• The FPT condition contains consecutive frames, the first-frame subject observation, and the subject trajectory, with Xi denoting frames, Ofirst the initial observation, and Y the trajectory.Source: §3.2 · p. 6
• FPT data represent location, appearance, and action observations as conversational queries whose answers describe subject movement trajectories.Source: §3.2 · pp. 6–7

3.3 Foresight Instruction Tuning

• Foresight Instruction-Tuning adds future observation Z after the trajectory, using frames, the initial observation, and trajectory as conditions for causal future prediction.Source: §3.3 · p. 8
• FIT can infer a likely interaction by presenting trajectories of related players before predicting a resulting event, such as a tackle causing both players to fall.Source: §3.3 · p. 9

Properties Evaluation of Foresight Minds

• Merlin is the first evaluated MLLM reported to perform tracking, achieving performance comparable to expert models and surpassing them on some metrics.This evaluation uses LaSOT and GOT10K, with only a small amount of tracking data used for training.Source: §4.2 · p. 11, Table 2 · p. 11

General Comprehension

• Merlin significantly outperforms comparative methods on MM-Vet and MMBench while preserving or improving general visual perception despite fewer resources than some methods.The comparison includes MMBench development and test sets, and some competing methods use more parameters or inaccessible in-house data.Source: Table 3 · p. 11, §4.3 · p. 12

Object Hallucination

• Merlin achieves the best hallucination-evaluation performance across Random, Popular, and Adversarial scenarios, improving by up to 5 points over Shikra.The evaluation uses zero-shot object hallucination on the COCO validation set; “Yes” denotes the proportion of positive answers.Source: §4.4 · p. 12, Table 4 · p. 12

Ablative Analysis of FPT & FIT

• Both FPT and FIT improve the reported future-reasoning and GOT10K metrics, while combining them produces Merlin’s best performance.Removing image-text pair data during pre-training considerably harms general ability, supporting its role in maintaining modality alignment and combined capabilities.Source: §4.5 · p. 13, Table 5 · p. 13

Ablative Analysis of Model Configuration

• High-resolution inputs and an unfrozen visual encoder benefit visual perception and fine-grained spatial information, while Conv2d compresses tokens for high-resolution multi-image training.More visual tokens do not improve future-reasoning performance, and the authors speculate that they may make supervision sparser.Source: Table 6 · p. 13, §4.6 · p. 14

Visualization Analysis

• Attention visualizations show trajectory-coordinate embeddings attending to corresponding objects across frames, supporting trajectories as an interface between language and dynamic visual contexts.The analysis is used to explain Merlin’s visual capability and lower hallucination relative to previous baselines.Source: §4.7 · p. 14

5 Limitation and Conclusion

• Merlin combines Foresight Pre-Training and Foresight Instruction-Tuning to improve MLLMs’ future reasoning from image observations.The approach uses trajectories as structured learning objectives for dynamic information.Source: §5 · p. 15
• Merlin supports spatial-coordinate or tracklet outputs from single images or multiple frames while retaining performance across traditional vision-language tasks.Source: §5 · p. 15
• The paper reports that Merlin demonstrates powerful future-reasoning capacities alongside visual-comprehension performance.Source: §5 · p. 15
• The authors identify long sequential videos and more comprehensive future-reasoning evaluation as remaining limitations.Source: §5 · p. 15

A Appendix

• The supplementary material provides additional information about Merlin because of the paper’s page limit.Source: A Appendix · p. 15
• Section B explains the constructed dataset, including its data sources and creation process.Source: A Appendix · p. 15
• Sections C and D detail the FPT and FIT data formats and the training approach, while Section E expands the method discussion.Source: A Appendix · p. 15

B Dataset Details

• The authors randomly sample 10M image-text pairs from LAION-400M for Foresight Pre-Training.LAION-400M is described as a CLIP-filtered dataset containing 400 million image-text pairs.Source: B Dataset Details · p. 15
• The training collection includes image-text pairs, detection, tracking, visual reasoning, referring, and dialogue data across FPT and FIT.Table 7 summarizes the training-data categories, including captioning, referring, detection, tracking, reasoning, and dialogue.Source: B Dataset Details · pp. 15, 18, Table 7 · p. 16
• Detection data are built from public datasets by extracting object categories and boxes, filtering tiny objects, and limiting categories per image.The stated category limit is n = 15 when an image contains more categories than the set threshold.Source: B Dataset Details · p. 16
• Tracking data use open-source SOT and MOT datasets and are organized into short clips for causal trajectory modeling in FPT.The listed sources include GOT10K, LaSOT, MOT17, Sompt, DanceTrack, SportsMOT, BDD100K, and MeViS.Source: B Dataset Details · pp. 16–17
• The tracking format represents a query object with its bounding box across successive frames, using structured trajectory outputs.The format is expressed as frame-indexed boxes associated with object categories or identifiers.Source: B Dataset Details · pp. 17–18
• GPT-4 generates 30K trajectory-and-action question-answer pairs for T-CoT conversation data.The dialogues use trajectories and associated descriptions or actions to formulate questions and answers containing trajectory coordinates.Source: B Dataset Details · p. 18

C Data Format Details

• Figure 6 presents an exhaustive visualization of data formats used in both pretraining and instruction tuning.Source: C Data Format Details · p. 18
• Negative samples are included in question-answer pairs to train the model to produce negative answers when appropriate.Source: C Data Format Details · p. 18

• The section directs readers to Table 8 for detailed training settings.Source: D Training Details · p. 18
• Table 8 reports Merlin’s training hyperparameters, including settings shared across both training stages.Source: Table 8 · p. 19
• Table 9 compares Merlin with and without precise task descriptions using GOT10K AO and average Future Reasoning scores.Source: Table 9 · p. 19

• Merlin cannot support video sequences exceeding 8 frames, limiting its ability to model extended motion information.The constraint is attributed to using an image encoder rather than a video encoder, which produces too many vision tokens for the LLM.Source: E More Discussions · p. 19
• A more efficient tokenizer for long-range videos is identified as an important direction for future research.Source: E More Discussions · p. 20
• The paper’s future-reasoning benchmark is not yet comprehensive or fully accurate, motivating development of a more robust evaluation benchmark.Source: E More Discussions · p. 20

Effect of Precise Task Description

• Without precise task descriptions, tracking performance drops from 51.4% to 28.4%.The comparison supports precise task descriptions as important for multi-task learning in MLLMs.Source: F More Experiments · p. 20

The effectiveness of trajectory CoT (T-CoT)

• Trajectory CoT substantially improves prediction reasoning beyond FPT alone.The passage states that FPT provides future-reasoning capability, while prediction-reasoning performance remains limited without trajectory CoT.Source: F More Experiments · p. 20

• The supplementary experiments include ablations in Tables 9 and 10 and qualitative analyses in Figures 7, 8, and 9.Source: F More Experiments · p. 20