MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models

• MINED addresses the gap in multimodal evaluation of time-sensitive knowledge, including temporal misalignment and conflicting information.Source: §1 · pp. 1–2
• MINED evaluates temporal awareness across six dimensions and eleven challenging tasks.The dimensions cover cognition, awareness, trustworthiness, understanding, reasoning, and robustness.Source: Figure 1 · p. 1, §1 · p. 2
• 63.07 CEM was Gemini-2.5-Pro’s highest score among 15 evaluated LMMs, while most open-source models showed notable time-sensitive knowledge deficiencies.Source: §1 · p. 2
• Knowledge editing effectively updated missing time-sensitive knowledge in single-edit scenarios.The experiments used LLaVA-v1.5 (7B) and Qwen-VL (7B) knowledge they initially failed to answer.Source: §1 · p. 2

2.1 Large Multimodal Model

• Recent LMMs integrate visual encoders with language models, while systems such as Gemini-2.5-Pro and Kimi-Latest have enhanced reasoning and long-context capabilities.Source: §2.1 · pp. 2–3

2.2 Temporal Reasoning Benchmarks

• Textual temporal benchmarks largely ignore time-sensitive knowledge, and multimodal efforts overlook its critical influence; MINED provides fine-grained evaluation of LMMs’ time-sensitive knowledge understanding.Source: §2.2 · p. 3

3 Benchmark Construction

• Two professional annotators collected original data from Wikipedia across country, sport, company, university, organization, and competition domains.Each time-sensitive fact is represented as a quadruple (S, H, P, A): subject, hypernym, property, and a time-varying list of attribute values.Source: §3 · p. 3

3.1 Cognition of Time-Sensitive Knowledge

• Cognition tasks probe models’ parameterized knowledge using an entity image and property, with time-agnostic, interval-aware, and timestamp-aware prompts.Time-agnostic prompts request the latest answer without a clear time node; the other formats specify a period or date.Source: §3.1 · p. 3

3.2 Awareness of Temporal Misalignment

• Awareness tasks test whether models handle conflicts between query timestamps and external contexts describing current or outdated attributes.Future-misaligned contexts pair a past query time with current information; past-misaligned contexts pair a current query with outdated information.Source: §3.2 · p. 3

3.3 Trustworthiness of Unanswerable Date

• Trustworthiness tasks test whether models recognize timestamps outside the valid period for a knowledge attribute and treat those queries as unanswerable.The benchmark constructs past and future unanswerable dates by moving earlier than the earliest record or later than the latest record.Source: §3.3 · pp. 3–4

3.4 Understanding of Temporal Concept

• Understanding tasks evaluate temporal concepts expressed in different formats, including implicit periods inferred from historical events.For example, “when Jeff Bezos served as CEO of Amazon” is mapped to July 5, 1994, through July 5, 2021.Source: §3.4 · p. 4

3.5 Temporal Reasoning

• Temporal reasoning uses chronological ranking and date calculations to assess temporal logic and numerical precision.The calculation task gives a date and the interval between two events, requiring the model to infer the target event’s date.Source: §3.5 · p. 4

3.6 Robustness of Time-Sensitive Knowledge

• Robustness tasks test whether models can self-correct wrong answers after being told their original answer is incorrect.The adversarial temporal error task uses samples all LMMs answered incorrectly across three cognitive subtasks, then presents a rephrased question.Source: §3.6 · p. 4

3.7 Benchmark Analysis

• 4,208 questions span six key dimensions and six fine-grained categories in MINED.The benchmark analysis points to Table 2 and Figure 3 for category distribution and key statistics.Source: §3.7 · p. 4

4.1 Experimental Setup

• The evaluation uses exact-match responses, averages CEM across subtasks, and averages scores across four semantically equivalent prompts.The prompt configurations are “Question”, “Generalization Question”, “Image”, and “Generalization Image”.Source: §4.1 · pp. 4–5

4.2 Analysis of Main Results

• LMMs perform best on Timestamp-Aware tasks, yet Gemini-2.5-Pro still fails to recall approximately 15% of the knowledge.Timestamp-Aware questions specify a point in time, unlike Time-Agnostic or interval-aware questions.Source: §4.2 · p. 5
• Temporally misaligned context impedes correct knowledge recall, though open-source models show greater robustness; smaller open-source models remain vulnerable, with Qwen2-VLI (7B) declining 43.84% on F.M.C and 56.43% on P.M.C.Source: §4.2 · pp. 5–6

4.3 Analysis of Exploratory Results

• Organization, company, and country-leader queries outperform athlete and competition-champion queries; increasing model size generally helps most tasks.The model-size trend has exceptions in R.K, P.U.D, F.U.D, and A.T.E.Source: Figure 5 · p. 6, §4.3 · pp. 6–7, Figure 6 · p. 6

4.2 Analysis of Main Results

• Models perform poorly on implicit temporal concepts, while open-source models lead on ranking and closed-source models on calculation.Gemini-2.5-Pro recalls less than 20% of relevant knowledge on implicit temporal concepts; Qwen2.5-VL ranking CEM falls from 50.3 (3B) to 38.9 (7B) to 11.4 (72B).Source: §4.2 · p. 6
• Models generally reject unanswerable dates, slightly favor future-date refusals, and show limited error correction; Gemini-2.5-Pro corrects fewer than 40% of prior errors.Qwen-VL (7B), LLaVA-Next (7B), and InternVL2.5 (8B) correct none in A.T.E.Source: §4.2 · p. 6

4.3 Analysis of Exploratory Results

• Open-source models produce few latest answers and many irrelevant ones, while closed-source models reduce irrelevant answers but retain many outdated ones.Figure 7 approximates temporal knowledge distribution: closed-source models have broader temporal coverage, while open-source knowledge is concentrated in more recent periods.Source: Table 4 · p. 7, §4.3 · p. 7, Figure 7 · p. 7
• Even with relevant context, models sometimes give irrelevant responses or incorrect attribute values instead of using that context.The error analysis distinguishes context-based, other, and irrelevant answers; red and green values indicate negative and positive effects.Source: Table 5 · p. 7, §4.3 · p. 7

5 Can we update LMMs with time-sensitive knowledge?

• Single editing effectively updates time-sensitive knowledge, and FT-LLM performs strongly across evaluated tasks.SERAC and MEND are comparatively weaker; all methods except SERAC perform excellently on A.T.E, while updating improves complex I.T.C and C.A performance.Source: Table 6 · p. 8, §5 · p. 8
• The study compares parameter-modifying methods FT-LLM, FT-VIS, and MEND with parameter-preserving SERAC and IKE across single and lifelong editing settings.Single editing restores weights after each edit; lifelong editing applies the entire dataset before evaluation.Source: §5 · p. 8
• In lifelong editing, FT-LLM, FT-VIS, and SERAC lose performance on most tasks, while SERAC has only 10.4% loss.SERAC's A.T.E performance improves by 12.6%; the authors suggest its memory-based architecture and explicit caching may mitigate catastrophic forgetting.Source: Table 7 · p. 8, §5 · p. 8

6 Conclusion

• MINED evaluates LMMs' time-sensitive knowledge capability, and the results identify continuing temporal-awareness difficulties despite Gemini-2.5-Pro's strong performance.The conclusion highlights awareness, implicit temporal understanding, and robustness as areas needing improvement.Source: §6 · p. 8
• Knowledge editing effectively updates missing knowledge in single-edit scenarios, while the conclusion calls for better temporal consistency, implicit-concept understanding, and self-correction.The paper frames these as technical hurdles for advancing LMMs toward dynamic knowledge systems.Source: §6 · p. 8

Limitations

• MINED uses static visual data rather than video and covers six representative domains, leaving fields such as law and medicine for future exploration.The stated scope excludes complex temporal dynamics such as video.Source: Limitations · p. 9

Ethical Considerations

• The authors link integrity of time-sensitive multimodal knowledge to preventing misinformation and report that editing outdated information can enhance model reliability.They state that knowledge editing can effectively mitigate identified limitations in current LMMs.Source: Ethical Considerations · p. 9

A The Use of Large Language Models in MINED

• GPT-4o generates temporal-misalignment contexts for MINED’s Awareness dimension.The authors say this use is consistent with current academic research norms.Source: A The Use of Large Language Models in… · p. 11
• Five listed models are used to evaluate MINED, while an LLM polishes grammar in a human-authored draft.The evaluation list comprises Kimi-Latest, Gemini-2.5-Pro, Doubao-1.5-Vision-Pro, Seed-1.6-Vision, and GPT-4.1; the paper says it was not generated entirely by AI.Source: A The Use of Large Language Models in… · p. 11

B.1.1 Original data construction pipeline

• Two annotators screen Wikipedia candidates for visual, time-sensitive entities and record each accepted entity’s subject, hypernym, property, changing attributes, and original image.Candidate domains are country, sport, company, university, organization, and competition.Source: §B.1.1 · p. 11
• The pipeline tests image-based entity perception with 15 LMMs, discards entities 10 models fail to identify, and selects a CLIP-matched generalization image.Google image candidates are compared with the original image by cosine similarity; samples with similarity 1 are excluded before selecting the top-ranked image.Source: §B.1.1 · p. 11

B.1.2 Task data construction pipeline

• Task examples include time-agnostic, interval-aware, and timestamp-aware cognition questions, plus generated future- and past-misaligned contexts and unanswerable dates.Implicit temporal questions substitute uniquely matched historical events for explicit time periods; reasoning tasks ask models to rank events or calculate a target date.Source: §B.1.2 · pp. 11–12
• Reasoning tasks test chronological ranking and date calculation, while the adversarial temporal-error task uses examples all models failed on in Cognition.The adversarial questions ask for a Yes/No response to a claim about the entity’s property over a time interval.Source: §B.1.2 · p. 12

B.1.3 Inter Annotator Agreement Numbers

• The agreement subsection describes expert agreement in terms of observed agreement and chance agreement, and classifies samples passing screening as positive.Samples that do not meet the screening criteria are classified as negative.Source: §B.1.3 · pp. 12–13

B.1.4 MINED ’s Quality

• Original data quality relies on human verification and CLIP image matching; GPT-4o-generated misaligned contexts receive instructions and examples intended to limit distortion and bias.Two annotators compare generated contexts with manually written contexts on 20 sampled examples from each misalignment task, scoring quality from 0 to 10.Source: §B.1.4 · p. 13

B.1.5 MINED ’s Evolvability

• MINED plans quarterly attribute-list updates so its time-sensitive benchmark can evolve while continuing to probe models’ internal knowledge boundaries.The update pipeline retrieves Wikipedia text, extracts club affiliations with GPT-4o under parsing rules, and checks new data against current records.Source: §B.1.5 · p. 13

B.1.6 MINED ’s Quantity

• Table 10 presents the quantity of time-sensitive knowledge for each task.The supplied passage identifies the table’s scope but does not provide its cell values.Source: Table 10 · p. 13

C Experiment Resources about MINED

• Experiments use four A100 GPUs for models up to 38B parameters, four H100 GPUs for larger models, and one H100 for knowledge editing.Each A100 has 40 GiB and each H100 has 96 GiB.Source: C Experiment Resources about MINED · p. 14

D.1.1 F1-Score

• F1-Score conclusions are consistent with the conclusions obtained using CEM.The appendix introduces F1-Score as an additional evaluation metric and reports complete F1 results in Table 11.Source: §D.1.1 · pp. 14–15, Table 11 · p. 14

D.1.2 Cover Exact Match

• The appendix provides complete CEM results in Table 12 for reference.The supplied passages identify the table as a CEM performance comparison but do not provide its cell values.Source: §D.1.2 · p. 15, Table 12 · p. 15

D.1.3 LLM as judge

• LLM-as-judge evaluation improves all LMMs’ scores by capturing nuanced semantic matches, and its results remain consistent with the Section 4.2 observations.The judge model is GPT-4o; the authors present this analysis to address aliases that surface-level CEM and F1 can score poorly despite semantic correctness.Source: §D.1.3 · p. 15

D.2.1 Model Size

• Performance generally scales with model size across most tasks, except R.K, P.U.D, F.U.D, and A.T.E.The analysis uses seven InternVL-2.5 model sizes.Source: §D.2.1 · pp. 15–16

D.2.2 Foundation LLMs

• With identical architecture, LLaVA-Next variants using different foundation LLMs diverge; LLaVA-NextV. (7B) scores 31.2 CEM on A.T.E while LLaVA-NextL. (8B) and LLaVA-NextM. (7B) perform poorly.Source: §D.2.2 · p. 16, Figure 10 · p. 16

D.3 Experimental results based on prompt agreement about MINED

• Prompt variations cause subtle result changes and performance fluctuations, indicating limited model robustness and motivating prompt agreement for more reliable findings.Each sample uses four semantically equivalent prompts, also enabling assessment of how image contexts affect performance.Source: §D.3 · p. 16, Table 14 · p. 17

E.1 Editing Setting

• Single editing evaluates each edit immediately and restores pre-edit weights, whereas lifelong editing applies all dataset edits before comprehensive evaluation.Figure 11 presents the complete workflow.Source: §E.1 · pp. 16–17, Figure 11 · p. 17

E.2 Knowledge Editing Methods and Parameters

• The editing methods include retrieval-based in-context IKE, gradient-based FT, low-rank-gradient MEND, and SERAC’s scope-classifier routing to a counterfactual model or the original model.Source: §E.2 · p. 17, SERAC · p. 18, MEND · p. 18

E.3 Editing Quantity

• Table 15 tabulates editing-sample quantities by task, while Table 16 reports sequential-editing performance for LLaVA-v1.5 (7B).Source: Table 15 · p. 18, Table 16 · p. 18

E.4 Editing Analysis

• In lifelong FT-LLM editing across six tasks, performance on an initial edit steadily declines as the number of subsequent edits increases, consistent with interference in retaining prior and original knowledge.The authors attribute the trend to overfitting and weight disruption from converging on individual samples.Source: §E.4 · p. 18

F Case Studies about MINED

• Figures 12–22 show model outputs for the same query across tasks, and Figures 23–28 provide case studies for six reported observations.Source: F Case Studies about MINED · p. 18

G.1 Chat templates for each task

• Figures 29–39 provide chat templates for each MINED task to clarify the experimental setup.Source: §G.1 · p. 18

G.2 Chat templates for LLM as judge

• Figure 40 supplies the complete chat template for reproducing the LLM-as-judge evaluation.Source: §G.2 · p. 18