Seeking and Updating with Live Visual Knowledge

1 Introduction

• LIVEVQA introduces a large-scale benchmark for seeking and updating live visual knowledge from recent news, YouTube videos, and academic papers.Its instances pair distinctive event images with Level 1 entity-recognition and Level 2 cross-modal, multi-hop reasoning questions.Source: §1 · p. 2
• 17 MLLMs struggle to identify the latest visual knowledge, while multimodal search tools substantially improve performance.Text-based online searching does not improve performance, whereas multimodal retrieval helps address dynamic visual information.Source: §1 · p. 2
• 4.2-point improvement is achieved when short-term visual factuality-oriented QA fine-tuning updates knowledge-intensive capability.The update occurs through parameter-efficient fine-tuning within only one epoch, although visual perception capability is harmed.Source: §1 · p. 2
• The dataset, benchmark, and findings are intended to support future research on improving MLLMs’ live visual knowledge seeking and updating.The experimental materials are publicly available online.Source: §1 · p. 2
• The data engine collects illustrated visual data, applies multi-level filtering, and generates basic and detailed question-answer pairs.The pipeline contains raw data collection and Q&A-pair generation modules.Source: Figure 2 · p. 3

2 LIVEVQA: The dataset

• LIVEVQA contains 107,143 questions and 28,488 unique images collected from content dated April 2024 to early May 2025.The dataset uses a multi-stage LLM/MLLM-in-the-loop pipeline with strict filtering and human validation above a 97% pass rate.Source: §2 · p. 3
• The dataset combines visually distinctive event images with Level 1 recognition and Level 2 multi-hop cross-modal reasoning questions.Both levels are available in open-ended and multiple-choice formats with synthetic reasoning trajectories.Source: §2 · p. 3

2.1 Raw data collection

• News, YouTube, and arXiv sources are filtered for relevant, distinctive, and current visual content before question generation.News images undergo URL, size, event-relevance, and semantic-similarity filtering, while videos and papers receive source-specific preprocessing and image selection.Source: §2.1 · pp. 3–4
• YouTube processing restricts videos to English content, at most 10 minutes, and videos with English subtitles before subtitle-based segmentation and keyframe selection.Keyframes are deduplicated, cropped, and selected for topical relevance, visual informativeness, and representativeness.Source: §2.1 · p. 4

2.2 Visual question answering generation

• Generated questions target substantive entities at Level 1 and seven types of multi-hop contextual reasoning at Level 2, with filtering for answer quality and verifiability.Level 1 questions avoid simple visual attributes, while Level 2 questions connect image details with textual metadata.Source: §2.2 · pp. 5–6
• The collection excludes bioRxiv and medRxiv papers because anti-scraping mechanisms prevented completely synthetic data collection.The authors abandoned those sources after attempting to collect them.Source: §2.2 · p. 5

2.3 Dataset statistics and split

• The evaluation split contains 1,500 unique images and 3,000 questions selected for human-searchability and model-based answerability.Each query was required to be searchable by a human within 10 minutes using web browsing and reverse visual search.Source: §2.3 · p. 6

3 How well do current MLLMs seek out visual knowledge?

• The study frames live visual knowledge seeking as necessary for assistants and evaluates whether models can access current information through external tools.The human-validated test set also supports measuring visual factuality calibration.Source: §3 · p. 6

3.1 Experiment setups

• The evaluation tests diverse state-of-the-art proprietary and open-source MLLMs, including models with native image-search capabilities and text- or GUI-based search frameworks.Responses, confidence scores, correctness, attempted answers, and F-scores are averaged over three evaluations.Source: §3.1 · pp. 6–7, Table 4 · p. 8

3.2 Results and analysis

• Larger models generally improve visual factuality recognition, but proprietary models retain an advantage over open-source counterparts.GPT-4.1-nano reaches 8.6% on Level 2 News questions, compared with 14.8% for GPT-4.1-mini and 16.5% for GPT-4.1.Source: §3.2 · p. 7
• Multimodal search more than doubles GPT-4.1’s average accuracy from 16.5% to 33.4%, with especially large gains on challenging Level 2 questions.GPT-o3’s Level 2 accuracy also rises from 3.0% to 15.8% with tool use.Source: §3.2 · pp. 7–8
• All evaluated models exhibit substantial overconfidence because their stated confidence falls below the ideal y = x calibration line.Calibration is positively related to accuracy, with GPT-4.1 better calibrated than its smaller variants.Source: §3.2 · p. 8, Figure 5 · p. 8

4 Can we update MLLMs with new visual knowledge?

• The central update question is whether MLLMs can efficiently acquire new visual concepts while retaining prior visual understanding and general multimodal reasoning.The paper positions PEFT as an efficient alternative for updating knowledge when retrieval introduces latency or struggles with semantically similar visual inputs.Source: §4 · p. 8

4.1 Experiment setups

• The update experiments compare LoRA and DoRA across ranks from 16 to 64 using Qwen2.5-VL-3B/7B-Instruct.The study also evaluates QA, MCQA, and QAR instruction formats, with a rephrased 1,500-sample MCQA validation set and MMMU and MMstar general benchmarks.Source: §4.1 · pp. 8–9

4.2 Results and analysis

• Training on LIVEVQA with concise multiple-choice answers improves MMMU by 4.2%, while direct multiple-choice training learns faster than the other formats.Performance continues rising into the second epoch, whereas QA and QAR formats saturate early.Source: §4.2 · p. 9, Figure 6 · p. 9
• Visual knowledge updating consistently reduces MMStar performance, with simple-answer training dropping from 65.80% to 58.16%.The decline persists across rank, training-step, and format variations and plateaus with continued training.Source: §4.2 · p. 9
• Higher-rank LoRA configurations outperform lower-rank counterparts, yielding an average 5.4% improvement on the validation subset.The gain is particularly evident when assimilating recent visual entities.Source: §4.2 · p. 9, Figure 7 · p. 9

5 Conclusion

• LIVEVQA provides up-to-date visual content paired with cross-modal, multi-hop reasoning questions for research on MLLM knowledge seeking and updating.The paper presents the dataset, benchmark, and findings as a foundation for future work in dynamic visual contexts.Source: §5 · p. 9

Ackownledgement

• The paper thanks contributors from MM-Search, MM-search-R1, and the University of Washington for their insights and discussion.Source: Ackownledgement · p. 10

A Limitations

• Crawling older news and video data can create date-category imbalance, while arXiv CAPTCHAs can produce incomplete downloads.Source: A Limitations · p. 14
• Despite human validation, occasional QA-generation or filtering errors may leave flawed dataset samples.The validation used over 500 balanced samples per category and reported error rates below 3%.Source: A Limitations · p. 14
• Update experiments were restricted to LoRA, DoRA, and the Qwen2.5-VL family, excluding full fine-tuning and broader model coverage.Source: A Limitations · p. 14

B Related work

• Live visual knowledge covers current news, emerging events, cultural phenomena, and other temporally evolving information for timely visual-query responses.Source: B Related work · p. 14
• Knowledge updating seeks to inject new information while retaining prior capabilities, but catastrophic forgetting motivates high-quality synthetic data and specialized techniques.Source: B Related work · p. 14
• The paper presents an automated engine for collecting visual knowledge and synthesizing multimodal update data, while future work will add baselines and more efficient techniques.Source: B Related work · p. 14, A Limitations · p. 14

C.1 News

• News processing validates and canonicalizes URLs, extracts article text and image candidates, removes duplicates, and retains a small set of relevant visuals.Selection combines domain-sensitive URL checks, HTML extraction, similarity-based deduplication, and area thresholds.Source: §C.1 · pp. 15–16
• News image curation keeps at most four prominent images per article and discards images below a fraction of the largest image’s area.Source: §C.1 · p. 16

C.2 Videos

• YouTube processing searches dated English videos, limits duration to ten minutes, downloads subtitles, converts formats, segments topics, and extracts filtered keyframes.Source: §C.2 · p. 17
• Video keyframes undergo perceptual deduplication, clarity selection, layout-based cropping, and GPT-4.1 relevance filtering before QA synthesis.Source: §C.2 · p. 17

C.3 Academic papers

• Academic-paper processing retrieves arXiv metadata and HTML, removes unusable images, and uses an MLLM plus human validation to select key visuals.Source: §C.3 · pp. 17–18
• Academic QA generation combines template-based Level-1 questions with detailed summaries and Level-2 questions requiring image context, abstracts, or higher-level reasoning.Source: §C.3 · p. 18

• Four authors independently annotate LLM/MLLM outputs using detailed instructions and Streamlit interfaces to reduce bias and assess agreement.Source: §C.4 · p. 18
• GPT-4.1 subtitle parsing and event segmentation achieves a 98% annotated passing rate, while video image selection achieves a 96% passing rate.Source: §C.4 · p. 18
• GPT-4.1 text filtering for video QA generation achieves 99% accuracy, and academic-paper key-image selection achieves 94% accuracy.Source: §C.4 · p. 18
• News-image filtering achieves 96% accuracy by retaining visuals that add essential information beyond text while excluding generic or decorative images.Source: §C.4 · p. 19
• Level-1 and Level-2 QA generation and filtering use GPT-4.1 with annotation tools to enforce meaningful answers, verifiability, and multimodal context.Reported GPT-4.1 accuracies are 93% for Level-1 generation, 97% for Level-1 filtering, 98% for Level-2 generation, and 99% for Level-2 filtering.Source: §C.4 · p. 19
• Detailed answer generation produces answers, event information, and reasoning processes, with GPT-4.1 reaching 99% annotated accuracy.Source: §C.4 · p. 19

• Table 5 documents evaluated model names, creators, versions, access times, licenses, and experimental purposes.Source: D Detailed experimental setups · p. 25, Table 5 · p. 25
• F-score rewards guessing when the probability of correctness exceeds 1/2, because guessing then has a higher expected score than abstaining.The metric uses c for correct answers, i for incorrect answers, and n for non-answered questions.Source: D Detailed experimental setups · p. 25
• Fine-tuning uses Qwen2.5-VL-3B/7B-Instruct with LoRA, a frozen visual encoder, mixed precision, and 8 A800 GPUs.Source: D Detailed experimental setups · p. 25

• 152 images were classified as NSFW, but every classification was judged incorrect.Source: §D.1 · p. 26
• The dataset used an NSFW detector7 for quality validation.Source: §D.1 · p. 26
• A normal photograph of Taylor Swift was incorrectly labeled NSFW as an example of the detector’s unreasonable judgments.Source: §D.1 · p. 26

E Additional experimental results

• Fallbacks improve stability by defaulting failed Stage 1 queries to the original question, skipping CAPTCHA screenshots, and directly querying the model when Stage 3 fails.These strategies are designed to prevent error amplification across the multimodal reasoning chain and improve answer quality in complex visual-language scenarios.Source: E Additional experimental results · p. 26
• Level 2 problems show a significant performance decline versus Level 1, except in the arXiv subset, where abundant visual information can support direct answers.The video-subset table reports that visual searching dramatically enhances performance on Level 2 questions.Source: E Additional experimental results · pp. 26–27, Table 7 · p. 27

F Prompt

• The synthetic-data prompts filter images for current, indispensable social relevance and require captions to use only verifiable article information without inference.The default stance is exclusion, retaining images only when they provide unique visual evidence directly tied to the article’s core event.Source: F Prompt · pp. 27–29
• Level 1 question-generation prompts require image-and-text questions with explicit short answers, six permitted categories, temporal context, and specific identifiers beyond computer vision alone.They prohibit questions about visible text, counting, generic descriptions, obvious public figures, chart data, and images lacking temporal context.Source: F Prompt · pp. 30–31
• The Level-1 QA prompt requires concise image-grounded questions with specific answers, non-revealing wording, distinct topics, and 3–5 multiple-choice options with randomized correct-answer positions.It also requires plausible distractors, multiple valid answer phrasings, English output, and an error response when no suitable question can be generated.Source: F Prompt · pp. 33–34
• The Level-1 filtering prompt removes short news and questions with generic, news-platform, food-only, or otherwise weak answers while retaining specific people, locations, events, products, and organizations.Questions are evaluated using both mandatory keep conditions and discard criteria, with JSON output identifying removals and acceptable items.Source: F Prompt · p. 34
• The Level-2 prompt generates near-impossible multi-hop questions requiring Level-1 answers, at least 4–5 inference steps, precise entities, natural visual references, deceptive options, and exactly one of seven answer categories.It prohibits invented facts and leakage cues while requiring 3–5 questions spanning different categories and privately verified reasoning chains.Source: F Prompt · p. 35
• YouTube preprocessing segments transcripts by topic continuity, adds punctuation without changing words, and selects QA images for relevance, clarity, visual richness, safety, and minimal textual interference.Segmentation minimizes fragments and prioritizes semantic continuity over timestamp gaps, while image scoring penalizes blurry content, prominent answer-revealing text, and visible personal or sensitive identifiers.Source: F Prompt · pp. 36–38
• The visual curator prompt selects up to five high-quality, relevant, nonredundant, and visually diverse images while rejecting textual interference and unsuitable sets.It requires outputting preferred image indices or “No suitable images found.”Source: F Prompt · p. 40
• The academic QA prompt generates 1–2 image-grounded multiple-choice questions whose answers derive from supplied paper context, with five options, a correct letter, paraphrases, and reasoning.Questions must avoid describing the image or leaking article content, while distractors remain plausible but text-refutable.Source: F Prompt · pp. 41–42
• The SimpleQA grader labels predictions CORRECT, INCORRECT, or NOT_ATTEMPTED based on semantic completeness, contradiction, and whether the answer includes the gold information.The prompt permits harmless omissions or typos when the requested information remains clear, but treats unsupported contradictions as incorrect.Source: F Prompt · pp. 42–44
• The calibration prompt requests an answer and a confidence score in a JSON object, and Figure 20 provides an example of the dataset.The prescribed confidence range is 0% to 100%.Source: F Prompt · p. 44, Figure 20 · p. 45

G Case study

• The case study covers event, location, time, person, object, organization, count, reason, title, author, and data-oriented questions with concrete ground-truth answers.Source: G Case study · pp. 45, 47, 48, 49, 50, 51
• Multimodal retrieval enables GPT-4o to answer a knowledge-intensive visual question correctly, whereas it fails without retrieval.Source: Figure 21 · p. 46
• Examples include identifying a paper title, its lead researcher, and the number of proposed HMFNet components.Source: G Case study · pp. 55–57

NeurIPS Paper Checklist

• The checklist reports that the abstract and introduction accurately reflect the paper’s contributions and scope, and that the paper discusses its limitations.Source: NeurIPS Paper Checklist · p. 59

• The authors state that they fully disclose experimental details and provide source code and datasets to support reproducibility.Source: §2 · pp. 60–61
• The paper reports that experiments use APIs or 8 × NVIDIA A800 (80G) GPUs and that training and test details are disclosed.Source: §2 · p. 62
• Error bars are not reported because they would be too computationally expensive, particularly for some advanced reasoning models.Source: §2 · p. 62
• The authors state that they discuss broader societal impact, filter data using advanced models and humans, and release the dataset under a CC-4.0-NC-BY license.Source: §2 · p. 63

Asset documentation

• The paper states that newly introduced assets are documented in the supplementary materials.The checklist also emphasizes providing licensing, copyright, and terms-of-use information for released or repackaged assets.Source: §2 · p. 64

Crowdsourcing and Research with Human Subjects

• For crowdsourcing and human evaluations, the paper reports providing participant instructions, screenshots, and human agreement or pass rates.The supplied passage does not state compensation details beyond the checklist question.Source: §2 · pp. 64–65

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

• The paper marks IRB approval as not applicable because the research does not require IRB review.The checklist states that IRB requirements depend on the country and institution for human-subjects research.Source: §2 · p. 65