EviScene: Beyond "Is It AI?" for Agentic Image Factuality Assessment

1 Introduction

• Without a supplied claim, initial inspection may not clarify the image’s factual situation, and evidence about the same person or place may concern a different activity, year, or stage.The system must refine questions using visual content and newly obtained evidence.Source: §1 · pp. 1–2
• EviLens contains 10,174 supported and refuted images and evaluates both verdict accuracy and evidence sufficiency.The benchmark combines controlled construction and web collection and covers twelve factual error types.Source: §1 · p. 2
• EviScene is a multimodal ReAct agent that interleaves visual inspection, retrieval, and evidence analysis to refine questions and revisit relevant image details.Its final report cites the observations supporting its verdict.Source: §1 · p. 2
• The compact policy is trained with supervised fine-tuning on reviewed teacher trajectories, followed by privileged on-policy self-distillation on selected decisions from its own investigations.The distillation uses verified hint-guided continuations without exposing the hint to the student.Source: §1 · p. 2
• 79.05% balanced accuracy and 52.26% SESR exceed the same-backbone base agent by 6.71 and 9.37 percentage points, respectively.The full SFT+PSD Qwen3.5-9B policy achieved the highest SESR among evaluated agents on the held-out test set.Source: §1 · p. 2

2 Related Work

• Visual authenticity research includes both AI-generated image or video detection and evaluation of whether depicted content fits factual knowledge or commonsense.Related work also addresses factually grounded refinement and reasoning about semantic visual anomalies in generated images.Source: §2 · p. 2
• Contextual verification recovers image source, time, or location, while evidence-based multimodal fact-checking verifies supplied claims; Image Factuality Assessment receives no textual claim.EviScene refines image-grounded questions and returns an evidence-grounded verdict.Source: §2 · p. 3
• Prior multimodal agents use intermediate findings to guide retrieval and visual analysis, while policy learning uses interaction trajectories or on-policy distillation.EviScene applies these ideas to image-centered verification, using new evidence to refine questions and guide visual reinspection.Source: §2 · p. 3

3 EviLens

• EviLens contains 10,174 images—3,114 supported and 7,060 refuted—from controlled construction and web collection.The task evaluates both the factual situation conveyed by an image and justification with external evidence.Source: §3 · p. 3

3.1 Twelve Factual Error Types

• The twelve error types group discrepancies into event and relation, attribute and detail, and context, with each refuted image assigned its primary deciding discrepancy.The type describes the factual error, not the image’s acquisition route, production method, or visual style.Source: §3.1 · pp. 3–4

3.2 Construction and Collection

• Curation plans searches and prepares candidates; controlled construction specifies and checks depicted facts, while web collection recovers evidence for existing images.Constructed refutations alter a selected detail or relation or depict a conflicting occurrence; collected material includes fact-checked images and circulating synthetic images.Source: §3.2 · p. 4, Figure 3 · p. 4
• Automated checks screen clarity, evidence alignment, duplication, and leakage; two domain experts review each candidate, with a third resolving disagreements.Experts also confirm that the intended discrepancy is visible in constructed refuted images.Source: §3.2 · p. 4
• The dataset has 8,647 training and 1,527 test images, with related images from the same source or event grouped when splits are constructed.Evaluation inputs contain only an image and generic verification instruction; private references check verdicts and stated grounds.Source: §3.2 · p. 4

4 EviScene

• EviScene combines image-centered ReAct investigation with full-trajectory SFT followed by PSD on selected decisions from the policy’s own investigations.The agent shares an investigation history across visual inspection and external evidence gathering.Source: §4 · pp. 4–5

4.1 Image-Centered Multimodal Investigation

• Three tool groups cover scene and text inspection, web and image retrieval, and source-page or reference-image checking; observations refine questions and can prompt targeted visual reinspection.The policy repeats investigation until it completes or exhausts its action budget, then uses the image and history to produce a verdict, explanation, and cited observations.Source: §4.1 · pp. 5–6

4.2 Full-Trajectory Supervised Fine-Tuning

• SFT uses 4,872 reviewed teacher trajectories, weighting tool-call and final-report tokens 2, reasoning tokens 1, and instruction and tool-observation tokens 0.Instructions and tool observations remain in context but do not contribute to the loss; images remain conditioning inputs.Source: §4.2 · p. 6

4.3 Privileged On-Policy Self-Distillation

• PSD locates recoverable decision errors in current-policy investigations, adds procedural hints without answers, and accepts only corrected, correct, evidence-supported continuations that pass audit.Selected completions are distilled from the frozen policy’s token probabilities, while successful investigations provide preservation targets; updated policies generate new trajectories for later rounds.Source: §4.3 · pp. 6–7

5.1 Experimental Setup and Evaluation Protocol

• The 1,527-image test compares direct judgment with investigation on five shared backbones, reporting BAcc, macro F1, class-wise recall, and SESR.SESR requires both a correct verdict and sufficient grounds for the decisive image-grounded fact.Source: §5.1 · pp. 7–8

5.5 Qualitative Analysis

• Figure 5 pairs each image and label with GPT-5.5 and EviScene verdicts and a decisive factual comparison, including records, schedules, and construction-stage evidence.The examples include certification and launch records, an exhibition schedule, and distinguishing a museum opening from its documented construction stage.Source: Figure 5 · p. 8, §5.5 · p. 10

5.4 Source and Generator Analysis

• Table 2 reports BAcc/SESR by source and generator; Full reaches 52.75% SESR on constructed images and 50.17% on web-collected images.Full also improves SESR over Base across all three generators, including GPT Image 2 and Seedream 5.0 Lite, which were not used to construct training images.Source: Table 2 · p. 9, §5.4 · p. 9

5.2 Main Results

• 79.05% BAcc and 52.26% SESR make Full the highest-SESR evaluated agent; it exceeds Qwen3.5-397B-A17B by 7.33 BAcc and 8.97 SESR points.Across five shared backbones, EviScene improves SESR over direct judgment by 27.11 to 46.82 points.Source: §5.2 · pp. 8–9

5.3 Training and Tool Analysis

• SFT raises Qwen3.5-9B SESR from 42.89% to 50.49%; removing web search, image retrieval, or evidence inspection lowers both BAcc and SESR.The corresponding SESR drops from Full are 16.37, 12.25, and 6.22 points, respectively.Source: §5.3 · p. 9, Table 3 · p. 9

6 Discussion and Conclusion

• EviScene refines factual questions through visual inspection and external evidence gathering, while its compact policy uses full-trajectory SFT followed by PSD on verified decisions from its own investigations.EviLens combines controlled construction and web collection to evaluate verdict accuracy and evidence sufficiency.Source: §6 · p. 10
• Across five shared backbones, EviScene improves evidence sufficiency over direct judgment; trained Qwen3.5-9B achieves 79.05% BAcc and 52.26% SESR, the highest SESR among evaluated agents.Removing web search, image retrieval, or evidence inspection reduces both metrics.Source: §6 · p. 10

AI use statement

• Generative AI created synthetic images for controlled-construction branches and helped polish manuscript language; authors reviewed the outputs and revisions and take responsibility for the dataset and results.The appendix details EviLens construction, factual annotation, error types, expert review, and composition.Source: AI use statement · p. 10, A EviLens Construction and Annotation · p. 14

A.1 Relation to Existing Benchmarks

• EviLens combines no-claim image verification, external reference evidence, origin-independent labels, and evaluation of both verdict accuracy and evidence sufficiency.“World fact” denotes a verdict about the depicted situation, while “evidence output” requires evidence-bearing justification.Source: §A.1 · p. 14

A.2 Construction and Collection

• Controlled construction renders supported situations or introduces selected factual errors, while web collection recovers source information for existing images and their factual labels.Counterfactual construction changes a detail or relation in a documented situation; fabrication depicts an occurrence checked against records or constraints.Source: §A.2 · p. 14

A.3 Factual Annotation and Evidence

• Annotations describe the relevant subjects, activity or relation, context, and evidence; generated-image review checks rendered content against factual references and refuted explanations identify conflicting relations.Visual references assess appearance and composition, while factual interpretation follows the associated evidence.Source: §A.3 · pp. 14–15

A.4 Error-Type Definitions

• The taxonomy has twelve factual error types, with the primary discrepancy resolving overlap between neighboring categories such as identity substitution and false official record.Numerical alteration concerns a changed value; spatial alteration concerns local arrangement, while geographic displacement changes the scene's location.Source: §A.4 · pp. 15–16, Table 6 · p. 15

A.5 Expert Review

• Two domain experts review each image, description, and retained records; they check visual clarity, target grounding, evidence, and label, with a third expert resolving disagreements.Cases may be accepted, accepted with edits, or rejected; unclear or unresolved examples are rejected.Source: §A.5 · p. 16

A.6 Composition, Splits, and Availability

• The dataset contains 10,174 images across five pathways, all retained images underwent expert review, and the partitions include 8,647 training images and 1,527 test images.Controlled construction supplies most refuted cases, whereas web collection has a more even label split; the twelve intended error types each account for 502 to 524 controlled refuted images.Source: §A.6 · p. 16

• Appendix B records the Qwen3.5-9B policy’s configuration, runtime, and training procedures; prompt templates appear in Appendix E.Source: B Implementation and Training Details · p. 16
• Table 8 reports the error-type distribution of 6,173 refuted images from controlled construction.Source: Table 8 · p. 17

B.1 Model and Training Configuration

• Evaluated model sources follow official releases or model cards, and Table 9 collects implementation and training settings for the Qwen3.5-9B full policy.Source: §B.1 · p. 17, Table 9 · p. 17

Table 9 continued

• PSD starts from the three-epoch SFT checkpoint, uses a frozen Qwen policy and Gemini 3.7 Flash intervention model, and trains for five epochs with LoRA rank 32.The continued settings specify AdamW optimization, BF16 precision, and training on four NVIDIA H20 GPUs.Source: Table 9 continued · p. 18

B.2 Agent Runtime and Tools

• The runtime validates native tool calls under a 24-action budget, retains results for subsequent turns, and sends the accumulated interaction to terminal judgment with citations to successful observations.Search, page reading, OCR, and visual inspection provide retrieval and evidence-checking operations; search candidates are checked before citation.Source: §B.2 · pp. 18–19, Table 10 · p. 19
• The tool ablations disable one of three tool families while keeping the remaining tools and the Full checkpoint, prompt, action budget, and terminal-judgment procedure fixed.Source: §B.2 · p. 18, Table 11 · p. 19

B.3 SFT Data and Supervision

• Qwen3.5-9B is fully fine-tuned on 4,872 reviewed teacher investigations, with weighted supervision emphasizing tool calls and terminal reports while retaining their preceding reasoning.Each retained example includes the action-observation sequence and a reviewed terminal report.Source: §B.3 · p. 19

B.4 PSD Procedure

• In each PSD round, Gemini 3.7 Flash proposes and reviews interventions, while the frozen Qwen policy supplies continuations and token distributions for repair and preservation targets.Repair targets pair a hint-augmented teacher context with the original student context; preservation targets use identical contexts, and both receive the same row weight.Source: §B.4 · p. 19, Table 12 · p. 20

C.1 Inputs and Evaluation Settings

• The test images are separate from teacher trajectories and SFT or PSD training data.Table 13 collects the test-set and evidence-evaluation settings.Source: §C.1 · p. 20, Table 13 · p. 20

C.2 Metrics

• Balanced accuracy and macro F1 give equal weight to both classes, and all metrics are reported as percentages.The class supports are 377 supported and 1,150 refuted images; evaluation permits up to three attempts, with content-based refusals on all attempts counted as incorrect.Source: §C.2 · pp. 20–21
• SESR counts an image only when the verdict is correct and the response provides sufficient grounds for the decisive image-grounded fact.The evaluator requires verdict correctness, factual alignment, and sufficient evidence, while accepting equivalent wording, alternative sources, and different investigation paths.Source: §C.2 · p. 20, §C.3 · p. 21

C.3 Assessment of Complete Responses

• Complete responses are assessed against private reference material, and SESR credit requires all three evaluator criteria to be satisfied.Source: §C.3 · p. 21

D.1 Additional Quantitative Analysis

• Training moves the compact Qwen policy toward stronger verdict accuracy and evidence sufficiency, while investigation raises SESR across systems and affects BAcc differently across backbones.Source: Figure 6 · p. 21, §D.1 · p. 21
• For Qwen3.5-9B, SFT raises supported recall and lowers refuted recall, closing the gap between the two factual labels.Figure 6 separates supported and refuted recall across systems.Source: §D.1 · p. 21, Figure 6 · p. 21
• PSD further improves BAcc and SESR while narrowing the gap between class recalls for the Qwen3.5-9B policy.Figure 7 separates overall decision quality, label recall, and evidence sufficiency across training stages.Source: §D.1 · p. 21, Figure 7 · p. 22

E Prompt Templates

• The prompt appendix maps the implementation labels real and fake to SUPPORTED and REFUTED, respectively.Source: E Prompt Templates · p. 22

E.1 Active Policy Prompts

• The investigation prompt organizes policy notes as Established, Open, and Action, while the runtime appends validated tool-call results to the interaction history.Source: §E.1 · p. 22