Self-Evolving Multimedia Verification through Memory Consolidation of Contestation Experiences

1 Introduction

• SEMV addresses multimedia verification beyond authenticity alone, including whether accompanying narratives correctly describe what happened, where and when, and who was involved.COSMOS studies image-caption consistency, while the Multimedia Verification Grand Challenges call for evidence-rich investigation.Source: §1 · p. 1
• SEMV represents evidence-grounded support and attack arguments with subclaims, provenance, verification status, and strengths, then uses A-QBAF to govern decisions, uncertainty, contestation, and memory updates.The argumentative state is used directly rather than serving only as a post-hoc explanation.Source: §1 · p. 2
• Human feedback targets the earliest affected stage, recomputes only its downstream dependency cone, and preserves independent results.Candidate experiences are staged and checked for provenance, support, duplication, and conflict before consolidation.Source: §1 · p. 2
• SEMV formalizes cooperative agents, structured case state, typed communication, provenance-constrained arguments, A-QBAF propagation, uncertainty estimation, and deterministic decision aggregation.The framework's evaluation spans predictive performance, significance, report quality, calibration, contestability, memory-transfer safety, and computational cost across four VLM backbones.Source: §1 · p. 2
• SEMV develops verification-gated self-evolution with staged experience checks, multi-case consolidation, conflict retention, frozen evaluation snapshots, and measures of beneficial and negative transfer.It also introduces adaptive causal contestation with localized recomputation, preserved traces, and intervention-locality measures.Source: §1 · p. 2

2.1 Multimedia Verification

• Multimedia verification has progressed from image-text consistency toward evidence-grounded investigation, but the preliminary A-QBAF framework does not address selective correction propagation or safe reuse of verified experience.Prior directions include multimodal feature fusion, external-evidence retrieval, and LLM/VLM contextual reasoning; grand challenges add structured reporting and case-level investigation.Source: §2.1 · p. 3

2.2 From Explainability to Contestability

• SEMV routes contestation to the earliest pipeline stage that can repair it—such as claim decomposition, retrieval, or validation—instead of contesting only the argument graph.The table compares explicitly reported mechanisms, distinguishing them from partial or implicit support and capabilities not documented.Source: Table 1 · p. 4, §2.2 · p. 5

2.3 Self-Evolution through Memory Consolidation

• Earlier memory-based agents store episodic lessons or abstract cross-episode insights, while repeated rewriting and poor applicability motivate tracking both transfer benefits and harms.Long-term memory should preserve provenance, scope, conflicts, and reuse outcomes.Source: §2.3 · p. 5
• SEMV defines self-evolution as a closed loop that verifies and consolidates contestation- and outcome-derived candidate knowledge for future decisions while keeping model parameters and agent structure fixed.Its lifecycle captures experience, reflects on outcomes, verifies and stages candidates, consolidates across cases, promotes supported knowledge, and records later reuse.Source: §2.3 · pp. 5–7
• Memory consolidation filters weakly grounded records, merges duplicates, aggregates independent support, preserves unresolved conflicts, abstracts recurring patterns, and tracks effects on later decisions.Figure 1 contrasts potentially noisy, redundant, unstructured, or conflicting accumulated interactions with structured reusable knowledge and failure lessons.Source: §2.3 · p. 5, Figure 1 · p. 6

3.1 Problem Definition

• A verification output comprises a predicted label distribution and label, confidence, justification, execution trace, and case-level uncertainty and escalation codes.The input includes media and a main textual claim, with optional contextual metadata and supplied evidence.Source: §3.1 · p. 7
• An argument is epistemically admissible only if it cites resolvable current-case evidence with provenance and passes grounding verification.Memory may guide planning and argument construction but cannot replace current-case evidence; failed arguments remain trace-visible and marked inadmissible as verified evidence.Source: §3.1 · p. 8

3.2 Multi-Agent Transition System

• The multi-agent transition system uses local agent observations and actions, a dependency graph, state transitions, and invariants to coordinate actions over shared case state.Independent extraction, retrieval, and claim-level reasoning may run concurrently, but results are committed sequentially after validation.Source: §3.2 · pp. 8–9
• Committed states must preserve unique identifiers, evidence provenance, argument admissibility and weighting, valid argument-graph edges, append-only traces, hidden-supervision restrictions, frozen evaluation memory, and dataset-compatible labels.An agent action is rejected if it does not preserve the system-wide constraints.Source: §3.2 · p. 9

4.1 Module A: Input Canonicalization

• Module A canonicalizes benchmark-specific inputs into a standardized bundle of text, media, source groupings, available context, claims, and run settings.The adapter validates paths and schemas, maps internal decisions to benchmark labels, and keeps evaluation-only annotations inaccessible until the first-pass report is committed.Source: §4.1 · p. 11

4.2 Module B: Multimodal Media Evidence Extraction

• Module B extracts localized, provenance-bearing observations using metadata, OCR, ASR, visual, forensic, and reverse-image tools, representing tool failures as uncertainty rather than evidence of manipulation.Video processing samples scene midpoints, retaining at most Kf = 8 representative frames by default, and transcribes audio with timestamps.Source: §4.2 · p. 11

4.3 Module C: Claim, Memory, and External Evidence Retrieval

• Module C decomposes broad claims into scoped subclaims, retrieves external evidence, and uses active memory to guide queries without adding it to current-case evidence.Evidence is normalized into provenance-linked records, ranked per subclaim, and the ten highest-ranked items are passed to argument agents by default.Source: §4.3 · pp. 11–12

4.4 Module D: Argumentation and Reasoning

• SEMV’s A-QBAF combines evidence-grounded support and attack arguments to score subclaims and apply priority-ordered rules for case-level decisions.Arguments that fail verification remain traceable but receive reduced influence; uncertainty checks flag conditions such as missing or conflicting evidence for review.Source: §4.4 · pp. 12, 14, §4.5 · p. 15

4.5 Module E: Human-Contestable Decision Verification

• Human contestation intervenes on the evidence–argument–decision computation, routing changes to the earliest affected stage and recomputing downstream dependencies.The append-only trace preserves the initial report and records the review, revision plan, affected items, and revised output.Source: §4.5 · p. 16

4.6 Module F: Self-Evolution through Memory Consolidation

• Module F transfers verified experience to later cases without updating model parameters, and only after the first-pass or contested report is finalized.Candidates from validation or test cases cannot pass the promotion gate; equivalent observations from independent cases can add support.Source: §4.6 · pp. 17–18
• Only verified candidates participate in promotion; verified contradictions are retained as counter-evidence, and short-term records past retention are archived rather than silently deleted.Verified and under-review candidates are persisted in short-term memory.Source: §4.6 · p. 18
• Every 25 learning cases, compatible observations are grouped by memory type, task, claim, applicability scope, and semantic relation; promotion thresholds vary by memory type.Episodic observations may be promoted on a single case, making confidence the remaining safeguard; failure lessons and semantic rules require two and three independent cases and sources, respectively. Equivalent or mutually entailing observations are merged.Source: §4.6 · p. 18
• A conflict ratio above 0.30 moves an active record under review; deprecation requires both observed contradiction and confidence below 0.45.Source: §4.6 · p. 18
• Only active long-term records are retrievable, and retrieval, planning citation, argument use, success, and contestation are logged separately; evaluation uses a frozen, bit-identical memory snapshot.Evaluation usage events are written outside the snapshot, preventing committed transitions from altering it.Source: §4.6 · p. 19

5.1 Datasets and Evaluation Protocols

• COSMOS evaluation uses all 1,700 test images, with validation for configuration and a fixed 1,200-case training subset for memory learning.Perceptual image hashes and normalized caption fingerprints are checked to reduce near-duplicate leakage.Source: §5.1 · p. 19
• Full-scale COSMOS conditions use one decoding seed; principal conditions also run with three seeds on a stratified 500-case subset to estimate variation.The authors report that repeating all full-scale conditions exceeded the compute budget.Source: §5.1 · p. 19
• MV2026 has 50 training, 10 validation, and 10 live challenge cases; 10 training cases are held out for primary validation because official validation labels are unavailable.The dataset evaluates multimedia verification report quality.Source: §5.1 · p. 19, Table 3 · p. 20
• CTR uses 10 MV2026 validation cases to test contestation selection, error localization, revision, routing, locality, and recovery of final decisions and reports.Three annotators define and adjudicate target states; 40 corrupted episodes span five error families, alongside uncorrupted controls.Source: §5.1 · p. 20

5.2 Consolidated Memory Construction

• The frozen memory snapshot is built from 1,200 class-balanced, source-stratified COSMOS training cases processed before gold labels are revealed, with no validation or test candidates.The active snapshot contains 29 episodic, 12 failure, and 7 semantic records, plus 8 conflict records retained under review.Source: §5.2 · pp. 20–21

5.3 Agent Backbones and Baselines

• Four end-to-end SEMV runs substitute a shared multimodal backbone across agent roles and the VLM analyzer, with matched inputs, visual settings, thinking budgets, and tool-call limits.Published baseline scores are compared numerically only within the full-test protocol block because some prior work used a 400-record holdout.Source: §5.3 · p. 21

6.1.1 Predictive Evaluation

• The COSMOS evaluation reports accuracy, precision, recall, and F1, with OOC as the positive class, and uses Wilson 95% accuracy intervals and unpaired two-proportion z-tests against Tran et al..The baseline test is unpaired because per-record baseline predictions are unavailable.Source: §6.1.1 · p. 22
• 91.88% accuracy and 93.00% precision make Qwen3.6 SEMV’s strongest COSMOS accuracy profile, while Nemotron3 leads recall (92.47%) and F1 (91.82%).All four backbones match or exceed Tran et al. in accuracy; three improve F1. Figure 4 compares the backbones with the baseline on the full test set.Source: §6.1.1 · p. 22, Figure 4 · p. 22, Table 4 · p. 23

6.1.2 Confidence Calibration

• Confidence is evaluated separately from predictive accuracy because it measures how closely predicted confidence corresponds to empirical correctness.The section uses ECE and Brier Score to assess calibration and probabilistic prediction error.Source: §6.1.2 · p. 23, Metrics · pp. 23–24

Metrics

• Qwen3.6 has the lowest Brier score (0.053), while Nemotron3 has the lowest ECE (0.045), showing that probability accuracy and calibration distinguish backbones.Gemma4 has the highest ECE (0.085) and weakest Brier score (0.091), despite a competitive F1 of 89.30%.Source: Metrics · p. 24

6.2.1 Official MV2026 Judge Evaluation

• The official MV2026 assessment scores reports across summary and classification, verified evidence, forensic analysis, cross-checking, and clarity and structure.The challenge evaluation is the paper’s primary real-world measure of report performance.Source: §6.2.1 · pp. 24–25
• 714.95 is the official aggregate judge score for the submitted A-QBAF system, compared with 544.11 for Team matchalatte, 461.37 for Team dashlab, and 441.54 for Deep Verify.The system obtained the highest reported score on each of ten cases, ranging from 59.49 to 83.00.Source: §6.2.1 · p. 24, Table 5 · p. 25

6.2.2 Held-Out Validation Evaluation

• On ten held-out MV2026 validation cases, Qwen3.6 correctly verifies 9/10 with no abstentions and leads Evidence-URL F1 (83.5%), temporal grounding (9/10), and required-field coverage (97.1%).The evaluation compares predictive performance, report quality, and efficiency across backbones.Source: §6.2.2 · pp. 25–26
• Qwen3.6 has the lowest median wall-clock time at 4.8 minutes per case, while InternVL3.5 takes the longest at 9.1 minutes.Nemotron3 generates 12.4k output tokens and uses 32 LLM calls; Gemma4 generates the fewest tokens (10.6k).Source: §6.2.2 · p. 26

6.3.1 Metrics

• Consolidation metrics track candidate verification and promotion, merging, conflict retention, and rejection; reuse metrics assess relevance and successful use, while transfer metrics compare matched memory-on/off runs.Positive transfer counts cases corrected by memory; negative transfer counts cases that were correct without memory but become incorrect with it.Source: §6.3.1 · p. 27

6.3.2 Results

• Of 184 reflected candidate records, 61.4% pass grounding, provenance, and consistency checks; 42.5% of grounded candidates are promoted, 18.6% merged, 7.1% retained as conflicts, and 31.8% rejected.Consolidation filters candidates rather than directly accumulating all reflected experiences.Source: §6.3.2 · pp. 27–28, Table 7 · p. 28
• Unrestricted memory updates yield 46.4% positive transfer but 5.7% negative transfer, whereas verified consolidation and conflict retention reduce harmful transfer without eliminating beneficial reuse.The full configuration maintains 43.0% positive transfer while reaching 0.2% negative transfer.Source: §6.3.2 · p. 28, Table 8 · p. 28
• 91.88% accuracy with the full memory policy coincides with negative transfer of 0.2% and positive transfer of 43.0%.Adding verified consolidation reduces negative transfer from 5.7% to 0.5%; adding conflict retention raises relevance to 93.6% and successful use to 88.5%.Source: §6.3.2 · p. 28, Table 8 · p. 28

6.4 Contestability

• U globally reruns after an edit; C restarts from the earliest affected stage, and C+S limits recomputation to its dependency closure.The comparison replays the same adjudicated feedback across configurations.Source: §6.4 · p. 28, Figure 5 · p. 29, §6.4.1 · p. 29

6.4.1 Quantitative Evaluation

• The 50 matched CTR episodes come from 10 source cases; uncertainty is estimated with a source-case cluster bootstrap.The bootstrap resamples source cases with their associated outcomes in 10,000 replicates.Source: §6.4.1 · pp. 29–30
• Apply rises from 77.5% under U to 95.0% under C, while Trace rises from 72.0% to 98.0%.Localization increases from 61.1% to 72.4% with C, then to 97.6% with C+S.Source: §6.4.1 · pp. 31–32, Figure 6 · p. 32
• 96.7% of initially erroneous decisions are corrected under C+S; it also achieves 68.4% Recomputation Avoided and 52.8% Compute Saved.Median Revision Time falls to 2.27 minutes under C+S.Source: Table 9 · p. 32, §6.4.1 · p. 32

6.4.2 Qualitative Trace Evaluation

• Consolidated memory supplies a reusable rule: without original recording metadata, publication time gives an upper bound, not the event time.The case also shows human challenges redirecting temporal checks and prompting independent verification of candidate coordinates.Source: §6.4.2 · pp. 34–35
• Contestation changes the Rafah case’s reasoning more than its verdict: location is corroborated and event time bounded, while onset, cause, and responsible actor remain unresolved.The revised analysis treats 14:36 on May 10 as the earliest located fire publication, not an established event time.Source: §6.4.2 · pp. 34–35

7.1 Main Findings

• Across the full COSMOS test set, all four SEMV backbones match or exceed the strongest directly comparable baseline in accuracy.The strongest variants also improve F1 and provide useful confidence estimates.Source: §7.1 · p. 36
• 91.88% accuracy and 0.2% negative transfer are reported when verified consolidation retains unresolved conflicts, versus 87.59% accuracy and 5.7% negative transfer with unrestricted updates.The unrestricted update has 46.4% positive transfer; verified consolidation without conflict retention reduces negative transfer to 0.5%.Source: §7.1 · p. 36

7.2 Limitations

• COSMOS is a controlled binary image-caption benchmark; MV2026 has 10 held-out and 10 private-test cases, while CTR has 50 episodes from 10 source cases.CTR replays adjudicated corruptions rather than longitudinal interactions with independent professional fact-checkers.Source: §7.2 · p. 36
• Full COSMOS conditions are single-run evaluations, with three-seed stability measured only on a 500-case subset.Paired experiments use a fixed retrieval cache, and the reported results cover four VLM backbones on one hardware platform.Source: §7.2 · p. 36

8 Conclusion

• SEMV combines provenance-constrained A-QBAF reasoning, causal contestation, scoped revision, and verification-gated memory consolidation.The paper reports competitive verification performance, safer knowledge reuse, more faithful corrections, and lower recomputation across COSMOS, CTR, and MV2026.Source: §8 · p. 37
• Automated multimedia verification is dual-use: false false-context decisions may suppress accurate reporting, while false verified decisions may legitimize misinformation.The system is intended for decision support rather than autonomous moderation; memory-poisoning risk is mitigated, not eliminated.Source: §8 · p. 37
• The paper calls for evaluation on broader multilingual settings, live retrieval, larger human-contestation studies, and stronger attacks on memory and provenance.Performance may vary across languages, regions, and source ecosystems because retrieval coverage is uneven.Source: §8 · p. 37

Data Availability

• COSMOS is available through its official project under a research-use access agreement; the authors cannot redistribute the original data.Source: Data Availability · p. 38
• MV2026 access is administered by the organizers through the official challenge website, and its original multimedia cases are not redistributed with the article.Source: Data Availability · p. 38
• The reviewer-based CTR dataset is available upon reasonable request, excluding the underlying MV2026 multimedia content.Source: Data Availability · p. 38

• The supplement covers notation, component roles, tools, configuration constants, stability evaluation, and sensitivity analysis.These topics are summarized in Tables S1–S6.Source: Supplementary Materials · p. 38
• Figure S1 presents sensitivity to configuration constants on the COSMOS validation sensitivity subset.Source: Supplementary Materials · p. 38
• Table S6 reports one-factor-at-a-time sensitivity results on the COSMOS validation subset.Source: Supplementary Materials · p. 38

A1 Notation

• Table S1 groups SEMV’s principal notation by case and output, evidence and arguments, multi-agent state, A-QBAF reasoning, contestation, and verified memory consolidation.Source: A1 Notation · p. 43, Table S1 · p. 43

A2 Functional Roles

• Table S2 summarizes functional roles through their local observations, principal actions, and primary outputs or state updates.Source: Table S2 · p. 44

A3 Tool Configuration

• The supplement summarizes tool and runtime configurations, with settings held fixed across evaluations and optional components indicated.Source: A3 Tool Configuration · p. 45, Table S3 · p. 45

A5 Stability Evaluation

• Across three COSMOS runs, backbone accuracy standard deviations were 0.20–0.42 percentage points and memory-policy standard deviations were 0.20–0.53 percentage points.The largest memory-policy variation was for unverified append-all and the smallest for full verified consolidation.Source: A5 Stability Evaluation · p. 46, Table S5 · p. 46
• On the 10-case MV2026 held-out set, correct-decision counts varied by at most one case for Gemma4 and Qwen3.6, while Evidence-URL F1 standard deviations ranged from 0.5 to 0.9 points.The authors characterize these findings as descriptive stability evidence, not population-level uncertainty estimates, given the small run and case counts.Source: A5 Stability Evaluation · p. 46, Table S5 · p. 46

A6 Sensitivity to Configuration Constants

• Decision-time perturbations held evidence and the reference-built memory snapshot fixed, whereas memory-policy perturbations rebuilt memory from a fixed 1,200-case subset before evaluation on the same 500-case subset.All constants were frozen before the sweep, and the analysis subset was not used to revise them afterward.Source: A6 Sensitivity to Configuration… · p. 47, Table S6 · p. 47
• 13 exploratory perturbations produced 91.0%–92.0% accuracy, within 0.8 percentage points of the 91.8% validation reference.Figure S1 presents the sensitivity sweep; Table S6 distinguishes conditional decision-time tests from memory reconstruction.Source: A6 Sensitivity to Configuration… · p. 47, Table S6 · p. 47, Figure S1 · p. 48
• The one-factor-at-a-time sweep does not characterize interactions among constants or robustness to larger configuration changes, and point-estimate differences are not treated as evidence of statistical necessity.The reference configuration was selected on a separate development pool and frozen before analysis.Source: A6 Sensitivity to Configuration… · pp. 47–48