PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-VeriRec: Failure-Guided Fusion for Verifiable Agentic Multimodal Recommendation

Yufeng Wang

arXiv:2609.31718v1cs.IRcs.AIcs.CLcs.CV

TL;DR

Recommendation tasks can depend on image constraints that text metadata only hints at, and some requests have no valid answer. MM-VeriRec builds a verifiable benchmark and routes diagnosed failures to targeted fusion repairs; across the reported settings, failure-guided fusion improves performance, while the benchmark’s visual attributes remain deliberately simple.

  • Problem

    Recommendation evaluation may miss whether agents use decisive image evidence, detect text-image mismatch, or abstain when hidden visual constraints cannot be met.

    Source: §1 · p. 1

  • Method

    MM-VeriRec builds image-backed tasks, verifies recommendations against deterministic visual constraints, and uses diagnosed failure types to guide fusion repairs.

    Source: §1 · pp. 1–2, §3.2 · p. 2

  • Results

    Across movie-poster and product-image settings, adaptive failure-guided fusion improves, while stronger embeddings alone leave residual failures.

    Source: §1 · p. 2, §6 · pp. 6–7

  • Takeaways & Limitations

    The benchmark pairs correctness with actionable failure explanations, supporting diagnosis of which repair fits the observed failure.

    Source: §6 · pp. 6–7

    The benchmark uses simple verifier-friendly visual attributes, so its scores do not establish performance on richer or open-ended multimodal recommendation.

    Source: §6 · p. 7

Abstract

from arXiv · show

Images often carry recommendation constraints that text metadata only hints at. A movie may need to look dark, a product may need a minimal style, and a visually impossible request should be rejected. Agentic multimodal recommenders must reason over text-image evidence, decide when visual evidence is decisive, and abstain when no valid action exists. We introduce MM-VeriRec, a verifiable multimodal recommendation protocol and failure-guided fusion method for hidden visual constraints, image-text mismatch, and impossible-task abstention. MM-VeriRec builds tasks from real movie-poster and product-image datasets, verifies each recommendation with deterministic visual attributes, and converts failures into actionable labels: text-trap following, visual ignorance, and false acceptance. Fusion should not merely concatenate modalities, but should diagnose which modality failed and route to the appropriate repair. Across MM-ML 1M and Amazon Reviews datasets, stronger text and vision embeddings improve retrieval but do not remove these failure modes, whereas failure-guided fusion does. The adaptive attribute gate reads the same tags the verifier checks and its scores are verifier-aligned upper bounds testing whether the taxonomy routes to the correct repair. More informative is transfer under a non-aligned gate: an independently derived leave-one-out CLIP detector still reaches 0.7028 and 0.6111 visual-grounded success, above both a VBPR baseline and plain fusion. The text-versus-visual gap reproduces across two LLM families, and the repair that helps differs by domain. MM-VeriRec is both a benchmark and a practical diagnostic loop for trustworthy agentic multimodal recommendation.

1 Introduction

MM-VeriRec addresses recommendation tasks where decisive constraints are visual, text can mislead, and some requests have no valid answer. It pairs deterministic verification with failure-guided fusion, using diagnosed failures to select repairs; stronger embeddings alone leave residual failures.

1 Introduction

  • Text can plausibly imply unsupported visual properties, while visually impossible requests require abstention; aggregate accuracy blurs these distinct failure cases.The paper distinguishes text-trap following, visual ignorance, and false acceptance as failures that call for different repairs.Source: §1 · p. 1
  • MM-VeriRec combines text and visual evidence, deterministic checks of recommendation constraints, and failure labels that guide fusion decisions.The protocol covers candidate membership, text and rating constraints, consumed items, hidden visual conditions, and abstention when no valid candidate exists.Source: §1 · pp. 1–2
  • Failure-guided fusion routes false acceptance to verification and abstention, text-trap following to contradiction penalties, and visual ignorance to attribute-verified gating.The benchmark indicates which lightweight repair to try rather than only recording that an agent failed.Source: §1 · p. 2
  • Text-only agents miss hidden visual constraints, generic image fusion can hurt when representations are misaligned, and stronger MiniLM/CLIP embeddings still leave residual failures.The paper reports that verification and failure-conditioned repair address failures that stronger embeddings do not remove.Source: §1 · p. 2
  • The contributions include a protocol for hidden visual constraints, image-text mismatch, and impossible-task abstention, plus a taxonomy-to-intervention fusion controller.The controller maps text-trap following, visual ignorance, false acceptance, and policy failures to interventions.Source: §1 · p. 2

2 Related Work

Prior recommender work incorporates visual evidence into ranking, while agentic benchmarks study interactive recommendation and deterministic checks. MM-VeriRec focuses on whether agents satisfy hidden visual constraints, abstain when requests are impossible, and diagnose visual-grounding failures.

Agentic and multimodal recommender evaluation

  • Agentic benchmarks evaluate interactive recommendation or deterministic verification, while MM-VeriRec targets multimodal item evidence and visual grounding or abstention.AgentRecBench evaluates interactive scenarios [9], tau-Rec emphasizes deterministic verification and policy checks [6], and A/B Agent provides a poster-based movie sandbox [13].Source: §2 · p. 2

Multimodal recommender systems

  • Visual recommendation methods combine image and textual features with interaction signals, primarily optimizing ranking quality under observed interactions.VBPR introduced visual factors for implicit-feedback recommendation [2], and later systems used graph propagation and related methods [12] [15].Source: §2 · p. 2

Pretrained embeddings and failure-aware fusion

  • Text and image-text embeddings improve some retrieval baselines, but still leave false acceptance, text-image contradiction, and visual ignorance without failure-conditioned repair.MM-VeriRec compares compact color/style features, CNN and CLIP embeddings, and attribute-verified gates under a shared verifier.Source: §2 · p. 2

3 Benchmark Design

MM-VeriRec constructs image-backed recommendation tasks with visible preferences, hidden visual constraints, and policy requirements, then deterministically checks recommendations and abstentions. It evaluates real movie-poster and product-image datasets using cached visual tags and reports both success and failure explanations.

3.1 Task Format

  • Each task combines user history, a candidate catalog, visible text preferences, hidden image-derived constraints, and policy constraints; impossible tasks require abstention.Hidden attributes include brightness, dominant color, contrast, and visual style density.Source: §3.1 · p. 2

3.2 Verifier

  • The verifier accepts a recommendation only when it meets candidate, consumption, text/rating, and hidden-visual constraints, and accepts abstention only when no valid candidate exists.Other outcomes include policy violations, text-trap following, visual ignorance, false acceptance, hallucinated items, and over-abstention.Source: §3.2 · p. 2, Figure 1 · p. 3

3.3 Datasets

  • The benchmark uses MM-ML-1M movie posters and Amazon Reviews 2023 All_Beauty product images to create image-backed tasks.The datasets pair interactions or reviews with item metadata and images, from which visual tags are derived.Source: §3.3 · p. 3

3.4 Constraint Generation

  • Fixed image thresholds produce cached categorical tags, making verifier checks deterministic; adaptive-gate results using the generator’s own tags are verifier-aligned upper bounds.The non-aligned detector is a leave-one-out k-nearest-neighbor predictor over CLIP features, with measured tag accuracy of 0.72 on MM-ML-1M and 0.82 on Amazon.Source: §3.4 · p. 3

3.5 Evaluation Metrics

  • Tasks are generated from the full catalog, with fixed-size candidate lists and a 0.12 probability of impossible cases; repair rules are fixed before evaluation rather than adapted online.The paper reports visual-grounded success alongside failure counts, policy pass rate, abstention accuracy, image-text mismatch pass rate, and visual evidence use when applicable.Source: §3.4 · p. 4, §3.5 · p. 4

4 Methods

The methods compare text-only, visual-tag, and embedding-based agents, then use diagnosed failure modes to select fusion repairs. The controller is kept simple, with fixed main-experiment parameters, and the repair contribution varies by dataset.

4.1 Agents

  • Text-only, visual-tag, and embedding agents test metadata, poster-derived visual tags, and text-image similarity, including MiniLM, CLIP, ResNet-18, and aligned color/statistics embeddings.Plain fusion combines text and image similarity; the lexical retrieval baseline uses TF-IDF metadata embeddings.Source: §4.1 · p. 4

4.2 Failure-Guided Fusion

  • Visual verification masks candidates below a threshold and abstains if none remain; text-trap suppression penalizes unsupported metadata claims, while adaptive guided fusion adds an attribute-verified gate.The main experiments fix the fusion weight at α=0.5, visual threshold at 0.50, and text-trap penalty at 0.25; the aligned attribute analyzer makes the final step a controlled upper-bound intervention.Source: §4.2 · p. 4

4.3 From Failure Labels to Fusion Decisions

  • The controller maps false acceptance to rejection, text-trap following to contradiction handling, and visual ignorance to a stronger visual gate before ranking.The controller is intentionally simple so failure labels can guide repair choices rather than relying on a monolithic reranker or a fixed fusion weight.Source: §4.3 · p. 4
  • On MM-ML-1M, the text-trap penalty raises success from 209 to 229, then the attribute gate raises it to 323; on Amazon, the gate alone raises success from 135 to 242.The penalty removes all 44 trap failures on MM-ML-1M, while the gate removes the remaining 94 visual-ignorance failures; on Amazon, the penalty is a no-op and the gate removes all 107 visual-ignorance failures.Source: §4.3 · pp. 4–5

5 Experiments

Across movie and product-image tasks, the experiments test whether agents use hidden visual constraints and whether stronger embeddings or failure-guided repairs address the remaining errors. Non-aligned CLIP-detector gates outperform plain fusion on both datasets, while the aligned-gate results are verifier-aligned upper bounds.

5.1 Setup and Metrics

  • The evaluation uses three random seeds, cached image features, and visual-grounded task success as its primary metric, with policy, abstention, mismatch, visual-use, and failure-count metrics as secondary measures.Baselines include TF-IDF, MiniLM, ResNet-18, CLIP, and compact color/statistics features.Source: §5.1 · p. 5

5.2 MM-ML-1M Main Results

  • On MM-ML-1M, visual-tag prompting raises DeepSeek success from 0.0444 to 0.9222, while aligned color-statistics fusion raises retrieval success from 0.3611 to 0.5806.Gemini 2.5 Flash reproduces the text-only to visual-tag gap, moving from 0.0222 to 0.9444; adaptive guided fusion’s perfect score is a verifier-aligned upper bound.Source: §5.2 · p. 5, Table 3 · p. 5

5.3 Transfer and Stronger Embeddings

  • On Amazon, adaptive guided fusion reaches 0.8963 visual-grounded success, but stronger embeddings alone leave structural failures and this aligned-gate score is an upper bound.The gate reads the constraint generator’s tags; with MiniLM text and CLIP image features on MM-ML-1M, plain fusion still has 37 false acceptances and 125 visual-ignorance failures.Source: §5.3 · pp. 5–6, Table 5 · p. 6, Table 6 · p. 6

5.4 Non-Aligned Attribute Detector

  • 0.7028 on MM-ML-1M and 0.6111 on Amazon visual-grounded success with a CLIP detector, versus plain-fusion baselines of 0.5806 and 0.5000, respectively.The detector is independently derived; the oracle-gate results are verifier-aligned upper bounds.Source: §5.4 · p. 6, Table 7 · p. 6

6 Discussion, Limitations, and Conclusion

MM-VeriRec evaluates recommendation failures by type and links each failure to a conditional repair. Its experiments support this diagnostic approach while bounding claims to simple visual attributes and limited recommender comparisons.

6 Discussion, Limitations, and Conclusion

  • Offline image embeddings and color/style summaries can be computed once per item, while online decisions score candidates, apply masks, and choose whether to abstain.A generic ResNet-18 embedding can miss the tested visual condition, whereas compact color/style attributes align with hidden constraints.Source: §6 · pp. 6–7
  • MM-VeriRec can guide pre-deployment engineering by matching dominant false-acceptance, text-trap, or visual-ignorance failures to a verifier, contradiction-aware fusion, or better visual signals.For visual ignorance, the proposed options are a better visual representation or an explicit attribute detector.Source: §6 · p. 7
  • MM-VeriRec is a stress-test rather than a complete recommender benchmark, using simple verifier-friendly attributes and leaving learned modern multimodal recommenders unevaluated as recommendation candidates.The adaptive analyzer is aligned with the benchmark attributes, and the taxonomy's contribution is selecting repairs per dataset rather than adding accuracy on its own.Source: §6 · p. 7
  • Across movie-poster and product-image settings, adaptive failure-guided fusion improves recommendation by mapping failure modes to visual verification, contradiction penalties, and attribute-verified gating.The appropriate repair differs by domain, and stronger embeddings alone do not resolve impossible-task abstention, text-image contradiction, or visual ignorance.Source: §6 · pp. 6–7
  • The paper concludes that multimodal evaluation should explain failures well enough to guide repairs, with a staged agent that escalates uncertain cases proposed as future work.The suggested agent would use the failure-guided controller for clear cases and a larger multimodal model for uncertain ones.Source: §6 · p. 7

Related papers

Loading 2609.31718v1…