Source-linked AI summary
CVE-SAI: Counterfactual Visual Evidence-Guided Selective Attribute Indexing for Risk-Controlled E-commerce Search
Xiaolong Sun, Qichao Wang, Hangyu Li, Liang Chen
TL;DR
Existing multimodal attribute systems do not jointly verify visual support and control whether predictions enter persistent indexes. CVE-SAI freezes an ontology-constrained visual candidate, audits its evidence, and uses independent calibration for admission; across five ABO attributes, it improves inference, certified coverage, and controlled retrieval while reducing unsafe exposure.
Problem
Existing methods do not jointly verify support from the current product image and control whether an attribute prediction enters a persistent index.
Method
CVE-SAI freezes an ontology-constrained candidate from the primary image before evidence audits, then uses calibrated admission under a 5% unsafe-admission budget.
Results
Across five ABO attributes, CVE-SAI achieves the highest certified admission coverage and strongest controlled retrieval performance with the lowest unsafe exposure among automatic-admission systems.
Takeaways & Limitations
Separating inference from admission supports visually supported attribute completion while limiting persistent index contamination.
Takeaways & Limitations
The 5% operating point is a prespecified deployment budget, not evidence that the same threshold is optimal for every catalog or application.
Abstract
from arXiv · showhide
Multimodal product models can complete missing e-commerce attributes, yet current methods still optimize attribute-answer accuracy without verifying visual support, conflate transient prediction with persistent index admission, and lack explicit risk control over factually incorrect or visually unsupported values. We address these gaps with Counterfactual Visual Evidence-Guided Selective Attribute Indexing (CVE-SAI), which first infers and freezes an ontology-constrained candidate from the primary image and attribute question without catalog text, and then decides whether that candidate should enter the index. Focus-Zone Distortion (FZD) constructs an attribute-specific visual-dependence proxy through a controlled counterfactual intervention, and Evidence-Guided Attention Redistribution (EGAR) uses the proxy to refine ontology-constrained scoring. The canonical candidate is frozen before evidence necessity, evidence retention, nuisance-transformation stability, and candidate-specific catalog-text conflict audits; catalog text can only tighten admission and cannot revise the candidate. Independent family-level calibration selects one policy with a simultaneous one-sided finite-sample bound under a 5% unsafe-admission budget. Experiments on five visual attributes derived from Amazon Berkeley Objects show that CVE-SAI improves attribute inference and evidence localization, achieves the highest certified admission coverage under the shared risk protocol, and yields the strongest controlled retrieval performance with the lowest unsafe auto-induced exposure among automatic-admission systems. Separating inference from admission therefore enables visually supported attribute completion to improve retrieval while limiting persistent index contamination.
1 Introduction
CVE-SAI separates visual attribute inference from persistent index admission, addressing the need to verify image support and control unsafe catalog updates. It combines counterfactual evidence modeling, candidate audits, and calibrated admission, improving inference, certified coverage, and retrieval outcomes.
- Persistent indexing makes unsupported attribute predictions consequential because stored values are repeatedly reused for retrieval and ranking.
- CVE-SAI separates attribute inference from admission, allowing the model to abstain and defining unsafe-admission risk for admitted canonical values.
- Catalog text is unavailable during visual candidate generation and can only affect the later admission decision, not revise the frozen candidate.
- CVE-SAI achieves the highest Macro VAA, Ans.-F1, Certified Write Coverage, and controlled retrieval performance while minimizing unsafe exposure among evaluated systems.The benchmark uses five visual attributes derived from Amazon Berkeley Objects and a fixed retrieval evaluation.
- FZD derives an attribute-specific visual-dependence proxy, while EGAR uses it to refine ontology-constrained candidate scoring before evidence and catalog-text audits.
2 Related Work
Prior work supports product attribute prediction and multimodal representation learning, while separate decoding methods target visually unsupported generation. These directions do not jointly address visual verification and persistent index admission.
- MAVE documents incomplete product attributes across diverse categories, while MOON uses multimodal content for attribute prediction and cross-modal retrieval.
- POPE evaluates whether mentioned objects appear in images, whereas VCD and M3ID reduce unsupported multimodal content through visual-prompt interventions.
3 Problem Formulation
The formulation defines visual answering and selective admission over product–attribute pairs, distinguishing image truth from trusted item truth. Risk certification then evaluates unsafe admitted values and selects coverage-maximizing policies under finite-sample constraints.
- Each applicable product–attribute pair receives a visual-answer space containing ontology values, an out-of-ontology outcome, and abstention.The applicability table fixes the evaluation population, and refusals, failures, fallbacks, and withholding remain in its denominator.
- The visual stage uses only the primary image, target attribute, and frozen ontology, while catalog text becomes available only after the candidate is frozen.
- Image truth records what the primary image supports, whereas item truth records the trusted product fact using structured fields and auxiliary views.
- Risk certification uses a prespecified family-level sample and retains only observable cases with complete truth and evidence annotations.Incomplete truth, indeterminate item truth, or incomplete evidence annotation causes unreplaced attrition.
- For admitted canonical candidates, unsafe risk combines factual disagreement with absent visual grounding, counting an admission once when both occur.
- The selected policy maximizes calibration coverage subject to a simultaneous upper-confidence risk bound and a minimum-admission constraint, with α = 0.05 and nmin = 172.
4 Methodology
CVE-SAI separates product-text-free candidate inference from admission by freezing an ontology-constrained candidate, auditing its visual support and stability, and calibrating index entry under a 5% risk budget.
- Overview: Figure 1 organizes the system into product-text-free visual inference, frozen-candidate auditing, and risk-controlled indexing.Only admitted values enter the isolated auto-attribute field for retrieval.
- Counterfactual Visual Candidate Generation: FZD uses a question-conditioned counterfactual distortion to derive an attribute-specific visual-dependence proxy, while EGAR uses it to refine ontology-constrained scores.The proxy is computed before candidate freezing and retains positive attention decreases caused by weakening relevant regions.
- Frozen-Candidate Auditing: The canonical gate freezes the candidate, original-image probability, visual proxy, and redistribution strength before later views rescore that same value.Out-of-ontology responses, abstentions, fallbacks, and invalid routes are withheld before auditing.
- Frozen-Candidate Auditing: Evidence auditing combines necessity, retention, nuisance-transformation stability, and candidate-specific catalog-text conflict after visual scores are fixed.The evidence mask reaches 0.70 cumulative proxy mass with one-patch dilation; catalog text can tighten admission but cannot revise the candidate.
- Risk-Controlled Index Admission: Only candidates passing both adjusted evidence and stability thresholds are admitted; otherwise, values are withheld from the searchable auto-attribute field.Audit scores and version identifiers remain in a nonsearchable provenance record, while admitted canonical values emit searchable tokens.
- Risk-Controlled Index Admission: Independent finite-sample calibration selects a feasible policy using a Bonferroni-corrected one-sided Clopper–Pearson bound under prespecified coverage and risk criteria.The complete finite policy grid is fixed before risk labels are opened, and the selected policy is applied to test without refitting or recertification.
5 Experimental Setup
The evaluation uses five visual-attribute tasks derived from ABO, family-level splits and calibration, a primary Qwen2.5-VL backbone, shared baselines, and controlled Lucene retrieval metrics.
- Benchmark: The benchmark covers color, pattern, item shape, finish type, and style using label-blind product-family splits.The test split contains 12,656 eligible product–attribute pairs from 4,000 families.
- Benchmark: Certification fixes one pair per family before annotation or inference and retains records with complete image truth, item truth, and evidence annotation.Incomplete or indeterminate truth and evidence cause unreplaced attrition, while visual unanswerability remains eligible.
- Models and Implementation: Qwen2.5-VL-3B-Instruct is the primary backbone, with InternVL3-2B reserved for cross-backbone analysis.The primary setup averages question-token attention over visual tokens from decoder layers 28–35 and all 16 attention heads.
- Baselines: The study compares contrastive encoders, multimodal embedding models, and multimodal language models under the same candidate-freezing and family-level risk-selection protocol.CVE-SAI additionally uses FZD, EGAR, and four audits, whereas external systems use scalar confidence.
- Retrieval Protocol: Controlled retrieval uses Lucene 9.12.1 with BM25, fixed field tokenization and boosts, three-level qrels, and paired No-Auto runs.The reported metrics include NDCG@10 and Unsafe Auto-Induced Exposure@10.
6 Results and Analysis
CVE-SAI improves image-only attribute inference and evidence localization, then uses calibrated, evidence-aware admission to increase certified coverage without increasing unsafe admissions. These gains transfer to controlled retrieval, where admitted attributes improve ranking and reduce unsafe exposure.
- Attribute Inference and Evidence Localization: 72.36% Macro VAA and 84.47% Ans.-F1 let CVE-SAI outperform MOON by 2.95 and 2.75 percentage points, respectively.Every eligible pair remains in the denominator, so fallback and failed-inference outcomes lower both scores.
- Attribute Inference and Evidence Localization: 49.16% Macro Patch AUPRC lets FZD exceed Attention Rollout by 10.89 points and Grounding DINO by 18.32 points.FZD estimates attribute-specific support through the positive decrease in question-conditioned visual attention caused by focus-zone intervention.
- Risk-Controlled Index Admission: 44.50% CWC@5% versus 33.19% for MOON gives CVE-SAI an 11.31-point coverage gain while UnsafeWWR falls from 2.84% to 2.36%.The test counts show 420 additional admissions, including 416 safe and four unsafe values.
- Retrieval Effectiveness: CVE-SAI reaches 0.6749 NDCG@10, exceeding MOON by 0.0162 and the No-Auto reference by 0.0621 under a fixed Lucene protocol.The paired runs change only the admitted auto-attribute tokens, rather than the retrieval pipeline.
- Retrieval Effectiveness: 0.50% UAIE@10 is the lowest among automatic-admission systems, 0.41 percentage points below MOON, while NDCG@10 improves.The result indicates that admission gains persist downstream as better ranking with fewer unsafe exposures.
- Component Analysis: Removing FZD, EGAR, necessity, retention, nuisance stability, or text-conflict audits lowers CWC@5% and NDCG@10, while uncertified thresholding reaches 50.03% coverage but 5.82% UnsafeWWR.Necessity and retention form the core visual-support test; the other audits address complementary failure modes.
7 Conclusions and Future Work
The paper separates attribute inference from persistent index admission through CVE-SAI, combining visual-evidence auditing with calibrated risk-controlled policy selection. Across five ABO attributes, it reports improved inference and evidence localization, highest certified coverage under the 5% budget, and strongest controlled retrieval outcomes among automatic-admission systems.
- CVE-SAI separates attribute inference from persistent index admission and admits only values passing a calibrated policy under a 5% unsafe-admission budget.The framework freezes the canonical candidate before auditing and selects a unique admission rule using independent one-sided calibration.
- FZD estimates attribute-specific visual dependence, while EGAR refines ontology-constrained scoring before evidence and catalog-text conflict audits.Catalog text is used during candidate-specific auditing after the canonical candidate is frozen.
- Across five ABO attributes, CVE-SAI improves attribute inference and evidence localization while achieving the highest CWC@5% and lowest observed UnsafeWWR.Admitted values enter only a reversible auto-attribute field.
- CVE-SAI provides the highest NDCG@10 and lowest UAIE@10 among automatic-admission systems.The reported retrieval evaluation uses the controlled setting described for the automatic-admission comparison.
- Future work will extend the framework to multi-image products, evolving ontologies, and cross-catalog calibration while preserving inference, auditing, and admission separation.