PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videos

Tong Wu, Ming Cheng, Jiazhen Hu, Jiaying Gong, Hoda Eldardiry

arXiv:2609.06410v1cs.CLcs.CV

TL;DR

Static product images can miss temporal, multi-angle, and fine-grained attribute cues, while video VLMs face challenges with noisy sampling and domain knowledge. ViS-CoT combines representative-frame selection, visual search, and interleaved multimodal reasoning in a training-free, plug-and-play pipeline. Across five state-of-the-art video VLMs, it improves average micro-F1 on VideoAVE by 17.91 percentage points.

  • Problem

    Static product images can miss temporal, multi-angle, and fine-grained details, while video VLMs may sample noisy or redundant content and lack domain knowledge for product AVE.

    Source: §1 · pp. 1–2

  • Method

    ViS-CoT selects representative video frames, retrieves similar-product knowledge, iteratively refines hypotheses with captions and ASR, and integrates the evidence into attribute predictions.

    Source: §1 · p. 2

  • Results

    17.91 percentage points average improvement in micro-F1 on VideoAVE across five state-of-the-art open-source video VLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ViS-CoT offers a training-free, resource-efficient, plug-and-play approach for video-to-text e-commerce AVE with open-source video VLMs.

    Source: §1 · p. 2

    The iterative search and reasoning procedure adds inference latency, which may hinder real-time or large-scale production use.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Existing approaches to visual attribute value extraction (AVE) primarily rely on static product images, failing to capture temporal cues, multi-angle views and fine-grained visual details. Directly applying video vision-language models (VLMs) to product AVE results in limited performance due to the lack of domain knowledge, and fine-tuning them requires extensive high-quality data and substantial computational resources. Thus, we propose visual search augmented chain-of-thought reasoning (ViS-CoT), a training-free, plug-and-play pipeline that can be easily applied to any open-source video VLM for video-to-text AVE in e-Commerce. Specifically, ViS-CoT employs visual clustering to identify representative frames, followed by visual search to retrieve semantically similar product knowledge that can enrich attribute cues. Next, an interleaved CoT reasoning module iteratively refines reasoning through visually-aligned auxiliary texts derived from captioning and automatic speech recognition. Finally, the integrated information guides the model toward accurate and fine-grained attribute predictions. Extensive experiments across 14 product categories on the VideoAVE dataset show that ViS-CoT consistently enhances multiple state-of-the-art video VLMs, achieving an average improvement of 17.91 percentage points in micro-F1.

1 Introduction

Static product images can miss fine-grained details, while uniform video sampling may include noisy or redundant content. ViS-CoT combines visual search and staged multimodal reasoning, improving VideoAVE performance across video VLMs.

1 Introduction

  • Product videos reveal details such as closure and protection type that multi-image sets may miss, while rotation and changing conditions expose subtle properties.Source: §1 · p. 1
  • Uniform frame sampling can capture irrelevant, noisy, or redundant content that results in hallucinations; product video understanding targets item properties rather than narratives.Source: §1 · p. 2
  • 17.91 percentage points average micro-F1 improvement on VideoAVE across five state-of-the-art open-source video VLMs, evaluated across 14 product categories, with competitive performance versus fine-tuned VLMs.The experiments validate the contribution of each component.Source: §1 · p. 2
  • ViS-CoT extracts representative frames through visual clustering, retrieves similar-product attribute priors, iteratively refines hypotheses using captions and ASR, then integrates multimodal evidence for predictions.The four stages are key-frame extraction, visual search, interleaved chain-of-thought reasoning, and integration and generation.Source: §1 · p. 2
  • ViS-CoT is a training-free, plug-and-play pipeline designed to enhance video VLMs for e-commerce video-to-text attribute extraction.The paper describes it as broadly applicable and resource-efficient.Source: §1 · p. 2
  • The two-stage design—vision-only hypothesis followed by retrieval-grounded refinement—is reported as more effective than visual search or chain-of-thought reasoning alone.Source: §1 · p. 2

2 Related Work

Related work progresses from text-based and image-based attribute extraction toward video understanding and multimodal reasoning. The paper situates visual search as a way to ground product-focused video reasoning.

2 Related Work

  • Early AVE used text-only inputs, while VLM-based AVE captures image information but remains limited by static views that miss temporal and multi-angle cues (Gong and Eldardiry, 2024; Fang et al., 2024a; Gong et al., 2025b; Trabelsi et al., 2025).Source: §2 · p. 2
  • Video VLMs still struggle with fine-grained task-specific understanding, and specialized video methods span classification, captioning, and localization.Source: §2 · pp. 2–3
  • Retrieval-augmented video comprehension incorporates contextual knowledge, which the paper applies through visual search to e-commerce AVE (Jeong et al., 2025; Zheng et al., 2025; Ren et al., 2025).Source: §2 · p. 3
  • Multimodal chain-of-thought work includes prompt-, plan-, and learning-based approaches, with expert tools and external knowledge also used to support reasoning.Source: §2 · p. 3
  • General-purpose multimodal reasoning pipelines designed for visual question answering can include redundant or noisy information for e-commerce product-video understanding.Source: §2 · p. 3

3 Methodology

ViS-CoT is a training-free visual-search-augmented reasoning pipeline for video-language models, with stages for key-frame extraction, visual search, interleaved reasoning, and final generation. It formulates product AVE both with predefined attributes and as open attribute-value pair extraction.

3 Methodology

  • ViS-CoT combines key-frame extraction, visual search, interleaved chain-of-thought reasoning, and integration and generation in a training-free pipeline.The pipeline is described as integrable into video-language models.Source: §3 · p. 3

3.1 Problem Formulation

  • The task covers both extracting values for predefined attributes and open extraction of attribute-value pairs when no attributes are specified.The input is a product video; auxiliary transcripts, retrieved metadata, and contextual cues can enrich it.Source: §3.1 · p. 3

3.2 Key Frame Extraction

  • Frame-quality filtering removes blurry, overly dark or bright, and low-contrast frames, while semantic clustering selects representative frames and reduces redundancy.K-means groups the cleaned frames, and the frame closest to each cluster centroid is selected as a key frame.Source: §3.2 · pp. 3–4

3.3 Visual Search

  • Visual search encodes training product images and video key frames with SigLIP2, averages query-frame similarities, and retrieves top-k products with associated attribute values.The retrieved products form the query product's candidate aspect set.Source: §3.3 · p. 4

3.4 Interleaved Chain-of-Thought Reasoning

  • Interleaved reasoning starts from visual-only evidence, then revisits and refines the reasoning using captions, refined ASR transcripts, and retrieved product knowledge.Captions describe key-frame content, while refined transcripts are summarized from Whisper ASR output.Source: §3.4 · p. 5

3.5 Integration and Generation

  • The final VLM input combines key frames with captions, refined transcripts, similar-product data, and the interleaved reasoning outcome to generate the result.The integrated input is described as preserving complementary multimodal cues and reducing ambiguity.Source: §3.5 · p. 5

4 Experiments

The experiments evaluate ViS-CoT on VideoAVE across five video-language models and report consistent gains in both attribute-conditioned and generalized settings. Ablations, a case study, sensitivity tests, and error analysis examine how the system works and where predictions remain difficult.

4.1 Experimental Setup

  • The study evaluates VideoAVE’s 14 product categories and 172 unique attributes using five open-source video-language models.It reports category- and attribute-level fuzzy F1 for both attribute-conditioned prediction and open attribute-value extraction.Source: §4.1 · pp. 5–6

4.2.1 Main Results

  • ViS-CoT consistently improves all five base VLMs across attribute-conditioned and generalized settings, with generally higher performance when attributes are provided.The gains also hold in the more challenging generalized setting, where attributes must be discovered automatically.Source: §4.2.1 · p. 6
  • ViS-CoT improves fine-grained visual and semantic attributes, while added reasoning can hurt readily identifiable attributes such as brand and color.Retrieved product knowledge helps with ambiguous or visually absent attributes such as country of origin and suggested users.Source: §4.2.1 · pp. 6–7, Table 2 · p. 7

4.2.2 Ablation Study

  • 46.5% F1 for the single-pass variant is 11.9 pp below full ViS-CoT; clean-test visual-search gains remain +13.3 pp versus +14.9 pp on the full set.Clustering outperforms random frame sampling, and full ViS-CoT performs best across almost all categories while remaining competitive with a full-size finetuned model.Source: §4.2.2 · p. 7, Figure 3 · p. 7

4.2.3 Case Study

  • For a Cocoon organizer sleeve, ViS-CoT corrects material to Nylon and pattern to Solid, but predicts Zipper rather than the gold Magnetic closure.Close-up weave cues and retrieved products support the material correction; captions disambiguate textured weave from a printed pattern, while transcript phrases bias closure reasoning.Source: §4.2.3 · p. 8, Figure 4 · p. 8

4.2.4 Parameter Sensitivity

  • Among tested settings, k = 10 gives the best overall performance, while increasing cluster count generally slightly reduces performance across four categories.k = 20 does better in Arts, Grocery, and Pet, but can add noise and latency elsewhere; m = 5 is the selected moderate cluster count.Source: §4.2.4 · p. 8, Figure 5 · p. 9, Figure 6 · p. 9

4.2.5 Error Analysis

  • Errors comprise 28.6% visually ambiguous cases, 28.7% knowledge-dependent cases, and 42.7% hallucinations; hallucinations account for roughly 37–50% of failures across seven categories.Knowledge-dependent errors are more frequent in Pet and Toys, while visually ambiguous errors occur more often in Patio and Grocery.Source: §4.2.5 · pp. 8–9
  • Proposed improvement directions include adding structured product knowledge, adapting reasoning depth to attribute difficulty, and improving confidence calibration.Source: §4.2.5 · p. 9

5 Conclusion

ViS-CoT combines visual clustering, retrieved product information, captions, transcripts, and interleaved reasoning for video-to-text e-Commerce AVE. It offers a training-free, plug-and-play approach and reports gains across five video VLMs in both AVE settings.

5 Conclusion

  • ViS-CoT is a resource-efficient, training-free plug-and-play solution applicable to open-source video VLMs for e-Commerce AVE.The framework reports performance gains across five video VLMs under both attribute-conditioned and open AVE settings.Source: §5 · p. 9
  • The pipeline combines visual clustering for key frames with retrieved similar-product data, captions, transcripts, and interleaved chain-of-thought reasoning.Source: §5 · p. 9
  • ViS-CoT shows strong performance gains across five state-of-the-art video VLMs in both attribute-conditioned and open AVE settings.Source: §5 · p. 9

6 Limitations

The authors identify inference latency and evaluation scope as limitations of ViS-CoT. Their evaluation focuses on video-to-text e-Commerce AVE, VideoAVE, and open-source video VLMs.

6 Limitations

  • The iterative search and reasoning pipeline adds inference latency that may hinder real-time or large-scale production use.The authors analyze this latency in Appendix A.Source: §6 · pp. 9–10
  • The study does not compare ViS-CoT with chain-of-thought methods for storytelling, captioning, or summarization because those tasks use different objectives and metrics.Source: §6 · p. 10
  • Evaluation centers on video-to-text e-Commerce AVE and VideoAVE, which the authors identify as the only publicly available benchmark directly aligned with this task.The authors also focus on open-source video VLMs; commercial-system evaluation is constrained by API cost, limited transparency, and changing versions.Source: §6 · p. 10

A Inference Latency Analysis

The latency analysis reports both the efficiency of ViS-CoT’s performance gains and its per-query runtime. Keyframe clustering is a one-time preprocessing cost, while visual search adds little latency.

A Inference Latency Analysis

B Ablation Study: Component Contributions

The ablation study finds that the full ViS-CoT system outperforms the baseline and visual search alone, while visual search alone substantially improves average F1. Visual search also improves all seven tested categories.

B Ablation Study: Component Contributions

C Single-Pass vs. Interleaved Reasoning

Single-pass tests whether ViS-CoT’s evidence can be used effectively without staged reasoning. Interleaved reasoning performs better when both approaches use the same retrieved and auxiliary evidence.

C Single-Pass vs. Interleaved Reasoning

D Label-Leakage Robustness

Visual search retains most of its gain on a test subset without product overlap with training. The authors also describe safeguards and retrieval design choices intended to distinguish attribute transfer from direct label copying.

D Label-Leakage Robustness

  • Product_id overlap ranges from 19% to 37% across 14 categories, reflecting products shared across the marketplace splits.The clean subset retains 63–81% of samples per category.Source: D Label-Leakage Robustness · p. 17
  • Retrieval drops neighbors with the query’s product_id, and the clean-subset evaluation removes every test product appearing in training.The skip_self=True safeguard prevents trivial self-matches but does not rule out neighbors with the same attribute labels.Source: D Label-Leakage Robustness · p. 17
  • The clean-subset visual-search gain is +13.3 pp, only 1.6 pp below the +14.9 pp full-set gain.The clean subset excludes test products whose product_id appears in training; the authors conclude the gain is not driven by near-duplicate label leakage.Source: Table 8 · p. 17, D Label-Leakage Robustness · p. 17
  • The authors attribute most improvement to attribute-level transfer from visually similar but distinct products, rather than direct label copying.Retrieved context aggregates attribute-value frequencies across top-k neighbors instead of presenting one nearest-neighbor label.Source: D Label-Leakage Robustness · p. 17
  • Stage 1 forms a visual hypothesis before retrieved labels appear, which the authors say mitigates label-anchoring effects in the reasoning trace.Source: D Label-Leakage Robustness · p. 17

E Open-Source Reproducibility

The authors replace two GPT-4.1 modules with Qwen2.5-7B and report similar end-to-end performance. Qwen2.5-7B matches or slightly exceeds GPT-4.1 in three categories.

E Open-Source Reproducibility

  • Qwen2.5-7B replaces GPT-4.1 for transcript summarization and attribute definition generation (Adef) in the full ViS-CoT pipeline.The model summarizes raw ASR transcripts and generates structured attribute descriptions for each category.Source: E Open-Source Reproducibility · p. 17
  • Performance differences across categories are small, and Qwen2.5-7B matches or slightly exceeds GPT-4.1 in Arts, Pet, and Toys.Source: E Open-Source Reproducibility · p. 17
  • Qwen2.5-7B yields 58.3% average micro-F1 versus 58.5% with GPT-4.1, a difference of −0.2 percentage points.Table 9 reports end-to-end scores across seven representative categories.Source: Table 9 · p. 17, E Open-Source Reproducibility · p. 17
Loading 2609.06410v1…