PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Efficient Multimodal and Multilingual Opinion Extraction for STI: A QLoRA-Based Fine-Tuning Approach

Sheng Hong, Xuanqi Wang, Jiacheng Wang, Yuwei Wang

arXiv:2608.14152v1cs.AI

TL;DR

STI opinion extraction needs focused, reliable structured outputs across multilingual and multimodal information streams. The paper uses QLoRA-adapted VideoLLaMA models with visual grounding and case-level value assessment, achieving 64.98% Precision and 51.14% F1-score in the selected Image-Augmented setting.

  • Problem

    Multimodal STI opinion extraction remains challenged by extraction defocus, language drift, and limited output stability in non-English settings.

  • Method

    The framework fine-tunes VideoLLaMA2 models with QLoRA on a four-language multimodal dataset and uses visual anchors for JSON core-opinion extraction and case-level assessment.

  • Results

    64.98% Precision and 51.14% F1-score are achieved by fine-tuned VL2.1 in the selected Image-Augmented setting, with 74.00% sample-level accuracy.

  • Takeaways & Limitations

    The framework provides structured core opinions and case-level value signals that can support STI databases and downstream policy analysis.

  • Takeaways & Limitations

    Dense multi-opinion inputs remain challenging because the model can omit valid secondary opinions, over-extract peripheral statements, or drift toward incorrect related statements.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have reshaped semantic analysis. Opinion Extraction (OE) for Science and Technology Intelligence (STI) requires concise core opinions from large information streams. Off-the-shelf models struggle to filter noise from these streams and show limited structured-output reliability in zero-shot multilingual and multi-modal settings. To address information overload and extraction defocus, this study proposes a multimodal core-opinion extraction framework in which visual evidence serves as a contextual anchor for textual judgment. Using VideoLLaMA2 (VL2) and VideoLLaMA2.1 (VL2.1) as the base models, we apply Quantized Low-Rank Adaptation (QLoRA) fine-tuning on a curated dataset of 2,194 multilingual and multimodal samples. Under the selected Image-Augmented setting, fine-tuned VL2.1 generates structured JSON core-opinion outputs, achieving 64.98% Precision, 42.15% Recall, 51.14% F1-score, and 74.00% sample-level accuracy. Relative to the zero-shot VL2.1 setting, it raises the F1-scores of Spanish and Russian from 4.83% and 0.45% to 46.05% and 51.93%, respectively. The framework further incorporates a Fuzzy Cumulative Prospect Theory-based post-extraction triage module for case-level value assessment, providing a case-level value signal for downstream STI screening.

I. INTRODUCTION · II. METHODOLOGY · A. Overall Framework

The paper targets focused multilingual and multimodal STI core-opinion extraction from noisy information streams. It proposes a VL2/VL2.1-based framework that anchors textual judgments with visual evidence and produces structured intelligence cues for downstream screening.

  • I. INTRODUCTION: STI data increasingly combines heterogeneous news, social media, and video sources across Chinese, English, Russian, and Spanish.This motivates multilingual and multimodal opinion analysis.
  • I. INTRODUCTION: MLLM-based STI opinion extraction faces extraction defocus and language drift when processing complex multimodal information.Prior multimodal analysis supports fusing heterogeneous evidence for robustness in complex media understanding.
  • I. INTRODUCTION: The proposed framework uses VideoLLaMA2 and VideoLLaMA2.1 to convert broad semantic reading into focused core-opinion extraction.Images and videos serve as contextual anchors that help distinguish core opinions from peripheral textual statements.
  • A. Overall Framework: Visual cues provide contextual evidence that helps the model identify the core opinion expressed in text.This anchoring mechanism links subjective textual statements to supporting multimodal context.
  • I. INTRODUCTION: 2,194 annotated instances across four languages provide multimodal and textual grounding for training the model to focus on core opinions.The dataset spans English, Chinese, Spanish, and Russian.
  • II. METHODOLOGY: QLoRA adapts a 7B-parameter model with parameter-efficient updates while constraining extraction outputs to JSON-formatted core opinions.The passage identifies this as a central methodological contribution, although its reported experimental continuation is truncated.
  • A. Overall Framework: The integrated pipeline transforms unstructured, noisy STI data streams into structured core opinions and screening-oriented intelligence cues through four components.The overall framework is explicitly presented as a four-component pipeline.
  • A. Overall Framework: 2,194 annotated samples cover four languages and three modalities—text, image, and video—for multimodal data anchoring.Aligning visual cues with textual statements helps ground subjective opinions and reduce drift toward irrelevant details.

B. Dataset Construction · C. Base Model Selection

The study constructs a 2,194-sample multilingual, multimodal core-opinion extraction dataset spanning four languages and three retained modalities. It selects VL2 and VL2.1 as practical base models for the STI setting based on multimodal support, context, multilingual behavior, benchmark performance, and training feasibility.

  • B. Dataset Construction: The finalized dataset retains text, image, and video from raw collections that may also include occasional audio-bearing pages.The four target languages were selected according to geopolitical relevance and data availability.
  • B. Dataset Construction: 0.5186, 0.7899, and 0.7056 were the unweighted Kappa values for Chinese, Spanish, and English, respectively; linearly weighted values were 0.8325, 0.9216, and 0.8710.These values compare finalized human sentiment annotations with independently generated DeepSeek-V3.2 labels under the same positive/neutral/negative schema.
  • B. Dataset Construction: 2,194 OE samples span text, image, and video across English, Chinese, Russian, and Spanish.The dataset contains 1,198 text, 858 image, and 138 video samples, with 608 English, 610 Chinese, 482 Russian, and 494 Spanish samples.
  • C. Base Model Selection: Four 7B multimodal LLMs—Video-LLaVA, Video-LLaMA, VideoLLaMA2, and VideoLLaMA2.1—were compared, leading to selection of VL2 and VL2.1.The comparison considered the practical requirements of the STI opinion-extraction task.
  • C. Base Model Selection: Model selection considered text, image, and video support, context length, multilingual behavior, relevant benchmarks, and training feasibility on two A100-40GB GPUs.The comparison focused on English, Chinese, Spanish, and Russian behavior and is summarized in Table II.
  • C. Base Model Selection: VL2 and VL2.1 provide the most practical balance of context capacity, decoder quality, and training feasibility for multilingual multimodal OE.This balance motivated their use as base models for the studied STI setting.

D. Parameter-Efficient Fine-Tuning Strategies

The study uses QLoRA for memory-efficient adaptation of multimodal 7B backbones, combining NF4-quantized frozen weights with trainable low-rank updates. Training updates only the adapter parameters using autoregressive crossentropy and AdamW.

  • QLoRA configuration: QLoRA combines LoRA-style low-rank updates with 4-bit NF4 quantization of the frozen backbone, enabling multimodal 7B fine-tuning on two A100-40GB GPUs.This configuration targets limited memory and compute budgets.
  • QLoRA configuration: Low-rank adapters are attached to dense transformations in self-attention and feed-forward sublayers while the base weights remain NF4-quantized and frozen.The trainable factors are A and B, applied to representative linear transformations in the transformer blocks.
  • Adapter initialization: B0 = 0 makes the initial adapter update zero, while Gaussian initialization of A provides a stable starting direction for adaptation.The rank r and scaling factor α are selected empirically in later hyperparameter analysis.
  • Optimization: Training minimizes autoregressive crossentropy over gold JSON core-opinion sequences while updating only Θ = {A, B} with AdamW.Maximum token length, batch size, and epoch count are specified in the experimental setup.

E. Case-Level Value Assessment Based on Fuzzy Cumulative Prospect Theory · 1) Evaluation Dimensions:

The framework assesses the intelligence value of extracted-opinion cases for STI monitoring using Fuzzy Cumulative Prospect Theory. Each case combines the predicted opinion with source, temporal, and subject information and is evaluated across five dimensions.

  • E. Case-Level Value Assessment Based on Fuzzy Cumulative Prospect Theory: Each predicted opinion is organized with its source, temporal information, and subject to form a case record.Sentence embeddings support contextual matching between the predicted opinion and these source, time, and subject cues.
  • E. Case-Level Value Assessment Based on Fuzzy Cumulative Prospect Theory: The framework uses Fuzzy-CPT-based value assessment to characterize each case’s intelligence value for STI monitoring.It follows core-opinion extraction and aligns with indicator-based quality assessment work for generative AI.
  • 1) Evaluation Dimensions:: Five dimensions structure the assessment: credibility, relevance, local heat, consistency, and recency.These dimensions cover source authority, STI-topic association, strategic attention, contextual alignment, and temporal characteristics.
  • 1) Evaluation Dimensions:: Credibility evaluates the authority and evidential characteristics of the source context.This dimension focuses on the source context underlying the extracted opinion.
  • 1) Evaluation Dimensions:: Relevance evaluates the semantic association between the extracted opinion and STI priority topics.The dimension measures how closely the opinion relates semantically to priority STI topics.
  • 1) Evaluation Dimensions:: Local heat captures the strategic attention associated with the topic.It represents the topic’s strategic attention within the assessment.
  • 1) Evaluation Dimensions:: Consistency measures alignment between the opinion and its textual context, while recency reflects temporal characteristics.Together, these dimensions assess contextual fit and time-related properties of the case.

2) Fuzzy-CPT Fusion: · 3) Value Assessment Outputs: · F. Evaluation Metrics

The framework converts five dimensions into a unified Fuzzy-CPT case value score and four star-priority categories, then combines value assessment with predicted core opinions in structured JSON. Core-opinion extraction is evaluated through similarity-based Precision, Recall, F1-score, and sample-level accuracy, including a threshold-tolerant matching rule.

  • 2) Fuzzy-CPT Fusion:: Fuzzy-CPT integrates five dimensions into a unified value score V ∈[0, 1] that incorporates uncertainty and asymmetric decision preferences.Each case receives a case-level value score for subsequent categorization.
  • 2) Fuzzy-CPT Fusion:: Cases are assigned to four value categories: 5-Star Priority, 4-Star High Value, 3-Star Reference, or 2-Star Limited Value.The category thresholds are summarized in Table III.
  • 3) Value Assessment Outputs:: Value assessment outputs include dimension scores, an aggregated value score, a star category, and assessment guidance in a structured case representation.The resulting JSON record combines the predicted core opinion with contextual information and the corresponding value assessment.
  • 3) Value Assessment Outputs:: Case-level assessments can be aggregated by topic, source, and time to support issue watchlists, cross-source analysis, and screening summaries.These aggregations support downstream STI screening and monitoring workflows.
  • F. Evaluation Metrics: Four complementary metrics evaluate core-opinion extraction: Precision, Recall, F1-score, and sample-level accuracy.Opinion-level evaluation uses similarity-threshold matching rather than exact string identity.
  • F. Evaluation Metrics: τ = 0.5 is required for one-to-one matching between a predicted and unmatched ground-truth core opinion, allowing paraphrase while preventing loose many-to-one matches.Sample-level accuracy additionally indicates whether a sample contains at least one successfully matched core opinion.
  • F. Evaluation Metrics: Recall measures recovered ground-truth opinions, Precision reflects extraction reliability, and F1-score summarizes their balance through the harmonic mean.A sample is correct when at least one predicted core opinion is successfully matched under the same threshold, corresponding to TP > 0.
  • F. Evaluation Metrics: Sample-level accuracy measures whether the model recovers at least one valid core opinion from an input, supporting downstream STI analysis that begins with sample-level screening.This provides a coarse indicator of sample-level utility alongside opinion-level metrics.

III. EXPERIMENTAL RESULTS AND ANALYSIS … 2) Implementation Details:

The study evaluates multimodal core-opinion extraction on a multilingual STI dataset using four complementary metrics. VL2 and VL2.1 are fine-tuned with QLoRA under a specified Image-Augmented benchmark configuration.

  • 1) Datasets and Evaluation Metrics:: 2,194 samples span four languages and three modalities in the multimodal STI dataset.The languages are Chinese, English, Spanish, and Russian; modalities are text, image, and video.
  • 1) Datasets and Evaluation Metrics:: 200 held-out test samples are evenly distributed across the four languages, with 50 samples per language.
  • 1) Datasets and Evaluation Metrics:: Four complementary metrics evaluate core-opinion extraction: Precision, Recall, F1-score, and sample-level accuracy.
  • 2) Implementation Details:: VL2 and VL2.1 serve as the backbones and are fine-tuned using QLoRA.
  • 2) Implementation Details:: The final configuration uses LoRA rank r = 256 and scaling factor α = 512.These settings follow the hyperparameter analysis reported later.
  • 2) Implementation Details:: Unless otherwise specified, benchmark results use the Image-Augmented setting, pairing available static visual evidence with one representative visual cue.Text-only samples remain text inputs, and modality ablation includes the Text-Centered and Image-Augmented settings.

3) Prompt-Based Baselines: · B. Zero-Shot Baseline · C. Prompt-Based Baseline Comparison

The study establishes off-the-shelf zero-shot VL2/VL2.1 as reference baselines, then compares 4-Shot, Chain-of-Thought, and Tree of Thoughts prompting. Zero-shot performance favors English and Chinese, while 4-Shot improves F1-score and sample-level accuracy across both backbones.

  • 3) Prompt-Based Baselines: Zero-shot prompting uses direct instructions without exemplars and serves as the efficiency baseline at approximately 465 tokens.The comparison includes four prompt baselines, with prompt-based reasoning variants used for multimodal reasoning and structured generation.
  • B. Zero-Shot Baseline: The off-the-shelf VL2 and VL2.1 models are evaluated without fine-tuning or specialized prompting as the structured core-opinion extraction reference.Table IV reports the zero-shot results for both models.
  • B. Zero-Shot Baseline: VL2.1 zero-shot F1-score falls from 35.64% in English to 4.83% in Spanish and 0.45% in Russian, while its overall F1-score is 17.95%.VL2 follows the same cross-lingual pattern, declining from 36.78% in English to 14.72% in Spanish and 6.21% in Russian, with 18.92% overall F1-score.
  • B. Zero-Shot Baseline: VL2.1 zero-shot extraction reaches 68.00% Recall, 84.00% sample-level accuracy, and 24.15% Precision in English.In Chinese, VL2.1 reaches 65.63% Recall, 66.00% sample-level accuracy, and 17.21% Precision; these results form the reference baseline.
  • C. Prompt-Based Baseline Comparison: The prompt comparison evaluates 4-Shot, Chain-of-Thought, and Tree of Thoughts strategies on both VL2 and VL2.1 before parameter updates.The comprehensive results are presented in Table V.
  • C. Prompt-Based Baseline Comparison: 4-Shot increases VL2 F1-score from 18.92% to 33.38% and sample-level accuracy from 36.00% to 73.00% relative to zero-shot.For VL2.1, 4-Shot increases F1-score from 17.95% to 33.71% and sample-level accuracy from 42.00% to 84.50%.
  • C. Prompt-Based Baseline Comparison: ToT achieves the best prompt-only VL2 F1-score at 36.69%, whereas VL2 CoT reaches 14.88% F1-score and 28.50% sample-level accuracy.VL2 CoT Spanish and Russian F1-scores are 3.14% and 1.25%, respectively.
  • C. Prompt-Based Baseline Comparison: On VL2.1, CoT reaches 18.09% F1-score, while ToT reaches 23.27% F1-score with 36.50% sample-level accuracy.Reasoning-oriented prompts therefore produce different results from 4-Shot across the two backbones.

D. Impact of QLoRA Hyperparameters · E. Performance of Multimodal QLoRA Fine-Tuning

The study selects r = 256 and α = 512 as the strongest tested QLoRA configuration, then applies it to Image-Augmented fine-tuning of VL2 and VL2.1. Fine-tuning improves precision and F1-score over prompt-only adaptation, while dense multi-opinion inputs remain challenging.

  • D. Impact of QLoRA Hyperparameters: The hyperparameter ablation varies VL2 adapter capacity across r = 64 (α = 128), r = 128 (α = 256), and r = 256 (α = 512), keeping α/r = 2.The search was conducted under the Text-Centered setting to isolate adapter capacity from visual grounding.
  • D. Impact of QLoRA Hyperparameters: 41.92% Recall and 47.04% F1-score are achieved at r = 256, up from 29.74% and 37.63% at r = 64, while Precision changes modestly from 51.21% to 53.59%.Sample-level accuracy also rises from 60.00% to 72.00% across these settings.
  • D. Impact of QLoRA Hyperparameters: 51.23% Russian F1-score and 45.33% English F1-score are obtained at r = 256, while Spanish reaches 43.21% and Chinese peaks at 51.01% under r = 128.At r = 256, Chinese Recall rises while Precision decreases from 71.70% to 54.67%.
  • E. Performance of Multimodal QLoRA Fine-Tuning: The selected r = 256 and α = 512 configuration is used to fine-tune VL2 and VL2.1 under the Image-Augmented setting for cross-lingual core-opinion extraction.The fine-tuning stage is intended to improve performance beyond prompt-based baselines.
  • E. Performance of Multimodal QLoRA Fine-Tuning: 64.98% Precision and 51.14% F1-score are achieved with QLoRA versus 21.65% and 33.71% for VL2.1 under 4-Shot, while Recall falls from 76.11% to 42.15%.Sample-level accuracy decreases from 84.50% to 74.00%, indicating a precision-oriented shift relative to prompt-only adaptation.
  • E. Performance of Multimodal QLoRA Fine-Tuning: 64.98% Precision and 51.14% F1-score for VL2.1 exceed VL2’s 55.52% and 48.82%, whereas VL2 retains higher Recall and sample-level accuracy at 43.56% and 75.00%.VL2.1 is particularly stronger in Chinese, reaching 73.68% Precision and 54.90% F1-score versus 58.33% and 50.00% for VL2.
  • E. Performance of Multimodal QLoRA Fine-Tuning: 44 of 50 English test cases contain at least one valid core opinion, but errors include partial misses, peripheral additions, and incomplete recovery in dense multi-opinion passages.The model may preserve a central strategic statement while omitting secondary opinions, and residual over-extraction also remains.

F. Ablation Study on Modalities

The ablation compares text-only, image-augmented, and full-modality inputs for fine-tuned VL2 and VL2.1. Image-Augmented is selected as the main benchmark because it improves or maintains performance, while Full Modality remains supplementary.

  • Experimental settings: The ablation evaluates Text-Centered, Image-Augmented, and Full Modality settings using fine-tuned VL2 and VL2.1 on the test set.Text-Centered uses text only, Image-Augmented adds one representative static visual cue, and Full Modality includes text, static images, and temporal videos.
  • Image-Augmented Inputs Are the Selected Main Configuration: 64.98% Precision is achieved by VL2.1 with Image-Augmented inputs versus 63.32% with Text-Centered inputs, while F1-score remains 51.14% versus 51.12%.For VL2, Image-Augmented inputs also raise F1-score from 47.04% to 48.82% and Precision from 53.59% to 55.52%.
  • Full Modality Results: 48.75% F1-score is obtained by VL2 under Full Modality versus 48.82% with Image-Augmented inputs, while VL2.1 reaches 51.06% versus 51.14%.Full Modality is retained as a supplementary modality comparison.

G. Comparison with External Multimodal Models · H. Value Assessment Demonstration

Under shared zero-shot prompts and strict structured-output constraints, fine-tuned VL2.1 outperformed the external multimodal models in Precision and F1-score. The value-assessment module then evaluated 289 predicted opinion cases, organizing them into value categories and topic-level STI monitoring signals.

  • G. Comparison with External Multimodal Models: 64.98% Precision and 51.14% F1-score were achieved by fine-tuned VL2.1, the highest values among all compared systems.The comparison used DeepSeek-V3.2, Qwen3-Omni-Flash (VideoFrames), and Claude-Sonnet-4-6 under identical zero-shot prompts and strict structured-output constraints.
  • G. Comparison with External Multimodal Models: 10.10 percentage points separated fine-tuned VL2.1’s 51.14% F1-score from Claude-Sonnet-4-6’s 41.04%.The difference from DeepSeek-V3.2’s 31.25% F1-score was 19.89 points.
  • G. Comparison with External Multimodal Models: 19.37% Precision, 80.80% Recall, 31.25% F1-score, and 85.50% sample-level accuracy were achieved by DeepSeek-V3.2.Claude-Sonnet-4-6 achieved 28.46% Precision, 73.53% Recall, and 41.04% F1-score, while Qwen3-Omni-Flash achieved 29.11% Precision, 45.20% Recall, and 35.41% F1-score.
  • H. Value Assessment Demonstration: 289 predicted opinion-level cases were assessed by combining each predicted core opinion with its source, temporal information, and subject context.The framework evaluates five dimensions and produces a value score and star category for each case.
  • H. Value Assessment Demonstration: 180 cases were categorized as 3-Star Reference, 66 as 4-Star High Value, 3 as 5-Star Priority, and 40 as 2-Star Limited Value.This distribution provides a structured overview of the value characteristics of the extracted opinions.
  • H. Value Assessment Demonstration: 69 cases categorized as 4-Star High Value or 5-Star Priority were summarized by topic and source coverage.Frontier AI, electric and autonomous vehicles, clean energy transition, and digital transformation were recurrent high-value topics.
  • H. Value Assessment Demonstration: 0.8109 was the value score assigned to an English India IT-sector case, which was categorized as 5-Star Priority.Its attributes were Industry Report, fiscal year 2024, Digital Transformation, and the opinion “We are witnessing a digital revolution in India”.
  • H. Value Assessment Demonstration: The value-assessment module integrates extracted opinions with contextual information to support the organization, comparison, and monitoring of STI-related opinions.The resulting categories provide a structured representation of case value for STI monitoring.

IV. CONCLUSION

The study presents a multimodal and multilingual STI core-opinion extraction framework combining contextual anchoring, QLoRA-based parameter-efficient fine-tuning, and case-level value assessment. Fine-tuned VideoLLaMA2.1 achieves strong held-out performance and substantially improves Russian and Spanish extraction over zero-shot VL2.1, while supporting downstream STI analysis.

  • Framework: The framework integrates multimodal contextual anchoring, QLoRA-based PEFT, and case-level value assessment for STI core-opinion extraction.It is designed to address language drift and the computational cost of traditional zero-shot MLLMs.
  • Performance: 64.98% Precision, 42.15% Recall, 51.14% F1-score, and 74.00% sample-level accuracy are achieved by fine-tuned VideoLLaMA2.1 under the selected Image-Augmented setting.The results come from a held-out 200-sample test set drawn from a self-constructed STI dataset of 2,194 samples.
  • Performance: 46.05%–51.93% F1-scores are obtained for Russian and Spanish after fine-tuning, compared with 0.45%–4.83% in the zero-shot VL2.1 setting.The reported language-specific gains cover Russian and Spanish.
  • Applications: The framework produces structured core opinions with contextual case information and value assessment results for integration with STI databases and downstream policy analysis.It is positioned for automated global intelligence monitoring in complex cross-domain information environments.
  • Future work: Future work will extend case-level assessment toward topic-level trend synthesis and broader crossplatform STI monitoring.This identifies the planned direction for expanding the assessment and monitoring scope.
Loading 2608.14152v1…