PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Requirements Traceability Link Recovery in Industrial Settings

Haoming Huang, Shinpei Hayashi, Kazuya Kohata, Myeongki Kim, Yuki Kurihara

arXiv:2610.34503v1cs.SEcs.AIcs.IR

TL;DR

Automated TLR must recover correct links while limiting false positives, but binary classifier outputs offer no direct adjustment to that balance. The paper extends RAG-based TLR with multimodal retrieval and confidence filtering; on Settest, F1 is .310 versus .114 for the baseline.

  • Problem

    Automated TLR must recover correct links while limiting false-positive candidates for engineer review, but a fixed binary classifier provides no direct way to adjust the balance between missed and incorrect links.

    Source: §I · pp. 1–2

  • Method

    The approach retrieves text and image candidates, classifies each pair with a vision-language model, and reports verbalized confidence for filtering predicted links.

    Source: §I · p. 2

  • Results

    On Settest, the approach achieves F1 = .310, compared with F1 = .114 for the baseline.

    Source: 3) Results and Discussion · p. 7

  • Takeaways & Limitations

    On Settest, full-input runs suggest the pipeline can extend recovery to image-involving links while retaining text–text capability.

    Source: 3) Results and Discussion · p. 8

    The confidence score is self-reported rather than a calibrated probability, and its values cluster at a few levels near 1.0.

    Source: A. Construct Validity · p. 10

Abstract

from arXiv · show

Requirements traceability links support activities such as change-impact analysis, coverage checking, and validation. Industrial requirements can include traceable images, which text-only approaches cannot process directly. For a fixed classifier configuration, binary decisions alone offer no direct way to adjust the balance between missed and incorrect links. We extend a retrieval-augmented generation (RAG) approach for inter-requirements traceability link recovery (TLR). Our extension retrieves text and image candidates separately, classifies each retrieved pair with a vision-language model, and filters the predicted links by verbalized confidence. We evaluate our approach on three non-overlapping industrial requirement sets and six public text-only datasets. On the industrial test sets, our approach achieves an F1 score of 0.310, compared with 0.114 for our replication of the baseline approach by Hey et al. Our approach recovers links in all four modality combinations while improving text-only F1 on the industrial data and maintaining similar performance on the public datasets, but recovery remains limited by failures to retrieve related text requirements for image queries. Confidence filtering produces link sets with different precision and recall, removing false positives and some correctly recovered links from the unfiltered link set. This study provides industrial evidence on the potential and limitations of a current RAG-based TLR approach for multimodal requirements and identifies cross-modal retrieval as the component to prioritize in further optimization.

I. INTRODUCTION

Industrial traceability must account for image elements and differing precision–recall priorities, while binary classifier outputs offer no direct way to adjust that balance. This study extends RAG-based inter-requirements TLR with multimodal retrieval and confidence filtering.

I. INTRODUCTION

  • Industrial trace links support change-impact analysis, coverage checking, validation, testing, and navigation, and accurate links improved completion time and correctness in selected maintenance tasks,.Source: §I · p. 1
  • When image elements are omitted, links involving them cannot be recovered, leaving engineers to check those elements separately; industrial specifications include graphical artifacts in many projects.Source: §I · p. 1
  • Different industrial tracing tasks may require different precision–recall balances, but a fixed classifier's binary output provides no direct way to adjust them.Source: §I · pp. 1–2
  • The extension retrieves text and image candidates, classifies pairs with a vision-language model, and uses verbalized confidence to form different subsets of positive predictions.Source: §I · p. 2
  • The study evaluates a replicated text-only baseline and the multimodal approach on non-public industrial datasets and six public text-only datasets.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior work spans classical retrieval, RAG-based link classification, visual-document processing, and confidence-based filtering, but the cited studies leave multimodal trace-link recovery unaddressed.

A. Pre-LLM TLR Approaches

  • Classical TLR ranks target artifacts by textual similarity, while other approaches combine token patterns and embeddings or map fine-grained requirements and code units,,.Source: A. Pre-LLM TLR Approaches · p. 2

B. LiSSA and RAG-Based Inter-Requirements TLR

  • LiSSA and Hey et al. use retrieval followed by LLM classification, while this study adopts their element-level RAG setting; trace-link semantics depend on the artifact relation and oracle,.Prompt formulation can substantially change precision and recall in TLR.Source: B. LiSSA and RAG-Based… · p. 2

C. Multimodal Requirements and Software Documents

D. Human Review and LLM Confidence

  • Human review remains part of traceability workflows, and verbalized confidence may vary in reliability across tasks, so it should not be treated as calibrated probability without task-specific evidence,, [17].The selected-study comparison positions this work as combining image processing with trace-link recovery and confidence filtering,,,,.Source: D. Human Review and LLM Confidence · pp. 2–3

III. METHODOLOGY

The method represents requirements as text or image elements, retrieves candidate low-level elements, and uses a VLM to classify pairs and threshold positive predictions by confidence.

III. METHODOLOGY

  • A pretrained generative VLM extends prompt-based classification to image inputs using its trained visual interface, without requiring the authors to develop and train a separate connection to a text-only LLM.Source: §III · p. 3

A. Task and Representation of Requirement Elements

  • The task represents HLR and LLR elements as text or images and predicts element-level links; a VLM classifies retrieved pairs, and positive decisions meeting the confidence threshold become links.Source: A. Task and Representation of… · p. 3

B. Multimodal Candidate Retrieval

  • Base retrieval selects k LLR candidates per HLR, whereas Modality separately selects k text and k image candidates; experiments use k=4, and the classifier receives original content.Embeddings rank candidates but are not passed to the classifier.Source: B. Multimodal Candidate Retrieval · p. 3

C. VLM Classification and Confidence Filtering

The classifier processes retrieved text–image pairs in any modality combination and can filter its positive decisions by verbalized confidence. The paper also defines configurations that vary retrieval or classifier input while holding other settings fixed.

C. VLM Classification and Confidence Filtering

  • Text and image elements are supplied as text blocks or direct visual input, allowing classification of text–text, text–image, image–text, and image–image pairs.Each retrieved HLR–LLR pair is classified separately.Source: C. VLM Classification and Confidence… · p. 3
  • A confidence threshold retains only parsed positive decisions whose verbalized confidence meets the threshold, producing binary link sets.The threshold is selected before test-set evaluation; missing confidence is assigned 1.0, while malformed confidence is assigned 0.0.Source: C. VLM Classification and Confidence… · pp. 3–4

D. Configuration Options

  • The reference configuration and five single-factor variants change retrieval or classifier input while keeping the model, decoding, output format, and parsing rules fixed.Variants add relation rules, artifact-language instructions, neighboring context, section headings, or modality-specific retrieval.Source: D. Configuration Options · p. 4

IV. EMPIRICAL STUDY

The study asks whether multimodal retrieval and classification can improve industrial link recovery while preserving text-only performance, support different precision–recall priorities, and fit smaller-model constraints. It evaluates industrial and public datasets using distinct oracle scopes.

IV. EMPIRICAL STUDY

  • The empirical study examines four questions: industrial recovery versus a text-only baseline, retained text-only recovery, confidence filtering, and quality with smaller VLMs.The questions are motivated by multimodal industrial artifacts, differing precision–recall priorities, and hardware constraints.Source: §IV · pp. 4–5

1) Datasets and Oracle Scopes:

  • The public evaluation uses six English text-to-text datasets with their distributed gold-link oracles to assess text-only compatibility.The datasets were evaluated by Hey et al. and distributed in their replication package.Source: 1) Datasets and Oracle Scopes · p. 5
  • Three non-overlapping Japanese requirement sets from different parts of one system are split into validation Set A and test Sets B and C, with choices frozen before testing.Configurations and confidence thresholds are selected on Set A.Source: 1) Datasets and Oracle Scopes · p. 5
  • The industrial documents are represented as ordered text and image elements, and each oracle labels every HLR–LLR pair for refinement or predecessor–successor relations.Two practitioners annotated the industrial oracles, with one practitioner labeling each set.Source: 1) Datasets and Oracle Scopes · p. 5
  • 150 of 356 industrial gold links (42.1%) involve images, and Set C has no text–text gold links.Image-involving links include text–image, image–text, and image–image pairs.Source: 1) Datasets and Oracle Scopes · p. 5

2) Metrics:

The study evaluates link recovery with precision, recall, and Fβ, and distinguishes text-only coverage from full-task coverage. It compares an executable replication of Hey et al. with the multimodal approach.

2) Metrics:

  • Precision, recall, and Fβ are computed from true positives, false positives, and false negatives, with β values of 0.5, 1, and 2 weighting precision or recall differently.Combined-set metrics sum TP, FP, and FN before calculation because negative pairs dominate the Cartesian product.Source: 2) Metrics · pp. 5–6
  • Text-only coverage measures the share of full-oracle gold links whose two endpoints are text, while text-only predictions are assessed against both the text–text subset and the full oracle.The full-oracle assessment also counts unrecovered image-involving links as false negatives.Source: 2) Metrics · p. 6

3) Compared Approaches:

  • The multimodal approach retrieves text and image candidates with a multimodal embedding model and uses a Gemma 4 VLM to produce a binary decision and confidence score.For the industrial text-only comparison, it retrieves the four highest-similarity text candidates using the reference classifier input.Source: 3) Compared Approaches · p. 6
  • .451 and .449 are the arithmetic mean F1 values across six datasets for Hey et al.’s reported and replicated results, respectively.The authors use the executed replication as the proxy baseline in subsequent comparisons.Source: Table V · p. 6, 3) Compared Approaches · p. 6

4) Evaluation Procedure:

The evaluation selects configurations and confidence thresholds on validation data before test evaluation. For the first comparison, the text-only baseline is run on inputs with images removed and evaluated against text-only and full oracle scopes.

4) Evaluation Procedure:

  • The procedure uses unquantized bfloat16 runs on one 80 GB NVIDIA H100 with fixed temperature zero and a maximum output length of 1,024.The stated software setup is PyTorch 2.8, CUDA 12.9, and Transformers 5.14.Source: 4) Evaluation Procedure · p. 6

B. Configuration Selection

  • Configurations and thresholds are selected on Setvalid by comparing maximum overall F1 across thresholds from zero to one in 0.05 increments.Thresholds for F0.5, F1, and F2 are selected on the chosen configuration’s stored responses, with higher thresholds breaking ties.Source: B. Configuration Selection · p. 6
  • .517 was the highest validation maximum overall F1 for B+M, compared with .481 for B; candidate recall rose from .438 to .492.B+M also raised candidate recall for text–image and image–text links from zero to .500 and .200, respectively, and was fixed before Settest evaluation.Source: B. Configuration Selection · p. 6, Table VI · p. 7

2) Design:

  • The text-only baseline is run after removing image elements and evaluated against both the text–text oracle and the full oracle, where unrecovered image links count as false negatives.The full oracle contains text–text, text–image, image–text, and image–image links.Source: 2) Design · p. 7

3) Results and Discussion:

On the industrial test sets, the multimodal approach recovered more links than the text-only baseline, including image-involving links, while retaining similar text–text and public-dataset performance.

3) Results and Discussion:

  • The text-only baseline recovered 27.9% of links accessible to its input but only 9.9% of the full industrial oracle.64.3% of Settest links involved at least one image element.Source: 3) Results and Discussion · p. 7

2) Design:

  • The evaluation compared full-input filtered and unfiltered predictions, industrial text-only recovery, and unfiltered results on six public text-to-text datasets.The selected thresholds were evaluated against the full oracle for full-input runs and against the TT oracle for text-only runs.Source: 2) Design · p. 7, 3) Results and Discussion · p. 7

3) Results and Discussion:

  • 49 of 171 industrial gold links were recovered, with F1 = .310 versus .114 for the baseline across the complete executed systems.The systems differ in modalities, models, prompts, parsers, filtering, and candidate counts.Source: 3) Results and Discussion · p. 7, Table VII · p. 8
  • 37 of 110 image-involving links were recovered, including 9 of 14 text–image, 7 of 60 image–text, and 21 of 36 image–image links.The 16 recovered links in the two cross-modal directions were outside the text-only input.Source: 3) Results and Discussion · p. 7
  • On Settest, unfiltered text–text F1 was .215 for full-input runs and .214 for text-only runs.On the public data, the approach obtained .383 compared with .389 for the baseline.Source: 3) Results and Discussion · p. 8

E. RQ3: Confidence Trade-Offs

The study tests whether confidence thresholds selected on validation data can provide different precision–recall balances on industrial test sets. It applies each threshold to stored classifier responses without retuning.

1) Motivation:

  • Different tracing activities may prioritize precision and recall differently, motivating adjustable predicted link sets.The study examines whether validation-selected thresholds provide useful balances on Settest.Source: 1) Motivation · p. 8

2) Design:

  • Table IX reports the selected confidence thresholds for the approach on Setvalid.Figure 3 shows validation criteria and resulting F1 curves for the three industrial sets.Source: Table IX · p. 8, 2) Design · p. 8
  • The classifier responses are stored once, then confidence thresholds from zero to one in 0.05 increments determine which positive decisions are retained.For each Fβ measure, the threshold maximizing that measure on Setvalid is applied unchanged to SetB and SetC; ties use the higher threshold.Source: 2) Design · p. 8

3) Results and Discussion:

Confidence filtering changes the precision–recall balance, while smaller classifier checkpoints retain different amounts of overall F1. The reported threshold control is coarse, and a 12B checkpoint retains most of the 31B checkpoint’s F1.

3) Results and Discussion:

  • At τ = 1.00, filtering raised combined F1 from .253 to .310 while removing 196 false positives and 18 true positives.On the combined Settest curve, precision increased from .187 at τ = 0 to .338 at τ = 1, while recall changed from .392 to .287.Source: 3) Results and Discussion · p. 8, Figure 3 · p. 9
  • Confidence scores were concentrated at .90, .95, and 1.00, yielding a few distinct link sets rather than a continuous trade-off.The curves were nearly flat below τ = .90.Source: 3) Results and Discussion · p. 9, Figure 3 · p. 9

1) Motivation:

  • Smaller checkpoints were considered as an option for deployment under tighter local hardware limits.The 12B checkpoint had the highest overall F1 after 31B; E2B retained 58.1% of 31B F1 but gave the highest precision.Source: 1) Motivation · p. 9, 3) Results and Discussion · p. 9

2) Design:

  • The checkpoint comparison changed only the classifier checkpoint while keeping the pipeline settings and τ = 1.00 fixed.The threshold was selected for the primary 31B model on Setvalid and was not separately tuned for smaller checkpoints.Source: 2) Design · p. 9

3) Results and Discussion:

  • The 31B checkpoint achieved combined F1 = .310, while the 12B checkpoint retained 94.2% of that F1 at 38.7% of its nominal scale.The 31B checkpoint produced 37 fewer false positives than 12B, a reduction of 27.8%.Source: 3) Results and Discussion · p. 9, Figure 4 · p. 9

V. DISCUSSION

The discussion highlights cross-modal task coverage and retrieval limits, describes how confidence thresholds support different tracing priorities, and reports that threshold effects vary across requirement sets and VLM checkpoints.

A. Industrial Implications

  • 64.3% of the test oracle is image-involving, so text-only execution leaves that portion for manual checking; the final pipeline recovers links in both cross-modal directions.Source: A. Industrial Implications · p. 9
  • Only 8 of 60 image–text gold links enter candidate sets, and retaining 7 yields final image–text recall of .117.The classifier cannot recover gold links absent from the candidate set.Source: A. Industrial Implications · p. 10

B. Outputs for Different Tracing Priorities

  • Thresholding yields smaller, higher-precision link sets for review-burden priorities; F0.5, F1, and F2 offer selection points, but activity-specific choice needs practitioner evaluation.Source: B. Outputs for Different Tracing… · p. 10

C. Differences between Requirement Sets

D. VLM Selection

  • The 12B checkpoint retains similar overall link quality to 31B, while E2B has highest precision despite lower recall, supporting deployment under tighter hardware limits.The results also provide a basis for studying lower-latency and interactive TLR.Source: D. VLM Selection · p. 10

VI. THREATS TO VALIDITY

The conclusion reports industrial link-recovery results and checkpoint trade-offs, while the validity discussion qualifies the evidence through construct, internal, conclusion, and external validity constraints.

A. Construct Validity

  • The confidence signal is self-reported, clustered near 1.0, and not a calibrated probability; semi-manual conversion may also omit source layout.Source: A. Construct Validity · p. 10

B. Internal Validity

  • Single-practitioner labels preclude inter-annotator agreement; validation-set threshold selection may favor Setvalid, and the approach comparison changes multiple pipeline components.The selected candidate allocation may not suit other modality distributions, and its effect is not isolated from the larger candidate budget.Source: B. Internal Validity · p. 10

C. Conclusion Validity

  • Each configuration ran once; checkpoint comparisons do not isolate parameter count, and a threshold chosen for the primary VLM may change rankings.Source: C. Conclusion Validity · p. 10

D. External Validity

  • Industrial evidence covers three sets from different parts of one system in one organization; results may vary across organizations, languages, structures, granularity, image types, and model families.Source: D. External Validity · p. 10

VII. CONCLUSION

  • 49 of 171 industrial test links are recovered (precision .338, recall .287, F1 .310), versus baseline F1 0.114; links span all four modality combinations.Source: §VII · p. 10
  • 31B gives highest overall F1, 12B retains much performance, and E2B has highest precision; image-query text retrieval and practitioner evaluation remain needed.Source: §VII · p. 10
Loading 2610.34503v1…