PaperlayerView exact revision on arXiv

Source-linked AI summary

Intelligent Semantic Matching (ISM) for Video Tutorial Search using Transformer Models

Ahmad J. Tayeb, Sonia Haiduc

arXiv:2609.12921v1cs.SE

TL;DR

Programming video search is challenged by growing tutorial collections and keyword methods that miss semantic meaning and user intent. ISM uses Transformer-based transcript embeddings, chunk re-ranking, and GPT-4 summaries; it outperformed TechTube in retrieval and fragment identification, while users preferred its semantic matching and summaries.

  • Problem

    Growing programming-video collections and keyword-based search methods leave semantic meaning, user intent, and context insufficiently captured.

  • Method

    ISM segments video transcripts, encodes chunks and queries with SBERT-based Transformer embeddings, re-ranks retrieved chunks, and generates GPT-4 summaries.

  • Results

    ISM achieved Hit@5 0.95 versus TechTube’s 0.58 and F1 0.70 versus 0.52, while users preferred ISM and its GPT-4 summaries.

  • Takeaways & Limitations

    ISM offers more semantically aligned video and fragment retrieval and user-preferred summaries for programming tutorials.

  • Takeaways & Limitations

    The approach may miss important nonadjacent video segments because it currently identifies one continuous segment.

Abstract

from arXiv · show

The rise in the number and diversity of available software development video tutorials has enhanced digital learning for developers but also introduced challenges in locating relevant content efficiently. Existing video search methods, including keyword-based approaches and tools like CodeTube and TechTube, rely primarily on retrieval algorithms such as BM25, which fail to capture the semantic nuances and user intentions behind search queries. To address these limitations, we introduce ISM, an approach that uses SBERT to generate semantically rich vectors from video tutorial transcripts to improve the search for programming video tutorials. By segmenting transcripts and implementing a re-ranking process, ISM effectively preserves context and enhances the relevance of search results. Additionally, ISM generates informative video summaries using GPT-4, allowing developers to quickly assess the relevance of video content. To evaluate our approach, we first performed a quantitative study comparing ISM with the baseline TechTube. The results revealed that ISM performs better in both video retrieval and fragment identification, achieving a Hit@5 score of 0.95 and an average F1 score of 0.70 compared to the baseline's 0.58 and 0.52, respectively. We also performed a user study, which revealed that users strongly preferred the semantic matching capabilities and AI-generated summaries of our approach. This work advances the state-of-the-art in programming video tutorial search and summarization by offering more nuanced and user-aligned retrieval and summarization mechanisms.

I. INTRODUCTION

Programming video tutorials are increasingly valuable but difficult to search with keyword-based methods that miss semantic meaning and user intent. ISM addresses this gap with Transformer-based transcript embeddings, re-ranking, and GPT-4 summaries, and is evaluated against TechTube through quantitative and user studies.

  • I. INTRODUCTION: The work targets a growing discovery problem caused by the expanding volume of programming tutorials and the limitations of BM25-style retrieval.Existing approaches can miss unusual terminology, broader context, and nuanced needs in complex queries.
  • I. INTRODUCTION: ISM segments transcripts and re-ranks relevant chunks to preserve continuity when multiple chunks from one video match a query.Segmenting addresses Transformer input limits, while re-ranking helps retain broader context across relevant chunks.
  • I. INTRODUCTION: 0.95 Hit@5 versus 0.58 for TechTube, while fragment identification reached an F1 score of 0.70 versus 0.52.These results come from evaluation on the TechTube dataset.
  • I. INTRODUCTION: Users preferred ISM’s semantic matching and GPT-4-generated summaries, which they considered accurate and informative.The user study compared ISM with TechTube and compared GPT-4 summaries with creator-written descriptions.
  • I. INTRODUCTION: ISM uses Transformer-based models to create semantically dense transcript vectors, addressing keyword search engines’ limited semantic understanding.The approach also uses SBERT embeddings and cosine similarity to encode documents and queries separately.

III. METHODOLOGY

ISM prepares tutorial metadata and transcripts for semantic retrieval by segmenting transcripts into model-compatible chunks, embedding them with SBERT, and indexing the resulting vectors.

  • III. METHODOLOGY: ISM’s indexing pipeline downloads video metadata and transcripts before structuring them for search and retrieval.Metadata includes video IDs, titles, and descriptions.
  • III. METHODOLOGY: 80-word transcript chunks yielded the best retrieval performance in empirical testing and were embedded with the multiqa-distilbert-cos-v1 SBERT model.Chunking accommodates Transformer input-length constraints while representing each segment semantically.
  • III. METHODOLOGY: The resulting chunk embeddings and metadata are indexed in Qdrant, enabling scalable multidimensional vector search.Qdrant supports efficient retrieval of relevant video chunks during query processing.

B. Video Retrieval

Video Retrieval embeds user queries, retrieves transcript chunks by cosine similarity, aggregates chunks by video, and re-ranks videos to balance relevance with coverage.

  • B. Video Retrieval: ISM retrieves the top-k transcript chunks by comparing the query embedding with chunk embeddings using cosine similarity.The same SBERT model generates the query embedding used for retrieval.
  • B. Video Retrieval: The system aggregates retrieved chunks from each video and re-ranks videos using chunk relevance scores or ranks.This reduces the impact of videos with excessive numbers of retrieved chunks while preserving relevant content.
  • B. Video Retrieval: For most strategies, ranking considers up to nmax top chunks per video, selected by similarity score.The subset is defined from the total retrieved chunks for each video.
  • B. Video Retrieval: MaxSim uses only each video’s single highest-similarity chunk, whereas AvgRank, AvgScore, WeightRank, and WeightScore combine multiple chunk rankings or scores.Weighted strategies give earlier retrieved chunks greater influence.
  • B. Video Retrieval: Weighted Score Adjustment provided the best overall results by prioritizing highly relevant chunks without letting videos with many chunks dominate rankings.Earlier retrieved chunks receive greater influence in the weighted scoring strategy.

C. Result Display

The Result Display component presents final search results through relevant-segment highlighting and video-summary generation.

  • C. Result Display: Result Display contains two submodules: Relevant Segment Highlighting and Video Summary Generation.Together they provide the user-facing presentation of retrieved content.

1) Relevant Segment Highlighting:

ISM identifies relevant transcript segments by thresholding semantic similarity, grouping consecutive qualifying chunks, and presenting contextually expanded results.

  • 1) Relevant Segment Highlighting:: ISM computes cosine similarity between the query embedding and 80-word chunk embeddings using the all-mpnetbase-v2 SBERT model.The segment-retrieval configuration uses 80-word chunks to remain within token limits while preserving local context.
  • 1) Relevant Segment Highlighting:: Similarity thresholding retains transcript chunks whose scores reach a percentage α of the video’s maximum similarity.The example threshold is 45%; C′_i denotes retained chunks and Simmax the maximum chunk similarity.
  • 1) Relevant Segment Highlighting:: The system groups adjacent qualifying chunks into sequences and prioritizes the two longest sequences as likely contextually complete segments.Longer uninterrupted sequences are treated as more likely to capture complete relevant content.
  • 1) Relevant Segment Highlighting:: Selected segment boundaries are expanded by subtracting seconds from the start and adding seconds to the end to preserve contextual information.The adjusted timestamps aim to capture the full scope of relevant content.
  • 1) Relevant Segment Highlighting:: The final output presents the most relevant video segments with adjusted timestamps for efficient access to contextually complete content.This allows users to reach relevant portions without watching the entire video.

2) Video Summary Generation:

ISM uses GPT-4 to summarize transcript segments and, when necessary, combine those summaries into one coherent video summary for user-facing display.

  • 2) Video Summary Generation:: GPT-4 generates concise natural-language summaries for transcript segments, limiting each segment summary to 150 words.Transcript splitting accommodates GPT-4 input-length constraints and supports readable summaries.
  • 2) Video Summary Generation:: When a transcript produces multiple segment summaries, GPT-4 condenses them into a single coherent summary of the entire video.The second prompting stage combines the segment-level summaries.
  • 2) Video Summary Generation:: The Result Display component presents re-ranked videos, relevant segments, and GPT-4-generated summaries in one user-facing interface.This combination helps users assess tutorial content and identify relevant videos quickly.
  • 2) Video Summary Generation:: ISM is evaluated against TechTube, which retrieves videos and pertinent segments using query expansion and transcript-based similarity.TechTube’s pipeline uses pseudo-relevance feedback for queries and silence-based segmentation for video fragments.

B. Dataset

The evaluation uses the TechTube benchmark of programming-related queries and videos, with exclusions reducing the available samples for each research question.

  • B. Dataset: The dataset contains 98 natural-language queries covering repository maintenance and Java and Python programming tasks.Queries came from a benchmark and highly rated Stack Overflow threads.
  • B. Dataset: The source collection included 400 English-audio YouTube videos longer than three minutes, from which 98 query-matching videos were selected.The selected videos corresponded to the benchmark queries.
  • B. Dataset: The replication package supplied initial and reformulated queries for RQ1 and ground-truth relevant fragments for RQ2.These materials supported comparisons between ISM and TechTube on the same benchmark.
  • B. Dataset: Twelve unavailable videos reduced the dataset to 86 videos, while exclusions left 85 queries for RQ1 and 84 for RQ2.RQ1 omitted one query without a reformulation; RQ2 omitted two queries because of TechTube-package processing issues.
  • B. Dataset: Video retrieval is evaluated with Hit@k and MRR, whereas fragment retrieval uses Precision, Recall, and F1-Score.Hit@k checks whether a relevant video appears among the top k results, and MRR captures how early the first relevant video appears.

D. RQ1: Video Retrieval Performance

Across video-retrieval strategies, ISM outperforms Lucene and TechTube on Hit@k and MRR, while its segment retrieval shows especially strong recall and F1 improvements.

  • D. RQ1: Video Retrieval Performance: ISM’s retrieval evaluation compares AvgRank, AvgScore, MaxSim, WeightRank, and WeightScore against Lucene and TechTube baselines.Lucene uses original queries, while TechTube uses reformulated queries.
  • D. RQ1: Video Retrieval Performance: At k = 5, ISM reached Hit@5 of 0.98 across several strategies, compared with TechTube’s 0.82, and all strategies reached Hit@10 of 0.98.WeightRank also achieved the highest Hit@1 of 0.80.
  • D. RQ1: Video Retrieval Performance: Semantic transcript representations, chunk aggregation, and re-ranking support ISM’s reported gains over Lucene and TechTube.The paper attributes improved contextual prioritization to dense vectors and strategies such as Weighted Score Adjustment.
  • D. RQ1: Video Retrieval Performance: WeightRank achieved MRR of 0.87 for the top 2 videos, compared with TechTube’s 0.59, indicating earlier ranking of relevant videos.ISM consistently outperformed both baselines across the reported k values.
  • E. RQ2: Video Segment Retrieval Performance: For segment retrieval, ISM achieved Average Precision of 0.69 and Average Recall of 0.82, compared with TechTube’s 0.67 and 0.55.The recall difference indicates that ISM captures more relevant segments.
  • E. RQ2: Video Segment Retrieval Performance: ISM achieved F1-Score of 0.70 versus TechTube’s 0.52, with the difference statistically significant at p-value = 1.39e-05.The reported improvement follows ISM’s higher recall in segment retrieval.

V. USER STUDY

The user study compared ISM with TechTube for video search and evaluated preferences and perceived quality for creator-written versus GPT-4-generated summaries.

  • Study Design: Participants used a web application across three phases to compare ISM with TechTube, choose between summary types, and rate GPT-4 summary quality.Search results and summaries were randomized to reduce positional bias, and participants provided reasons for their choices.
  • Phase 1: Video Search Evaluation: Phase 1 asked participants to formulate queries from short Python tasks, compare side-by-side ISM and TechTube results, and rate which engine was more helpful on a 1–4 scale.Participants also marked individual videos they considered relevant.
  • Phase 2: Textual Summary Preference Evaluation: Phase 2 compared creator descriptions with GPT-4-generated summaries, asking participants to rate how well each described video content and select their preferred summary.Participants watched videos before rating the summaries, and summary placement was randomized with sources hidden.
  • Phase 3: Quality of GPT-4-Generated Summaries: Phase 3 asked participants to rate GPT-4 summaries for accuracy and completeness on a 1–4 scale and optionally provide written feedback.Accuracy concerned correctness, whereas completeness concerned coverage of key information.
  • Study Materials: The study used 32 Python exercises collected from GeeksforGeeks to provide varied programming tasks and retrieve relevant YouTube videos.Python was selected because students and novices were a primary target population for tutorial use.

B. Participants

The study included 91 participants, mainly undergraduate computer science students, who compared ISM and TechTube through Python search tasks and relevance judgments.

  • Participants: The 91 participants were predominantly undergraduate computer science students, with 57% reporting less than one year of Python experience and 93% identifying as undergraduates.The sample also included participants with no Python experience and a smaller group of software developers.
  • Phase 1 Results: 64% of 179 searches favored ISM results, compared with 20% favoring TechTube, while ISM received higher average relevance ratings of 3.42 versus 2.54 out of 4.ISM also had a higher median and tighter interquartile range in Figure 5, indicating more consistent relevance ratings.
  • Search Efficiency: 76% of participants completed their tasks with one query, while 14% needed two and 6% needed three queries.The remaining participants required between four and twelve queries, with 2% requiring more than five.
  • Relevant-Result Analysis: Participants actively marked at least one relevant video in 171 queries after 91 of 179 queries were excluded because neither list received a relevance mark.The filtering reflected the optional nature of relevance marking or cases where participants found no relevant results.
  • Relevant-Result Analysis: ISM’s average marked-result relevance rate was 0.587, compared with 0.27 for TechTube, across 171 queries with at least one marked relevant video.The relevance rate was the proportion of results in each list marked relevant by participants.
  • Result Overlap: ISM and TechTube shared 209 videos across their result lists, representing 17.46% of ISM’s results and 16.18% of TechTube’s results.Despite this overlap, participant feedback showed clear differences in perceived relevance.

D. Results of Phase 2: Summary Preference Evaluation

Phase 2 compared creator descriptions with GPT-4-generated summaries and found stronger participant preference and usefulness ratings for the AI-generated summaries.

  • Preference and Usefulness: 84% of 270 evaluations favored GPT-4-generated summaries, compared with 11% favoring video creator descriptions.Five percent of evaluations found both or neither summary useful.
  • Preference and Usefulness: GPT-4-generated summaries received an average usefulness rating of 3.38 out of 4, compared with 1.8 out of 4 for creator descriptions.Figure 6 shows a higher median and tighter interquartile range for GPT-4 summaries.
  • User Feedback: Participants praised GPT-4 summaries for detailed, clear, readable, and organized overviews that captured videos’ key points and overall ideas.Many participants criticized creator descriptions for restating titles, resembling transcripts, or containing irrelevant links and advertisements.
  • User Feedback: Some participants preferred creator descriptions for conciseness, directness, and formatting, while others found GPT-4 summaries too long or wordy.A minority considered both types equally useful or found neither satisfactory.

E. Results of Phase 3: Quality of GPT-4 Summaries

GPT-4-generated summaries received strong accuracy and completeness ratings, helping users assess video relevance without watching the entire video. Feedback also identified missing details, code examples, and presentation improvements.

  • Participants rated summaries after watching or skimming random short videos and judging accuracy and completeness on scales from 1 to 4.
  • 3.39/4 accuracy and 3.08/4 completeness were the average ratings for GPT-4-generated summaries.These corresponded to 85% and 77%, respectively.
  • 25 of 69 respondents considered the summaries good, accurate, or complete, while 22 reported missing key points, insufficient detail, or inadequate explanations.
  • 6 respondents recommended adding code examples to improve clarity and practical value in programming-related summaries.
  • Feedback also requested better formatting, grammar, organization, and summary lengths matched to video depth.
Loading 2609.12921v1…