PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Language Model Agents Learn by Watching Tutorial Videos?

Haoyu Zhao, Zafeirios Fountas, Jun Wang

arXiv:2611.01460v1cs.LGcs.AIcs.CLcs.CV

TL;DR

LLM agents lack a mechanism to turn tutorial videos into persistent, reusable knowledge, while continual improvement through parameter updates is prohibitively expensive. TAL consolidates videos into a structured tutorialbook for frozen agents; on VidLearn, it achieves a maximum relative training-set gain of 112.72% over the vanilla GPT-5.2 baseline.

  • Problem

    LLM agents lack a mechanism to convert abundant tutorial videos into persistent, reusable knowledge, while continual improvement through parameter updates is prohibitively expensive.

    Source: §1 · pp. 1–2

  • Method

    TAL consolidates tutorial videos into a structured tutorialbook through adversarial question harvesting and memory refinement, then uses TOC-based retrieval for frozen LLM agents.

    Source: §1 · pp. 2–3

  • Results

    112.72% maximum relative gain on VidLearn’s training set over the vanilla GPT-5.2 baseline; TAL also achieves the best performance on both splits and strongest scaling with tutorial count.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    Across VidLearn and a software-tutorial evaluation, the results demonstrate the potential of instructional videos as a scalable resource for continual learning without parameter updates.

    Source: §5 · p. 13

    The setting assumes a frozen solver whose parameters remain unchanged throughout the learning stream.

    Source: §3 · p. 4

Abstract

from arXiv · show

Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduce Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that consolidates tutorial videos into a persistent, structured tutorialbook memory through a two-stage adversarial process that harvests hard, video-grounded questions and uses them to refine the memory. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model's reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL outperforms all evaluated baselines and improves monotonically as more videos are consumed, achieving up to 113% and 91% relative gains over the corresponding vanilla MLLMs on the training and test splits, respectively. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in 69% of head-to-head games. Evaluation in a second domain further demonstrates the generality of TAL. Together, these results highlight instructional videos as a scalable resource for parameter-update-free agent continual learning.

1. Introduction

The introduction proposes learning from tutorial videos as a way for frozen LLM agents to accumulate reusable knowledge without parameter updates. It presents TAL, its tutorialbook consolidation and TOC-based retrieval, and evaluates it on chess and software tutorials.

Introduction

  • Conventional parametric learning largely fixes LLM capabilities after training, and repeated fine-tuning is prohibitively expensive at scale.The introduction contrasts this with humans’ routine learning from books and tutorial videos.Source: §1 · p. 1
  • RAG and agent skills depend on human-curated knowledge, while case-based reasoning reuses past cases that may offer limited guidance in novel contexts.The introduction motivates tutorial videos as an internet-scale source of demonstrations and explanations that can be consolidated without agent-specific curation.Source: §1 · pp. 1–2
  • TAL consolidates hard, video-grounded questions into a structured tutorialbook through a two-stage process, then retrieves relevant chapters through a TOC tree guided by model reasoning.The process retains questions the frozen model cannot answer without context, validates the missing knowledge, and rewrites the corresponding chapters.Source: §1 · pp. 2–3
  • VidLearn is a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 classical openings.Chess combines structured knowledge, visual-language grounding, and multi-step reasoning.Source: §1 · p. 2
  • TAL beats fine-tuning-free baselines by up to 113% on VidLearn’s training split and 91% on its test split, with stronger opening-phase positions in 69% of head-to-head games.Performance scales monotonically with the number of consumed videos.Source: §1 · p. 3
  • In software tutorials, TAL reaches 49.7% held-out accuracy, 10.9 percentage points above the strongest baseline, and achieves the highest mean accuracy in all eight chess settings.The comparisons use stronger retrieval and memory baselines.Source: §1 · p. 3

2. Related Work

Prior work improves LLM agents through external-memory reuse of skills or experiences, while demonstration and video-understanding methods focus on imitation or inference-time comprehension. TAL instead turns human tutorial videos into structured external knowledge reusable across future tasks.

2.1. Continual Learning for LLM Agents

  • Skill-based methods store reusable workflows, while experience-based and reflection-based methods reuse prior cases or feedback; TAL instead consolidates human tutorials into a reusable knowledge substrate.Skill construction can require manual curation, while case-level reuse depends on similar past interactions and the model's ability to reason from retrieved cases.Source: §2.1 · p. 3

2.2. Learning from Demonstrations

  • Robotics demonstrations typically become policies or action sequences, while video retrieval and in-context methods target inference-time comprehension; TAL converts tutorials into structured knowledge for future tasks.Long-form video understanding becomes challenging as videos grow longer and denser; VideoRAG and Demo-ICL address this by retrieving relevant segments before generation.Source: §2.2 · pp. 3–4

3. Tutorial Agentic Learning

Tutorial Agentic Learning (TAL) learns from instructional videos without updating model parameters, consolidating tutorial knowledge into a persistent memory that supports later queries. It refines that memory through hard, video-grounded questions and retrieves relevant chapters through a reasoning-driven table of contents.

3. Tutorial Agentic Learning

  • TAL keeps the base solver frozen and acquires reusable knowledge from tutorial videos through a persistent tutorialbook memory.The learning substrate changes over the stream, but the solver parameters do not.Source: §3 · p. 4, §3.1 · p. 5, §3.2.3 · p. 7

3.1. Tutorialbook Memory as a Learning Substrate

  • Unlike parameter updates or episodic case replay, TAL consolidates multimodal tutorial content and supporting evidence into structured, reusable tutorialbook knowledge.The memory is organized as textbook-style chapters with retrieval keys and summaries.Source: §3.1 · p. 5

3.2. Two-Stage Adversarial Consolidation

  • TAL uses two-stage adversarial consolidation: it harvests hard, video-grounded questions, then turns failures on those questions into targeted memory edits.The stages use a read–write loop to refine the tutorial memory.Source: §3.2 · p. 5

3.2.1. Stage I: Video-Grounded Agent–Model Adversarial Consolidation

  • Stage I aligns video frames and subtitle windows to generate questions, retaining only those grounded in tutorial evidence and difficult for the frozen solver.Each question is tied to a timestamped teaching moment, with its reference answer derived from the sampled evidence.Source: §3.2.1 · p. 5

3.2.2. Stage II: Question–Memory Adversarial Consolidation

  • Stage II tests the current memory against the question bank and uses unanswered questions to target edits to relevant chapters.Candidate edits are accepted only when they strictly improve question-bank performance, ensuring monotonic improvement and preventing destabilization of previously correct knowledge.Source: §3.2.2 · p. 6

3.2.3. Learning in Tutorial Memory

  • TAL operationalizes an Information Bottleneck tradeoff [42] through a coupled process that harvests difficult, evidence-supported questions and edits memory to improve answers.Its LLM-based rewrite and score-based acceptance test constrain edits to chapters implicated by failures; both question-generation and tutorial memory are refined without gradient descent.Source: §3.2.3 · p. 7

3.3. Inference with Tutorial Memory

  • At inference, TAL conditions the frozen answer model on tutorial context selected from the refined memory using deterministic top-k retrieval.The retrieval distribution collapses to the selected context for answering.Source: §3.3 · p. 8

3.4. TOC Retrieval

  • TAL retrieves through a hierarchical table of contents, selecting chapters by having the frozen LLM reason over TOC entries rather than using embedding similarity.The TOC is organized into volumes, parts, and chapters, which are expanded into the retrieved context.Source: §3.4 · p. 8

4. Experiment

The experiments evaluate TAL on chess tutorials, stronger retrieval and memory baselines, practical chess play, and ablations of its learning and retrieval components. TAL achieves the strongest reported results across benchmark settings, and the ablations indicate that its gains depend on structured, visually grounded memory and retrieval choices.

4.1. Dataset

  • VidLearn contains 72 chess tutorials across 12 openings, with 48 training videos and 110 manually verified multimodal questions from 24 held-out videos.At test time, the agent answers from one video frame, a natural-language query, and multiple-choice options.Source: §4.1 · p. 8

4.2. Experimental Setup

  • The study compares four prompting and retrieval methods across four model families, then benchmarks TAL against stronger retrieval and memory systems using Gemini-3.5-Flash.The matched-backbone comparison uses temperature 0.7 and includes Transcript-RAG, two VideoRAG variants, mem0, and ReasoningBank.Source: §4.2 · p. 8

4.3. Main Results

  • TAL achieves the best performance on both VidLearn splits, with maximum relative gains of 112.72% on GPT-5.2 training and 91.42% on Kimi K2.5 test.With 48 videos, TAL reaches 84.1% on training questions and 71.8% on held-out questions, versus 64.6% and 64.5% for the strongest respective baselines.Source: Table 1 · p. 9, §4.3 · pp. 9–10, Table 2 · p. 11
  • TAL’s accuracy improves as videos accumulate, and its 105,703-token memory is 82.32% smaller than the 598,027-token raw subtitle corpus.The GPT-5.2-built tutorial memory also transfers to other answer models; on Photoshop tutorials, 16 videos yield 49.7% held-out accuracy, 10.9 points above the strongest baseline.Source: Figure 3 · p. 10, §4.3 · pp. 10–11, Table 3 · p. 11

4.4. Head-to-Head Evaluation

  • Across all tested move horizons, TAL achieves higher win rates and lower Average CentiPawn Loss than vanilla GPT-5.2 in head-to-head chess games.The evaluation covers all 12 VidLearn openings, with fixed opening positions and 20 color-swapped games per opening.Source: §4.4 · pp. 11–12, Table 4 · p. 12

4.5.1. Effect of Two-Stage Consolidation

  • Full TAL reaches 71.82%, exceeding the 63.63% additive prediction from the Stage I-only and Stage II-only variants.Stage I only scores 55.45% versus vanilla’s 48.18%, while Stage II only scores 56.36%, suggesting the stages interact synergistically.Source: §4.5.1 · p. 12

4.5.2. Effect of Retrieval Strategy

  • With the tutorialbook and base model fixed, TOC retrieval achieves 50.00% Recall@3 versus 28.18% for dense retrieval using OpenAI’s text-embedding-3-small, a 21.82-point improvement.Source: §4.5.2 · p. 12

4.5.3. Effect of Visual Information

  • With 48 videos, full TAL reaches 71.8% on held-out questions, versus 62.1% for transcript-only TAL.The matched comparison isolates visual evidence: full TAL constructs memory from event-aligned frames and transcripts, while transcript-only TAL uses subtitles alone.Source: Table 7 · p. 13, §4.5.3 · p. 13

5. Conclusion

Tutorial Agentic Learning (TAL) lets frozen LLM agents consolidate instructional videos into a structured tutorialbook memory accessed through TOC retrieval. It outperforms representative fine-tuning-free baselines across VidLearn and a second software-tutorial evaluation, with performance scaling as more tutorials are consumed.

5. Conclusion

  • TAL enables frozen LLM agents to consolidate instructional videos into a structured tutorialbook memory accessed through TOC retrieval.The paper presents TAL as a nonparametric continual-learning paradigm.Source: §5 · p. 13
  • TAL outperforms representative fine-tuning-free baselines across VidLearn and a second software-tutorial evaluation, and scales monotonically with the number of tutorials consumed.The results support instructional videos as a scalable resource for continual learning without parameter updates.Source: §5 · p. 13

A. Implementation and Evaluation Details … A.3. Judge Behavior and Verification

The implementation pairs transcript- and frame-based retrieval baselines with TAL’s reflective question generation, chapter editing, and TOC retrieval. A frozen judge verifies answer grounding, while exact-match evaluation governs reported metrics and edit acceptance.

A. Implementation and Evaluation Details; A.1. Baseline and Experimental Configuration

  • The transcript RAG baseline retrieves five chunks from overlapping 220-token subtitle segments, while Video-RAG variants pair event-frame captions with local transcripts.The transcript baseline uses BM25 and text-embedding-3-small, fuses 30 candidates from each retriever with RRF, and supplies the top five chunks; the frame-aware variant also supplies frames.Source: §A.1 · p. 17

A.2. TAL Operations and Prompts

  • Across rounds, Gϕ uses its external history to replace correctly answered questions with harder, nonduplicate questions and retain incorrect answers as knowledge gaps.Hard rewrites target the same event or frame and favor plausible distractors, tactical refutations, move-order traps, or subtle plan choices.Source: §A.2 · p. 17, §P.1 · p. 17

P.1 Reflective revision of the question generator

  • The tutorialbook editor routes failed questions to implicated chapters, adds or corrects missing knowledge, and commits revisions only when exact-match QA-bank accuracy strictly improves.Otherwise, it restores the previous chapter.Source: §P.1 · p. 17

P.2 Stage II tutorialbook editor (Write operator)

  • 50.0% Recall@3 for TOC retrieval exceeded 28.2% for dense retrieval, with the frozen LLM selecting up to three chapters from a compact tree index.TAL expands only validated chapter IDs into the answer context; each added video increases the TOC by about 240 tokens.Source: §P.2 · p. 18, §P.3 · p. 18

P.3 TOC retrieval

  • The TOC retrieval prompt asks the frozen LLM to return JSON chapter IDs and confidence using only the supplied index, question, and options.Source: §P.3 · p. 18

A.3. Judge Behavior and Verification

  • A manual audit found 100% agreement between the frozen grounding judge and human checks across 50 decisions, and two people independently checked all 110 held-out questions.The judge gates Stage I answer grounding but does not score reported metrics; exact MCQ matching determines accuracy, difficulty filtering, and Stage II edit acceptance.Source: §A.3 · p. 18

A.4. Construction Cost

Tutorialbook construction is a one-time cost, and the resulting memory can be reused across different answer models without reconstruction.

A.4. Construction Cost

  • $0.38 per video ($0.12 input and $0.26 output), or $0.03 per minute, is the measured Gemini-3-Flash construction cost.Construction averaged 245k input tokens and 88k output tokens over 102 calls per video, for videos averaging 12.9 minutes.Source: §A.4 · p. 18

A.5. Robustness to Noisy or Irrelevant Videos

TAL’s test accuracy improves as automatically collected video budgets grow, while Transcript-RAG’s fluctuates; the results suggest TAL tolerates naturally occurring variation in source quality and relevance.

A.5. Robustness to Noisy or Irrelevant Videos

  • Across 12/24/36/48-video budgets, TAL test accuracy rises 48.2%/57.3%/61.8%/64.6%, while Transcript-RAG fluctuates 42.7%/38.2%/43.6%/37.3%.Videos were automatically selected from top-k highly viewed matches to topic-keyword searches; robustness to controlled corruptions remains future work.Source: §A.5 · p. 19

B. VidLearn Details

VidLearn comprises 72 instructional videos across 12 classical chess openings, split into training and held-out test sets. The release provides per-question metadata and reproduction code, but not the raw videos.

B. VidLearn Details

  • 72 videos cover 12 classical openings, with 4 training and 2 held-out test videos per opening, totaling approximately 22.5 hours and 724K subtitle words.The training set has 48 videos (∼14.9 hours; ∼462K words), and the test set has 24 videos (∼7.7 hours; ∼262K words).Source: B. VidLearn Details · p. 19, Table 9 · p. 19
  • The release includes per-question metadata and question-generation and validation code for reproduction, but does not redistribute raw video files.The metadata includes video ID, timestamp, question, options, answer index, and verified rationale.Source: B. VidLearn Details · p. 19

C. Human Performance

Both human and TAL performance improve with more tutorial videos on held-out questions, although TAL remains below the human reference. The human evaluation is a small-scale qualitative comparison, not a representative estimate of human performance.

C. Human Performance

  • Humans answered held-out questions after watching 12, 24, 36, or 48 training videos in separate sessions under closed-book conditions using an interface matching the model’s test instances.The evaluation disallowed engines, books, and web search; it was designed to check for a similar learning trend, not estimate representative human performance.Source: C. Human Performance · p. 19, Figure 4 · p. 20
  • Both human and TAL (avg) curves rise with more demonstration videos, while TAL accuracy remains below the human reference and scales monotonically in the tested regime.The comparison uses the held-out test set.Source: Figure 5 · p. 20, C. Human Performance · p. 21
  • Humans show strong consolidation efficiency, often learning an opening’s shared logic from only a handful of tutorials—or a single high-quality demonstration.The tutorials may present the same opening in stylistically and pedagogically diverse ways.Source: C. Human Performance · p. 21

D. Train–Test Similarity Analysis · E. Online Self-Evolution

The analysis finds no near-duplicate overlap between test questions and training material, while TAL can expand its memory online by discovering videos for topics missing from its TOC. In VidLearn, it builds the training corpus for all 12 openings without a pre-curated corpus or human-filtered video list.

D. Train–Test Similarity Analysis

  • The leakage analysis compares 110 held-out test questions with 174 training questions and 2,564 deduplicated subtitle chunks from 48 training videos.Subtitle chunks use 220 tokens with a 50-token overlap.Source: D. Train–Test Similarity Analysis · p. 21
  • For each test question, the analysis reports maximum similarity to either reference pool using four metrics, including TF-IDF and 128-dimensional LSA cosine similarity.It also reports mean±SD, bootstrap 95% confidence intervals, median, P90, and the proportion of scores ≥0.8.Source: D. Train–Test Similarity Analysis · p. 21
  • No test question has a near-duplicate match (similarity ≥0.8) to training questions or subtitle chunks; lexical overlap is low, while LSA shows moderate topic relatedness.The splits draw from the same 12 openings, but no near-duplicate overlap appears under any metric or reference pool.Source: Table 10 · p. 21, D. Train–Test Similarity Analysis · p. 21

E. Online Self-Evolution

  • When a target topic has no matching TOC entry, TAL treats it as a knowledge gap, generates search queries for instructional videos, and updates memory without parameter changes.The loop is keyword → gap detection → video discovery → memory update; retrieved videos undergo the same Stage I and Stage II refinement used for initial construction.Source: E. Online Self-Evolution · p. 22
  • TAL’s online-learning mode lets its memory grow without parameter updates or manually curated sources, with potential application to broader, faster-changing domains.The paper frames this as an opportunity for deployment beyond the VidLearn experiment.Source: E. Online Self-Evolution · p. 22
  • Given only the names of 12 openings, TAL constructs its entire VidLearn training corpus through TOC-based gap detection, without a pre-curated corpus or human-filtered video list.This end-to-end experiment demonstrates that TAL can extend its knowledge base in open environments.Source: E. Online Self-Evolution · p. 22

Related papers

Loading 2611.01460v1…