SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System

1. Introduction

• Existing MLLMs have limited ability to use external resources and tools, while agentic systems struggle with rigorous scientific reasoning and workflow self-optimization.The paper also identifies hallucinations and unstable LLM capabilities as contributing problems.Source: Educational E-booklet Generation · pp. 1–2

Educational E-booklet Generation

• SciEducator applies the Deming Cycle through iterative workflow optimization, repeatedly planning, executing, evaluating, and refining workflows to converge toward high-confidence, accurate interpretations and reliable educational content for complex scientific activities.Source: Educational E-booklet Generation · p. 2
• Its understanding stage retrieves domain knowledge, evaluates candidate workflows, and uses execution feedback to decide whether to finalize results or refine the solution pool.The educating stage retrieves procedures, safety precautions, equipment images, and shopping links to create instructions, visual guides, audio narration, and interactive e-booklets.Source: Educational E-booklet Generation · p. 2
• SciEducator consistently outperforms leading commercial MLLMs and state-of-the-art multi-agent systems in both understanding and educating.The comparison is reported on SciVBench using metrics for both capabilities.Source: Educational E-booklet Generation · p. 2
• SciVBench contains 500 expert-validated, literature-grounded QA pairs across physics experiments, chemistry experiments, and daily-life phenomena, categorized into five question types.The types are terminology, principle, prediction, reading, and design.Source: Educational E-booklet Generation · p. 2

2.1. Multimodal Large Language Models

• Video understanding includes temporal-aware VTimeLLM [10], TimeSuite [28], and Slowfast-LLava [25], alongside widely adopted closed-source GPT-4o [15], Gemini, and Claude.These systems are limited in leveraging external resources and integrating specialized tools, which curtails their practical applicability.Source: §2.1 · p. 2

2.2. Multi-Agent Systems

• VideoAgent [23], memory-enhanced MAS [7], and VideoAgent2 [31] explore interactive planning, spatiotemporal memory, and uncertain reasoning, yet lack viable initial plans and systematic self-evolution for scientific tasks.They also face LLM hallucinations and instability, while scientific videos require external expertise and rigorous step-by-step reasoning.Source: §2.2 · p. 3

3.1. Problem Definition and Overview

• SciEducator seeks a self-consistent answer from a query and scientific video through a planner- and evaluator-led PDSA loop, with optional educational booklet generation.The cycle retrieves knowledge and builds a solution pool, assesses candidate solutions, investigates insufficient answers, and updates the pool before another iteration.Source: §3.1 · p. 3

3.2. Tools & Agents Configuration

• The system comprises 16 specialized components—10 agents and six tools—split between dynamically invoked and fixed-execution components.Dynamic components support planning, acquisition, evaluation, retrieval, and safety prompting; fixed components handle knowledge-base maintenance, multimodal synthesis, and booklet generation.Source: §3.2 · p. 3

3.3. Deming Cycle within SciEducator

• In each video-understanding cycle, SciEducator builds candidate plans from video captions and retrieved knowledge, then evaluates and executes the best trade-off before checking answer confidence.The plan stage samples video at one frame per second and retrieves knowledge once in the first cycle; the do stage combines objective and LLM-based criteria, using 20 randomized probe calls per tool or agent to estimate latency, token use, and success probability, plus IDF-based relevance and a subjective quality judgment.Source: §3.3 · pp. 3–4
• When confidence is low, SciEducator diagnoses answer failures, adds useful evidence to its knowledge base, and replans before repeating the confidence check.Replanning can adjust plan structure, granularity, or query specificity—for example, by increasing captioning frame rate for missed actions—and continues until confidence reaches a threshold or the chosen cycle limit.Source: §3.3 · pp. 4–5

3.4. Educational E-booklet Generation

• When requested, SciEducator generates a child-friendly e-booklet with experiment guidance, equipment links, illustrated procedures, safety precautions with audio prompts, and a summary of principles.Entity Recognition, Procedure Search, and Safety Alert agents assemble content; equipment search, illustration, and speech tools add modalities before an agent integrates them into a booklet. The paper states that, compared with a single-LLM baseline, this design offers richer modalities, higher relevance and instructional quality, and improved engagement.Source: §3.4 · p. 5

4.1. SciVBench Dataset

• SciVBench contains 500 expert-authored and validated, scientifically grounded QA pairs across physics, chemistry, and daily-life phenomena; answers require visual comprehension and coherent scientific reasoning.The pairs are distributed as 160 physics, 148 chemistry, and 192 daily-life items; videos are evaluated without subtitles or audio narrations.Source: §4.1 · p. 5

4.2. Evaluation Metrics

• Understanding evaluation uses Qwen3-Max-scored relevance and accuracy; education evaluation compares shared textual outputs using four Qwen-VL-Plus-rated criteria on 40 videos.Education criteria are relevance, instructional quality, attractiveness, and educational value.Source: §4.2 · pp. 5–7

4.3. Experimental Results

• SciEducator surpasses all baselines in relevance and accuracy across the physics, chemistry, and daily-life tracks, and shows a clear advantage over MLLMs in educational-content generation.Source: Table 1 · p. 7, §4.3 · p. 7, Table 2 · p. 7
• Qualitatively, search tools and planning/replanning yield more comprehensive, organized answers, while educational materials combine text, images, hyperlinks, and audio in structured, visually guided e-learning content.Source: Figure 4 · p. 6, §4.3 · p. 7, Figure 5 · p. 8

4.4. Ablation Studies

• Increasing PDSA maximum rounds from 1 to 5 improves understanding; education variants with 1, 3, and 5 cycles improve relevance and educational value, supporting higher-quality guidance images.Source: §4.4 · p. 7, Table 3 · p. 7, Figure 6 · p. 8
• Evaluator Agent elements increase the likelihood of selecting superior solutions, reduce resource use, and improve final accuracy within the five-round limit.Complete Evaluator Agent time and token consumption are normalized to 1.00.Source: §4.4 · pp. 7–8, Table 4 · p. 8
• New knowledge (Knew) and failure analysis (F) improve the Study Stage’s solution pool and thereby enhance both relevance and accuracy.Removing both leaves no feedback, so updates only discard executed solutions.Source: Table 5 · p. 8, §4.4 · p. 8

5. Conclusion

• SciEducator is presented as the first iterative self-evolving multi-agent system for scientific video comprehension and education.It reformulates the Deming Cycle’s Plan–Do–Study–Act philosophy into a self-evolving reasoning and feedback mechanism.Source: §5 · pp. 8–9
• Extensive experiments show SciEducator significantly outperforms current popular models.The authors construct SciVBench to ensure comprehensive performance assessment.Source: §5 · p. 9
• The authors expect SciEducator to inspire future multi-agent systems and broader domains.Its feedback mechanism progressively enhances interpretation of complex scientific activities to deliver accurate and reliable results.Source: §5 · p. 9

A. Tools & Agents

• The system integrates 10 agents and 6 tools, divided into dynamically invocable and fixed-execution components with defined input and output specifications.Source: A. Tools & Agents · p. 11

A.1. Dynamically Invocable Tools/Agents

• The Planner generates and adjusts solution pools, consolidates information, evaluates confidence, and produces final answers, while the Captioner describes video frames and the Evaluator selects solutions.The Planner and Evaluator use GPT-4o; the Captioner uses Gemini 2.0 Flash. The Evaluator considers subjective and objective metrics and initiates the selected solution's execution.Source: §A.1 · p. 11
• SciEducator’s Web Search organizes web information, Paper Search searches and summarizes academic papers, Procedure Search finds experimental procedures, Safety Alert searches for precautions, Entity Recognition identifies instruments and materials, and VideoSR enhances blurry frames.Web Search uses SegGPT, based on the Google Search engine; Paper Search and Entity Recognition use GPT-4o; Procedure Search and Safety Alert use searchGPT; VideoSR is based on an open-source model.Source: §A.1 · p. 11

A.2. Fixed-Execution Tools/Agents

• Fixed-execution tools archive knowledge texts and embedding vectors before DCAgent operation, while a GPT-4o RAG Agent retrieves and summarizes relevant scientific knowledge from video-derived keywords.The small knowledge base includes basic science and material properties; when it contains no relevant knowledge, the RAG Agent reports that absence.Source: §A.2 · p. 11
• The IDF Calculator computes knowledge-base inverse document frequency for solution keywords whenever the Evaluator assesses the solution pool.The values indicate keyword uniqueness and importance to video content.Source: §A.2 · p. 11
• The Equipment Search Tool returns an image and a most relevant purchase link for each experimental equipment item.Source: §A.2 · p. 11
• Fixed-execution content-generation tools create procedure illustrations and speech, while the E-booklet Agent combines experiment text, images, links, and audio in an HTML booklet.Illustrations use the Gemini 2.5 Flash Image (Nano Banana) API, speech uses KittenTTS, and the booklet agent is implemented in JavaScript/CSS.Source: §A.2 · p. 12

B. Details about Empirical Prior

• The empirical prior is estimated from 20 randomized probe calls per tool or agent, measuring average latency, token usage, and success probability.Token usage is converted into the corresponding platform’s API money cost.Source: B. Details about Empirical Prior · p. 12
• Probe results report latency, cost, and success rates for Web Search Agent, Paper Search Agent, Captioner Agent, VideoSR Tool, Procedure Search Agent, Entity Recognition Agent, and Safety Alert Agent.Web Search Agent averages 21.89s per call, costs $0.0139, and has an 88% success rate; Paper Search Agent averages 17.41s per call, costs $0.010, and has a 75% success rate. Captioner Agent takes about 25s at 1 FPS, costs $0.0001, and has a 93% success rate; VideoSR Tool takes approximately 53s at 1 FPS, costs $0.0001, and has a 100% success rate. Procedure Search Agent averages 28.17 seconds per call, costs $0.0178, and has an 87% success rate; Entity Recognition Agent averages 10.45 seconds per call, costs $0.0064, and has a 99% success rate; Safety Alert Agent averages 26.29 seconds per call, costs $0.0153, and has a 72% success rate.Source: B. Details about Empirical Prior · p. 12
• The Evaluator Agent uses t + 1000c as a reference, weighting resource consumption and feasibility equally to select the overall optimal solution.Here, t is time consumption and c is financial cost.Source: B. Details about Empirical Prior · p. 12

C.1. Understanding

• Relevance scores are 1 for answers entirely relevant to the question’s scientific subfield, 0.5 for partly irrelevant or broadly related answers, and 0 for irrelevant answers.Qwen3-Max assesses relevance against reference answers and scientific background, independently of correctness.Source: §C.1 · p. 12
• Accuracy scores are 0 for absolute errors or no covered key points, 0.5 for partial coverage without errors, and 1 for full key-point coverage without errors.Responses may use different wording if the meaning matches; additional correct content without errors can still receive full credit.Source: §C.1 · pp. 12–13

C.2. Educating

• Education responses are compared on relevance, instructional quality, attractiveness, and educational value, with illustrations included in SciEducator’s attractiveness assessment.Qwen-VL-Plus evaluates whether procedures and precautions fit the experiment, guide children clearly and safely, engage them, and support scientific interest and understanding.Source: §C.2 · p. 13

D.1. Plan Stage

• The Plan stage prompts agents to generate multiple specific, logically sequenced tool-call solutions that use video information and retrieved knowledge to clarify the phenomenon.Plans specify descriptions, tool sequences, and inputs; the prompt cautions against concluding the phenomenon before obtaining sufficient information.Source: §D.1 · p. 14

D.2. Do Stage

• The Do stage selects the best plan by weighing feasibility, relevance, completeness, coherence, keyword representativeness, and tool time, cost, and success probability.It combines time and financial cost as t + 1000c and gives resource consumption and feasibility equal weight.Source: §D.2 · pp. 14–15

D.3. Study & Act Stage

• The Study & Act stage diagnoses poor executions, summarizes existing results and new knowledge, and adjusts prior solutions or generates new plans.The prompt allows failed plans to be discarded without generating replacements every time.Source: §D.3 · p. 15

E. More Visualization Result

• Additional visualizations demonstrate SciEducator’s capabilities in scientific video understanding and educational content generation.Source: E. More Visualization Result · p. 15

• An iterative Plan-Do-Study-Act mechanism underpins SciEducator’s rigorous step-wise reasoning about scientific videos (Fig. 7).Source: §E.1 · p. 15
• By integrating external professional knowledge and analyzing failures, SciEducator refines its understanding rather than producing potentially hallucinated or superficial answers.Source: §E.1 · pp. 15–16
• This refined understanding establishes accurate scientific principles, such as light refraction or chemical reactions, as the foundation for subsequent educational materials.Source: §E.1 · p. 16

E.2. Educational E-booklet Generation

• The e-booklet organizes text, diagrams, and safety alerts into an engaging introduction, illustrated materials and procedures, safety notes, and a concise summary.Materials include pictures and shopping links, while the introduction uses accessible language to make scientific concepts engaging for learners.Source: §E.2 · p. 16

F.1. Knowledge Base

• The knowledge base covers middle- to high-school physics and chemistry fundamentals in 84 themed chapters, while excluding advanced knowledge and uncommon phenomena.Topics include Newton’s second law, electromagnetic induction, thermal expansion and contraction, metal combustion, and properties of gases in air.Source: §F.1 · p. 23
• The knowledge base serves as a foundational reference corpus for IDF calculation and provides basic scientific context to mitigate plan-stage language-model hallucinations.Source: §F.1 · p. 23
• SciEducator embeds each knowledge-base chapter with text-embedding-3-large for rapid vector retrieval in later runs without recomputing the document embeddings.Source: §F.1 · p. 23
• The knowledge-base approach is described as applicable beyond scientific video understanding to other fields.Source: §F.1 · p. 23

F.2. Average Cost Per Question

• With maximum PDSA rounds of 1, 3, and 5, average time is about 105s, 158s, and 206s, while money cost is $0.0542, $0.0783, and $0.1051, respectively.A fixed Captioner Agent call for an initial video description is required beforehand; API costs are calculated automatically by the calling platform.Source: §F.2 · p. 23

F.3. SciVBench Dataset

• SciVBench analyzes average question and answer character lengths separately for its physics, chemistry, and daily-life categories.Figures 10 and 11 present the question- and answer-length statistics, respectively.Source: §F.3 · p. 23, Figure 10 · p. 23, Figure 11 · p. 23
• SciVBench reports average video duration statistics for its physics, chemistry, and daily-life categories.Figure 9 presents the average-duration statistics by category.Source: Figure 9 · p. 23, §F.3 · p. 23