PaperlayerView exact revision on arXiv

Source-linked AI summary

SLATE: Are AI-Generated Slides Educationally Effective? A Benchmark for Language Teaching Quality and Learner Knowledge Acquisition

Jingzhuo Wu, Jiajun Zhang, Liu Yi, Leqi Zheng, Yuheng Jing, Xinyuan Zhou, Quan yang

arXiv:2609.06212v1cs.CL

TL;DR

Existing evaluations often assess the polish and structure of AI-generated teaching slides without measuring what learners acquire or transfer. SLATE benchmarks slide effectiveness using low-resource language puzzles and matched learning tests; it finds that pedagogical design is more strongly associated with learning outcomes than content validity, while transfer remains weaker at greater distance.

  • Problem

    Existing evaluations score slide surface properties without measuring learner knowledge acquisition, conceptual understanding, or transfer.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    SLATE turns low-resource linguistics puzzles into instructional units and evaluates generated decks with pretest, study, posttest, and matched near- and far-transfer tests.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Pedagogical design is more strongly associated with learning outcomes than content validity, while all evaluated systems show a near/far transfer gap and several produce negative gains.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Artifact-only evaluation risks missing whether generated materials function as intended, so evaluation should include learner outcomes.

    Source: §5.5 · p. 8

    SLATE evaluates one-shot, non-interactive study and does not capture feedback, learner questions, or spaced repetition.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLMs have achieved remarkable capabilities in generating language teaching slides. However, a critical mismatch persists between visual polish and actual instructional effectiveness. To address this gap, we introduce SLATE (Slide-based Learning Assessment for Teaching Effectiveness), the first benchmark that evaluates AI-generated language teaching slides through instructional effectiveness and learner knowledge acquisition. SLATE transforms linguistics olympiad puzzles from low-resource languages with negligible web presence into 90 standardized instructional units comprising 1,133 assessable items, paired with a structured course outline and matched near- and far-transfer test sets. This pretest-posttest design eliminates pretrained knowledge leakage, ensuring gains reflect learning rather than prior recall. Using VLMs as scalable learner proxies and directionally supported by a three-system human pilot, our results show that content validity exhibits a weak association with learning gain, while pedagogical design exhibits a robust positive association. Moreover, most systems show a significant gap between near- and far-transfer accuracy, and even frontier models can produce negative learning gains. SLATE reveals a dissociation between artifact quality and instructional effectiveness, calling for a paradigm shift in how generative teaching systems are built, evaluated, and deployed.

1 Introduction

Existing slide evaluations emphasize artifact quality, leaving learner acquisition and transfer unmeasured. SLATE evaluates instructional effectiveness directly and finds that pedagogical design is more strongly associated with learning outcomes than content validity.

1 Introduction

  • Existing evaluations emphasize surface properties without measuring learner acquisition, conceptual understanding, or transfer, so polished decks can yield no gain or negative outcomes.Prior artifact metrics include content coherence, visual design, and structural completeness.Source: §1 · p. 1
  • SLATE introduces a benchmark for evaluating AI-generated language-teaching slides through instructional effectiveness and learner transfer.It unifies learning-gain measurement, near/far-transfer evaluation, pretraining-leakage control, and human validation.Source: §1 · pp. 1–2
  • SLATE converts low-resource-language linguistics olympiad puzzles into 90 instructional units with 1,133 assessable items, course outlines, and matched transfer tests.Its four dimensions assess linguistic validity, cognitive-pedagogical design, visual communication, and learning and transfer outcomes.Source: §1 · p. 2
  • Across 10 frontier systems, pedagogical design correlates strongly with learning gain while content validity correlates weakly, and several systems show negative gains.The evaluation also finds a systematic dissociation between artifact quality and instructional effectiveness, supported by a 30-participant human pilot.Source: §1 · p. 2

2 Related Work

Prior work advances slide generation and evaluates generated decks as artifacts, while educational benchmarks largely assess learners or interactive tutoring. SLATE shifts evaluation toward whether generated teaching materials improve learner outcomes.

Slide Generation and Evaluation

  • Existing slide-generation work includes agentic pipelines, but evaluation frameworks score content, design, and coherence as artifact properties.Existing work does not assess whether studying generated decks improves learner outcomes.Source: §2 · pp. 2–3

Educational Benchmarks and Learning Transfer

  • Educational benchmarks mainly test model knowledge and reasoning, while tutoring benchmarks assess model behavior rather than teaching-artifact effectiveness.SLATE instead evaluates generated artifacts through a pretest–study–posttest–transfer protocol.Source: §2 · p. 3

3 The SLATE Benchmark

SLATE treats slide generation as creating instruction from puzzle-derived units and evaluates both deck properties and learner outcomes. Its benchmark construction and four-dimensional framework support controlled assessment of learning and transfer.

3.1 Task Definition

  • A generated deck must teach a puzzle-derived unit correctly, organize it into a learnable sequence, support rule induction visually, and improve performance on source and transfer items.Unlike standard text-to-slide generation, the source is an instructional unit distilled from a rule-learning problem, and evaluation includes learner outcomes as well as artifact quality.Source: §3.1 · p. 3

3.2 Benchmark Overview

  • SLATE covers seven linguistic task types and uses low-resource puzzles to reduce prior-knowledge and retrieval risks while enabling precise near- and far-transfer evaluation.Units are assessed on five factors and assigned to one of three difficulty bands.Source: §3.2 · p. 3, Table 2 · p. 3

3.3 Construction Pipeline

  • The construction pipeline parses puzzles into assessable items, converts them into concept-centered units, creates outlines that withhold transfer answers, and generates rendered teaching decks.Annotators extract 2–6 teachable rules per puzzle and create matched near- and far-transfer items; generators receive a common outline and produce HTML decks rendered as PNG pages.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4 Evaluation Framework

  • The framework evaluates linguistic validity (D1), cognitive-pedagogical design (D2), visual communication (D3), and learner gains and transfer (D4).D3 is an automated coarse filter for basic visual scaffolding; D4 compares source-item pretest and posttest performance with matched near- and far-transfer accuracy.Source: §3.4 · p. 4

4 Experiments

Across 10 frontier systems, SLATE evaluates learner outcomes alongside artifact measures and finds that pedagogical design is more strongly associated with learning gain than content validity. Learning and transfer outcomes vary across systems, student-model capacity, and task types.

4.1 Experimental Setup

  • All 10 systems received the same outlines for 90 units; Qwen3-VL-8B-Instruct completed source-item pretests, studied PNG decks, then took posttests and transfer tests.Source: §4.1 · pp. 4–5

4.3 Ablation Study

  • Slides produced positive gains for three of four student VLMs, scaling with model capacity; context was strongest, and full-frame beat 3-frame sampling by ∼3%.The smallest model, Qwen2.5-VL-3B, had a ↓1.4% slide effect; puzzle context contributed ↑5.6% to ↑12.5%.Source: §4.3 · p. 5, Figure 3 · p. 6, Table 4 · p. 6

4.2 Main Results

  • ↑5.46% (Claude Opus 4.7) and ↑5.14% (Qwen3.6-Max) are the largest D4 gains; four systems lose, and D1 rank diverges from gain.GPT-5.4 scores highest on D1 (4.20) but ranks fourth on gain, while MiniMax-M2.7 scores below average on D1 (3.77) but ranks third.Source: §4.2 · p. 5, Table 3 · p. 6

4.4 Analysis

  • ↑3.5% mean gain makes paradigm completion the most teachable task type, while translation has the weakest gains (↓1.1%).The near/far gap is 30.7% for paradigm tasks and 9.6% for Other; conclusions for Morphophonology and Other are exploratory because of small samples.Source: §4.4 · p. 6, Table 5 · p. 7
  • ρ = 0.72 (p = 0.019) for D2 versus gain, compared with ρ = 0.38 (p = 0.28) for D1; with N = 10, the correlations are suggestive, not conclusive.GPT-5.4's highest D1 score but fourth-place gain, and MiniMax-M2.7's below-average D1 score but third-place gain, illustrate the mismatch.Source: §4.4 · p. 6
  • ↑68% on Wik-Mungkan compound nouns contrasts with ↓50% on Jahai smell semantics; GPT-5.4 also reaches 100% near-transfer yet has a 30% posttest decline.The successful puzzle rises from 4% pretest to 72% posttest, while the failure case drops from 72% to 22%.Source: §4.4 · pp. 6–7
  • Near-transfer accuracy (∼35–42%) exceeded far-transfer (∼11–16%) by 20+%, including for the strongest systems.Figure 4 places systems in a high-near, low-far region, consistent with stronger pattern reuse than conceptual generalization.Source: §4.4 · pp. 5–6, Figure 4 · p. 7

5 Discussion

The discussion finds that pedagogical design is more strongly associated with learning gains than content validity, while reliability checks and human validation support the evaluation approach. It identifies recurring instructional failures and limits SLATE’s scope to explicit rule-based instruction.

5.1 Evaluation Reliability

  • Human–human ICC is 0.80 for D1 and 0.72 for D2, but human–model agreement is more moderate on subjective pedagogical sub-dimensions.Model judges can over-score D2 when decks look structurally complete without genuine pedagogical depth, motivating multi-agent panels.Source: §5.1 · p. 7, Table 6 · p. 7

5.2 Construct Validity: Error Injection

  • Error injection selectively lowers its target dimension, while assessment-integrity corruption raises posttest scores by 0.5 points and reduces far-transfer performance by 5.0 points.These results support the interpretation that SLATE measures instructional properties rather than surface fluency alone.Source: §5.2 · pp. 7–8

5.3 Failure Modes and Hypothesized Mechanisms

  • D2 correlates more strongly with learning gain than D1 (ρ = 0.72 vs. ρ = 0.38), while all systems show near-transfer performance exceeding far transfer and several produce negative gains.The far-transfer items require cross-domain generalization, and current decks rarely provide the abstraction support needed for novel contexts.Source: Figure 5 · p. 7, §5.3 · p. 8

5.4 VLM Student Validity

  • The human pilot showed significant posttest gains, 85.7% per-puzzle directional agreement, and perfect cross-system ranking consistency with VLM results.The 30 participants covered systems with high, intermediate, and negative gains; content perturbation also drove gains below the zero-information baseline.Source: §5.4 · p. 8

5.5 Broader Implications

  • SLATE currently covers explicit rule-based instruction with discrete rules and clear input-output mappings, not context-dependent pragmatics or communicative interaction.The authors identify extending evaluation to those less rule-governed domains as future work.Source: §5.5 · p. 8

6 Conclusion

The conclusion argues that artifact quality and instructional effectiveness are dissociated: well-structured-looking slides do not reliably yield learning gains or transfer. It identifies scaling human validation and adding transfer-oriented generation objectives as next steps.

6 Conclusion

  • Pedagogical design is more strongly associated with learning outcomes than content validity, while all systems show a near/far transfer gap.The authors conclude that current approaches support pattern reuse but not the deeper abstraction needed for conceptual generalization.Source: §6 · p. 8
  • The authors identify scaling human validation and incorporating transfer-oriented objectives into generation pipelines as the most important next steps.Extending SLATE to interactive, multi-session formats is also described as promising for closing the transfer gap.Source: §6 · pp. 8–9

Limitations

The limitations concern both the VLM learner proxy and SLATE’s static, single-session protocol. The authors call for validation with more architectures and larger cohorts, alongside evaluation of interactive instruction.

Limitations

  • The VLM proxy differs from human learners in episodic memory, interactive question-asking, and static image processing, despite perfect ranking consistency in the three-system pilot.Validation across additional model architectures and larger participant cohorts remains future work.Source: Limitations · p. 9
  • SLATE evaluates one-shot, non-interactive study and does not capture iterative feedback, learner questions, or spaced repetition.The authors identify multi-turn and interactive formats as a natural extension.Source: Limitations · p. 9

Ethics Statement

The paper cautions that SLATE is an evaluation benchmark, not evidence that AI teaching slides are ready for unsupervised deployment. Its supporting material connects the benchmark’s scoring and construction choices to learning theory, controlled perturbations, and expert review.

Ethics Statement

  • SLATE is not evidence that AI-generated teaching slides are ready for unsupervised educational deployment; incorrect explanations or visualizations may harm learners, so human oversight remains necessary.Source: Ethics Statement · p. 9

A.1 LLM-as-Judge and Multi-Model Evaluation

  • Prior work motivates multi-agent scoring alongside learner-side transfer measurement because judge scores can diverge from expert human ratings on expertise-dependent evaluations.The cited work recommends multifaceted evaluation combining automatic scoring with outcome-based measures.Source: §A.1 · p. 13

A.2 Simulated Learners and VLM Proxies

  • SLATE uses low-resource languages with negligible web presence and a pretest–posttest design to distinguish learning from pretrained recall.This design addresses the risk that apparent VLM improvement reflects prior exposure rather than learning from the slides.Source: §A.2 · p. 13

A.3 Multimedia Learning and Visual Communication

  • Multimedia learning principles inform SLATE’s D2 pedagogical-design and D3 visual-communication dimensions, while cognitive load theory distinguishes material complexity, poor design, and schema formation.The expertise reversal effect predicts that supports beneficial to novices can become counterproductive as expertise grows.Source: §A.3 · p. 13, §A.4 · p. 13

A.5 Transfer of Learning and Assessment Design

  • Transfer is treated as a continuum across knowledge, physical, temporal, and functional contexts, and retrieval success may depend on how study and test processes match.The discussion also identifies normalized-gain pretest–posttest designs as a standard approach for controlled learning measurement.Source: §A.5 · p. 13

B.1 Instructional Unit Construction

  • Each source puzzle becomes an instructional unit containing a target knowledge point, rules, terms, exceptions, examples, and transfer items; rules aim to capture explanatory generalizations rather than answer templates.A teachable rule is the smallest explanatory statement needed to solve the source puzzle and generalize beyond it.Source: §B.1 · p. 14

B.4 Course Outline Fields

  • The course outline specifies learning objectives, required and prohibited content, visual aids, and transfer design, while keeping transfer answers hidden from the generator.The difficulty rubric measures how hard a concept is to teach, not merely how hard its source puzzle is to solve.Source: §B.4 · pp. 14–15, §B.2 · p. 14, Table 7 · p. 15

B.6 Error Injection Protocol, Expected Results, and Research Value

  • Controlled error injections test whether benchmark dimensions respond selectively to knowledge, pedagogical, visual, and assessment-integrity defects, providing construct validation and a diagnostic lens for future systems.The framework separates content correctness, teaching quality, visual support, and learner change rather than collapsing them into document quality.Source: §B.5 · p. 15, §B.6 · p. 16, C Theory-to-Metric Mapping · p. 16

D.4 Qualitative Disagreement Analysis

Human–model agreement is stronger for checkable content judgments than for interpretive pedagogical judgments, especially practice quality. The accompanying protocol and error-injection setup specify how learner outcomes and benchmark sensitivity are assessed.

D.4 Qualitative Disagreement Analysis

  • Practice quality is the weakest-aligned subdimension, while model-assisted scoring is more stable for checkable content than interpretive pedagogy.Judging whether exercises are well timed, targeted, and generative is harder than checking whether instructional components are present.Source: §D.4 · p. 16

D.5 D4 Protocol

  • Learner models complete source-item pretests and posttests, then near- and far-transfer tests after studying each deck page by page.Main experiments report puzzle-level macro averages, with item-level micro values as supporting statistics.Source: §D.5 · p. 17

D.6 D3 Visual Design Breakdown

  • The D3 breakdown reports visual-design metrics including slide counts, words per slide, tables, colors, visual-aid completeness, and exercise inclusion.VA% summarizes visual-aid completeness in the main results table.Source: §D.6 · p. 17, Table 11 · p. 18

D.7 Difficulty-Stratified Gains

  • Posttest gains are stratified by difficulty band, but estimates for Basic are highly unstable because it contains only two units.The reported table presents gain percentages by difficulty band.Source: §D.7 · p. 17, Table 12 · p. 18

E Error Injection Study

  • The error-injection study tests whether four controlled instructional failures selectively affect their intended benchmark dimensions rather than merely lowering overall scores.The targeted families are knowledge, pedagogy, visual design, and assessment integrity.Source: E Error Injection Study · pp. 17–18

E.1 Experimental Setup

  • Ten high-scoring reference decks receive four controlled corrupted variants each, evaluated with the standard D1–D4 pipeline and five independent runs per variant.The clean decks have D1–D4 averages of at least 8.0/10; learner outcomes are aggregated with mean scores and 95% bootstrap confidence intervals.Source: §E.1 · pp. 18–19

E.2 Main Results

Controlled corruptions selectively damage their intended benchmark dimensions, while learner outcomes distinguish genuine learning from apparent posttest gains. The section also reports statistical support, annotation checks, and a high-gain teaching example.

E.2 Main Results

  • Knowledge corruption lowers D1 by 4.5 points, pedagogical corruption lowers D2 by 4.4 points, and visual corruption lowers D3 by 5.3 points.These selective drops support the intended dimension-specific sensitivity of the benchmark.Source: §E.2 · p. 19, Table 13 · p. 20

E.3 D4-Specific Outcomes

  • Far transfer falls by 4.5 points under knowledge corruption and by 5.0 points under assessment-integrity corruption, despite a 0.5-point apparent posttest increase in the latter.Visual corruption mainly reduces posttest recovery while leaving far transfer relatively less affected; the integrity condition also carries a 1.5-point D1 penalty.Source: §E.3 · p. 19, Table 14 · p. 20

E.4 Interpretation

  • Different perturbations produce distinct learning effects: knowledge errors harm factual accuracy and far transfer, pedagogy disrupts scaffolding and near transfer, and integrity violations separate apparent performance from learning.The authors interpret these patterns as aligned with SLATE’s theoretical design.Source: §E.4 · p. 19

E.5 Statistical Reporting

  • All target-dimension drops are significant (p < 0.001), while non-target drops are not (p > 0.1); advanced tasks show larger far-transfer drops under knowledge and pedagogical corruption (p < 0.01).Comparisons use paired two-tailed t-tests, with 95% bootstrap confidence intervals for score drops.Source: §E.5 · pp. 19–20

F.1 Annotator Qualifications; F.2 Annotation Protocol

  • Three trained graduate-level annotators built the units and outlines, with independent annotation followed by discussion and senior-annotator adjudication.Their backgrounds included linguistics or applied linguistics training, and calibration used five pilot puzzles.Source: §F.1 · p. 20, §F.2 · p. 20

F.3 Inter-Annotator Agreement; F.4 Quality Assurance

  • Annotators showed κ = 0.81 for difficulty bands and r = 0.87 for rule counts; a senior verification pass led to revision of 12% of units.Heldout review judged 94% of near-transfer and 89% of far-transfer items answerable; revisions primarily addressed transfer-item clarity and rule boundaries.Source: §F.3 · p. 20, §F.4 · p. 20

G Case Study: Contrasting Slide Decks; G.1 Success Case: Wik-Mungkan Compound Nouns

  • In the Wik-Mungkan success case, Claude Opus 4.7’s P187 deck had the benchmark’s largest single-puzzle gain, with 100% near transfer and 0% far transfer.Figure 6 reports a 68% gain; the deck uses color-coded tables, explicit rules, and worked examples to teach compound-noun formation.Source: §G.1 · p. 21, Figure 6 · p. 21

G.2 Failure Case: Jahai Smell Verb Semantics

This appendix examines instructional effectiveness through case studies of slides that can reduce posttest accuracy, while documenting the human pilot, instructional-unit examples, and ablation-study model family. It also discusses challenges in aligning pedagogical scoring with learning and outlines validation and benchmark setup.

G.2 Failure Case: Jahai Smell Verb Semantics

  • A Jahai deck from Gemini-3.1-Pro caused the benchmark’s largest negative gain: posttest accuracy fell by 50%.The deck clearly labeled smell categories but introduced confusion through interacting instructional choices.Source: §G.2 · p. 21, Figure 7 · p. 22
  • The Jahai deck’s rigid six-category schema conflicted with prior knowledge, while dense, similar verb forms, mismatched assessment representations, and merged summary entries compounded confusion.Its 11 slides introduced 11 phonologically similar verbs, with three or more on each rule slide; teaching used semantic features while the puzzle required direct description-to-verb matching.Source: §G.2 · p. 21

G.3 Artifact-Quality Gap: Dutch Spelling Alternations

  • GPT-5.4’s Dutch deck scored highest on D1 (4.20/5) yet caused a 30% posttest-accuracy drop, despite 100% near-transfer and 50% far-transfer accuracy.The transfer results indicate that the deck conveyed spelling rules even though posttest accuracy declined.Source: §G.3 · p. 22, Figure 8 · p. 23
  • The Dutch deck may have prompted format-induced response shifts and disrupted prior knowledge by replacing surface forms and direct mappings with intermediate decompositions.The posttest expected surface orthographic forms, whereas worked examples used decomposed notation; the 70% pretest baseline suggested already-functional prior knowledge.Source: §G.3 · p. 22

H.1 Why D2 Is Harder Than D1

  • Human-verification results indicate that D2 is harder to align than D1 because judgments of sequencing, scaffolding, overload, and practice depend more on assumptions about learners.D1 judgments about correctness, alignment, and completeness are comparatively objective.Source: §H.1 · p. 23

H.2 What Error Injection Can Validate

  • The proposed error-injection tests predict that knowledge corruption lowers D1 and D4, pedagogical corruption weakens D2 and learning outcomes, and visual corruption lowers D3 and interpretability.Corrupting exception handling or abstraction support should especially harm far transfer, while leakage may inflate posttest performance and weaken evaluation validity.Source: §H.2 · p. 23

H.3 Why Agreement Analysis Matters

  • Model-assisted D1/D2 scoring is defensible, but pedagogical quality remains harder to score robustly than content validity.Agreement analysis matters because it tests whether model-assisted scores align with expert linguistic and pedagogical judgment.Source: §H.3 · p. 23

K Example Instructional Units

  • Example units cover Wik-Mungkan body-part and quality compounding and Turkish vowel harmony in agentive -CI and privative -sIz suffixes.The examples include original, near-transfer, and far-transfer items, with difficulty assessed across rule complexity, exception and repetition density, transfer abstraction, and distractor strength.Source: K Example Instructional Units · p. 26

L Student Model Abbreviations

M Human Pilot Study Results

  • The human pilot assigned 30 participants (10 per condition) to Claude Opus 4.7, Claude Sonnet 4.6, or Gemini-3.1-Pro under blinded, stratified assignment.The conditions spanned high, intermediate, and negative D4 gains (ranks 1, 5, and 10); participants studied three puzzles stratified by difficulty and task type. Tests were two-tailed with α = 0.05; effect sizes used Cohen’s d for t-tests and η2 for ANOVA.Source: M Human Pilot Study Results · pp. 26–27

M.1 Aggregate Results and Baseline Equivalence

Thirty participants gained overall, but learning outcomes varied sharply by system; VLMs matched human system rankings perfectly, with puzzle-level directional disagreements concentrated near the measurement floor.

M.1 Aggregate Results and Baseline Equivalence

  • Across 30 participants, posttest gain was ↑8.2% (t(29) = 6.78, p < 0.001, d = 1.24), while pretest scores did not differ significantly across conditions.Baseline equivalence: F(2, 27) = 0.32, p = 0.73, η2 = 0.02.Source: §M.1 · p. 27

M.2 Per-Participant Results

  • All participants gained under Claude Opus 4.7 (d = 3.24) and Claude Sonnet 4.6 (d = 2.48), while Gemini-3.1-Pro had negative mean gain (d = 0.74), with 7 of 10 showing zero or negative gains.The first two systems’ mean effects were significant and positive; Gemini-3.1-Pro’s mean effect was significantly negative.Source: §M.2 · p. 27, Table 18 · p. 28

M.3 Cross-System Ranking Validity

  • Human and VLM system rankings were identical—Claude Opus 4.7 > Claude Sonnet 4.6 > Gemini-3.1-Pro—with Spearman ρ = 1.0 (p < 0.001).Learning gains differed significantly across system conditions: F(2, 27) = 42.15, p < 0.001, η2 = 0.76, and all pairwise comparisons were significant.Source: §M.3 · p. 27, Table 17 · p. 27

M.4 VLM–Human Directional Agreement

  • VLM and human gain directions agreed on 72 of 84 puzzle-system pairs (85.7%); all 12 disagreements had at least one gain below 3% and near-transfer accuracy below 40% for both learner types.No discordant pair involved an absolute gain above 5% for either learner type, and system-level rankings remained identical.Source: §M.4 · p. 27

M.5 Task-Type Stratified Results

  • Human and VLM learners showed perfectly consistent system rankings across the major instructional task categories.The task-type-stratified results are reported in Table 20.Source: §M.5 · p. 28, Table 20 · p. 28

N Human Study Details

  • The 30-person, three-system human pilot was designed to validate VLM proxy rankings across the range of instructional effectiveness observed in the main experiment.The appendix documents recruitment, design, procedure, participant instructions, compensation, consent, and ethical review.Source: N Human Study Details · p. 28

N.1 Participants

  • Participants were 30 Mandarin-speaking Chinese university linguistics undergraduates, ages 19–24, with no prior exposure to the puzzle languages.They had at least two years of linguistics coursework; 17 were female and 13 male.Source: §N.1 · p. 28

N.2 Instructions Given to Participants

  • Participants studied three randomly sampled puzzles, completing pretests, slide instruction, posttests on the same items, and near- and far-transfer tests for each.The session lasted about 45 minutes, and participants were told no prior knowledge of the puzzle languages was expected.Source: §N.2 · p. 29

N.3 Experimental Design and Procedure

The human study standardized puzzle assignment and session procedures across three system conditions. Its analyses were preregistered, and participant consent and data handling were specified.

N.3 Experimental Design and Procedure

  • Participants and experimenters were blinded to system identity; each participant received three stratified puzzles, and sessions followed the same timed protocol with free slide navigation but no return to pretesting.Puzzle sampling balanced difficulty and task type across conditions, and no puzzle was assigned to more than one participant per condition.Source: §N.3 · p. 29

N.4 Recruitment and Compensation

  • Each participant received 60 RMB for a single 45-minute session, regardless of task completion.The compensation was standardized across pilot and validation participants.Source: §N.4 · p. 29

N.5 Data Consent

  • Participants gave written informed consent covering study purpose, collected data, anonymized academic use, voluntary withdrawal, and seven-year data storage before destruction.Each participant signed two copies, one for the research team and one to keep.Source: §N.5 · p. 29

N.6 Ethics Review

  • Experimental data were collected anonymously without personally identifiable information and were designated solely for academic analysis, not external disclosure.Participants were represented only by numeric labels.Source: §N.6 · p. 30

N.7 Data Analysis Plan

  • All analyses were preregistered; the primary outcome was posttest-minus-pretest gain, with near- and far-transfer accuracy as secondary outcomes.Tests were two-tailed at α = 0.05; no outliers or missing data were recorded.Source: §N.7 · p. 30

Related papers

Loading 2609.06212v1…