PaperlayerView exact revision on arXiv

Source-linked AI summary

From Sentiment Classification to Actionable and Responsible Feedback: A Scoping Review and Evidence Map of NLP in Student Evaluation of Teaching, 2015-2026

Jeff Eicher, Rafael da Silva

arXiv:2609.27939v1cs.CLcs.LG

TL;DR

It is unclear whether NLP’s technical diversification in student-evaluation comments corresponds to gains in educational value and evidence robustness. This scoping review maps evidence across five research-question axes and finds a 49.7-percentage-point drop from demonstrated output or stronger to intended-user evaluation or stronger.

  • Problem

    It remains unclear whether NLP’s technical diversification in SET comment analysis corresponds to gains in educational value and evidence robustness.

    Source: §1.1 · p. 2

  • Method

    The review’s core approach is an evidence and gap map organized along five research-question axes.

    Source: §3.1 · p. 6

  • Results

    A2+ was 258/421 (61.3%), versus A3+ at 49/421 (11.6%), a 49.7-percentage-point drop in the review’s sharpest quantified gap.

    Source: §7 · p. 62

  • Takeaways & Limitations

    The map’s axes should be read together; predictive performance alone does not establish whether technical evolution serves teaching.

    Source: §7 · p. 62

    35.5% (527/1484) of records retained for full-text retrieval had no PDF retrieved, and non-retrieved records differed from retrieved or included studies on measured bibliographic dimensions.

    Source: §6 · p. 59

Abstract

from arXiv · show

Natural language processing (NLP) applied to open-ended teaching-evaluation comments (Student Evaluation of Teaching, SET) has tracked the field's technical evolution--from lexicons and conventional classifiers to transformers and large language models (LLMs)--but it is not evident that this technical diversification has been accompanied by corresponding gains in educational value and robustness of the evidence. This scoping review (PRISMA-ScR) maps 421 studies (2015-2026, 2026 partial) along a technical axis (RQ1) and four value dimensions (RQ2-RQ5). Dual mutually blinded LLM screening with sampled human adjudication coded seven extraction domains, with targeted codebook-boundary review at synthesis. The joint map's sharpest quantified gap is the actionability discontinuity: demonstrated output or stronger (A2+: 258/421; 61.3%) versus intended-user evaluation or stronger (A3+: 49/421; 11.6%), a 49.7 percentage-point drop. Sentiment analysis remains the modal task (300/421); diagnostic and generative depth is a substantial minority (D4-D5: 28.2% of resolved cases); a formal fairness metric is rare (1.9%). The findings are descriptive and do not support causal claims of progress: technological coexistence and uneven reporting are part of the map, but the A2+ to A3+ cliff is the contribution, not a quality ladder.

1 Introduction · 2 Prior reviews and positioning · 3 Method

The review examines whether technical sophistication and analytical depth, methodological support, actionability, and responsible-use evidence evolve jointly, rather than treating complexity as a proxy for value. It documents a staged LLM-assisted workflow with human adjudication and supplementary extraction, followed by descriptive, qualitative, and evidence-gap synthesis.

1.1 The core problem

  • The review’s contribution is an integrated map of technical evolution and four value dimensions, not a catalog of algorithms or performance metrics.The dimensions are analytical depth, methodological validation and reporting, actionability, and evidence of responsible use.Source: §1.1 · p. 2, §1.4 · p. 3

2 Prior reviews and positioning

  • Among the related reviews examined, none jointly characterized all five axes for NLP applied to open-ended SET comments.Adjacent reviews cover parts of the landscape; the claimed contribution is their joint characterization across temporal and technological evolution, not novelty of each axis in isolation.Source: §2 · p. 5

3.1 Review design and protocol

  • The scoping review mapped heterogeneous evidence and gaps along RQ1–RQ5, following JBI guidance and PRISMA-ScR reporting.The evidence and gap map was the review’s main product.Source: §3.1 · pp. 5–6

3.3 Data sources and search dates

  • Searches covered Consensus, Lens, Dimensions, OpenAlex, and Google Scholar on July 6, 2026, followed by three rounds of bidirectional citation snowballing.Snowballing ran July 9–19, 2026; citation links were discovery mechanisms, not automatic inclusion criteria.Source: §3.3 · p. 6, §3.4 · pp. 6–7

3.5 Record management and deduplication

  • The corpus comprised 421 studies after eight confirmed older versions were removed from 429 extraction records.Eligible studies empirically applied NLP or related computational methods to student free-text comments about teaching, instructors, courses, modules, or related teaching experiences, and were published from 2015 onward; specified out-of-scope records were excluded.Source: §3.5 · p. 7, §3.6.1 · pp. 7–8, §3.6.2 · p. 8

3.7 AI-assisted title and abstract screening

  • Two mutually blinded LLMs screened abstracts, with human review for disagreements and borderline records; 303 of 1105 abstract-bearing records entered the overlay.The overlay kept 80 and dropped 223; 802 were auto-retained, while 602 of 1484 unified records followed the no-abstract route to full-text retrieval assessment.Source: §3.7 · pp. 8–9, §3.8 · pp. 9–10

3.9 Full-text retrieval

  • Of 1484 records retained for retrieval, 957 (64.5%) had a PDF on disk and 527 (35.5%) did not; non-retrieval was an access failure, not an eligibility exclusion.The records without PDFs were reported as sought for retrieval but not retrieved, rather than as lost includes.Source: §3.9 · p. 10

3.10 AI-assisted full-text eligibility screening

  • Qwen and DeepSeek agreed on 58/60 records overall and 51/53 include/exclude cases (Cohen’s κ = 0.92).In the 53-case primary set, each model matched the gold-standard set in 90.6% of cases; inclusion recall was 89.5% for Qwen (34/38) and 86.8% for DeepSeek (33/38).Source: §3.10 · p. 10
  • 4/38 records were excluded by both models, and consensus inclusion recall was 33/37 = 89.2% on the agreement set; 429 studies proceeded to extraction and version consolidation yielded N = 421.The five human-only inclusions recovered four agreement-stage false exclusions and one disagreement Keep.Source: §3.10 · p. 11

3.11 Human adjudication

  • Human screening adjudication was performed by J.E.; the n = 60 eligibility confidence sample was single-rater, while the KEEP/DROP coverage overlap showed 36/51 agreement (Cohen’s κ = 0.11; PABAK = 0.41).The overlap was operational coverage, not a second gold standard or a basis for recoding N = 421.Source: §3.11 · pp. 11–12

3.12 Data-extraction framework

  • Across 429 extraction studies, the third judge adjudicated 1,545 disputed labels, forcing 347 proposed positives negative when verbatim evidence was absent.Combined agreement on found flags was 90.1% across the original and supplementary fields. The framework covered seven domains, and human review targeted codebook-boundary cases rather than a random full-extraction audit or dual-human IRR.Source: §3.12 · pp. 12–14

3.12.1 Supplementary extraction after the original 38

  • The 32-question supplementary extraction addressed gaps in the original form, preserving the original 38 questions and their adjudicated results rather than changing eligibility or the research questions.The 32 questions were frozen on 13 August 2026, before execution; unreported context variables were not inferred from affiliation, venue, title, or external geolocation, and higher-order constructs were assigned later under explicit coding rules.Source: §3.12.1 · pp. 15–16

3.13 Piloting and codebook refinement

  • The extraction schema was piloted on 100 deliberately diverse studies, with changes versioned; the pilot was not dual-human extraction IRR or codebook-boundary adjudication.The stated JBI minimum was piloting with two or more team members on at least two to three studies; the pilot was cited as exceeding that size benchmark, not as human double-coding.Source: §3.13 · p. 16

3.14 Evidence-synthesis methods

  • Synthesis combined descriptive frequencies, relational cross-tabulations, qualitative thematic synthesis, and an evidence-and-gap map; cross-tabulations describe this mapped corpus rather than an external population.Contingent logistic or ordinal models were not fitted: explicit external validation had 33 TRUE cases in N = 421, and the family×period×indicator layer contained 149 sparse cells.Source: §3.14 · pp. 16–17

3.15 Data and code availability

  • A versioned repository provides frozen bases and code to reconstruct classifications and prevalence tables, alongside the 421-study metadata list, extraction questions, prompts, and adjudication records.The primary-study PDFs are not redistributed, and reconstruction does not require calling the models or claim bit-for-bit re-execution on deprecated proprietary weights.Source: §3.15 · pp. 17–18

3.16 Protocol deviations

  • The supplementary extension was documented as a protocol deviation: its 32 questions were frozen after the original extraction but before supplementary execution, and did not revise the original results, eligibility, or N = 421.Headlines dependent only on supplementary questions are tagged (v2); sentiment analysis, D1–D5, and A0–A5/A2+/A3+ are not.Source: §3.16 · p. 18

4 Results

The results map technological approaches, analytical depth, methodological and evaluation evidence, actionability, responsible-use reporting, and their combinations across 421 studies. They also characterize future-research recommendations by conceptual pertinence and investigability, with mapped patterns presented descriptively.

4.1 Study selection and corpus composition

  • Of 1,707 post-snowballing include-labelled records, version consolidation yielded 421 studies published from 2015 to 2026 for analysis.Of 957 retrieved records, 527 (35.5%) could not be retrieved; the 528 retrieved records that did not proceed to Step 2 are a distinct eligibility loss. The 602 records without abstracts entered by the snowballing route, not title-and-abstract model screening.Source: §4.1 · pp. 19–21

4.2.1 Expansion without technological substitution

  • All five technological families coexisted in 2023–2025, showing accumulation rather than replacement of earlier approaches.Traditional / Lexicon / Rule-based methods and Classical Machine Learning first appear in the corpus in 2015, Deep Learning in 2016, Transformers in 2019, and Large Language Models / Generative AI in 2022; these are first observed corpus occurrences, not historical first uses. In 2025, counts were 37 Classical ML, 29 Transformers, 25 Deep Learning, 15 LLM / Generative AI, and 16 Traditional studies.Source: §4.2.1 · pp. 22–23, Figure 4 · p. 23

4.2.2 A broad task repertoire, still concentrated on sentiment

  • Sentiment Analysis remained the most prevalent task, appearing in 300 studies (71.3%), within a broad but uneven technology–task repertoire.Of 90 possible task-family combinations, 79 were observed; 11 tasks appeared with all five families. Empty cells indicate combinations not observed in this corpus, not research priorities or impossibilities.Source: §4.2.2 · pp. 24, 26, Figure 5 · p. 25

4.2.3 Technological diversification in a concentrated empirical context

  • Context reporting was incomplete: country was resolved for 200/421 studies, and all four context dimensions were jointly resolved for just 93/421 (22.1%).Among resolved cases, 190/200 (95.0%) were from one country, 232/284 (81.7%) from one institution, and English accounted for 89/195 (45.6%) resolved comment-language bases. The 527 non-retrieved records over-represented closed-access status, the no-abstract route, and recent years; their country and comment language were not inferred.Source: §4.2.3 · p. 26

4.3.2 Most of the corpus remains between D1 and D3

  • D4–D5 outputs accounted for 116/411 (28.2%) resolved studies, while D1–D3 remained the majority.The five mutually exclusive levels classify the highest output demonstrated, not quality, rigor, merit, or actionability; D2 is a distinct thematic-structure output without linked valence, and its ordering above D1 does not drive the main collapsed contrast.Source: §4.3.1 · p. 27, §4.3.2 · p. 28

4.3.3 More recent models do not uniformly produce deeper outputs

  • D4–D5 shares did not rise monotonically over time or by technology family, and LLM / Generative AI studies were not uniformly deep.Across complete periods, D4–D5 ranged from 20.2% in 2019–2021 to 32.5% in 2024–2025, with D1–D3 the majority throughout. By family, D4–D5 was highest for LLM / Generative AI (52.9%) and lowest for Deep Learning (8.3%); 47.1% of resolved LLM studies remained D1–D3. The multi-label family comparison uses resolved-depth denominators (104/175/108/88/34).Source: §4.3.2 · p. 28, §4.3.3 · p. 29, Figure 7 · p. 29

4.3.4 Depth is not equivalent to pedagogical grounding

  • Pedagogical constructs appeared in 199/421 studies (47.3%), but only 23/421 (5.5%) named an external pedagogical framework; depth and pedagogical grounding were distinct.Ten of the 28 D5 studies were not construct-positive. Inferred constructs are not automatically validated measures and require additional human and external-theory or empirical validation.Source: §4.3.4 · p. 30, Figure 8 · p. 30

4.4 RQ3: technical sophistication does not imply methodological maturity

  • Methodological reporting was uneven: dataset descriptions appeared in 410/421 studies (97.4%), but inter-annotator agreement in only 54/421 (12.8%).Instrument provenance appeared in 193/421 (45.8%), and explicit open-field evidence in 117/421 (27.8%). Overall response rate was sufficient to determine in 25/421 (5.9%); written-comment response rate in 27/421 (6.4%); nonresponse-bias discussion in 22/421 (5.2%); and missing-data treatment in 90/421 (21.4%). Only six and seven studies, respectively, reported a single direct percentage for overall and comment response rates; among the 90 missing-data studies, deletion/exclusion/complete-case was identified in 68/90 (75.6%).Source: §4.4.1 · pp. 31–32, §4.4 · p. 31, §4.4.2 · pp. 34–35

4.4.3 Evaluation is common but concentrated on classification metrics

  • 350/421 studies (83.1%) reported usable evaluation metrics, but classification performance dominated (279/421; 66.3%); methodological reporting varied substantially across indicators.Dataset description appeared in 410/421 (97.4%), compared with external validation in 33/421 (7.8%), inter-annotator agreement in 54/421 (12.8%), and code availability in 43/421 (10.2%). Human evaluation as a metric (8/421) is distinct from M11 human evaluation of outputs (58/421).Source: §4.4.3 · pp. 35, 37, Figure 11 · p. 36, §4.4.4 · p. 37

4.4.5 RQ3 synthesis: selective reporting, without uniform maturation

  • Methodological reporting changed selectively, not uniformly: 6 of 11 indicators shifted by at least 10 percentage points between 2015–2018 and 2024–2025, while external validation remained approximately stable.M10 prompt and/or model-version documentation increased from 4.5% to 32.5%. Missing or unreported commenter characteristics limit assessment of item nonresponse and self-selection; the comment corpus should not be presumed representative of all respondents.Source: §4.4.4 · p. 39, §4.4.5 · pp. 39–40

4.5.3 More recent technology also does not produce an actionability gradient

  • A3+ remained a minority in every technological family, and its shares showed no monotonic gradient with technological sophistication.In full-family membership, A3+ ranged from 2.7% in Deep Learning and 7.7% in Classical Machine Learning to 18.9% in Traditional / Lexicon / Rule-based and 23.5% in LLM / Generative AI. Family contrasts are confounded with publication period; LLM / Generative AI studies were concentrated in 2024+.Source: §4.5.3 · pp. 41–43, Figure 14 · p. 42

4.5.4 Depth and actionability are related but are not the same construct

  • Among 411 studies with resolved depth, D1 was modal at A1, whereas D2–D5 were modal at A2; depth and actionability co-occurred but did not guarantee one another.A3+ shares across D1–D5 were 6.6%, 0.0%, 8.0%, 21.6%, and 28.6%, not monotonic; Kendall tau-b was 0.344, a descriptive ordinal association.Source: §4.5.4 · pp. 43, 45, Figure 15 · p. 44

4.5.5 User studies, deployment, and pedagogical grounding

  • Pedagogical grounding is distinct from analytical depth and actionability: generated recommendations can lack pedagogical evidence, and fluent LLM outputs may require human verification.The passage cautions that this is not a prevalence estimate of confabulation in the SET-NLP corpus or a claim that D5 outputs are necessarily harmful.Source: §4.5.5 · p. 45

4.5.2 Demonstration is common; evaluation and authentic use are rare

  • 258/421 studies (61.3%) reached A2+, versus 49/421 (11.6%) at A3+; the 49.7-point drop marks the boundary between demonstrating outputs and intended-user evaluation.A2 is the modal mutually exclusive level (209/421; 49.6%); A3+ additionally includes authentic deployment or pilots and measured impact. The gap is invariant to how A2 is partitioned internally.Source: §4.5.2 · p. 41, Figure 13 · p. 41, §4.5.6 · p. 45

4.6.5 RQ5 synthesis

  • Responsible-use evidence ranged from study limitations (70.5%) to formal fairness metrics (1.9%); institutional-use risk was explicitly reported in 18/421 studies (4.3%).Across actionability levels, 19 of 20 dimensions showed no monotonic increase. Institutional-use risk is distinct from ethical implications, oversight, and transparency; six cases remained uncertain after adjudication and were not counted as positive evidence.Source: §4.6.1 · pp. 46–47, Figure 16 · p. 48, §4.6.3 · p. 49

4.7 Integrated evidence and gap map

  • The integrated map brings together technological family, analytical depth, methodological indicators, actionability, and responsible-use constructs, alongside technology-by-task and depth-by-actionability crosses.Figure 18 presents category density within each axis and the two crossed combinations; the depth-by-actionability cross covers 411 studies with resolved depth.Source: §4.7 · p. 51
  • Fig. 18 maps within-dimension density and crossed combinations across NLP in SET (N = 421), with hatching marking low counts (n < 5), not retrieval.Panel A shows density; panels B and C show crossed combinations, with bubble area or cell color representing n.Source: Figure 18 · p. 52
  • A cell is promoted to a future-research recommendation only when the combination is conceptually pertinent and investigable.The qualitative synthesis of the five research questions remains in Sections 4.2–4.6 and the Discussion.Source: §4.7 · p. 53

5 Discussion

The map shows technological diversification alongside uneven depth, validation, actionability, and responsibility reporting, not a monotonic quality ladder. Its clearest quantified gap is between demonstrated outputs and intended-user evaluation; the findings describe this corpus, not causal effects.

5.1 Overview: did the four value dimensions keep pace with the technical axis?

  • 258/421 studies (61.3%) reached A2+ versus 49/421 (11.6%) at A3+, a 49.7-percentage-point actionability discontinuity.A2+ denotes demonstrated output or stronger; A3+ denotes intended-user evaluation or stronger. The review identifies this gap as its central contribution, not a quality ladder.Source: §5.1 · pp. 53–54

5.2 Did model sophistication increase analytical depth? (RQ1 × RQ2)

  • D4–D5 comprise 28.2% of resolved cases, but depth does not rise monotonically with technology: LLM/Generative-AI reaches 52.9% and Deep Learning 8.3%.The LLM/Generative-AI share is temporally confounded and exposed to a retrieval gap; 47.1% of that family remains in D1–D3. Pedagogical constructs appear in 47.3% of studies, while named frameworks appear in 5.5%; neither measure is equivalent to analytical depth.Source: §5.2 · p. 54

5.3 Did methodological rigor keep pace with model development? (RQ1 × RQ3)

  • Fairness and group-performance metrics appear in 2/421 studies, while reporting of methodological indicators varies by technology and period rather than showing uniform maturation.Some temporal contrasts remain visible within adequately sized family strata, while others attenuate after stratification; these are reporting patterns, not a rigor score.Source: §5.3 · p. 55

5.4 The distance between prediction and actionable teaching improvement (RQ2 × RQ3 × RQ4)

  • A2 is modal in four of five technological families and every complete period, while A3+ remains a minority in every family and resolved-depth category.Kendall tau-b=0.344 and Cramér’s V=0.253 describe corpus co-occurrence, not population effects; no actionability series progresses monotonically by technology, period, or depth.Source: §5.4 · p. 56

5.5 Does responsible use grow with consequence? (RQ4 × RQ5)

  • Responsible-use reporting is not universally monotonic across actionability levels (1/20), and A3+ is higher than A0–A2 on 18/20 dimensions.Institutional-use risk is distinct and explicit in 18/421 studies; small A3 and other cells constrain comparisons.Source: §5.5 · p. 56

5.6 Multilingual and cross-institutional limitations (RQ1)

  • Country was unreported in 221/421 studies and 232/284 resolved institutional scopes were single-institution; instrument provenance was unclear in 228/421.Instrument structure was unreported in 299/421 studies, and an open field was explicitly identified in 117/421. Pooling semantically distinct comment fields without preserving their prompts can obscure the unit of analysis, but measured model bias from pooling is not established.Source: §5.6 · p. 57, §5.7 · p. 57

5.8 Participation, representativeness, and missing voices

  • Explicit overall response-rate evidence appears in 25/421 studies, written-comment response-rate evidence in 27/421, nonresponse-bias discussion in 22/421, and missing-data treatment in 90/421.These are reporting frequencies, not proof of poor response rates, biased NLP models, or absent practices when evidence is unreported.Source: §5.8 · p. 58

5.9 Research agenda

  • The research agenda calls for explicit context, language, validation, missingness, and reproducibility reporting, and for distinguishing demonstrated outputs from evaluated or deployed use.For instructors, common demonstrated outputs do not imply usable recommendations; institutions should not read sparse institutional-risk reporting as absence of risk, and researchers should avoid maturity scores and causal progress claims.Source: §5.9 · p. 58, §5.10 · p. 59

6 Limitations

The review’s evidence is bounded by incomplete full-text retrieval, screening and coding uncertainty, search constraints, and uneven primary-study reporting. These limitations qualify what the mapped corpus can establish without showing that selection bias was measured or that mapped counts should be recoded.

Incomplete retrieval of full texts

  • 527/1,484 retained records (35.5%) lacked retrieved PDFs, limiting full-text eligibility and leaving recent LLM/Generative-AI estimates especially exposed to retrieval gaps.Non-retrieved records were more recent: 80.0% versus 54.3% had known publication years ≥2023, and 90.3% versus 36.3% were Unpaywall-closed. Country, language, technology, and institutional scope were unobservable for those records; 2026 is partial, and the included corpus does not represent the whole field.Source: §6 · p. 59

Metadata and corpus heterogeneity

  • Screening depended on indexed title and abstract metadata, while differing definitions and publication types reduce comparability across studies.The corpus includes journals, conferences, preprints, and theses, which have distinct peer-review and reporting patterns.Source: §6 · p. 59

LLM-assisted screening

  • Automated screening missed 4/38 gold-standard includes (~10.5%), but the human-Keep overlay recovered all four; residual completeness risk remains in unreviewed dual-EXCLUDE records.The 10.5% is an estimate from 38 gold includes, not a census of the field; single-model inclusion recall was 86.8–89.5%. LLM performance depends on model, prompt, and criteria wording, and difficult records may fail non-randomly.Source: §6 · pp. 59–60

Human adjudication and codebook reliability

  • Human validation and codebook checks were limited: the gold sample had one rater, codebook adjudication had one rater, and dual-human D/A extraction IRR was not yet frozen.On the 51-record KEEP/DROP overlap, human–human κ was 0.11 (PABAK = 0.41; Po = 36/51); dual-LLM agreement on original flags was 14,477/15,998 = 90.5%, a consistency floor rather than human-validated accuracy. In a 145-study coverage sample, exact A-level agreement was 73/113 and A3+ binary κ = 0.72; these diagnostics do not recode mapped counts.Source: §6 · pp. 60–61

Supplementary extraction

  • The +32 supplementary questions were added after the original 38 answers existed, creating residual risk that visible form gaps shaped which dimensions were queried independently.The questions were frozen before supplementary execution, not based on mapped prevalences or supplementary frequencies; original results were not rewritten, and supplementary-only headlines are a complementary layer.Source: §6 · p. 61

Search and temporal coverage

  • English-only search strings constrain non-English studies entering the funnel, snowballing may favor Anglophone networks, and coverage of 2026 is incomplete.The English share among resolved language bases was 89/195 = 45.6%, not a majority; the review does not estimate the volume of omitted SET-NLP.Source: §6 · pp. 59, 61

Comparability and primary-study reporting

  • Performance metrics were not compared across studies, and incomplete reporting of SET instruments, response rates, and written comments limited analysis of representativeness and nonresponse bias.The sentiment of students who provided no textual comments is unobservable.Source: §6 · p. 61

7 Conclusion

The map shows coexistence and uneven change across NLP technologies and educational-value dimensions, with a pronounced gap between demonstrated feedback outputs and evaluation by intended users. The review argues that technical sophistication must be read alongside depth, rigor, actionability, and responsibility.

7 Conclusion

  • Sentiment Analysis remains modal (300/421), while D4–D5 constitute 28.2% of resolved cases, illustrating expanded analytical scope without replacement.Technological families accumulate and coexist, and temporal evolution is non-monotonic; base or source English is 89/195 (45.6%), country reporting is resolved for 200/421, explicit pedagogical constructs are 47.3%, and named external frameworks are 5.5%.Source: §7 · pp. 61–62
  • A2+ outputs appear in 258/421 studies (61.3%), versus A3+ intended-user evaluation in 49/421 (11.6%), a 49.7-percentage-point gap.Mutually exclusive A2 remains modal (209/421 = 49.6%), A4+ is 10.0%, and demonstration predominates over authentic use; no monotonic progression by technology, period, or depth, nor causal evidence, is observed.Source: §7 · p. 62
  • Changes in methodological reporting, validation, and reproducibility are selective and uneven, not evidence of a global increase in rigor with model sophistication.LLM/Generative-AI has the highest observed D4–D5 share, a descriptive association only; depth without rigor may be fragile rather than an advance.Source: §7 · p. 62
  • Formal fairness metrics are reported in 8/421 studies (1.9%), amid heterogeneous evidence on safeguards and sparse institutional-use risk reporting.Limitations are reported in 297/421 (70.5%), privacy in 178/421 (42.3%), and bias/fairness evidence in 139/421 (33.0%); institutional-use risk is reported in 18/421, with 6 uncertain and 397 without reporting. No monotonic increase of safeguards with A0–A5 is observed (1/20).Source: §7 · p. 62
  • The review’s contribution is to read technical evolution together with four value dimensions that determine whether NLP advances serve teaching.The axes are presented as interdependent: predictive performance alone measures technical capacity, while depth, rigor, actionability, and responsibility shape its educational relevance.Source: §7 · p. 62

Declarations

The declarations state that the review required no ethics approval, describe data availability and funding, and report competing interests, author contributions, and generative AI use.

Ethics approval and consent to participate

  • Ethics approval was not required because the scoping review recruited no human participants and analysed no identifiable student records.Consent to participate was not applicable.Source: Ethics approval and consent to… · p. 62

Availability of data and materials

  • The review datasets and materials are available on GitHub, but the 421 primary-study full texts are not redistributed.Available materials include frozen bases, the canonical list of included studies as metadata and identifiers, prompt texts, and adjudication sheets.Source: Availability of data and materials · p. 63

Competing interests

Funding

  • The authors received no organizational support for the submitted work; the journal article-processing charge is covered by Universitat Oberta de Catalunya.Source: Funding · p. 63

Authors’ contributions

  • Both authors approved the manuscript, and the large language models used for first-pass screening and extraction are not authors.J.E. contributed to methodology, investigation, and review and editing; R.S. contributed to conceptualization, methodology, data curation, formal analysis, software, and writing.Source: Authors’ contributions · p. 63

Use of generative AI in writing

  • The authors used generative AI tools to refine text and translate from Portuguese into English, then reviewed and edited the content and retained responsibility for it.Generative AI tools are not authors.Source: Use of generative AI in writing · p. 63

Additional file 1 — From Sentiment Classification · 1 Additional file 1 (SM-A). Crosswalk from search questions to research questions

The supplement maps six discovery search questions to five synthesis research questions: RQ1 is the technical axis, while RQ2–RQ5 are value dimensions. The mapping is not one-to-one, and RQ3 had no dedicated search question.

Additional file 1 — From Sentiment Classification

1 Additional file 1 (SM-A). Crosswalk from search questions to research questions

  • The crosswalk distinguishes six search questions (R1–R6) from five research questions (RQ1–RQ5) that organize the synthesis.Source: §1 · p. 72
  • RQ1 is the technical axis, while RQ2–RQ5 are the four value dimensions tested against it.Source: §1 · p. 72
  • The mapping is not one-to-one: a search question may contribute to more than one research question, and RQ3 had no dedicated search question.Source: §1 · p. 72
  • The correspondence between original search questions and final research questions is recorded in Table 1.Source: §1 · p. 72, Table 1 · p. 73

2 Additional file 1 (SM-B). Search strings by question and source · 3 Additional file 1 (SM-C). Models, configuration, and prompt texts · 1. Parse of search-engine output

The supplementary materials document how discovery searches were executed, how models and prompts were configured, and how bibliographic responses were parsed into structured records and seed priorities.

2 Additional file 1 (SM-B). Search strings by question and source

  • Discovery questions R1–R6 were submitted to five sources on July 6, 2026; search syntax and year filters varied by source, and R1–R6 are distinct from RQ1–RQ5.Consensus received natural-language questions, while Lens, Dimensions, and OpenAlex received Boolean strings; the search strings are recorded by question and source.Source: §2 · p. 73
  • The supplied query examples cover sentiment analysis, BERT and LLM applications, actionable feedback, and aspect-based sentiment analysis.R2–R4 include source-specific strings, while the supplied R5 passage states its discovery question.Source: §2 · p. 74

3 Additional file 1 (SM-C). Models, configuration, and prompt texts

  • The logged model identifiers come from frozen artefacts, but vendor aliases and later model-weight changes limit claims of bit-for-bit endpoint reproducibility.The audit reconstructs classifications from frozen response bases in the public repository; several listed identifiers are explicitly unpinned aliases.Source: §3 · pp. 75–76
  • On the original 38 found flags, primary judges agreed in 14477/15998 study×field cells (90.5%); a second-instance judge handled only adjudicable disagreements, while R.S. reviewed extraction and synthesis boundaries.The 60-record confidence sample labelled by J.E. concerned full-text eligibility, and no second-instance judge was used at title-and-abstract screening.Source: §3 · pp. 75–76
  • Prompts for stages 1–4 were operationally written and run in Portuguese, while stages 5–12 used English; stages 1–2 named Consensus but were applied across all five sources.English renderings of stages 1–4 were editorial translations, not the strings sent to models; prompts were not rewritten per database.Source: §3 · pp. 77–78

1. Parse of search-engine output

  • The parse prompt converts search responses or batched reference lists into JSON paper records with metadata and seed-priority labels, not final inclusion decisions.The same prompt was used across all five discovery sources.Source: §1 · p. 79
  • Priority labels were based chiefly on abstracts when available, with title and context used otherwise; the categories distinguish central, complementary, and peripheral papers.The two input variants share the JSON fields and priority criteria; the batched-list variant additionally directs use of an Abstract: line when present.Source: §1 · pp. 79–80

2. Seed ranking

Seed ranking classified structured records as candidate starting points for citation snowballing, not as final inclusions. Papers were assigned one of three seed categories using their relevance to the research questions.

2. Seed ranking

  • The seed-ranking step selected candidate starting papers for citation snowballing from structured search records, rather than making final inclusion decisions.The same prompt was used for structured hits from Consensus, Lens, Dimensions, OpenAlex, and Google Scholar.Source: §2 · p. 80
  • ESSENCIAL papers directly answered at least one research question and were central, highly cited, or methodologically referential; RECOMENDADO papers were relevant complementary seeds, while OPCIONAL papers were peripheral, overly general, weakly related, or had insufficient metadata.Source: §2 · p. 81
  • The prompt required one exact-format block per paper, with its classification and a single objective sentence explaining the reason.Source: §2 · p. 81

3. Snowball relevance screen · 4. Post-snowball per-question abstract review

The two post-snowball reviews used distinct prompts: one made a binary relevance decision against the combined research objective, while the other assessed abstract relevance to each research question. Missing or ambiguous evidence was handled conservatively, with specific operating rules for records lacking abstracts.

3. Snowball relevance screen

  • Records with neither title nor abstract were rejected by default; title-only screening used a placeholder in the abstract slot.This default was an operating rule, not a model output.Source: §3 · pp. 81–82
  • The snowball screen classified each new record as relevant or not against the concatenated R1–R6 objective, truncated to 2,000 characters if needed.It allowed only APROVADO or REJEITADO; doubt was rejected.Source: §3 · pp. 81–82
  • The snowball prompt approved papers with sufficient direct fit and rejected off-objective, generic, unhelpful, or ambiguously documented records.It required a fixed verdict and one short, objective justification.Source: §3 · p. 82
  • No-abstract records in the include-labelled pool were retained through seed-passthrough or parent-promotion, not through a successful model screen.The section distinguishes this pathway from the snowball relevance prompt.Source: §3 · p. 83

4. Post-snowball per-question abstract review

  • For each included record, the post-snowball abstract review assessed relevance separately to one research question at a time, unlike the binary screen against the combined objective.The review was distinct from stage 5’s dual keep/drop screen with a JSON schema.Source: §4 · p. 83
  • The per-question review relied only on the supplied title and abstract, answering NAO when the abstract was missing or insufficient and SIM only with clear evidence of relevance.Its prescribed response also included one to three objective reasoning sentences in Portuguese.Source: §4 · p. 83

5. Title and abstract screening · 6. Discovery-stage full-text highlighter · 7. Full-text eligibility

The review used three distinct screening stages: blinded abstract screening, a discovery-stage full-text highlighter, and a separate full-text eligibility assessment. Each stage had a defined scope and decision protocol.

5. Title and abstract screening

  • Two mutually blinded models screened each available abstract against R1–R6, keeping a record if any question was relevant.The models evaluated abstracts only, using a shared prompt.Source: §5 · p. 84
  • Abstract screening required question-level relevance evidence and a yes, no, or unclear judgment before the overall keep-or-drop decision.A record was kept if at least one question was relevant; off-scope or too ambiguous records were dropped.Source: §5 · p. 84

6. Discovery-stage full-text highlighter

  • After PDFs were obtained through snowballing, a discovery-stage highlighter extracted verbatim quotes relevant to R1–R6 and returned an overall verdict.It used APROVADO, REJEITADO, or DÚVIDA and was explicitly distinct from the stage 7 eligibility protocol.Source: §6 · p. 85

7. Full-text eligibility

  • Stage 7 used a dual independent full-text screen to assess eligibility, not to extract study data or score evidence coverage.Screeners were instructed to use only the paper text and not infer missing information.Source: §7 · p. 86
  • The protocol assessed full-text completeness, required verbatim excerpts and reasoning, and allowed UNKNOWN when evidence was absent or inclusion could not be determined confidently.The final decision rules considered failures on E1–E5 and unresolved UNKNOWN answers.Source: §7 · pp. 86, 88, 89
  • Eligibility assessed whether studies analyzed student-produced teaching-evaluation comments with computational text analysis and constituted primary empirical research.The criteria covered student comments, their SET relevance, NLP application, and empirical data or systems; conceptual and opinion pieces were excluded.Source: §7 · pp. 86–87
  • The screen also checked evidence contribution across five synthesis domains, from NLP methods and tasks to methodological development, actionable feedback, and responsible-AI concerns.At least one of D1–D5 had to be supported for the contribution criterion.Source: §7 · p. 87

8. Original extraction (38 SubRQs)

Two mutually blinded models characterized each included full text across 38 SubRQs, with binary findings mapped to domain-specific scores. Multiple items within a SubRQ counted as one point.

Original extraction (38 SubRQs)

  • 38 SubRQs were used to characterize each included full text through two mutually blinded models.The 38 question wordings are provided in Additional file 1 (SM-D).Source: §8 · p. 90
  • Found=true mapped to 1 and found=false to 0; the model itself did not compute scores.Source: §8 · p. 90
  • Maximum scores were RQ1 7, RQ2 5, RQ3 14, RQ4 5, and RQ5 7, with a total maximum of 38.Multiple items under one SubRQ still counted as one point.Source: §8 · p. 90

9. Original extraction — skeptic challenge

When the primary models disagreed on whether evidence was found, the negative skeptic was re-queried; a positive flip was accepted only when every quoted span appeared verbatim in the paper.

Challenge procedure

  • On found=true/false disagreements, only the skeptic was re-queried with the affirming judge’s evidence and full text, and a flip required every span to occur in the paper.The affirming judge was not re-queried.Source: §9 · p. 91

Evidence guardrails

  • The auditor was instructed to prefer found=false, decide independently, avoid invented or altered quotes, and reject ordinary results reporting unless it evidenced the SubR.Weak, ambiguous, or other-judge-only evidence was grounds for found=false.Source: §9 · p. 92

Output format

  • The challenge requested a JSON object keyed by SubR ID, with found status, change from prior, items, verbatim evidence, reasoning, and confidence.The auditor reassessed only listed SubRs previously called found=false.Source: §9 · p. 92

10. Original extraction — third judge

The third judge serves as an independent auditor only when the primary judges disagree; existence decisions must be supported by the paper and, when found=true, by exact verbatim spans.

Purpose and decision rule

  • The third judge is invoked only on disagreement and does not replace the primary judges or human reviewers.A finding of existence requires non-empty verbatim spans copied exactly and contiguously from the paper; otherwise found is forced to false.Source: §10 · p. 92

Evidence standard

  • The auditor decides from the paper alone, without siding with either prior judge, and defaults to found=false when evidence is weak or ambiguous.Ordinary results reporting does not meet a SubR unless the paper uses it as evidence for that SubR’s definition.Source: §10 · pp. 92–93

Verbatim and output requirements

  • When found=true, each verbatim entry must be an exact contiguous copy; when found=false, verbatim must be empty.The response is JSON keyed by SubR IDs, with fields for found, items, verbatim, reasoning, confidence, and sides_with.Source: §10 · p. 93

11. Supplementary extraction (32 SubRQs) · 12. Supplementary extraction — third judge

The supplementary extraction adds 32 factual fields to the existing 421-paper corpus without replacing Protocol v1 or assigning downstream map scores. A third judge independently audits only disputed evidence fields, while the deposit’s stated exclusions limit what its screening stages establish.

11. Supplementary extraction (32 SubRQs)

  • The 32 supplementary SubRQs add factual fields for the same 421 papers; they do not re-extract the original 38 or replace Protocol v1.The protocol says the new answers supplement v1 and leave its historical extraction and found labels unchanged.Source: §11 · pp. 93–94
  • The supplementary protocol excludes D1–D5, A0–A5, maturity, and Responsible-NLP scores, which are described as deterministic downstream maps.These fields are not to be assigned in the additional extraction.Source: §11 · pp. 93–94
  • The extraction must use only paper text and avoid inferring or completing information that the source does not explicitly support.The protocol also bars following instructions embedded in the paper.Source: §11 · p. 94

12. Supplementary extraction — third judge

  • The third judge audits only disagreements on found status or adjudicable structured fields; agreed evidentiary fields and free-text differences are not reopened.The stated preference is false unless the paper supports true.Source: §12 · pp. 94–95
  • Any disputed true value requires an exact, contiguous supporting quotation; without one, the judge must return false and an empty verbatim array.The rule applies to found=true and disputed structured booleans set to true.Source: §12 · p. 95

What this deposit is not

  • Stages 1–4 and 6 are discovery or snowballing screens, not the unified corpus of 1,484 records; stage 6 is also not the 957-PDF eligibility protocol.The deposit also says its two LLMs are not two independent human screeners, and it does not recode frozen counts.Source: What this deposit is not · p. 95

4 Additional file 1 (SM-D). Inventory of extraction questions (38 original and 32 supplementary)

The inventory distinguishes 38 original extraction questions from a 32-question supplementary extension, which did not replace or rerun the original protocol. The questions cover technical methods, analytical outputs, evaluation, educational context, and actionability-related dimensions.

4 Additional file 1 (SM-D). Inventory of extraction questions (38 original and 32 supplementary)

  • 38 original questions and 32 supplementary factual questions were used; the extension did not rerun or replace the original extraction.The supplementary question identifiers and wording were locked on 13 August 2026, before execution later that day.Source: §4 · p. 96
  • The original inventory asks about NLP tasks and models, analytical outputs and comment-field handling, and evaluation datasets, annotation, and agreement.It also covers baselines, corpus language, dataset description, and training or adaptation strategy.Source: §4 · p. 96
  • The supplementary questions add educational-data context and finer-grained analysis of comment granularity, sentiment, aspect linkage, explanatory synthesis, and generated pedagogical recommendations.They also ask whether pedagogical constructs inform the analytical representation.Source: §4 · pp. 97–98
  • The extension inventories evaluation design; human evaluation of generative, interpretive, summarized, diagnostic, or recommendation outputs; reproducibility reporting for code, data, prompts, and model/version; and explicit actionability claims.Source: §4 · p. 98

5 Additional file 1 (SM-E). Chance-corrected agreement on the full-text eligibility confidence sample … 8 Additional file 1 (SM-H). PRISMA-ScR checklist

Across these supplementary sections, eligibility agreement is reported with explicit sample limitations, while the non-retrieved-record comparison finds substantial bibliographic differences from retrieved records. The supplement also distinguishes targeted codebook-boundary checks from accuracy audits and documents PRISMA-ScR reporting and the absence of critical appraisal.

5 Additional file 1 (SM-E). Chance-corrected agreement on the full-text eligibility confidence sample

  • The eligibility confidence sample used J.E.’s labels as the reference, with κ for chance-corrected agreement and PABAK shown because κ is prevalence-sensitive.Possible and No Abstract records were held out of the primary denominator; PABAK was close to κ given the sample’s moderate 38-versus-15 split.Source: §5 · p. 100

Qwen versus J.E. (primary, n = 53).

  • Qwen–J.E. agreement was κ = 0.78 (κ = 0.7808; PABAK = 0.81) on the primary n = 53 sample.The primary denominator excludes four Possible and three No Abstract records; the confidence sample was a convenience pilot rather than a random or powered funnel draw.Source: Qwen versus J.E. (primary, n = 53) · p. 100, §5 · p. 100

DeepSeek versus J.E. (primary, n = 53).

Consensus versus J.E., when the models agree (n = 51).

Qwen versus DeepSeek (primary, n = 53).

  • Qwen–DeepSeek agreement was κ = 0.92 (κ = 0.9181) on the primary n = 53 set and κ = 0.93 (κ = 0.9315) on the full n = 60 sample.The article’s headline LLM–LLM κ uses the primary-set coefficient; LLM–LLM κ measures model consistency, not accuracy.Source: §5 · p. 100, Qwen versus DeepSeek (primary, n = 53) · p. 101

6 Additional file 1 (SM-F). Targeted human codebook-boundary adjudication (R.S.)

  • Targeted R.S. adjudications addressed codebook-boundary cases using extracted evidence, not a census or random audit, and pipeline-key agreement was diagnostic rather than an accuracy gate.The separate dual-LLM found-flag agreement analysis measures consistency, not human-gold accuracy; these checks do not recode the review’s N = 421.Source: §6 · pp. 102–103, Table 9 · p. 102, Table 10 · p. 103

7 Additional file 1 (SM-G). Bibliographic comparison of non-retrieved reports (ANALYSIS RESULT 45)

  • Non-retrieved records were classified as DIFFERENT from retrieved records under the pre-specified rule, with closed-OA +54.0 pp and no-abstract share +29.5 pp; thesis and title-script triggers did not fire.The reviewer hypothesis that non-retrieved records were older, theses, or had non-English titles was not supported on the measured criteria; country, comment language, and institutional scope were not measured for these records.Source: §7 · p. 104, Table 11 · p. 104

8 Additional file 1 (SM-H). PRISMA-ScR checklist

  • The completed PRISMA-ScR checklist reports the flow diagram and selection counts in the main article; optional critical appraisal of individual evidence sources was not undertaken.The diagram’s reported sequence ends with 421 included studies from 429 eligible records.Source: §8 · p. 105, Table 12 · p. 105

Funding … 12 Figure-generation software

The supplementary material reports screening-agreement diagnostics, applicability-denominator rules, snowballing telemetry, corpus-boundary sensitivity, and figure-generation methods; these analyses retain the primary corpus denominator of N = 421.

9 Additional file 1 (SM-I). Applicability rules for methodological indicators M10 and M11

  • M10 was TRUE for 99/421 = 23.5% under the primary denominator and 99/368 = 26.9% among applicable studies; M11 was 58/421 = 13.8% and 58/141 = 41.1%, respectively.M10 is a union of prompt and model-version documentation, not limited to LLM studies; M11 concerns human evaluation of generated or interpretive outputs, not training labels or annotator-disagreement resolution. Not applicable denotes structural inapplicability, not missing reporting.Source: §9 · p. 107

10 Additional file 1 (SM-J). Snowballing discovery telemetry

  • Round 3 found no first-seen papers and ended with a terminal frontier of 0; summed LLM includes were 1328, not the final eligible-study N.The seed was a pass-through of 1070 records. Expansion-round counters are operational telemetry rather than PRISMA identification stages, and their rows need not close arithmetically.Source: §10 · p. 108, Table 13 · p. 108

11 Additional file 1 (SM-K). Coverage sample and dual-human overlap (R.S.; ANALYSIS RESULT 55); 12 Figure-generation software; Funding

  • Human coverage labels came from distinct samples across abstract, PDF, and included-study stages, and did not recode N = 421 or mapped counts.R.S. labelled 221/1105 abstract-bearing records, 191/957 retrieved PDFs, and 145/421 included studies; figure-generation used Python plotting with matplotlib, not generative-AI images. The search-question table, snowballing table, and conceptual selection figure are in the main article and not reproduced here.Source: Funding · p. 106, §11 · p. 109, §12 · p. 110

Human–human screening overlap (headline); AUTO_RETAIN (no κ versus pipeline).

HV1 dual consensus versus R.S. (audit, not recode).

  • 134/137 = 0.9781 agreement with dual consensus yielded κ = 0.9543 (0.95) and PABAK = 0.96 versus R.S.Versus Step-2 membership, κ = 0.74; sensitivity was not applied, and the four sampled records that would leave and 21 that would enter are not an alternative corpus N.Source: HV1 dual consensus versus R.S. (audit,… · p. 110

HV2 versus title-and-abstract models (not Additional file 1 (SM-E)).

HV3 codebook diagnostic (does not recode maps).

  • 73/113 = 64.6% exact A-level agreement was observed in the census-enriched HV3 sample, while A3+ binary agreement was κ = 0.7211 (0.72).Institutional-use κ = 0.5172 (0.52); on the labelled RQ5 subset, bias/role κ = 0.52, privacy κ = 0.82, and oversight κ = 0.88. These diagnostics do not replace the mapped counts.Source: HV3 codebook diagnostic (does not… · p. 110

Corpus-boundary sensitivity (ANALYSIS RESULT 56; not an alternative N).

  • After dropping five human-only Step-2 includes, the sensitivity corpus was n = 416, with A2+ 254/416 = 61.1% and A3+ 47/416 = 11.3%.SA was 299/416 = 71.9%; D4–D5 was 112/407 = 27.5% of resolved. The primary corpus remains N = 421, and the four gold-standard false-excludes are a subset (n = 417).Source: Corpus-boundary sensitivity (ANALYSIS… · p. 110
Loading 2609.27939v1…