Source-linked AI summary
An AI Factory for Technology Education: An Agentic Pipeline with Closed Loop Audience Feedback
Josue Hernandez, Hector R. Amado-Salvatierra, Rocael Hernandez Rizzardini
TL;DR
The paper addresses the limits of one-pass instructional-content generation, which often lacks external-source checks and a mechanism to learn from audience response. It presents a typed production pipeline with separate demand and authority research layers and a feedback cycle that updates rules. Over five months, it produced 26 guides and recorded engagement effects, including a hook style measured at 1.83 times baseline in the first analysis cycle.
Problem
Many content-generation systems make a single pass without checking documents against external sources or using audience response to change later pipeline behavior.
Source: §1 · p. 2Method
The pipeline uses typed workflow nodes, separates demand research from factual authority, and feeds post-level engagement telemetry into rules used by later production runs.
Source: §1 · p. 2, §3.2 · p. 5, §3.6 · p. 9Results
Over five months, the pipeline produced 26 documented guides and four measurable engagement findings; a contrarian hook style performed at 1.83 times baseline in the first analysis cycle.
Source: §5.5 · p. 15, §8 · pp. 17–18Takeaways & Limitations
Engagement findings can become binding defaults for later runs, while the operator retains authority to override those rules.
Source: §3.6 · p. 9The reported engagement figures measure adoption and demand, not guide completion or learning outcomes.
Source: §7 · p. 17
Abstract
from arXiv · showhide
Technology tools used for professional upskilling change on a timescale of weeks, and written guidance about them ages at the same pace. Large language models have made it inexpensive to draft a tutorial or a course outline, but most deployed generation systems still produce a piece of content once and never check it against what an audience actually does with it. This paper describes an agentic pipeline, organized as a typed workflow graph, that treats instructional content production as a manufacturing process with explicit checkpoints rather than a single generation call. Four node classes structure the graph: deterministic scripts, model steps bound to a fixed rubric, open-ended agent phases, and human approval gates, arranged along a forward production path and closed by a feedback cycle that rewrites generation rules from post level engagement telemetry. Production of content is split into two layers: a demand layer, built from video and community sources, that shapes topic selection and register, and an authority layer, built from official documentation, that every published claim must trace back to. Over five months of continuous operation, the deployed system published 26 documented guides at roughly 1.5 hours of operator time per week, and the feedback loop identified four measurable engagement effects that became binding rules for later runs, including a hook style that outperformed baseline by a factor of 1.83 in its first analysis window.
1. Introduction
Technology tools change quickly, making written guidance costly to keep accurate even as language models make tutorials and course outlines inexpensive to draft. The paper addresses the limits of one-pass generation with a typed production pipeline and feedback from audience engagement.
1. Introduction
- Technology tools and model releases change rapidly, so written guidance can lose accuracy within months.Prior work ranges from single-prompt course drafting [1] to agent-based research and instructional content generation.Source: §1 · p. 2
- Most deployed content-generation systems produce a document in one pass without checking it against external sources or learning from audience response.The paper also identifies an unclear division between judgment-intensive and mechanical work in these systems.Source: §1 · p. 2
- The proposed pipeline uses a typed workflow graph and a cross-run feedback cycle that rewrites rules used by topic selection and generation.It separates deterministic scripts, rubric-bound model steps, open-ended agent phases, and human approval gates.Source: §1 · p. 2
- Over five months, the system produced 26 documented multi-page guides, and its feedback cycle logged four measurable engagement effects.One effect reversed an earlier default after later data contradicted it.Source: §1 · p. 2
2. Related Work
Prior work covers AI-generated instructional content, agent orchestration, and closed-loop learning analytics, but the paper identifies a gap at their intersection. Its pipeline combines typed workflow nodes, separate demand and authority sources, and feedback that rewrites content-generation rules.
2. Related Work
- Instructional-content systems span single-prompt course drafting [1] and multi-agent lesson, course, and multimodal content generation [8] [10] [11].Adjacent retrieval-augmented work includes quiz generation and automated test generation and marking.Source: §2 · p. 3
- Agent-harness research studies orchestration, tool use, and governance [16] [17] [21] [22] [24], while closed-loop learning analytics sends learner or audience data back toward interventions [29] [31] [32].The reviewed adaptation work generally targets assessment or recommendations rather than rules for generating content.Source: §2 · pp. 3–4
- The paper combines typed workflow nodes, separate audience-demand and factual-authority sources, and engagement-driven rule updates with human override.It states that the reviewed systems do not combine these three properties.Source: §2 · p. 4
3. System Architecture
The system coordinates human-facing and external actions through an orchestrator and uses an executor for open-ended production work. Its workflow separates work by node type, researches topics for demand and factual grounding, applies layered generation controls, and closes a cross-run feedback cycle.
3.1 Orchestrator and Executor Decomposition
- The orchestrator handles operator interactions and external side effects, while one executor performs topic research, source-grounded outlining, and content generation sequentially.There is no dedicated reviewer agent; quality checks are embedded as executor checklists, and project-scoped persistent memory carries editorial corrections across runs.Source: §3.1 · pp. 4–5
3.2 The Workflow as a Typed Graph
- The workflow graph assigns deterministic scripts, fixed-rubric model steps, open-ended agent phases, and human approval gates according to the work performed.The operator approves topic selection, the outline, and the completed content bundle; after outline approval, generation proceeds on defaults with at most one consolidated question.Source: §3.2 · p. 5
- Across production runs, engagement telemetry rewrites a strategy file that topic discovery and generation must read on later runs.This feedback cycle distinguishes the workflow from a linear pipeline that only reports past engagement.Source: §3.2 · p. 5
3.3 Phase 0: Topic Discovery
- Topic discovery clusters material from configured video channels and subreddits, scores demand, documentation availability, and source depth, then applies novelty and guide-depth filters.The depth gate requires an authoritative source, at least three sources total, sufficient source length, multiple implementation paths, and material for four to seven substantive sections; scan-health failures can force a stop.Source: §3.3 · pp. 5, 8
3. System Architecture
- The system is organized as a forward production graph with a second, cross-run feedback cycle, as shown by the workflow figures.Figure 1 encodes node classes by shape and fill and distinguishes the forward path from the dashed feedback cycle; Figure 2 shows internal phase steps and checklist gates.Source: Figure 1 · p. 6, Figure 2 · p. 7
3.4 Phase 1: Grounding in Two Layers
- Research separates audience demand and register, drawn from video transcripts and community discussions, from factual authority established through official documentation and other primary sources.Video claims must be checked against primary sources, and video sources cannot appear in the published source list; the phase ends with a gap analysis, classification, and outline for human approval.Source: §3.4 · pp. 8–9
3.5 Phase 2: Generation Under Layered Quality Control
- Generation creates the guide bundle under exemplar calibration, source-verification rules, a style filter, artifact-specific checklists, and a pipeline-level checklist.The bundle includes a hub, four to seven subpages, distribution hooks, and direct-message templates.Source: §3.5 · p. 9
3.6 Publication, Telemetry, and the Feedback Cycle
- A hook style measured at 1.83 times baseline engagement became the default, while a narrative-hook default was later retired after it performed worst in a subsequent window.The operator can override rules, and the loop adapts to those overrides; engagement is treated as adoption and demand evidence, not learning outcome.Source: §3.6 · p. 9
4. Generated Artifacts and Content Design Model
Each run produces a linked instructional guide and audience-acquisition materials, with content structure selected by an automatic three-type taxonomy. The guides use instructional scaffolds and a self-sufficiency criterion, and the workflow is mapped to ADDIE.
4.1 Artifact Inventory per Pipeline Run
- Each of the 26 documented runs produced a guide-hub core, while the full multi-account distribution set was used through late June 2026.A run includes an instructional package and audience-acquisition instrumentation; the hub links to step-specific subpages and includes deliverables, audience descriptors, navigation, and evergreen authoritative sources.Source: §4.1 · p. 10
4.2 A Three Type Content Taxonomy with Automatic Classification
- The pipeline classifies material as a technical tutorial, strategic framework, or comparison guide, using surface signals and a 60-percent dominance rule for mixed material.Each type has its own template and reference exemplars; roughly 20 of 26 deployed guides were technical tutorials.Source: §4.2 · pp. 10–11
4.3 Embedded Instructional Scaffolding
- Guides embed instructional scaffolds including concrete deliverables, prerequisites, worked examples, expected-result feedback, troubleshooting, and transfer to new domains.Formatting rules also prescribe short paragraphs, semantic markers, and numbered lists for sequential steps.Source: §4.3 · p. 11
4.4 The Self-Sufficiency Criterion
- The quality checklist requires readers to be able to implement the guide’s procedure from the guide alone, but current templates include no formal learning assessment.A four-gate checklist covers content, formatting, tone, and lead-magnet quality; the implementation push serves as the practical exercise.Source: §4.4 · p. 12
4.5 Produced Examples
- Produced examples include technical tutorials, a strategic framework, and a comparison table embedded within a technical tutorial.The examples include a six-subpage voice-agent guide, a six-subpage Claude Code framework with more than 40 tips and commands, and a permission-modes tutorial with a five-mode comparison.Source: §4.5 · p. 12
4.6 Correspondence with the ADDIE Model
- The pipeline maps ADDIE’s Analysis, Design, Development, Implementation, and Evaluation stages to its research phases, outline approval, generation, publication, and feedback cycle.Its evaluation loop measures acquisition demand rather than learning gain.Source: §4.6 · p. 12
5. Production Output and Preliminary Results
Across five months, the pipeline published documented guide hubs with limited reported operator time, while its instrumented funnel and telemetry supplied engagement findings that became rules for later production. The evidence measures production, effort, adoption, and demand—not whether readers learned the procedures.
5.1 Content Production at Steady State
- The system published 26 documented multi-page guide hubs over five months; each hub has a landing page and subpages.The documented count is a registry floor; roughly 15 additional guide slots lacked registered hub identifiers.Source: §5.1 · p. 13
5.2 Operator Effort
- Approximately 1.5 hours of operator time per week covered three new guides end to end, including research kickoff, approval gates, publication, and scheduled distribution.This is an operator-measured self-report, not an instrumented time study; research, drafting, source verification, formatting, and packaging run without operator involvement.Source: §5.2 · p. 13
5.3 The Instrumented Funnel
- The funnel registered 3,014 keyword comments and sent 2,803 automated messages; the community added 612 members, with 47 percent of new members attributed by the platform to this channel.The platform’s member export lists current members only, so the counts are a floor.Source: §5.3 · pp. 13–14
5.4 Engagement Telemetry and Collection Method
- The first analysis window’s mean engagement score was 447.5, while the second window’s mean was 300.9, a 33 percent decline.Engagement score is reactions plus 3 times comments plus 2 times shares; comments correlated with score at 0.999 across 63 posts, reflecting the keyword-gated design.Source: §5.4 · p. 15
5.5 Findings from the Feedback Loop
- 1.83 times baseline in the first analysis cycle, the contrarian hook remained strongest in the second window; keyword posts averaged 314.1 versus 182.5 for posts without keywords.The second-window contrarian-hook mean was 387.5, down from 817; the keyword comparison is directional because the groups were unequal in size.Source: §5.5 · p. 15
- Reader- or tool-focused hooks averaged 340.5 or 317.0, versus 189.5 for creator-focused hooks; recently repeated topic families averaged 230.4 versus 346.2 for other topics.The loop made the first-sentence reader-or-tool preference an active rule and recorded the retired creator-story default; pure news posts averaged 154.8, with the window’s worst post scoring 6.Source: §5.5 · p. 15
5.6 Scope of the Evidence
- The reported outcomes measure production, operator effort, engagement, automated delivery, and membership growth—not whether readers complete guides or can perform the procedures.The deployment supports claims about production capability, operator effort, and demand, but does not measure learning.Source: §5.6 · p. 16
6. Discussion
The discussion argues that the pipeline’s node separation, source-grounding structure, and feedback cycle shape what can be audited and how future content is produced. It also identifies limits to what the five-month evidence can establish.
6. Discussion
- Deterministic scripts make scanning, scoring, and publishing auditable across 26 documented runs, concentrating evaluation on the three phases where agents exercise judgment.The paper argues that a single-agent-call pipeline would not provide the same separation or make failure-rate comparisons across node classes as easy to reconstruct.Source: §6 · p. 16
- Separating audience-demand sources from authority sources keeps informal platform claims out of the published-source list while requiring factual claims to trace to authority sources.The paper does not directly measure factual error rates, so this is presented as a design property rather than a validated outcome.Source: §6 · p. 16
- Feedback rules change later topic discovery and writing, but five months and two closed analysis windows cannot establish whether this adaptation generalizes or settles into rarely changing rules.The paper points to the retirement of a narrative-hook default as an example of a rule changing after later-window performance.Source: §6 · p. 16
7. Limitations and Future Work
The evidence is limited to adoption and demand measures, with learning outcomes unmeasured and operator time self-reported. Future work proposes assessment, instrumented time measurement, broader deployment, and telemetry that separates topic from timing.
7. Limitations and Future Work
- Engagement figures measure adoption and demand, not whether readers completed guides or learned to perform the procedures.The effect on learning outcomes remains unmeasured.Source: §7 · p. 17
- Operator effort is self-reported rather than measured in an instrumented time study.The later deployment figures are also estimates, and are not directly comparable because that deployment used a different human-loop scope.Source: §7 · p. 17
- The June engagement spike cannot be separated into topic and timing effects, while the between-window decline may reflect reach, fatigue, or topic selection.The current data cannot distinguish among these explanations.Source: §7 · p. 17
- Planned work would embed assessment modules to measure learning and deploy the pipeline outside artificial intelligence tools to test generalization.The current templates have no comparable learning-outcome instrument.Source: §7 · p. 17
- An instrumented time study and telemetry that records topic and timing separately are proposed to address the effort estimate and engagement confounds.The proposed telemetry would avoid inferring those variables from aggregate engagement scores.Source: §7 · p. 17
8. Conclusion
Over five months, the pipeline published 26 documented guides, and its engagement feedback produced findings that could revise later generation rules. The evidence demonstrates production and adoption, not learning outcomes.
8. Conclusion
- 26 documented guides were published over five months, with roughly 1.5 hours of operator time per week.The feedback cycle produced four measurable engagement findings, including one that reversed an earlier default.Source: §8 · pp. 17–18
- The pipeline uses four node classes and separates demand signals that shape topics from authority sources required to support published claims.Its feedback cycle rewrites generation rules from audience engagement.Source: §8 · p. 17
- A later analysis reversed an earlier generation default after new data contradicted it.The evidence is production capability, operator effort, and audience demand; it does not establish learning outcomes.Source: §8 · p. 18
- Whether the workflow generalizes beyond artificial intelligence tools remains untested, and an assessment layer is needed to measure what readers learn.A second deployment in another fast-changing technical domain would test generality.Source: §8 · p. 18