Source-linked AI summary
Engaging with Massive Online Courses
Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, Jure Leskovec
TL;DR
Because MOOC engagement remains poorly understood despite extensive activity data, the paper analyzes learner traces from large Coursera courses. It develops an engagement taxonomy, examines grades and forums, and tests badge presentation experimentally, finding distinct engagement styles and increased activity under some badge presentations.
Problem
Large-scale quantitative evidence about how students engage with MOOCs is limited, making student experience and completion concerns difficult to evaluate rigorously.
Method
The paper quantitatively analyzes activity traces from large Stanford Coursera courses, developing an engagement taxonomy and conducting randomized experiments varying forum-badge presentation.
Results
Students exhibit five distinct engagement categories, while varying badge presentation affects forum activity and engagement.
Takeaways & Limitations
Evaluating and designing MOOCs should account for diverse student behaviors rather than treating them as direct analogues of traditional university courses.
Takeaways & Limitations
The observational comparisons are insufficient to definitively establish that the badge system caused the observed engagement differences.
Abstract
from arXiv · showhide
The Web has enabled one of the most visible recent developments in education---the deployment of massive open online courses. With their global reach and often staggering enrollments, MOOCs have the potential to become a major new mechanism for learning. Despite this early promise, however, MOOCs are still relatively unexplored and poorly understood. In a MOOC, each student's complete interaction with the course materials takes place on the Web, thus providing a record of learner activity of unprecedented scale and resolution. In this work, we use such trace data to develop a conceptual framework for understanding how users currently engage with MOOCs. We develop a taxonomy of individual behavior, examine the different behavioral patterns of high- and low-achieving students, and investigate how forum participation relates to other parts of the course. We also report on a large-scale deployment of badges as incentives for engagement in a MOOC, including randomized experiments in which the presentation of badges was varied across sub-populations. We find that making badges more salient produced increases in forum engagement.
1. INTRODUCTION
The paper develops a trace-based framework for understanding MOOC engagement, showing that students exhibit diverse activity patterns not captured by offline-course intuitions. It also examines grades, forums, and badge presentation as an engagement intervention.
- Motivation: Large-scale quantitative evidence on how students engage with MOOCs remains limited, constraining rigorous evaluation of student experience and completion concerns.
- Approach: The paper analyzes activity traces from large Coursera courses to characterize lecture viewing, assignment submission, forum participation, and other engagement behaviors.
- Engagement styles: Students can be grouped into five recurring engagement styles—Viewers, Solvers, All-rounders, Collectors, and Bystanders—based on relative activity patterns.
- Engagement styles: MOOC engagement styles do not map precisely onto offline categories: some students mainly view or download lectures, while others complete assignments without viewing course content.
- Engagement and grades: Most students receive zero grades, but this does not necessarily indicate no effort; grade relationships also differ between Machine Learning and Probabilistic Graphical Models courses.
- Forums and badges: Forum threads grow mainly through new contributors, and randomized badge experiments found that presenting badges in different ways affected forum engagement.
2. PATTERNS OF STUDENT ACTIVITY
MOOC activity forms a small set of recurring engagement styles whose distribution depends on when students register or first act. Students also show distinct stopping patterns across lectures and assignments, with many interacting after the course ends.
- Engagement styles: Students are classified as Bystanders, Viewers, Collectors, All-rounders, or Solvers using total activity and the fraction of assignment questions attempted.Bystanders take very few actions; the other styles are separated by assignment-versus-lecture activity and whether lectures are viewed or downloaded.
- Activity patterns: Two large activity spikes occur at the total number of lectures and at the total number of lectures plus assignment questions.
- Activity patterns: Assignment stopping points appear as horizontal stripes, whereas lectures lack corresponding vertical stripes indicating clearly defined stopping points.Assignment and lecture engagement therefore exhibit different sequence patterns in the scatter plot.
- Time of interaction: 18% of students registered after the course ended, and these students interacted solely after the course ended under the archaeologist classification.The classification depends on first action time, not registration time, and remains separate from engagement style.
- Time of interaction: The fraction of Bystanders reaches 70% among students registering six months early, falls to 35% near the start date, and rises to 60% after the course ends.
- Time of interaction: A significant shift from All-rounders to Bystanders and Collectors occurs the day after the first assignment is due.
3. GRADES AND STUDENT ENGAGEMENT
Grades are strongly related to how students distribute activity, but the relationship differs across courses and includes substantial engagement among zero-grade students. High-achievers commonly watch many lectures and may submit assignments far more than once.
- Grade distribution: Two-thirds of ML2’s 60,000 registrants received a final grade of 0, while 10% achieved a perfect score and 20% received an intermediate grade.
- Zero-grade engagement: About 50% of registered students with zero grades watched at least one lecture, and 35% watched at least 10.The zero-grade population therefore includes Viewers who spent non-trivial amounts of time watching lectures.
- Grade and activity: In ML2, median assignment submissions, quiz submissions, lecture views, and forum thread views increase linearly with final grade.
- Grade and activity: In PGM2, activity rises until approximately 80% grades and then decreases, while perfect-grade students read less in the forum than lower-grade students.
- High-achievers: High-achievers typically watch many lectures, often exceeding the course’s lecture count through re-watching, although Solvers watch few or none.
- High-achievers: Among high-achievers, the mean number of assignment submissions was 57 for 42 questions, with some students submitting more than 200.
4. COURSE FORUM ACTIVITY
The paper analyzes how students use MOOC forums, finding diverse participation structures and distinct relationships between forum position, activity, grades, and engagement style. It also examines what students’ early forum language may reveal about later course activity.
- Engagement styles: Bystanders comprise over 50% of registered students but only 10% of the ML3 forum population, whereas 90% of All-rounders are forum readers.
- Forum structure: Forum participation is studied as interaction among students, with threads ranging from course discussions and questions to study-group coordination.
- Forum structure: The number of distinct contributors grows linearly with thread length, at roughly 2k/3 contributors for a thread with k posts across all six courses.
- Forum structure: Long conversational threads with very few contributors are extremely rare, indicating that lengthy threads usually involve many students contributing roughly once.
- Contributor dynamics: The forums show an initiator-response pattern: the second contributor is typically most active, while thread initiators tend to have lower grades than later contributors in ML classes.
- Contributor dynamics: In PGM classes, forum activity follows the same pattern, but grades are essentially flat across thread positions.
- Lexical analysis: Words associated with higher future assignment submission suggest prior familiarity with course terminology, while lower-associated words often concern study groups or non-English communication.
5. A LARGE-SCALE BADGE EXPERIMENT
The paper evaluates badges as incentives for MOOC forum activity using observational comparisons and a randomized experiment that varied badge presentation. Badge-targeted behaviors increased substantially, and presentation details also affected engagement.
- Study design: The study introduced forum badges in ML3 and randomized how badges were presented to different student groups.
- Badge system: Badges targeted cumulative thread reading and voting, but not authoring posts or threads, to avoid incentivizing low-quality content.
- Badge effects: Post and comment authoring distributions changed little across runs, unlike voting and thread reading, which were targeted by cumulative badges.
- Badge effects: Post quality did not suffer in ML3 by the measure of votes per item, whose distribution was very similar or slightly heavier than in earlier runs.
- Interpretation: Observational comparisons cannot definitively attribute ML3’s engagement increase to badges because unobserved factors may also differ between course runs.
- Presentation experiment: The badge-ladder treatment had significance value 0.036, while top-byline and thread-byline treatments each had significance value 0.095.
- Presentation experiment: The authors suggest that clear milestones, coherent badge groupings, and greater badge visibility may help explain presentation effects on engagement.
6. RELATED WORK
Prior quantitative MOOC studies examined activity time, demographics, or action timing, but offered limited analysis of engagement styles as defined here. Earlier online-learning research linked forum participation with academic performance, while prior badge studies lacked randomized experimentation in a newly designed system.
- MOOC research: Recent MOOC studies examined time spent, demographics, or action timing, whereas this paper defines engagement styles from students’ activity patterns.
- Educational forums: Earlier online-learning research found that participation in discussion forums was linked to higher academic performance.
- Badge research: Prior badge research analyzed existing systems, while this study designed a new badge system and used a randomized experiment to test how badges produce effects.
7. CONCLUSION
The paper concludes that MOOCs exhibit diverse student behaviors rather than simply replicating traditional university courses. It also reports that randomized variation in badge presentation affected forum activity, motivating further research on online learning environments.
- Conclusion: The analysis finds diverse, recurring MOOC engagement behaviors that do not map neatly onto traditional offline university-course assumptions.
- Conclusion: A badge system and large-scale randomized experiment were used to vary badge presentation across student sub-populations.
- Conclusion: Even small variations in badge presentation affected student activity in the forums.
- Future work: The authors identify predictive models of student behavior and grades, personalization, and recommendation as directions for future work.