PaperlayerView exact revision on arXiv

Source-linked AI summary

Influence of COVID-19 confinement in students performance in higher education

T. Gonzalez, M. A. de la Rubia, K. P. Hincz, M. Comas-Lopez, L. Subirats, S. Fort, G. M. Sacha

arXiv:2004.09545v1cs.CY

TL;DR

Adapting assessment to changed educational conditions is challenging, so this study compares student performance before and during COVID-19 confinement. It finds significantly better scores during confinement, including on tests whose format did not change.

  • Problem

    Adapting assessment to a new educational system and ensuring that it measures student performance adequately are central challenges.

  • Method

    The study compares assessment outcomes across student cohorts before and during confinement and examines students’ learning strategies.

  • Results

    COVID-19 confinement significantly improved student scores across tests, including assessments whose format remained unchanged.

  • Takeaways & Limitations

    Higher scores during confinement are attributed to improved learning performance associated with more continuous study habits.

  • Takeaways & Limitations

    The study cannot yet establish what specifically accounts for the observed difference in scores during confinement.

Abstract

from arXiv · show

This study explores the effects of COVID-19 confinement in the students performance in higher education. Using a field experiment of 458 students from three different subjects in Universidad Autonoma de Madrid (Spain), we study the differences in assessments by dividing students into two groups. The first group (control) corresponds to academic years 2017/2018 and 2018/2019. The second group (experimental) corresponds to students from 2019/2020, which is the group of students that interrupted their face-to-face activities because of the confinement. The results show that there is a significant positive effect of the COVID-19 confinement on students performance. This effect is also significative in activities that did not change their format when performed after the confinement. We find that this effect is significative both in subjects that increased the number of assessment activities and subjects that did not change the workload of students. Additionally, an analysis of students learning strategies before confinement shows that students did not study in a continuous basis. Based on these results, we conclude that COVID-19 confinement changed students learning strategies to a more continuous habit, improving their efficiency. For these reasons, better scores in students assessment are expected due to COVID-19 confinement that can be explained by an improvement in their learning performance.

3 Eurecat, Centre Tecnològic de Catalunya. Barcelona. Spain

This section identifies Eurecat, Centre Tecnològic de Catalunya, in Barcelona, Spain, and references the Escuela Politécnica Superior at Universidad Autónoma de Madrid and Computer Adaptive Testing.

  • Eurecat, Centre Tecnològic de Catalunya, is located in Barcelona, Spain.
  • The section also references the Escuela Politécnica Superior at Universidad Autónoma de Madrid, Spain.
  • Computer Adaptive Testing is listed as an additional section reference.

1. Introduction

The study addresses uncertainty in higher-education assessment during COVID-19 confinement by analyzing students’ learning strategies before and after confinement. Its data indicate that autonomous learning increased performance, supporting expectations of higher scores.

  • Research objective: The article aims to reduce uncertainty in higher-education assessment during the COVID-19 pandemic.It focuses on correctly interpreting student results when evaluation conditions differ from previous years.
  • Research objective: The study analyzes students’ learning strategies before and after confinement to explain performance changes.The analysis specifically considers autonomous learning, which may remain challenging without direct teacher supervision.
  • Main finding: The data indicate that autonomous learning increased students’ performance, so higher scores should be expected.The authors also discuss the reasons underlying this effect.
  • Study scope: The study covers more than 450 students enrolled in 3 subjects across three academic years at Universidad Autónoma de Madrid.It includes 2019/2020 data collected after COVID-19 restrictions were applied.

2. Background

The background frames e-learning and self-regulated learning as increasingly important for higher-education performance, particularly during the COVID-19 pandemic. It also reviews computer-adaptive testing (CAT) as a personalized assessment and learning tool while noting concerns about test-item exposure and question-bank requirements.

  • E-learning and higher education: E-learning platforms support lectures, assessments, self-evaluation, and analysis of greater amounts of information to improve teaching quality.MOOCs and other online tools have expanded the use of e-learning in higher education.
  • Learning strategies: Time-management tactics correlate with academic performance, while support for managing learning resources is critical to regular learning strategies.These findings motivate attention to students’ learning regularity and strategy management.
  • Self-regulated learning: Self-regulated learning makes students active and responsible for their learning, and stronger self-regulated learning skills are associated with better academic achievement in classroom and online settings.Such evaluation and self-evaluation tools became especially necessary during the COVID-19 pandemic to support performance in e-learning environments.
  • Computer-adaptive testing: CAT dynamically changes items according to prior answers and can personalize questions and feedback to individual learner characteristics.Unlike linear testing, CAT can provide more than a snapshot score and has been used to enhance learning.
  • Computer-adaptive testing: CAT requires knowledge of the learner to personalize subsequent question difficulty but risks continued test-item exposure that can enable memorization and sharing of answers.A large question bank has been suggested to limit exposure, yet most CAT models already require more items than comparable linear tests, making this solution often unfeasible.

3. Purpose

The study aims to identify how COVID-19 confinement affected students’ performance and assessment process. It also investigates whether confinement caused performance differences and which factors explain them.

  • The study aims to identify the effect of COVID-19 confinement on students’ performance.
  • H1 proposes that COVID-19 confinement has a significative effect on students’ performance.
  • A further analysis examines which factors of COVID-19 confinement are responsible for changes in performance.
  • H2 proposes that COVID-19 confinement has a significative effect on the assessment process.
  • The research questions address the direction, causal origin, reasons, and assessment consequences of differences in students’ performance.

4. Materials and methods · 4.1 Measurement Instruments

The study used e-valUAM and Moodle to administer adaptive, open-answer, and traditional assessments. Its CAT model normalized grades from answer scores and progressively selected question levels based on students’ prior responses.

  • 4.1 Measurement Instruments: Two online platforms supported assessment: e-valUAM implemented CAT tests, while the Moodle platform hosted tests without adaptive questions.e-valUAM was used for adaptive tests in Applied Computing and Design of Water Treatment Facilities; traditional tests were used in Metabolism.
  • 4.1.1 CAT theoretical model: The CAT model defines each normalized grade Sj as a weighted sum of normalized item scores ψi across NQ questions.Scores are based on question weights α and answer outcomes, with ψi=1 for correct answers and ψi=0 for incorrect answers.
  • 4.1.1 CAT theoretical model: CAT grades can be rescaled to a final grade FG between 0 and a maximum M using the factor K.The model states FGj=K Sj(α, ψ), with M typically taking values such as 10 or 100.
  • 4.1.1 CAT theoretical model: Adaptive question levels depend on the student’s full answer history and never decrease during a test.The level Lk is proportional to previous correct answers, inversely proportional to NQ, and satisfies Lk≥Lk-1.
  • 4.1.2 Multiple Answer Test (MA-T): MA-T assessments required selecting one correct answer from possible alternatives, with item scores adjusted for random guessing.The e-valUAM interface could optionally include images or sounds, while item level information remained hidden from students.
  • 4.1.3 Open Answer Test (OA-T): OA-T numerical problems varied at least one parameter on each execution and used Matlab code in e-valUAM to calculate solutions.Students entered numerical answers, and the statement could include multimedia files.
  • 4.1.4 Traditional tests: In Metabolism, course content was divided into 6 parts, each followed by online Moodle activities and face-to-face workshops.Activities included Exercises Workshop, Discussion Workshop, Self-assessment, and Test, with scores contributing to continuous assessment except discussion workshops.
  • 4.1.4 Traditional tests: Students completed 15 scored online activities: 5 Exercises Workshops, 6 Self-assessment activities, and 4 Tests.Exercises Workshops and Self-assessment were unsupervised, whereas Tests were conducted online under controlled examination conditions; Discussion Workshops were excluded because professors graded them manually.

4.2 Design of the experiment

The experiment compares pre-confinement control groups with a 2019/2020 experimental group, using assessment activities and autonomous-learning measures across three subjects. It separates the 2019/2020 course into pre-confinement and confinement periods and includes a three-stage longitudinal study of learning strategies and rewards.

  • Group design: The control group comprises students from academic years 2017/2018 and 2018/2019, while the experimental group comprises students from 2019/2020.The groups include Applied Computing, Metabolism, and Design of Water Treatment Facilities, with the latter also examined longitudinally.
  • Measurement procedure: Autonomous learning is measured with adaptive tools used in both learning and assessment, with students informed in advance about evaluation formats and available e-Learning tools.Theory uses MA-T through e-valUAM and OA-T for final evaluation, whereas numerical problems use OA-T for both learning and evaluation.
  • COVID-19 comparison: The confinement experiment compares assessment results between control and experimental groups across two periods: before March 11 and after March 11.Before March 11, measurable activities occurred under similar conditions; after March 11, some activities used a different format.
  • Subject-specific design: Applied Computing uses continuous self-evaluation with the same OA-T for training and final examination, and its test format and questions remained unchanged across the three years.The application is available from the beginning of the course, enabling analysis over a continuous self-evaluation process.

4.3 Statistical analysis

The analysis compared student scores and pass proportions across academic years using tests selected according to data distribution and sample relatedness. Results were reported as mean±SD, with statistical significance set at p<0.05.

  • Measures: Student performance was measured using activity scores (0-10) and the proportion passing each activity (score≥5) across 2019/2020 and the two previous academic years.In one subject, performance was also assessed using scores from 1-10 on all self-evaluation tests.
  • Between-year comparisons: Differences among the three academic years were evaluated with one-way ANOVA or Kruskal-Wallis tests, followed by unpaired t-tests or Mann-Whitney post hoc tests.The preceding academic years were first checked for similarity before comparison with 2019/2020.
  • Statistical tests: Normality was assessed with the D’Agostino and Pearson omnibus test before selecting parametric or non-parametric comparisons.Non-normal data were analyzed with Mann-Whitney or Kruskal-Wallis procedures, whereas normal data used t-tests or one-way ANOVA.
  • Pass-rate analysis: Pass proportions were compared across years using a z-test after confirming that the data met the central limit theorem.The threshold defining a passing activity was score≥5.
  • Reporting: Statistical analyses were performed using GraphPad Prism 6, and statistical significance was set at p<0.05.Data were presented as mean±SD.
  • Related-sample analysis: Related samples involving the same students were compared with a Wilcoxon Signed Rank test when normality could not be assumed.These results were reported as mean±SD with statistical significance set at p<0.05.

4.4 Participants

Participants were drawn from three Universidad Autónoma de Madrid courses across the 2016/2017–2019/2020 academic years. The sample included first-year and fourth-year students in Chemical Engineering and first-year students in Human Nutrition and Dietetics, with COVID-19 disruption affecting the 2019/2020 cohorts.

  • Applied Computing: Applied Computing included 97, 73 and 91 students in 2016/2017, 2017/2018 and 2019/2020, respectively.The first-year, 6 ECTS course combined theory lessons with practical computer-laboratory classes in Chemical Engineering.
  • Metabolism: Metabolism included 64, 63 and 47 enrolled students in 2017/2018, 2018/2019 and 2019/2020, respectively, with confinement beginning approximately halfway through the course.This compulsory, 6 ECTS first-year course in Human Nutrition and Dietetics was strongly affected by COVID-19 mobility restrictions.
  • Design of Water Treatment Facilities: Design of Water Treatment Facilities involved 23 fourth-year students during 2017/2018, with the study focused on the subject’s theoretical component.The 6 ECTS optative Chemical Engineering course included theoretical classroom teaching and practical laboratory classes.

5. Results

Students’ pre-confinement activity was concentrated near examinations, while rewarding more persistent use increased the time span of application use. During confinement, 2019/2020 students achieved significantly higher scores than control cohorts, including on unchanged online activities.

  • Self-learning strategies: More than 33% of tests were performed in the last 6 days before the final exam, with more than 50% of those occurring the day before.The platform was used 688 times in May versus 486 times from February through April.
  • Self-learning strategies: Students showed very low willingness to work continuously; only two OA-T cases involved using the application for more than one day.This pattern motivated the rewarded stage designed to analyze more persistent tool use.
  • Self-learning strategies: A rewarded stage increased average application use over time by 23%, from 1.9 days in stage 2 to 2.4 days in stage 3.Total attempts decreased from 123 to 102, while attempts per student fell from 9.5 to 5.4.
  • Confinement and performance: 4.5±1.6 was the 2019/2020 mean score before confinement, versus 3.9±1.5 in both control years, with significant differences from each control cohort.The comparisons were p=0.0003 versus 2017/2018 and p=<0.002 versus 2018/2019; the control cohorts did not differ (p=0.997).
  • Confinement and performance: 8.1±0.2 was the 2019/2020 mean score for activity 10, versus 6.5±0.2 and 6.7±0.3 in 2017/2018 and 2018/2019, respectively.For activity 11, the experimental-group mean was 7.8±0.2 versus 6.8±0.2 and 6.1±0.3; 95.6% passed activity 10 and 97.7% passed activity 11.

6. Discussion and conclusions

COVID-19 confinement was associated with significantly better student performance, including on assessments whose format did not change. The findings attribute this improvement primarily to changed learning methodology and more continuous autonomous study rather than altered assessment procedures.

  • Research question 2: Students also improved significantly on tests previously conducted in distant format, indicating that changed assessment format was not sufficient to explain the gains.The improvement appeared after confinement, whereas no significant differences were found in comparable distant-format tests performed before it.
  • Research question 3: Performance increased significantly under both additional e-Learning tasks and multimedia classes, suggesting that the improvement was independent of teachers’ specific learning strategies.Because no common instructional element was identified, the study proposes a general change in autonomous learning.
  • Research question 4: Higher scores are expected after COVID-19 confinement, including in activities whose format did not change, because they reflect improved learning performance.The authors conclude that confinement produced a measurable improvement in students’ learning performance.
Loading 2004.09545v1…