Source-linked AI summary
What Are Cybersecurity Education Papers About? A Systematic Literature Review of SIGCSE and ITiCSE Conferences
Valdemar Švábenský, Jan Vykopal, Pavel Čeleda
TL;DR
Cybersecurity education spans many concepts and teaching contexts, motivating a synthesis of research trends. The review analyzes 71 SIGCSE and ITiCSE papers from 2010–2019, finding varied topics but common teaching and evaluation patterns, alongside limited sharing of outputs and datasets.
Problem
Method
The authors systematically reviewed 71 cybersecurity education papers from SIGCSE and ITiCSE and organized extracted information in a public dataset.
Source: §3.3 · p. 2, §5 · p. 6Results
Typical papers address secure software development, network security, cyber attacks, cryptography, or privacy, while evaluations commonly use student questionnaires and pre/post-tests; only 31% share outputs and none provide supplementary datasets.
Source: §5.1 · p. 6Takeaways & Limitations
The review provides a trend synthesis and a framework of evaluation goals, evidence types, and analysis methods for future researchers and practice.
Source: §5 · p. 6The review is limited to SIGCSE, ITiCSE, and ICER, although other conferences and journals also publish cybersecurity education papers.
Source: §3.5 · p. 3
Abstract
from arXiv · showhide
Cybersecurity is now more important than ever, and so is education in this field. However, the cybersecurity domain encompasses an extensive set of concepts, which can be taught in different ways and contexts. To understand the state of the art of cybersecurity education and related research, we examine papers from the ACM SIGCSE and ACM ITiCSE conferences. From 2010 to 2019, a total of 1,748 papers were published at these conferences, and 71 of them focus on cybersecurity education. The papers discuss courses, tools, exercises, and teaching approaches. For each paper, we map the covered topics, teaching context, evaluation methods, impact, and the community of authors. We discovered that the technical topic areas are evenly covered (the most prominent being secure programming, network security, and offensive security), and human aspects, such as privacy and social engineering, are present as well. The interventions described in SIGCSE and ITiCSE papers predominantly focus on tertiary education in the USA. The subsequent evaluation mostly consists of collecting students' subjective perceptions via questionnaires. However, less than a third of the papers provide supplementary materials for other educators, and none of the authors published their dataset. Our results provide orientation in the area, a synthesis of trends, and implications for further research. Therefore, they are relevant for instructors, researchers, and anyone new in the field of cybersecurity education. The information we collected and synthesized from individual papers are organized in a publicly available dataset.
1 INTRODUCTION
Cybersecurity’s growing workforce need has prompted formal curricular guidance and informal learning initiatives. This review surveys a decade of cybersecurity education research at SIGCSE and ITiCSE and synthesizes trends for educators, researchers, and the conference community.
1 INTRODUCTION
- Cybersecurity’s global job vacancy reached 2.93 million, prompting a need to train more workers and a range of educational initiatives.ACM/IEEE curricula included cybersecurity in 2013, and the JTF published curricular guidance in 2017; extracurricular events and competitions are also gaining popularity.Source: §1 · p. 1
- The review examines cybersecurity education research presented at ACM SIGCSE and ACM ITiCSE from 2010 to 2019.The authors selected the conferences as leading computing education venues and the only two in the field ranked CORE A at the time.Source: §1 · p. 1
- The review maps taught topics and approaches, evaluation methods, further-research implications, and accepted-paper trends for instructors, researchers, and the conference community.It also offers practical recommendations for researchers.Source: §1 · p. 1
2 RELATED WORK
Prior cybersecurity education syntheses cover related topics but not SIGCSE and ITiCSE specifically. The review situates its contribution alongside work on cybersecurity curricula, Capture the Flag, and broader computing education surveys.
2.1 Cybersecurity Education
- Existing cybersecurity education reviews and syntheses had not focused on SIGCSE and ITiCSE.Related work covered qualitative methods, cybersecurity Master’s programs, professional competencies, and curricula.Source: §2.1 · p. 1
- Prior Capture the Flag studies mapped implementations, analyzed 3,600 challenges from 160 competitions, and described an event-organizing framework.CTF competitions and educational events let participants practice cybersecurity skills.Source: §2.1 · pp. 1–2
2.2 Other Areas of Computing Education
- Computing education surveys have reviewed programming education, educational data mining, CS1 topics, and automated feedback tools.Examples include reviews of 1,666 papers across 15 years and 481 papers across 50 years of SIGCSE conferences.Source: §2.2 · p. 2
3 METHOD OF CONDUCTING THE REVIEW
The review follows established SLR/SMS guidance to extract and analyze data from 71 papers. It searches and filters conference papers, pilots extraction criteria, and notes limits from conference scope and manual coding.
3 METHOD OF CONDUCTING THE REVIEW
- The review’s procedures follow established systematic literature review and systematic mapping study guidelines.The authors treat the approaches’ differences as negligible for this paper’s purposes.Source: §3 · p. 2
3.1 Research Questions
- The authors extracted and analyzed data from 71 papers to address five research questions about topics, teaching context, evaluation, impact, and community.The questions aimed to identify well- and underrepresented topics, describe teaching practices and research methods, assess adoption, and characterize the community.Source: §3.1 · p. 2
3.2 Paper Search and Automated Filtering
- A broad ACM Digital Library query, cybersecur* OR secur*, yielded 209 SIGCSE and 52 ITiCSE results before filtering.The search was restricted to conference proceedings since 2010.Source: §3.2 · p. 2
3.3 Pilot Reading and Manual Filtering
- After removing short submissions, two authors screened 90 full-paper candidates and excluded 19 false positives, selecting 71 papers.Independent screening reached Cohen’s kappa 0.87; discrepancies were resolved through discussion.Source: §3.3 · p. 2, §3.2 · p. 2
3.4 Full Text Reading and Data Extraction
- The authors piloted extraction criteria on 30 papers, then read all 71 selected papers and discussed unclear cases to reach agreement.The documented extraction results formed the dataset.Source: §3.4 · p. 2
3.5 Limitations
- The review covers SIGCSE, ITiCSE, and ICER, although other conferences and journals also publish cybersecurity education work.The authors consider SIGCSE and ITiCSE representative of community trends because they are field flagships.Source: §3.5 · p. 3
- Because data extraction was manual, readers may have overlooked or misinterpreted material, though cross-author discussion and validation aimed to minimize the risk.The authors report confidence in their search and double-checked false-positive filtering.Source: §3.5 · p. 3
4 RESULTS
The results map cybersecurity education topics using curricular knowledge areas and open-coded topic categories. Secure programming and network security are among the most frequent topics, alongside offensive security and human aspects.
4 RESULTS
- The results synthesize data extracted from 71 papers, with the full-record dataset publicly available.Representative papers are included where needed.Source: §4 · p. 3
4.1 RQ1: What Topics Are Taught?
- The authors identified general topic areas from the JTF Cybersecurity Curriculum and open-coded topics discussed in individual papers.A paper could map to multiple categories, so category counts may sum to more than 71.Source: §4.1 · p. 3, §4.1.2 · p. 3
4.1.1 Which Knowledge Areas from the JTF Cybersecurity Curricu-
- The most frequent JTF Knowledge Units were Network Defense (19 papers), Implementation (18), System Control (15), and Cryptography (14).Only 10 papers referenced a standardized curriculum to define learning objectives, and descriptions of teaching content varied in detail.Source: §4.1.1 · p. 3
4.1.2 What are the primary cybersecurity topics?
- Secure programming appeared in 24 papers, network security in 23, and cyber attacks and human aspects in 17 each.Cryptography appeared in 15 papers, and authentication and authorization in 13.Source: §4.1.2 · p. 3
7 Organizational security
Figure 1 reports how often the selected papers discussed each of the eight JTF Cybersecurity Curriculum Knowledge Areas. Independent topic coding produced findings similar to the most frequent curriculum Knowledge Units.
7 Organizational security
- Figure 1 maps how often each of the eight JTF Cybersecurity Curriculum Knowledge Areas appeared in the selected papers.Source: Figure 1 · p. 3
- Independent topic coding yielded findings similar to the most frequent curriculum Knowledge Units, supporting the validity of the results.Source: §7 · p. 3, Figure 1 · p. 3
4.2 RQ2: How Are the Topics Taught?
Most interventions described in the papers were applied in teaching practice and aimed at university or college students. Hands-on learning was common, and many interventions lasted more than a month; participant counts varied widely.
4.2.1 Whom are the teaching interventions aimed at?
- 54 papers targeted university or college students; instructors and managers, K–12 students, and professional learners were each represented in 7, 7, and 3 papers.Eight papers did not explicitly report their target group.Source: §4.2.1 · p. 3
4.2.2 Which teaching methods are used?
- 64 papers (90%) described teaching applied in practice, and hands-on learning was the most common method, appearing in 51 papers.Other methods included lectures in 24 papers, long-term projects in 10, discussion in 8, and writing in 7; exactly half of the 64 papers mentioned pair or group work.Source: §4.2.2 · p. 3
4.2.3 What is the time frame?
- 23 papers studied interventions lasting more than one month, while 27 described one run, 12 described two runs, and 17 described three or more.Other reported intervention durations ranged from up to an hour to several weeks or a month.Source: §4.2.3 · pp. 3–4
4.2.4 How big are the participant populations?
- The median intervention-participant count was 62.5; reported counts ranged from 17 to 14,000, and 9 papers did not report a count.For repeated interventions, the authors summed participants; ranges were averaged, and evaluation sample size served as a lower bound when it was the only figure reported.Source: §4.2.4 · p. 4
4.3 RQ3: How Is Evaluation Performed?
The review examined evaluation goals, collected evidence, analysis approaches, sample sizes, and dataset availability. Evaluations most often addressed participants’ perceptions, used descriptive statistics, and reported no link to an analyzed dataset.
4.3.1 What is the goal of the evaluation?
- Subjective perceptions were the most common evaluation goal, appearing in 50 papers; 21 examined objective learning, 8 student artifacts, and 5 other artifacts.Eight papers presented no evaluation.Source: §4.3.1 · p. 4
4.3.2 What types of data are collected?
- Questionnaires were collected in 40 papers and test data in 22; student artifacts appeared in 8, other artifacts in 4, interviews or focus groups in 7, and anecdotal evidence in 7.Among questionnaire studies, 26 used post-intervention questionnaires only and 14 used pre- and post-questionnaires; test studies split into 10 post-test-only and 12 pre- and post-test designs.Source: §4.3.2 · p. 4
4.3.3 How are the collected data analyzed?
- Descriptive statistics were used in 54 papers, inferential statistics in 19, and qualitative analysis in 10; 15 papers used no analysis method.Four additional papers attempted inferential statistics but reported insufficient information; the review describes reporting that includes the test, p-value, and sample size.Source: §4.3.3 · pp. 4–5
4.3.4 What is the sample size?
- The median evaluation sample size was 40.5, with reported counts ranging from 8 to 1,974; 11 papers did not report a number.The review notes a visible trend that not all students participating in an intervention took part in its evaluation; two large outliers were removed from the plot for readability.Source: §4.3.4 · p. 5
4.3.5 Can the evaluation be replicated?
- None of the examined papers provided an available link to the analyzed dataset.One paper linked to a Gitlab repository, but it was unavailable; another reproducibility aid was including the full wording of evaluation questions.Source: §4.3.5 · p. 5
4.4 RQ4: What Is the Impact of the Papers?
The review assessed impact through shared outputs, downloads, and citations. Many papers offered materials, but access was often unavailable and citation counts were low.
4.4 RQ4: What Is the Impact of the Papers?
- The review measured impact using publicly available outputs, paper downloads, and citation counts.Citation counts and downloads are imperfect indicators, and tool download data are usually private.Source: §4.4 · p. 5
4.4.1 Do the papers provide output usable by other educators?
- 49 of 71 papers (69%) lacked available supplementary materials, including papers whose linked websites were no longer accessible.Thirty papers referenced output for educators, but only 22 of those linked websites remained available.Source: §4.4.1 · p. 5
4.4.2 How much are the papers downloaded and cited?
- The median ACM Digital Library download count was 176, while the median Scopus citation count was 2 and fell to 1 after removing self-citations.The median number of non-self-citations from the SIGCSE or ITiCSE community was 0; the authors note that older papers have more time to accrue downloads and citations.Source: §4.4.2 · p. 5
4.5 RQ5: Who Forms the SIGCSE and ITiCSE Cybersecurity Community?
The review examined authors and affiliations to characterize the SIGCSE and ITiCSE cybersecurity education community. Most authors appeared only once, while university and U.S. affiliations predominated.
4.5 RQ5: Who Forms the SIGCSE and ITiCSE Cybersecurity Community?
- The review examined authors’ affiliations and countries to describe the community’s demographics.Source: §4.5 · p. 5, §4.5.2 · p. 6
4.5.1 Who publishes cybersecurity education research?
- Of 202 unique authors, 175 appeared in only one paper, while 14 appeared in two and 13 in three or more.The review interprets the large one-paper group as indicating a lack of dedicated cybersecurity education researchers publishing at these conferences, alongside a small stable community.Source: §4.5.1 · pp. 5–6
- The author analysis covered 251 author listings and 202 unique authors, with names manually checked for variations across papers.For example, Heather Richter Lipford and Heather Lipford-Richter were treated as the same author.Source: §4.5.1 · p. 5
- Ching-Kuang Shene and Jean Mayo were the most prolific authors, with five visualization papers in an ITiCSE series.Source: §4.5.1 · p. 6
4.5.2 What are the authors’ affiliations?
- Most of the 251 author listings were affiliated with universities (190) or colleges (31), followed by military institutions (22), research centers (7), and private companies (2).Source: §4.5.2 · p. 6
- The USA accounted for 203 author listings, followed by Canada (17) and the Czech Republic (10).The review relates this pattern to the prevalence of studies of higher-education interventions in the USA.Source: §4.5.2 · p. 6
5 DISCUSSION AND CONCLUSIONS
The review synthesizes trends across 71 cybersecurity education papers and offers recommendations for research and teaching. It finds a recurring pattern of university courses and student-perception evaluations, alongside limited sharing of reusable outputs.
5 DISCUSSION AND CONCLUSIONS
- The review covers 71 cybersecurity education papers from SIGCSE and ITiCSE published from 2010 to 2019, and its dataset is publicly available.It also provides a framework of possible evaluation goals, evidence types, and analysis methods.Source: §5 · p. 6
5.1 Summary of the Observed Trends
- A typical paper covered secure software development, network security, cyber attacks, cryptography, or privacy through a course, hands-on exercise, or teaching tool at a North American university.These studies usually reported one semester of teaching experience involving several dozen undergraduate students.Source: §5.1 · p. 6
- Evaluations commonly examined students’ subjective experiences through questionnaires, with pre- and post-tests also used to assess learning gains.Evaluations often involved only a subset of students; results were usually presented with descriptive statistics and sometimes inferential statistics.Source: §5.1 · p. 6
- Only 31% of papers made teaching outputs available to others, no paper supplied a dataset, and almost 87% of unique authors contributed only once.The review notes that low citation counts may suggest fragmentation, possibly because researchers explore disjoint topics and rarely use others’ results.Source: §5.1 · p. 6
5.2 Implications of this Literature Review
- Because K–12 education was underrepresented, the review suggests examining cybersecurity teaching interventions with younger learners.Source: §5.2 · p. 6
- The authors recommend exploring educational data mining and learning analytics to better understand students’ learning processes beyond commonly used questionnaires and tests.They also suggest investigating ways to motivate more students to participate in evaluations and comparing cybersecurity-conference trends with computing-education conferences.Source: §5.2 · p. 6
- Future research should use more rigorous evaluations, share datasets, and report methods more thoroughly.For teaching practice, the authors recommend standardized curriculum documents to define learning outcomes and stable public repositories for materials; tools from 8 of 30 papers were no longer accessible.Source: §5.2 · p. 6