PaperlayerView exact revision on arXiv

Source-linked AI summary

Design, development, and preliminary validity and reliability evidence of the Software Engineering Self-Efficacy Scale (SESES)

Albert D. Ritzhaupt, Jhonathan Sora-Cardenas, Priya Ganapathy Prasad, Kai A. Hackney, Neha Rani, Pedro Guillermo Feijóo-García

arXiv:2609.29068v1cs.SE

TL;DR

Measuring software engineering self-efficacy calls for a multidimensional approach that reflects distinct practices rather than one global belief. The paper develops the SESES from curriculum-aligned “I can” items and pilots it with undergraduate computing students. The pilot yielded a five-factor model aligned with the conceptual framework, while the authors note limits in the study’s scope and measure.

Abstract

from arXiv · show

The purpose of this research is to design, develop, implement, and provide preliminary validity and reliability evidence of the Software Engineering Self-Efficacy Scale (SESES). Framed by a conceptual framework using guidance in software engineering curriculum and concepts along with the notion of self-efficacy, we generated an initial item pool of n = 87 items to operationalize and measure software engineering self-efficacy among undergraduate computing students. The conceptual framework traces five dimensions: 1) Requirements Engineering, 2) Teamwork and Collaboration, 3) Software Quality Management, 4) Software Design and Architecture, and 5) Software Agile Methodologies. We pilot tested the SESES with n = 527 undergraduate computing students who had completed a software engineering course in the current semester or a previous academic semester. We employed Exploratory Factor Analysis (EFA) with the Principal Axis Factoring method and an oblique (Promax) rotation to examine the underlying structure of the SESES, resulting in the same five internally consistent latent constructs in the conceptual framework with minimal cross-loading and a simple structure in the pattern matrix, explaining approximately 57% of the variability in these data. Our findings suggest that software engineering self-efficacy is a multidimensional construct of five theorized and correlated, yet distinct latent factors. We unpack the limitations and delimitations of the research while exploring undergraduate computing students' software engineering self-efficacy using necessary domain-specific measurements.

I. INTRODUCTION

Software engineering courses span technical and collaborative demands, yet computing education has lacked a domain-specific measure of students’ confidence across software engineering practice. This study develops the SESES and examines its underlying constructs and preliminary validity and reliability evidence.

I. INTRODUCTION

  • Undergraduate software engineering courses combine lifecycle, technical, teamwork, and communication demands, motivating measurement of students’ beliefs about their capacity to meet them.Course content and organization can vary across programs.Source: §I · p. 1

A. Foundations of Self-Efficacy and Its Measurement in Computing

  • Self-efficacy concerns belief in one’s capacity to perform required actions, and domain-specific measures commonly use task-focused “I can” statements.The paper identifies a lack of a domain-specific, valid, and reliable measure for undergraduate computing students’ software engineering self-efficacy.Source: §I · p. 2

II. PURPOSE AND RESEARCH QUESTIONS

  • The study develops the SESES and asks which latent constructs underlie software engineering self-efficacy and what validity and reliability evidence supports its use.It also examines differences associated with students’ demographics and backgrounds.Source: §II · p. 2

B. Self-Efficacy in the Software Engineering Classroom

  • Prior software engineering classroom studies examined self-efficacy generally or in individual areas, leaving no measure spanning students’ beliefs across multiple areas of practice.Examples include communication and debugging self-efficacy, as well as general measures in project-based settings.Source: B. Self-Efficacy in the Software… · p. 3

C. Curricular Alignment to Software Engineering

The framework draws on professional, academic, and research characterizations of software engineering to identify domains for measuring self-efficacy. It organizes the construct into five related dimensions, each reflecting a distinct area of practice.

C. Curricular Alignment to Software Engineering

  • The framework’s five domains converge across professional, academic, and research sources, while representing course-relevant areas rather than the full scope of software engineering.SWEBOK, CS2023, and a review of software engineering competencies supply complementary characterizations.Source: C. Curricular Alignment to Software… · p. 3

IV. CONCEPTUAL FRAMEWORK

  • Five dimensions—Requirements Engineering, Teamwork and Collaboration, Software Quality Management, Software Design and Architecture, and Software Agile Methodologies—organize the framework and guide item development and analysis.The framework draws on software engineering literature, curriculum guidelines, and mastery-experience “I can” statements.Source: §IV · pp. 3–4, Figure 1 · p. 4

A. Software Agile Methodologies

  • Agile self-efficacy concerns applying iterative principles amid uncertainty, including changing requirements, scope, timelines, and deliverables.The framework distinguishes internalizing an agile mindset from merely executing practices.Source: A. Software Agile Methodologies · p. 4

B. Software Quality Management

  • Software quality management concerns ensuring products meet functional and non-functional requirements through systematic constraints and process standards.The passage connects this domain to gaps in graduates’ knowledge of configuration and quality processes.Source: B. Software Quality Management · p. 4

C. Requirements Engineering

  • Requirements engineering self-efficacy concerns confidence in eliciting, documenting, and validating stakeholder needs to establish a technical baseline.The passage frames this domain as relevant to navigating early project ambiguity.Source: C. Requirements Engineering · pp. 4–5

D. Software Design and Architecture

  • Software design and architecture self-efficacy concerns developing solutions for architecturally significant requirements while managing technical trade-offs.The passage characterizes this as confidence in making high-level structural decisions and balancing business goals with technical sustainability.Source: D. Software Design and Architecture · p. 5

E. Teamwork and Collaboration

The study developed the SESES through curriculum analysis, expert and student review, and a pilot administration to undergraduate computing students. The instrument covers five software engineering domains and was administered across three southeastern U.S. institutions.

E. Teamwork and Collaboration

  • Software engineering teamwork involves collaboratively constructing multi-version programs through technical synchronization and interdependence.The paper also describes role ambiguity and communication breakdowns as challenges students encounter in projects.Source: E. Teamwork and Collaboration · p. 5

V. DESIGN AND DEVELOPMENT PROCESS

  • The SESES development process produced 87 curriculum-aligned “I can” items, then reviewed and refined them with instructors, research-team experts, and four students before pilot testing.The development drew on coursework at two southeastern public research-intensive universities.Source: §V · p. 5

VI. METHOD

  • The pilot study examined the SESES’s underlying constructs and preliminary validity and reliability evidence, following Classical Test Theory guidance.This was the first pilot administration of the scale.Source: §VI · p. 5

A. Educational Context

  • The three participating institutions placed software engineering at different points in the curriculum, but course materials at two shared topics that informed the framework and item pool.All three institutions had ABET-accredited computing programs in the southeastern U.S.Source: A. Educational Context · p. 5

B. Participants

  • The sample comprised 527 undergraduates who had completed a software engineering course at one of three southeastern U.S. institutions.Participants varied in class year, computing coursework, and internship experience; approximately 73% were male and 97% were full-time students.Source: B. Participants · pp. 5–6

C. Instruments

  • The SESES contained 87 self-efficacy statements across five domains, rated on a five-point agreement scale.Participants also provided demographic and computing-background information.Source: C. Instruments · p. 6

D. Data Collection Procedures

  • Students completed the online survey confidentially through course learning-management systems over a three-week period, at their own pace and on their own devices.The survey was completed in uncontrolled environments.Source: D. Data Collection Procedures · p. 6

E. Data Analysis Procedures

The analyses examined the SESES factor structure and related student characteristics, retaining a five-factor solution with mixed fit-index evidence.

E. Data Analysis Procedures

  • Analyses included descriptive statistics, internal consistency reliability, Pearson correlations, and EFA after checks of assumptions and factor-analysis suitability.EFA used principal axis factoring and Promax rotation, with factor retention informed by the Kaiser criterion, parallel analysis, scree review, and parsimony.Source: E. Data Analysis Procedures · pp. 6–7, A. Exploratory Factor Analysis · p. 7

VII. RESULTS

  • The results section examines the factor structure, describes and relates the latent constructs, then investigates student characteristics that may moderate self-efficacy.For item assignment, the analysis used a > 0.4 loading threshold and removed items with problematic cross-loadings.Source: §VII · p. 7, E. Data Analysis Procedures · p. 7

A. Exploratory Factor Analysis

  • 57.2% of variability was explained by a five-factor model retaining 83 items, with zero cross-loadings after four items were removed.Two items had troubling cross-loadings, and two others did not meet the > 0.4 loading threshold.Source: A. Exploratory Factor Analysis · p. 7
  • RMSEA = 0.055 and SRMR = 0.03 indicated reasonable fit, whereas CFI = 0.852 and TLI = 0.832 fell short of good-fit criteria.The authors caution against judging the model solely by the sample-size-sensitive chi-square statistic and recommend weighing the fit evidence holistically.Source: A. Exploratory Factor Analysis · pp. 8–9

B. Descriptive Statistics

Students reported their highest self-efficacy in Teamwork and Collaboration and lowest in Software Design and Architecture; the five factors were positively related, with demographic and experience-related differences varying by construct.

B. Descriptive Statistics

  • 4.28 (SD = 0.63) for Teamwork and Collaboration was highest, followed by 4.03 (SD = 0.60) for Requirements Engineering; Software Design and Architecture was lowest at 3.50 (SD = 0.75).All five construct means exceeded 3.0; Teamwork and Collaboration was slightly leptokurtic, and scores spanned the four-point range between minimum and maximum ratings.Source: B. Descriptive Statistics · p. 9, Table IV · p. 10
  • The five highest-rated items all measured Teamwork and Collaboration, while the five lowest-rated items measured Software Design and Architecture; none fell below 3.0.Lower-rated items had noticeably larger standard deviations, indicating greater variability among students.Source: B. Descriptive Statistics · p. 9, Table V · p. 10, Table VI · p. 10

C. Correlation Analysis

  • All five factors correlated positively and significantly; associations were large (r > 0.50) except the medium Software Design and Architecture–Teamwork and Collaboration relationship.Requirements Engineering–Software Quality Management was the strongest relationship; the authors caution that these correlations do not establish cause and effect.Source: C. Correlation Analysis · p. 10, Table VII · p. 11

D. Individual Differences on Constructs

  • Females reported higher Teamwork and Collaboration and Software Agile Methodologies self-efficacy, while males reported higher Software Design and Architecture self-efficacy.Seniors had lower scores than juniors and sophomores in Software Quality Management and Software Design and Architecture, and lower scores than juniors in Software Agile Methodologies.Source: D. Individual Differences on Constructs · pp. 10–11, Table VIII · p. 11, Table IX · p. 11
  • No construct differed significantly by high-school computer science coursework; internship participants scored higher on all five factors, while prior college computing courses had only small positive relationships with Teamwork and Collaboration and Requirements Engineering.The relationships with prior courses were not discernible across the other factors; Software Quality Management approached but did not meet the p < .05 criterion.Source: D. Individual Differences on Constructs · p. 11, Table X · p. 12, Table XI · p. 12
  • Asians reported higher Software Quality Management self-efficacy than Mixed-race students and higher Software Agile Methodologies self-efficacy than White/Caucasian students; no other race-category differences were significant.Asians represented approximately 46% of the participants.Source: D. Individual Differences on Constructs · pp. 11–12

VIII. DISCUSSION

The discussion presents the SESES as a promising preliminary measure aligned with a five-factor view of software engineering self-efficacy. It also describes differences in students’ ratings and cautions that the study’s scope and self-report approach constrain interpretation.

A. Limitations and Delimitations

  • The framework may not cover all software engineering concepts or approaches, and it was aligned intentionally with courses at two institutions.Data were cross-sectional and collected at three southeastern U.S. institutions; students may have completed the focal course several semesters earlier.Source: A. Limitations and Delimitations · p. 12
  • The measure captures mastery experiences through self-report, leaving other self-efficacy dimensions unmeasured and scores exposed to response biases.The authors identify social-desirability, recall, and poor-self-awareness biases as possible concerns.Source: A. Limitations and Delimitations · p. 12

B. Software Engineering Self-Efficacy

  • Findings support a multidimensional, interrelated five-factor account of software engineering self-efficacy, with the SESES showing internally consistent factors and preliminary support for low-stakes use.The authors describe the framework as a strong starting point while noting that further validation is needed.Source: B. Software Engineering Self-Efficacy · pp. 12–14
  • The authors question whether traditional undergraduate courses provide enough experience with diverse teams and complex existing codebases for students to master software engineering concepts.They note that some concepts may be better appreciated after students enter the profession and gain firsthand experience.Source: B. Software Engineering Self-Efficacy · p. 13
  • Teamwork and Collaboration had the highest ratings, while Software Quality Management and Software Design and Architecture had the lowest and were strongly related.Requirements Engineering was the second-highest-rated construct; the authors question whether coursework provides deep stakeholder-engagement experience.Source: B. Software Engineering Self-Efficacy · p. 13
  • Internship participants had higher self-efficacy scores across all five constructs, while gender differences appeared in specific constructs.Females scored higher in Teamwork and Collaboration and Software Agile Methodologies; males scored higher in Software Design and Architecture.Source: B. Software Engineering Self-Efficacy · pp. 13–14

C. Implications for Practice

The authors discuss using SESES to assess and inform software engineering education, while describing further validation needed before broader applications. They also raise curriculum and assessment questions amid rapid GenAI adoption.

C. Implications for Practice

  • GenAI tools challenge educators to rethink programming assignments and how to integrate these tools responsibly without jeopardizing learning outcomes.The authors describe GenAI as capable of solving authentic introductory and intermediate programming assignments with clever prompting.Source: C. Implications for Practice · p. 14
  • SESES can be used before or after a course to assess students’ self-efficacy across five factors and inform lessons, curriculum changes, and intervention evaluation.Pre-course use could also prime students for course expectations.Source: C. Implications for Practice · p. 14

D. Implications for Future Research

  • Further work should validate the model across populations, conditions, and learning sequences, including CFA, item-level DIF, and stronger convergent and discriminant validity evidence.The authors also propose test-retest studies and comparisons with academic performance.Source: D. Implications for Future Research · pp. 14–15
  • Future applications include longitudinal tracking, studies of educational programs, and industry talent-development uses, with new data needed for each target population.Repeated administration could provide test-retest reliability evidence beyond internal consistency coefficients.Source: D. Implications for Future Research · pp. 14–15
  • The study supports a preliminary five-factor conceptualization and low-stakes use of SESES, while identifying further validation as necessary.The authors report notable gender differences across specific constructs and higher scores among students who had participated in computing internships.Source: D. Implications for Future Research · p. 15

APPENDIX A

The listed Requirements Engineering items cover stakeholder needs, requirements documentation and management, and links between requirements and implementation.

Requirements Engineering

  • Items address eliciting stakeholder needs, writing user stories and use cases, prioritizing requirements, and negotiating stakeholder trade-offs.They also include breaking requirements into modules and tasks.Source: §1 · p. 18
  • Items cover translating, modeling, checking, and tracing requirements, including identifying missing or conflicting specifications and linking requirements to code, design, and testing.The list also includes updating documents and managing requirement changes.Source: §1 · p. 18
  • The item set also includes ethical design decisions, professional standards, software architecture, maintainable code, and quality-risk prioritization.These items extend beyond requirements elicitation and documentation into design, implementation, and quality concerns.Source: §1 · p. 18

2. Teamwork and Collaboration

The teamwork items describe coordination, communication, and collaborative software development, including GitHub workflows and interpersonal support.

2. Teamwork and Collaboration

  • Items cover timely task completion, choosing communication channels, communicating with teammates, and organizing meetings and progress checks.Source: §2 · p. 18
  • GitHub-related items address version control, branches, commits, pull requests, issues, comments, notifications, and collaborative planning.Source: §2 · p. 18
  • Items include reviewing teammates’ code, resolving merge or integration conflicts, and offering constructive suggestions.Source: §2 · p. 18
  • Interpersonal items cover presenting ideas, building trust, mediating disagreements, and supporting teammates facing technical or organizational challenges.Source: §2 · p. 18
  • One item asks students to adapt when new requirements are identified.Source: §2 · p. 18

3. Software Quality Management

The Software Quality Management items cover testing, code health, standards, security, reliability, deployment, and quality monitoring. Together, they frame quality as both evaluation and ongoing improvement across contexts.

3. Software Quality Management

  • Other items address reliability, performance and stability, privacy and security, and detecting vulnerabilities across platforms and conditions.Source: §3 · p. 19
  • Quality assessment spans code analysis, identifying code smells, refactoring, and monitoring software engineering metrics across projects.Source: §3 · p. 19
  • The items assess software testing through creating test cases from specifications or existing software and automating tests across environments.Source: §3 · p. 19
  • The item set also covers maintaining quality through standards, version control and continuous delivery, DevOps practices, and deployment.Source: §3 · p. 19

4. Software Design and Architecture

The Software Design and Architecture items assess architectural selection, modeling, and trade-off reasoning alongside design principles and system decomposition. They include both named architecture styles and practical design representations.

4. Software Design and Architecture

  • The scale covers designing and describing event-driven, blackboard, peer-to-peer, pipe-and-filter, N-tier, layered, and client-server architectures.Source: §4 · p. 19
  • Other items address cohesion and coupling, decomposing systems into reusable components, and understanding MVC and MVVM differences.Source: §4 · p. 19
  • Items assess selecting architectures for projects, comparing architectural trade-offs, and justifying design decisions through documented rationale.Source: §4 · p. 19
  • Modeling skills include creating system diagrams, including flowcharts and entity-relationship diagrams.Source: §4 · p. 19

5. Software Agile Methodologies

The Software Agile Methodologies items cover applying Agile principles, understanding methods and artifacts, and reasoning about trade-offs. The passages also identify items dropped for low loadings or removed for cross-loadings.

5. Software Agile Methodologies

  • The reported Agile items include applying Agile principles, distinguishing Agile from waterfall, and identifying the utility of Agile artifacts.Source: §5 · p. 20
  • Items also address gathering stakeholder needs, trade-offs in Scrum, and coordinating work through GitHub issues, comments, and notifications.Source: §5 · p. 20
  • The item pool includes using DevOps practices and applying lessons from earlier projects to enhance software quality.Source: §5 · p. 20
  • Customer value and effort estimation items were dropped for low factor loadings, while object-oriented design and Git troubleshooting items were removed for cross-loadings.Source: §5 · p. 20

APPENDIX B

Appendix B presents scale items and descriptive statistics across Agile methods, architecture, teamwork, requirements, and quality management. The listed items range from self-rated practices to technical and collaborative tasks.

APPENDIX B

  • Agile items include Scrum roles and activities, backlog maintenance, lifecycle knowledge, boards, and distinguishing Scrum from other Agile methods.For example, differentiating Scrum from other Agile methods had a reported mean of 3.53 (SD 1.11).Source: Appendix B · p. 21
  • Architecture items cover selecting and modeling common styles, diagrams, design trade-offs, cohesion and coupling, and system decomposition.The reported mean for selecting an architecture that fits a project was 3.52 (SD 0.91).Source: Appendix B · p. 21
  • Teamwork items address communication, collaborative planning, GitHub, conflict mediation, timely responsibilities, code review, and trust.Communication had a reported mean of 4.40 (SD 0.79).Source: Appendix B · pp. 21–22
  • Requirements items span eliciting, documenting, prioritizing, tracing, and managing changing requirements and checking features against stakeholder expectations.Source: Appendix B · p. 22
  • Quality items include code review, metrics, professional standards, risk prioritization, testing, continuous quality tools, and learning from prior projects.Source: Appendix B · p. 22
  • Additional quality items cover security, reliability, performance, maintainability, deployment platforms, ethics, and DevOps practices.Source: Appendix B · p. 22
Loading 2609.29068v1…