PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding LLM Usage Among Early-Career Software Engineers in Practice

Julia Alencar, Ronnie de Souza Santos, Italo Santos, Cleyton Magalhaes, Danilo Monteiro Ribeiro

arXiv:2609.27973v1cs.SE

TL;DR

Evidence is limited on how novice software engineers develop effective LLM practices as they move from university into industry. This paper surveys 75 early-career professionals about workplace use, perceived skills, and educational preparation; it finds that effective use involves active human judgment and that participants report a preparation gap.

  • Problem

    Limited evidence describes how novice software engineers experience the transition from university to professional settings where LLMs are embedded in everyday work.

    Source: §I · p. 1, §II · p. 2

  • Method

    The authors conducted a mixed-methods survey of novice professionals, combining descriptive analysis of closed-ended responses with iterative qualitative coding of open-ended responses.

    Source: §I · p. 1, D. Filtering · p. 3, E. Data Analysis · p. 4

  • Results

    LLMs are part of early-career workflows, but effective use involves prompt refinement, output verification, contextual adaptation, and critical judgment rather than simple automation.

    Source: §I · p. 1, §VII · p. 10

  • Takeaways & Limitations

    Participants reported a gap between workplace expectations and university preparation, and emphasized realistic scenarios, hands-on projects, and guided evaluation activities as practical preparation.

    Source: §I · p. 2, A. Comparing Results with the Literature · p. 9

    The convenience sample comprises active LLM users, relies on self-reported experience that may differ from actual practice, and does not support causal claims.

    Source: §IV · p. 4

Abstract

from arXiv · show

Despite the rapid adoption of Large Language Models in professional software engineering, limited research has investigated how early career professionals develop effective AI assisted work practices during their transition into industry. We report findings from a mixed methods survey with 75 novice software engineers who actively use LLM supported tools in their daily work. Our results show that LLMs are embedded in routine software engineering activities, including coding, debugging, testing, documentation, and problem solving. Effective use depends on traditional software engineering competencies, such as debugging, testing, and architectural reasoning, together with critical thinking, output verification, prompt engineering, and continuous human oversight. We also identify a gap between workplace expectations and university preparation, with most participants reporting limited formal education on practical LLM use. These findings have implications for software engineering education, organizational onboarding, and workforce development in AI assisted software engineering.

I. INTRODUCTION

Although LLMs are embedded in software engineering work, limited evidence describes how novice engineers develop effective practices during the university-to-industry transition. This study examines their workplace practices, perceived skill needs, and preparation for AI-assisted development.

I. INTRODUCTION

  • 75 early-career professionals who actively use LLM tools were surveyed; effective practice involved prompt refinement, output verification, contextual adaptation, critical thinking, and human oversight.The study focuses on novices transitioning into industry, rather than students or developers generally.Source: §I · p. 1
  • The study identifies limited university preparation for workplace LLM use and discusses implications for software engineering education, organizational onboarding, and workforce development.Participants often learned AI-assisted development practices independently during their first professional experiences and favored realistic scenarios, hands-on projects, and guided evaluation activities.Source: §I · p. 2

II. RELATED WORKS: SKILLS AND PRACTICES IN AI-ASSISTED SOFTWARE ENGINEERING

  • LLM-assisted work still requires prompt construction, output review, debugging, architectural assessment, and contextual adaptation, alongside traditional technical and interpersonal competencies.Developers increasingly supervise and refine generated artifacts rather than producing every implementation directly.Source: §II · p. 2
  • Prior work has examined LLM use in software engineering tasks, but less is known about novice professionals’ transition from university to AI-assisted workplaces,,.The study addresses this gap through early-career engineers’ perspectives on skills, practices, and educational preparation.Source: §II · p. 2

III. METHOD

The authors used a cross-sectional mixed-methods questionnaire to examine novice engineers’ professional LLM use, perceived skills, and views of university preparation. They combined closed- and open-ended questions and piloted the instrument before deployment.

III. METHOD

  • A cross-sectional mixed-methods survey combined closed-ended measures of usage and perceptions with open-ended accounts of work practices, learning, skills, and educational gaps.The study focused on current practices, perceived technical and non-technical skills, and university preparation for AI-assisted development.Source: §III · p. 2

A. Survey Design

  • The anonymous online questionnaire covered LLM use, competencies, educational preparation, and professional background, with questions on prompting, output evaluation, project adaptation, and learning.Closed-ended items included Likert scales and multiple-choice questions; open-ended items elicited reflective accounts.Source: A. Survey Design · p. 2

B. Pilot

  • Two software engineering education researchers reviewed the survey for clarity, interpretability, relevance, and alignment, prompting revisions to wording and question order.The pilot also refined questions on competencies, workplace practices, and educational preparation.Source: B. Pilot · pp. 2–3

C. Recruitment

The study recruited early-career software engineers who actively used LLM tools, screened responses for eligibility and quality, and analyzed survey data descriptively and inductively. Participation was voluntary, and findings were anonymized and reported in aggregate.

C. Recruitment

  • Participants were recruited through Prolific and qualified through self-reported education, professional characteristics, up to two years of industry experience, and active professional LLM use.Eligible participants included software engineering students working in software-related roles and professionals with up to two years of experience.Source: C. Recruitment · p. 3

D. Filtering

  • Of 110 responses, screening excluded 16 mid-level or senior professionals and quality checks reduced the dataset to 75 novice software engineers.Quality checks included attention questions, removal of blank or nonsensical responses, and review for possible AI-generated text; none were identified as AI generated.Source: D. Filtering · p. 3

E. Data Analysis

  • Closed-ended responses were summarized with descriptive statistics, while open-ended responses were condensed, inductively coded by two authors, and grouped into broader categories through consensus discussions.The quantitative analysis emphasized frequency distributions and descriptive characterization rather than inferential statistics.Source: E. Data Analysis · p. 4

F. Ethics

  • Participation was voluntary with informed consent; no personally identifiable information was collected, and findings were reported in aggregate and anonymized form.Participants were informed about data handling and their right to withdraw.Source: F. Ethics · p. 4

IV. THREATS TO VALIDITY

The study reports a sample of 75 novice LLM users, primarily in their first two years of professional experience, and qualifies interpretation because of self-report, cross-sectional design, and convenience sampling.

IV. THREATS TO VALIDITY

  • Self-reported experiences may differ from actual practices, and the cross-sectional design does not support causal claims.The Prolific convenience sample comprises novice engineers who actively use LLM tools and is not statistically representative of all software engineers.Source: §IV · p. 4

A. Participant Demographics

  • 75 active LLM users were surveyed; 86.7% (n=65) had one to two years of experience, and the largest reported roles were Full Stack Developer (28.0%, n=21), Backend Developer (17.3%, n=13), and Data Scientist or Machine Learning Engineer (14.7%, n=11).Participants came from multiple countries; 62.7% (n=47) identified as men, and 32.0% (n=24) as belonging to groups underrepresented in software engineering.Source: A. Participant Demographics · pp. 4–5

B. How Novice Software Engineers Use LLMs at Work

Novice software engineers reported routine LLM use across varied professional tasks, alongside controlled, iterative, and exploratory interaction patterns. Some also used LLMs for learning and reasoning support.

B. How Novice Software Engineers Use LLMs at Work

C. Skills Required for LLM Usage in Software Engineering

Participants associated effective LLM use with traditional software engineering expertise and competencies for prompting, supervising, and validating AI-generated outputs. They also emphasized cognitive and self-management skills.

C. Skills Required for LLM Usage in Software Engineering

D. LLM Education in Software Engineering Programs

Respondents reported limited and inconsistent formal preparation for practical LLM use, while identifying workflow skills, foundational literacy, and applied teaching formats as educational needs.

D. LLM Education in Software Engineering Programs

  • 41 respondents reported little or no LLM coverage, minimal or highly theoretical instruction, or left the question unanswered; preparation was described as limited and inconsistent.Many described learning practical strategies independently during work.Source: D. LLM Education in Software… · p. 8
  • Respondents wanted programs to teach prompting, output evaluation and verification, debugging, testing, security, workflow integration, responsible use, and critical reasoning.Source: D. LLM Education in Software… · p. 8
  • Respondents called for practical instruction on integrating LLMs into realistic workflows and professional development activities.Source: D. LLM Education in Software… · p. 8
  • Respondents also wanted foundational instruction on what LLMs are, how they function, and how to begin using them before advanced practices.Source: D. LLM Education in Software… · p. 8
  • Preferred applied formats included hands-on and industry-based projects, case studies, lab sessions, and guided prompt-evaluation exercises.Source: D. LLM Education in Software… · p. 8
  • Other proposed formats included integrating content into lectures, providing reading materials, ethics workshops, and dedicated LLM courses.Seven respondents did not answer the question about ethics workshops.Source: D. LLM Education in Software… · p. 8

E. Answer to the Research Question

Novice engineers described integrating LLMs into routine work while adapting to their first professional roles. They associated effective use with combined engineering and oversight skills, amid limited practical university preparation.

E. Answer to the Research Question

  • Novice engineers described using LLMs in implementation, debugging, testing, documentation, reasoning, and learning while adapting to their first professional roles.Source: E. Answer to the Research Question · pp. 8–9
  • Effective participation was perceived to require traditional engineering competencies together with prompt refinement, output verification, contextual adaptation, and human oversight.Source: E. Answer to the Research Question · p. 9
  • Participants reported limited practical LLM exposure during university and described learning work practices independently in their first professional experiences.Source: E. Answer to the Research Question · p. 9

VI. DISCUSSION

The discussion considers the findings in relation to existing literature and their implications for software engineering research, education, and practice.

  • The discussion relates the study’s findings to existing literature and considers implications for software engineering research, education, and practice.Source: §VI · p. 9

A. Comparing Results with the Literature

The findings align with prior accounts of LLM use and its risks, while showing that effective practice combines traditional engineering skills with supervision and judgment. The study’s distinct contribution is its focus on how novices develop these practices during the transition into industry.

A. Comparing Results with the Literature

  • Findings align with prior work on LLM-supported workflows and the need to validate outputs before integrating them into software systems,,,,,,,.Participants described iterative prompting, contextual adaptation, and sustained human oversight,.Source: A. Comparing Results with the Literature · p. 9
  • Participants relied on programming, debugging, testing, architectural reasoning, infrastructure knowledge, and communication alongside prompting and contextual adaptation,.Source: A. Comparing Results with the Literature · p. 9
  • Participants used LLMs for learning and exploration while recognizing risks of reduced critical engagement and overreliance,,,,,.Analytical reasoning, critical thinking, and continuous learning remain important for interpreting generated outputs,,,.Source: A. Comparing Results with the Literature · p. 9
  • Novices often learned AI-assisted practices independently after entering industry, and emphasized practical preparation through realistic scenarios, hands-on projects, and guided evaluation.The study characterizes the transition from university to professional environments rather than AI-assisted development in general.Source: A. Comparing Results with the Literature · p. 9

B. Implications

The paper identifies educational and workplace support needs for early-career engineers using LLMs. Participants report limited practical university preparation and independent learning after entering industry, while the conclusion describes LLMs as routine tools requiring active oversight.

B. Implications

  • Many participants reported limited university exposure to practical LLM use, and identified hands-on projects and realistic development scenarios as ways to build verification and contextual reasoning.Source: B. Implications · pp. 9–10
  • Future research could examine how competencies evolve, compare novice and experienced engineers or organizations, and evaluate interventions for the transition into AI-assisted practice.Source: B. Implications · p. 9
  • Organizations could support early-career professionals with structured onboarding, mentorship, code review of AI-generated artifacts, and guidelines for appropriate LLM use.These practices may foster responsible development, reduce overreliance, and support professional growth.Source: B. Implications · p. 10

VII. CONCLUSION

  • LLM-supported tools are part of early-career workflows for coding, debugging, testing, documentation, and learning, but effective use involves verification, contextual adaptation, and critical judgment.The conclusion notes that the survey will be expanded across roles, backgrounds, and workplace contexts, with interviews planned to explore reported challenges.Source: §VII · p. 10
Loading 2609.27973v1…