Source-linked AI summary
Empirical Software Engineering in Practice: Insights from Google
Roberto Verdecchia, Justus Bogner
TL;DR
Industry practice of empirical software engineering is less understood than academic practice, including how teams choose methods, questions, and uses for results. This interview examines Google’s Developer Intelligence team, which adapts methods to problems and combines evidence across sources. The account highlights mixed-method triangulation, practical decision support, persistent measurement limits, and the need for methodological flexibility.
Problem
Empirical software engineering is well documented in academia but less understood in industry, including how practitioners choose studies and methods, use results, and encounter impediments.
Method
The Developer Intelligence team adapts empirical methods to research questions, combines quantitative, qualitative, survey, interview, diary, and log evidence, and supports intervention decisions.
Results
The interview reports that mixed methods and triangulation can increase confidence, while narrow logs and the absence of objective code-quality metrics remain important challenges.
Takeaways & Limitations
The account supports methodological breadth, acceptance of more than one valid way to conduct research, and an empirical lens that includes non-technical aspects of software engineering.
Takeaways & Limitations
The interview notes that objective measurement remains elusive for code quality and that software engineering research captures context-dependent slices of a complex human system.
Abstract
from arXiv · showhide
While it is fairly well known how empirical software engineering (ESE) is used in the academic world, we have limited knowledge of how ESE is practiced in industry. As part of our regular column on empirical software engineering (ACM SIGSOFT SEN-ESE), we want to dedicate a series of articles to interviewing ESE practitioners from various companies. Among other things, we want to understand how ESE processes are implemented in industry, e.g., different research methods, how practitioners decide on what to study, how research results are used within companies and beyond, and if they face recurrent impediments to using ESE methods in industrial contexts. In the first edition of "ESE in Practice", we are joined by Ciera Jaspan and Collin Green from the Developer Intelligence team at Google. This article is a faithful account of our conversation from August 13, 2026, which we edited for our column.
1. ESE AT GOOGLE
Google’s Developer Intelligence team adapts empirical methods to research questions, combines quantitative and qualitative evidence, and connects studies to decisions about developer experience. The team also faces limits from narrow logs, elusive metrics, stakeholder skepticism, and the context-dependent nature of software engineering.
- Research mission: The team studies developer happiness and productivity, identifies pain points, and helps tool and onboarding teams evaluate interventions.Its work supplies metrics and data for decisions about whether new features, tools, or onboarding changes are worth continuing.
- Methods and triangulation: Quantitative methods describe what happens, while qualitative methods help explain why it matters; triangulating surveys, interviews, logs, and other evidence increases confidence when findings converge.A code-review median of five minutes alone does not indicate whether the experience is good, bad, or a pain point.
- Methods and triangulation: Logs can be too narrow because team dynamics and external events, including the World Cup, elections, and the pandemic, may influence productivity beyond measured engineering interventions.The team therefore treats qualitative and survey work as necessary for capturing drivers absent from logs.
- Measurement challenges: Objective logs-based metrics remain elusive for constructs such as code quality and flow, despite curated surveys, interviews, diary studies, and qualitative benchmarks.The team seeks scalable objective measures but must balance convenience and speed against meaningful representation of engineers’ experiences.
- Choosing what to study: Research questions come from surveys, leadership concerns, engineers’ reports, and foundational questions, with more potential studies than available time.Technical debt, for example, can recur as a concern raised by either leadership or engineers.
- Using and applying results: The team prefers to know a study’s intended decision in advance, publishes selectively rather than chasing academic outputs, and sometimes pursues foundational questions without an immediate customer.Publication decisions are made against a high bar, while foundational work may be funded for future teams.
- Using and applying results: Applying results can provoke skepticism when findings conflict with stakeholders’ expectations, while long-term or narrowly distributed benefits can make prioritization difficult.Complementary diary evidence helped persuade one stakeholder that her experience was not typical, illustrating how multiple methods can address disbelief.
- Broader perspective: Software engineering research should accept multiple valid methods and examine technical and non-technical factors such as team dynamics, communication, psychological safety, and prioritization.The interview frames software engineering practice as a complex human system whose effectiveness is intertwined with these factors.
2. CONCLUSION
The interview highlights flexible, mixed-method empirical research and argues that ESE should balance rigor with methodological judgment rather than rigid dogma.
- 2. CONCLUSION: A broad empirical-method toolbox lets practitioners adapt their approach to different software-engineering problems.The interview emphasizes choosing methods according to the problem and combining quantitative and qualitative evidence.
- 2. CONCLUSION: Mixed-method research combines quantitative evidence about what happens with qualitative evidence about why, strengthening confidence and addressing method-related skepticism.Triangulation is especially valuable when different evidence sources align or when stakeholders hold preconceived expectations about acceptable methods.
- 2. CONCLUSION: Publishing negative results, tackling bold problems, and presenting holistic papers are presented as useful practices for the broader ESE community.The authors contrast these practices with highly specialized niche cases and salami-sliced research.
- 2. CONCLUSION: Methodological rigor remains important, but empirical standards should not replace expert judgment with inflexible rubrics.The conclusion calls for reflection on methodological rigidity and recognizes that more than one approach may be acceptable.