PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions

Carlo A. Furia, Richard Torkar, Robert Feldt

arXiv:2301.07524v6cs.SE

TL;DR

Empirical software-engineering studies often rely on observational correlations because large controlled experiments can be impractical, yet correlations do not establish causal effects. This paper applies DAG-based causal analysis to Code Jam data and compares causal and predictive analyses of programming-language effects. Language effects are small relative to other factors, but the two analyses yield different conclusions, and the causal analysis can help isolate effects and confounders.

  • Problem

    Observational software-engineering data can reveal associations, but those associations do not necessarily establish causal effects; causal understanding matters for estimating contributions and generalizing effects.

    Source: §1 · p. 1

  • Method

    The paper uses causal directed acyclic graphs to assess possible relations among variables and guide a statistical model intended to estimate causal effects from Code Jam observational data.

    Source: §1 · p. 2

  • Results

    Language effects are small relative to other factors, while causal and associational analyses of the same data reach nearly opposite language-related conclusions.

    Source: §1 · p. 2, §8 · p. 27

  • Takeaways & Limitations

    Even imperfect causal analysis can help isolate general interactions and improve the precision and robustness of answers to research questions about observational data.

    Source: §1 · p. 2, §8 · p. 27

    Selecting experienced participants may restrict the validity of results to that category of participants.

    Source: §7.2.1 · p. 25

Abstract

from arXiv · show

There is abundant observational data in the software engineering domain, whereas running large-scale controlled experiments is often practically impossible. Thus, most empirical studies can only report statistical correlations -- instead of potentially more insightful and robust causal relations. To support analyzing purely observational data for causal relations, and to assess any differences between purely predictive and causal models of the same data, this paper discusses some novel techniques based on structural causal models (such as directed acyclic graphs of causal Bayesian networks). Using these techniques, one can rigorously express, and partially validate, causal hypotheses; and then use the causal information to guide the construction of a statistical model that captures genuine causal relations -- such that correlation does imply causation. We apply these ideas to analyzing public data about programmer performance in Code Jam, a large world-wide coding contest organized by Google every year. Specifically, we look at the impact of different programming languages on a participant's performance in the contest. While the overall effect associated with programming languages is weak compared to other variables -- regardless of whether we consider correlational or causal links -- we found considerable differences between a purely associational and a causal analysis of the very same data. The takeaway message is that even an imperfect causal analysis of observational data can help answer the salient research questions more precisely and more robustly than with just purely predictive techniques -- where genuine causal effects may be confounded.

1 Introduction

The paper asks whether causal analysis can sharpen conclusions drawn from observational software-engineering data, using programming languages and Code Jam performance as a case study. Its causal and associational analyses reach nearly opposite language-related conclusions, despite the associational model predicting better.

1 Introduction

  • Observational studies often identify associations without establishing causal effects; causal understanding matters when estimating specific contributions and generalizing them to different scenarios.Prediction alone may suffice for some engineering applications, but it does not distinguish mere correlations from causal effects.Source: §1 · p. 1

1.1 Programming: Competitions and Languages

  • The study asks how language use relates to experienced Code Jam participants’ results, focusing on popular languages and participants who competed successfully across editions.Programming competitions offer measurable performance data under constrained conditions, although those conditions do not necessarily reflect professional software development.Source: §1.1 · p. 2

1.2 Correlation vs. Causation

  • The causal analysis reverses nearly all language-related associations from the predictive analysis: Python shifts from better-than-average to worse-than-average results, C++ from no consistent association to better results, and Java from worse results to no consistent association.The associational model nevertheless outperforms the causal model in predictive accuracy, while failing to isolate causal effects.Source: §1.2 · p. 3
  • Language effects are small relative to other factors, such as individual skill, so the case study’s main contribution is methodological rather than recommending a particular language.The authors argue that ignoring causal structure can undermine result robustness and generalizability.Source: §1 · p. 2, §1.2 · p. 3

1.3 Contributions

  • The paper demonstrates DAG-based causal analysis on a case study and argues that identifying confounders can clarify factor importance and disentangle spurious correlations.The authors make available data, scripts, additional results, and visualizations for reproducibility.Source: §1.3 · p. 3

1.4 Organization

  • The paper moves from causal and language-study background through Code Jam data, associational and causal models, their comparison, and validity threats to its conclusions.The roadmap identifies Sections 4–5 as the two analyses and Section 6 as their reconciliation.Source: §1.4 · pp. 3–4

2 Related Work

The related work spans causal-inference frameworks, applications in software engineering, and empirical studies of programming languages. It highlights the challenges of drawing robust conclusions about language effects from observational data, where confounding factors may complicate interpretation.

2.1 Causal Inference

  • Causal inference distinguishes observed associations from factors that cause outcomes; DAGs represent causal structure, and can guide interventions and effect estimation.The paper focuses on DAG-based inference and leaves comparison with other approaches, including potential outcomes, to future work.Source: §2.1 · p. 4

2.2 Causal Inference and Analysis in Software Engineering

  • A survey identified 31 causal-analysis studies in software engineering, spanning fault localization, testing, performance analysis, and other topics; many used statistical techniques without DAGs.The reviewed work also includes applications to AI development, open-source popularity, and configurable software.Source: §2.2 · p. 5

2.3 Empirical Studies of Programming Languages

  • Programming-language studies use controlled experiments, surveys, or repository data, each suited to different questions and carrying different limits on generalizability or robustness.Repository analyses can be informative for measurable characteristics such as conciseness, while noisy or heterogeneous data and confounding can complicate other comparisons.Source: §2.3 · p. 5
  • Prior studies found language-related defect associations small or context-dependent, and this paper likewise anticipates modest language effects relative to participant skills and experience.The only other identified empirical study of Google Code Jam compared language submissions on size, running time, and memory use.Source: §2.3 · p. 6

3 The Google Code Jam Data

The study analyzes public submission data from Google’s annual Code Jam contest, treating each round as a separate performance comparison. It narrows the dataset to three popular languages and experienced participants to make the analysis more focused and manageable.

3 The Google Code Jam Data

  • Code Jam is an annual online programming contest with elimination rounds, where participants submit solutions and advance according to their ranked scores.Contest details vary by year, but the general structure remains sufficiently stable for consistent analysis across editions; the study does not model progression between rounds.Source: §3 · p. 6

3.1 The Code Jam Dataset

  • The dataset covers the first seven yearly Code Jam editions, 2008–2014; each datapoint reports a participant’s results in one round, with challenge, nickname, language, and total submission size in bytes.Source: §3.1 · pp. 6–7

3.2 Selecting Programming Languages

  • The analysis retains C++, Java, and Python, the three most-used languages among 75 represented, which together account for an order of magnitude more submissions than all other languages combined.They constitute 50%, 20%, and 13% of datapoints, respectively.Source: §3.2 · p. 7

3.3 Selecting Experienced Participants

  • The language-filtered data contain 150,539 datapoints from 49,808 participants, but most participants entered only one or two editions and advanced at most to round two.The figure describes participant counts by minimum editions entered and participation rounds, with shading for the percentage of all participants and an ellipse for the analysis subset.Source: §3.3 · p. 7, Figure 2 · p. 8
  • The main analysis retains 1,896 datapoints from 105 experienced participants who entered at least two editions and submitted to at least six rounds in one edition.The authors describe this as a reasonable but non-unique selection; analyses on larger and smaller samples mostly preserve the overall answer.Source: §3.3 · p. 8

4 Correlational Analysis of Programming Language Effects

The section builds and compares Bayesian regression models to analyze programming-language associations with Code Jam rank, then checks whether the conclusions hold under frequentist analysis.

4.1 Modeling

  • The models use a negative binomial likelihood for positive-integer rank; predictors progress from language to nickname, challenge, and submission size, with m4 using log(size).The negative-binomial mean uses a logarithmic link; submission size ranges from 457 to 202 965 bytes, so m4 uses its logarithm to smooth wide variability.Source: §4.1 · pp. 9–10
  • The models use weakly informative priors focused on language intercepts, and validation checks find plausible priors, workable fits, and reasonable data adequacy.The authors note that alternative prior choices would lead to very similar overall estimates.Source: §4.1 · p. 10

4.2 Model Comparison

  • PSIS-LOO compares the models’ estimated out-of-sample predictive capabilities, and the section selects m4 for accurate prediction of associations with rank.The criterion reports uncertainty for score differences; the authors distinguish predictive model selection from the later causal analysis.Source: §4.2 · p. 11
  • 14.6 standard errors separate m4’s predictive score from m3’s; m4 outperforms all simpler models under PSIS-LOO.Model m3 also outperforms m2 by 18.5 standard errors, while m1 is 2.1 standard errors behind m2.Source: Table 1 · p. 11, §4.2 · p. 11

4.3 Analysis: Programming Language Associations

  • At 50% probability, model m4 associates Python with better results, Java with worse results, and shows no consistent C++ association.The study focuses on experienced Code Jam participants and interprets smaller rank ordinals as better results.Source: §4.3 · p. 12
  • Models m1, m2, and m3 predict language associations nearly opposite to m4, so refining the model changes the qualitative findings.The authors treat such flips as a sign that measured associations may be weak and motivate examining causal relations.Source: §4.3 · pp. 12–13

4.4 Frequentist Statistics

  • Frequentist counterparts and variable selection support the Bayesian analysis’s language associations and identify f4 as the best predictive model.The frequentist analysis also finds f4 and f3 predictions nearly opposite; the authors conclude high-level findings are largely independent of the regression approach.Source: §4.4 · p. 13
  • The frequentist analysis uses counterparts with the same predictors, outcome, negative binomial likelihood, and log-linear parameterization as the Bayesian models.It differs in technical choices including contrast encoding and flat rather than weakly informative priors.Source: §4.4 · p. 13

5 Causal Analysis of Programming Language Effects

This section develops an analysis of Code Jam data intended to identify causal effects of programming languages and compares it with the preceding associational analysis.

5 Causal Analysis of Programming Language Effects

  • The section aims to identify causal effects of programming languages in Code Jam and compare those results with the prior associational analysis.It describes the causal analysis as an analysis of the Code Jam dataset.Source: §5 · p. 13

5.1 From Correlation To Causation

Because Code Jam data are observational, associations between language and rank may mix direct language effects with confounding; the paper proposes a causal-model-guided analysis to separate them.

5.1 From Correlation To Causation

  • Statistical associations in observational data cannot by themselves identify causal relations, which depend on how the data were generated.The paper says causal effects cannot be inferred from statistical analysis alone without assumptions about the data-generation process.Source: §5.1 · p. 13
  • In a hypothetical randomized Code Jam experiment, assigning programmers, challenges, languages, and solution sizes at random would make language-rank association measure the direct language effect.The paper contrasts this with actual Code Jam, where participants choose whether to join and which language to use.Source: §5.1 · p. 13
  • Because skilled programmers may prefer a language, an observed language-rank association can mix language effects with indirect effects of skill.The paper illustrates this with skilled programmers preferring C++ and potentially achieving top ranks even if required to use Java.Source: §5.1 · p. 14
  • The proposed workflow designs plausible causal models, partially checks them against the data, then uses a consistent model to guide statistical adjustment for confounding.The resulting association is intended to estimate a causal effect subject to data uncertainty and the accuracy of the qualitative causal model.Source: §5.1 · p. 14

5.2 Qualitative Causal Models: DAGs

The paper uses causal DAGs to encode plausible relations among Code Jam variables, including unobserved skill, and to guide which variables should be adjusted for.

5.2 Qualitative Causal Models: DAGs

  • A causal DAG represents variables as nodes and direct causal relations as arrows; the Code Jam graphs include unobserved participant skill.The graphs highlight language and rank as the target causal relationship.Source: §5.2 · pp. 14–15
  • The proposed graphs include language→rank, nickname→language, language→size, nickname→skill, and skill→rank relationships.They also represent challenge as affecting rank and skill, and potentially which participants appear in a challenge’s data.Source: §5.2 · p. 15
  • The DAGs exclude arrows between size and rank because contest rules do not use submission size to determine rank, and rank follows submission.The authors also consider challenge→nickname to represent selective submissions to congenial challenges.Source: §5.2 · p. 15
  • The three DAG variants encode whether size depends on programmer preference, skill, or both.These alternatives correspond to nickname→size, skill→size, or both arrows entering size.Source: §5.2 · p. 15
  • The analysis tentatively assumes language may directly affect rank while seeking to reduce bias, and rules out rank→language because language is chosen before rank is determined.The causal effect may ultimately be significant or negligible.Source: §5.2 · p. 15

5.3 Validating DAGs on Empirical Data

The authors partially validate candidate causal DAGs by testing their implied conditional independencies against the data. This supports using d1 or d2, rather than d0, as the basis for causal analysis.

5.3 Validating DAGs on Empirical Data

  • For an implied independence X ⊥⊥Y | Z, regression tests whether Y remains associated with X after conditioning on Z; persistent association invalidates the independence.The authors caution against treating validation as a binary check, recommending that parameter uncertainty be assessed in context.Source: §5.3 · p. 16
  • The models mid1, mid2, and mid3 test the three implied independencies using distributions suited to their outcomes.mid1 and mid3 use negative binomial distributions; mid2 uses a normal distribution.Source: §5.3 · p. 16, Figure 6 · p. 17
  • Only 2% of 51 αc coefficients differ consistently from zero in mid1, versus 71% in mid2; mid3’s β interval is (−1.02, −0.95).These results support the tests: challenge and language are negligibly associated in mid1, challenge and size are associated in mid2, and size and rank are associated in mid3.Source: §5.3 · pp. 16–17, Figure 7 · p. 17
  • DAGs d1 and d2 pass the partial validation and are used as the basis for causal analysis; d0 fails.The three DAGs share one testable conditional independence, while some additional independencies involve unobserved variables.Source: §5.3 · pp. 16–17

5.4 Choosing Predictors to Measure Causality

The authors use the causal DAG to choose regression predictors that block open backdoor paths without opening closed ones. For Code Jam, m2 and m3 are suitable for estimating language’s causal effect on rank, with m3 preferred.

5.4.1 Causal and Non-Causal Paths

  • A path from language to rank contributes to their observed correlation; paths entering language are backdoor paths that can create spurious correlation, as with challenge affecting both language and rank.In DAG d2, paths I through N are backdoor paths.Source: §5.4.1 · p. 18

5.4.2 Open and Closed Paths

  • Choosing regression predictors can close open backdoor paths, but conditioning on a collider such as size can open a path and bias the estimate.In the size example, conditioning can induce a spurious language–skill association that confounds the estimated language effect on rank.Source: §5.4.2 · pp. 18–19

5.4.3 Adjustment Sets

  • An adjustment set conditions on variables that close every open backdoor path without opening any closed one; such sets may be unavailable or multiple.The constraints can be unsatisfiable, and multiple valid adjustment sets may exist.Source: §5.4.3 · p. 19

5.4.4 Unbiased Estimation for Code Jam Data

  • For DAG d2, {nickname} and {nickname, challenge} are adjustment sets; neither includes size, which should not be conditioned on.Other adjustment sets include unobserved skill and therefore cannot be used as predictors.Source: §5.4.4 · p. 19
  • Models m2 and m3 are suitable for estimating language’s causal effect on rank; m1 omits adjustment variables, while m4 conditions on size and adds confounding.The conclusion is robust across the considered DAGs; m3 is preferred because it outperforms m2 in predictive accuracy and challenge may improve precision.Source: §5.4.4 · p. 19

5.5 Analysis: Programming Language Effects

Under the causal interpretation, C++ is associated with better Code Jam results, Python with worse results, and Java has no consistent effect. This differs sharply from the correlational answer.

5.5 Analysis: Programming Language Effects

  • The causal interpretation identifies m3 as the model capturing language effects most accurately for Code Jam results.The authors revisit the model m3 intervals for each language’s contribution relative to the average language contribution.Source: §5.5 · p. 20
  • C++ is associated with better results, Python with worse results, and Java has no consistent effect, though its interval leans toward better ranks.These are the model m3 results at the 50% probability level.Source: §5.5 · p. 20
  • The causal answer differs completely from the answer based on purely correlational associations.The authors describe the two answers as nearly mirror images.Source: §5.5 · p. 20

5.6 Correlation vs. Causation

The correlational and causal answers to the language question are nearly mirror images. Predictive model preference also conflicts with causal model suitability, underscoring the need to select predictors for the intended analysis.

5.6 Correlation vs. Causation

  • The correlational and causal answers about programming languages are nearly mirror images.The causal interpretation reverses the associations reported in the correlational analysis.Source: §5.6 · p. 20
  • Model m4 leads in predictive capability but is unsuitable for causal estimation; causally consistent m2 and m3 agree on predictions essentially opposite to m4.Model m1 agrees with m2 and m3’s predictions but ranks last in the predictive comparison.Source: §5.6 · p. 20
  • Because Code Jam is not a randomized experiment, estimating causal effects requires careful selection of information in the statistical model.The authors distinguish observed associations from effects that are genuinely causal.Source: §5.6 · p. 20

6 The Modest Impact of Languages

In model m4, language effects on predicted rank are modest compared with effects associated with participants, challenges, and submission size. Centering these effects makes their relative contributions comparable, while the analysis highlights how confounding can complicate language comparisons.

6.1 Centered Effects in Model m4

  • Centered effects δℓ, δn, δc, and δ(s) express each predictor’s contribution relative to the average contribution in its group, allowing comparison on a common scale.Model m4 combines language, nickname, challenge, and size components; the plotted effects are estimates with 50% probability intervals.Source: §6.1 · pp. 21–22, Figure 9

6.2 Comparison of Centered Effects

  • The language-interval endpoints differ by 1.26, or about 3.5 on the rank outcome scale, indicating a modest and uncertain language effect.Figure 8 shows the language effects centered near zero with overlapping intervals.Source: Figure 8 · p. 21, §6.2 · p. 22
  • Nickname effects span 3.1, or 22.4 on the outcome scale—about an order of magnitude more than the corresponding language range.Changing language for a fixed participant is expected to shift rank modestly, whereas differences between participants remain substantial.Source: §6.2 · p. 22
  • Challenge effects span 5.0, or 153.2 on the outcome scale, exceeding the range attributed to language and nickname.Source: §6.2 · p. 22
  • Across the observed size range, the size effect spans 6.6, or 752.7 on the outcome scale; size is treated as a proxy for effort, not a direct cause of rank.Because size is logarithmic, a difference of s units corresponds to exp(s) times the size in bytes; increasing code alone does not improve rank.Source: §6.2 · pp. 22–23

6.3 Language vs. Other Effects

  • Knowing nickname, size, and challenge affects predicted rank more strongly than knowing language; the pattern is similar in model m3 without size.The authors argue that language effects can be hard to detect reliably because stronger factors may confound them.Source: §6.3 · p. 23
  • Explicitly modeling causal effects helped distinguish language’s limited role from its relations to other factors in the Code Jam data.Source: §6.3 · p. 23

7 Robustness and Threats to Validity

The authors test whether their findings persist across alternative Code Jam data selections and discuss threats tied to the data, models, and analysis. The robustness checks retain the contrast between causal models and the associational model, while the paper notes boundaries on generalizability and model completeness.

7.1 Analysis Robustness

  • Across 21 datasets, causal models m1–m3 usually associate C++ and Java with better-than-average ranks and Python with worse-than-average ranks; m4 shows the opposite pattern for Python and Java.The contrast generally persists, although smaller datasets have wider intervals and larger ones include more heterogeneous participants.Source: §7.1 · pp. 24–25, Figure 10 · p. 24

7.2 Threats to Validity and Analytics Smells

  • The threats discussion maps relevant analytics smells to the subsections and sections or references addressing them.Source: §7.2 · p. 25, Table 4 · p. 25

7.2.1 Data

  • The analysis uses experienced participants and three popular languages; this selection may restrict validity to that participant category, despite checks on larger and smaller subsets.Code Jam rank is computed by organizers under predefined rules, and submissions must use an available language compiler.Source: §7.1 · p. 24, §7.2.1 · p. 25

7.2.2 Modeling

  • The considered models are not definitive: richer or extended models could reveal more nuanced trends, although tested variants did not change the qualitative language conclusions.The authors also report that adding or removing some assumed causal links did not change the causal models’ implications.Source: §7.2.2 · p. 26

7.2.3 Analysis

  • Small programming-language effects limit strong generalizability claims; the authors used 50% intervals, varied the data subset, and report further sensitivity and multilevel-model checks.The additional analyses confirmed the modest relative language effect and corroborated the plain-regression results.Source: §7.2.3 · p. 26

7.2.4 Other Issues

  • A replication package supports reporting and visualization, while the authors note that causal analysis remains uncommon in software engineering data studies.Source: §7.2.4 · p. 26

8 Discussion and Conclusions

Programming-language effects are small relative to other variables, but causal analysis can still clarify interactions and improve external validity. The authors also distinguish prediction-first goals, where correlational models may suffice, from questions about causal contributions, while noting that causal hypotheses require validation and revision.

8 Discussion and Conclusions

  • Programming-language effects are small relative to other variables, weakening robustness, but causal analysis may isolate interactions with greater impact in other scenarios.The authors suggest that this can improve external validity.Source: §8 · p. 27
  • For prediction-first goals, correlational models may suffice or outperform causally consistent ones, while causal reasoning can guard against misinterpretation and overgeneralization.The paper gives predicting the next Code Jam winner as an example where model m4, or a more sophisticated model, may be preferable.Source: §8 · p. 27
  • Bayesian and frequentist models yield largely similar high-level contrasts between causal and non-causal analyses, but Bayesian posterior distributions make some analyses more flexible and straightforward.The authors prefer Bayesian statistics, while noting the analysis would still be possible with a frequentist model.Source: §8 · p. 27
  • Causal reasoning is not always the priority: predictive accuracy may matter more, causal and predictive models often agree, and causal hypotheses need validation and revision as processes change.The authors note that hypotheses may also change as the data-generating process is better understood.Source: §8 · p. 27
  • Ignoring causal relations may miss important parts of the picture, while causal reasoning prompts scrutiny of confounders and more sound interpretation.Source: §8 · p. 27
Loading 2301.07524v6…