PaperlayerView exact revision on arXiv

Source-linked AI summary

Following Breadcrumbs in Code: What Accidentally Committed Ad-Hoc Logs Reveal about Developer Comprehension

Yi-Hung Chou, Boyuan Jiang, Vidit Jain, Yiyang Min, April Yi Wang, James A. Jones

arXiv:2610.11041v1cs.SE

TL;DR

Ad-hoc logs are widely used to understand runtime behavior, but their temporary, local nature has limited systematic study. This paper mines accidental commits and analyzes live-streamed sessions to study their use across Java, JavaScript, and Python. Logs are concentrated in difficult runtime contexts and higher-complexity functions, with production logs showing a similar complexity association.

  • Problem

    Ad-hoc logs remain understudied because their transient, local nature constrains systematic investigation.

    Source: §1 · p. 2

  • Method

    The study mines accidental commits containing removed logs and analyzes live-streamed programming sessions to investigate ad-hoc logging.

    Source: §8 · p. 28

  • Results

    Ad-hoc logs cluster in difficult runtime contexts and higher-complexity functions, while production logs also tend to occur in higher-complexity functions.

    Source: §8 · p. 28

  • Takeaways & Limitations

    The findings suggest opportunities for tools that recommend logging locations, improve log readability, or visualize execution traces.

    Source: §8 · p. 28

    Commit validation estimates precision among matched commits but cannot measure recall among unmatched commits, and the stricter filter does not guarantee perfect precision.

    Source: §5.1 · p. 11

Abstract

from arXiv · show

Developers frequently insert temporary print or log statements, known as **ad-hoc logs**, to better understand program behavior at runtime, particularly when facing unexpected issues or complex control flows. Despite being a nearly universal practice, systematic study has been limited because these logs are ephemeral: they usually remain only in local environments and are removed before code is committed, making them difficult to capture. In this work, we addressed this challenge by mining accidental commits where developers unintentionally left ad-hoc logs and later deleted them, and by analyzing live-streamed programming sessions to observe their use in practice. Using these methods, we constructed a large dataset across three major programming languages (Java, JavaScript, and Python), enabling the large-scale investigation of logging practices. Our analysis reveals both common and language-specific patterns in where and how developers rely on ad-hoc logs. Across languages, ad-hoc logs tend to appear in program regions that are harder to reason about at runtime. We also identify distinctive language-level patterns, such as frequent use in asynchronous and callback functions in JavaScript and in thread-related classes in Java. In addition, functions containing ad-hoc logs generally have higher cyclomatic complexity than the overall function population. Production logs show a similar association, consistent with logging serving as a means of observing runtime behavior in structurally complex functions. Together, these findings provide empirical insight into developers' runtime comprehension practices and offer a valuable dataset for researchers and tool builders seeking to better support debugging and logging.

1 Introduction

Ad-hoc logs are temporary debugging and comprehension aids that have been difficult to study systematically. This study uses accidental commits and live-streamed sessions to examine their use and patterns across three languages.

1 Introduction

  • Ad-hoc logs are typically removed before deployment, leaving their transient and local nature a barrier to systematic study.The study uses corrective commits that remove mistakenly committed logs as recoverable traces.Source: §1 · p. 2
  • The study combines GitHub commits that capture mistakenly preserved logs with live-stream observations of temporary logging in practice.The commit data provides broad coverage, while the live-stream data provides depth and qualitative understanding.Source: §1 · p. 3
  • Across Java, JavaScript, and Python, ad-hoc logs concentrate in challenging runtime contexts, including JavaScript asynchronous and callback functions and Java thread-related classes.They also occur more often in high-cyclomatic-complexity functions than in simpler ones.Source: §1 · p. 4
  • The study contributes a dataset of 882,438 removed logs across JavaScript, Java, and Python, alongside live-coding analysis and a mining method for recovering hidden logs.The commit collection includes associated metadata from GitHub APIs.Source: §1 · p. 4

2 Background & Motivation

The background motivates studying ad-hoc logs as traces of how developers understand dynamic behavior, distinguishing them from persistent production logs. It positions this study’s scalable analysis against prior work and its limits.

2.1 Comprehending Dynamic Behaviors of Software

  • Understanding dynamic behavior, such as evolving program states and complex execution flows, is a demanding part of software comprehension.Prior evaluation approaches trade qualitative nuance and scalability, while structural proxies may not reflect developers’ actual cognitive challenges.Source: §2.1 · pp. 4–5

2.2 Tracking Back in the Forest: Ad-hoc vs. Production Logs

  • Ad-hoc logs are temporary debugging breadcrumbs, whereas production logs are persistent markers for monitoring and maintenance.Prior ad-hoc logging research included observational work with 12 JavaScript programmers; this study instead analyzes version-control data at scale.Source: §2.2 · pp. 5–6
  • Research on production logs spans why, how, where, and what developers log, providing frameworks and methods that inform this study.Prior production-log placement studies also inform the study’s comparison of ad-hoc log locations.Source: §2.2 · pp. 5–6

3 Research Questions

The study organizes its analysis around four questions about ad-hoc logs: their locations, content, motivations, and manipulation.

3 Research Questions

  • RQ1 asks where in the codebase developers place ad-hoc logs.Source: §3 · p. 7
  • RQ2 asks what information developers record in ad-hoc logs.Source: §3 · p. 7
  • RQ3 asks why developers rely on ad-hoc logs.Source: §3 · p. 7
  • RQ4 asks how developers manipulate ad-hoc logs, including inserting, revising, or removing them.Source: §3 · p. 7

4 Dataset Curation

The study curates commit-based ad-hoc log data and live-stream observations, then assembles production-log and repository samples for comparison. Its collection combines large-scale mining with qualitative analysis.

4 Dataset Curation

  • The study curates ad-hoc logs from Java, Python, and JavaScript/TypeScript projects and live-stream recordings of developers logging for debugging or comprehension.The datasets include commit, repository, and before-and-after code metadata.Source: §4 · p. 7
  • A commit-message regular expression, developed from sampled removals and clustered message patterns, enables large-scale identification of commits removing ad-hoc logs.The researchers sampled 2,000 repositories per language, selected 100 representative messages, and used their common linguistic structure to construct the filter.Source: §4 · p. 7
  • Commit collection combines GitHub Archive PushEvents from 2011–2020 with supplementary BigQuery data extending coverage through October 2024.For Java and Python, BigQuery PushEvents from 2015–2022 were also collected; GitHub APIs supplied commit files and diffs.Source: §4 · pp. 7–8
  • Production logs from the same repositories provide a like-for-like location comparison with ad-hoc logs.The collection yielded 390,191 Python, 163,033 JavaScript, and 807,189 Java production logs.Source: §4 · p. 8
  • Repository samples establish baselines for assessing whether log placement reflects complexity, threading, and asynchrony rather than convention alone.The live-stream analysis used all 15 curated videos, totaling 36:01:30, and identified log insertion through debugging-episode segments.Source: §4 · pp. 8–9

5 Data Analysis

The section prepares mined ad-hoc logs for analysis by validating candidate removals and enriching code context, then analyzes coded live-stream episodes to characterize logging behavior.

5.1 Analyzing Collected Ad-hoc Logs

  • Deleted log lines were checked against diffs, then source ASTs supplied enclosing blocks, functions, and metadata including function complexity.For JavaScript anonymous functions, the analysis inferred names for later function-name analysis.Source: §5.1 · pp. 9–10, Figure 2
  • The loose regex required a removal verb and debugging-statement noun, while the strict regex additionally required an adjective indicating debugging or accidental intent.For manual validation, statements were classified from their changes and surrounding code; temporary checkpoints and variable prints counted as ad-hoc, while user-facing output did not.Source: §5.1 · p. 10
  • Manual validation classified 98.14% of loose JavaScript, 86.52% of loose Java, and 80.87% of loose Python sampled statements as ad-hoc.Under strict filtering, the corresponding proportions were 100%, 92.77%, and 95.83%; the loose scheme collected 548,880 JavaScript logs versus 15,090 under the strict scheme.Source: §5.1 · p. 11
  • Validation estimates precision among regex-matched commits, not recall among unmatched commits, and even strict filtering may include inaccurate or recycled commit messages.The authors therefore describe the loose-to-strict reduction as a coverage tradeoff, not a measured recall value.Source: §5.1 · p. 11

5.2 Qualitative Analysis of Live Streaming Data

  • The live-stream analysis replayed labeled log-related episodes and used guiding questions about why logs were added, removed, and what they captured.Two coders calibrated on five videos with Cohen’s κ = 0.849; the analysis covered 15 videos totaling more than 36 hours and identified 92 ad-hoc logs.Source: §5.2 · pp. 11–12

6 Results

Across JavaScript, Python, and Java, ad-hoc logs cluster in function bodies and runtime-complex contexts, while their contents and uses reflect developers’ efforts to inspect program state and execution. Production logs show different placement patterns, and live-stream observations reveal varied approaches to creating and removing ad-hoc logs.

6.1 RQ1: Where in the codebase do developers place ad-hoc logs?

  • Ad-hoc logs most often appear directly in function or method bodies, whereas production logs are more concentrated in conditional and exception-handling blocks.The study cautions against comparing absolute placement percentages across language datasets because their collection and validation differ.Source: §6.1 · pp. 13, 14, 16
  • 47.6% of ad-hoc logs appeared in asynchronous functions by August 2024, compared with 4% of functions in the repository sample being asynchronous.JavaScript ad-hoc logs also occurred in callback and lifecycle functions, including render, constructor, and componentDidMount.Source: §6.1 · pp. 17–19
  • 2.21% of classes containing removed ad-hoc logs were concurrency-type classes, versus 0.42% in the Java baseline, a risk ratio of 5.30.These percentages reflect direct extends/implements relationships detected by the study, not all forms of Java concurrency.Source: §6.1 · pp. 17–19
  • Functions containing removed ad-hoc logs had higher observed cyclomatic complexity than sampled overall functions across the studied datasets.The comparison is summarized in Table 4 and Figure 8.Source: Table 4 · p. 19, §6.1 · p. 20, Figure 8 · p. 20

6.2 RQ2: What information do developers record in ad-hoc logs?

  • Developers often labeled and formatted ad-hoc log output to connect values to their variables and understand execution order.When two arguments were used, 84.7% of JavaScript and 76.8% of Python logs included at least one literal; live-stream logs showed a similar label-and-variable format.Source: §6.2 · pp. 21–22

6.3 RQ3: Why do developers rely on ad-hoc logs?

  • Developers used ad-hoc logs to inspect unclear program states and verify execution flow or assumptions, especially around asynchronous functions and third-party libraries.The live-stream analysis identified 68 instances concerning program states and 33 concerning execution flow.Source: §6.3 · pp. 22–23
  • Developers iteratively refined log content or placement and used multiple logs to compare values and execution order.The analysis observed 27 iteration instances and 22 comparison instances.Source: §6.4 · pp. 23–24
  • Most observed ad-hoc logs were removed after serving their immediate purpose, though removal approaches varied and some logs were committed.Many were deleted after output inspection (n=61); 80 were observed or confirmed eventually removed, while five were committed, including two accidental commits.Source: §6.4 · pp. 23–24

7 Discussion

Ad-hoc logs cluster in code that is harder to reason about at runtime, and functions containing them show higher observed cyclomatic complexity. The discussion connects these patterns to the limits of the mined data and to opportunities for better measurement and developer support.

7 Discussion

  • Revision-control data provides scale but captures only a snapshot and not developers’ rationale; live-stream analysis adds context about log creation, evaluation, and deletion.The live-stream analysis was used to gain insight into context and reasoning that surrounding source code cannot capture.Source: §7 · pp. 24–25
  • Ad-hoc logs occur in threading-related code, asynchronous functions, callbacks, and third-party lifecycle methods; functions containing them also show higher cyclomatic complexity across all three languages.The complexity association also appears among production logs, so it is not unique to ad-hoc logging.Source: §7 · p. 25
  • The dataset offers a complementary empirical lens for examining code complexity measures and may help identify challenging code for refactoring or instruction.The authors suggest further validation of established metrics and note React component lifecycles as a possible case for API reconsideration or visualization.Source: §7 · p. 25
  • Developers often place logs iteratively, treating initial placements as tentative probes and refining them until they provide enough information about program states and flows.The authors identify recommendations for more effective initial logging locations and variables as future work.Source: §7 · p. 26
  • Logs often include labels, formatting, custom helpers, anchor strings, and selectively structured objects to compare program states and trace execution over time.The authors suggest IDE support for discriminative anchors, relevant variable groupings, and readable object summaries.Source: §7 · p. 26
  • Exploratory recursion checks found Python recursion prevalence of 1.067% versus 0.884%; excluding recursive functions shifted the complexity difference only from 32.65% to 32.45%, while Java means remained 4.82 versus 1.87.The recursion analysis was post hoc and not part of the confirmatory RQ1 findings.Source: §7 · pp. 26–27
  • The qualitative observations are exploratory: the live-stream sample is relatively small, may involve performance bias, and is skewed toward personal or inactive projects.The authors also note that a second researcher and iterative discussions helped mitigate coding uncertainty.Source: §7 · pp. 25, 27

8 Conclusion

The study combines accidental-commit mining and live-stream observations to examine ad-hoc logging at scale. It finds these logs cluster in difficult runtime contexts and complex functions, and suggests ways to support debugging and logging.

8 Conclusion

  • The study combines more than 800,000 removed log statements across JavaScript, Java, and Python with 36 hours of qualitative observations.It presents a large-scale empirical study of ad-hoc, or “printfstyle,” logging using accidental commits and live-streamed programming sessions.Source: §8 · p. 28
  • Ad-hoc logs concentrate in hard-to-reason-about execution contexts, and both ad-hoc and production logs tend to occur in functions with higher cyclomatic complexity.Language-specific patterns include JavaScript asynchronous and callback functions and Java thread-related code.Source: §8 · p. 28
  • The findings suggest tools could recommend logging locations, improve log readability, or visualize execution traces to reduce printf debugging’s trial-and-error burden.The approach also shows how accidental version-control artifacts can reveal otherwise hidden software practices.Source: §8 · p. 28
  • Aggregated ad-hoc logs offer researchers and tool builders a window into developers’ comprehension struggles and strategies.They connect lightweight, local debugging practices with more formalized approaches to runtime understanding.Source: §8 · p. 28

9 Declarations

The declarations identify study support, author contributions, data availability, conflicts of interest, and clinical trial status.

9.1 Funding

  • The study received partial support from the Dieter Schwarz Stiftung Foundation, and Yi-Hung Chou received partial support from the Taiwan Government Scholarship to Study Abroad.Source: §9.1 · p. 28

9.4 Author Contributions

  • Yi-Hung Chou led the study and developed shared scripts, while other authors conducted language-specific experiments, qualitative analysis, or supervised the work.Source: §9.4 · p. 29

9.5 Data Availability Statement

  • The study datasets, manual checks, replication package, and additional language-specific materials are available through figshare.Source: §9.5 · p. 29

9.6 Conflict of Interest

9.7 Clinical Trial Number

  • The clinical trial number is not applicable, and the authors acknowledge reviewers and other colleagues for feedback and support.Source: §9.7 · p. 29
Loading 2610.11041v1…