Source-linked AI summary
Do AI Coding Agents Log Like Humans? An Empirical Study
Youssef Esseddiq Ouatiti, Mohammed Sayagh, Hao Li, Ahmed E. Hassan
TL;DR
Software logging is essential for observability, but how AI coding agents produce and follow logging requirements remains underexplored. The paper empirically compares agentic and human pull requests, analyzes explicit instructions, and tracks post-generation regulation, finding lower agent logging prevalence, weak instruction compliance, and substantial human repair work.
Problem
The study addresses the gap in evidence about whether AI agents match human logging practices and whether developers can govern observability through explicit instructions.
Method
The authors analyze 4,550 agentic and 3,276 human pull requests across 81 repositories using quantitative logging metrics plus qualitative instruction and review-comment analysis.
Results
Agents change logging less often than humans in 58.4% of repositories, explicit instructions appear in 4.7% of PRs, agents ignore requests 67% of the time, and humans perform 72.5% of repairs.
Takeaways & Limitations
The findings indicate a specification gap and a compliance gap, suggesting that optional natural-language prompts are insufficient to ensure observability.
Takeaways & Limitations
The study covers Python, Java, and JavaScript/TypeScript repositories with at least 100 stars, limiting generalization to small-scale repositories.
Abstract
from arXiv · showhide
Software logging is essential for maintaining and debugging complex systems, yet it remains unclear how AI coding agents handle this non-functional requirement. While prior work characterizes human logging practices, the behaviors of AI coding agents and the efficacy of natural language instructions in governing them are unexplored. To address this gap, we conduct an empirical study of 4,550 agentic pull requests across 81 open-source repositories. We compare agent logging patterns against human baselines and analyze the impact of explicit logging instructions. We find that agents change logging less often than humans in 58.4% of repositories, though they exhibit higher log density when they do. Furthermore, explicit logging instructions are rare (4.7%) and ineffective, as agents fail to comply with constructive requests 67% of the time. Finally, we observe that humans perform 72.5% of post-generation log repairs, acting as "silent janitors" who fix logging and observability issues without explicit review feedback. These findings indicate a dual failure in natural language instruction (i.e., scarcity of logging instructions and low agent compliance), suggesting that deterministic guardrails might be necessary to ensure consistent logging practices.
1 Introduction
AI coding agents must satisfy observability requirements, but their logging behavior and responsiveness to logging instructions remain uncertain. This study compares agents with humans, measures instruction compliance, and examines post-generation repairs.
- Motivation: Observability is a critical non-functional requirement because logging supports failure diagnosis and system-health monitoring.Logging must balance sufficient context against overhead and noisy, difficult-to-analyze output.
- Study Scope: 4,550 agentic PRs and 3,276 human PRs across 81 repositories form the empirical basis for comparing logging practices and instruction effects.The study combines quantitative metrics with qualitative analysis of instructions and review comments across the logging lifecycle.
- RQ1: 58.4% of repositories show agents changing logging less often than humans, although agents add 30% more logs per 1,000 lines of code where both groups add logs.Agents mimic human error-logging patterns but less consistently match informational context such as INFO statements.
- RQ2: 4.7% of PRs contain explicit logging instructions, and agents fail to comply with logging requests 67% of the time.The reported noncompliance is largely independent of instruction specificity.
- RQ3: Humans perform 72.5% of post-generation logging repairs, mostly through subsequent commits rather than explicit code-review requests.This creates a hidden maintenance burden in which humans regulate agentic logging after generation.
2 Background & Related Work
Prior research portrays human logging as contextual, informal, and unstable, while existing AI studies largely evaluate isolated logging tasks or broader agent outcomes. This paper instead examines logging in real agentic pull requests against human baselines.
- Human Logging: Human logging commonly appears in exception handling, return-value verification, and critical logic branches, while practices remain informal and expertise-driven.Prior work also reports critical coverage gaps and substantial lifetime modification of logging statements.
- Research Gap: The paper asks whether agents replicate human logging behaviors, including churn rates and coverage gaps, or develop distinct machine-native practices.It frames the comparison against established human patterns.
- AI Logging Studies: Prior LLM logging benchmarks report 65.9% placement accuracy but only 15.2% exact match for log messages.These results motivate examining logging behavior beyond isolated generation benchmarks.
- Contribution: The study complements benchmark evaluations with in-situ analysis of logging produced in active agentic workflows requiring human review and merging.It investigates whether over-logging and verbosity manifest in real projects.
- Contribution: Unlike studies centered on functional correctness, code structure, or acceptance rates, this work analyzes logging as an observability-focused non-functional requirement.It examines how logging is produced, instructed, and regulated in AI-authored pull requests.
3 Data Collection and Processing
The study builds a filtered dataset of human and agentic pull requests, detects logging changes with language-specific regexes, and analyzes agent instructions and developer intents.
- Data Collection: 4,550 agentic and 3,276 human PRs across 81 repositories constitute the final collection described for the study.The data come from AIDev and are filtered toward mature repositories with both contributor types.
- Data Filtering: The project-level comparison initially retains 130 repositories with at least 10 agentic and 10 human PRs, then restricts languages to Python, Java, and JavaScript/TypeScript.The language restriction supports well-defined logging-statement identification.
- Logging Detection: Logging changes are identified from pull-request patches using language-tailored regexes that exclude build artifacts, binaries, minified code, and generic print statements.Patterns cover Python, Java, and JavaScript/TypeScript logging idioms.
- Validation: 96% precision and 94% recall are obtained from manual validation of 380 diffs for the regex-based detection approach.The validation used a representative sample with a 95% confidence interval and 5% margin of error.
- Data Filtering: The final dataset contains 77 repositories after excluding four with no logging changes in either agentic or human PRs.This exclusion supports analyses requiring observable logging changes.
- Instruction Analysis: Instructions are collected from linked issues, repository-level instruction files, and review comments, then classified by logging intent as Add, Remove, or Modify.A multi-agent LLM-as-judge protocol assigns labels from independent model votes after prompt refinement against manually annotated samples.
4 Results
Agents differ from humans in how often they modify logging, the density of logs they add, and their use of logging conventions. Explicit logging instructions are uncommon and do not reliably change agent behavior, while repository-instruction compliance is also constrained by uncertain visibility.
- RQ1: Agents add 65% more logs per 1,000 LOC in the 48 repositories where their changes are smaller than humans’ changes.Overall density differences are largely a composition effect because log density decreases as pull-request size increases for both groups.
- RQ1: Agents broadly match human log-message length, levels, and placement, but diverge most for INFO and WARN levels and conditional or loop contexts.INFO is the level where humans exceed agents in 24.7% of repositories, while WARN has 48.1% overall similarity; humans log more in conditionals in 28.6% and loops in 32.5%.
- RQ1: 58.4% of repositories show agents changing logging in fewer pull requests than humans, with a median prevalence score of 0.45.The typical repository therefore shows lower agent logging-change prevalence than human logging-change prevalence.
- RQ2: Only 4.7% of 1,308 observable agentic pull requests have any logging instruction across linked issues or repository instruction files.The two instruction sources have zero overlap: 15 pull requests receive issue-only instructions and 46 receive repository-file-only instructions.
- RQ2: 67% of logging requests are unmet, and detailed wording yields only 27% compliance for issue instructions and 6.5% for repository instructions.At the issue level, 11 of 15 instructions are strong but only 3 comply; all 46 repository-file instructions are strong, with 3 complying.
- RQ2: Logging instructions do not increase logging changes: instructed agents changed logging in 14.8% of cases versus 20.8% for uninstructed agents, with χ2=1.32 and p=0.25.Repository-instruction non-compliance may reflect either instructions not being surfaced to agents or being surfaced but ignored.
RQ3. Is agentic logging regulated post generation, and by whom?
Post-generation logging regulation is common for both agentic and human PRs, occurs mainly early in the PR lifecycle, and remains heavily human-driven. Regulation is concentrated in larger PRs, while explicit review feedback is rare.
- Method: The study reconstructs each agent-introduced logging statement from initial commit to final merged state and attributes modifications or deletions to humans or bots.It combines authorship metadata, review-comment classification, and Kaplan–Meier survival analysis.
- Revision timing: Logging revisions occur primarily in the first few follow-up commits for both human and agentic PRs, with human survival curves dropping faster and lower.Kaplan–Meier analysis tracks PRs whose first commit already contains logging changes and marks the first later logging edit as the event.
- Explicit review feedback: 2.18% of agentic PRs and 2.17% of human PRs contain explicit logging feedback in review comments.Among PRs whose initial commit already contains logging changes, the rates remain low at 5.80% and 6.00%, respectively.
- PR size: Post-generation logging regulation is concentrated in larger PRs: agentic PRs with log changes have median 2,702 LOC versus 231 LOC when logging remains unchanged.The difference is statistically significant (p < 0.001) with a large effect size (Cliff’s δ = 0.688).
- Post-generation regulation: 77.2% of agentic PRs and 81.6% of human PRs with introduced logs undergo post-generation logging revisions.These revisions occur before merging and do not significantly reduce the effort required to regulate logging.
- Who regulates logging: 54.5% of revised agentic PRs are changed exclusively by humans, who perform 72.5% of all post-generation log modifications.Human intervention therefore remains the dominant source of logging regulation after generation.
5 Implications
The findings expose a disconnect between agent-generated code and maintained observability. The paper therefore recommends deterministic enforcement, stronger logging-oriented training, and explicit review accountability.
- Natural-language guidance: 4.7% of the time, developers provide explicit logging instructions, while agents fail to comply with constructive requests 67% of the time.The paper concludes that natural-language guidance alone is unreliable for enforcing observability requirements.
- Guardrail-driven development: Tool builders should replace optional logging prompts with deterministic guardrails such as observability-focused linters or CI/CD checks before pull-request submission.The proposed design treats logging as a hard, verifiable constraint.
- Logging behavior: Agents mimic human error-logging patterns but significantly underuse INFO-level logging compared with human developers.The paper interprets this skew as evidence that current models treat logging primarily as reactive failure capture rather than proactive state tracking.
- Training implications: Future training datasets or reward models should emphasize the semantic value of state-transition logging and runtime system narratives.The discussion names specialized training and RLHF as possible directions.
- Maintenance burden: Humans perform 72.5% of post-generation log repairs, acting as silent janitors through implicit fixes in subsequent commits.The paper characterizes this as a hidden maintenance tax associated with agentic coding.
- Review practice: Observability should become a first-class PR review checklist item, with reviewers encouraged to reject uninstrumented agentic changes rather than silently absorbing technical debt.This recommendation directly targets the implicit human repair burden.
6 Threats to Validity
The study’s validity is bounded by classification subjectivity, unobserved ephemeral instructions, regex-based logging detection, and a repository sample limited to three language ecosystems and projects with at least 100 stars.
- Internal validity: Intent classification for logging instructions is potentially subjective, although an LLM Jury achieved Cohen’s κ = 0.83 against manually annotated ground truth.The jury used GPT-4o, GLM-4.7, and DeepSeek to triangulate labels.
- Internal validity: Non-compliance may reflect agents failing to access repository instruction files because of context-window limits, although the authors argue behavioral alignment is more likely.The dataset uses frontier models with 128k+ token context windows and modern context-compaction strategies.
- Internal validity: IDE chat instructions are not captured, creating a blind spot for ephemeral guidance delivered before PR generation.The authors argue that persistent logging gaps nevertheless suggest such instructions, if present, do not drive observability strongly.
- Construct validity: Regex-based static analysis may miss dynamic logging patterns or custom wrappers, despite achieving 96% precision and 94% recall on 380 validated code diffs.Language-specific patterns were used for Python, Java, and JavaScript/TypeScript while build artifacts were excluded.
- External validity: The study covers Python, Java, and JavaScript/TypeScript repositories with at least 100 stars, so findings may not generalize to small-scale repositories.The authors note that specific compliance rates may change as LLM capabilities evolve.
7 Conclusion
This empirical study compares agentic and human logging across 4,550 pull requests in 81 repositories, examining implementation, instruction response, and post-generation regulation. Agents modify logging less often than humans in 58.4% of repositories, ignore explicit instructions 67% of the time, and leave humans performing 72.5% of repairs.
- Scope and approach: The study analyzes 4,550 pull requests from 81 open-source repositories to compare agent logging with human practice and examine post-generation regulation.It investigates implementation patterns, responses to instructions, and subsequent maintenance.
- Principal findings: Agents modify logging less often than humans in 58.4% of studied repositories, while explicit instructions appear in only 4.7% of PRs and are ignored 67% of the time.The paper describes this as a specification gap combined with a compliance gap.
- Conclusion: Humans perform 72.5% of post-generation log repairs, acting as silent janitors who maintain observability through later commits.The findings suggest that optional prompts are insufficient and motivate deterministic enforcement mechanisms such as CI/CD linters.