PaperlayerView exact revision on arXiv

Source-linked AI summary

A systematic literature review on logging smell detection

Nora Madi, Manal Binkhonain

arXiv:2609.04215v1cs.SE

TL;DR

Logging smells can reduce the usefulness and quality of software logs, but existing detection research lacks a systematic synthesis. This paper conducts an SLR of 21 studies, maps addressed smells to a taxonomy, and examines techniques, datasets, and evaluation; it finds fragmented coverage and inconsistent validation, motivating standardized benchmarks, improved tools, and LLM-based approaches.

  • Problem

    Existing logging-smell detection research lacks a systematic understanding of its targeted issues, techniques, datasets, and evaluation practices.

  • Method

    The study conducts an SLR of 21 studies, maps logging smells to an existing taxonomy, and analyzes detection techniques, datasets, and evaluation strategies.

  • Results

    The literature is fragmented, with variation in addressed issues and a lack of shared benchmarks, standardized datasets, consistent validation methods, and unified task definitions.

  • Takeaways & Limitations

    Future work should prioritize community-agreed benchmarks, shared tasks, tooling support, and exploration of LLM techniques for log issue detection.

  • Takeaways & Limitations

    The review focuses on detection aspects even when primary studies also address repair, which may overemphasize detection elements that were not their central contribution.

Abstract

from arXiv · show

Context:Logging is an important part of software development that helps developers monitor systems, understand behavior, and fix problems. But when logging is done poorly, it can introduce logging smells, which are defects that reduce the usefulness of logs or even make them problematic. Objective:This study looks at how logging smells are currently detected. The goal is to better understand the existing research on automatic detection techniques, datasets, and evaluation methods. Method:We conducted a systematic literature review (SLR) of 21 studies focused on detecting logging smells. In this review, we define key logging-related terms, identify and map the types of smells to an existing taxonomy, and examine the detection techniques, datasets, and evaluation strategies used across the studies. Results:We found that the research is still scattered and inconsistent. For example, there is no common benchmark or standardized approach for evaluating results, making it difficult to compare studies. In addition, we observe inconsistencies in the way log smells are addressed, as studies differ in the types and number of smells they target. Conclusion:There is still room for improvement in how logging smells are studied and detected. We point out several challenges and suggest future directions, such as developing better tools, using large language models (LLMs), and building more standardized datasets for evaluation.

1. Introduction

This study systematically reviews automatic logging-smell detection research to characterize targeted issues, techniques, datasets, and evaluation practices. It identifies inconsistent coverage and assessment practices and proposes directions including LLM-based detection, benchmarking, and tooling.

  • An SLR of 21 studies examines logging-smell types, detection techniques, datasets, and evaluation strategies.
  • Different studies target different issue types and numbers, use varied terms and datasets, and apply inconsistent evaluation methods.
  • These inconsistencies hinder comparison, reproducibility, and consistent extension of existing work.
  • The review maps existing software log smells to a structured taxonomy, exposing underexplored types for future research.
  • Future directions include LLM-based detection, benchmarking, and integration with tooling.

2. Background

The background defines logging-related concepts and frames logging smells as quality problems within software logging. It introduces an existing taxonomy of smells, causes, and consequences, while noting gaps in current tool support.

  • Software Logging: A log statement emits runtime information using a verbosity level, static message text, and dynamic runtime variables.
  • Software Logging: Software logging spans deciding what and where to log, instrumenting source code, storing generated logs, and analyzing log data.
  • Logging Smells: Logging smells are treated as a subset of logging issues involving poor practices that can reduce log readability, maintainability, or effectiveness.
  • Software Log Smell Taxonomy: The taxonomy includes ten smells, including inconsistent formats, missing identifiers, incorrect severity levels, misleading variables, and missing or scarce log entries.
  • Software Log Smell Taxonomy: Reported causes include missing guidelines and insufficient tooling, while consequences include information leakage, incorrect event ordering, and performance degradation.
  • Software Log Smell Taxonomy: Existing tools support some smells but leave others, such as Format turmoil, inadequately supported and often address only specific issue types.

3. Related Work

Earlier reviews address broad logging quality and practice concerns, whereas this study focuses specifically on log-smell detection. It synthesizes detection issues, techniques, and research gaps to provide a more focused overview.

  • Chen and Jiang reviewed 69 papers, including 19 on logging quality, and grouped challenges into usability, diagnosability, logging code quality, and security compliance.
  • He et al. surveyed 158 papers, including 25 on logging, emphasizing diagnosability, maintenance, performance, and logging anti-patterns.
  • Gholamian and Ward reviewed 14 primary studies within logging practices and issues, identifying opportunities for automated detection and benchmark-quality logging statements.
  • Gu et al. found that research emphasizes where-to-log and what-to-log, while why-to-log and how-well-to-log receive less attention.
  • Batoun et al. reviewed 204 logging studies, including seven focused on log statement quality and related security aspects.
  • This study differs by concentrating on log-smell detection and consolidating techniques, challenges, and future opportunities across the reviewed literature.

4. Methodology

The study uses a systematic literature review guided by Wohlin’s snowballing approach to identify and analyze research on log issue detection. It maps reported smells to an existing taxonomy and extracts data addressing the review questions.

  • Research design: The review follows Wohlin’s snowballing approach to identify relevant research on logging issues and detection techniques.The process examines seed studies, backward citations, and forward citations iteratively.
  • Snowballing execution: The initial seed set contained 4 papers, and backward snowballing added 10 papers after four iterations examining 895 references.The resulting seed set comprised 14 papers.
  • Snowballing execution: Forward snowballing examined 826 studies across four iterations and added 7 relevant papers, producing a final review set of 21 papers.Studies were screened using inclusion and exclusion criteria before final selection.
  • Data extraction: The review extracts paper metadata and additional data items designed to address all research questions.Extracted metadata includes titles, authors, publication year, publication type, venue, source database, and citation counts.
  • Data synthesis: Reported logging smells are mapped to Saarimäki et al.’s taxonomy to organize categories, identify patterns and gaps, and support cross-study comparison.The mapping was refined through repeated reviews of the taxonomy and primary-study descriptions.
  • Problem definition: The review also examines how studies frame their research problems because logging research addresses diverse issues and combinations of issues.This analysis considers how researchers ground and justify the specific problems they investigate.

5. Results

The reviewed literature has grown in recent years, but reproducibility resources remain inconsistent across the 21 studies. Publication activity increased after 2017 and peaked in 2023.

  • Publication venues: The reviewed studies span conference and journal venues, with a conference-to-journal ratio of 13:8.Most studies appeared in established software engineering venues.
  • Publication trends: Research on log smell detection increased after 2017 and reached its publication peak in 2023.The review describes steady attention and continued contributions in later years.
  • Reproducibility: 13 studies provided accessible datasets, whereas 8 studies did not include dataset links.Dataset availability was assessed as part of the review’s reproducibility analysis.
  • Reproducibility: 10 of 21 studies shared both code and data, while 8 provided no dataset links and 8 provided no code links.The review treats shared code and data as supporting replication and extension.

5.1. RQ1: What types of logging smells do researchers aim to study? And how do they ground their problem?

The review identifies 17 logging smells and finds that research concentrates on incorrect log levels and message-related problems. Most studies ground their problem definitions in empirical analyses rather than literature gaps.

  • Logging-smell coverage: The review identifies 17 logging smells, 3 consequence types, and 2 cause types across the selected studies.The taxonomy also links smell categories with possible causes and consequences.
  • Logging-smell coverage: Incorrect log levels appeared in 8 papers, while wrong or imprecise messages appeared in 6 and language issues or typos in 5.These were the most frequently reported smell facets in the review.
  • Logging-smell coverage: Most other smell types appeared in only one or two papers, indicating uneven coverage across known log quality issues.The literature concentrates on a small set of recurring smell types.
  • Problem definition and grounding: Only 3 of 21 studies, or 14%, primarily select target smells by referring to gaps in prior literature without explicit empirical investigation.This approach forms a minority of the identified problem-definition strategies.
  • Problem definition and grounding: 85% of reviewed studies define log smell problems through empirical analysis, including preliminary, qualitative, or quantitative investigations.These investigations use artifacts such as logging code, commit history, and issue reports.

5.2. RQ2: What techniques are most commonly used to detect logging smells?

The reviewed studies use static analysis, machine learning, deep learning, and LLMs to detect logging smells, with newer work increasingly adopting deep learning and LLMs. Approaches differ in whether they use separate components or unified solutions for multiple smell types.

  • Detection techniques: Four main detection techniques are used: static analysis, machine learning, deep learning, and LLMs.Static analysis examines source code without execution, while learning-based approaches identify patterns from data.
  • Machine learning: Machine-learning studies commonly use supervised algorithms such as Random Forest, Naïve Bayes, Logistic Regression, and Decision Trees.These models predict attributes such as verbosity levels or the presence of logging anomalies.
  • Deep learning: Deep-learning studies use neural architectures including BiLSTM and Transformer encoders for classification and linguistic assessment of log messages.QuLog also uses explainable-AI techniques to highlight influential tokens for improving log clarity and correctness.
  • Technique trends: From 2012 to 2021, static analysis and traditional machine learning dominated, while deep learning increased from 2022 and LLMs appeared in 2024.
  • Multiple smells: Some recent LLM frameworks detect multiple defects with one tool, but S01 and S02 do not handle cases where a log statement has multiple smell labels.Other studies instead use separate models or components for each smell type.

5.3. RQ3: How do existing studies assess the validity and effectiveness of their detection techniques for log-related issues in software systems? And what datasets are used?

Studies assess detection techniques through manual validation, ground truth, metrics, real-world projects, baselines, and developer feedback, while datasets are assembled and labeled from open-source projects. Evaluation practices, project selection, and programming-language coverage vary substantially, limiting comparability.

  • Assessment methods: 17 of 21 studies use manual author validation, while 13 evaluate against ground truth and 11 incorporate developer feedback.Ground truth includes annotated datasets, known fixes, manually labeled examples, and synthetic data.
  • Assessment methods: 17 studies report quantitative metrics such as precision, recall, F1score, accuracy, or AUC, but metric choices differ across studies.Only 8 studies compare against existing baselines, and those baselines are inconsistent.
  • Datasets: Open-source projects such as Apache Tomcat, Hadoop, and CloudStack are commonly used, but researchers customize extraction and manual labeling for log issue detection.Dataset construction typically includes project selection, logging-statement or change collection, and labeling.
  • Programming languages: Approximately 86% of reviewed studies focus on Java-based projects, with only a few addressing Python, Angular, Ruby, PHP, or C/C++.
  • Comparability: The review suggests that established benchmarks are lacking because datasets, baselines, metrics, and task definitions are not shared or standardized.This makes it harder to reproduce results and compare findings fairly.

5.4. RQ4: What are the challenges, research gaps, and future directions in logging issue detection?

The review identifies challenges spanning inconsistent guidelines, diverse defects, extraction errors, classification errors, language bias, cross-project variation, and data imbalance. It recommends shared benchmarks, stronger NLP and adaptive methods, broader language and project coverage, dynamic information, feedback integration, and LLM-based tooling.

  • Challenges: Logging smell detection lacks standardized guidelines for what constitutes a good or bad log statement.This inconsistency complicates both development practice and automated quality assessment.
  • Challenges: Logging issues range from incorrect verbosity and duplicate messages to readability, inconsistency, and missing statements, requiring multiple analysis techniques.
  • Challenges: Log-statement extraction can miss instances: S13 used 26 regular expressions, while Log Sculptor initially detected only 10.6% with GPT-4.Log Sculptor improved detection after adding deterministic preprocessing.
  • Challenges: Detection methods face false positives, false negatives, language constraints, cross-project generalization problems, and imbalanced data.Models trained on one project may struggle on another because vocabulary, numerical feature distributions, and coding conventions differ.
  • Future directions: Recommended technical directions include advanced NLP, adaptable cross-project models, LLM-integrated tools, dynamic runtime information, and multi-language detection.Broader evaluation should include closed-source systems and languages beyond Java.
  • Future directions: Future work should develop shared datasets, unified task definitions, common baselines, and community-accepted logging standards.These measures are proposed to improve consistency and fair comparison across studies.

6. Threats to Validity

The review identifies validity concerns related to study selection, selective attention to detection, subjective categorization, and mapping logging issues to an existing taxonomy.

  • Study selection: The snowballing search may have missed some important studies, despite testing different search strings to reduce this risk.The authors believe any missing studies are likely few and would not significantly affect the findings.
  • Scope of analysis: Focusing on detection aspects in studies that also address repair may overemphasize detection elements that were not the original work’s central contribution.The review explicitly limits its analysis to detection-related information.
  • Categorization: Categorizing studies based on interpretation may introduce subjectivity, although the authors refined categories consistently and checked them against the original studies.Alternative interpretations may exist even though the review used observed patterns and existing terminology.
  • Taxonomy mapping: Mapping logging issues to an existing taxonomy may introduce interpretation bias when studies cover multiple areas or authors’ intentions are unclear.The authors reduced this threat by studying the taxonomy and the context of each primary study.
  • Scope boundary: The conclusions apply only to detection-focused research and should not be generalized to repair-focused research.Some included studies address repair, but the review concentrates solely on detecting logging issues.

7. Conclusion

The review provides a structured overview of logging-issue detection research and finds substantial fragmentation in issues, benchmarks, datasets, validation methods, and task definitions. It points toward LLM-based approaches, shared benchmarks, and tooling support as future directions.

  • Findings: The review examines 21 primary studies to characterize logging issues, detection techniques, and evaluation practices.Its analysis covers how the problem is framed, which issues are addressed, and how detection techniques are evaluated.
  • Findings: The literature varies in the number and types of issues addressed and lacks shared benchmarks and evaluation practices.These inconsistencies contribute to a fragmented research landscape.
  • Future directions: Future research should explore intelligent detection approaches while prioritizing community-agreed benchmarks, shared tasks, and tooling support.The paper identifies LLM techniques as a potential opportunity for advancing log issue detection.
  • Conclusion: The review concludes that further improvement and collaboration are needed in logging-issue detection research.Its stated direction is toward more unified, practical, and scalable solutions for improving log quality.

Declaration of Generative AI and AI-assisted Technologies in the Writing Process

The authors used ChatGPT to improve the clarity, readability, and language of the paper, then reviewed and edited the text and retained responsibility for its content.

  • AI-assisted writing: ChatGPT was used to enhance the paper’s clarity, readability, and language.The authors reviewed and edited the resulting text.
  • Author responsibility: The authors state that they take full responsibility for the published content after using these tools.
Loading 2609.04215v1…