PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations

Subrata Das, Ali Aman, Muhammad Asaduzzaman, Kawser Wazed Nafi, Salimur Choudhury

arXiv:2609.17844v1cs.SE

TL;DR

Argument-swap bugs can be difficult to detect in Python because swapped arguments may have compatible types, while static type information is typically unavailable. BugProbe uses contextual representations without call-to-definition mappings and outperforms DeepBugs across all call sites.

  • Problem

    Argument-swap bugs can be hard to detect when swapped arguments have compatible types, especially in dynamically typed languages where static type information is typically unavailable.

    Source: §I · p. 1

  • Method

    BugProbe frames argument-swap detection as binary classification and learns identifier representations from call-site contexts without requiring call-to-definition mappings.

    Source: §I · pp. 1–2, §VII · p. 10

  • Results

    Across all call sites, BugProbe achieves 91.1% accuracy versus DeepBugs’ 81.8%, with similar gains in precision, recall, and F1-score.

    Source: 3) Results · p. 7

  • Takeaways & Limitations

    BugProbe requires no explicit call-site-to-definition mappings, making it well suited to dynamically typed languages.

    Source: §VII · p. 10

    The evaluation centers on synthetic swaps that may not be bugs, with only 55 real-world cases and open-source Python evidence that may not generalize elsewhere.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Names of source code elements convey rich semantic information and have been widely used in software engineering tasks such as bug detection, code completion, type prediction, and code classification. Prior studies exploit lexical similarity between method arguments and formal parameter names to detect bugs caused by incorrectly ordered arguments, typically relying on establishing mappings between method calls and their corresponding definitions. However, such mappings are often difficult to obtain in dynamically typed languages like Python. In this paper, we present BugProbe, a learning-based approach for detecting incorrectly ordered arguments in Python method calls that does not require call-to-definition mappings. Our approach leverages multiple sources of contextual information, including local context and argument usage context, and combines name-based similarity with machine learning to construct expressive representations of method arguments. We collect a new dataset of 132,739 Python source files from the top-1,000 starred GitHub repositories, yielding 3,371,244 synthetic training examples, and contribute a curated benchmark of 55 real-world argument-swap bugs manually verified from commit histories. We evaluate our approach on this dataset and show that it achieves high accuracy and consistently outperforms a state-of-the-art baseline across standard evaluation metrics. These results demonstrate that effective detection of argument-ordering bugs is possible without relying on explicit call-to-definition resolution, making the approach well suited for dynamically typed language settings.

I. INTRODUCTION

Argument-swap bugs can silently produce incorrect behavior when arguments have compatible types, and detecting them is difficult in dynamically typed Python. BugProbe represents call-site arguments using broader context without requiring call-to-definition mappings, and is evaluated against DeepBugs on synthetic and real-world data.

I. INTRODUCTION

  • DeepBugs requires call-to-definition mappings and uses narrow identifier-level contexts, both of which can be problematic for Python argument-swap detection.In the authors’ study, 89% of Python call sites had definitions in different files, often third-party libraries.Source: §I · p. 1
  • BugProbe formulates detection as binary classification and builds argument embeddings from call-site contexts to capture long-term dependencies and structural usage patterns.Its approach avoids requiring call-to-definition mappings.Source: §I · pp. 1–2
  • BugProbe’s F1 score is approximately 11.3% higher (relative) than DeepBugs across all call sites.The evaluation also includes synthetic and real-world benchmarks.Source: §I · p. 2
  • The authors contribute a dataset of 132,739 Python files from the top-1,000 starred GitHub repositories, containing 3,371,244 synthetic examples.Source: §I · p. 2
  • The authors contribute a curated benchmark of 55 real-world argument-swap bugs manually verified from commit histories across 36 open-source repositories.Source: §I · p. 2

II. RELATED WORK

Prior work detects argument-ordering defects through anomaly patterns, lexical similarity, heuristics, or learned local-context embeddings; BugProbe instead combines broader usage contexts without requiring call-definition mappings. The paper also describes its Python corpus, synthetic swaps, and manually verified real-world bug benchmark.

II. RELATED WORK

  • Prior argument-ordering approaches use recurring call-site patterns, name similarity, or handcrafted checks, while DeepBugs learns from local token context but requires call-to-definition mappings.Related work also covers argument recommendation and type inference, whose outputs may inform argument-related defects but do not directly evaluate ordering-bug detection.Source: §II · pp. 2–3

III. DATA COLLECTION

  • The corpus starts from Python repositories among GitHub’s top 1,000 by stars, then filters files for parsing, qualifying calls, duplication, and size.The initial collection contained 381,244 files; preprocessing removed files that were unsuitable for training or computationally feasible analysis.Source: §III · p. 3
  • The final dataset contains 132,739 files, 1,685,622 qualifying call sites, and 3,371,244 examples generated as one positive and one swapped negative per call site.Each five-fold split uses 2,696,686 examples for training and 674,558 for testing.Source: §III · p. 3, Table I · p. 3
  • Negative sampling swaps only the first two positional arguments; 22.4% of qualifying call sites have higher arity, so generalization to their other position pairs remains untested.The authors identify arbitrary argument-position swaps and transfer to higher-arity swaps as future work.Source: §III · p. 3
  • The real-world benchmark contains 55 manually verified argument-swap bugs mined from commit histories and drawn from 36 repositories.The candidates were checked to distinguish bug fixes from signature refactors, then examined by two annotators.Source: §III · pp. 3–4

IV. METHODOLOGY

The methodology reproduces DeepBugs’ learning pipeline for Python and extracts name-based features from call sites with at least two arguments. It forms balanced examples from observed calls and negatives created by swapping the first two arguments.

IV. METHODOLOGY

  • The methodology reproduces DeepBugs for Python while preserving its problem formulation and learning pipeline.DeepBugs was originally designed for JavaScript, and its core embedding-and-classification idea is language-agnostic.Source: §IV · p. 4, A. DeepBugs Reproduction · p. 4

1) Data Extraction:

  • For each parsed call with at least two arguments, the extraction records the base-object and callee names, the first two argument names and literal types, and available formal-parameter names.Source: 1) Data Extraction · p. 4
  • Each observed call becomes an assumed-correct positive example, and swapping the first two argument names and types generates a negative example.The resulting training examples are balanced positive-negative pairs.Source: 1) Data Extraction · p. 4

2) Embedding Generation:

Identifier and literal embeddings are learned from AST-derived token sequences, then combined into fixed-length vectors for supervised classification. BugProbe broadens embedding context to capture usage and structural signals and avoids explicit call-to-definition mappings.

2) Embedding Generation:

  • Identifier and literal embeddings are learned with Word2Vec from AST-derived token sequences containing identifiers and literals, with other syntax discarded.The model uses CBOW, a context window of 20, 200-dimensional embeddings, and a vocabulary capped at 10,000 tokens.Source: 2) Embedding Generation · p. 4

3) Classifier:

  • The classifier concatenates learned identifier embeddings and literal-type vectors into fixed-length inputs for a supervised feedforward binary classifier.The network follows DeepBugs with one hidden layer of size 200 and a sigmoid output layer.Source: 3) Classifier · p. 4

B. Proposed Framework

  • BugProbe changes embedding context to capture long-range usage and structural information, and redesigns example tuples to match that broader scope without explicit call-to-definition mappings.The framework combines unsupervised representation learning with supervised classification.Source: B. Proposed Framework · p. 4

1) Data Extraction:

BugProbe extracts call-site and argument context features to represent calls without relying on definitions. It pairs extracted examples with examples formed by swapping the first two arguments.

1) Data Extraction:

  • Each call-site tuple includes the callee’s local context, both arguments’ usage contexts and parent/latest usage contexts, and their types when available.Literal arguments receive type information; otherwise the type fields are empty.Source: 1) Data Extraction · p. 4
  • Extracted tuples are treated as positive examples under the assumption that most corpus code is correct; swapping the first two argument contexts and types generates balanced negative examples.The transformation is intended to introduce an argument-ordering bug while preserving surrounding context.Source: 1) Data Extraction · pp. 4–5
  • Local Context (LC) captures nearby tokens around an argument or call site, while Argument Usage Context (AUC) records earlier uses of the argument name in its enclosing scope.AUC includes references from assignments, updates, and prior calls.Source: 1) Data Extraction · p. 5
  • Parent Context (PC) captures the call site’s syntactic container, while Latest Usage Context (LUC) captures the closest preceding same-scope line mentioning the argument.Examples of parent contexts include loop headers, conditional headers, and function definitions.Source: 1) Data Extraction · p. 5
  • Because PC is identical for both arguments at a call site, BugProbe combines it with LUC as PCLU to give each argument a more discriminative representation.PCLU concatenates the parent and latest-usage token sequences.Source: 1) Data Extraction · p. 5

2) Embedding Generation:

BugProbe learns embeddings from separate corpora of argument contexts, optionally enriching tokens with expression types. It combines context and type representations into feature vectors for a classifier.

2) Embedding Generation:

  • Each extracted LC, AUC, LUC, and PC context instance is treated as a sentence, yielding a separate token-sequence corpus for each context type.The corpora contain sequences derived from argument usages across the codebase.Source: 2) Embedding Generation · pp. 5–6
  • Expression-type encoding appends a lightweight syntactic role to an identifier, distinguishing identical names used in different expression contexts.Examples of roles include assignment target, call argument, and binary operand.Source: 2) Embedding Generation · p. 6
  • The context corpora are combined for CBOW Word2Vec training with a window size of 5 and 100-dimensional embeddings.Aggregated context signals are used rather than isolated identifiers.Source: 2) Embedding Generation · p. 6

3) Classifier:

  • The classifier concatenates nine 100-dimensional vectors into a 900-dimensional feature vector, then uses a one-hidden-layer perceptron to classify argument orderings.The nine vectors represent callee context, argument contexts, and type information; the hidden layer has size 100.Source: 3) Classifier · p. 6

V. EVALUATION

The evaluation asks whether BugProbe detects argument-swap bugs, how name-based checks and context sources contribute, and how it performs on real-world bugs. Its synthetic evaluation uses file-level five-fold cross-validation on a large Python corpus.

V. EVALUATION

  • The evaluation poses five questions about BugProbe’s effectiveness, name-based checks, information sources, pre-trained embeddings, and performance on real-world bugs.The real-world bug dataset and replication source are identified as available to support future reproductions.Source: §V · p. 6

A. Experimental Setup

  • The dataset contains 132,739 Python files and 3,371,244 examples from 1,685,622 qualifying call sites; 2,696,686 examples are used for training and 674,558 for testing across five folds.The files come from the top-1,000 starred GitHub repositories.Source: A. Experimental Setup · p. 6
  • File-level five-fold cross-validation uses one fifth of files for testing in each fold, with embeddings trained only on that fold’s training files.Files containing real-world benchmark bugs are excluded from the splits and embedding training.Source: A. Experimental Setup · p. 6

B. Evaluation Metrics

The section defines the reported precision, recall, and F1 measures, motivates evaluation of name similarity and call-definition mapping, and compares BugProbe with DeepBugs across call-site partitions. BugProbe has higher reported accuracy on the complete dataset.

B. Evaluation Metrics

  • Precision is the proportion of correct predictions among predictions made, recall is the proportion of true instances correctly identified, and F1 is their harmonic mean.Precision, recall, and F1 are reported separately for each class.Source: B. Evaluation Metrics · p. 7

1) Motivation:

  • In the Python study, 89% of call sites have definitions in different files; the section also asks whether argument–parameter lexical similarity alone reliably detects swaps.The difficulty of mapping definitions motivates assessing detection without call-definition mappings.Source: 1) Motivation · p. 7

2) Approach:

  • The comparison separates resolvable same-file call sites from all call sites; BugProbe’s features stay identical across partitions, while DeepBugs’ available information changes.The all-call-sites partition includes sites regardless of whether a definition is available.Source: 2) Approach · p. 7

3) Results:

  • On all call sites, BugProbe accuracy is 91.1% versus DeepBugs’ 81.8% (9.3 percentage points; approximately 11.4% relative); AUC rises from 92.0 to 97.5.On mapped call sites, accuracy is 87.8% for BugProbe and 84.6% for DeepBugs.Source: 3) Results · p. 7

2) Approach:

The section evaluates lexical similarity as a lightweight argument-ordering check and finds that it catches most swapped arguments but produces notable false positives. Its reliance on formal parameter names also limits its availability in Python.

2) Approach:

  • The check normalizes identifier names into terms and compares lexical overlap between call-site argument names and corresponding formal parameter names.It uses no learned representations or contextual information.Source: 2) Approach · pp. 7–8
  • A two-argument call is classified as correctly ordered when its average similarity exceeds 0.5; synthetic negative examples swap the first two arguments.The method depends on corresponding formal parameter names being available.Source: 2) Approach · p. 8, 3) Results · p. 8

3) Results:

  • For incorrect calls, lexical similarity achieves 75.3% precision and 94.5% recall (F1 = 83.8%), but correctly ordered calls have only 69.0% recall (F1 = 79.1%).Correct-call precision is 92.6%.Source: 3) Results · p. 8

1) Motivation:

BugProbe combines contextual signals because isolated sources may be insufficient, and investigates whether expression-typed token encoding adds value. The full context configuration has the best reported F1 and AUC, while ET encoding yields modest gains, especially in context-rich models.

1) Motivation:

  • BugProbe combines local, usage, parent/latest usage, and lightweight type information because each source captures different aspects of argument context.The section investigates the individual and combined contributions of these feature groups and ET encoding.Source: 1) Motivation · p. 8

2) Approach:

  • The ablation compares five feature-group combinations, with M5 retaining all available context types; models are evaluated both with and without ET encoding.The training and evaluation protocol is held constant across those comparisons.Source: 2) Approach · p. 8

3) Results:

  • M5, the full BugProbe configuration, achieves the highest overall performance: 90.71% F1 and 97.26% AUC.Its context sources are callee and argument local context, argument usage context, parent/latest usage context, and argument type information.Source: 2) Approach · p. 8, 3) Results · p. 9
  • ET encoding raises M1’s F1 from 62.07% to 67.86%, while gains for context-rich M4 and M5 are approximately 0.5 percentage points.The paper describes ET gains as consistent but modest across most configurations.Source: 3) Results · p. 9

2) Approach:

  • BugProbeCodeBERT replaces BugProbe’s Word2Vec component with CodeBERT while keeping the rest of the framework unchanged.The benefit of pre-trained code embeddings for argument-ordering detection remained unclear.Source: 2) Approach · p. 9, 1) Motivation · p. 9

3) Results:

On 55 real-world bugs, BugProbe outperforms DeepBugs across the reported metrics. The section also reports CodeBERT-variant results, error patterns, and runtime measurements for the framework.

3) Results:

  • BugProbeCodeBERT achieves 81.5% F1 for incorrect examples and 89.6% ROC-AUC, but falls short of original BugProbe across metrics at greater computational cost.Its accuracy is 81.0%.Source: 3) Results · p. 9

2) Approach:

  • The real-world evaluation pairs each manually confirmed buggy call with its fixed version and tests models trained on synthetic data without benchmark fine-tuning.A bug counts as detected when the buggy call receives a higher incorrectness likelihood than its corrected pair.Source: 2) Approach · p. 9, 1) Motivation · p. 9

H. Error Analysis

  • False positives often involve related or co-occurring arguments such as width/height, while false negatives often involve generic names or names derived from similar expressions.In these false negatives, lexical similarity and contextual usage may lack sufficient discriminative signal.Source: H. Error Analysis · p. 9

3) Results:

  • On the 55-bug benchmark, BugProbe achieves 80.9% accuracy versus DeepBugs’ 53.6%, with incorrect-class F1 of 79.6% versus 59.2% and AUC of 90.8 versus 55.1.The correct-class F1 scores are 82.1% for BugProbe and 46.3% for DeepBugs.Source: 3) Results · p. 9, Table VII · p. 10

A. Efficiency and Practical Applicability

  • Parsing takes 15 ms on average and context collection adds 1.1 ms per example; testing 674,558 examples takes 4 minutes total.The reported average per-instance time to determine ordering or generate a warning is 1.1 ms.Source: A. Efficiency and Practical… · p. 10

B. Threats to Validity

BugProbe consistently outperforms DeepBugs and other baselines without explicit call-site-to-definition mappings. The evaluation has limits involving synthetic swaps, measurement scope, dataset coverage, and embedding choice.

B. Threats to Validity

  • Synthetic negatives may not all represent real bugs, and the 55-bug real-world benchmark remains indicative rather than statistically comprehensive.The study generates negatives by swapping the first two arguments; applying the same procedure to all approaches is reported not to bias comparative results.Source: B. Threats to Validity · p. 10
  • Precision, recall, and F1 capture detection effectiveness, but not developer-perceived usefulness or bug severity.The authors identify user studies as a way to gain further insight into practical impact.Source: B. Threats to Validity · p. 10
  • The open-source GitHub Python evaluation may not generalize to proprietary codebases or other languages with different typing or naming conventions.The authors describe Python as a challenging and widely used dynamically typed language, making it a meaningful study target.Source: B. Threats to Validity · p. 10
  • Results involving pre-trained embeddings are specific to the evaluated Word2Vec models; alternative embedding architectures may yield different outcomes.Word2Vec was chosen to ensure a fair comparison with DeepBugs.Source: B. Threats to Validity · p. 10

VII. CONCLUSION

  • BugProbe consistently outperforms DeepBugs and other baselines on synthetic and real-world datasets without explicit call-site-to-definition mappings.The conclusion attributes strong performance to combining contextual feature groups, with modest but consistent gains from expression-typed token encoding.Source: §VII · p. 10
  • Future work will explore richer type inference, other semantic bug patterns, comparisons with LLMs, and additional bug-detection baselines.The named baselines include SWAPD and NALIN.Source: §VII · p. 10

Related papers

Loading 2609.17844v1…