PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioral Regression Detection for Pull Requests via Structured Oracles

Yiwen Fu, Binhang Qi, Xiao Li, Runlin Liu, Hailong Sun, Xiang Gao

arXiv:2610.35223v1cs.SE

TL;DR

PR-level regression detection remains difficult when tests miss modified code and existing methods lack structured behavioral reasoning about preserved and intended changes. ORION combines intent-aware structured oracles with generated tests and differential execution; it outperforms evaluated baselines on RegBench-Py and finds maintainer-confirmed candidates in a larger PR study.

  • Problem

    PR regression detection is difficult because tests may not cover changed code, and existing methods can lack structured behavioral boundaries or miss deeper implications.

    Source: §I · p. 2

  • Method

    ORION extracts PR intent and impacted functions, synthesizes structured behavioral oracles, generates and executes tests across pre- and post-change code, and combines deterministic rules with constrained LLM reasoning.

    Source: §I · p. 2

  • Results

    ORION achieves F1-scores of 0.6125 with DeepSeek-Chat-V3.2 and 0.5578 with Claude-4.5-haiku, both higher than evaluated baselines on RegBench-Py.

    Source: §I · p. 2

  • Takeaways & Limitations

    In an analysis of 1,459 merged PRs, ORION identified nine reproducible bug candidates, eight confirmed by maintainers, and its approximately 15-minute runtime is described as practical for asynchronous CI/CD or high-risk PR analysis.

    Source: D. RQ3: Average Runtime and Cost… · p. 10, §VIII · p. 12

    The evaluation uses Python projects and a limited-size benchmark, so results may not generalize to other languages, ecosystems, or workflows.

    Source: §VI · p. 11

Abstract

from arXiv · show

Software regressions are unintended behavioral changes introduced by code modifications and are hard to detect at pull-request (PR) level. Existing methods rely on incomplete tests or directly analyze PRs via LLMs, lacking structured behavioral reasoning and explicit oracles. We present ORION, a staged framework for PR-level regression detection. ORION extracts PR intent and identifies impacted functions via AST-based impact analysis, then synthesizes function-level structured behavioral oracles encoding preserved/changed behaviors, setup constraints and control-flow hints from PR intent, static risk signals and code changes. Guided by these oracles, ORION generates executable tests, conducts oracle-coverage-guided augmentation and static repair, and differentially runs tests on pre- and post-change code. It classifies PRs with deterministic rules and constrained LLM reasoning. We build RegBench-Py, the first Python PR regression benchmark with 154 verified real-world PRs. ORION achieves F1-scores of 0.6125 (DeepSeek-Chat-V3.2) and 0.5578 (Claude-4.5-haiku), outperforming baselines including Testora, Codex and Claude Code. Compared with PatchGuru, ORION supports broader multi-function PRs while remaining competitive on single-function cases. On 1,459 merged PRs, ORION finds 9 reproducible regression candidates, 8 confirmed by maintainers. It takes ~15 minutes per PR, showing practicality for CI/CD screening.

I. INTRODUCTION

ORION addresses PR regression detection’s gaps in structured behavioral oracles and coverage of subtle behavioral implications. It combines intent-aware oracle construction and test execution; on RegBench-Py it outperforms evaluated baselines, while a motivating example shows it detecting a regression missed by a tool focused on new logic.

I. INTRODUCTION

  • Existing approaches often lack structured boundaries between preserved and intentionally changed behavior, and LLM-based methods may overlook deeper implications and regression-triggering conditions.Test suites may also fail to cover code modified by a PR, even in well-tested software.Source: §I · pp. 1–2
  • ORION extracts PR intent, identifies impacted functions via AST analysis, synthesizes function-level behavioral oracles, generates tests, and classifies pre/post-change differences using deterministic rules and constrained LLM reasoning.Its oracles encode preserved and changed behaviors, setup requirements, and control-flow hints.Source: §I · p. 2
  • RegBench-Py contains 154 Python PRs with reproducible regression evidence and independently re-inspected clean PRs.The benchmark is balanced between regression and non-regression PRs.Source: §I · p. 2
  • ORION achieves F1-scores of 0.6125 with DeepSeek-Chat-V3.2 and 0.5578 with Claude-4.5-haiku, both higher than evaluated baselines on RegBench-Py.The scope-aware PatchGuru comparison found neither method consistently dominated on commonly completed single-function PRs, while ORION supported broader multi-function PRs.Source: §I · p. 2

II. MOTIVATING EXAMPLE

  • For yt-dlp PR #9613, ORION’s test of a working Vimeo embed passed before the change and failed afterward, exposing a regression that Testora’s invalid-embed-focused tests missed.The resulting exception was captured during differential execution and classified as a regression with LLM-assisted assessment.Source: §II · pp. 2–3

III. METHODOLOGY

ORION organizes regression detection into four stages, moving from PR intent and impact analysis through oracle-guided testing to regression identification.

III. METHODOLOGY

  • The four stages are PR intent and impact-region identification, oracle synthesis, test generation, and regression identification.Tests run on pre- and post-change versions, and behavioral differences are classified using logic-based rules and LLM reasoning.Source: §III · p. 3

A. PR Intent and Impact Region Identification

ORION builds a structured context from PR intent and localized code changes, then uses it to synthesize function-level behavioral oracles. The oracles encode preserved and evolving behavior alongside setup and control-flow guidance for test generation.

1) PR Global Intent Extraction:

  • PR intent is normalized into change type, expected behavior change, backward compatibility, and high-level expectations to distinguish intended evolution from behavior that should remain preserved.Expected behavior change sets the scope for enforcing preserved-behavior constraints, while backward compatibility captures declared breaking changes and migration paths.Source: 1) PR Global Intent Extraction · pp. 3–4

2) Impact Region Identification:

  • Impact analysis maps changed diff lines to AST functions and records structural risks such as removed parameters, changed defaults, exception changes, silent **kwargs swallowing, and function-to-generator conversion.Immediate callers, callees, and sibling methods provide lightweight call context, while oracle synthesis remains scoped to directly impacted functions.Source: 2) Impact Region Identification · p. 4

B. Oracle Synthesis

  • ORION places an intermediate oracle layer between change analysis and test generation because direct LLM generation must otherwise combine risk reasoning with executable test construction.The structured oracle records behavior constraints without producing executable code.Source: B. Oracle Synthesis · p. 4
  • The function-level oracle encodes preserved behavior, intended changes, setup requirements, and control-flow hints, with static risk signals injected as standardized generation requirements when present.Oracle synthesis targets functions with meaningful behavioral impact and uses budget-based ranking when impacted functions exceed the analysis budget.Source: B. Oracle Synthesis · pp. 4–5

C. Test Generation

ORION’s test-generation pipeline aims to produce executable tests that cover oracle-defined behavioral boundaries and new control-flow paths.

C. Test Generation

  • Test generation uses a three-stage pipeline of generation, coverage augmentation, and static repair to produce executable tests that validate oracle constraints.The tests are intended to cover critical behavioral boundaries and newly introduced control-flow paths under executability requirements.Source: C. Test Generation · p. 5

1) Oracle-Guided Generation:

ORION generates tests from oracle-defined behavioral requirements rather than raw diffs, then tracks uncovered oracle regions to guide augmentation. It statically repairs executability without changing test semantics.

1) Oracle-Guided Generation:

  • Oracle-guided generation prompts the LLM with verification targets for preserved behavior, changed behavior, and new paths, plus relational context for feasible invocations.Each test includes an oracle reference linking it to a specific oracle region and index for deterministic coverage tracking.Source: 1) Oracle-Guided Generation · p. 5
  • A heuristic prioritizes functions by changed- and preserved-behavior counts, using fixed weights α = 5 and β = 2 rather than empirically optimized values.The initial-generation cap is 24 base tests per PR; the weighting affects allocation, not the relative importance of behaviors in regression classification.Source: 1) Oracle-Guided Generation · p. 5

2) Coverage-Guided Augmentation:

  • Coverage-guided augmentation records uncovered oracle entries and uses input mutation or oracle-guided regeneration to target those gaps, within a global cap of 40 additional tests.Functions with more uncovered items are prioritized.Source: 2) Coverage-Guided Augmentation · p. 5

3) Static Repair for Executability:

  • Static repair fixes formatting, identifiers, imports, and syntax, while avoiding logic-level changes so identical tests remain comparable across program versions.Repair is limited to minimal executability guarantees and does not alter generated tests’ logical intent.Source: 3) Static Repair for Executability · p. 5

D. Regression Identification

ORION classifies PRs by running identical tests in isolated before-and-after environments, applying deterministic rules first and constrained LLM reasoning to unresolved cases. RegBench-Py provides 154 balanced Python PRs, with regression labels supported by developer evidence and clean labels treated as operational rather than proof of absence.

D. Regression Identification

  • ORION runs tests independently in matched before-and-after environments, captures structured execution outcomes, and uses deterministic rules before constrained LLM reasoning for unresolved cases.The LLM assesses differences against PR intent and oracle constraints, and PR-level regression follows when at least one executable case is classified as regression.Source: D. Regression Identification · pp. 5–6
  • A case is labeled regression when execution differs and at least one standardized answer identifies a failure against preserved behavior or an unintended performance slowdown.Identical executions are clean; infrastructure or setup failures are excluded, and slowdowns count only at twice the prior runtime absent noise or an intended tradeoff.Source: D. Regression Identification · p. 6

IV. REGBENCH-PY: A BENCHMARK FOR PR-LEVEL REGRESSION DETECTION

  • RegBench-Py contains 154 PRs—77 regression-inducing and 77 clean—collected from 207 active Python projects and spanning 65 repositories.Regression samples require explicit developer attribution and reproducible builds; clean samples are screened using repository history and independent manual review.Source: §IV · pp. 6–7
  • Clean PRs are defined by no later bug-fix PR changing their modified lines within 12 months and no concrete regression evidence on manual review.Annotators initially agreed on 75/77 cases (97.4%); the two disagreements were retained only after joint adjudication found no concrete evidence of regression.Source: §IV · p. 7
  • Regression PRs have a wider churn spread and heavier tail than clean PRs, although the groups’ typical churn is comparable.Churn is the sum of added and deleted lines, shown on a logarithmic scale.Source: Figure 3 · p. 6, §IV · p. 7

V. EXPERIMENT

The experiments evaluate ORION’s benchmark accuracy, component contributions, per-PR runtime and cost, and effectiveness on newly merged PRs. Comparisons include regression-detection tools, direct LLM judgment, and agentic coding systems, with some baselines evaluated on restricted subsets.

V. EXPERIMENT

  • The experiment poses four questions covering RegBench-Py classification accuracy, component impact, average runtime and cost, and detection of unknown regressions in newly merged PRs.Source: §V · p. 7

1) Baselines and Configuration:

  • Baselines include Testora in adapted and native configurations, a direct LLM judge using PR metadata and diffs, Codex, and Claude Code.Claude Code is evaluated on the full benchmark.Source: 1) Baselines and Configuration · p. 7
  • PatchGuru comparisons use only the common RegBench-Py subset where PR diffs modify exactly one source function and both systems produce valid outputs.Source: 1) Baselines and Configuration · p. 7
  • ORION, Testora, and the direct LLM baseline are tested with DeepSeek-Chat-V3.2 and Claude-4.5-haiku; Codex and Claude Code use their own agent backends.Experiments use consistent API parameters, prompt settings, and execution environments where applicable.Source: 1) Baselines and Configuration · pp. 7–8

2) Metrics:

The section defines the classification metrics, describes the evaluation setup, and reports ORION’s performance on RegBench-Py. ORION leads the shared-backend comparisons in Recall and F1-score, while its results vary across runs.

2) Metrics:

  • Precision, Recall, and F1-score are calculated from true positives, true negatives, false positives, and false negatives for regression-PR classification.True positives and false negatives concern regression-inducing PRs; true negatives and false positives concern non-regression PRs.Source: 2) Metrics · p. 8

3) Implementation:

B. RQ1: Classification Accuracy on RegBench-Py

  • With DeepSeek, ORION reaches Recall 0.6364 and F1-score 0.6125; with Claude, it reaches Recall 0.5325 and F1-score 0.5578, the highest values for both metrics under each shared backend.The direct LLM baseline has higher Precision under both backends, but lower Recall; Table I covers all 154 PRs and uses Testora’s adapted filtering setting.Source: Table I · p. 8, B. RQ1: Classification Accuracy on… · p. 8
  • On the 94 PRs retained by Testora’s native filters, ORION’s F1-scores are 0.5000 with DeepSeek and 0.4211 with Claude, compared with Testora’s 0.1967 and 0.1667.The subset contains 41 regression-inducing and 53 clean PRs.Source: B. RQ1: Classification Accuracy on… · pp. 8–9, Table II · p. 9
  • Among the 33 PRs completed by both PatchGuru and ORION, neither method consistently dominates, while PatchGuru’s single-function requirement limits its applicability to broader PRs.Only 48 of 154 PRs meet PatchGuru’s requirement, and it completes 33 of those.Source: B. RQ1: Classification Accuracy on… · p. 9, Table III · p. 9
  • Across three DeepSeek runs, the standard deviations are 0.040 for Precision, 0.057 for Recall, and 0.048 for F1-score; Table I reports the first run.The analysis of 49 true positives finds 38 cases consistent with developer-reported regressions.Source: B. RQ1: Classification Accuracy on… · p. 9

C. RQ2: Impact of Key Components

The ablation study compares the complete ORION pipeline with variants that remove or weaken key stages. Its results show the largest performance loss when oracle-guided test generation is removed, with augmentation also contributing to performance.

C. RQ2: Impact of Key Components

  • The ablation variants remove analysis-guided oracle generation, oracle-guided test generation, oracle-driven augmentation, or intent and oracle context from final judgment.All variants are evaluated on the full RegBench-Py dataset under identical execution settings.Source: C. RQ2: Impact of Key Components · p. 9
  • Removing oracle-guided test generation reduces Recall from 0.6364 to 0.2597 and F1-score from 0.6125 to 0.3279, the largest degradation among the reported ablations.The complete pipeline has the best overall performance in the study.Source: C. RQ2: Impact of Key Components · p. 9
  • Without oracle-driven augmentation, Recall falls to 0.4805 and F1-score to 0.5103, as the initial tests leave some oracle-specified behaviors uncovered.This variant omits oracle-coverage checking and oracle-driven test augmentation.Source: C. RQ2: Impact of Key Components · pp. 9–10
  • Removing PR intent and oracle context from final judgment lowers Recall to 0.6104 and F1-score to 0.5987, a smaller decrease than removing test generation or augmentation.The ablation retains deterministic rules and the five-question protocol.Source: C. RQ2: Impact of Key Components · pp. 9–10

D. RQ3: Average Runtime and Cost Overhead

ORION averages about 870 seconds per PR, with LLM-enabled stages and environment provisioning accounting for most runtime. In a separate real-world study, it identified reproducible bug candidates in recent PRs, most of which were confirmed by maintainers.

D. RQ3: Average Runtime and Cost Overhead

  • About 870 seconds (approximately 15 minutes) per PR are dominated by LLM-enabled stages and environment provisioning; static analysis contributes negligibly.Test generation is the most expensive LLM stage, while environment setup consumes much of test-execution time.Source: D. RQ3: Average Runtime and Cost… · p. 10
  • Each PR consumes approximately 60K–100K tokens, costing about $0.01–$0.02 with DeepSeek-Chat or $0.12–$0.23 with Claude Haiku 4.5.Runtime, rather than monetary cost, remains the main practical bottleneck.Source: Figure 5 · p. 10, D. RQ3: Average Runtime and Cost… · p. 10

E. RQ4: Effectiveness on Real-World Regressions

  • Across 1,459 PRs from NetworkX, SymPy, FastAPI, and Keras, ORION identified 9 reproducible bug candidates; maintainers confirmed 8 and one remains under discussion.The study aims to assess whether ORION can surface concrete issues, not estimate regression prevalence.Source: E. RQ4: Effectiveness on Real-World… · p. 10
  • The identified candidates span boundary-condition errors, backward-compatibility breaks, semantic logic changes, error-handling drift, and runtime exceptions.Examples include a removed public attribute and a NameError while parsing stringified type hints under Python 3.10+.Source: E. RQ4: Effectiveness on Real-World… · pp. 10–11

VI. THREATS TO VALIDITY

ORION’s validity is constrained by environment reconstruction, uncertain intent and oracle quality, and nondeterministic LLM stages. Its evaluation also has dataset, leakage, and measurement boundaries that limit how broadly its results should be interpreted.

Internal validity

  • LLM nondeterminism can change ORION’s oracles, tests, and judgments across runs; repeated DeepSeek-V3.2 evaluation showed moderate variability.The authors advise considering this variability when interpreting single-run performance.Source: §VI · p. 11
  • ORION may miss regressions involving complex state, remote services, caches, asynchronous behavior, or optional dependencies, and may flag intended API evolution or sensitive exception differences.It is more effective for explicit observable differences such as return values, exceptions, CLI/config/IO, and serialization behavior.Source: §VI · p. 11
  • Unavailable LLM training corpora prevent ruling out prior exposure to public RegBench-Py PRs, so executable pre/post evidence does not eliminate leakage risk.The authors also note that environment reconstruction can fail for dependency or toolchain reasons unrelated to regressions.Source: §VI · p. 11

External validity

  • Results may not generalize beyond Python projects or workflows; RegBench-Py is limited in size, and regression PRs have a wider churn spread and heavier tail than clean PRs.Balanced sampling and churn-distribution analysis mitigate, but do not remove, PR size as a potential confounder.Source: §VI · p. 11

Construct validity

  • Precision, recall, and F1 measure PR-level detection, not oracle quality, intent fidelity, exact localization, or full bug reproduction.Manual inspection found that 38 of 49 PRs predicted by ORION at least partially covered the developer-reported regression.Source: §VI · p. 11

VII. RELATED WORK

Prior regression and test-generation methods often expose differences or optimize general coverage without explicitly modeling evolving behavior. ORION instead structures intent-aware behavioral oracles; on the full benchmark it outperformed evaluated baselines, while supporting broader multi-function PRs than PatchGuru.

A. Regression Detection and Behavioral Differences

  • Prior regression-detection methods expose version differences or analyze risk, but ORION indexes structured preservation, evolution, setup, and control-flow requirements by function to guide tests.PatchGuru also operationalizes intent into assertions and inputs, but its current implementation targets single-function PRs.Source: A. Regression Detection and Behavioral… · pp. 11–12

B. Automated Test Generation

  • General automated test-generation methods improve coverage but are often change-semantics-agnostic; ORION separates oracle synthesis from test generation and targets regression-inducing conditions.The related work notes that optimizing general coverage or mutation score can limit regression targeting and interpretability.Source: B. Automated Test Generation · p. 12

C. Modeling Code Changes, Intent, and Oracles.

  • Existing oracle approaches usually assume stable expected behavior, whereas ORION combines intent and structural signals to distinguish preserved from changed behavior.Prior work uses specifications, invariants, metamorphic relations, or oracle-aware regression-test selection.Source: C. Modeling Code Changes, Intent, and… · p. 12

D. LLM-Based Program Analysis

  • Few prior systems couple static risk signals, structured behavioral oracles, and differential execution for PR-level regression detection; ORION uses this staged design.The paper positions this as improving robustness and interpretability without heavy analysis overhead.Source: D. LLM-Based Program Analysis · p. 12

VIII. CONCLUSION

  • 0.6125 F1 with DeepSeek-Chat-V3.2 and 0.5578 with Claude-4.5-haiku; ORION outperformed evaluated baselines on the full benchmark under corresponding settings.In 1,459 PRs, it identified 9 reproducible bug candidates, 8 confirmed by maintainers.Source: §VIII · p. 12
  • ORION and PatchGuru did not consistently dominate on commonly completed single-function PRs, while ORION supports multi-function PRs and takes about 15 minutes per PR.The authors describe that runtime as practical for asynchronous CI/CD or high-risk PR analysis.Source: §VIII · p. 12

Related papers

Loading 2610.35223v1…