Source-linked AI summary
Discovering Performance Archetypes: Critical-Path-Aware Pattern Analysis and Regression Detection
Kaveh Shahedi, Heng Li, Maxime Lamothe, Foutse Khomh
TL;DR
The paper addresses the limited ability of static metrics and single-signal profiling to explain runtime performance. It combines critical-path analysis with static features and kernel-level resource data to discover transferable archetypes and detect regressions. Across six C/C++ applications, the framework discovers recurring archetypes and achieves F1-score 0.867, a 60.4% improvement over resource-only methods.
Problem
Static complexity metrics and single-signal profiling provide incomplete accounts of runtime performance shaped by execution behavior and resource interactions.
Method
The methodology synthesizes static code features, dynamic critical paths, and kernel-level resource data to cluster performance archetypes and detect regressions.
Results
F1-score 0.867 and 60.4% improvement over resource-only methods accompany 13 discovered archetypes, including five near-universal patterns covering 56.4% of observed paths.
Takeaways & Limitations
Recurring runtime performance behavior appears structured and transferable across diverse applications, supporting archetype-based characterization and multi-signal regression analysis.
Takeaways & Limitations
The evaluation uses six open-source C/C++ applications, synthetic injected degradations, and modeling choices that may limit generalizability to other settings.
Abstract
from arXiv · showhide
Software performance analysis and prediction requires integrating multiple signals, as code structure alone cannot capture runtime behavior shaped by execution frequency, resource contention, and I/O patterns. We present a critical-path-aware performance analysis methodology that automatically discovers recurring performance patterns by synthesizing static code features, dynamic execution traces, and kernel-level resource data. In a preliminary study across six real-world C/C++ applications (SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq), we first empirically confirm that static complexity metrics explain only 10.4% of the variance ($ρ^2$) in critical path execution time, quantifying a gap that, while theoretically expected, had not been measured systematically across applications. Motivated by this finding, we analyze nearly 80,000 critical execution paths and address two research questions. First, we discover 13 distinct performance archetypes: recurring behavioral patterns that appear consistently across different applications, independent of their domain or implementation. Five of these patterns are near-universal and appear in at least five of the six applications studied. Notably, three of these archetypes are present in all six applications, and together, these common patterns account for 56.4% of all observed paths. Each archetype maps to specific resource profiles and optimization strategies that transfer across domains. Second, we leverage these archetypes within a multi-signal regression detection framework that triangulates path structure, resource consumption, and archetype deviations, achieving an F1-score of 0.867 and a 60.4% improvement over resource-only methods.
1 Introduction
The introduction frames a gap between static code metrics and runtime behavior, motivating a multi-signal, critical-path-aware methodology for transferable archetypes and regression detection.
- Motivation: Static analysis misses execution frequency, instruction and library costs, and loop behavior, whereas single-signal profiling overlooks interactions among computation, memory, and I/O.
- Research challenges: The study targets recurring cross-application patterns, multi-signal regression detection, and identification of functions that matter beyond static complexity.
- Approach: The methodology combines static code analysis, dynamic critical-path extraction, and kernel-level resource monitoring across SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq.It analyzes top-k critical paths across multiple executions and diverse inputs.
- Motivation: Static complexity metrics explain only 10.4% of performance variance, while 14.7% of functions exhibit misaligned static and dynamic behavior across six applications.These findings motivate dynamic analysis of code segments that materially affect execution time.
- Contributions: Clustering discovers 13 performance archetypes, including five near-universal patterns, while triangulated regression detection achieves F1-score 0.867 and improves 60.4% over resource-only methods.Three archetypes appear in all six applications and the common patterns cover 56.4% of observed paths.
2 Related Works
Related work spans static and dynamic performance analysis, regression detection, workload characterization, tracing, critical-path analysis, and trace visualization, but these perspectives remain insufficiently integrated.
- Performance Regression Detection: Regression-detection approaches can produce false positives or identify regressions without pinpointing function-level causes.
- Performance Characterization and Patterns: Existing performance characterization mines workloads and sessions, but the related literature does not establish transferable, resource-coupled execution archetypes across applications.
- The Static-Dynamic Divide: Prior work treats static code structure and runtime behavior largely separately, despite dynamic analysis revealing behavior invisible to static metrics.
- The Multi-Level Tracing Landscape: Tracing tools trade off overhead, detail, and safety, motivating synchronized application- and system-level visibility.
- Critical Path Analysis: From Parallelism to Sequential Code: Critical-path methods expose bottlenecks missed by aggregate metrics, while trace-analysis tools remain primarily visualization-focused.
3 Methodology
The methodology integrates static, dynamic, and resource perspectives around critical paths, then clusters their multi-dimensional behavior into archetypes for characterization and anomaly detection.
- 3.1 Overview: The framework combines static code characteristics, dynamic execution behavior, and system-resource measurements for each program execution.This integration targets interactions that single-perspective analyses can overlook.
- 3.1 Overview: Performance is measured primarily as wall-clock execution time, complemented by CPU utilization, memory allocation rate, and I/O throughput.Performance variance denotes explained variance of execution time.
- 3.2 Background: Critical Paths and Archetypes: A critical path is a root-to-leaf dynamic call-graph sequence contributing the most wall-clock time, preserving caller context unlike aggregate profiling.The top-k paths compact large call graphs into performance-relevant sequences.
- 3.2 Background: Critical Paths and Archetypes: Critical paths combine structural, temporal, and resource information and support cross-input analysis of workload-dependent performance behavior.
- 3.2 Background: Critical Paths and Archetypes: Clustering multi-dimensional critical-path features yields performance archetypes with distinct resource profiles and optimization strategies, while archetype shifts can signal deviations.
3.3 Subject Programs and Data Collection
The study evaluates six open-source C/C++ applications with diverse workloads, generating randomized inputs and repeated executions to support broad and stable performance measurements.
- 3.3 Subject Programs and Data Collection: The dataset covers SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq, spanning database, cryptographic, compression, multimedia, networking, and text-processing workloads.
- 3.3 Subject Programs and Data Collection: Each application receives 500 distinct inputs designed to trigger diverse execution paths and workload characteristics.
- 3.3 Subject Programs and Data Collection: Inputs are generated by uniformly randomizing valid, documented parameters such as codecs, database operations, cipher suites, compression levels, protocols, and filters.
- 3.3 Subject Programs and Data Collection: Each input runs three times after pilot checks showed low variance, balancing measurement stability with collection cost.
3.4 Multi-Level Trace Collection
The methodology collects synchronized static, application-level, and kernel-level signals to characterize application behavior across executions.
- Three synchronized data streams provide a comprehensive view of each execution.
- Static analysis extracts LOC, cyclomatic complexity, loop nesting depth, unique calls, and I/O presence for every function.
- uftrace captures function entry and exit events, timestamps, and call relationships with low instrumentation overhead.
- LTTng records fine-grained CPU, memory, disk I/O, system-call, and network events at nanosecond precision.
- Process and thread identifiers align kernel events with application traces, including user-space memory allocations.
3.5 Critical Paths Extraction and Correlation
Critical paths are extracted as longest wall-clock call sequences within threads, then correlated with kernel resource activity through time-aligned windows.
- A critical path is the longest root-to-leaf call sequence in a thread, measured by wall-clock time.
- The extraction greedily follows the longest-duration child and reports paths across threads while preserving each path’s thread identity.
- The study extracts up to 10 paths per execution across 1,500 executions per application, yielding 79,789 paths overall.
- Path selection targets peak behavior from the longest single invocation chain rather than cumulative repeated-call time.
- Kernel events are aggregated into 10 μs windows and associated with path functions when their execution intervals overlap.
3.6 Discovering Performance Archetypes
The methodology represents critical paths with structural, temporal, resource, and transition features, then clusters standardized vectors into interpretable performance archetypes.
- Each critical path is represented by a 33-dimensional vector spanning structural, temporal, resource, and transition characteristics.
- Resource features summarize CPU utilization, memory activity, and I/O throughput using statistical and behavioral indicators.
- Standard scaling and k-means clustering group approximately 80,000 paths into scalable, centroid-based archetypes.
- Archetypes are evaluated for cross-application prevalence, iteration stability, and within-cluster homogeneity.
3.7 Multi-Signal Regression Detection
The regression detector combines path-structure, resource, archetype, and bounds signals using baseline-only models and evaluates them against injected degradations and established baselines.
- Regression scenarios modify one traced function per build with degradation code targeting resource spikes, duration increases, path elongation, or new functions.
- Baseline modeling uses 70% of non-regressed executions split by input to avoid leakage across iterations.
- Anomaly detection combines path-signature, resource, archetype-distribution, and bounds-violation scores into S_anomaly.
- Equal signal weights and a threshold τ = 0.65 are determined using baseline training data, and executions exceeding the threshold are flagged.
- The framework evaluates injected regressions using 30% of normal executions plus anomalous executions, comparing against seven baseline techniques.
3.8 Automation and Practical Workflow
The methodology provides an automated workflow for collecting data, extracting critical paths, discovering archetypes, and detecting regressions, with limited setup required for new C/C++ applications.
- 3.8 Automation and Practical Workflow: All stages from data collection through regression detection are fully automated after initial compile flags and optional one-time threshold calibration.The pipeline covers data collection, critical path extraction, static analysis, archetype discovery, and regression detection.
- 3.8 Automation and Practical Workflow: New C/C++ applications require function instrumentation, a workload-generator subclass, and execution of the pipeline entry point.The entry point handles trace collection, critical path extraction, archetype assignment, and regression detection.
- 3.8 Automation and Practical Workflow: The replication package includes scripts and a step-by-step guide for extending the framework to new applications.
4 Results and Evaluation
Across six applications, static complexity weakly predicts dynamic criticality, while critical paths converge into transferable archetypes that support strong multi-signal regression detection. The results also expose runtime variability and harder-to-detect behavioral shifts.
- 4.2 Preliminary Analysis: The Static-Dynamic Gap: Static metrics explain only 10.4% of performance variance on average, and 14.7% of functions show misaligned static and dynamic rankings.The misaligned groups are hidden bottlenecks at 7.0% and misleading complexity at 7.6%.
- 4.3 RQ1: Universal Performance Archetypes: 13 archetypes span application boundaries, with five near-universal patterns covering 56.4% of 44,995 critical paths across the six applications.Three archetypes appear in all six applications, while two more appear in at least five.
- 4.3 RQ1: Universal Performance Archetypes: Call-stack depth does not reliably predict runtime duration: A3 takes 0.17ms at depth 14.2, whereas shallower A1 takes 5.49ms at depth 6.3.The reported comparison represents a 32× difference inversely correlated with depth.
- 4.3 RQ1: Universal Performance Archetypes: Application performance signatures differ: diversity ranges from 0.31 in Zstandard and cURL to 0.54 in SQLite, OpenSSL, and FFmpeg.jq is moderately diverse at 0.38 but concentrates 74.1% of paths in the application-specific A4 archetype.
- 4.3 RQ1: Universal Performance Archetypes: Only three archetypes are application-specific, while 10 of 13 appear in at least two applications, indicating broad pattern transferability.The application-specific archetypes collectively represent 21.4% of paths, whereas the remaining archetypes represent 76.9%.
- 4.3 RQ1: Universal Performance Archetypes: Archetype stability is 43.4%, with the remaining 56.6% of inputs showing iteration changes attributed to scheduling, cache state, or I/O timing.These changes are characterized as runtime non-determinism rather than methodology instability.
- 4.4 RQ2: Triangulated Regression Detection: 0.867 F1 and 60.4% improvement over resource-only methods demonstrate that critical-path-aware multi-signal detection outperforms isolated resource analysis.The complete method achieves F1 0.867 versus 0.540 for the best resource-only baseline and also exceeds a threshold-based critical-path detector at 0.697.
- 4.4 RQ2: Triangulated Regression Detection: Detection is strongest for resource spikes at 99.6% and structural changes such as path elongation at 85.5%, but archetype shifts reach only 53.3%.Archetype shifts are described as out-of-distribution changes in the learned feature space.
5 Threats to Validity
The study’s validity is constrained by its application, hardware, workload, regression-injection, evaluation-split, labeling, and modeling choices.
- 5.1 External Validity: The study covers six open-source C/C++ applications on one Intel Core i7-11700K platform, limiting direct generalization across languages, proprietary systems, and microarchitectures.Randomly generated inputs may also differ from production workloads.
- 5.2 Internal Validity: The evaluation uses synthetic degradations, one regression type per build, a single 70/30 input split, and system-level noise may add measurement variance.These choices may not fully represent interaction effects, algorithmic changes, configuration shifts, or multiple simultaneous regressions.
- 5.3 Construct Validity: The methodology’s constructs depend on alternative metric formulations, selected path and archetype counts, quartile thresholds, and a longest-single-invocation path algorithm.These choices may miss cumulatively important short invocations or yield different results under alternative formulations.
- 5.3 Construct Validity: The 33-dimensional feature vector omits explicit cache and branch-prediction modeling, leaving finer microarchitectural behavior for future validation.The authors also propose validating archetypes against developer-identified bottlenecks.
6 Conclusions
The study finds recurring, transferable critical-path performance archetypes across six applications and uses them to improve regression detection over resource-only analysis.
- 6 Conclusions: 13 archetypes cluster nearly 80,000 critical-path executions across six C/C++ applications, with 10 appearing in multiple applications and five near-universal.The five near-universal archetypes collectively cover 56.4% of observed paths.
- 6 Conclusions: The regression detector triangulates path structure, resource consumption, and archetype deviations, outperforming resource-only baselines by 60.4% with F1: 0.867.The result supports multi-signal analysis over single-perspective approaches within the studied setting.
- 6 Conclusions: Recurring archetypes suggest that runtime performance behavior is structured and transferable across diverse application domains.The conclusion points toward archetype-informed optimization strategies that generalize beyond individual applications.
- 6 Conclusions: Future work should broaden language and domain coverage, add finer microarchitectural signals, and validate archetype transferability on additional hardware.