PaperlayerView exact revision on arXiv

Source-linked AI summary

FastCI: Efficient GPU-Intensive CI for LLM Training Frameworks

Tianshuo Qiao, Naiqian Zheng, Xiaopeng Liu, Shuguang Wang, Diandian Gu, Xuanzhe Liu, Xin Jin

arXiv:2610.01967v1cs.LGcs.SE

TL;DR

GPU-intensive end-to-end tests make CI a bottleneck for LLM training frameworks. FastCI uses runtime evidence to select and prune tests, then prioritizes risky tests and optimizes workloads. On the evaluated CI workload, it reduced latency and GPU use while improving modified-code coverage retention.

  • Problem

    GPU-intensive end-to-end testing makes CI for LLM training frameworks time-consuming and resource-intensive, creating a bottleneck for fast-paced development.

    Source: §1 · pp. 1–2

  • Method

    FastCI uses sampled runtime evidence to select affected tests and prune equivalent contexts, then schedules high-risk tests and reduces out-of-scope workloads.

    Source: §1 · p. 2

  • Results

    FastCI reduces CI latency by 77.5% and GPU resource usage by 63.9%, while improving modified code coverage retention by 3.2% over deployed CI pipelines.

    Source: §6 · p. 10

  • Takeaways & Limitations

    FastCI now handles production CI at ByteDance, with no reported omitted-test regressions across 784 CI runs over three weeks.

    Source: §5 · p. 7

Abstract

from arXiv · show

As large language models (LLMs) keep growing in size and complexity, their training frameworks evolve at a rapid pace as well. Therefore, continuous integration (CI) is critical for maintaining the quality and stability of these frameworks. However, unlike traditional software, CI for LLM training frameworks relies on GPU-intensive tests, which usually involve complete model training or evaluation. This leads CI itself to become a new bottleneck for fast-paced development. In this paper, we introduce FastCI, a framework that improves the efficiency of CI for LLM training frameworks. FastCI leverages runtime evidence to select affected tests and prune tests that execute changed code in equivalent contexts. Then FastCI prioritizes high-risk tests to expose potential failures earlier, and optimizes test workloads along dimensions outside the intended validation scope of each test. Evaluated on the CI workload of our LLM training framework, FastCI reduces the CI latency by 77.5% and the GPU resource usage by 63.9%, while improving the modified code coverage retention by 3.2%, compared with the currently deployed CI pipelines. FastCI has now been integrated into the CI pipelines of our LLM training framework at ByteDance.

1 INTRODUCTION

GPU-intensive end-to-end tests make CI costly for complex LLM training frameworks, while existing selection and execution practices leave inefficiencies. FastCI targets both test selection and execution.

1 INTRODUCTION

  • LLM-framework CI runs take more than four hours on average despite a dedicated pool of thousands of GPUs, and many tests do not cover changed code.The framework receives more than 200 merge requests per week, with each potentially triggering CI multiple times.Source: §1 · pp. 1–2
  • File-level path mappings over-select tests and can miss affected tests when manually maintained mappings are incomplete.The example contrasts selecting all tests depending on a modified file with runtime evidence identifying only the test that executes the changed function.Source: §1 · p. 2
  • FIFO scheduling delays failure exposure, while oversized end-to-end workloads consume GPU resources without proportionally improving regression validation.FastCI instead prioritizes high-risk tests and reduces workload outside each test’s intended validation scope.Source: §1 · p. 2
  • FastCI uses sampled runtime traces to select tests reaching changed functions, prune equivalent execution contexts, prioritize high-risk tests, and reduce out-of-scope workloads.Its runtime evidence graph records observed functions and calls for each test.Source: §1 · p. 2
  • FastCI cuts CI latency 77.5% and GPU resource usage 63.9%, while raising modified code coverage retention 3.2% over deployed path-based CI.It also cuts GPU usage before the first failure by 89.2% versus FIFO scheduling; production integration reduced average CI latency by 58.4%.Source: §1 · p. 2

2 RELATED WORK

Prior test-selection approaches trade precision, analysis cost, and coverage, while prediction-based methods commonly prioritize failures without guaranteeing that tests can be skipped.

2 RELATED WORK

  • File- and module-level static analysis remains coarse, while function-level analysis can time out on LLM training frameworks; NameRTS may over-select tests sharing entry points.FastCI uses lightweight runtime sampling, though incomplete evidence makes selection mechanism design challenging.Source: §2 · p. 3
  • Exhaustive runtime tracing can substantially increase CI runtime and GPU consumption, motivating FastCI’s use of lightweight sampling for test dependencies.The sampling approach must handle potentially incomplete runtime evidence.Source: §2 · p. 3
  • Prediction-based selection lacks a clear guarantee for skipping tests and is commonly used in fail-stop pipelines; FastCI instead uses historical information to schedule tests in non-fail-stop CI.The cited prediction signals include historical outcomes, developer information, learned representations, and LLM reasoning.Source: §2 · p. 3

3 OVERVIEW

FastCI maintains runtime evidence and execution history across CI runs, then uses that state to select and prune tests before optimizing their execution.

3 OVERVIEW

  • FastCI continuously updates a runtime evidence graph and test execution history, using this state to optimize selection and execution for each merge request.Its modular components are designed to integrate into existing CI pipelines with minimal changes.Source: §3 · p. 3
  • FastCI selects tests that reach changed graph entities, broadens selection when evidence is missing, and removes tests executing changed code in equivalent contexts.Figure 3 depicts direct selection from runtime evidence and within-class expansion when evidence is absent.Source: §3 · p. 4, Figure 3 · p. 4
  • For execution, FastCI prioritizes high-risk tests using change-related failure associations and recent flakiness, then reduces workloads outside each test’s validation scope.When workload reduction is not possible, it retains the original workload to preserve validation capability.Source: §3 · p. 4

4 DESIGN

FastCI's design combines runtime-evidence test selection and context-aware pruning with risk-based scheduling and workload optimization. The four techniques aim to reduce redundant testing, surface failures earlier, and avoid workload dimensions outside a test's validation scope.

4.1 EVIDENCE-GUIDED TEST SELECTION

  • FastCI selects tests using function-level dependencies in a runtime evidence graph, broadening selection when sampled evidence for modified entities is missing.For changes outside explicit function or method scope, the changed file's functions and methods are included; missing graph evidence triggers expansion to same-class methods and caller functions.Source: §4.1 · pp. 4–5
  • The runtime evidence graph is updated as the codebase evolves, by default sampling each CI run and updating after an MR is merged.For a stable codebase, updates may instead occur periodically, such as every three commits.Source: §4.1 · p. 5

4.2 CONTEXT-AWARE TEST PRUNING

  • FastCI prunes tests only when their execution contexts and validation metadata match, retaining a test with a different context.Contexts are caller-and-callee subgraphs within three hops of a modified entity; pruning is performed per modified entity, then retained tests are merged.Source: §4.2 · p. 5

4.3 RISK-AWARE TEST SCHEDULING

  • FastCI ranks tests by a risk score combining code-change failure associations with recent test instability, scheduling higher-risk tests first.The weighting of code-change risk increases with the number of historical observations for the current change-test associations.Source: §4.3 · pp. 5–6

4.4 WORKLOAD-OPTIMIZED TEST EXECUTION

  • FastCI reduces unrelated workload dimensions automatically when identifiable, but leaves dimensions unchanged when their relevance to a test's validation scope is unclear.It uses test metadata and historical execution paths to identify relevant dimensions, then applies reductions as configuration overrides without modifying test code; periodic comparisons with the original workload can withdraw reductions if outcomes diverge.Source: §4.4 · p. 6

5 EVALUATION

FastCI is evaluated across end-to-end CI performance, test selection, scheduling, workload optimization, and runtime-sampling tradeoffs. The evaluation reports lower CI and GPU costs alongside retained modified-code coverage.

5 EVALUATION

  • 77.5% lower CI latency and 63.9% lower GPU usage versus deployed CI accompany 3.2% higher modified-code coverage on ByteDance’s framework workload.After production integration, latency fell 58.4%, with no reported regressions caused by omitted tests across 784 CI runs over three weeks.Source: §5 · p. 7

5.1 END-TO-END CI PERFORMANCE

  • 18.6%, 6.1%, and 1.8% of full-CI GPU usage, latency, and queueing time, respectively, are FastCI’s averages.The evaluation sampled production MRs in shadow CI over two days; FastCI outperformed path-based CI and NameRTS at every percentile for latency and queueing.Source: §5.1 · pp. 7–8
  • 79.2% lower median CI latency and 97.3% lower median queueing time versus path-based CI show gains across MRs, not just inexpensive cases.At the 90th percentile, latency and queueing reductions were 78.3% and 89.0%, respectively.Source: §5.1 · p. 8

5.2 EFFECTIVENESS OF TEST SELECTION

  • Evidence-guided selection improves efficiency and coverage over path-based selection; context-aware pruning further reduces selected tests with little modified-code coverage loss.Versus path-based selection, evidence guidance changes selected-test ratio from 62.6% to 35.9%, normalized GPU cost from 67.6% to 39.4%, affected-test recall from 76.1% to 84.5%, and modified-code coverage retention from 96.0% to 99.8%. Pruning reduces selected-test ratio to 25.7% and normalized GPU cost to 28.6%; coverage retention is 99.2%, while affected-test recall is 66.2%.Source: §5.2 · p. 8, Table 1 · p. 8
  • LLM reasoning retains 54.6% of affected tests and 83.1% of modified-code coverage; NameRTS reaches 94.0% recall but 80.8% normalized GPU cost.The paper attributes LLM reasoning’s coverage shortfall to difficulty inferring complete end-to-end execution paths, and NameRTS’s cost to over-selection.Source: §5.2 · p. 8

5.3 EFFECTIVENESS OF TEST SCHEDULING

  • 63.6% lower median first-failure rank and 89.2% lower median GPU consumption before the first failure versus production FIFO are reported for FastCI.Against COLEMAN, median first-failure rank falls from 9 to 4 and GPU consumption before the first failure from 12.6 to 2.3 card-hours; instability and code-change indicators offer complementary benefits.Source: §5.3 · p. 9, Figure 7 · p. 9

5.4 EFFECTIVENESS OF WORKLOAD OPTIMIZATION

  • 11.8% lower average single-test execution time and 19.3% lower GPU consumption accompany end-to-end reductions versus FastCI-select.With workload optimization, FastCI further reduces GPU usage by 18.3%, average CI latency by 28.6%, and test queueing time by 46.6%.Source: §5.4 · p. 9

5.5 OVERHEAD AND COVERAGE OF RUNTIME SAMPLING

  • 50 Hz retains 99.2% of full-suite modified-code coverage with 4.1% overhead; 100 Hz adds 0.4% retention but raises overhead to 17.0%.The evaluation describes 50 Hz as a suitable tradeoff and reports that sampling overhead does not become a new CI bottleneck.Source: §5.5 · p. 9, Table 2 · p. 9

6 CONCLUSION

FastCI improves CI efficiency for LLM training frameworks, with lower latency and GPU use and higher modified-code coverage retention than the deployed pipelines. It is integrated into ByteDance production CI; the authors also describe their use of generative AI tools.

6 CONCLUSION

  • CI latency fell 77.5% and GPU resource usage fell 63.9%, while modified code coverage retention rose 3.2% over the deployed CI pipelines.These production CI results are reported alongside FastCI's integration into ByteDance production CI.Source: §6 · p. 10
  • FastCI uses a runtime evidence graph to select and prune tests while optimizing their scheduling and execution.The framework is integrated into production CI at ByteDance.Source: §6 · p. 10

AI USE STATEMENT

  • Generative AI assisted with language editing, literature search, reference organization, and plotting-script implementation, but not experiments or core methodology.The authors report that they reviewed and revised AI-assisted textual suggestions and take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A RISK SCORE COMPUTATION FOR TEST SCHEDULING

FastCI computes test risk from historical code-change associations and recent test instability, then orders tests by decreasing risk. Its signals use prior CI outcomes, with specified smoothing, attenuation, and replay safeguards.

A RISK SCORE COMPUTATION FOR TEST SCHEDULING

  • The code-change risk signal uses statistics from previous CI runs, including the file with the highest positive historical failure rate for a test.A file-test pair without previous observations contributes zero.Source: A RISK SCORE COMPUTATION FOR TEST… · p. 14
  • λ smooths how quickly code-change risk gains confidence; when no changed file has a positive historical failure rate, code-change risk and its confidence are zero.In that case, the score is determined solely by the instability indicator; λ is set to 5 in the evaluation.Source: A RISK SCORE COMPUTATION FOR TEST… · p. 14
  • The instability signal uses up to W recent test outcomes, exponentially weights failures, and attenuates the rate after consecutive passes.The evaluation uses W = 5 and γ = 0.8; attenuation factors are 0.85 and 0.6 after two and four consecutive passes, respectively.Source: A RISK SCORE COMPUTATION FOR TEST… · p. 14
  • Tests are ordered by decreasing risk, while offline replay processes MRs chronologically and uses the first 30 MRs only to warm up history.Statistics are updated only after each MR has been scored to avoid future-information leakage.Source: A RISK SCORE COMPUTATION FOR TEST… · p. 14

B MAINTAINING THE RUNTIME EVIDENCE GRAPH

FastCI samples test execution and converts traces into a per-test runtime evidence graph. It updates graph evidence after CI runs so later MRs can use the refreshed graph for test selection.

B MAINTAINING THE RUNTIME EVIDENCE GRAPH

  • FastCI adds lightweight sampling to the existing CI launch script rather than modifying individual tests.It samples the rank-0 Python process with py-spy; because short-lived functions and native execution may be missed, absent entities trigger the expansion mechanism.Source: B MAINTAINING THE RUNTIME EVIDENCE GRAPH · p. 14
  • Trace processing removes frames outside the repository, normalizes paths, resolves Python methods to class-qualified names when possible, and records caller-callee edges.Frames also record which test executed each function, and traces for a test are combined into one partial graph.Source: B MAINTAINING THE RUNTIME EVIDENCE GRAPH · p. 14
  • After CI, FastCI replaces graph evidence for successfully traced tests, preserves evidence for unexecuted tests, and stores the merged graph under the new commit.A later MR uses the updated graph when its merge base matches that commit, continuing the collection-selection-update loop.Source: B MAINTAINING THE RUNTIME EVIDENCE GRAPH · p. 14

C.1 LLM SELECTION BASELINE

The LLM selection baseline uses a code agent to choose candidate tests from merge-request changes and repository context. Its fixed instruction favors recall when impact is uncertain, and the agent receives no current or future execution outcomes.

C.1 LLM SELECTION BASELINE

  • The baseline uses a GPT-5.5 code agent given the MR diff, evaluated repository, and CI test definitions.It operates on the same MR revisions and candidate tests as the other evaluated methods.Source: §C.1 · p. 15
  • Its fixed instruction asks the agent to inspect related code and CI entry points and select candidate tests that may reach changed behavior.When impact is uncertain, it favors recall while keeping selections within the MR-triggered candidate set.Source: §C.1 · p. 15
  • The baseline uses the same instruction and repository-access scope for every evaluated MR and provides no current or future execution outcomes.Source: §C.1 · p. 15

C.2 COLEMAN SCHEDULING BASELINE

This section describes the COLEMAN scheduling baseline and the offline reference used to measure modified-code coverage retention. It also reports FastCI’s staged deployment, safeguards, and operational changes.

C.2 COLEMAN SCHEDULING BASELINE

  • COLEMAN ranks tests using prior outcomes, combining normalized TimeRank rewards with an exploration bonus and preventing current-MR outcomes from leaking into the ranking.It uses FRRMAB with a 100-cycle history window, decay factor 1, and ten fixed seeds.Source: §C.2 · p. 15

C.3 OFFLINE COVERAGE REFERENCE

  • Modified-code coverage retention measures the fraction of full-suite-covered executable lines in modified functions that selected tests also cover.The reference excludes native C++ and CUDA execution; 100% retention does not ensure the same inputs or assertions.Source: §C.3 · p. 15

D.1 DEPLOYMENT PROCESS

  • FastCI was validated end to end on shadow CI before production, then rolled out from limited tracing and gated MRs to default use for eligible MRs.The rollout was reviewed with the development team before enabling production workloads.Source: §D.1 · p. 15

D.2 DEPLOYMENT SAFEGUARDS

  • During rollout, YAML-only changes stayed on existing CI rules, and configuration controls allowed FastCI to stop handling MRs or be disabled.Both rollback levels required only a configuration update.Source: §D.2 · p. 16

D.3 OPERATIONAL LESSONS

  • On-demand graph materialization made trace-processing overhead per MR O(1) in accumulated traces, while local environment-based settings nearly eliminated configuration overhead.Eagerly maintaining graphs for every version had caused test-selection timeouts as traces accumulated.Source: §D.3 · p. 16
Loading 2610.01967v1…