PaperlayerView exact revision on arXiv

Source-linked AI summary

From GPU to TPU with Coding Agents: What Works, What Fails, and What It Costs?

Hui Sun, Anderson Uchôa, Wesley K. G. Assunção

arXiv:2610.34422v1cs.SEcs.DCcs.LG

TL;DR

GPU-to-TPU migration offers deployment flexibility amid cost and vendor-capacity concerns, but verifying behavioral preservation is difficult when tests are scarce and accelerator outputs legitimately differ. In ten PyTorch projects migrated with Claude Code and independently assessed, eight reached L3, but 49% of failures raised no error, underscoring the need for independent verification.

  • Problem

    GPU-to-TPU migration offers deployment flexibility amid cost and vendor-capacity concerns, but verifying behavioral preservation is difficult when tests are scarce and accelerator outputs legitimately differ.

    Source: §I · p. 1

  • Method

    Claude Code migrated ten PyTorch research projects from GPUs to TPUs, and each migration was checked level by level using the L0–L4 consistency framework.

    Source: §VI · p. 10

  • Results

    8 of 10 projects reached L3, with 2.4 human interventions per migration on average; 49% of failures raised no error.

    Source: §I · p. 2

  • Takeaways & Limitations

    Agent-driven hardware migration needs independent, execution-based verification because successful execution alone does not establish correctness.

    Source: §I · p. 2, §VI · p. 10

    Evidence is limited to one agent, one TPU v5e target, PyTorch, and ten research projects that fit on one chip; other settings may behave differently.

    Source: §V · p. 10

Abstract

from arXiv · show

A coding agent can migrate a machine learning workload, report success, and still change how the model trains. Rising compute costs, limited accelerator capacity, and vendor dependence are pushing organizations toward alternatives to graphics processing units (GPUs); a key question is whether agent-generated migrations to tensor processing units (TPUs) can be trusted and at what cost. We build a dataset of 148 PyTorch ML research projects and use Claude Code to migrate ten of them from NVIDIA GPUs to Google Cloud TPUs. We independently assess whether each migration runs, preserves numerical behavior, executes on the intended hardware, maintains the project's goal, and delivers acceptable performance. We also examine failures, human effort, and costs. Eight projects meet our project-goal criterion, and migrations require 2.4 human interventions on average. Of 70 execution attempts, 53 fail (76%), and 26 of these failures (49%) raise no error. The most frequent cause is repeated compilation and other execution inefficiencies, while interactions with third-party libraries can silently change the computation: in one four-chip migration, the agent reports success despite a change in gradient clipping that reduces accuracy from 86.53% to 84.89%. Execution costs vary widely: in one workload, TorchTPU, a PyTorch execution backend for TPUs, trains 1.86 times faster and 25% cheaper per run than the GPU. For practitioners, these findings motivate independent reference runs, verification under the intended hardware configuration, and workload-specific acceptance and stopping criteria that account for both computational and human costs.

I. INTRODUCTION

This study asks whether agent-generated GPU-to-TPU migrations preserve workload behavior and whether their operational benefits justify migration and verification costs. Using independent checks of ten PyTorch research projects, it finds that project-goal preservation and execution success do not by themselves establish a trustworthy migration.

I. INTRODUCTION

  • On four chips, a pinned training library changed gradient clipping and reduced accuracy from 86.53% to 84.89% despite successful execution and an agent report of success.The library clipped gradients before averaging them across chips.Source: §I · p. 1
  • The study examines ten PyTorch research projects selected from 148 repositories, using Claude Code and independent checks across execution, numerical behavior, hardware integrity, project goals, and performance.The paper contributes migration evidence, effort and cost data, practitioner lessons, and a replication package.Source: §I · p. 2
  • Eight of ten projects reached L3, while 49% of failures raised no error and agent-reported outcomes were not always supported by independent checks.Migrations required 2.4 human interventions on average.Source: §I · p. 2
  • Overall, 49% of failures raised no error; the most frequent cause involved lazy-tensor execution, including repeated recompilation and host synchronization.Library interactions could also silently alter computation, while the agent’s diagnoses sometimes obscured the underlying cause.Source: §I · p. 2
  • In one GPT-2 experiment, TorchTPU trained 1.86 times faster and 25% cheaper per run than the GPU, with accuracy within 0.16% of the GPU’s.The case studies show that hardware configuration and backend choice can change migration outcomes.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

The paper distinguishes hardware-backend migration from conventional language or version migration and focuses on execution, training semantics, dependencies, and cost. It argues that agent-reported success and ordinary compilation or tests alone do not establish preserved workload behavior.

II. BACKGROUND AND RELATED WORK

  • PyTorch/XLA uses lazy-tensor graphs compiled via XLA, where unsupported operators may fall back to CPU and data-dependent shapes may trigger repeated compilation.TorchTPU executes eagerly by default on a native TPU device while also supporting compilation; neither backend guarantees bitwise GPU agreement.Source: §II · p. 2
  • Unlike migrations between languages or versions, this study retains PyTorch while changing hardware and backend, so acceptance requires evidence that relevant workload behavior is preserved.Research papers’ key claims provide workload-specific references, not universal correctness criteria.Source: §II · p. 2
  • Prior work includes agent-generated and optimized operators for Huawei Ascend NPUs, while execution-grounded checks have also been used for repository transformations.This study focuses on accelerator execution, training semantics, dependency interactions, migration costs, and failures without runtime errors.Source: §II · pp. 2–3
  • Hardware-migration feasibility may emerge only through resource-consuming attempts, while stopping rules can prematurely reject migrations that might succeed.The study examines early stopping as a trade-off between computational cost and abandoning a viable migration.Source: §II · p. 3

III. METHODOLOGY

The methodology selects a small, challenge-diverse sample of PyTorch research projects and evaluates migrations using five complementary consistency dimensions. It separates measured numerical agreement from categorical judgments and leaves acceptance thresholds specific to each workload.

A. Project Selection

  • The ten-project sample was stratified by research area and anticipated migration challenge, supporting comparisons across challenges but not representing industry ML workloads.Projects were drawn from 100 selected candidates after screening a corpus of 148.Source: A. Project Selection · p. 3
  • Two authors manually checked each sampled project’s eligibility, lack of an existing TPU path, and accessibility for evaluation.The checks constrained the sample to publicly accessible, resource-limited projects requiring an actual migration.Source: A. Project Selection · p. 3

B. The Consistency Assessment Framework

  • The assessment covers executability (L0), numerical agreement (L1), execution integrity (L2), project-goal preservation (L3), and performance (L4).L1 is a measurement, whereas L4 is judged only after L3.Source: B. The Consistency Assessment Framework · p. 3
  • The dimensions are ordered assessments, not a strict correctness hierarchy; repeated recompilation is a performance problem unless it prevents execution.Faster execution cannot compensate for a migration that changes the intended computation.Source: B. The Consistency Assessment Framework · p. 3
  • Because no universal numerical tolerance separates floating-point variation from defects across workloads, the study reports observed differences and leaves acceptance thresholds to application requirements.Smoke-test thresholds trigger investigation rather than establish correctness.Source: B. The Consistency Assessment Framework · pp. 3–4

C. Agent Setup

Each migration uses a fresh Claude Code session and a traceable repository of attempts, then selects and runs a paper-relevant experiment. The agent records claimed consistency levels, while the researchers independently inspect evidence and apply fail-fast rules.

C. Agent Setup

  • A fresh Claude Code session receives the paper, repository, and instructions, while a public migration repository records the original code and each subsequent attempt and failure.Attempts and failures are labeled with their level, cause, and whether they were silent.Source: C. Agent Setup · p. 4
  • The agent prioritizes an experiment relevant to the paper’s key claim, executable, feasible on one TPU v5e chip, and supported by accessible data and a baseline when needed.The target run is limited to approximately four hours; additional permissions require approval rather than selection of a weaker alternative.Source: C. Agent Setup · p. 4
  • GPU reference runs and a matching TPU smoke test precede migration attempts, which are documented with changes, expected consistency levels, and observed outcomes.The GPU runs use an NVIDIA L4, fixed seeds, and include both a smoke test and a full run.Source: C. Agent Setup · p. 4
  • After full runs, researchers independently inspect evidence for L0–L4, including numerical differences, device placement, recompilation, optimization semantics, and cost.The agent stops at ten attempts or three hours without L0, on an unavoidable blocker, or when smoke-test differences or projected runtime require a decision.Source: C. Agent Setup · p. 4

D. Data Collection and Manual Analysis

The study combines execution records, independent code and outcome inspection, and matched GPU–TPU runs to assess migration consistency. Eight of ten projects reached L3, while performance and numerical agreement varied across workloads.

D. Data Collection and Manual Analysis

  • The researchers collected pull requests, failure reports, agent logs, code changes, and GPU/TPU execution records, then independently inspected outcomes and assigned failure causes.The first author based root-cause labels on errors, agent diagnoses, and subsequent fixes.Source: D. Data Collection and Manual Analysis · p. 4
  • The assessment compares published metrics with single full GPU and TPU runs under matching settings and the same seed, distinguishing expected, claimed, and independently measured levels.When the GPU run also missed the paper’s claim, a matching TPU run counted as preserving original behavior; failures were classified by cause, and costs were measured using token use, elapsed migration time, and per-run training time times machine list price.Source: D. Data Collection and Manual Analysis · p. 4, 1) RQ1 Consistency · p. 4

1) RQ1 Consistency:

  • Eight of ten projects reached L3, three also reached L4, and migrations required an average of 2.4 human interventions.Three projects preserved original GPU behavior without reproducing their papers’ claims; matching the original run alone was not enough for acceptance.Source: 1) RQ1 Consistency · p. 5, Table I · p. 5
  • Across ten projects, the median number of attempts before completion or stopping was 6.5 (range 3–11), and only MEND reached L2 on its first migrated attempt.The authors recommend budgeting for iterative debugging and validation rather than expecting a one-shot migration.Source: 1) RQ1 Consistency · p. 5
  • Across nine projects reaching L0, GPU–TPU metrics differed by a median of 0.43% (range 0.01%–3.43%), while the median TPU-to-paper difference was 0.95%.Differences were below 1% in seven projects; the authors recommend establishing a GPU reference before attributing discrepancies to migration.Source: 1) RQ1 Consistency · p. 5
  • Full-run time after migration was a median of 0.62 times the original (range 0.33 to 2.58); five projects ran faster and four slower.Only MELoRA, TDPO, and MEND reached L4; shorter runs for LM-Steer and UniPELT reflected faster evaluation or early stopping, respectively.Source: 1) RQ1 Consistency · p. 5
  • The agent’s final claims matched independent assessments in six completed projects and were lower in three, yet the four-chip case still exposed a false success.In that case, the agent reported success despite a silent accuracy regression; final claims therefore still required independent verification under the intended hardware configuration.Source: 1) RQ1 Consistency · p. 5

2) RQ2: Failure Causes:

Across ten projects, most failed attempts raised no explicit error. Repeated compilation was the most frequent cause, while library interactions could silently change the computation.

2) RQ2: Failure Causes:

  • 53 of 70 attempts failed (76%), including 26 failures (49%) without explicit errors.The agent most often labeled failures as device, numerics, or other, without distinguishing lazy-tensor execution or problems inherited from the original code.Source: 2) RQ2: Failure Causes · p. 6
  • Environment drift and hard-coded CUDA assumptions appeared in seven and eight failures, respectively, and typically produced errors that helped diagnose them.The agent’s broad labels obscured mechanisms such as recompilation and host synchronization, while predicted challenges matched the most frequent observed cause in only three of eight applicable projects.Source: 2) RQ2: Failure Causes · p. 6
  • Lazy-tensor execution caused 13 failures across seven projects, all without errors.Repeated recompilation and host synchronization could make steps tens of times slower or increase host memory while training continued.Source: 2) RQ2: Failure Causes · p. 6
  • Among five third-party-library failures, CoFi retained stale parameter copies and MELoRA failed to reload its best checkpoint, both silently changing outcomes.Their effects became apparent only by comparison with the GPU reference; other cases produced explicit errors involving parameter replacement or checkpoint restoration.Source: 2) RQ2: Failure Causes · p. 6
  • Reliable migration acceptance requires runtime monitoring and behavioral verification beyond successful execution and agent-generated diagnoses.Nearly half of failed attempts raised no error, and third-party interactions could silently alter computation.Source: 2) RQ2: Failure Causes · p. 6

3) RQ3: Cost and Failing Fast:

Migration costs include more than agent tokens: accelerator runs, validation, and human interventions also contribute. Stopping rules can reduce wasted computation but may falsely reject viable migrations.

3) RQ3: Cost and Failing Fast:

  • Agent token charges had a median of $9.74 (range $2.00–$22.55), but five of nine completed TPU runs cost less than GPU runs and four cost more.Failed attempts, validation runs, and human interventions add costs beyond a single successful execution; equivalent workloads also matter because early stopping lowered UniPELT’s TPU cost.Source: 3) RQ3: Cost and Failing Fast · p. 6
  • Three stopping rules triggered nine distinct stops, and six of the seven smoke-test stops were false alarms.Only one smoke-test stop detected a migration-induced error; the budget stop could not be assessed, and time-estimate stops flagged cost rather than incorrect computation.Source: 3) RQ3: Cost and Failing Fast · p. 6
  • A CoFi smoke-test gap fell from 5.3% to 0.29% when both executions used the same teacher, showing that the reference setup itself can trigger rejection.The initial discrepancy came from independently trained teacher models; the first stop, in contrast, correctly flagged pruning and fine-tuning losses above the GPU reference.Source: 3) RQ3: Cost and Failing Fast · pp. 6–7
  • The agent stopped excessively slow attempts, but numerical gates produced false alarms and required human review of ambiguous outcomes.The authors call for workload-aware stopping criteria rather than fixed thresholds alone.Source: 3) RQ3: Cost and Failing Fast · p. 7

B. Case Study: Single- vs Multi-chip TPU Migration

The single-chip migration closely matches the GPU’s accuracy, while the four-chip run completes but silently changes training behavior and misses the project-goal criterion. The single-chip result is slower and more expensive than the GPU.

Results

  • 84.89% on four chips fell below the GPU’s 86.53% and the published LoRA baseline of 85.6%, failing the L3 criterion despite successful completion.Accuracy was below the GPU at all 73 evaluation points, and training loss ended higher at 0.446 versus 0.378.Source: B. Case Study: Single- vs Multi-chip… · p. 7
  • 86.54% on one chip closely matched the GPU’s 86.53%, with the final training loss also matching at 0.378.Accuracy remained within 0.62% of the GPU at every evaluation from step 2,000 onward.Source: B. Case Study: Single- vs Multi-chip… · p. 7

Migration on one chip v5e-1

  • 0.89 times the GPU’s throughput and 1.59 times its per-run cost made the one-chip migration unattractive on L4 performance and cost.The chip was busy about 13% of a sampled interval, compared with 91% for the GPU.Source: B. Case Study: Single- vs Multi-chip… · p. 7

Silent failure on four chips v5e-4

  • Per-chip gradient clipping before cross-chip averaging changed the optimization procedure, unlike the GPU’s clipping of the full batch gradient once.The agent preserved the global batch but did not verify the order of clipping and averaging; the dependency lacked an XLA-specific clipping path.Source: B. Case Study: Single- vs Multi-chip… · pp. 7–8

Finding

  • A migration validated on one TPU chip can silently alter distributed optimization on multiple chips, so verification must match the intended hardware configuration.The study identifies distributed training semantics and dependency behavior as acceptance checks.Source: B. Case Study: Single- vs Multi-chip… · p. 8

C. Case Study: Migrating to Different Backends

On the same single-chip TPU, PyTorch/XLA and TorchTPU both stayed close to GPU accuracy, but only PyTorch/XLA passed the study’s L3 criterion. TorchTPU trained faster and at lower per-run cost than the GPU, while the comparison is limited by single-seed runs and different PyTorch versions.

Design

  • Different PyTorch versions (2.6 and 2.14) mean the backend comparison does not isolate backend effects.The runs otherwise kept the global batch size, training steps, and other hyperparameters the same.Source: C. Case Study: Migrating to Different… · p. 8

Results (L1, L3)

  • 81.80% on MNLI-matched let PyTorch/XLA pass L3, while TorchTPU’s 81.23% fell 0.16 percentage points below the GPU’s 81.39%.TorchTPU reached 82.70% on MNLI-mismatched, slightly above the GPU’s 82.66%, but the two TPU backends differed by an average of 0.50% across 73 evaluations.Source: C. Case Study: Migrating to Different… · p. 8
  • A single seed and different PyTorch versions prevent attributing backend accuracy differences to the backend alone, so acceptance thresholds must account for workload requirements and numerical variability.The small TorchTPU shortfall does not establish a backend-induced defect.Source: C. Case Study: Migrating to Different… · pp. 8–9

Migration on each backend

  • Three added files and no modified originals made TorchTPU’s migration smaller than PyTorch/XLA’s five-file change, but device execution still required verification.TorchTPU compatibility issues had silently placed the model and batches on the CPU when CUDA was unavailable.Source: C. Case Study: Migrating to Different… · p. 9

Cost and performance (L4)

  • 1.98 hours made TorchTPU 1.86 times faster than the GPU’s 3.68 hours, and its $2.37 run cost was 25% below the GPU’s $3.14.The TPU hourly price was 41% higher; TorchTPU kept the chip busy 88% of training time versus 11% for PyTorch/XLA.Source: C. Case Study: Migrating to Different… · p. 9

Room for optimization

  • 186 ms per step spent tracing and dispatching in PyTorch/XLA, against 26 ms of chip work, points to host overhead; GELU fusion cut TorchTPU’s step time from 150 to 112 ms.PyTorch/XLA barely changed with the fusion.Source: C. Case Study: Migrating to Different… · p. 9

D. Implications for Practice

The study’s research-project setting exposes practical migration challenges, but does not establish how often they occur in production. Its implications emphasize independent acceptance evidence, hardware-specific semantic checks, and accounting for migration, validation, and execution costs.

Implications for Practice

  • Only seven of the 148 collected repositories contained author-written tests, and dependency compatibility repeatedly complicated migration.The findings inform engineering practice but do not establish how frequently the problems occur in production.Source: D. Implications for Practice · p. 9

For practitioners

  • Teams should set workload-specific acceptance criteria and compare training curves and evaluation metrics with an independent GPU reference under comparable settings and seeds.When the original execution misses its expected outcome, teams should distinguish inherited discrepancies from migration-induced changes.Source: D. Implications for Practice · p. 9
  • Checks should match the intended hardware configuration, including device placement, checkpoint restoration, optimizer state, effective batch size, and gradient operations.The four-chip case shows that validation on one chip may not reveal altered distributed optimization on multiple chips.Source: D. Implications for Practice · p. 9
  • TorchTPU reduced one workload’s training time by 1.86× and per-run cost by 25%, while TPU/GPU run costs ranged from 0.47 to 3.6 times across projects.Agent token charges, failed attempts, verification runs, and an average of 2.4 human interventions also contribute to total migration cost.Source: D. Implications for Practice · p. 9

For migration tool builders

  • Tools should verify execution integrity, numerical agreement, and project-specific goals before reporting success, using GPU/TPU comparisons and runtime evidence.Relevant evidence includes device placement, CPU fallback, compilation activity, training curves, and operations such as checkpoint restoration and gradient reduction before clipping.Source: D. Implications for Practice · p. 9
  • Smoke tests caught problems before full runs, but fixed thresholds produced six false alarms among seven smoke-test stops.Configurable acceptance thresholds, computational budgets, stochastic-evaluation handling, and human approval can make stopping decisions evidence-driven.Source: D. Implications for Practice · p. 10

Implications for Practice

  • Seven of nine completed TPU runs had accelerator utilization below 25%, and GELU fusion reduced TorchTPU fixed-batch step time by 25%.The proposed workflow separates consistency checks from subsequent optimization and reruns checks after optimization.Source: D. Implications for Practice · p. 10

V. THREATS TO VALIDITY

The study’s consistency judgments, run-to-run comparisons, and cost estimates have methodological and scope limits. Its results are descriptive for a small, specific set of projects, tools, and hardware configurations.

Construct validity

  • A single seed cannot distinguish before-and-after differences below 1% from run-to-run variation, and another analyst might classify some failures differently.The authors report L1 as a measurement rather than a verdict; their L3 judgment relaxes the requirement and could let a weaker migration pass.Source: §V · p. 10

Internal validity

  • Dependency upgrades may account for part of an observed difference, because the agent changes pinned library versions to support the target.The agent is nondeterministic, and experimenter decisions such as approving a full run after a smoke-test gate also influence outcomes.Source: §V · p. 10

External validity

  • Ten research projects, one agent, PyTorch, and a target that was mainly one TPU v5e chip bound generalization; the authors make no claim of statistical generalization.Production code, larger models, multi-chip targets, other agents or accelerators, and newer software versions may behave differently.Source: §V · p. 10

VI. CONCLUSION AND FUTURE WORK

The study finds that migration success requires independent, execution-based verification, because running code can still change workload behavior. Future work targets migration tooling, budget-aware stopping rules, failure diagnosis, and broader cross-hardware benchmarking.

VI. CONCLUSION AND FUTURE WORK

  • 8 projects reach L3, including 3 whose matching GPU runs themselves miss the paper’s claim, with 2.4 human interventions per migration on average.The authors conclude that agent-driven hardware migration needs independent, execution-based verification.Source: §VI · p. 10
  • 49% of failures raised no error, and interactions between migrated code and third-party libraries were among the hardest problems to find.Whether migration pays off depends on the target and its configuration.Source: §VI · p. 10
  • Future work will turn migration and verification into a tool for submitting only L3 replication-package migrations as pull requests.The plan targets replication packages of newly published papers.Source: §VI · p. 10
  • The authors plan budget- and target-aware early stopping, a failure-diagnosis benchmark and knowledge base, and a cross-hardware ML migration benchmark scored by consistency level.The proposed benchmark would ask agents to migrate research projects to target accelerators and compare against original-hardware reference runs.Source: §VI · p. 10

Related papers

Loading 2610.34422v1…