Backstitch: Restoring Request Causality Across a Production Microservice Fleet

• In one fleet snapshot, 673 of 1,133 services carried at least one break, fragmenting records even though downstream work continued and requests succeeded.The incomplete history complicates incident diagnosis, traffic attribution, and capacity planning.Source: §1 · p. 1
• Backstitch replays target workloads against an idle baseline to identify request-correlated calls, traces the responsible handoff through source code, makes a bounded repair, and validates it with replay.The repair restores the causality, lifetime, and attribution required by the handoff.Source: §1 · p. 2
• Over 112 days, the actionable break rate fell from 20.12% to 8.55% across 26,136 calls in 2,587 services, while the monitored service count grew 13.7%.In a cohort of 240 calls followed through first production deployment, the break rate fell from 90.46% to 4.69%.Source: §1 · p. 2
• Shared frameworks and transport standards propagate context on recognized paths, but unrecognized handoffs leave a residual long tail that is costly to repair manually.Backstitch complements framework-level prevention by making one-at-a-time repairs affordable.Source: §1 · pp. 1–2

2.1 One Request, Many Services

• A request can span many independently developed services and asynchronous tasks, while observability records its history to support diagnosis, dependency analysis, traffic attribution, and capacity planning.No engineer has complete knowledge of every participating service and execution path.Source: §2.1 · p. 2

2.2 How We Observe the System

• Distributed tracing connects recorded spans through trace identifiers and parent–child relationships, with trace context identifying the trace and current parent span.W3C Trace Context defines interoperable encodings for carrying trace context across service boundaries.Source: §2.2 · p. 2

2.3 Context Propagation

• Context propagation transfers request metadata along execution paths; local handoffs must pass context to workers, and RPC frameworks encode and recover it across process boundaries.Context may be passed explicitly or made available through execution-local state.Source: §2.3 · p. 3

3.1 Failure Semantics and Symptoms

• A root-like downstream call is only a diagnostic candidate because legitimate periodic, startup, maintenance, or detached work can also lack a visible request parent.Passing incoming context everywhere is unsafe: retries may outlive deadlines, and batches may combine multiple requests.Source: §3.1 · p. 3

3.2 Why Missing Context Matters

• Missing context can distort dependency records and traffic attribution, while live control decisions also depend on correct request context.A trustworthy causal chain must preserve topology, attribute latency and errors correctly, exclude unrelated work, and respect concurrency, timeout, and lifecycle semantics.Source: §3.2 · pp. 3–4

3.3 Production Measurement

• 673 of 1,133 services (59.4%) had at least one propagation break; among inspected root-like calls, 15.6% were lost request continuations, while 53.3% were non-request or intentionally detached work.For SDK-linked broken-call samples, 91.1% were at graph depth 5 or greater and 80.5% at depth 8 or greater; median manual repair took 90 minutes before review and rollout.Source: §3.3 · p. 4

3.4 Why Diagnosis and Repair Are Difficult

• Diagnosis is difficult because breaks are silent, can occur across heterogeneous handoffs and code ownership boundaries, and cannot be established or validated by source inspection or functional tests alone.A repair must preserve causality, lifetime, and attribution rather than uniformly copying context.Source: §3.4 · pp. 4–5

3.5 Limits of Existing Approaches

• Existing mechanisms automate propagation only across recognized paths, while source transformation and trace analysis do not determine whether a symptom is a defect or validate the deployed repair.Backstitch complements prevention by diagnosing residual symptoms, locating uncovered handoffs, and validating repairs in production.Source: §3.5 · p. 5

4.1 Design Overview: Execution as the Reference

• Replay distinguishes request-driven work from independent activity, source analysis finds the earliest unsupported handoff, and a repeated replay checks that the repair restores causality without capturing unrelated work.A request-scoped task may require context capture at submission, restoration only during execution, and clearing before the worker accepts another task.Source: §4.1 · pp. 5–6

4.2 Diagnosing and Repairing One Break

• Diagnosis compares idle and target-request traffic, traces execution paths to the repair anchor, and selects a bounded change that restores the handoff’s causality, lifetime, and attribution.The smallest sufficient repair belongs in the module that owns the violated contract; runtime validation checks that the target call remains, the root-like observation disappears, and unrelated work does not inherit context.Source: §4.2.1 · p. 6, §4.2.2 · p. 7, §4.2.3 · p. 7

4.3 Fleet-Wide Repair

• The fleet campaign groups candidates by service, repairs bounded waves, and re-queries telemetry to expose remaining, newly visible, or newly introduced breaks.Recurring validated repairs can become knowledge-base rules, but every later match must still pass runtime validation.Source: §4.3 · p. 7

4.4 Why a Specialized Agentic System

• Backstitch specializes a coding agent’s work by scoping cases before coding, constraining allowable repairs, preserving evidence, and continuing through runtime validation.The system selects cases and controls repair state; the agent supplies source understanding and modification.Source: §4.4 · pp. 7–8

5 Implementation

• The implementation comprises about 9,100 lines of Go code for MCP integrations and 32,400 lines of Skills and repair-knowledge specifications.Figure 5 presents the implementation around an interchangeable coding agent.Source: §5 · p. 8

5.1 Agent Architecture

• Workflow Skills define stages and exit conditions, while specialized Skills handle localization, repair, dependency updates, audits, and code-deploy validation.Operational tools expose runtime, source, dependency, build, deployment, replay, and review functions; service workspaces preserve evidence and checkpoints for pause and resume.Source: §5.1 · p. 8

5.2 Diagnose: Replay and Localize

• Controlled replay compares idle and target-request windows, while read-only localization traces accessible callsites backward through application and dependency code to identify the repair anchor.If accessible source cannot establish an anchor, the localization process records the missing evidence and abstains.Source: §5.2 · pp. 8–9

5.3 Fix and Validate

• A repair pattern must match the complete execution path and justify the required contract; a separate audit checks the recorded source claims and diff before replay-based validation.Validation reuses the diagnosis workload and observation parameters, and validated patterns can be promoted for later reuse.Source: §5.3 · p. 9

5.4 Cross-Repository and Fleet Operation

• Cross-repository repairs preserve path evidence and validation state, while fleet campaigns run isolated service workflows in bounded waves and re-query telemetry after owner-approved rollout.Repository owners review, approve, and release merge requests manually; interactive and batch modes share the same repair logic and validation standard.Source: §5.4 · p. 9

6.2 Repairs Restore the Missing Causal Chain

• Across 112 days, the 27-scenario actionable break rate fell from 20.12% to 8.55%; among 240 rollout-aligned tuples, the break rate fell from 90.46% to 4.69%.The rollout-aligned break rate also fell to 8.70% in the intermediate window; all 16 services improved in both post-deployment windows.Source: Figure 6 · p. 10, §6.2 · p. 10

6.1 Methodology

• The evaluation combines 27 monitored request scenarios, a personalized content-feed call graph, a 240-tuple rollout-aligned cohort, and an offline benchmark of 14 services.The benchmark scores target reach and source-location agreement against human repairs, without live deployment or production feedback.Source: §6.1 · p. 10

6.3 Specialization Drives Repair Capability

• Removing dependency-source access reduced targets reached from 78 to 55; removing knowledge-base guidance left all 78 targets reachable but produced clearly wrong repairs for at least 39.Removing candidate-commit builds and source-level checks reduced targets reached to 73.Source: §6.3 · p. 11

6.4 The Fleet Loop Sustains Repair

• Owners approved 96 of 114 reviewed merge requests (84.2%); shared-library repairs were approved more often than caller-service repairs, 94.1% versus 76.2%.Interactive mode repaired a higher share of deployed cases, while batch mode processed almost twice as many breaks.Source: §6.4 · p. 11

6.2 Repairs Restore the Missing Causal Chain

• With Codex and GPT-5.4, Backstitch reached all 78 production-confirmed targets and matched the human repair location in 77; without Backstitch, the pair reached 6.Every tested condition left all eight leave-unchanged cases untouched.Source: §6.2 · p. 11, §6.3 · p. 11, Figure 7 · p. 12

6.3 Specialization Drives Repair Capability

• Across matched agent-model pairs, Backstitch increased targets reached for all six models, including GPT-5.4 from 6 to 78 and GLM-5 from 0 to 5.The full configuration also raised OpenCode from 24 targets to all 78, while the scripted fixer reached 32.Source: §6.3 · p. 11, Figure 7 · p. 12

6.4 The Fleet Loop Sustains Repair

• After the second-round peak, candidate services fell 94.19%; no request-causality-preserving repaired calls recurred at 30 days (0/82) or 60 days (0/78).Fifteen recurring root-like calls were periodic, self-initiated, or uncontrolled refresh work.Source: §6.4 · p. 12, Figure 8 · p. 12
• Repeated remeasurement reveals newly visible breaks, while deployed repairs restore causal records without interrupting downstream execution.The fleet loop carries repairs through owner review and adoption, then identifies subsequent repair work.Source: §6.4 · p. 12

Context propagation

• Tracing infrastructure, transport standards, and related mechanisms presume recognized handoffs; Backstitch targets uncovered ones from production symptoms.Other compared approaches include context libraries, asynchronous middleware, source transformation, and sidecars.Source: §7 · p. 12

Trace reconstruction

• Casper, TraceWeaver, and related systems infer request structure offline, whereas Backstitch repairs propagation in the live execution.Offline reconstruction cannot govern worker deadlines, cancellation, identity, or labels during execution.Source: §7 · p. 12

Automated repair and validation

• Unlike Dr.Fix and ORCA [25], which begin from defect or incident evidence, Backstitch faces successful requests and passing tests, so acceptance relies on deployed-revision runtime validation.A weak patch signal can pass without fixing the propagation break.Source: §7 · p. 13

Coding agents and fleet change

• Backstitch’s contribution is the system around the coding agent: repository repair mechanics [11] [29], staged fleet change, and governed execution [7] [20].Unlike STRATUS, Backstitch produces reviewed source changes remeasured after rollout.Source: §7 · p. 13

8 Conclusion

• Backstitch turns surviving execution into a repair reference, from production symptom through bounded source change to validating replay.The conclusion says the same approach applies wherever a record breaks while the behavior it describes survives.Source: §8 · p. 13
• Over 112 days, the fleet’s propagation-break rate more than halved.Source: §8 · p. 13
• For handoffs frameworks miss, the paper argues that a specialized agentic system with expert knowledge and runtime validation makes repairs affordable.Source: §8 · p. 13

A.1 Repository and Workspace Organization

• A.1 separates workflow policy, reusable repair knowledge, and platform integrations, while each service workspace preserves resumable progress and evidence.The workspace retains checkpoints, paths, revisions, deployments, and metric comparisons so work can resume without reconstructing evidence from conversation history.Source: §A.1 · p. 16, Figure 9

A.2 MCP Server

• A.2 keeps platform authentication and API details outside the agent prompt; registered tools expose operations, while Skills govern when they may be used.Table 1 lists representative MCP-server operations.Source: §A.2 · p. 16, Table 1 · p. 17

A.3 Skill-Governed Agent Execution

• A.3 constrains movement from investigation to modification with Skills, and only the repair stage may modify a repository.Read-only localization and audit help prevent a plausible patch from serving as evidence that its own diagnosis was correct.Source: §A.3 · pp. 16–17, Table 2 · p. 17

A.4 Recording Candidate Execution Paths

• A.4 records candidate paths as inspectable source claims, checks them against recognition rules, and leaves evidence gaps explicit when no path supports repair.The service-level index links each call to its own evidence record and status, enabling resumption at the first incomplete call.Source: §A.4 · p. 18

A.5 Knowledge Rules and Negative Constraints

• A.5 encodes applicability conditions as positive evidence, repairs, and exclusions, so superficially similar worker executions can require different parents and lifetimes.Negative rules are checked before modification and again during audit.Source: §A.5 · p. 19

A.6 Checkpointed Validation Evidence

• A.6 retains step-indexed validation state and operation history, binding metric comparisons to candidate and baseline environments and supporting resumable runs.The final report classifies calls as repaired, remaining, or newly observed, and the overall result as full, partial, or failed closure.Source: §A.6 · p. 19, Figure 14 · p. 19

B Evaluation Details

• The appendix supplies benchmark construction rules, comparison tables, and supporting denominators for the main evaluation.Source: B Evaluation Details · p. 19
• Production measurements, rollout-aligned analysis, governance records, repeated content-feed measurements, and the offline benchmark are distinct datasets.Source: B Evaluation Details · p. 19
• The appendix identifies the records included and the unit counted in each subsection.Source: B Evaluation Details · p. 19

B.1 Controlled Benchmark

• The benchmark covers 14 real services with frozen pre-repair source and no live deployment or production feedback.Source: §B.1 · p. 19
• The 90 scored tuples include 78 production-confirmed targets, four still-broken tuples, and eight requiring no source change.The primary metric counts confirmed targets reached at a reference repair location; that match alone does not establish correctness.Source: §B.1 · p. 19
• The scripted fixer deterministically applies five knowledge-base repair patterns and reverts rewrites that fail parsing or formatting checks.Source: §B.1 · p. 19
• 72 targets reached are added for GPT-5.4 with Backstitch; the full system has 24/78 exact-diff matches.Source: §B.1 · pp. 19–20
• Without repair guidance, target and location matches remain 78/78, yet the audit finds 39/78 targets with conservatively counted wrong edits.The benchmark also leaves all eight leave-unchanged tuples untouched across completed conditions.Source: §B.1 · pp. 19–20
• Because benchmark inputs include the observed call tuple, the evaluation does not test production candidate discovery and only partially exercises diagnosis.Source: §B.1 · p. 20

B.2 Production Measurement and Deployment Indexing

• The production series covers a fixed registry of 27 business-critical scenarios and counts observed application-code services, excluding infrastructure targets.At the latest snapshot, the scope contains 26,136 tuples and 2,587 services.Source: §B.2 · p. 20
• Actionable tuple break rates fall from 20.12% to 8.55%, and actionable QPS-weighted rates from 6.90% to 1.87%, as observed services increase by 13.7%.Table 8 records the corresponding absolute counts.Source: §B.2 · p. 21, Table 8 · p. 22
• The final 28-day actionable tuple-rate mean is 13.16% versus 20.45% in the first 28 days, but changing traffic, topology, and other work make the series production context rather than a treatment estimate.Source: §B.2 · p. 21
• All four implementation-language groups with at least 100 tuples at both endpoints show improved actionable tuple break rates.Table 9 reports rates by primary implementation language.Source: §B.2 · p. 21, Table 9 · p. 22
• The rollout-aligned cohort retains 240 tuples across 16 services from 424 tuples with a post-deployment record; the event confirms a deployment after merge, not inclusion of the exact repaired commit.Source: §B.2 · p. 22
• Each of 16 services has a lower post-deployment break rate, with even the lower endpoint of each 95% interval showing a reduction above 60 percentage points.Both 95% intervals for presence include no change, so disappearing target calls do not explain the reduction.Source: §B.2 · pp. 22–23, Table 12 · p. 23

B.3 Workflow Adoption and Remaining Cases

• The governance snapshot contains 1,667 interactive records and 3,057 batch cases, not every production call or raw detector output.The two modes use separate records and labels, so their cases are reported separately rather than pooled.Source: §B.3 · p. 23
• 830 of 1,667 interactive records (49.8%) are marked fixed; among 442 records with explicit production verification, 424 are positive (95.9%).For 280 deployed batch cases with post-deployment results, 192 were repaired (68.6%).Source: §B.3 · p. 23
• 112 of 123 calls linked to shared or dependency repairs have an identified repair (91.1%), while downstream adoption is complete for 90 calls.The calls span 33 caller groups and 47 shared repositories; closure is recorded at the affected call because callers must still move to the new shared-library version.Source: §B.3 · p. 23
• 1,276 batch records fell outside campaign scope; five stated reasons account for 89.66% of that cohort.The five largest reasons were sidecar-originated calls (482), no static execution path (222), no released fixed dependency (210), insufficient break evidence (116), and a required breaking dependency upgrade (114).Source: §B.3 · pp. 23–24

B.4 Repeated Measurement of the Content-Feed Graph

• The content-feed graph’s newly visible candidate services fell from 1,223 in round 2 to 37 in round 6, a 94.19% reduction.The six rounds measured one production region’s graph, part of the 27-scenario production scope; rounds 5 and 6 contributed 2.76% and 1.97% of cumulative services.Source: §B.4 · p. 24

B.5 Durability Rule

• Zero of 82 observable request-continuation calls recurred through 30 days, and zero of 78 recurred through 60 days; approximate 95% upper bounds were 3.66% and 3.85%, respectively.At 60 days, 15 of 184 observable calls reappeared (8.15%); 14 were periodic or self-initiated and one was an uncontrolled dependency refresh.Source: §B.5 · p. 24

B.6 Operational Efficiency

• Across 16 caller services with at least one completed repair, nine repairs passed audit; the median service took 21.95 minutes, with half taking 12.87–30.89 minutes.The services used 43 agent sessions and 9.99 hours of agent time in total.Source: §B.6 · p. 24
• For fully accounted sessions, repair used 169.75 million tokens: 169.20 million sent and 0.55 million returned.Cached context accounted for 153.60 million input tokens (90.78%); median service use was 10.04 million tokens.Source: §B.6 · p. 24

B.7 Rollback Case Study

• A June 11, 2026 rollout raised reported request latency by about 300% after a tracing-library upgrade caused periodic calls to be measured as incoming requests.Removing the library and recording periodic calls under a separate span changed the metric; traffic-free test environments then stopped reporting those measurements.Source: §B.7 · p. 24

B.8 How the Audit Compares with Production Outcomes

• In a 27-repair set, all later recovered in production; the audit agreed on 21 of the 23 repairs it ruled on, but none failed in production.The set therefore leaves open how often an audit-passing repair fails in production.Source: §B.8 · p. 25