PaperlayerView exact revision on arXiv

Source-linked AI summary

Independent Verification Paths Are Not Independent: A Case Study of Common-Mode Failure in a Satellite Catalogue Pipeline

Fabio Rovai

arXiv:2609.37603v1cs.SEcs.AIcs.DB

TL;DR

Redundant computation is used to check pipeline outputs by comparing routes built with different technology, but this paper examines a case where agreement did not ensure correctness. The authors trace the failure in a satellite-catalogue pipeline and conclude that shared definitions left errors of meaning unchecked.

  • Problem

    Redundant computation offers a cheap, mechanical check without ground truth for pipelines whose findings are generated faster than they can be audited.

    Source: §1 · p. 1

  • Method

    The paper analyzes a cross-catalogue pipeline whose gate compares Python counts from source files with SPARQL counts over an emitted RDF graph, then examines its definitions and source-based checks.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Three of seven published counts were wrong because both paths shared constants encoding a misreading of the source vocabulary, so the gate agreed while the error remained.

    Source: §1 · p. 2, §10 · p. 8

  • Takeaways & Limitations

    In this case, redundant computation checked implementation but could not test shared definitions; each shared definition needs a check against the source, though that is not sufficient by itself.

    Source: §10 · p. 8

    The evidence is one incident in one pipeline and one defect family, so the paper makes no claim about how often this structure occurs or fails elsewhere.

    Source: §9 · p. 8

Abstract

from arXiv · show

A common safeguard for a data pipeline is redundant computation: derive each published number by two routes built on different technology and refuse to exit when they disagree. We report one such gate failing, in a cross-catalogue integrity study of two open registers of Earth-orbiting objects. A gate comparing a set-based Python path with SPARQL queries over the emitted RDF graph printed ALL CROSS-CHECKS AGREE on seven counts. Three were wrong, one overstated more than fourfold (932 against 220). Both paths imported the same constants, which encoded a misreading of the source's status vocabulary, so the error was common-mode and the gate could not see it. We give the mechanism, an object-level ledger reconciling every figure, and three checks that go back to the source's documentation, measured on the defective code and on its correction. We then checked that correction against each object's phase history, held in a source file the pipeline never read. The correction was also wrong: 42 of its 261 disagreements are artefacts, and none of our three checks flagged them. Finally, in a controlled replication with three pinned models and tools disabled, 72 of 75 paths generated on request as independent checks computed the defective count, 29 of 30 even when the prompt carried the source's own definitions of the codes. The evidence is one pipeline and one defect family. Within it, redundancy verified implementation, and the errors that reached publication were errors of meaning.

1 Introduction

Redundant computation is appealing because it offers a cheap correctness check without ground truth, but this paper documents one pipeline where agreement concealed errors. The case traces the shared misreading behind the failure and what the authors’ checks did and did not catch.

1 Introduction

  • The study presents one pipeline incident and does not claim to estimate how often redundancy gates fail.Its aim is to show this failure completely: the gate, the wrong and corrected numbers, the source definition, and what caught or missed the error.Source: §1 · p. 1
  • Three of seven published counts were wrong, with the largest overstated more than fourfold, because both paths shared constants encoding the same misreading of status codes.The Python and SPARQL implementations agreed, so the gate could not detect the common-mode error.Source: §1 · p. 2
  • The paper traces the failure mechanism with an object-level ledger, tests three documentation-based checks on defective and corrected code, and examines a later correction against object histories.The authors also test whether a code-generating model produces an independent verification path when asked.Source: §1 · p. 2
  • Redundancy can verify implementation, but paths built on shared definitions cannot test whether those definitions correctly capture meaning.Source: §1 · p. 2

2 Setting

The pipeline compares two independently maintained registers of artificial Earth-orbiting objects without treating either as ground truth. Its verification setup includes graph validation and a gate that recomputes published counts through two routes.

2.1 Two registers of one population

  • Two open registers describe one population, allowing disagreements between them to be treated as findings rather than resolving them against a presumed ground truth.The pipeline retrieved CelesTrak and GCAT on 18 August 2026 and read four GCAT files.Source: §2.1 · p. 2

2.2 The pipeline and its gate

  • Three checks ran on every build; the paper focuses on a gate that recomputed each published count in Python from source files and by SPARQL over the emitted RDF graph.The pipeline reconciles objects by catalogue number and emits typed defect assertions in an RDF graph.Source: §2.2 · p. 2
  • The two counting routes agreed on all seven counts in the published run, despite using different representations and data paths.One path counted Python set members; the other counted SPARQL bindings over a materialized graph.Source: §2.2 · p. 2

3 The failure

The published gate agreed on every compared count, yet three of seven were wrong. The largest error was a disposition disagreement count that changed substantially under later readings of the source data.

3 The failure

  • 932 disposition disagreements became 261 after the first correction and 220 after checking object histories; the initial count treated every status outside GONE as still in orbit.Of 769 mismatches in one direction, 640 involved transition codes rather than a fate; the ledger accounts for objects entering and leaving the count.Source: §3 · p. 3
  • 900 coverage-gap cases became 622 after the pipeline’s missing fourth main catalogue was included; 278 objects had been counted absent despite appearing in that file.Source: §3 · p. 3
  • The undisclosed-tracking-loss count changed from 1,104 to 1,094 because ten counted objects carried the field whose absence the count concerned.The field is maintained and set on 1,292 objects across the catalogue.Source: §3 · p. 3
  • All three errors concerned what source data meant or contained, which the gate could not detect by comparing implementation outputs.Only the disposition error was common-mode in the strict sense.Source: §3 · p. 3

4 Why the gate could not see it

The gate diversified its final computation but shared the definitions that gave the counts their meaning. Its default classification and shared constants made agreement evidence of consistent implementation, not independent interpretation.

4.1 The diversity was in the wrong place

  • Both paths imported identical classification vocabularies and loaders, so they implemented the same reading of the source status codes.The Python counts and SPARQL queries differed in computation, but not in where their meanings came from.Source: §4.1 · pp. 3–4

4.2 The defective definition

  • Unlisted statuses—including E, C, DK, and ATT—were silently treated as meaning an object remained in orbit.The rule counted an object as gone when any status appeared in GONE; every other status fell into the in-orbit default.Source: §4.2 · p. 4
  • Raw-string matching mishandled uncertain dates such as R?, moving 15 objects in or out of the count.This was another defect in the shared module, and comparing outputs could not reveal the mistaken source reading.Source: §4.2 · p. 4

4.3 Relation to multiversion programming

  • Unlike multiversion designs that share a specification, these paths shared executable constants, leaving no opportunity to disagree about a status code.The paper relates this structural sharing to prior work on correlated failures [Knight and Leveson, 1986; Eckhardt and Lee, 1985, Littlewood and Miller, 1989].Source: §4.3 · p. 4

5 What did catch it

Three source-facing checks, rather than comparing two derivations, exposed the defective reading. They examined the source’s documentation, the unexplained disagreements, and whether the source already represented the supposedly missing information.

5 What did catch it

  • Reading GCAT’s field documentation showed that each record describes a flight-history phase and its status names the event ending that phase.This separates an object’s location from the event that ends a particular record.Source: §5 · p. 4
  • Decomposing the 769 unexplained disagreements showed that 640 paired a CelesTrak impact or landing with a GCAT transition code.In 500 cases the code was docking or attachment; Gemini 8 illustrates how both catalogues could correctly describe different events.Source: §5 · pp. 4–5
  • Checking CelesTrak directly for a lost-tracking field found that the field was in use, contrary to the finding’s implicit premise.The check tested the claim about what the source represented, rather than the count derived from it.Source: §5 · p. 5
  • Each check returned to the source’s own account instead of comparing two derivations of a quantity.The results were found on a second pass prompted by thin-feeling results, not by a failing gate.Source: §5 · pp. 4–5

6 Checking the correction against object histories

Checking the first correction against object phase histories reduced the disposition disagreement from 261 to 220, showing that the correction still misclassified cases. A separate reading of GCAT’s derived latest-phase catalogue gave 219, with every difference accounted for.

6 Checking the correction against object histories

  • GCAT defines C as destroying an object only when no subsequent phase follows; a collision survivor starts a new phase.After E, the next phase belongs to a debris fragment, so E marks a boundary rather than establishing the object’s fate.Source: §6 · p. 5
  • 261 disagreements fell to 220 after adding event-catalogue phases; 42 objects were removed and one payload was added.The 42 removals were all 30 explosions and 12 of 14 collisions; 40 had later in-orbit phases, while two explosions had no later phase recorded.Source: §6 · p. 5
  • GCAT’s latest-phase catalogue gave 219 disagreements, with 208 objects common to both readings and all 23 differences assigned named causes.Those causes were 12 reentries, 3 record revisions, and 8 deep-space objects excluded by the pipeline’s design.Source: §6 · p. 5
  • The first correction misread two codes and omitted the history file needed to settle them; none of the three checks detected the 42 artefacts.The histories were held in the event catalogue, a source file the pipeline had not read.Source: §6 · p. 5

7 Three gates, implemented and measured

The three implemented gates exposed different limits: a completeness check could pass a wrong assignment, residue decomposition offered no rejection rule, and inventory reconciliation depended on its file list. A SHACL total that appeared corroborative was only an arithmetic identity.

7 Three gates, implemented and measured

  • The total-function check caught every implicated status on defective code but passed the first correction, which assigned E and C explicitly to the wrong category.Applied to unrelated controlled columns, it also fired on 70,292 orbit-type, 70,130 object-type, and 1,292 data-status rows.Source: §7 · p. 6
  • Residue decomposition made both errors legible but had no rejection criterion, so the reader treated a pattern as confirmation.On the correction, 30 explosions and 14 collisions appeared alongside CelesTrak objects in orbit.Source: §7 · p. 6
  • Inventory reconciliation found the omitted fourth catalogue but missed the event catalogue because its file list was scoped to the main-catalogue index section.Widening the list would also flag files the pipeline had no reason to read; the check corrected a build report from 280 recovered objects to 278.Source: §7 · p. 6
  • The SHACL validation returned 3,564 results, matching the sum of six counts because the shapes selected the same graph nodes those counts counted.The matching total therefore repeated the graph’s errors rather than providing an independent check.Source: §7 · p. 6

8 Asking a model for an independent check

The authors tested whether code-generating models would produce an independent check whose meaning came from somewhere other than the shown path. In a controlled replication, most generated paths reproduced the defective count, including when given the source’s definitions.

  • The experiment asked whether models would derive a check’s meaning independently, although reusing a definition is ordinarily correct practice.The authors note that generated test oracles can encode the behavior of the code they see rather than its intended behavior [Konstantinou et al., 2024].Source: §8 · p. 6
  • The preliminary 30-trial run could not establish whether outputs had read files, and its prompt’s paraphrase returned 208 rather than the defective 932.The model aliases were not resolved to identifiers, and the loader bodies were elided.Source: §8 · p. 7
  • The controlled replication pinned three models, disabled tools, and tested generated paths on a synthetic catalogue and frozen real files.The synthetic probe encoded status–state pairings as distinct powers of two so the counts revealed which pairings each path included.Source: §8 · p. 7
  • 72 of 75 generated paths returned the defective 932; only one would have exposed the defect by listing unclassified status codes.Two other paths refused to produce a count, and one failed in its hand-written parser.Source: §8 · p. 7
  • 29 of 30 paths given GCAT’s definitions returned 932 and counted the status codes as the shown path did.One path correctly described E and C but still grouped them with docking codes as the defective path did.Source: §8 · p. 7
  • On this task, requesting independence, forbidding code sharing, and supplying source definitions still yielded paths that computed the shown path’s result.The authors conclude that independence must be specified through inputs and verified by computed behavior, not merely requested; they make no claim about models generally.Source: §8 · pp. 7–8

9 Limitations

The evidence is bounded to one pipeline, one defect family, and a small model-generation experiment. The reference count and correction also depend on catalogue records with unresolved timing and interpretation limits.

  • The study covers one incident and one defect family, so it does not estimate how often redundancy structures occur or fail elsewhere.The three gates were measured on one codebase against two errors; the second error was a single prospective test, not an evaluation.Source: §9 · p. 8
  • The reference count of 220 is not an oracle: it counts objects with incompatible catalogue claims, and later records could revise it.The event catalogue arrived six weeks later, and corrections to older phases in that window could not be separated; another GCAT reading gave 219, while treating two explosions as destroyed gives 222.Source: §9 · p. 8
  • The generation experiment tested one task with three models from one family and five trials per cell, without retrieval of source records.A second model family was dropped, and the event catalogue that resolves E and C was not supplied.Source: §9 · p. 8

10 Conclusion

The case shows how two paths sharing a misread definition can agree on wrong counts, while source checks alone may still miss meaning errors. The object-level ledger and phase-history comparison track how the disposition count changed across readings.

10 Conclusion

  • The gate agreed on seven published counts, three wrong, because both paths shared constants encoding the same misreading of the status vocabulary.The authors conclude that diversity below a shared definition cannot detect an error in that definition; redundant computation verified implementation, not meaning.Source: §10 · p. 8

A Object-level ledger of the disposition count

  • The ledger follows objects present in both main catalogues, keyed by catalogue number, through all three versions of the count.It distinguishes 998 objects removed from the comparison from disagreements, and the 500 docking and attachment objects are a subset of 769.Source: A Object-level ledger of the… · p. 9, Table 4 · p. 10

B The full-history check

  • The full-history check uses each object’s latest eligible phase, applying GCAT’s event-ending definitions and resolving docking or transfer through the destination object.It excludes 19 phases begun after 18 August 2026, sets aside 28 objects whose latest phase leaves the present state unrecorded, and requires both records to agree for three duplicated catalogue numbers.Source: B The full-history check · p. 9
  • The derived-catalogue comparison gives 219 across 69,249 comparable objects, with 208 in common with the 220-object phase-history reading.The 42 objects removed from the first correction are listed in the derived catalogue as 40 in orbit and two exploded; CelesTrak gives none a decay date.Source: B The full-history check · p. 10

C Generation experiment: protocol and per-trial results

The appendix describes the controlled generation experiment and its earlier, flawed preliminary run. It also records the replication setup, trial outcomes, and released artifacts.

C Generation experiment: protocol and per-trial results

  • The behavioral probe encoded 14 status-and-decay cells plus two controls as distinct object counts, then ran generated code on the frozen real files.It called eligible zero-argument functions and recorded the count returned or printed by the selected function.Source: C Generation experiment: protocol and… · p. 10
  • In the preliminary run, 22 of 30 outputs returned the shown path’s 208, while the prompt never exposed the 932 defect it was scored against.When code sharing was forbidden, seven outputs diverged from 208, including one that returned 795 after treating a phase end date as a decay date.Source: C Generation experiment: protocol and… · pp. 10–11
  • The replication pinned three model identifiers, disabled tools, used an empty working directory, and appended GCAT definitions in two conditions.The CLI’s fixed setup and prompts, raw outputs, and per-trial metadata are recorded in the replication artifacts.Source: C Generation experiment: protocol and… · p. 11
  • 72 of 75 paths had the shown path’s signature on the probe, counting all six codes as in orbit when CelesTrak recorded a decay.Exceptions included a failing parser, a path using an unclassified-status rule, and a path that stopped on statuses it had not classified.Source: C Generation experiment: protocol and… · p. 11

D Artifacts

  • The repository includes code, gate implementations, prompts, raw model outputs, per-trial metadata, and the behavioral probe.The artifact inventory records retrieval dates and SHA-256 prefixes for the source catalogue files.Source: D Artifacts · p. 11

Related papers

Loading 2609.37603v1…