PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignment

Zhen Zhang, Ahmad Hafez, Amr Alanwar

arXiv:2608.17713v1cs.LG

TL;DR

Cross-view correspondence in agent evaluation and credit assignment is a measurement intervention, yet its downstream consequences are not identified when optimal choices are hidden. The paper develops two-sided validation and all-optima analysis, finding that hidden exact completions can alter diagnoses, signed credit, and rankings under fixed objectives.

  • Problem

    Cross-view evaluation pipelines treat correspondence as neutral even though multiple optimal matchings can support opposite downstream scientific conclusions.

  • Method

    The paper validates cross-view transport, propagates all exact matching choices to conclusions, and reports uncertainty or abstention when optima disagree.

  • Results

    Hidden exact completions altered diagnoses, signed credit, and a public ranking under fixed primary objectives.

  • Takeaways & Limitations

    One returned optimum does not justify one conclusion; correspondence contracts should expose legal optima and report a point update, common direction, or abstention.

  • Takeaways & Limitations

    The audits cover one agent family, checkpoint, and compiler family, establish neither final-policy degradation nor universal affectedness, and retain a Weak downstream consequence.

Abstract

from arXiv · show

Agent evaluations and trace-based learning often compare outputs across transformed views through a post-response correspondence treated as neutral preprocessing. We show that this correspondence is a measurement intervention: omitting it can manufacture sensitivity, an over-aggressive map can manufacture invariance, and multiple optimal correspondences can leave mechanism labels and signed learning credit unidentified. We develop a validity theory and audit with three components: two-sided validation of nuisance removal and response preservation, all-optima identification of downstream conclusions, and uncertainty propagation after validity is established. We characterize the linear feasibility boundary for response-preserving nuisance removal, compute sharp ranges over exact-optimum correspondence sets, and give a distribution-free certificate that retains a credit coordinate only when all exact optima agree on its nonzero sign. Across public code and SQL pipelines, two deterministic optimal tracebacks disagree on temporal localization for 55.9% of 1,586 nonzero trajectory pairs; two frozen 800-rollout tool-use audits, including a task-and-seed-disjoint replication, expose exact-optimum reversals of intended turn-level credit, although a clean public quick-start subset shows none. A pre-registered transport gate failed on natural responses; frozen corrected and held-out controls then show that a map calibrated only on benign examples erases every retained harmful response, while two-sided validation selects response-preserving alternatives. Cross-view correspondence must therefore be declared, validated, and propagated into uncertainty before agent evaluation or credit assignment supports a point conclusion.

1 Introduction

Cross-view correspondence is a measurement intervention rather than neutral preprocessing, so scientific conclusions must account for transport validity and every legal exact optimum. A point conclusion is warranted only under a declared, validated correspondence contract.

  • Core problem: An optimal correspondence can preserve the same scalar score while assigning opposite local credit, leaving the downstream conclusion unidentified.A reference tool call tied with two predicted calls, and either matching preserved total score while crediting a different turn.
  • Core problem: Correspondence spans cross-view transport, within-output matching, and exact-tie completion across transformed prompts, interfaces, schemas, repositories, or observations.The paper treats this full interface as a measurement intervention, not neutral preprocessing.
  • Failure modes: Omitting transport can manufacture sensitivity, benign-only calibration can manufacture invariance, and multiple exact optima can reverse mechanism labels or signed learning credit.These are the three failure directions identified for correspondence validity.
  • Correspondence contract: The correspondence contract combines two-sided transport validation, all-optima identification, and uncertainty propagation before returning a point, common direction, or abstention.Transport must remove nuisance without erasing response, while all exact-optimal correspondences must support the same downstream conclusion.
  • Contributions and scope: The paper contributes legal-update geometry, a compiler-dependent credit frontier, natural audits of diagnosis and credit policies, and a declared-contract requirement for point conclusions.The scope is not that every pipeline fails, but that point conclusions require a declared and validated correspondence contract.

2 Correspondence Contracts and Two-Sided Validity

Correspondence must be treated as a declared measurement contract: validate the transport for both nuisance removal and response preservation, then propagate any exact matching ties into the conclusion. The theory shows that benign-only calibration cannot certify invariance, while linear nuisance removal with retained response is feasible exactly when the nuisance and response subspaces intersect trivially.

  • Correspondence contracts: A transport must be validated before downstream matching choices, because an undeclared correspondence convention leaves response claims unidentified even with more samples.The population response is a sharp set over all declared transports, and point claims require a declared map or agreement across that set.
  • Correspondence contracts: A correspondence contract declares the matching objective, optimizer-set policy, and downstream readout, with deterministic, stochastic, or all-optima-invariant consumption as distinct targets.Soft matching changes the objective and therefore defines a different contract rather than identifying the original exact-optimum claim.
  • Two-sided validity: Small null leakage prevents false sensitivity but not false invariance; two-sided certification therefore requires low null leakage, positive response gain, and preservation of every required behavioral witness.The certificate is relative to declared null controls, response-bearing controls, thresholds, and witnesses, not a claim of global semantic equivalence.
  • Two-sided validity: Benign-only calibration cannot certify invariance, because an unrestricted transport can leave all benign controls unchanged while collapsing an untested response to its baseline.This construction preserves benign error while reducing response gain to zero and failing q-faithfulness.
  • Two-sided validity: N ∩ R = {0} is the exact feasibility condition for a linear transport that removes nuisance while retaining positive response gain, with orthogonal-projection gain sin θmin(R, N).A shared nonzero response–nuisance direction must be annihilated, whereas projection onto N ⊥ remains injective on R.

3 Solver-Independent Credit: Geometry and Complexity

The section makes solver-independent credit a property of the full update body induced by every exact correspondence, then characterizes when common progress, coordinate retention, and normalization are well-defined. It shows that compiler composition—not matching alone—determines auditability, with affine and low-treewidth cases tractable but shared standardization reaching hardness.

  • Geometry of solver-independent credit: Every exact correspondence is compiled into an update body Kθ; the update is solver-identified iff Kθ is a singleton, while 0 ∈ Kθ rules out any strict common-progress direction.When 0 ∉ Kθ, the closest point to the origin supplies a certified common direction with margin equal to its norm.
  • Geometry of solver-independent credit: Coordinate j is retained only when every exact optimum has the same nonzero sign, a pointwise-maximal rule that need not itself be a legal full-gradient update.The rule distinguishes robust directional credit from selecting one solver completion.
  • Relative-credit normalization: Translation neutrality plus current-peer isolation forces a differentiable reward compiler to be constant; scale neutrality also prevents a nonconstant compiler from having a uniformly Lipschitz extension to zero dispersion.The trilemma exposes trade-offs among peer dependence, stability near ties, and when correspondence uncertainty is consumed.
  • Complexity by compiler: Affine compilers admit polynomial-size exact formulations, while local nonlinear compilers are tractable through min-sum elimination when the compiler interaction graph has bounded treewidth.The relevant graph belongs to the composed reward pipeline, not to the matching problems considered separately.
  • Complexity by compiler: Shared group standardization is an Ising compiler: inverse-polynomial-additive lower-support certification is NP-hard and universal all-optima threshold certification is coNP-hard.The hardness arises from composing tie choices with normalization, even though one exact matching and its update remain polynomial to compute.

4 Natural Consequences and Exact Repairs

Exact-optimum correspondences create substantial ambiguity in temporal diagnoses and turn-level credit, while source metrics can also misclassify responses. Two-sided validation and all-optima analysis expose these failures and identify conservative repairs that preserve only conclusions stable across valid alternatives.

  • Temporal diagnosis: 55.9% of 1,586 non-null trajectory pairs receive different temporal labels under deterministic left- versus right-priority optimal tracebacks.Mean sharp-range width is .466 for Codex versus .094 for the OpenHands control, and 47.5% are alignment-unidentified under the preregistered rule.
  • Temporal diagnosis: 32 exact Hungarian optima for one released trajectory produce Order Consistency ranging over [0, 1] and Step Coherence over [.43, 1], showing structural mechanism ambiguity.Repeated names and canonicalized arguments create cosine-one permutations, so the spread is not attributable to model, judge, or sampling noise.
  • Turn-level credit: 18.8% of multi-call trajectories have material exact-optimum reward width, and the intended multi-turn advantage reverses sign in 14/20 affected task groups.The all-optima certificate retains 76.0% of canonical nonzero coordinates, while the task-and-seed-disjoint replication confirms the reversal pattern.
  • Turn-level credit: Earlier-call priority shifts credit by .196 [.149, .246] and .174 [.133, .219], while a uniform law flips 20.1% of canonical nonzero coordinates in expectation.Reversing only the selector’s call order preserves the objective but changes learning credit; random input permutations reduce mean positional bias to about −.001.
  • Strict trajectory score: 14.23% of natural task-configuration cells receive source score one despite non-exact traces, with the defect appearing in 22/26 configurations and 12/13 model families.The two-sided repair s2 = ℓ/max{m, n} equals one if and only if complete name sequences agree, and it creates nine pairwise ranking reversals.
  • Transport validity: The natural-response transport gate failed: transported distance .019 was equivalent at τ, so benign reconstruction alone could not establish response preservation.Later controlled and frozen checks diagnose the failure rather than retroactively passing the gate.

5 Related Work

The paper builds on established alignment, matching, process-conformance, optimization, and partial-identification tools, but applies them to validated cross-view maps and sharp downstream diagnoses. It distinguishes this approach from probabilistic matching and tests of already-declared statistics or updates.

  • Correspondence and alignment: Established unknown-map identification, extremal alignment, robust matching, and process-conformance methods provide tools for analyzing all optimal correspondences.The paper redirects these tools toward sharp downstream diagnosis or compiled updates after validating the cross-view map.
  • Correspondence and alignment: Probabilistic or soft matching specifies a law or modified objective, whereas this paper studies exact correspondences and downstream conclusions after validation.The cited passage also notes known tie-breaking impossibilities in this related literature.
  • Optimization and learning tools: Classical convex, MAX-CUT, elimination, and GRPO-gradient tools are composed through a source-faithful interface for the paper’s analysis.Related work also establishes non-singleton lower levels and normalization pathologies.
  • Identification before testing: Unlike perturbation and metamorphic tests of a declared statistic, the paper first tests whether correspondence identifies that statistic or update using partial-identification machinery.This reframes validation around identification before assessing the resulting statistic or update.

6 Limitations and Conclusion

The paper concludes that one returned correspondence optimum cannot support a unique evaluation or credit conclusion: contracts must expose all legal optima and report agreement, direction, or abstention. Its claims remain bounded by post hoc, synthetic, and provider-control limitations, with no final-policy guarantee inferred.

  • Limitations: The natural gradient audit is post hoc and uses one 2,560-parameter tensor, while the worst-case construction is synthetic and finite-source executions are conformance checks.The paper establishes neither final-policy degradation nor universal affectedness, and provider outputs lack immutable digests and controllable seeds.
  • Conclusion: One returned optimum does not justify one conclusion; a correspondence contract records the objective and exact-optimum set, then reports a point update, common-progress direction, or abstention over legal optima.Hidden choices can change diagnosis and signed credit under a fixed objective, while exact repair can move a public ranking.
  • Claim boundaries: 44.6% and 47.5% are preserved as separate frozen-endpoint and source-state results, respectively, with both reversing the same two aggregate cells.The frozen sweep spans 43.2–49.4%; the reconstructed quick-start is reported separately as 6/20, 1/9, and 0/7.
  • Claim boundaries: Nine pairwise model comparisons and the top-ranked public-benchmark configuration survive all 59 leave-one-task-out omissions, but no final-policy effect is claimed.Section C explicitly excludes unmeasured training and policy effects.
  • Conclusion: The claimed technical novelty is correspondence-induced composition: a compiler-dependent frontier and finite-bit construction in Theorem 2, plus natural audits where hidden exact completions alter diagnoses and signed credit.The appendix distinguishes this contribution from standard partial-identification, principal-angle, and classical multi-objective-optimization machinery.

A Proofs

The proofs show that response identification is relative to the admissible transport class: differing admissible transports yield a sharp identified set rather than a point conclusion. They also formalize correspondence contracts, distinguishing selector-, law-, and solver-invariant targets over exact optima.

  • Identification relative to transport: The sharp population identified set is the set of expected responses over all admissible transports, and a singleton class point-identifies the response.If two admissible transports yield different expectations, the observation law alone cannot point-identify the response.
  • Identification relative to transport: For compact transport classes and continuous functionals, the identified-set extrema are attained, supporting sharp interval-hull reporting for threshold decisions.The identified set itself need not be an interval when the transport class is nonconvex.
  • Correspondence-contract semantics: With an exact optimizer set, a declared selector targets h(s(A⋆Q)), while a declared law targets EA∼π[h(A)].These contracts define policy-indexed targets rather than solver-independent properties of the objective solution.
  • Correspondence-contract semantics: A solver-invariant scientific claim is the set {h(A) : A ∈ A⋆Q}, sharp when every exact optimum is admissible and no selector or law narrows the contract.Replacing Q with a soft objective changes the estimand rather than selecting within the original hard-optimum set.

A.2 Two-sided correspondence validation.

Two-sided correspondence validation requires testing both nuisance removal and response preservation: benign-only calibration can erase responses while raw comparison can conflate nuisance with response. Linear nuisance-killing transports preserve response with positive gain exactly when nuisance and response subspaces intersect only at zero, while sign-agreement diagnostics retain coordinates only when all exact optima agree.

  • A.2 Two-sided correspondence validation: Benign-only transport calibration does not certify response retention or witness faithfulness.A transport faithful for an L-Lipschitz behavioral witness controls witness change through transported distance, but null leakage alone provides no positive response guarantee.
  • A.2 Two-sided correspondence validation: A linear nuisance-killing transport with positive response gain exists exactly when the nuisance and response subspaces intersect only at zero.Orthogonal projection achieves gain equal to sin θ_min(R, N), the sine of their smallest principal angle.
  • A.2 Two-sided correspondence validation: Two-sided validation admits a transport only when it separately passes nuisance-leakage and response-retention tests.Raw comparison conflates nuisance and response, while null-control-only calibration can collapse harmful and benign states.
  • A.3 Coordinatewise sign agreement as a diagnostic: The coordinatewise sign-agreement rule retains a canonical credit coordinate only when every declared exact optimum preserves its sign and nonzero status.The rule is a diagnostic sign certificate, not a probability law over optima or an automatically valid full-gradient update.

A.4 Relative-credit normalization tradeoffs and constructive bounds.

Relative-credit normalization faces an unavoidable tradeoff: translation neutrality with current-peer isolation forces constancy, while exact scale neutrality creates inverse-dispersion instability near zero. Damping controls amplification but sacrifices exact scale neutrality and does not certify sign preservation.

  • Normalization tradeoffs: Translation-neutral, current-peer-isolated normalization is constant on a connected domain of nonconstant reward groups.Differentiating translation neutrality makes the Jacobian annihilate the common direction; peer isolation then forces every diagonal entry to vanish.
  • Normalization tradeoffs: Exact scale-neutral normalization has Jacobian scaling DN(ay) = a^-1DN(y), causing instability as dispersion approaches zero.A nonconstant C1 map is therefore not uniformly Lipschitz near zero dispersion and cannot have a continuous nonconstant extension there.
  • Normalization tradeoffs: These inverse-dispersion and peer-influence tradeoffs apply at the class level rather than only to the standard-deviation formula.The section distinguishes structural properties of normalization classes from behavior specific to one formula.
  • Normalization tradeoffs: Damping preserves translation neutrality and globally caps Jacobian amplification at 1/τ, but gives up exact scale neutrality near ties and does not certify a sign.An independently estimated origin and scale instead gives 1/σ-Lipschitz behavior and zero current-peer derivatives, while changing within-group origin and scale properties.

A.7 Compiler tractability frontier.

The section shows that group standardization can compile correspondence ambiguity into an Ising optimization whose exact optima preserve scalar reward while changing the downstream objective. It also identifies tractable affine and bounded-treewidth cases, alongside hardness of support approximation and universal all-optima certification.

  • Ising compilation: Each ambiguous assignment fiber has exactly two exact optima that preserve the same scalar trajectory reward, enabling compilation of graph structure into an Ising objective.The construction uses polynomial-size literal tool-call similarity instances and equal trajectory totals, with only two deterministic anchors carrying the queried policy-score derivative.
  • Hardness frontier: Adjacent cut sizes stay Ω(m−14) apart in support, while inverse-polynomial-additive lower-support approximation is NP-hard and universal all-optima threshold certification is coNP-hard.The finite-bit promise formulation preserves a polynomially representable separation for the hardness reduction.
  • Tractable formulations: For affine compiled updates, the exact correspondence uncertainty is the image g(P1 × · · · × Pk) of product exact-optimum faces, with directional support given by an LP and closest-point computation by a convex QP or epigraph SOCP.Assignment integrality makes the vertices of each Pi exactly the integral exact optima; the contribution is treating this correspondence face as the uncertainty source consumed by the reward compiler.
  • Tractable formulations: For local nonlinear factors on a tree decomposition of width w, variable elimination runs in time poly(n)q^(w+1) and memory poly(n)q^w, with backpointers recovering an exact minimizing correspondence.Replacing min by max computes upper support, so bounded treewidth provides a tractable support-oracle method.
  • Ising compilation: The compiled objective’s minimizers are exactly maximum cuts because the cut and uncut variance terms satisfy αcut < αuncut.The construction yields JG = (αuncut − αcut)/2 > 0, while the global branch vanishes because all trajectory totals are equal.

B Additional Experimental Evidence and Robustness

Additional audits show that exact-optimum correspondences can make temporal and structural agent metrics non-identified across domains, while disclosed convention changes materially affect reported ambiguity. The evidence therefore supports reporting sharp ranges and distinguishing metric-identification failures from demonstrated model-ranking reversals.

  • B.1 Sharp temporal localization over all optimal alignments: 47.5% ambiguity is reported under the corrected source-state convention, versus 44.6% under the endpoint convention.The preregistered threshold sweep gives 43.2%, 47.5%, and 49.4% ambiguity at cutoffs (.35, .45), (.40, .50), and (.45, .55), with the same two aggregate reversals.
  • B.2 Second natural audit: global matching (M3-Bench): All 260 task/assignment combinations in the M3-Bench audit match the pinned official evaluator, but 14 of 213 released rows contain cross-step exact ties and non-point structural scores.For Task 00130004, 32 certified optima yield Order Consistency [0, 1], Step Coherence [.43, 1], and Merge Purity [.31, 1].
  • B.2 Second natural audit: global matching (M3-Bench): The M3-Bench finding is a metric-identification failure, not a demonstrated model-ranking reversal, because multiplicity resides in the fixed reference and prediction outputs are unavailable.The audit’s exact sharp-bounding methods apply to additive statistics on Levenshtein optimal-path DAGs and linear functionals over Hungarian optimal faces, not universally to nonlinear statistics.

B.3 Natural learning audit: exact-optimum MatchTIR credit.

The natural learning audit finds exact-optimum MatchTIR correspondences can reverse intended advantage signs while preserving the matching objective, so solver-invariant credit requires an all-optima sign mask. The result passes preregistered criteria in discovery and held-out samples, remains stable to small slack and hostile checks, but is bounded by public-configuration and source-coefficient limitations.

  • Natural learning audit: 93 of 495 eligible rollouts are material, and 14 of 20 material four-rollout groups admit an intended advantage-sign reversal despite exact-optimal matching.The released greedy objective is never suboptimal; both original and reversed completions attain the exact matching score, with total rewards differing by at most 8.9 × 10−16.
  • Near-optimal sensitivity: At relative slack ρ = .05, material counts rise from 93 to 107 and from 90 to 116, while the median slack needed to create a .10 coordinate range is .20 in both samples.No new material trajectory appears through ρ = .01 in either sample, supporting stability to small solver tolerances.
  • Hostile verification: After removing tool errors and requiring distinct predicted-call JSONs, 24/394 = 6.1% remain material; among nine clean all-distinct material rollouts, 5/7 affected groups still reverse sign.The mask retains 150/159 = 94.3% canonical updates in that clean subset, indicating duplicates amplify but do not create the result.
  • Public-configuration boundary: The public quick-start reconstruction yields 6/20 sign-flip groups over all material drivers, 1/9 for all-distinct drivers, and 0/7 after additionally removing tool errors.The audit makes no claim about an undocumented author override, final-policy degradation, or reported benchmark scores.
  • Validation criteria: Discovery and held-out samples pass all frozen criteria, with 13/20 and 14/22 material groups showing intended advantage-sign disagreement, respectively.The promotion rule required exact-objective preservation, a positive early-minus-late contrast, at least 65% negative slopes, and disagreement in at least 20% of affected groups.

B.10 Two-sided transport-admissibility gates.

Two-sided transport-admissibility gates distinguish response-preserving transports from maps that manufacture invariance. Frozen controls reject destructive transport, while natural-response testing exposes deletion and motivates conservative abstention or call-site repair.

  • Transport construction: Across four leave-one-family-out folds, the corrected call-site transport produces zero false invariance while preserving wrapper-side preprocessing, postprocessing, exceptions, and return flow.This transport substitutes the local implementation body at its actual call site.
  • External validation: All seven external-matrix conditions pass for the corrected transport: benign views map to zero, harmful mutations preserve failure signatures and positive distance, and destructive transport erases all eight.The matrix was frozen before observing requests or pytest outcomes.
  • Frozen protocol: The audit freezes implementation, candidate order, targets, tests, four mutation families, and seven conditions before observing requests or pytest outcomes.A separately frozen extension tests local alpha-renaming and branch inversion.
  • Natural-response validation: The natural-response gate fails because response deletion occurs in all 18 xarray states, despite occurring in 0/18 pylint states.The preregistered 75% and both-substrate conditions therefore fail.
  • Gate decisions: A conservative exact-delegation guard accepts 2/2 benign bases, rejects 8/8 harmful wrappers, and abstains on all 36 evolved wrappers.The call-site transport covers all 36 evolved wrappers while preserving their witnesses.

C Extended Limitations and Scope … D.4 Reporting rule.

The paper limits its claims to declared validation contracts, audited populations, and tested computational regimes, while requiring solver-independent reporting that preserves exact-optimum uncertainty. Its audits are tractable, but broader semantic, learning, complexity, and cross-model conclusions remain unestablished.

  • C Extended Limitations and Scope: Two-sided certification is contract-relative: passing declared controls, witnesses, and transport-family checks does not prove global semantic equivalence.The linear theorem gives a sharp feasibility boundary but does not independently validate nonlinear source transformations, which require frozen tests and witnesses.
  • C Extended Limitations and Scope: The temporal-alignment result conditions on nonzero-edit released code and SQL pairs, while MatchTIR fractions condition on groups with material exact-optimum reward width.Discovery, task-and-seed-disjoint, and post-hoc natural update-body analyses are reported separately rather than as unrestricted deployment or preregistered training estimates.
  • C Extended Limitations and Scope: MatchTIR identifies turn rewards and intended advantage signs compatible with all exact-optimal correspondences, not demonstrated final-performance effects.The public quick-start does not visibly enable the intended multi-turn mask, published traces do not expose every assignment, and later updates or trajectory-level credit may offset local reversals.
  • C Extended Limitations and Scope: The prospective natural-response gate failed, the corrected-witness replication followed diagnosis, and only the later held-out validation tested the frozen transport-selection rule.These evidence levels are not pooled; the held-out result validates deterministic transformations across four repositories, not new natural trajectories or a second agent family.
  • C Extended Limitations and Scope: The complexity theorem separates compiler regimes without establishing natural-case or fixed-precision hardness, and the audited low-treewidth interaction graphs are exactly solvable rather than ordinary hard MAX-CUT instances.The shared-standardization reduction is a worst-case finite-bit construction, while executable conformance checks detect algebraic and source-realizability errors.
  • C Extended Limitations and Scope: Cross-model and cross-training-algorithm generality remains untested because provider outputs are not bitwise reproducible and the audits use one agent family, checkpoint, and compiler family.Most reanalyses are deterministic from released or frozen samples.
  • D Computational Cost of Solver-Independent Auditing: For additive diagnostics, exact-optimum envelopes are computed with two-pass dynamic programs on shortest-path DAGs or two min/max-cost matchings on integral assignment faces.The same procedures decide whether a point diagnosis is solver-determined; both audited pipelines fall within this tractable region.
  • D.2 Polynomial envelopes for the audited backends.: Report the exact envelope when a polynomial backend applies; otherwise report a randomized estimate with sampling error or abstain instead of hiding a tie-break.Near-optimal additive auditing can be NP-hard, while exact fair averaging may be intractable even when sharp min/max guarding is polynomial.
Loading 2608.17713v1…