PaperlayerView exact revision on arXiv

Source-linked AI summary

Witness Overlap: Directional Provenance Inside Open-Weight Model Families

Siyuan Li, Haoxuan Zeng, Xin Luo, Fernando Jia, Florence Li, Zhengyang Geng, Zico Kolter, Tai Sing Lee, Tianqin Li

arXiv:2609.31784v2cs.AI

TL;DR

Symmetric pairwise evidence can establish checkpoint relatedness but cannot determine which checkpoint came first. WITNESS OVERLAP uses a third same-family checkpoint to compare anchor-conditioned update geometry. Its one-witness test achieves strong directional-provenance results across LLM, VLM, and diffusion evaluations.

  • Problem

    Existing provenance methods often test relatedness from a known base, while symmetric pairwise scores cannot orient parent–child relationships within a family.

    Source: §1 · p. 1

  • Method

    WITNESS OVERLAP compares update-direction overlap from each candidate endpoint to the other checkpoint and a third same-family witness using prompt-free weight-space scores.

    Source: §1 · p. 2

  • Results

    One-witness parent–child orientation exceeds 93% on LLMs and reaches 100% on VLMs and 90% on diffusion models, alongside strong root, sibling, and chain-ordering results.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The evaluated signal transfers across LLM, VLM, and diffusion architectures when each family has a shared aligned parameter subspace.

    Source: §6 · p. 10

    The validated setting is mostly direct base-to-descendant releases; complex lineage reconstruction and adversarial tampering remain outside it.

    Source: §6 · p. 10

Abstract

from arXiv · show

Open-weight models are often released, fine-tuned, aligned, merged, and re-released, making provenance audits ask not only whether checkpoints are related, but also which checkpoint came first. Many existing model-provenance methods are designed for a base-known audit setting: given a victim or source model, they test whether a suspect model is related to it. Although these audits are framed as source-to-suspect tests, their underlying evidence is often symmetric, relying on representation similarity, weight similarity, behavioral fingerprints, or correlation statistics. Symmetric pairwise comparisons can detect relatedness, but they cannot by themselves orient relationship between checkpoints A and B. We therefore introduce a local geometric comparison: instead of comparing two checkpoints directly, we add a third same-family checkpoint as a witness and compare the geometry around each candidate endpoint. Direction is inferred by asking which candidate behaves more like a branching parent. Motivated by this idea, and by the empirically observed asymmetry between parent-anchored and child-anchored witness-overlap distributions, we propose Witness Overlap, a prompt-free, training-free white-box test for directional provenance. On 176 LLM checkpoints from 16 families, our one-witness test orients 95.3\% of parent-child decisions using Frobenius cosine. We further evaluate root identification, sibling discrimination, generalizations to VLM and diffusion families, and chain-structured ordering. The signal is robust to weight noise and sparse pruning, with a proposed SVD weight reduction variant showing greater robustness than Frobenius cosine.

1 Introduction

Existing provenance methods can test whether checkpoints are related, but symmetric pairwise evidence cannot determine which came first. WITNESS OVERLAP uses a third same-family checkpoint to compare anchor-conditioned update geometry and infer direction.

1 Introduction

  • Symmetric pairwise provenance scores can detect relatedness but cannot orient a parent–child edge, a limitation as audits increasingly face many same-family checkpoints.Existing methods compare weights, representations, activations, outputs, prompts, likelihoods, or generated-text statistics.Source: §1 · p. 1
  • Parent-anchored updates to distinct descendants tend to have low overlap, while child-anchored updates toward a parent and sibling tend to be more aligned.The lower-overlap endpoint is predicted as the parent.Source: §1 · p. 2
  • WITNESS OVERLAP compares update directions from an anchor to a candidate and a third same-family witness, then compares scores anchored at both endpoints.The proposed prompt-free, training-free scores are Frobenius cosine and top-k SVD subspace overlap.Source: §1 · p. 2
  • The method frames four family-internal tasks: root identification, parent–child orientation, sibling discrimination, and ordering checkpoints along a single fine-tuning chain.It evaluates these tasks across LLM, VLM, and diffusion checkpoints.Source: §1 · p. 2
  • On 176 LLM checkpoints from 16 families, one-witness WITNESS OVERLAP exceeds 93% parent–child orientation accuracy and reaches 100% LLM root accuracy.It also supports sibling discrimination, single-chain ordering, and evaluations on VLM and diffusion families.Source: §1 · p. 2

2 Related Work

Related work separates detecting whether checkpoints belong to a family from recovering directional lineage within a known family. WITNESS OVERLAP targets the latter with anchor-conditioned update overlap rather than global scalar checkpoint scores.

2 Related Work

  • Provenance work spans two stages: family detection constructs same-family candidate sets, while lineage recovery asks for directional relationships within those sets.The second stage assumes the family set is already known.Source: §2 · p. 2
  • Family detection uses metadata and supply-chain records as well as model-derived white-box and black-box evidence, but external records may be incomplete or inconsistent.Model-derived signals include representations, weights, gradients, activations, outputs, prompts, likelihoods, and generated-text traces.Source: §2 · pp. 2–3
  • Relatedness evidence alone does not determine directional relationships among checkpoints already known to belong to one family.Nikolic et al. [2025] reports 99.32% precision and 91.25% recall for relatedness screening in the study's LLM models.Source: §2 · p. 3
  • Prior lineage methods predict candidate parents, orient parent–child pairs, or recover model trees using weight-derived signals and learned or learning-free detectors.Examples include Neural Lineage, Neural Phylogeny, and MoTHer.Source: §2 · p. 3
  • WITNESS OVERLAP targets training-free lineage recovery inside an identified family by using a third checkpoint and anchor-conditioned overlap between weight deltas.This contrasts with assigning each checkpoint a global scalar score.Source: §2 · p. 4

3 Method: Witness Overlap

WITNESS OVERLAP treats directional provenance as a comparison of update geometry around candidate endpoints, using a third same-family checkpoint as a witness. Its scores and task-specific aggregation rules turn anchor-conditioned overlap into root, edge, pair-type, and chain decisions.

3.1 Setting and notation

  • The setting assumes a known same-family checkpoint set and defines root identification, parent–child orientation, pair-type discrimination, and single-chain ordering as its tasks.The chain task assumes a known base and successive SFT checkpoints.Source: §3.1 · p. 4
  • The method aggregates updates over dense projection-role blocks, excluding embeddings, language-model heads, and normalization weights in LLM experiments.VLM and diffusion checks use analogous architecture-specific projection blocks.Source: §3.1 · p. 4

3.2 Per-tensor anchor-conditioned statistics

  • Frobenius cosine measures alignment of two outgoing deltas in full tensor space; SVD overlap compares dominant update subspaces using average squared cosine of principal angles.The SVD variant uses rank-k truncated SVD, with default k = 16, and aggregates dominant directions to smooth tensor-level noise.Source: §3.2 · p. 4

3.3 Weight aggregation and task decision rules

  • Per-tensor scores are averaged within parameter blocks and then across blocks; the resulting triplet score measures similarity between two deltas from the same anchor.The two scores are anchor-asymmetric: swapping the anchor need not preserve overlap.Source: §3.2 · p. 4, §3.3 · p. 5
  • Root identification selects the candidate with the lowest mean witness-pair overlap, while parent–child orientation predicts the lower-overlap endpoint as parent.The parent anchor points toward distinct descendants; child-anchored deltas share a reverse-update component.Source: §3.3 · p. 5
  • Pair-type discrimination uses a low–high endpoint-score pattern for parent–child pairs and a high–high pattern for sibling pairs, with a validation-calibrated threshold.The smaller endpoint score is the primary pair-type statistic.Source: §3.3 · p. 5
  • For a known-base fine-tuning chain, the method averages witness overlaps for each SFT checkpoint and orders checkpoints by increasing chain score.Later checkpoints are described as sharing reverse-update components with more earlier SFT witnesses.Source: §3.3 · p. 5

3.4 Algorithm and properties

  • The procedure iterates over ordered anchor–target–witness triplets, computes a selected per-tensor statistic, and aggregates the scores using the task rules.An update-vector explanation for chain ordering is provided separately.Source: §3.4 · p. 5

4 Proposition: When Parent Anchors Have Lower Witness Scores

This section gives a Frobenius-cosine explanation for why parent anchors are expected to score below child anchors, and reports that observed descendant-update alignments typically fall in the theory’s relevant low-alignment regime.

4 Proposition: When Parent Anchors Have Lower Witness Scores

  • The analysis flattens trainable-weight tensors and defines ρ as descendant-update alignment and λ as their relative update norm.Source: §4 · p. 6
  • For nonzero, non-collinear descendant updates, Proposition 4.1 gives the parent-anchored Frobenius witness score as their update alignment ρ.Source: §4 · p. 6
  • Observed descendant-update alignments ρ are typically small, placing real checkpoints in the regime where Proposition 4.1 expects child anchors to score higher.Source: §4 · p. 6

5 Experiments

Experiments compare two weight-space overlap statistics with directional baselines and test family-internal provenance tasks, perturbations, merges, and quantization.

5.1 Setup

  • The evaluation covers 176 LLM checkpoints across 16 families, plus four VLM families and five Diffusion families; direct-edge labels follow public HuggingFace lineage metadata.Directional baselines include Neural Phylogeny’s parameter-norm rule and MoTHer’s kurtosis direction component and directed tree recovery.Source: §5.1 · p. 6

5.2 Main result

  • SVD identifies 16/16 LLM roots, 4/4 VLM roots, and 4/5 Diffusion roots; Frobenius cosine reaches 95.3% LLM parent–child direction accuracy.The root-identification results exceed endpoint-only baselines, while both metrics outperform those baselines on LLM parent–child orientation.Source: §5.2 · p. 7
  • On LLMs, cosine reaches 0.898 AUROC and 82.4% balanced accuracy for sibling discrimination; all five chain scores rise in order, alongside higher child-anchor overlap.Pair typing uses a leave-one-family-out logistic regressor, while the chain ordering follows continued fine-tuning from sft1 through sft3.Source: §5.2 · p. 8

5.3 Nw-witness ablation

  • Pooling witnesses improves LLM direction accuracy: SVD rises from 94.1% at Nw=1 to 100.0% at Nw=all, while cosine rises from 95.6% to 99.4%.The same trend holds for VLM and Diffusion families when all eligible witnesses are averaged.Source: §5.3 · p. 9

5.4 Failure case analysis

  • With one witness, SVD misorients 100/1542 LLM triplets and cosine misorients 73/1542; cosine failures are most frequent in SmolLM2-135M at 34/182.The analysis links failures to high child-witness alignment and a large witness-to-child scale.Source: §5.4 · p. 9

5.5 Noise perturbation and pruning robustness tests

  • Direction accuracy does not decrease under tested child-side Gaussian noise; child-only pruning stays stable through 50% sparsity, while severe all-descendant pruning degrades performance.In the all-descendant pruning setting, SVD is more robust than Frobenius cosine at moderate sparsity.Source: §5.5 · p. 9

5.6 Same-family model merges

  • With one witness, root–merge direction accuracy is 96.88%–97.03% for TIES and 95.70%–96.56% for SLERP; the test does not identify merge contributors.With all witnesses, both metrics reach 100% for TIES; SVD also reaches 100% for SLERP.Source: §5.6 · p. 9

5.7 Quantization robustness test

  • When all checkpoints in three complete LLM families are quantized, the method identifies 3/3 roots and orients 37/37 edges with all witnesses.When only queried children are quantized, both metrics reach 794/800 (99.25%) with one witness and 65/65 with all witnesses; these results assume a common dense parameterization.Source: §5.7 · p. 10

6 Discussion and Limitations

The method transfers across evaluated LLM, VLM, and diffusion families, while its reliability depends on a same-family witness and shared aligned parameter space. The discussion also bounds its validated scope and reports Stage-1 screening counts.

6 Discussion and Limitations

  • One-witness parent–child direction reaches 95.3% on LLMs, 100% on VLMs, and 90% on diffusion models; root identification reaches up to 100%, 100%, and 80%, respectively.It recovers all five tested single-chain orders; pair-type discrimination requires a calibrated threshold, and the method assumes each family has a shared aligned parameter subspace.Source: §6 · p. 10
  • Weakly aligned updates make the sufficient condition in Proposition 4.1 easier to satisfy, consistent with the observed signal.Source: §6 · p. 10
  • The method requires an approximately correct same-family set and at least one additional same-family witness, and should abstain when none exists.One injected false positive preserved 22/22 root predictions and 220/220 aggregated directions, but this does not establish tolerance to arbitrary Stage-1 errors.Source: §6 · p. 10
  • Under the evaluated INT8 rule, root predictions and all-witness directions were retained; same-family TIES and SLERP root-to-merge direction also remained stable.Cross-family merges violate the same-family premise, so the authors make no directional claim for them.Source: §6 · p. 10
  • The signal can weaken when a child update is much smaller than a witness update or when descendant updates are highly aligned.Pair-type discrimination is less robust than root identification and direction, with best balanced accuracy below 90%; complex lineage reconstruction and adversarial tampering remain outside the validated setting.Source: §6 · p. 10

A Evidence for Stage-1 Screening

  • Table 4 records TP=146, FN=14, FP=1, and TN=2,399 across 160 true root-derived pairs and 2,400 unrelated controls.The evaluation applies black-box outputs and multiple hypothesis testing to the 176-checkpoint LLM subset.Source: Table 4 · p. 13, A Evidence for Stage-1 Screening · p. 13

B Stage-1 False-Positive Stress Test

The stress test injects architecture-compatible false positives between the Llama-3-8B and Mistral-7B families. The method preserves root identification and aggregated edge orientation in these tested contaminated pools.

B Stage-1 False-Positive Stress Test

C Derivations for Witness Asymmetry

The derivations formalize how anchor-conditioned witness overlap can reveal direction under specified update-geometry conditions. The appendix also gives the decision procedure and examines chain ordering, per-family geometry, and witness averaging.

C.1 Frobenius-cosine witness asymmetry

  • For non-collinear updates, the Frobenius-cosine asymmetry condition reduces to ρ < λ/2; the collinear case is degenerate for this comparison.The parent-anchor score is ρ, and the derivation compares it with the child-anchor score.Source: §C.1 · p. 14

C.2 SVD subspace extension

  • If every paired right-subspace direction satisfies ρt < min{1, σi,t/(2σj,t)}, child-anchor SVD overlap exceeds parent-anchor overlap.Here ρt is the principal cosine between right subspaces, under the shared-left-subspace SVD model.Source: §C.2 · p. 14

D Algorithm

  • The algorithm averages witness overlaps, predicts the root by minimum root score, orients each pair toward the lower-overlap anchor, and uses a threshold for pair type.Its inputs include a same-family set, shared 2D tensor set, statistic choice, and an SVD rank when applicable.Source: D Algorithm · p. 16

E Single-Chain Ordering Details

  • In controlled fine-tuning chains, anchor geometry changes from low/low at sft1 to high/low at sft2 and high/high at sft3.The checkpoints are formed by sequential supervised fine-tuning from the base on MMLU, UltraChat, and PubMedQA.Source: E Single-Chain Ordering Details · p. 16

F Per-Family Single-Witness Geometry

G VLM and Diffusion Witness Averaging

  • SVD overlap shows a slight improvement as more witnesses are averaged, while orientation remains strong beyond the single-witness setting.The figure describes the cross-domain VLM and diffusion witness-averaging ablation; the reported SVD trend suggests pooling can reduce noise in dominant update subspaces.Source: G VLM and Diffusion Witness Averaging · p. 17, Figure 4 · p. 17

H Same-Family Model Merging

The merge evaluation tests whether same-family merges of two descendants remain on the descendant side of their shared root. Both TIES and SLERP are tested, with one-witness and all-witness decisions.

H Same-Family Model Merging

  • The evaluation merges two descendants sharing a root using TIES or SLERP, with 0.50/0.50 balanced and 0.25/0.75 asymmetric nominal weights.Root identification uses four-checkpoint candidate sets; direction tests place each merge relative to its root using one or all eligible witnesses.Source: H Same-Family Model Merging · p. 18, Table 7 · p. 18
  • With one witness, root-to-merge direction accuracy is 96.88%–97.03% for TIES and 95.70%–96.56% for SLERP; all-witness results reach 100% for both TIES metrics and SLERP with SVD.SLERP with Frobenius cosine reaches 97.50% for balanced merges and 98.13% for asymmetric merges using all witnesses.Source: H Same-Family Model Merging · p. 18, Table 7 · p. 18
  • The experiment tests root-relative placement only; it does not identify merge contributors or orient individual contributor-to-merge edges.Source: H Same-Family Model Merging · p. 18

I Quantization Robustness Test

INT8 quantization preserves or improves witness-overlap direction decisions in the tested settings. The section also reports parent–child anchor asymmetry and characterizes single-witness failures.

I Quantization Robustness Test

  • With all checkpoints quantized, one-witness accuracy is 423/436 (97.0%) for cosine and 406/436 (93.1%) for SVD, while all 37 all-witness decisions and three root predictions are preserved.The checkpoints were recovered into a common dense parameterization before computing weight differences.Source: I Quantization Robustness Test · p. 18, Table 8 · p. 18
  • Child-only INT8 causes no clean-correct direction decision to become incorrect and corrects 42 cosine and 63 SVD decisions that were previously wrong.The authors attribute the pattern in this setting to a shared quantization residual when the child is the anchor, which strengthens overlap asymmetry.Source: I Quantization Robustness Test · p. 19

J Anchor-Role Asymmetry

  • Parent-anchor overlap is usually smaller than child-anchor overlap across the aggregated LLM, VLM, and Diffusion evaluations.Table 10 aggregates per-pair averages over all in-family witnesses for each anchor choice.Source: Table 10 · p. 19

K Single-Witness Failure Cases

  • The single-witness failure table identifies 73 misoriented LLM triplets and flags the top 10% of cosine triplets by ρ and η using thresholds 0.0247 and 45.16.Here i is the evaluated child update, j the witness update, ρ = C(p; i, j), and η = ∥uj∥/∥ui∥ = 1/λ.Source: K Single-Witness Failure Cases · p. 19, Table 11 · pp. 19–20

L Pruning Stress Test

Pruning robustness depends on which descendants are sparsified: child-only pruning is stable in the tested GPT-2 setting, while pruning all descendants is harder. Child-side Gaussian noise does not reduce accuracy in the tested setting.

L Pruning Stress Test

  • At 50% child-only pruning on GPT-2, SVD and Frobenius cosine both achieve 80.0%, 97.5%, 100.0%, and 100.0% direction accuracy for Nw = 1, 2, 3, all.The stress test zeros the smallest fraction of entries by absolute value while preserving tensor names and shapes.Source: L Pruning Stress Test · p. 20, Table 12 · p. 21
  • At 10% all-descendant pruning, SVD direction accuracy is 77.8%, 76.1%, 74.3%, and 70.0%, versus 53.3%, 52.5%, 49.5%, and 50.0% for Frobenius cosine across Nw = 1, 2, 3, all.Pruning all descendants degrades both scores at high sparsity; averaging more witnesses improves accuracy in milder settings.Source: L Pruning Stress Test · p. 21, Table 14 · p. 22

M Child Gaussian Noise Stress Test

  • Child-side Gaussian perturbations do not reduce Frobenius-cosine direction accuracy in the tested two-family setting.Larger perturbations tend to make direction easier because the noisy child-anchor deltas share a strong noise component, while the parent-anchor comparison pairs the noisy child update with a clean witness update.Source: M Child Gaussian Noise Stress Test · p. 22, Table 15 · p. 22

N Per-Family Composition and Model Registry

The evaluation uses public checkpoints for local academic evaluation, with LLM families spanning multiple model sizes and post-training types and separate VLM and Diffusion checks. The checkpoint sets include diverse fine-tuning objectives.

N Per-Family Composition and Model Registry

  • The listed repositories are used solely for local academic evaluation, and the authors state that they do not redistribute or otherwise provide checkpoint files.The LLM registry identifies repositories by family and checkpoint type, including base, SFT, and RLHF/alignment.Source: N Per-Family Composition and Model… · pp. 23, 25, Table 16 · p. 23
  • The LLM evaluation includes families ranging from GPT2 at 124M parameters to Meta-Llama-3-8B and Qwen3-8B, with one base and 5–14 post-training children per family, totaling 130 SFT and 30 RLHF descendants.Table 17 reports the per-family composition and parameter scale.Source: N Per-Family Composition and Model… · p. 25, Table 17 · p. 25
  • The separate cross-domain check set covers four VLM and five Diffusion families, each with one base and four fine-tuned descendants, for 36 descendants total.These checkpoints are kept separate from the LLM headline counts.Source: N Per-Family Composition and Model… · p. 26, Table 19 · p. 26
  • The checkpoint set spans varied LLM SFT and alignment objectives, VLM tasks including OCR and robotics-oriented fine-tuning, and Diffusion tasks including inpainting and style adaptation.The listed objectives also include code generation, translation, reasoning, DPO, GRPO, and PPO.Source: N Per-Family Composition and Model… · p. 26

Related papers

Loading 2609.31784v2…