Source-linked AI summary
Tracing Generated Samples to Training-Data Clusters in Flow-Matching Models
Rania Briq, Ohad Fried, Michael Kamp, Stefan Kesselheim
TL;DR
The paper asks how training samples influence generated images in flow-matching models when local velocity perturbations do not directly determine final counterfactual effects. It combines the analytical closed-form velocity solution with learned trajectories, evaluates cluster-level scores using LOCO models across latent spaces, and finds that semantic similarity is strong while trajectory-based attribution is competitive and attribution depends on representation and trajectory dynamics.
Problem
Training-sample influence in flow matching is difficult to assess because endpoint effects depend on how velocity-field changes propagate along the generation trajectory, not only on local perturbations.
Method
The paper derives cluster-level trajectory-based attribution by combining the analytical flow-matching closed-form solution with trained trajectories and evaluates it against LOCO counterfactual models and baseline methods.
Results
Semantic similarity is a strong baseline, while closed-form trajectory-based attribution is competitive in some metrics and attribution varies with latent representation, counterfactual metric, and trajectory dynamics.
Takeaways & Limitations
Attribution in flow matching is not explained by endpoint similarity or local velocity perturbations alone; representation and whole-trajectory dynamics are also relevant.
Takeaways & Limitations
The experiments are limited to image-domain large-group deletion, and the closed-form solution applies only to rectified flow with linear flow paths.
Abstract
from arXiv · showhide
Understanding which training samples influence a generated image is an important problem in generative modeling. In flow matching, training samples influence the generated image through the velocity field along the generation trajectory. Removing samples to examine their counterfactual influence changes the velocity field, and the resulting effect on the final image depends on how the change propagates through the trajectory. Consequently, local changes in the velocity field do not necessarily predict the final counterfactual effect. This work investigates attribution in flow-matching models through a hybrid analytical--learned approach, and uses it to derive trajectory-based attribution scores at the cluster level. We evaluate these attribution scores using independently retrained leave-one-cluster-out (LOCO) models, and compare with several attribution baselines using two different flow-matching latent spaces. Our experiments show that semantic similarity constitutes a strong baseline, while the closed-form trajectory-based attribution is competitive in some metrics without requiring counterfactual retraining or model gradients. Our results show that attribution in flow matching depends not only on semantic similarity to training samples, but also on the latent representation, trajectory dynamics, and how influence is propagated to the final output.
1 Introduction
The paper studies cluster-level training-data attribution in flow-matching models, where local velocity-field changes may not predict final counterfactual effects because perturbations propagate along trajectories. It combines analytical and learned flow information, evaluates attribution with LOCO counterfactual models, and compares existing baselines.
- Motivation: Flow-matching attribution must account for how velocity-field perturbations propagate along the entire generation trajectory, since local changes can cancel before the endpoint.This makes endpoint effects computationally difficult to infer from local velocity changes alone.
- Attribution setting: The study partitions training data into clusters and uses leave-one-cluster-out models as counterfactual oracles for evaluating cluster-level attribution.The setup reflects evidence that removing an entire semantic cluster can change generated identity more distinctly than removing individual samples.
- Related work: The paper situates its contribution among influence-function and gradient-based attribution methods adapted to generative models.These related approaches estimate training-sample effects through parameter influence or gradients rather than the proposed trajectory-based formulation.
- Approach: The proposed approach combines the flow-matching closed-form velocity solution with the trained model trajectory to derive trajectory-based attribution scores.The closed-form velocity is expressed as a combination of training-sample contributions, providing a basis for attribution.
- Evaluation: Experiments compare the proposed attribution with gradient-based methods and semantic similarity using two flow-matching latent representations.The evaluation is designed to test attribution across different latent geometries and against established baselines.
2 Methodology
The method combines rectified-flow dynamics with a closed-form velocity representation to attribute generated samples to training-data clusters along the full generation trajectory. It evaluates these scores against leave-one-cluster-out counterfactual models, whose endpoint deviations provide a pseudo-ground-truth.
- Flow Matching Preliminaries: Rectified flow uses linear interpolants between Gaussian source samples and data samples, with generation obtained by integrating a learned velocity field.The path is x_t = (1 − t)x_0 + tx_1, and the target velocity is x_1 − x_0.
- Closed-Form Optimal Velocity: The optimal velocity is a conditional expectation, and its finite-dataset closed form expresses the field as weighted contributions from individual training samples.Each positive coefficient λ_i(x,t) represents a sample’s influence at a location and timestep.
- Trajectory-Based Cluster Attribution: The approach evaluates closed-form sample contributions on the model trajectory and aggregates posterior mass from samples belonging to each cluster.Cluster attribution is based on the integral of a cluster’s contribution over the whole trajectory.
- Trajectory-Based Cluster Attribution: Closed-form cluster mass predicts the most influential cluster as the one with the largest accumulated cluster contribution.The prediction is k̂ = argmax_k A^mass_k.
- Velocity-Perturbation Attribution: A second closed-form variant measures the velocity-field perturbation caused by deleting each cluster, renormalizes the remaining weights, and integrates the perturbation along the trajectory.This perturbation-based score performs nearly equally to posterior mass but requires more computation.
- Counterfactual Evaluation: LOCO models are trained separately without each cluster, and the cluster producing the largest endpoint RMS deviation is treated as the pseudo-ground-truth.The evaluation averages endpoint deviations over five counterfactual-training seeds; in the illustrated example, cluster C9 produces the largest deviation.
3 Experiments
The experiments compare closed-form and baseline attribution methods against cluster-deletion counterfactuals across flow-matching latent spaces. Evaluation uses cluster scores, counterfactual effects, Top-1 agreement, and ranking correlation.
- Evaluation metrics: The study evaluates cluster attribution using counterfactual effect, Top-1 agreement, and Spearman rank correlation.The oracle is defined from leave-one-cluster-out models, and each query uses the same initial noise as the full model.
- Attribution methods: Methods include closed-form attribution, endpoint semantic similarity, gradient similarity, and a random-cluster baseline.Semantic similarity uses nearest centroids or members in DINOv2 and CLIP spaces.
- RAE-model evaluation: DINOv2-centroid similarity achieves the highest counterfactual RMS and Top-1 agreement in the RAE evaluation.Closed-form group mass has slightly better Spearman correlation, while nearest-member similarity performs worse than nearest-centroid.
- RAE-model evaluation: None of the evaluated methods recover the oracle counterfactual better than about 50% in the RAE evaluation.This indicates that the methods leave additional attribution information unexplained.
- Latent-space comparison: Attribution performance drops in the VAE latent space, where distances are less suitable for measuring similarity than in semantically structured DINOv2 latents.The VAE-based flow-matching evaluation uses N = 128.
4 CLIP-defined clusters
The CLIP-defined cluster experiment tests whether DINOv2’s performance depends on alignment between the attribution representation, clustering space, and model latent space. DINOv2-based similarity remains stronger than CLIP-based similarity, while overall agreement declines relative to DINOv2-defined clusters.
- CLIP-defined clusters: DINOv2-based similarity still outperforms CLIP-based similarity when clusters are defined in CLIP embedding space.The comparison uses K = 12 CLIP-defined clusters and retrained leave-one-cluster-out models.
- Comparison with DINOv2-defined clusters: All methods achieve lower Top-1 and Spearman correlation for CLIP-defined clusters than for DINOv2-defined clusters.The result suggests that DINOv2’s structured organization helps predict the counterfactual effect.
- Interpretation: DINOv2’s advantage is not explained solely by alignment among the attribution, clustering, and model latent spaces.The VAE experiment supports the same broader dependence on latent-space geometry.
- Interpretation: DINOv2 captures geometric and lower-level features that appear important for image similarity, unlike CLIP’s stronger text-alignment emphasis.This provides a proposed explanation for DINOv2’s stronger organization of samples for attribution.
5 Discussion
The discussion presents closed-form attribution as a strong, retraining-free signal while emphasizing that attribution depends on latent representation, training stochasticity, and the definition of influence. The study remains limited to large group deletion in image models and to rectified flow with linear paths.
- Contributions: The work introduces cluster-level attribution for flow-matching models using LOCO models and a hybrid closed-form plus trajectory-based approach.It evaluates the approach against gradient-based and semantic-similarity baselines.
- Main findings: The closed-form method provides a strong attribution signal without retraining or fine-tuning, enabling adaptation to dynamically defined clusters.It performs only slightly worse than static DINOv2-based similarity methods.
- Main findings: Attribution depends strongly on latent representation: RAE achieves considerably better results, while VAE distances and similarities are less meaningful.The discussion attributes this difference to the VAE’s reconstruction-oriented objective.
- Main findings: Different training seeds produce different rankings, so disagreement is not necessarily a failure of the attribution method.The counterfactual is affected by both training samples and training stochasticity.
- Limitations: The experiments are limited to image-domain large-group deletion, and the closed-form solution applies only to rectified flow with linear flow paths.Other modalities, finer-grained attribution, and alternative flow paths remain open questions.