PaperlayerView exact revision on arXiv

Source-linked AI summary

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Zelong Xu, Yan Li, Wenhe Hu, Xiyang Hu

arXiv:2609.34363v1cs.CVcs.AIcs.MM

TL;DR

Omni-modal models can struggle to connect sounds with the visual observations occurring at the same moment, and answer-level supervision leaves local pairing implicit. SyncRA uses input timing to align intermediate audio–visual representations during QA fine-tuning; it improves correspondence-sensitive answering and mean accuracy across evaluated model–benchmark combinations.

  • Problem

    Local audio–visual pairing can remain weakly represented and implicit in answer-level supervision, while a spoken cue paired with the wrong scene can still yield a plausible answer.

    Source: §1 · p. 1

  • Method

    SyncRA jointly trains QA and an auxiliary objective that aligns simultaneous intermediate audio–visual states using timing-derived pairs, without additional annotations or inference-time components.

    Source: §1 · p. 3

  • Results

    SyncRA improves mean accuracy on all 20 model–benchmark combinations and raises Qwen3-Omni swap-test AllFour accuracy from 34.00% to 62.50% over answer-only fine-tuning.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Controlled comparisons show that aggregate QA scores can conceal whether answers follow the current pairing, and local temporal correspondence is directly supervisable.

    Source: §8 · p. 10

    The swap test is multiple-choice over 100 source videos; whether models track pairing changes in open-ended answers remains untested.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.

1 INTRODUCTION

Omni-modal models can recognize audio and visual content yet fail to track which observations coincide in time. SyncRA makes local audio–visual correspondence an explicit training target and improves results across evaluated models and benchmarks.

1 INTRODUCTION

  • Weak local correspondence can link a spoken cue to the wrong scene while producing a fluent, plausible answer that is difficult for users to detect.Both modalities may be individually correct even when their temporal pairing is not.Source: §1 · p. 1
  • Controlled swaps show that current models often answer as if the original audio–visual pairing remained in place, even after answer-only fine-tuning.A model must follow the pairing in the input, rather than either modality alone or global video semantics, to answer all versions correctly.Source: §1 · p. 2
  • The paper reports that local pairing is weakly represented and under-supervised: temporal information remains decodable, but direct matching of simultaneous audio and visual states is weak.Answer-level supervision can leave local pairing implicit when a question is answerable from global semantics or one modality.Source: §1 · p. 2
  • SyncRA aligns simultaneous intermediate audio and visual states and separates mismatched ones using timing already present in the input, without additional annotations.It jointly optimizes this representation-level target with ordinary answer supervision.Source: §1 · pp. 2–3
  • SyncRA raises mean accuracy on all 20 model–benchmark combinations and improves swap-test AllFour accuracy over answer-only fine-tuning on all four backbones.On Qwen3-Omni, AllFour accuracy rises from 34.00% to 62.50%.Source: §1 · p. 3

2 METHODOLOGY

SyncRA adds an auxiliary representation objective to ordinary QA fine-tuning, using processor timing to form audio–visual pairs within shared temporal intervals. It contrasts matching and mismatching intervals within each video while leaving inference unchanged.

2 METHODOLOGY

  • SyncRA adds local audio–visual correspondence supervision to intermediate backbone states during ordinary QA fine-tuning.The added projection head is training-only and discarded after training.Source: §2 · p. 3, §2.3 · p. 4

2.1 TEMPORAL INTERFACE AND INTERVAL POOLING

  • At a selected block, SyncRA mean-pools audio and visual token states assigned by native timing metadata to each shared interval.The valid intervals are processor sampling units, not annotated semantic events; a same-interval pair is positive because the observations co-occur in the input.Source: §2.1 · p. 3

2.2 WITHIN-VIDEO SYNCHRONY CONTRAST

  • A shared projection and symmetric contrastive objective align corresponding interval states and distinguish them from other intervals in the same video.Keeping candidates within one video fixes source identity and global context, so the objective discriminates among times within a recording.Source: Figure 2 · p. 4, §2.2 · p. 4

2.3 JOINT TRAINING WITH ANSWER SUPERVISION

  • The auxiliary objective is added to final-answer token cross-entropy; samples with fewer than two valid intervals receive answer supervision alone.The projection head adds 64d training parameters, is discarded after training, and adds no inference-time component.Source: §2.3 · p. 4

3 EXPERIMENTAL SETUP

Experiments use a shared, balanced 10K-question training corpus and compare four dense and mixture-of-experts backbones under matched fine-tuning conditions. Evaluation covers five video benchmarks and a controlled correspondence test.

3.1 DATA

  • All fine-tuning uses the same balanced 10K-question subset of OmniVideo-100K, with shared data splits and media inputs across methods.The subset contains 10,000 questions from 1,782 videos across ten task categories.Source: §3.1 · p. 5

3.2 BACKBONES AND BASELINES

  • The study evaluates two dense backbones and two mixture-of-experts backbones, comparing Base, answer-only Vanilla SFT, Text CoT-SFT, and SyncRA.Within each backbone, update scope is held fixed across training objectives.Source: §3.2 · p. 5

3.3 TRAINING CONFIGURATION

  • Fine-tuned methods share an optimization recipe and select checkpoints using validation answer-token negative log-likelihood rather than benchmark scores.The shared recipe uses AdamW, learning rate 10^-5, effective batch size 12, and two epochs.Source: §3.3 · p. 5

3.4 EVALUATION

  • Evaluation reports accuracy on five public video benchmarks and their equally weighted Avg-5, using matched questions, media, and denominators within each backbone.Table 1 reports AllFour accuracy on the same 100 source videos and defines Δ as SyncRA minus Vanilla SFT in percentage points.Source: §3.4 · p. 5, Table 1 · p. 6

4 A CONTROLLED SWAP TEST FOR TEMPORAL PAIRING

The controlled swap test checks whether answers follow the current audio–visual pairing rather than a modality alone or the clip’s global semantics. It tests the same spoken cues across four versions of each source video.

4 A CONTROLLED SWAP TEST FOR TEMPORAL PAIRING

  • The test uses 100 source videos, each contributing four unique spoken cues in nonoverlapping windows with a stable visible scene during each utterance.Independent ASR and human inspection establish the annotations before inference.Source: §4 · p. 6
  • Each cue is tested on original, audio-exchanged, video-exchanged, and jointly exchanged inputs, with windows 1 ↔ 3 and 2 ↔ 4 swapped in designated streams.A single-stream exchange changes the correct scene, while exchanging both streams preserves the pairing at a new position.Source: §4 · p. 6
  • AllFour credits a cue only when all four versions are answered correctly; overall accuracy averages the 1,600 question–version evaluations per checkpoint.Paired 95% bootstrap intervals resample source videos with their cues and versions.Source: §4 · p. 6

5 RESULTS

SyncRA improves correspondence-sensitive answering across four backbones and raises mean accuracy across all evaluated model–benchmark combinations. Its gains are largest or second-largest on benchmarks that most directly require audio and vision together.

5.1 CORRESPONDENCE IMPROVES ACROSS ARCHITECTURES

  • SyncRA raises AllFour accuracy over Vanilla SFT on all four backbones, with gains from 5.00 to 28.50 percentage points.All four paired 95% source-video bootstrap intervals exclude zero.Source: §5.1 · p. 6
  • For every backbone, the largest version-specific improvement occurs under a single-stream exchange, where the correct answer must follow a changed audio–visual pairing.Table 1 reports AllFour accuracy on the same diagnostic of 100 source videos.Source: §5.1 · p. 6

5.2 SYNCRA IMPROVES GENERAL VIDEO QA ON EVERY MODEL–BENCHMARK COMBINATION

  • SyncRA achieves higher mean accuracy than Vanilla SFT in all 20 model–benchmark combinations.The comparison holds under full understanding fine-tuning for dense models and partial parameter updates for MoE models.Source: §5.2 · p. 7, Table 2 · p. 7
  • On every backbone, the largest or second-largest gain falls on Daily-Omni or AVUT-Human, which most directly require audio and vision together.Table 2 reports benchmark accuracy and equally weighted Avg-5 across five benchmarks.Source: §5.2 · p. 7, Table 2 · p. 7
  • Text CoT-SFT trails Vanilla SFT on Avg-5 for every backbone, while SyncRA improves over both.On Qwen3-Omni, Text CoT-SFT trails Vanilla SFT on all five benchmarks.Source: §5.2 · p. 7

6 ABLATION STUDY AND ANALYSIS

The analysis finds that SyncRA’s gains depend on contrasting local temporal pairs and that it strengthens same-time correspondence in the backbone’s own representations. This correspondence rises at the supervised depth and remains visible in the final block.

6.1 LOCAL CONTRASTIVE SUPERVISION DRIVES THE GAINS

  • Direct local pairing beats the strongest alternative temporal target, Fixed time codes, by 17.75 percentage points on AllFour.The alternatives’ Daily-Omni accuracies are close to SyncRA’s, while their AllFour scores diverge.Source: §6.1 · p. 8
  • All tested λ values improve Qwen3-Omni AllFour over Vanilla SFT by at least 24.00 points, with λ = 0.1 scoring highest.The tested values are λ ∈ {0.03, 0.1, 0.3}.Source: §6.1 · p. 8
  • InfoNCE reaches 62.50% AllFour accuracy versus 46.25% for pairwise ranking, despite similar Daily-Omni scores.The comparison keeps co-occurring positive pairs and the training setup fixed.Source: §6.1 · p. 8

6.2 SYNCRA ALIGNS THE BACKBONE’S OWN STATES IN TIME

  • After SyncRA training, interval-pooled states in Qwen3-Omni and Qwen2.5-Omni show a more concentrated same-time diagonal.Figure 3 shows cosine similarity between unprojected interval-mean audio states (rows) and video states (columns) on a fixed validation clip, using a shared 0.4–1 scale.Source: §6.2 · p. 8, Figure 3 · p. 8
  • Same-interval retrieval rises from 13.39% unprojected R@1 to 96.91% with SyncRA on 100 Qwen3-Omni validation videos.With a two-interval audio delay, SyncRA retrieves the current interval at 96.45% R@1 but the content origin at 0.01%.Source: §6.2 · p. 8
  • SyncRA’s correspondence gain peaks at the supervised block and remains visible in the final block, unlike the decline at the final block for Base and Vanilla SFT.Figure 4 shows the depth profile for Qwen3-Omni; supervision at block 24 or 36 improves AllFour over Vanilla SFT, while location shapes where correspondence concentrates.Source: §6.2 · p. 9, Figure 4 · p. 9

7 RELATED WORK

SyncRA draws on audio–visual synchrony and temporal contrastive learning, but applies local temporal contrast within an answering model. Its analyses examine temporal organization in the model’s own states and whether answers track changed pairings.

7 RELATED WORK

  • Prior audio–visual synchrony work uses same-time frame–audio pairs as positives and pairs from other videos or shifted times as negatives; CAV-MAE Sync explicitly refines frame–audio alignment (Araujo et al., 2025).SyncRA contrasts same-input-time pairs inside an answering model and measures temporal organization in its own states.Source: §7 · p. 9
  • LatentOmni combines temporal InfoNCE, sensory feature supervision, and text prediction, whereas SyncRA contrasts intermediate states during QA fine-tuning without latent reasoning tokens at inference (Dai et al., 2026).SyncRA tests whether answers track changes in the current pairing.Source: §7 · p. 9
  • TimeLens studies timestamp representations for continuous video temporal grounding, while SyncRA’s diagnostic asks for scene selection during a spoken cue.REPA and iREPA align diffusion-model states to pretrained visual features; SyncRA instead uses co-occurring input intervals to jointly train the answering model’s audio and visual states.Source: §7 · p. 9

8 CONCLUSION

The conclusion frames local temporal correspondence as a distinct capability that can be supervised directly. It reports that SyncRA improves correspondence-sensitive answering across dense and MoE backbones and mean accuracy across evaluated model–benchmark combinations.

8 CONCLUSION

  • SyncRA improves correspondence-sensitive answering across dense and MoE backbones and raises mean accuracy on every evaluated model–benchmark combination.The method makes local audio–visual correspondence an explicit target during QA post-training.Source: §8 · p. 9
  • Similar aggregate QA scores can conceal large differences in whether answers follow the current pairing.The conclusion characterizes local temporal correspondence as a distinct, directly supervisable capability.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

The appendices document the study’s data, optimization, temporal processing, layer selection, auxiliary objectives, and baseline procedures, alongside four stated limitations. Fine-tuning uses a fixed 10K-question corpus, and SyncRA obtains interval correspondences from native timing information.

REPRODUCIBILITY STATEMENT

  • The appendices document data splits, optimization, model configurations, diagnostic construction, scoring, uncertainty, and representation analyses.They also report per-run benchmark counts.Source: REPRODUCIBILITY STATEMENT · p. 11

A LIMITATIONS

  • The swap test covers 100 source videos with multiple-choice questions; whether pairing changes are tracked in open-ended answers remains untested.The study also uses one compact 10K-question fine-tuning corpus and does not measure how gains vary with data scale.Source: A LIMITATIONS · p. 14

B.1 DATA AND ANSWER SUPERVISION

  • The fixed, balanced OmniVideo-100K subset contains 10,000 audio–video question–answer examples from 1,782 source videos, including 5,000 multiple-choice and 5,000 open-ended questions.Source: §B.1 · p. 14

B.2 OPTIMIZATION AND TRAINABLE PARAMETERS

  • SyncRA and the comparison objectives share the media, questions, answers, example order, and native chat template; the auxiliary projection is removed for inference.Training checkpoints are selected by validation answer-token NLL, and evaluation uses the model’s native answer-generation path.Source: §B.1 · p. 14, §B.2 · p. 14

B.3 NATIVE TEMPORAL LAYOUTS AND MEDIA PROCESSING

  • Audio and visual tokens are assigned to shared temporal intervals using timestamps, interval boundaries, or temporal patch metadata, rather than sequence position.Intervals are processor sampling units, and Qwen3 intervals are approximately one second at 2 FPS, though frame-budget resampling can change duration on longer inputs.Source: §B.3 · pp. 14, 16

B.4 LAYER SELECTION ON FROZEN BASE MODELS

  • Frozen-base probes select the eligible candidate layer with the highest validation correspondence score; selected depths vary across backbones.The strongest probe occurs at three-quarters depth for both Qwen models, one-quarter for MiniCPM-o, and one-half for Nemotron.Source: §B.4 · p. 17, Table 10 · p. 17

B.5 AUXILIARY LOSS IMPLEMENTATION

  • SyncRA adds a symmetric interval contrastive loss to answer training, using same-interval positives; samples with fewer than two valid intervals use answer supervision alone.Controls include deranged within-video pairings and fixed timestamp codes that do not directly compare audio and visual feature vectors.Source: §B.5 · pp. 17–18

B.6 TEXT COT-SFT BASELINE

  • Text CoT-SFT trains on supplied reasoning followed by the final answer, sharing Vanilla SFT’s split and optimization settings and selecting epoch 2 for all four backbones.Validation NLL is computed over the complete reasoning-and-answer target on all 500 validation examples.Source: §B.6 · p. 18

B.7 BENCHMARK INPUTS, GENERATION, AND STATISTICAL SUMMARIES

SyncRA has positive mean-accuracy differences over Vanilla SFT across all 20 evaluated model–benchmark combinations. The appendices specify benchmark scoring, controlled swap construction, and comparisons among alignment objectives.

B.7 BENCHMARK INPUTS, GENERATION, AND STATISTICAL SUMMARIES

  • Benchmark accuracy uses a fixed effective denominator, retaining generation failures, parsing failures, and incorrect answers; responses are parsed with method-specific rules.Evaluation uses greedy generation with a maximum of 1,280 new tokens.Source: §B.7 · p. 19
  • 20 positive SyncRA-minus-Vanilla benchmark mean differences form a consistent pattern across the evaluated settings.The evaluation spans five benchmarks and four backbones, including two dense and two MoE models.Source: §B.7 · p. 19

B.8 ALIGNMENT-LOSS OBJECTIVES AND RATIONALE

  • InfoNCE uses temporal co-occurrence as the positive signal and within-video alternatives as candidates, training interval discrimination rather than positive-pair attraction alone.Its jointly normalized candidate scores distinguish it from pairwise ranking, which compares positive and negative scores independently.Source: §B.8 · p. 20
  • 62.50% AllFour accuracy for InfoNCE compares with 40.50% for cosine-only alignment and 46.25% for pairwise ranking.The SyncRA-minus-control differences are 22.00 and 16.25 percentage points, respectively; Daily-Omni accuracies differ by at most 0.75 percentage points among the three objectives.Source: §B.8 · p. 20

C.1 SOURCE MATERIAL AND ANNOTATION

  • The controlled diagnostic uses four spoken cues from each of 100 independent source videos, with human-checked scenes visible in the sampled frames.The 100 diagnostic source IDs are absent from the training and validation sources of the fine-tuning subset.Source: §C.1 · p. 21

C.2 INPUT VERSIONS AND QUESTION CONSTRUCTION

  • The swap test independently exchanges windows in audio, video, or both; swapping one stream changes the correct scene pairing, while swapping both preserves it.Questions are asked independently without timestamps, window positions, or version labels.Source: §C.2 · p. 21

C.3 CHECKPOINTS, METRICS, AND UNCERTAINTY

SyncRA predictions track the current audio–visual pairing more reliably across controlled input versions. The representation analyses examine correspondence within and across videos, under altered audio, and across model depths and readout spaces.

C.3 CHECKPOINTS, METRICS, AND UNCERTAINTY

  • The evaluation compares Base, Vanilla SFT, and SyncRA checkpoints, with additional text-reasoning and objective controls for Qwen3-Omni.Text CoT-SFT reaches 12.00% AllFour and 54.06% overall accuracy on Qwen3-Omni; AllFour requires correctness in every input version, and each source receives equal weight.Source: §C.3 · pp. 22–23, Table 14 · p. 22

C.4 PREDICTION MATRICES ACROSS INPUT VERSIONS

  • 91.25% of SyncRA predictions are correct when both modalities move together, versus 80.25% for Vanilla SFT; video-exchange accuracy is 76.50% versus 53.00%.Under video exchange, predictions retaining the original scene fall from 33.25% for Vanilla SFT to 14.50% for SyncRA; audio-exchange accuracy is 81.00% versus 67.00%.Source: §C.4 · pp. 23–24, Figure 5 · p. 24

D.1 CROSS-BACKBONE HEATMAPS

  • Similarity matrices for four backbones compare unprojected Vanilla SFT and SyncRA states across audio and video time for two clips.The shared color scale spans 0.4–1, with values below 0.4 shown at the lightest color.Source: §D.1 · p. 25, Figure 6 · p. 26

D.2 SAMPLE CONSTRUCTION, STATES, AND RETRIEVAL DEFINITIONS

  • The retrieval analysis pools audio and video states within each interval, normalizes them, and measures bidirectional same-interval matching across 100 validation videos.The analysis also compares pooled readouts and defines retrieval-time error by the difference between predicted and target interval start times.Source: §D.2 · p. 25

D.3 AUDIO REPLACEMENT, DELAY, AND THE PROJECTION COMPLEMENT

  • After removing the 64-dimensional SyncRA projection row space, SyncRA retains 96.87%, 96.59%, and 96.21% R@1 under original audio, donor replacement, and delay.The results indicate that the matching improvement extends beyond that readout subspace.Source: §D.3 · p. 27, Table 20 · p. 27

D.4 LINEAR DECODING OF ELAPSED TIME

  • The released Base checkpoint already supports linear within-modality time decoding: audio/video MAEs were 8.996/5.675 seconds, versus shuffled-label controls of 30.816/31.041 and a 29.276-second constant predictor.Scalar time recovery and cross-modal interval matching measure distinct properties; high same-interval retrieval need not imply good cross-modal time-decoder transfer.Source: §D.4 · pp. 28–29

D.5 CLIP-LEVEL RETRIEVAL ACROSS SOURCE VIDEOS

  • Clip-level retrieval rankings depend on readout space and temporal granularity: Clip-level AV leads with full-state retrieval, while SyncRA leads through either learned projection.These cross-video results complement rather than replace within-video interval retrieval.Source: §D.5 · p. 29, Table 24 · p. 29

D.6 CORRESPONDENCE ACROSS MODEL DEPTH

SyncRA strengthens interval correspondence across intermediate and later model representations, with the observed distribution depending on supervision depth. The evaluation uses a compact 10K-question corpus, and how gains vary with corpus size remains open.

D.6 CORRESPONDENCE ACROSS MODEL DEPTH

  • At block 36, SyncRA raises R@1 from 13.39% to 96.91% and lowers retrieval-time MAE from 7.176 to 0.044 seconds; block 24 and the final block also show correspondence gains.The block-36 configuration has higher point estimates for AllFour and Daily-Omni than block 24, but their AllFour difference has a paired interval containing zero.Source: §D.6 · p. 30, Table 25 · p. 30

E.1 COMPARISON WITH LATENTOMNI

  • A controlled comparison with LatentOmni requires its code, checkpoints, and training dataset; those materials were not available in its official repository as of September 26, 2026.The paper instead compares objectives under shared data, budget, and backbones.Source: §E.1 · p. 31

E.2 TRAINING-DATA SCALE

  • The experiments use a compact 10K-question corpus, and how SyncRA’s gains vary with corpus size remains an open question.The reported gains span all 20 model–benchmark combinations across four backbones, two architecture families, and two update scopes.Source: §E.2 · p. 31

E.3 NUMBER OF TRAINING RUNS

The authors report three independent runs per trained configuration and emphasize the consistency of paired differences across model–benchmark combinations. The appendices provide complete objective comparisons, run-level results, and per-run task-equal macro summaries.

E.3 NUMBER OF TRAINING RUNS

  • SyncRA exceeds Vanilla SFT in all 20 model–benchmark combinations, and the four diagnostic paired 95% source-video bootstrap intervals exclude zero.Each trained configuration has three independent runs; the authors identify the across-combination pattern, rather than any single run, as carrying the conclusion.Source: §E.3 · p. 31

F FULL BENCHMARK RESULTS AND OBJECTIVE COMPARISONS

Related papers

Loading 2609.34363v1…