PaperlayerView exact revision on arXiv

Source-linked AI summary

Past as State, Present as Attention: Persistent-State Blockwise Flow Matching for Long-Horizon Co-Speech Motion Generation

Yi Liu, Xiangyue Zhang, Jianfang Li, Jia Ma, Yisheng He, Jianqiang Ren

arXiv:2610.32373v1cs.CVcs.GRcs.MM

TL;DR

Long-horizon co-speech motion generation must preserve temporal coherence and style, while short seeds can drift and explicit histories become costly and difficult to learn from limited data. PASPA summarizes past motion in a persistent state and models the current block with bidirectional attention, using blockwise flow matching and tailored training and inference. It achieves state-of-the-art FGD on public benchmarks, with a reported 43.9% FGD reduction versus GlobalDiff on SHOW.

  • Problem

    Short-seed conditioning can drift in coherence and style, whereas retaining explicit histories incurs growing costs and is difficult to learn from limited co-speech data.

    Source: §1 · p. 2

  • Method

    PASPA combines an inter-block persistent state for past history with intra-block bidirectional attention, using blockwise flow matching and tailored training and inference.

    Source: §1 · pp. 2–3

  • Results

    PASPA achieves state-of-the-art FGD on public benchmarks; on SHOW, FGD is 0.00111, a 43.9% reduction from GlobalDiff’s 0.00198.

    Source: §4.1 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    PASPA’s persistent-state and current-block attention design preserves style and prevents temporal drift during extended motion generation.

    Source: §5 · p. 10

    The fixed-dimensional persistent state can overwrite older context in extremely long sequences, losing distant gesture details or long-range correspondences.

    Source: §G.1 · p. 28

Abstract

from arXiv · show

Co-speech motion generation remains challenging over long horizons due to the need for temporal coherence and consistent style. Existing methods typically generate long motions segment by segment, either conditioning on a short seed of recent frames, which gradually drifts, or retaining ever-longer explicit histories, which are costly to maintain and hard to learn from limited data. We trace these limitations to a key asymmetry between past and present: past motion carries slowly varying style that should be compactly summarized, whereas the current segment exhibits fast-changing local dynamics that require fine-grained modeling. This asymmetry motivates a simple Past-As-State, Present-as-Attention principle for long-horizon co-speech motion generation. Following this principle, we propose PASPA, which summarizes past history with an inter-block persistent state and models the current segment with intra-block bidirectional self-attention. We instantiate PASPA within a blockwise flow matching framework, with multi-block supervision for effective training over variable histories, prefill-decode state caching for efficient rollout, and hybrid classifier-free guidance for enhanced history and speech conditioning. Extensive public-benchmark experiments show that PASPA achieves state-of-the-art FGD performance and preserves motion style across long-sequence generation.

1 Introduction

Long-horizon co-speech motion generation must preserve temporal coherence and speaker style, but short-seed conditioning can drift and explicit histories grow costly and difficult to learn from. PASPA represents slowly varying past style with a persistent state and models fast-changing current motion with full attention, within a blockwise flow-matching framework.

1 Introduction

  • Short-seed conditioning can drift in temporal coherence and style, while retaining explicit histories grows costly and is difficult to learn from limited data.These limitations motivate asking how to represent and propagate past motion without attending explicitly to raw historical sequences.Source: §1 · p. 2
  • PASPA compresses past history into a persistent state and uses full attention for the current block, reflecting their distinct style and local-dynamics roles.Past motion carries slowly varying style; current-block dynamics change quickly and need fine-grained modeling.Source: §1 · p. 2
  • PASPA combines a block-level recurrent state based on Gated DeltaNet with bidirectional self-attention within each current block.The state summarizes preceding blocks; attention enables full interaction for fine-grained current-block modeling.Source: §1 · p. 2
  • Blockwise flow matching trains with distinct noise levels and parallel denoising across multiple blocks, while inference reuses cached history states and hybrid guidance controls historical dependence.The guidance augments audio-conditioned guidance with history-drop guidance.Source: §1 · p. 2
  • The paper identifies past-present asymmetry as the basis for PASPA and contributes tailored multi-block training, state-cached inference, and hybrid classifier-free guidance.These contributions instantiate the past-as-state, present-as-attention principle.Source: §1 · p. 3

2 Related Work

Prior co-speech motion work includes codebook-based and generative approaches, while diffusion and flow matching support diverse synthesis and blockwise sequence generation. Long-context models trade attention’s cost and limited-data inductive bias against recurrent state methods that offer efficiency and temporal biases.

2 Related Work

  • VQ-VAE methods generate motion through learned codebooks but often have limited diversity; diffusion-based methods better support diverse synthesis and have become dominant.Other approaches add semantic controllability through hierarchical designs and language-based guidance.Source: §2 · p. 3
  • Block Diffusion combines intra-block denoising with inter-block autoregression, while Diffusion Forcing assigns independent per-token noise levels between next-token prediction and full-sequence diffusion.This autoregressive diffusion paradigm has also been adapted to motion generation.Source: §2 · p. 3
  • Attention scales quadratically with sequence length and lacks inductive biases for data-limited regimes, while state-space and linear-attention models use efficient recurrent state updates.Gated DeltaNet’s gating and delta-rule updates yield a temporal-decay bias leveraged for continuous motion dynamics.Source: §2 · p. 3

3 Methodology

PASPA generates co-speech motion block by block, using persistent state for past history and bidirectional attention for current-block dynamics. Its blockwise flow-matching training and state-cached inference support long-horizon rollout.

3.1 Preliminary

  • Long-horizon co-speech generation partitions motion, expression, and aligned speech into corresponding blocks, while flow matching interpolates clean motion and Gaussian noise over time.Motion uses 55 joint rotations and global translation in a 333-dimensional representation; expression uses 100 FLAME coefficients.Source: §3.1 · p. 4

3.2 Past-as-State, Present-as-Attention Module

  • PASPA assigns persistent state to slowly varying history and bidirectional attention to fast-changing local motion dynamics.Style features remain correlated over long timescales, whereas local dynamics decay rapidly.Source: §3.2 · p. 4, Figure 3 · p. 4

3.2.1 Inter-Block Persistent State.

  • Blockwise Gated DeltaNet updates a persistent state from the noisy current block and encoded clean history, with gates and delta-rule updates accumulating informative trends.The method freezes reads at each block’s initial state to prevent intra-block future leakage and condition only on strict past history.Source: §3.2.1 · p. 4

3.2.2 Intra-Block Bidirectional Attention.

  • Blockwise multi-head self-attention enables bidirectional token interaction within each block while forbidding cross-block interaction.Reshaping blocks as a batch reduces attention complexity from O((NK)^2d) to O(NK^2d).Source: §3.2.2 · pp. 5–6

3.2.3 Network Architecture.

  • The PASPA network replaces a self-attention sublayer with sequential BGDN and BMHSA sublayers, using audio, expression, motion, and speaker information in its architecture.A frozen audio-to-expression network supplies expressions, and torso-with-translation and hands branches are merged into full motion.Source: §3.2.3 · p. 6

3.3 Blockwise Flow Matching

  • Blockwise flow matching factorizes generation across blocks and predicts clean motion blocks, with aligned speech and expression conditioning each block.Multi-block supervision denoises blocks in parallel with distinct noise levels, exposing the model to varied history lengths and noise levels.Source: §3.3 · p. 6

3.4 Inference for Long-Horizon Rollout

  • Inference prefills a fixed-size state from the seed, decodes each block with that state as a read-only condition, then updates the state with the denoised block.Hybrid classifier-free guidance combines unconditional, condition-only, history-only, and full predictions; time-dependent guidance scales activate during mid-noise intervals [0.1, 0.6].Source: §3.4 · p. 7

4 Experiments

Experiments compare PASPA with prior methods on BEAT2 and SHOW, then analyze its modeling choices, training, guidance, long rollouts, and inference efficiency. Results favor PASPA on generation quality and show reduced long-rollout degradation with efficient state caching.

4.1 Main Results

  • PASPA achieves the best FGD and MSE on BEAT2 in both 1 Speaker and All Speakers settings, and improves all metrics over FM-based GlobalDiff.It also has second-best Div and competitive BC in the 1 Speaker setting.Source: §4.1 · p. 7, Table 1 · p. 8
  • In a blind study with 18 participants and 167 ranking groups, PASPA has the highest top-1 win rate across four criteria, at 39.8% overall.The criteria are realism, semantic consistency, synchronization, and diversity; qualitative comparisons also report better speech alignment and more natural gestures than baselines.Source: §4.1 · p. 8, Figure 5 · p. 8, Figure 6 · p. 9
  • On SHOW, PASPA records FGD 0.00111, a 43.9% reduction from GlobalDiff’s 0.00198, with slightly higher BA and second-best Div.The evaluation uses the BEAT2 training recipe without dataset-specific hyperparameter tuning.Source: §4.1 · p. 8, Table 2 · p. 9

4.2 Ablations and Analysis

  • Past-state plus current-block attention yields the best FGD: more raw-history attention helps, while state-based modeling of the current block performs worse than no history.Removing history also substantially degrades FGD.Source: §4.2 · p. 9, Table 3 · p. 9
  • Multi-block supervision initially degrades FGD; weakly monotonic timesteps and adaptive weighting reduce it to 5.614, with SG and LogitN further lowering it to 5.216 and 4.450.The reported issues are unordered block timesteps and imbalanced losses across blocks and noise levels.Source: §4.2 · p. 9, Table 4 · p. 9
  • Full Hybrid CFG reaches FGD 3.532, outperforming condition CFG at 3.883; enabling it adds 4.9–9.1% latency and 0.01 GB peak memory.History dropout alone slightly degrades FGD, and history guidance without the joint term remains worse than condition CFG.Source: §4.2 · pp. 9–10, Table 5 · p. 10, Table 7 · p. 10
  • Across 10-, 30-, and 60-second rollouts, PASPA shows the weakest FGD degradation and highest diversity; prefill-decode separation makes it the fastest method.Without prefill-decode separation, PASPA is the slowest; its peak memory matches Short Seed and stays below Token Attention’s.Source: §4.2 · pp. 9–10, Table 6 · p. 10, Figure 7 · p. 10

5 Conclusion

PASPA applies persistent-state summaries to past history and bidirectional attention to the current segment within blockwise flow matching. Public-benchmark evaluations report state-of-the-art FGD and preserved style during extended generation.

5 Conclusion

  • PASPA combines a persistent state for past history with bidirectional attention for the current segment in blockwise flow matching.Its training and inference schemes include multi-block supervision, prefill-decode, and hybrid classifier-free guidance.Source: §5 · p. 10

A Empirical Motivation for the Asymmetry Between Past and Present

The autocorrelation analysis finds that style features persist much longer than local motion dynamics, motivating persistent summaries of history and finer modeling of the current block.

A Empirical Motivation for the Asymmetry Between Past and Present

  • Style autocorrelation remains substantially positive beyond 100 frames, while local-motion autocorrelation approaches zero within a few tens of frames; the pattern persists with raw-pose-mean style features.The authors interpret this as a data-intrinsic asymmetry: history carries slowly varying style, whereas current-block dynamics change quickly.Source: A Empirical Motivation for the… · p. 18

B Background: Chunkwise Parallel Form of Gated DeltaNet

  • Chunkwise Gated DeltaNet reads prior chunks from a block-start state and combines that inter-chunk term with token interactions within the current chunk.The chunkwise form partitions the sequence into non-overlapping chunks.Source: B Background: Chunkwise Parallel Form… · p. 19

C.1 Details of Sequence-Level Losses

  • The sequence-level FK losses supervise absolute joint positions and pairwise bone-angle similarities, with the structural loss discouraging unnatural bone stretching.The joint loss directly supervises absolute positions; the structural loss enforces consistency between predicted and ground-truth bone-angle similarity matrices.Source: §C.1 · p. 20

C.2 Details of Boundary Losses

  • Boundary losses are applied to a concatenated window spanning adjacent predicted blocks to address possible kinematic discontinuities from non-overlapping denoising.The boundary terms include rotation, velocity, and acceleration matching, supporting continuity across block transitions.Source: §C.2 · pp. 20–21

D Dataset Information

The evaluation uses BEAT2 as its primary benchmark and SHOW as an additional benchmark with a distinct motion annotation source.

D Dataset Information

  • BEAT2 provides synchronized speech and motion from 25 speakers, with both a single-speaker Scott protocol and an all-speakers protocol; SHOW adds in-the-wild videos from four speakers with pseudo-ground-truth motion.BEAT2 contains about 60 hours of data [45], while SHOW contains about 26.9 hours and follows the DiffSHEG gesture-evaluation protocol [11].Source: D Dataset Information · p. 21

E.1 Implementation Details of Main Results

The implementation uses a shared PASPA backbone with dataset-specific motion preparation and expression predictors. Training and inference settings specify optimization, conditioning, and rollout choices.

E.1 Implementation Details of Main Results

  • BEAT2 motion is represented as a 333-dimensional vector, while SHOW poses are aligned to BEAT2’s joint ordering; both datasets use 16 kHz speech processed into WavLM features.The expression predictors remain frozen during PASPA training, with a reused GlobalDiff predictor on BEAT2 and a TalkSHOW-based predictor on SHOW.Source: §E.1 · pp. 21–22
  • The six-layer backbone uses hidden dimension 384, eight attention heads, BGDN for history propagation, BMHSA for current-block modeling, and a default block size of 64.The BEAT2 configuration has 65.158 M parameters and 16.633 G MACs per forward pass.Source: §E.1 · pp. 21–22
  • Training runs for 1,000 epochs on each dataset, using a mixed Muon-AdamW optimizer and adaptive loss weighting across timesteps and blocks.The reported runs use 316,000 BEAT2 steps and 82,000 SHOW steps; the shared learning rate drops from 10^-3 to 10^-4 after 5,000 steps.Source: §E.1 · pp. 22–23
  • Inference uses 10 denoising steps, prefills an 8-frame seed into the state, and applies Hybrid CFG with condition and history scales of 1.5.Guidance is restricted to t in [0.1, 0.6].Source: §E.1 · p. 23

E.2 Implementation Details of Other Experiments

The experiments specify long-rollout style evaluation and a separate benchmark of inference latency and memory across rollout lengths.

E.2 Implementation Details of Other Experiments

  • Long-rollout style analysis evaluates terminal 10-second windows at 10-, 30-, and 60-second horizons on test sequences exceeding 60 seconds.It compares Short Seed, which keeps the last 8 frames, with Token Attention, which retains all past frames; confidence intervals use clip-level bootstrap resampling.Source: §E.2 · p. 23
  • Inference efficiency is measured across rollout lengths from 10 to 600 seconds, comparing PASPA with state caching against variants that re-prefill history and two baselines.Latency and peak memory are measured after warm-ups; the CFG-overhead benchmark is run separately on an H100.Source: §E.2 · p. 23

E.3 Implementation Details of the User Study

The user study used randomized, anonymized method rankings against a ground-truth reference, while the time-distribution study selected a moderately negative LogitN mean using FGD as its primary criterion.

E.3 Implementation Details of the User Study

  • Eighteen participants produced 167 valid ranking groups, ranking four anonymized methods on 20-second clips across realism, semantic consistency, synchronization, and diversity.The interface included a ground-truth reference that participants did not rank.Source: §E.3 · p. 23, Figure 8 · p. 24
  • PASPA achieved the best mean rank across all four user-study criteria.The interface randomized method order per clip and presented the ground-truth motion as a non-rankable reference.Source: §E.3 · p. 23, Figure 8 · p. 24

F.1.1 Time Distribution

  • With σ = 1.0, LogitN sampling at µ = −1.0 achieved the best FGD; a more negative µ = −1.2 worsened FGD, so the study adopted µ = −1.0.FGD was the primary selection criterion, despite other µ values being favored by some secondary metrics.Source: §F.1.1 · p. 24, Table 9 · p. 24

F.1.2 Block Size and Training Sequence Length

The sensitivity study selected K = 64 and T = 256, balancing local motion modeling and the difficulty of learning multi-block denoising over longer windows.

F.1.2 Block Size and Training Sequence Length

  • K = 64 achieved the best FGD at T = 256; smaller blocks degraded FGD and substantially increased MSE.The authors attribute this to short blocks fragmenting local motion and limiting dense bidirectional interactions for fast-changing dynamics.Source: §F.1.2 · p. 25
  • T = 256 gave the best FGD with K = 64; T = 128 remained competitive on some secondary metrics, while T = 512 did not further improve FGD.The authors attribute the longer-window result to more inter-block dependencies making multi-block denoising harder to learn.Source: §F.1.2 · p. 25
  • The study adopted K = 64 and T = 256 as its default block size and training-window length.The block-size sweep fixed T = 256, and the window-length sweep fixed K = 64.Source: §F.1.2 · p. 25

F.1.3 Classifier-Free Guidance

The guidance ablations favor Hybrid CFG, with its history and condition guidance improving FGD over condition-only guidance in the main BEAT2 settings. Ablations also identify a preferred guidance composition, scale, and activation interval.

Conditional Classifier-Free Guidance

  • Condition dropout reduced FGD from 4.450 to 3.937, and condition guidance at wcond = 1.5 further reduced it to 3.883; larger scales degraded FGD.The authors attribute the dropout gain to training-time data augmentation and the moderate-scale gain to inference-time guidance.Source: §F.1.3 · p. 25

Hybrid Classifier-Free Guidance

  • With wcond = 1.5, adding history dropout alone raised FGD from 3.883 to 3.989, while history guidance at whist = 1.5 lowered it to 3.532.Over-guidance along the history axis raised FGD to 5.603 at whist = 2.5; the authors adopted (wcond, whist) = (1.5, 1.5).Source: §F.1.3 · pp. 25–26

Hybrid CFG Decomposition

  • Interaction composition achieved the best FGD among the tested Hybrid CFG decompositions, adding a joint-effect term beyond the marginal guidance directions.The compared modes included sequential guidance in opposite orders, parallel marginal contributions, and Interaction.Source: Table 13 · p. 26, §F.1.3 · p. 26, Table 12 · p. 26

Classifier-Free Guidance Activation Interval

  • The [0.1, 0.6] CFG activation interval achieved the best FGD among the tested intervals; extending it toward either noise extreme degraded FGD.The adopted interval sets guidance scales to non-trivial values only within the interval and to 1 otherwise.Source: Table 14 · p. 26, §F.1.3 · p. 26

Effect of CFG on Main Results

  • Hybrid CFG improved FGD over no CFG: 3.76 vs. 3.71 in the 1 Speaker setting and 2.59 vs. 2.33 in the All Speakers setting.Hybrid CFG also improved BC and MSE in both settings, with slightly lower Div; FGD was the primary quality metric.Source: §F.1.3 · p. 26

F.2 Additional Analysis of Our Method

Additional analyses report weaker FGD degradation and highest diversity for PASPA over longer rollouts, limited variation across generation seeds, and the best mean ranking across four user-study criteria.

F.2 Additional Analysis of Our Method

  • Across rollout horizons, PASPA showed the weakest FGD degradation and maintained the highest Div; it also had competitive BC and slightly lower MSE at later horizons.Figure 9 plots endpoint metrics across T ∈ {10, 30, 60} s, evaluated on the terminal 10 s, with 95% Wald confidence intervals.Source: §F.2 · pp. 26–27, Figure 9 · p. 27
  • Under a 50-seed All Speakers evaluation of the same checkpoint, PASPA achieved mean FGD 2.237 with standard deviation 0.074.The seed sweep used freshly initialized generation seeds, unlike the original main-table evaluation.Source: §F.2 · p. 27

F.3 User Study

  • PASPA achieved the best mean rank on all four user-study criteria, with an overall rank of 2.14 versus 2.51 for second-best EMAGE.The study assessed perceptual quality beyond automatic metrics.Source: §F.3 · p. 27

G.1 Limitations and Future Work

This section discusses PASPA’s limitations in generalization, memory, training, evaluation, and global translation, alongside broader-impact and reproducibility disclosures. It also covers the method’s risk characterization, asset licensing and safeguards, human-subjects study, and use of LLMs.

G.1 Limitations and Future Work

  • PASPA’s principle has not been verified beyond co-speech motion, and its instantiation may need adaptation for domains such as music-to-dance [39] or text-to-motion [71].Testing PASPA on other motion-generation domains is left for future work.Source: §G.1 · p. 27
  • The fixed-dimensional persistent state can overwrite older context in extremely long sequences, losing distant gestures or long-range correspondences despite retaining slowly varying style statistics.The authors propose log-linear attention [23], which uses a logarithmically growing set of states, as future work; they describe the compression trade-off as fundamental to linear recurrent models [16] [22] [82] [84].Source: §G.1 · p. 28
  • Teacher forcing conditions training blocks on ground-truth history, unlike inference on generated history, creating exposure bias that may accumulate into repetition, posture collapse, identity drift, or translation artifacts.The authors leave integrating Self Forcing [29], which rolls out autoregressively during training, as future work.Source: §G.1 · p. 28
  • Automatic metrics do not fully capture long-term stylistic consistency or perceptual quality, and broader real-world deployment validation remains necessary.The paper complements metrics with a user study but leaves developing style or perceptual metrics to future work.Source: §G.1 · p. 28

G.2 Failure Cases

  • PASPA produces plausible local gestures but fails to recover a ground-truth large in-place circling trajectory, keeping motion mostly within a fixed spatial region.The authors attribute this to jointly learning Euclidean global translation and constrained rot6d rotations in one stream, which may underemphasize large-amplitude translation; more accurate translation modeling is future work [92] [97].Source: §G.2 · pp. 28–29, Figure 10 · p. 29

G.3 Broader Impacts

  • PASPA may support digital humans, virtual assistants, and animated characters while also enabling misleading avatars, impersonation, disinformation, or non-consensual digital likenesses.The paper says its motion-only model may pose these risks when combined with separately synthesized voice, face, or identity systems; experiments use public datasets without new identifiable data collection or user-facing deployment, and released materials will encourage non-deceptive use.Source: §G.3 · p. 29, §10 · p. 35

H Licenses of Used Assets

  • The listed assets use varied terms, including Apache 2.0 for BEAT2 and SANA, non-commercial research licenses for SHOW and SMPL-X, and MIT for FLA and WavLM.The paper states that assets are properly cited and used in compliance with their licenses for academic research.Source: H Licenses of Used Assets · pp. 29–30

NeurIPS Paper Checklist

  • The checklist reports that the paper provides reproduction details and plans to release code, while reporting uncertainty measures and compute resources for experiments.The disclosures cover method and sampling specifications, datasets and configurations, uncertainty or variability for key analyses, and training and inference resources.Source: NeurIPS Paper Checklist · pp. 32–34

10. Broader impacts

  • The authors say PASPA does not generate language, images, video, audio, photorealistic appearance, or identity information, and discuss broader misuse considerations in Sec. G.3.They state that the method focuses on motion generation for research purposes and does not release a new scraped dataset or high-risk generative model.Source: §10 · p. 36
  • The authors release PASPA code, trained checkpoints, and generated motion results with documentation covering setup, data preparation, training, inference, evaluation, licensing, and limitations.They report using public datasets and codebases, citing original papers, and complying with licenses including Apache 2.0, MIT, BSD 3-Clause, and SMPL-X’s non-commercial research license.Source: §10 · pp. 36–37
  • The user study used voluntary rankings of short generated-motion clips, collected no personal data, and posed no risks beyond ordinary computer use; informed participants were told participation was voluntary.The authors say formal IRB approval was not required under their institution’s review process for this low-risk study, which followed the NeurIPS Code of Ethics; participants received no monetary compensation.Source: §10 · p. 37
  • The authors state that LLMs were not important, original, or non-standard components of the method or experiments; any use was limited to writing, editing, or formatting.They say any such use did not affect methodology, experiments, scientific claims, or originality.Source: §10 · p. 38

Related papers

Loading 2610.32373v1…