Full-Duplex Interaction with Speech, Expressions and Gestures

• ELLSA 2 remains highly competitive on speech interaction and outperforms cascaded baselines on gesture and expression metrics, especially in generating semantic gestures.Source: §1 · p. 2
• The paper establishes a curated data and evaluation pipeline with expression-supervised conversational data, a semantic exemplar library, and a human-verified test set.It evaluates speaking-listening activity, prosodic alignment, full-duplex behavior, and gesture semantics, and investigates reference-based MLLM metrics whose system-level correlations with human judgments support preliminary assessment.Source: §1 · p. 2
• ELLSA 2 jointly optimizes full-duplex speech interaction and dialogue-conditioned gesture-expression generation in one end-to-end model.Its chunk-wise synthesizer uses acoustic features and distinct listening and speaking representations, and continues generating non-verbal behavior during verbal silence.Source: §1 · pp. 1–2
• Contextual semantic calls decide whether a semantic gesture fits the response, while template guidance uses a curated exemplar to guide its realization.The approach separates gesture invocation from realization and significantly improves semantic gesture quality.Source: §1 · p. 2

2.1 END-TO-END FULL-DUPLEX SPEECH INTERACTION

• ELLSA 2 contrasts with prior full-duplex systems by jointly optimizing speech, gestures, and expressions in one end-to-end framework.Prior systems include dual-model monitor–generator architectures (Fu et al., 2025; Wang et al., 2025), continuous speech generation (Défossez et al., 2024; Roy et al., 2026), and explicit dialogue-state prediction (Yu et al., 2025); ELLSA and ViBES also explore action-related modalities (Wang et al., 2026).Source: §2.1 · p. 2

2.2 CO-SPEECH GESTURE AND EXPRESSION GENERATION

• ELLSA 2 unifies speech, gesture, and facial-expression generation, adds semantically meaningful gestures, and investigates evaluation protocols for dyadic conversational gestures.Prior co-speech work spans skeleton- and joint-level representations, autoregressive, diffusion, and flow-matching methods; dyadic gesture and expression generation conditions on both interlocutors’ speech (Agrawal et al., 2025; Peng et al., 2026; Peng et al., 2025; Chu et al., 2026; Lim et al., 2026). Semantic gestures remain challenging for conventional co-speech models (Zhang et al., 2024; Liu et al., 2025a;b), and the evaluation investigation includes subsets, objective metrics, and MLLM-as-a-judge.Source: §2.2 · p. 3

3.1 STREAMING DIALOGUE GESTURE AND EXPRESSION GENERATION

• Autoregressive 0.8-second chunks interleave speaking audio, listening audio, and gesture-expression tokens for online, dialogue-aware generation.Motion uses region-specific RVQ-VAE codes for upper body, hands, and facial expression; each chunk predicts six motion steps from ten speaking- and ten listening-speech embeddings.Source: §3.1 · p. 3

3.2 SPEECH INTERACTION WITH GESTURE AND EXPRESSION GENERATION

• The dialogue backbone’s hidden states condition speech, gesture, and expression generation, while continuous listening enables nonverbal responses during verbal silence.Speaking-stream hidden states come from speech synthesis and are replaced by a learnable null representation during silence; listening-stream hidden states encode incoming speech and dialogue state.Source: §3.2 · p. 4

3.3 CONTEXTUAL SEMANTIC CALLS AND TEMPLATE GUIDANCE

• A binary semantic-call head decides whether context warrants a semantic gesture, then retrieved exemplar tokens guide its realization in the corresponding chunk.Look-ahead text representations allow the exemplar to be prepared before gesture generation, avoiding added latency in streaming interaction.Source: §3.3 · p. 5

3.4 MULTIMODAL CURRICULUM TRAINING

• Curriculum training first adds speech, gesture, and expression generation progressively, then integrates the modules through four stages, ending with joint optimization.The speech dialogue model is frozen while the gesture-expression module learns representation alignment in Stage 3; Stage 4 jointly optimizes the components using a lower learning rate for speech-dialogue parameters. Stages 1–2 follow SALMONN-omni (Yu et al., 2025).Source: §3.4 · p. 5

4.1 TRAINING AND TEST DATA PREPARATION

• Filtering retains 81 speakers for gesture supervision, and expression estimation, invalid-detection filtering, and crop stabilization produce temporally stable FLAME-based facial-expression supervision.The source is Seamless Interaction, filtered for visible speaking gestures and pronounced speaking–listening differences; the original dataset lacks off-the-shelf facial-expression annotations.Source: §4.1 · p. 6
• The curated benchmark contains 138 human-verified clips: 67 full-duplex examples and 71 semantic examples, alongside 3,227 semantic exemplars spanning 1,125 action families and 2,072 phrases.The full-duplex examples cover behaviors such as backchannels, overlaps, and interruptions; benchmark gestures serve as anchors for MLLM and human evaluation.Source: §4.1 · p. 6

4.2 EVALUATION MEASURES

• Speech evaluation reports S2T, transcribed S2S answer accuracy, GPTScore, and turn-taking success; expression measures include LVE, MHD, FDD, and JDD.Speech benchmarks are Llama Questions, Web Questions, TriviaQA, and AlpacaEval; expression measures assess speaking-related geometry and state-dependent dynamics.Source: §4.2 · p. 7
• Gesture evaluation pairs speaking–listening activity ratio and prosodic alignment with MLLM-based duplex and semantic scores on rendered interaction videos.Prosodic alignment contrasts matched-speech onset–gesture correlation with a negative sample from another session; the duplex score assesses timing and state-appropriate gestures, while the semantic score assesses conveyed meaning.Source: §4.2 · pp. 7–8
• Reference-based MLLM scores and human ratings assess duplex behavior and semantic meaning, with human judgments treated as the gold standard and score agreement analyzed.Gemini-2.5-Pro and Seed each judge clips twice at 5 fps using synchronized human references; reported scores average trials and judge models.Source: §4.2 · p. 8

5.1 SPEECH INTERACTION

• 78.0%, 47.3%, and 63.6%: ELLSA 2 achieves the highest S2S accuracy on all three QA benchmarks and averages 95.2% turn-taking accuracy.Turn-taking accuracy reaches 99.3% on both Llama Questions and Web Questions and 94.4% on TriviaQA; AlpacaEval S2T/S2S scores are 3.97/3.20, close to SALMONN-omni’s 4.01/3.22.Source: Table 1 · p. 8, §5.1 · p. 8

5.2 GESTURE GENERATION

• 87.8 F1 and 87.0 accuracy: ELLSA 2 predicts whether to invoke a semantic gesture, reflecting the dialogue backbone’s ability to determine when one is appropriate.The passage attributes this ability to the backbone’s semantic understanding and notes that speech interaction can benefit semantic gesture generation.Source: §5.2 · p. 8
• 2.711 vs. 2.014: semantic guidance raises ELLSA 2’s human semantic score and yields the highest semantic scores under both human and MLLM evaluations.The guided variant also achieves the best speaking/listening activity ratio; jointly optimized ELLSA 2 has the best alignment and highest human-rated duplex scores, and consistently beats the standalone generator on human duplex evaluation. The two ELLSA 2 variants remain comparable on other metrics.Source: §5.2 · pp. 8–9, Table 2 · p. 8
• 0.806/0.904: mean pairwise MLLM–human correlations reach these values for duplex/semantic quality, while aggregated judgments correlate at 0.853/0.919.Human annotator agreement exceeds 0.9 for both duplex and semantic evaluation; the results support aggregated MLLM judgments as a preliminary metric for efficient evaluation and model comparison.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 EXPRESSION GENERATION

• ELLSA 2 outperforms all cascaded expression baselines across six facial metrics, with guidance achieving the best overall performance.Both guided and unguided ELLSA 2 improve on the standalone generator on every metric.Source: Table 4 · p. 9, §5.3 · p. 9

6 CONCLUSION

• ELLSA 2 jointly generates speech, gestures, and facial expressions during speaking and listening, while semantic template guidance improves semantic gesture quality and reference-based MLLMs support preliminary system-level evaluation.The work also establishes a curated data and evaluation pipeline focused on full-duplex behaviors and semantic gestures.Source: §6 · pp. 9–10

AI USE STATEMENT

• AI tools assisted with language polishing, related-work identification, data curation, annotation interfaces, the demonstration page, appendix drafting, and Figure 2; authors reviewed generated code and content.No section of the main paper was drafted by generative AI, and MLLMs were used for semantic annotation and automatic evaluation.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The paper emphasizes mitigating harmful, discriminatory, or biased outputs, detecting AI-generated content, and clearly informing users when they interact with an AI system.It presents safety and transparency as priorities alongside improving human-like intelligence.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The authors provide model, training, and dataset details and state that models and datasets are publicly available, with code, data, and checkpoints planned for release upon acceptance.Further specifications are provided in Appendices A to D.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 FACTORIZED GESTURE AND EXPRESSION REPRESENTATION

• Gesture and expression use separate residual quantizers, while facial expression is factorized into mouth and orthogonal non-mouth components for encoding and recombination.The mouth basis is geometry-derived and captures 99.6836% of squared singular-value energy; the learned representation excludes lower-body joints.Source: §A.1 · p. 14

A.2 AUTOREGRESSIVE GESTURE AND EXPRESSION PREDICTION

• Gesture-expression prediction proceeds in 0.8-second intervals, with shared hidden states and separate categorical heads predicting 18 code indices without autoregression over quantization depth.Future intervals are masked, while previously generated gestures and expressions provide temporal context.Source: §A.2 · p. 15

A.3 PRESERVED ACOUSTIC FEATURES AND TEMPORAL ALIGNMENT

• Each interval aligns ten listening states at 12.5 Hz with twenty raw speaking states at 25 Hz, groups speaking features to the conditioning rate, and predicts six latent steps.Outgoing speech features retain their utterance intervals, with silence represented in the intervening gaps; generation context is capped at 20 seconds.Source: §A.3 · p. 15

A.4 SEMANTIC-CALL PREDICTION AND SUPERVISION

• The semantic-call router identifies lexical candidates, while its call head judges contextual appropriateness using causal response-prefix and dialogue-context states.Soft targets from two language-model judgments supervise contextual selection within the available gesture repertoire rather than reproduction of the observed speaker’s gesture.Source: §A.4 · pp. 15–16
• Contextual semantic-call prediction reached 87.8% F1 and 87.0% accuracy on a balanced, candidate-conditioned test set.The test measures whether an available gesture fits a proposed lexical endpoint; its locked confusion matrix is TP/FP/FN/TN = 144/30/10/124.Source: §A.4 · p. 16

A.5 TEMPLATE-GUIDED GESTURE GENERATION

• Template guides condition continuation rather than substituting exemplar poses, and guide conditions directly apply to upper-body and hand codebooks while facial targets remain dialogue-specific.Causal attention exposes the current guide and speech conditions to targets; after the verified event span ends, unguided continuation resumes.Source: §A.5 · p. 16

A.6 TRAINING OBJECTIVES AND CONFIGURATION

• Stage 4 jointly optimizes speech dialogue and gesture–expression generation after staged connection and adaptation, while tokenizers, the original acoustic pathway, and the LLM backbone remain frozen except for LLM LoRA.Stages 1–2 connect speech perception and synthesis following SALMONN-omni (Yu et al., 2025); Stage 3 adapts a separately pretrained gesture–expression generator with the speech system fixed, and Stage 4 enables joint optimization through added conditioning paths.Source: §A.6 · p. 17
• Stage 4 trains on speech dialogue, semantic calls, free generation, and guided generation at a 2:1:4:1 sampling ratio, using learning rate 2 × 10−5, batch size 64, and 20k steps.Stages 1–3 use ASR and QA for 40k steps, then ASR, QA, and multi-turn dialogue for 60k steps, followed by sampled free and guided gesture–expression generation at 4:1 for 2.5k steps.Source: §A.6 · p. 17
• The joint objective combines dialogue next-token cross-entropy, speech acoustic-token cross-entropy, grouped categorical gesture–expression loss, and semantic-call-head loss.Dialogue cross-entropy includes conversational control tokens.Source: §A.6 · p. 18
• Gesture–expression loss is averaged over valid target positions and weights 6 upper-body, 6 hand, 4 non-mouth expression, and 2 mouth-expression codebooks, decreasing weights with quantization depth.The decreasing depth weights prioritize earlier RVQ stages, which typically capture dominant components of reconstructed gesture or expression.Source: §A.6 · p. 18

A.7 MODEL SPECIFICATIONS

• The gesture–expression transformer is initialized from Qwen2.5-0.5B-Instruct and has 24 layers, hidden size 896, 14 query heads, two key/value heads, and feed-forward size 4,864.It uses 18 independent embeddings and output heads, each with a 2,048-code vocabulary.Source: §A.7 · p. 18
• ELLSA 2 has 10.78B parameters, with 776M trainable parameters across generator, connector, LoRA, semantic-head, and special-token groups.The trainable groups include 696M generator parameters, 15M gesture-connector, 13M LLM LoRA, 1M synthesizer LoRA, 38M speech-connector, 10M speech synthesis-connector, 2M semantic-head, and 16K added special-token parameters.Source: §A.7 · p. 18

B.1 SPEAKER FILTERING

• 294,700 of 735,705 assessed windows from 3,401 speakers passed the window-level checks.Windows were assessed in 60-second spans at a 30-second stride and 30 Hz; passing required hard checks, component minima, and a total score threshold.Source: §B.1 · pp. 18, 20
• Selection was performed at speaker level: 81 speakers were selected, and those speakers contributed 1,318 participant recordings.Source: §B.1 · p. 20

B.2 FACIAL SUPERVISION EXTRACTION

• All 1,318 selected recordings have expression files, and training targets come from contiguous jointly valid body-and-face spans.KaoLRM extracts facial-keypoint crops at 30 Hz, smoothing crop trajectories while retaining missing detections as invalid.Source: §B.2 · p. 20

B.3 SEMANTIC EXEMPLAR LIBRARY CONSTRUCTION

• Gemini-proposed semantic events were video-verified, yielding 29,248 recordings with 77,585 annotated events before pose and text screening.Candidate windows were 192 frames (6.4 seconds), aligned to a 24-frame generation grid; text classification assigned action families and exact spoken anchors.Source: §B.3 · pp. 20–21
• The final semantic inventory contains 3,227 exemplars, 1,125 families, and 2,072 uniquely assigned phrases.The broad library has 7,665 exemplars; a semantic-and-family confidence filter leaves 3,672 before phrase disambiguation. Construction limits each family to twelve examples and each speaker or session to two contributions per family.Source: §B.3 · p. 21
• Two language models judged whether candidate actions would be natural and informative at lexical endpoints, producing 18,621 soft targets across 10,145 text segments.Annotation covered 24,527 endpoints; balanced development and test sets contain 312 and 308 endpoints, respectively, with disjoint sessions and a shared exemplar inventory.Source: §B.3 · p. 21

B.4 HUMAN-VERIFIED DIALOGUE TEST SET

• The human-verified test set contains 138 twenty-second clips: 82 naturalistic and 56 improvised, including 71 accepted semantic-gesture clips.Duplex categories cover backchannels, overlapping speech, target interruptions, and partner interruptions; objective activity and facial metrics use 4–16 seconds of each clip.Source: §B.4 · p. 22

C.1 SPEECH INTERACTION

• Speech interaction evaluation uses 300 Llama Questions, 2,032 Web Questions, 1,000 TriviaQA questions, and 199 AlpacaEval instructions.QA scoring normalizes predicted and reference text and accepts any non-empty reference alias; S2T uses native generated text, while S2S uses Whisper-large-v3 audio transcriptions. Alpaca responses are rated for accuracy and reasonableness on a 1–5 scale, and turn-taking succeeds when a response arrives within two seconds after the user query ends.Source: §C.1 · p. 22

C.2 GESTURE ACTIVITY AND PROSODIC ALIGNMENT

• The activity statistic compares mean joint displacement during target speaking, including overlap, with displacement during target silence and partner speech, excluding mutual silence.Displacements use 13 upper-body joints on a common root-fixed skeleton and are evaluated at 30 Hz; values above or below one indicate greater speaking or listening activity, respectively.Source: §C.2 · p. 22
• A measures the matching audio–gesture onset-strength and angular-velocity correlation against an assigned mismatched-audio correlation, with the contrast scaled by a frozen development-set value.Correlations are maximized over lags from −0.4 to +0.4 s after smoothing; the frozen scale is 0.207.Source: §C.2 · pp. 22, 24
• The BeatAlign comparison uses the archived GestureLSM BEAT-style measure (Liu et al., 2022; 2025c) and reports matched scores plus controls using mismatched speech.The implementation fixes the Gaussian width, local-extremum order, and normalized-speed threshold; the controls include a mean matched-minus-mismatched margin and a sigmoid contrast.Source: §C.2 · p. 24
• A reverses only the last two systems in the human ranking, while standard BeatAlign and both paired controls agree less closely; mismatched speech improves BeatAlign’s rank agreement.The comparison uses seven generated systems and 616 matched/mismatched comparisons on the common pose-available subset.Source: Table 8 · p. 24, §C.2 · p. 24

C.3 REFERENCE-BASED MLLM SCORES

• The duplex MLLM score weights timing, conversational appropriateness, and visibility, while the semantic score combines visible evidence, meaning match, concreteness, and a generic beat/fidget penalty.The scores are D0 = 0.45T + 0.45P + 0.10V and S0 = clip(0.35V + 0.45M + 0.20K − 0.30F, 0, 1).Source: §C.3 · pp. 24–25
• Gemini-2.5-Pro and Seed score applicable clips twice using a reference-based rubric and 5 fps video input, with separate reference and candidate videos in each request.The protocol requests normalized component scores and validates ranges, timestamps, evidence intervals, target-speech absence, and nonoverlap of penalty types.Source: §C.3 · pp. 24–25
• Both MLLM scores penalize unjustified movement during target silence and overactivity elsewhere, with natural gesture preparation, returns, and listener feedback exempt.Each judgment is penalized before averaging four judgments, followed by equal-weight averaging over 67 duplex or 71 semantic items.Source: §C.3 · p. 25
• Table 10 reports Pearson correlation matrices for Duplex and Semantic scores, with each cell formatted as sample-level and system-level correlation.The supplied passages specify the matrix format but do not provide its correlation values.Source: Table 10 · p. 26, §C.3 · p. 26

C.4 HUMAN-RATING PROTOCOL AND CORRELATION ANALYSIS

• Two annotators each provided 966 valid 1–5 ratings covering seven systems on the same 67 duplex and 71 semantic clips, with human references anchoring the scale but not rated as systems.Both final exports covered all 966 trials without invalid ratings.Source: §C.4 · p. 26
• Correlation analysis paired individual human ratings with judge means across seven systems, using 469 duplex or 497 semantic sample pairs and seven system means per task.Human and MLLM scores were averaged over annotators or judge means, respectively, and then equally over clips within each task.Source: §C.4 · p. 26
• MLLM–human correlations were 0.806/0.904 for duplex/semantic; averaged judge–annotator scores correlated across systems at 0.853/0.919 and at sample level at 0.3966/0.5579.Human–human Pearson correlations were 0.6332/0.5906 at sample level and 0.9872/0.9400 at system level.Source: §C.4 · p. 26
• The anonymous-rating interface provided a reference and randomized trial order; annotators watched both 20-second clips, rated task response, then checked global activity.Annotators could replay video at 30 fps, while MLLMs sampled at 5 fps; the five anchors ranged from missing or unrelated response to reference-level function without unjustified activity.Source: §C.4 · p. 26

C.5 FACIAL GEOMETRY AND DYNAMICS

• Facial evaluation covers the central 4–16 seconds, with speaking frames following target voice activity including overlap and non-speaking frames comprising its complement, including mutual silence.Source: §C.5 · p. 26
• Facial metrics use common FLAME geometry with zero identity and fixed head, neck, and eye pose; LVE measures lip-vertex reconstruction error, MHD whole-head error, FDD upper-face temporal variation, and JDD jaw-opening variation.The setup isolates expression and jaw articulation.Source: §C.5 · p. 28
• Metrics are computed within each clip and state, then clips are equally averaged; temporal standard deviations use |T| −1 and require at least two valid frames.FDD sums coordinates before temporal standard deviation, while JDD measures jaw opening on the local jaw x axis in radians.Source: §C.5 · p. 28

D.1 SPEECH UNDERSTANDING AND INTERACTION

• Speech-data preparation follows SALMONN-omni (Yu et al., 2025), combining speech recognition, spoken question answering, and synthetic multi-turn conversation across instruction following, factual questions, longer answers, and conversational responses.Available question recordings are retained, while response speech is synthesized with CosyVoice2-0.5B (Du et al., 2024); for text-only sources, the inherited pipeline uses the base LLM to produce conversational answers and synthesizes both sides.Source: §D.1 · p. 28
• Multi-turn examples retain speaker turns and include interruption and competing-speech variants so the system can learn when to continue, yield, or respond.ASR-only examples are used in Stage 1; the spoken-QA and conversation pool connects synthesis and dialogue timing in Stage 2 and is reused to retain speech capability during joint optimization in Stage 4.Source: §D.1 · p. 28

D.2 GESTURE, EXPRESSION, AND SEMANTIC CALLS

• The gesture–expression pool contains 1,318 recordings from 81 selected Seamless Interaction speakers, paired with KaoLRM-derived facial parameters.Each recording preserves both participants’ audio and its timing relative to the target person’s body, hands, and face, supervising speaking and listening in the same representation.Source: §D.2 · p. 29
• The guided generation manifest contains 8,372 cached windows: 4,048 have one, 4,132 have two, and 1,942 have three.Source: §D.2 · p. 29
• Semantic-call training retains 10,145 text segments with 18,621 supervised endpoints; the call task supplies text context while other scheduled tasks supply audio and gesture–expression targets.Source: §D.2 · p. 29

E ABLATION STUDIES

• Ablations examine gesture training data, facial tokenization, and joint adaptation on 138 human-verified clips: 67 duplex and 71 semantic.They retain alignment, duplex, and semantic scores, where larger is preferred; expression metrics are lower-is-better. Expression speaking/non-speaking states are S/L, with facial L including mutual silence, while RSL listening excludes it.Source: E ABLATION STUDIES · p. 29

E.1 GESTURE TRAINING DATA

• Speaker selection improved RSL by 0.089 and prosodic alignment A by 0.1424.The comparison used gesture-only generators trained on the full pool of 121,054 recordings from 3,401 speakers versus a curated subset of 1,318 recordings from 81 selected speakers.Source: Table 12 · p. 29, §E.1 · p. 29
• Speaker selection improved duplex score D by 0.2513 and semantic score S by 0.1800.The comparison used gesture-only generators trained on the full pool of 121,054 recordings from 3,401 speakers versus a curated subset of 1,318 recordings from 81 selected speakers.Source: Table 12 · p. 29, §E.1 · p. 29

E.2 FACE EXPRESSION

• Factorized facial tokenization reduced speaking LVE by 0.6748 mm compared with a unified six-level facial RVQ.The factorized tokenizer used a four-level non-mouth RVQ and a two-level mouth RVQ; both designs used the same 81-speaker generator-training pool.Source: Table 13 · p. 29, §E.2 · p. 29
• Factorized facial tokenization reduced speaking MHD by 0.1223 mm compared with a unified six-level facial RVQ.The factorized tokenizer used a four-level non-mouth RVQ and a two-level mouth RVQ; both designs used the same 81-speaker generator-training pool.Source: Table 13 · p. 29, §E.2 · p. 29

F LIMITATIONS.

• ELLSA 2 lacks visual perception, and its semantic exemplar library draws only on Seamless Interaction, limiting multimodal understanding and semantic coverage.The authors propose integrating visual inputs and adding high-quality exemplars from larger, more diverse sources to improve coverage and robustness.Source: F LIMITATIONS · pp. 29–30

G.1 DUPLEX MLLM SCORE

• Duplex scoring compares synchronized reference and candidate videos, independently verifies visible candidate motion, and gives substantial credit for sparse functional fragments while capping ambiguous-cue Timing at 0.37.The metric focuses on the strongest event rather than averaging the clip; a local response can earn credit despite generic motion elsewhere.Source: §G.1 · p. 30

G.2 SEMANTIC MLLM SCORE

• Semantic scoring independently inspects candidate motion and credits a visible coarse property shared with the reference, even when the full gesture is ambiguous or incomplete.It is a sparse strongest-event metric: a useful fragment can earn credit, while arbitrary motion without a relevant visible property should remain low.Source: §G.2 · pp. 31–32

G.3 SHARED ACTIVITY ADDENDUM

• The shared-activity addendum preserves local event scores, separately assesses full-clip inappropriate activity, and defines a silence-motion penalty only for verified, sustained gesticulation during target silence.It exempts natural adjustments, meaningful listening responses, and justified gesture carry-over; the supplied passage introduces a separate overactivity penalty but does not provide its criteria.Source: §G.3 · pp. 33–34
• Global penalties compare candidate motion with the same-time human reference, separate silence and overactivity evidence, and compute Final = Base * (1 - 0.45*S - 0.35*O).Do not penalize lively or useful sparse gestures; silence penalties require eligible silent time, and evidence intervals for S and O must not overlap.Source: §G.3 · pp. 34–35

G.4 REQUEST CONTEXT AND OUTPUT EXAMPLE

• Requests provide causal dialogue, event timing, role and reference annotations, label reference and candidate videos separately, and require timestamped evidence for positive deductions.Low-energy intervals must be checked against target audio while allowing legitimate feedback to partner speech; format-recovery feedback validates the schema without changing the rubric or choosing among valid scores.Source: §G.4 · p. 35

G.5 CONTEXTUAL-CALL ANNOTATION

• Contextual-call annotators judge whether the supplied gesture family is useful after the trigger in causal context, using a 0–4 scale and accounting for mismatch, incompleteness, and repetition.They assess the specific family rather than the original speaker’s behavior, reject misleading literalization or unsupported action, and mark the endpoint incomplete if interpretation requires future words.Source: §G.5 · pp. 35–36

G.6 SOURCE-VIDEO SEMANTIC CURATION

• Source-video curation retains only unambiguous explicit emblems, deictic gestures, and iconic or mimetic gestures, excluding beats, fidgets, ambiguous meanings, and abstract gestures needing interpretation.Descriptions should match spoken content without speculation; a separate verification prompt checks whether the described gesture occurs within the original segment’s time window.Source: §G.6 · pp. 36–37