Ambient @ EgoLongQA 2026: Distilling Long-Video perception into a Sub-2B Model

1 Introduction

• EgoLongQA asks four-way questions about roughly ten-minute egocentric videos, often linking an anchor event to something that happened relative to it.The ≤2B division counts the entire multimodal checkpoint, including the vision tower.Source: §1 · p. 1
• Ambient’s agentic pipeline reaches 81.4% on the distribution-robust evaluation set, but is infeasible at 2B parameters.The pipeline samples three junior passes and uses a senior orchestrator with clip-retrieval tools.Source: §1 · p. 1
• The design distils the one-shot junior rather than the senior, whose multi-turn tool trajectory cannot be expressed in a single forward pass.The student receives the junior’s same input/output contract; the figure contrasts ∼393 s of orchestration with ∼32 s of generation.Source: Figure 1 · p. 2, §1 · p. 2
• Component-level distillation uses the junior’s inference-matched input protocol to avoid train/test mismatch.The contribution contrasts this with imitating the senior’s multi-turn tool trajectory.Source: §1 · p. 2
• 2.2132 B parameters are reduced to 1.9985 B by removing unused embedding rows, with zero logit change on retained rows.The embedding table accounts for 23% of the backbone; the authors note that this pruning is harder for generative models than classifiers.Source: §1 · p. 2
• Because language models carry option-identifier selection bias, the authors permute options to make gold answers uniform in training and held-out sets.The passage attributes the selection-bias observation to.Source: §1 · p. 2

2.1 The agentic teacher pipeline

• The junior is sampled three times; consensus answers are accepted, while disagreements are escalated to a senior model with high-fidelity clip tools.The agentic pipeline is submitted to the Large model division, and its junior perception module is distilled into the student.Source: §2.1 · pp. 2–3

2.2 The Student distillation

• The 2B Qwen3.5 vision-language student receives 100 sampled video frames, the question, and four options, then emits a timestamped description and structured answer.The model’s native dynamic-resolution vision tower and multimodal rotary position encoding support serving the adapter at a higher resolution than it was trained at.Source: §2.2 · p. 3

2.3 Oracle-filtered trace distillation

• 2,936 traces from 605 videos retain ground-truth-correct junior passes only when all three sampled passes agree; juniors solve two thirds of validation above 90%.All teachers are open-weight; no closed model is used in the pipeline.Source: §2.3 · pp. 3–4

2.4 Synthetic data

• 943 audited synthetic questions over 408 Ego4D videos raise the 2B student’s unseen-video score from 27.1% to 54.3%; generation rejects questions answerable without video or lacking visible evidence.The generation pipeline yields 26%; raw generator output assigns 42% of answers to “A”, motivating balancing.Source: §2.4 · p. 4, Figure 2 · p. 5

2.5 Training.

• Resolution transfer yields +5.8 points over low-resolution training and inference, while epoch 1 outperforms epoch 2 in all five distillation runs.Training uses LoRA with 3× choice-span weighting; evaluation uses held-out videos with options permuted to a uniform gold distribution.Source: §2.5 · p. 5

2.6 Vocabulary pruning to meet the parameter limit

• Pruning the multilingual embedding table reduces the backbone from 2.2132 B to 1.9985 B parameters, with exactly zero measured logit difference on retained rows.The embedding table is 23% of the backbone; the transformer, vision tower, projector, and tokenizer remain untouched.Source: §2.6 · pp. 5–6

3 Results

• 27.1% to 81.4% on the 2B backbone, while the 27B model rose from a 65% single-pass baseline to 85.7%.Replacing a three-vote, gated senior-agent pipeline with one greedy generation reduced per-sample latency from roughly 393 s to 32 s.Source: §3 · p. 6
• Synthetic data did not improve held-out accuracy but was reported to improve out-of-distribution robustness.Source: §3 · p. 6

3.1 Final leaderboard

• 0.8279 placed the entry first in the official ≤2B held-out division, 1.27 pp ahead of the runner-up.Source: §3.1 · p. 6

4.1 Reinforcement learning

• GRPO over the distilled model yielded no reproducible held-out gain; the authors suspect roughly 25 in-distribution, unseen rollout prompts were insufficient.Prompts seen during supervised training may yield memorization, while out-of-distribution prompts did not transfer.Source: §4.1 · p. 7

4.2 Frame budget: the sign of the effect depends on model capacity

• At ≤2B, increasing the frame count from 100 to 400 hurt accuracy on the same question indices with thinking enabled in both arms.Source: §4.2 · p. 7
• +11.2 pp accuracy at 35B with 400 rather than 100 frames, opposite the ≤2B result.This comparison used Qwen3.6-35B-A3B in the large-division agentic pipeline submission.Source: §4.2 · p. 7

• Distilling the agent’s perception component, rather than the agent, brought a 2B model to competitive long-form egocentric video QA.Source: §5 · p. 8
• Vocabulary pruning brought the model under the strict parameter limit using rows the task does not use.Source: §5 · p. 8
• The entry placed first, and its test score matched validation within 1.4 pp.Source: §5 · p. 8
• The authors attribute robustness to out-of-distribution videos to synthetic Ego4D data distilled into the student.Source: §5 · p. 8