End-to-end Listen, Look, Speak and Act

Motivation

• Current systems remain limited because conversational models lack physical action, while VLA models typically use rigid, half-duplex text-instruction frameworks without raw speech or spoken responses.These limitations prevent natural turn-taking and barge-ins.Source: §1 · pp. 1–2

Contributions

• ELLSA unifies vision, speech, text, and action in a streaming full-duplex model that processes inputs and generates speech and actions concurrently.It supports speaking-while-acting, context-grounded VQA while acting, and action barge-ins.Source: §1 · p. 2

Method

• SA-MoE routes modalities to specialized experts and integrates them through unified self-attention, preserving task specialization while enabling cross-modal relationships and reducing modality interference.The speech expert handles speech and text, while the action expert handles vision and action.Source: §1 · p. 2

Contributions

• SA-MoE achieves significantly superior performance to a single dense model with less training cost.Source: §1 · p. 2

Results

• ELLSA performs on par with specialized models across speech-interaction and robotic-manipulation benchmarks while enabling dialogue and action turn-taking, defective-instruction rejection, speaking-while-acting, and action barge-ins.Source: §1 · p. 2

Duplex Multimodal Interaction Models

• Earlier speech-dialogue systems support low-latency conversation and some visual perception, but remain unable to translate understanding into physical actions.Source: §2 · pp. 2–3

Vision-Language-Action Models

• ELLSA advances beyond these systems by supporting simultaneous listening, looking, speaking, and acting with end-to-end speech input, speech output, and precise action prediction.It can decide whether to answer, act, or be interrupted during execution.Source: §2 · p. 3
• Large VLA models perform robotic tasks but generally restrict outputs to actions and remain half-duplex, limiting natural language interaction and interruption handling.Some extensions add question answering or instruction rejection, but retain half-duplex operation.Source: §2 · p. 3

3.1 STREAMING FULL-DUPLEX MIMO

• ELLSA interleaves speech input, image input, text output, and action output within each time block to support streaming full-duplex MIMO interaction.Speech output is derived from text-output embeddings; default mode activates all four modalities, while speech-only mode uses speech and text.Source: Figure 1 · p. 3, §3.1 · p. 4

3.2 SA-MOE

• SA-MoE assigns modalities to designated experts and uses unified attention to fuse information across experts while activating only one expert’s weights at each step.The speech expert handles speech and text, whereas the action expert handles vision and action; attention-connected experts share cross-modal information.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
• SA-MoE preserves expert strengths, reduces modality interference, and provides a flexible framework that can incorporate additional modality-specific experts.ELLSA merges speech with text and vision with action to leverage pretrained knowledge.Source: §3.2 · p. 5

3.3 TRAINING STRATEGY

• Training proceeds in three stages: individual speech and action experts, integrated SA-MoE training on basic and advanced interaction tasks, then end-to-end connection of a streaming speech synthesizer.The final stage feeds speech-expert hidden states through a trainable connector to give ELLSA speech generation.Source: Figure 3 · p. 5, §3.3 · pp. 5–6

4.1 MODEL SPECIFICATIONS

• ELLSA processes one second of speech and a video frame per time block, then generates text, speech, and action outputs.The model produces eight text tokens, or <silence> when no verbal response is required, alongside one second of speech and action output.Source: §4.1 · p. 6
• The speech and action experts use separate backbones: the speech expert pairs a streaming Mamba encoder with LLama-3.1-8B-Instruct, while the action expert uses Emu3-Base with Emu3-VisionTokenizer and FAST.Source: §4.1 · p. 6

4.2 DATA AND TASK SPECIFICATIONS

• ELLSA is trained on basic speech and manipulation tasks plus advanced tasks requiring speaking while acting, context-grounded VQA, defective-instruction rejection, and action barge-ins.These tasks extend evaluation from individual capabilities to multimodal interaction scenarios.Source: §4.2 · pp. 6–7
• Speaking-while-acting requires verbal responses during uninterrupted action execution, whereas context-grounded VQA asks about the environment’s current state during execution.The VQA questions are derived from LIBERO LONG and depend on the stage reached in a sequential manipulation task.Source: §4.2 · p. 6
• Defective-instruction rejection covers visual, semantic, motion, and out-of-context errors, while action barge-in requires immediate cancellation after an interruptive command.For barge-in, ELLSA explicitly responds with “Action Cancelled” to indicate that the ongoing action has stopped.Source: §4.2 · p. 7
• Figure 4 illustrates a single interaction combining spoken instruction, action execution, context-grounded VQA, and action barge-in.The example demonstrates simultaneous multimodal input-output processing and conversational interruption handling.Source: Figure 4 · p. 7
• Evaluation reports speech-to-text, speech-to-speech, accuracy, GPTScore, and task success rate across speech interaction, VQA, manipulation, and duplex abilities.Task success rate is also used for conversational dynamics such as turn-taking and barge-ins.Source: §4.2 · p. 7

5 RESULTS

• SA-MoE substantially outperforms a dense model trained on all tasks, supporting the use of pretrained experts to reduce modality interference.Detailed comparisons are provided in Table 7.Source: §5 · p. 7

5.1.1 SPEECH INTERACTION

• ELLSA delivers performance comparable to current open-source full-duplex interaction models and achieves the highest S2S performance.The speech benchmarks include Llama Questions, Web Questions, TriviaQA, and AlpacaEval.Source: §5.1.1 · pp. 7–8, Table 1 · p. 8

5.1.2 SPEECH-CONDITIONED ROBOT MANIPULATION

• ELLSA achieves the highest average performance across all LIBERO task suites in speech-conditioned robot manipulation.Unlike conventional text-conditioned, turn-based VLA evaluation, ELLSA receives speech instructions and decides when to initiate actions.Source: §5.1.2 · p. 8, Table 2 · p. 8

5.2.1 FULL-DUPLEX ABILITY

• ELLSA successfully predicts dialogue and action turn-taking, rejects all four defective-command types with high accuracy, and largely preserves valid-instruction execution.The authors hypothesize that its longer one-second modeling block may contribute to its dialogue turn-taking advantage over shorter-block systems.Source: §5.2.1 · p. 8, Table 3 · p. 9
• During action execution, ELLSA distinguishes general questions, interruptive commands, and silence, respectively continuing while answering, cancelling the action, or proceeding silently.Table 3(c) reports success rates across these speech-input conditions.Source: §5.2.1 · pp. 8–9

5.2.2 SPEAKING-WHILE-ACTING

• ELLSA can generate speech while executing actions, but performance declines when simultaneous processing becomes difficult, especially on LIBERO LONG and TriviaQA.Table 4 averages results across question-answering and manipulation datasets.Source: §5.2.2 · p. 9, Table 4 · p. 9

5.2.3 CONTEXT-GROUNDED VQA

• ELLSA reaches approximately 80% average accuracy on context-grounded VQA during environmental interaction.The result indicates that the speech expert can use visual information despite not being trained on visual data.Source: §5.2.3 · p. 9, Table 5 · p. 9

• ELLSA coordinates vision, speech, text, and action in real-time full-duplex interaction through attention-connected modality-specific experts.Source: §6 · p. 10
• SA-MoE addresses modality interference while enabling fluid cross-modal communication across the unified architecture.Source: §6 · p. 10
• ELLSA achieves competitive performance on standard benchmarks while unlocking speaking-while-acting, context-grounded VQA, and action barge-ins.Source: §6 · p. 10

ETHICS STATEMENT

• The ethics statement emphasizes safeguarding against harmful, discriminatory, or biased outputs, detecting AI-generated content, and informing users when they interact with AI.Source: ETHICS STATEMENT · p. 10

A.1 WORKING MODE

• ELLSA supports default and speech-only modes using the order speech → image → text → action.Source: §A.1 · p. 15
• In speech-only mode, visual input is absent and the model produces dummy actions without actual movement, while default mode incorporates all modalities.Source: §A.1 · p. 15
• Mode-specific prompts support transcript generation and question answering, with prompts processed by the speech expert at the beginning of the multimodal sequence.Source: §A.1 · p. 15
• ELLSA preserves complete speech and text history but retains only the last two seconds of vision and action history to reduce sequence length for current-observation VLA and VQA tasks.Source: §A.1 · p. 15

• Matching 32-layer, 4096-hidden-size backbones with 32 attention heads and 8 key-value heads enable attention-based fusion without additional SA-MoE parameters.Source: §A.2 · p. 15
• The speech expert’s Mamba encoder produces 25-Hz embeddings and downsamples them to 5 Hz by concatenating each five-frame group before the speech backbone.Source: §A.2 · p. 15
• The experts retain distinct RoPE settings while sharing token indices across the multimodal sequence.Source: §A.2 · p. 15

• Stage 2 trains SA-MoE on ASR, speech QA, robot manipulation, speaking-while-acting, context-grounded VQA, defective instruction rejection, and action barge-ins for 500 steps.This stage uses batch size 1024 and learning rate 4 × 10−4.Source: B TRAINING DETAILS · p. 15
• Stage 3 continues mostly the same task mixture for 20k steps but omits speech-conditioned robot manipulation because it produces no textual output beyond <silence>.The stage uses batch size 256 and learning rate 2 × 10−4.Source: B TRAINING DETAILS · pp. 15–16
• The Mamba streaming encoder is pretrained for 300k steps, followed by 40k-step speech-expert training on ASR and speech QA.Pretraining uses LibriHeavy and GigaSpeech; stage 1 uses batch size 512 and learning rate 2 × 10−4.Source: B TRAINING DETAILS · p. 15
• Speech responses and instructions are synthesized with CosyVoice2-0.5B, while Whisper filters action-barge-in samples and Gemini-2.5-Pro supplies annotations for rejection and VQA tasks.Source: B TRAINING DETAILS · p. 16
• Table 6 summarizes the training datasets used by the system.Source: Table 6 · p. 16

C EVALUATION DETAILS

• Speech interaction uses four datasets, while robot manipulation reports average success across 500 episodes with 50 episodes per task.AlpacaEval responses receive GPTScore 2 ratings from 1 to 5, and speech-to-speech outputs are transcribed with Whisper-large-v3.Source: C EVALUATION DETAILS · p. 16
• Full-duplex turn-taking succeeds when responses occur within 1 second after a question ends, while speaking-while-acting inserts speech queries during ongoing action execution.Speaking-while-acting uses 100 episodes per task suite, and context-grounded VQA extends the insertion interval to 2–30 seconds.Source: C EVALUATION DETAILS · pp. 16–17

D.1 COMMANDS FOR ACTION BARGE-IN

• Action-barge-in evaluation uses commands such as “Stop now,” “Hold on,” and “Pause here” to test interruption handling during action execution.Source: §D.1 · p. 17

D.2 QUESTIONS FOR CONTEXT-GROUNDED VQA

• Context-grounded VQA pairs task scenes and instructions with questions about object locations, grasped items, task progress, and completed placements.Examples ask whether a stove is on, what the robot is holding, and how many moka pots have reached the stove.Source: §D.2 · pp. 17–18

D.3 DEFECTIVE INSTRUCTION REJECTION

• Defective instructions are categorized as visual, semantic, motion, or out-of-context according to absent properties, absent objects, impossible motions, or irrelevant commands.Source: §D.3 · p. 19
• ELLSA identifies and rejects defective instructions while providing reasonable justifications grounded in the current context.Illustrations cover visual color errors, absent objects, impossible motion through solids, and contextually incongruent commands.Source: §D.3 · p. 19, Figure 5 · p. 19, Figure 6 · p. 19

E.1 PROMPTS FOR LLAMA-3-8B-INSTRUCT TO GENERATED SPEECH QA RESPONSE

• The evaluation pipeline prompts models to generate conversational responses, annotate video-based VQA, judge answer correctness, and create defective robotic-instruction examples.Speech responses are rated from 1 to 5 solely for accuracy and reasonableness.Source: §E.1 · p. 20, §E.2 · p. 20, §E.3 · p. 21

F FURTHER DISCUSSION OF SA-MOE

• SA-MoE integrates pretrained experts across diverse modalities to address modality interference and supports data-efficient post-training rather than primarily serving as a pretraining architecture.Source: F FURTHER DISCUSSION OF SA-MOE · p. 22
• SA-MoE significantly outperforms a single dense model while largely preserving expert-level performance, with relative declines of 10.3% for speech and 6.4% for action.The dense model uses 3k fully finetuned steps on basic tasks, whereas SA-MoE uses LoRA for 500 steps across all tasks.Source: F FURTHER DISCUSSION OF SA-MOE · p. 22, Table 7 · p. 22, Table 8 · p. 22

G.1 TIME BLOCK DURATION

• Both time-block configurations complete inference within their respective blocks, while the smaller block naturally reduces interaction latency.At 0.48 seconds, speech experts remain comparable but the action expert degrades, likely because shorter action sequences reduce temporal coherence.Source: §G.1 · p. 23, Table 9 · p. 23

G.2 NUMBER OF EXPERTS

• The two-expert ELLSA performs on par with two three-expert configurations, so the paper adopts two experts for more efficient training.Source: §G.2 · p. 23, Table 10 · p. 24

G.3 SPEECH ENCODER

• Replacing the speech encoder with SPEAR reduces speaking-while-acting gaps from 13.3% to 2.9% on LIBERO LONG and from 17.0% to 7.0% on Web Questions.The authors attribute the remaining degradation primarily to speech-model capacity rather than the architecture itself.Source: §G.3 · p. 24, Table 11 · p. 24

H RESULTS ON CALVIN BENCHMARK

• The CALVIN variant combines a SPEAR-based speech expert with a CALVIN-specific UniVLA action expert under the ABCD→D protocol.It reports success at each task index and the average number of consecutively completed tasks across five-task sequences.Source: H RESULTS ON CALVIN BENCHMARK · p. 24, Table 12 · p. 25
• On CALVIN, SA-MoE performs comparably to and sometimes exceeds modality-specific baselines, supporting plug-in integration of task-specific action experts.Source: H RESULTS ON CALVIN BENCHMARK · p. 24

I FURTHER DETAILED RESULTS

• Context-grounded VQA evaluation may be misjudged because Gemini processes videos at only 1 fps and can miss essential contextual frames.Source: I FURTHER DETAILED RESULTS · p. 25, Table 15 · p. 25

J LIMITATIONS

• ELLSA currently covers only a limited range of natural communication scenarios and has not yet been validated in real-world settings.Backchanneling remains unaddressed, and the reported full-duplex results are within simulated environments.Source: J LIMITATIONS · p. 26

K THE USE OF LARGE LANGUAGE MODELS (LLMS)

• The experiments use LLM-generated training data and model-based evaluation for AlpacaEval and context-grounded VQA.Source: K THE USE OF LARGE LANGUAGE MODELS… · p. 26