PaperlayerView exact revision on arXiv

Source-linked AI summary

JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang

arXiv:2610.12957v1cs.ROcs.AIcs.LG

TL;DR

Embodied agents must translate multimodal observations and requests into executable behavior while adapting to product constraints and changing goals. JoyInside-VLM-Agent trains a product-conditioned multimodal policy with staged learning and token-level evidence routing; its results vary across public benchmarks, while its evaluated proprietary systems lead reported metrics.

  • Problem

    Embodied agents must turn multimodal observations and user requests into executable behavior while respecting product constraints and adapting to changing goals and execution failures.

    Source: §1 · p. 1

  • Method

    The approach combines verified product-conditioned trajectories and staged policy learning with DPER, which routes privileged token-level evidence to modulate reward-derived advantages without changing their signs.

    Source: §1 · p. 3

  • Results

    Public results are mixed despite 77.44 BFCL v4 overall; with domain-specific skill refinement, the evaluated system leads JoyInside-Devices and JoyInside-Robotics baselines on all reported metrics.

    Source: §6 · p. 16

  • Takeaways & Limitations

    Two quadruped case studies illustrate visually grounded search, replanning from execution feedback, and explicit failure reporting.

    Source: §6 · p. 16

    The results do not isolate data construction, reward design, or DPER routing, and proprietary evaluations include domain-specific skill refinement.

    Source: §7 · p. 16

Abstract

from arXiv · show

Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy's parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.

1 Introduction

Embodied agents must translate multimodal observations and user requests into executable behavior while adapting to product constraints, changing goals, and execution failures. JoyInside-VLM-Agent combines product-conditioned trajectories, staged policy learning, and multi-source rewards, with DPER routing privileged evidence at the token level.

1 Introduction

  • The training objective combines reasoning quality, product-constraint adherence, and progress toward the current goal, while distributing trajectory-level credit across tokens remains a challenge.The introduction motivates training data that preserves action–observation correspondence across product variation, goal revisions, and failure recovery.Source: §1 · pp. 1–2
  • DPER routes token-level evidence from a frozen privileged teacher and a privileged student sharing the rollout policy’s parameters, rescaling token advantages without changing their reward-derived signs.Evidence is selected according to disagreement and support for each sampled token; privileged information is used only during training.Source: §1 · p. 3
  • 77.44 BFCL v4 overall, outperforming evaluated baselines; quadruped case studies qualitatively show visually grounded search, feedback-based replanning, and honest failure reporting.The system was evaluated on public tool-use benchmarks and proprietary JoyInside datasets.Source: §1 · p. 3
  • Product-conditioned construction generates verified multimodal trajectories covering product variation, goal revisions, and failure recovery, followed by supervised fine-tuning and multi-expert on-policy distillation.Reinforcement learning then refines the policy using interaction-specific rewards.Source: §1 · pp. 2–3

• Multi-source interaction rewards.

  • Multi-source interaction rewards combine process-quality scores, multi-judge trajectory rankings, and task-queue assessments of goal alignment and cross-turn continuity.The framework is assessed on public benchmarks, proprietary evaluation sets, and qualitative quadruped case studies.Source: §1 · pp. 2–3, • Multi-source interaction rewards · p. 3

2 Related Work

Prior work addresses shared embodied capabilities, interaction-aware behavior, trajectory construction, and staged policy learning. JoyInside-VLM-Agent connects these strands by conditioning a shared policy on product specifications and interaction state, then refining it with interaction-specific rewards.

2.1 Embodied and Context-Aware Interaction

  • Embodied and context-aware systems study cross-platform transfer and grounding behavior in user intent and environmental feedback, while interactive benchmarks make policies and evolving goals explicit.JoyInside-VLM-Agent targets training a shared policy across robotic platforms and smart hardware, conditioned on product roles, capabilities, constraints, and interaction state.Source: §2.1 · p. 3

2.2 Multimodal Trajectory Construction

  • Synthetic trajectory methods provide coverage and validity, while revision trajectories and executable environments expose recovery behavior or environmental transitions.This pipeline couples execution feedback with compositional product specifications and visual observations, covering recovery and adaptation to revised goals.Source: §2.2 · pp. 3–4

2.3 Post-Training and Reward-Guided Optimization

  • Prior post-training combines instruction tuning, on-policy teacher distillation, and reward assessment; DPER joins sign-preserving advantage scaling with dual-source evidence routing.The paper distinguishes reward assessment of reasoning, product constraints, and goal consistency from the separate question of how to use those signals in optimization.Source: §2.3 · p. 4

3 Product-Conditioned Multimodal Data Construction

The data pipeline composes product specifications and executable constraints with visual feedback, then generates and validates closed-loop interaction trajectories. Its outputs include accepted demonstrations, preference pairs, and rule-preserving environments for online rollouts.

3 Product-Conditioned Multimodal Data Construction

  • Product-conditioned trajectory construction couples compositional product specifications, executable constraints, and state-consistent visual feedback to cover recovery from failures and adaptation to revised goals.Figure 2 links product composition and initialization, closed-loop interaction, and trajectory validation and reuse.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Product and Environment Construction

  • Each product specification P comprises identity and interaction style, workflows and recovery rules, permissions and constraints, and tools, skills, and perceptual or action capabilities.Source materials include developer configurations, product descriptions, images or screenshots, tool APIs, operational rules, and domain skills.Source: §3.1 · p. 4
  • Executable backends validate calls and apply state transitions, while compatibility checks match workflows, arguments, capabilities, and embodiment to each composed product.Product constraints inform both the developer prompt and machine-checkable environment predicates.Source: §3.1 · pp. 4–5

3.2 Closed-Loop Trajectory Generation

  • Scenarios pair a user goal and initial hidden state with visual assets; the agent receives the product prompt, interaction history, and current observation, while hidden states and future events remain withheld.The observation may include a user message, visual input, execution result, and error.Source: §3.1 · p. 5, §3.2 · pp. 5–6
  • The sandbox routes dialogue and action outputs, applies execution results and state changes, and produces updated visual feedback; dynamic events introduce goal revisions, failures, and environmental changes that require a response.The agent may update its plan, seek clarification, retry, or terminate after observing consequences.Source: §3.2 · p. 6

3.3 Trajectory Validation and Data Outputs

  • Accepted demonstrations pass hard execution checks and semantic review; matched audited violations form preference pairs, while executable product rules and validators support online rollouts.The supervised set includes appropriate refusals and compliant recovery, but excludes agent-side rule violations even if later corrected.Source: §3.3 · p. 6

4 Method

The shared policy is trained in product-conditioned stages, moving from supervised interaction behavior through distillation to reward-based refinement. Supervision covers multimodal demonstrations and assistant outputs, with observations and tool results serving as context.

4 Method

  • Product-conditioned training proceeds through supervised fine-tuning, multi-teacher distillation, and DPER refinement; deployment uses a single student.The policy is shared across robots and smart hardware.Source: §4 · p. 6, Figure 3 · p. 7

4.1 Product-Conditioned Supervised Fine-Tuning

  • Supervised fine-tuning combines general instructions, tool interactions, image-conditioned decisions, execution feedback, and verified sandbox trajectories.Consistent message roles encode the protocol, product specification, requests, observations, and execution results.Source: §4.1 · p. 7
  • Training minimizes cross-entropy over valid assistant-token positions, while user messages, observations, and tool returns remain conditioning inputs.Task queues, when present, are supervised as assistant outputs.Source: §4.1 · p. 7
  • At each ReAct decision, the policy emits reasoning and either a tool call or direct response, with an updated task queue when required.Completion of a pending tool action must be verified from execution feedback, not inferred from the call or response itself.Source: §4.1 · pp. 7–8

4.2 Multi-Teacher On-Policy Distillation

On-policy distillation transfers complementary text and vision capabilities into the student using its own generated prefixes. Capability-matched teachers supervise selected decisions before DPER training.

4.2 Multi-Teacher On-Policy Distillation

  • Text teachers supervise planning, tool selection, argument construction, and feedback interpretation; vision teachers supervise decisions requiring image evidence.Each teacher receives product constraints and the student-generated prefix, plus visual observations when needed.Source: §4.2 · p. 8
  • The objective minimizes reverse-KL divergence from the student to the capability-matched teacher on aligned token spaces.Teacher distributions are evaluated on the student’s sampled context with gradients stopped through the teacher.Source: §4.2 · p. 8
  • Distillation selects assistant positions and conditions both distributions on the same student-generated prefix.Rollout batches and teacher targets are fixed during each update; updated students collect subsequent batches, and teacher assignment remains training-only.Source: §4.2 · p. 8

4.3 DPER: Token-Level Evidence Routing

DPER combines group-relative trajectory rewards with token-level evidence routed from privileged teacher and student branches. The routed evidence scales token advantages while retaining their reward-derived signs.

4.3 DPER: Token-Level Evidence Routing

  • DPER uses group-relative reward comparisons to set token-advantage signs and selects privileged evidence to scale them, without separate KL or JS distillation losses.It combines sign-preserving advantage modulation with token-level source selection.Source: §4.3 · p. 8, §4.3.3 · p. 9

4.3.1 Dual-Source Privileged Scoring

  • A frozen teacher and a parameter-sharing privileged student score sampled tokens on the same rollout using a training-only reference.The generating policy never receives that reference, and fixed trajectories keep both scores tied to the same decisions and observed consequences.Source: §4.3.1 · p. 8

4.3.2 Token-Level Source Selection

  • Token-level routing uses batch-relative disagreement and support to select the privileged teacher or student across four cases.The cases distinguish low or high disagreement and, when disagreement is high, which source better supports the sampled token; probability support alone does not establish executability or goal consistency.Source: §4.3.2 · p. 9

4.3.3 Evidence-Weighted Policy Optimization

  • The selected evidence rescales group-relative advantages with a clipped, mixed multiplier that preserves each nonzero advantage’s sign.Evidence strength and clipping bound the effect; zero strength or zero clipping radius recovers the original trajectory advantage.Source: §4.3.3 · p. 9

4.4 Multi-Source Interaction Rewards

The reward combines process-quality evaluation, relative candidate rankings, and task-queue consistency, with format and length factors shaping the normal-path score. Each component evaluates a distinct aspect of behavior, from reasoning quality to product constraints and evolving goals.

4.4 Multi-Source Interaction Rewards

  • Normal-path rewards combine GPRM with available Judge and Queue scores, renormalize remaining weights, and apply format and length factors.Missing optional scores are excluded; unavailable GPRM triggers a fixed fallback and skips Judge and Queue evaluation. Required-queue omission is capped so renormalization alone cannot exceed the reward for a valid zero Queue score.Source: §4.4 · p. 10

4.4.1 Generative Process Evaluation

  • GPRM evaluates each reasoning prefix with a verification rationale and quality grade, then averages expected grade quality across evaluated steps.Its annotated training sequences include the task, reasoning prefix, rationale, and grade; these labels supervise process quality independently of candidate rankings.Source: §4.4.1 · pp. 10–11

4.4.2 Multi-Judge Ranking

  • Judges rank the same candidate group with ties; aggregated ballot rankings produce relative preferences that are mapped to rewards.If every candidate triggers a deterministic hard failure, the method skips relative ranking and DPER policy updates for that group.Source: §4.4.2 · p. 11

4.4.3 Task-Queue Consistency

  • TaskQueue scores initial-turn intent coverage and, on feedback turns, alignment with updated task requirements and continuity with the verified prior queue.Continuity requires exact batch matches in position, while the two terms balance adaptation to revised goals against preservation of pending task order.Source: §4.4.3 · pp. 11–12

5 Experiments

The experiments evaluate JoyInside-VLM-Agent on public and proprietary benchmarks, track training dynamics, and examine two quadruped interactions. Results vary across public benchmarks, while the evaluated system leads on the proprietary sets and the case studies illustrate feedback-guided behavior.

5.1 Experimental Setup

  • The evaluation uses approximately 120K training examples and covers four public benchmarks, two proprietary datasets, and both tool-use and visually grounded robotics tasks.The proprietary datasets focus respectively on multi-turn device dialogue and control, and long-horizon visually grounded robot reasoning and control.Source: §5.1 · p. 12
  • The evaluated model starts from Qwen3.5-35B-A3B and undergoes supervised fine-tuning, multi-teacher distillation, and DPER; privileged references are restricted to training.Domain skills are refined on the proprietary sets while model parameters remain fixed, so those results reflect the policy and refined skills together.Source: §5.1 · p. 12

5.2 Training Dynamics

  • DPER’s maximum reward rises rapidly and remains above baselines for most plotted steps, but this highest sampled reward does not establish performance across all rollouts.The figure compares actor entropy, response length, and maximum reward for DPER, GRPO, RLSD, and DOPD over the first 300 steps.Source: §5.2 · p. 13, Figure 4 · p. 13

5.3 Benchmark Results

  • 77.44 BFCL v4 overall score exceeds both evaluated models; public results are mixed, while the evaluated system leads on all reported metrics for JoyInside-Devices and JoyInside-Robotics.The proprietary-set results include domain-specific skill refinement; no trajectory exact-match accuracy is reported for those sets.Source: Table 1 · p. 14, §5.3 · pp. 14–15
  • On ToolBench, the model leads in trajectory accuracy (23.5%) and four of five tool-use and response metrics; on ToolACE, it leads in Plan.EM (96.86) and Act.EM (89.06), but trails on trajectory accuracy and other metrics.On MSAgent, its trajectory accuracy is 90.39%, between the larger model (92.41%) and base model (88.46%).Source: §5.3 · p. 14, Table 1 · p. 14

5.4 Real-World Case Studies

  • In one quadruped case, visual attribute matching leads from search to approach and greeting; in the other, a failed approach prompts renewed search and failure reporting after six steps without reacquisition.The examples are qualitative and illustrate visually grounded decisions, feedback-driven replanning, and explicit reporting of success or failure.Source: §5.4 · p. 15, Figure 5 · p. 15, Figure 6 · p. 15

6 Conclusion

JoyInside-VLM-Agent combines product-conditioned trajectories and staged learning to adapt behavior to robotic platforms and smart hardware. It scores 77.44 on BFCL v4, has mixed results on other public benchmarks, and leads on all reported metrics for two proprietary sets.

6 Conclusion

  • The framework combines verified product-conditioned trajectories with staged policy learning and multi-source rewards for reasoning quality, product-constraint adherence, and alignment with evolving goals.DPER routes token-level evidence between privileged teacher and student policies to scale token advantages while preserving their reward-derived signs.Source: §6 · pp. 15–16
  • 77.44 BFCL v4 overall score; results on other public benchmarks are mixed, while the evaluated system leads on all reported metrics for JoyInside-Devices and JoyInside-Robotics.The proprietary results include domain-specific skill refinement.Source: §6 · p. 16
  • Two quadruped case studies illustrate visually grounded search, replanning from execution feedback, and explicit failure reporting.Source: §6 · p. 16

7 Limitations and Future Work

The authors identify limits in generalization evidence and in attributing results to individual training components. They also note that reward signals are imperfect proxies for task success and outline further evaluations and calibration work.

7 Limitations and Future Work

  • Generalization across a broader range of product configurations and operating environments still requires systematic evaluation.Future work includes longer real-world interactions and more diverse benchmarks spanning multimodal understanding, reasoning, planning, and tool use.Source: §7 · p. 16
  • Current results do not isolate data-construction, reward-design, and DPER-routing contributions, and proprietary evaluations also include domain-specific skill refinement.The authors call for controlled ablations to separate these effects.Source: §7 · p. 16
  • Model-based rewards and embedding similarity are imperfect task-success proxies, and preserving reward-derived advantage signs does not resolve errors in the underlying reward.Planned work includes execution-grounded reward calibration and robustness to incomplete or noisy feedback.Source: §7 · p. 16

A Supplementary Formulations

The appendix specifies the policy and training objectives, then details how task-queue rewards represent intent, execution continuity, and score availability.

A.1 Policy Decisions and Sandbox Transitions

  • The sandbox policy conditions on product instructions, history, observations, and prior task state; backend execution updates hidden state and may produce checked visual feedback.Hidden states and event schedules remain unavailable to the policy, and some branches produce no visual feedback.Source: §A.1 · p. 19

A.2 Supervised Learning Objective

  • Supervised learning minimizes assistant-token negative log-likelihood, with more supervised tokens contributing proportionally more per trajectory.Assistant reasoning, responses, actions, and present task queues are targets; user messages, observations, and tool returns are not.Source: §A.2 · p. 19

A.3 DPER Policy Objective

  • DPER updates the unprivileged policy using evidence-weighted token advantages and clipped policy ratios, while holding rollout-derived quantities fixed during each batch update.Gradients update only the unprivileged policy; identical-reward rollout groups receive zero base advantages.Source: §A.3 · pp. 19–20

A.4 GPRM Grade Normalization

  • GPRM normalizes probabilities over four grade tokens at a fixed judgment position, then averages expected grade rewards across reasoning steps.The grade weights for A through D are (1, 2, 1/3, 0).Source: §A.4 · p. 20

A.5 Reward Availability and Scoring Paths

  • When GPRM input or scoring is unavailable, the reward path can fall back to format and length scores, while valid zero scores remain in normal aggregation.If no process steps are evaluable, GPRM, Judge, and Queue calls are skipped; group-level hard-failure exclusion still takes precedence over policy updates.Source: §A.5 · p. 20

A.6 Task-Queue Scoring

  • Initial queue scoring compares reconstructed queue intent with the request, measuring intent coverage rather than exact queue matching or continuity.The inverse rewrite preserves listed tasks, arguments, batch structure, and ordering constraints; Table 2 illustrates its prompt with the current queue substituted for {task_queue}.Source: §A.6 · p. 20, Table 2 · p. 21
  • On feedback turns, the semantic score compares the current queue with requirements updated from observation history, while continuity checks the execution-conditioned unfinished queue against ordered task batches.Execution status comes from feedback, and exact continuity matching allows neither reordering nor differing task entries.Source: §A.6 · p. 21
  • The trajectory-level TaskQueue reward averages evaluated turn scores, including zeros; unavailable queue scores are omitted and remaining reward weights are renormalized.If no turn is evaluable, normal content aggregation omits the Queue component.Source: §A.6 · pp. 20–21

B Product Prompt and Interaction Templates

The appendix describes a configurable product developer prompt and how it is serialized alongside shared instructions and ReAct-style assistant outputs.

B Product Prompt and Interaction Templates

  • The product developer prompt template specifies role, input, output, capabilities, and execution policies.Table 3 presents the structural template with configurable role, interaction interface, capabilities, and execution policies.Source: B Product Prompt and Interaction… · p. 22, Table 3 · p. 22
  • The prompt maps its role and communication interface to IP, its capability registry to SP, and execution policies to WP and CP.Configuration-specific placeholders include the instruction source and output recipient; concrete tool definitions and policy instances are omitted.Source: B Product Prompt and Interaction… · p. 22

B.1 Chat-Template Serialization

  • The product specification is serialized as text in a distinct developer message, separating it from shared system instructions without changing message delimiters.Table 4 shows the chat-template branch and a representative input sequence.Source: §B.1 · p. 23, Table 4 · p. 23
  • Tool results appear inside tool_response blocks within user messages, preserving their source through the serialization tags.The representative input omits optional tool definitions, visual-token expansion, and interaction history.Source: §B.1 · p. 23, Table 4 · p. 23

B.2 ReAct Assistant Output Template

  • A ReAct assistant message combines a shared prefix with one of two continuations, and includes the task block only when the protocol requires it.Table 5 illustrates the two continuations and the conditional task block.Source: §B.2 · p. 23, Table 5 · p. 23
  • Execution feedback conditions the next policy decision, whereas a direct response addresses the user without issuing a tool call.Source: §B.2 · p. 23
Loading 2610.12957v1…