Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agents

1 INTRODUCTION

• The paper models dLLM action bias as task-blind: contextually salient actions receive state–action-dependent probability inflation that does not depend on the task.The bias is apparent when an agent repeats an action after it fails.Source: §1 · pp. 1–2
• At failure states, adaptive masked decoding can commit the failed action as a confident fill before the deferred decision confronts failure feedback.Resampling still produces plausible alternatives, but the distribution concentrates on the contextually salient failed action.Source: §1 · pp. 1–2
• Reflect Reverse scores proposed actions by the reverse conditional, which masked dLLMs can evaluate by masking task tokens and denoising.The method samples candidates, scores them by reverse task evidence, then samples the next action proportionally to those scores.Source: §1 · pp. 2–3
• Under the stated assumptions, the reverse score equals the faithful model’s task posterior because the task-independent corruption factor cancels across tasks.The reverse conditional asks whether the state and candidate action explain the task.Source: §1 · p. 2
• Across four embodied benchmarks, Reflect Reverse improves task success and progression rates over forward-scoring baselines.The experiments cover ALFWorld, ScienceWorld, BabyAI, and Jericho with LLaDA and iLLaDA.Source: §1 · p. 3

2.1 MASKED DIFFUSION LANGUAGE MODELS

• Masked dLLMs progressively mask sequence tokens, then predict masked positions simultaneously from the unmasked context during reverse steps.Any span can be scored against another by masking one and clamping the other across denoising steps.Source: §2.1 · p. 3

2.2 TASK-BLIND CORRUPTION

• Task-blind corruption assumes the observed action conditional differs from a faithful conditional by a state–action factor that does not depend on the task.The assumption’s substance is task-independence: at fixed states, the measured log-ratio varies with state and action but is flat across goals.Source: §2.2 · p. 3

3.1 REFLECT REVERSE

• Reflect Reverse first samples and deduplicates candidate actions from the forward conditional using chain-of-thought proposals.The resulting candidate set is denoted A(s).Source: §3.1 · p. 3
• It reflects by estimating each candidate’s reverse conditional through random masking of task tokens, denoising, and reading masked-token log-likelihoods.Each masked estimate contributes to a Monte Carlo estimate of the reverse score.Source: §3.1 · p. 3

3. Act. Sample the next action from the corrected policy

• Reflect Reverse is training-free and adds |A(s)| × M denoising passes per decision.The method samples the next action from the corrected policy.Source: §3 · p. 4

3.2 INVARIANCE OF THE REVERSE SCORE TO TASK-BLIND CORRUPTION

• Under the stated assumptions, the reverse score recovers the faithful task posterior up to a positive task weight shared across candidate actions.Its pointwise identity preserves the faithful task-evidence ratio's candidate ordering and ties, and softmax normalization removes the shared weight at any temperature.Source: §3.2 · p. 4
• The guarantee compares candidates on the proposed set, which may omit actions in the faithful model’s support; the corrected policy matches the faithful conditional only when the marginal is flat across candidates.The unconditional guarantee is recovery of the task-evidence ratio's ordering and ties, not the normalized faithful conditional.Source: §3.2 · p. 4

3.3 HOW LARGE IS THE RESIDUAL BIAS?

• At β = 1, dTV(ρ, µA) ≤ tanh(Λm/4) ≤ tanh(Λtask/4), while forward-selection bias grows with the corruption gap ∆2 toward 1 − γ.The reverse bound is independent of corruption; when corruption is flat, forward selection is exact and the corrected policy retains only the m-bias.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation covers four embodied benchmarks, two dLLM backbones, two temperatures per backbone, and three random seeds, reporting success rate and progress rate.Episodes are limited to 30 interaction steps; each step generates 20 candidate actions.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

• On Jericho, progress improves for both backbones at both temperatures, although success decreases at each backbone’s lower evaluated temperature.The two PR decreases across all configurations occur in LLaDA settings and are each smaller than one percentage point.Source: §4.2 · p. 6
• Reflect Reverse has higher mean SR in 13 of 16 evaluation groups and higher mean PR in 14, with SR gains on three benchmarks and PR gains in all eight iLLaDA configurations.The reported SR gains occur on ALFWorld, ScienceWorld, and BabyAI for both backbones at both evaluated temperatures.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3.1 RERANK ON FORWARD SCORE

• Reflect Reverse generally outperforms Reflect Forward, with higher SR in 12 of 16 configurations and higher PR in 14.For iLLaDA, reverse scoring improves SR in seven of eight configurations and PR in all eight.Source: Table 1 · p. 7, §4.3.1 · pp. 7–8

4.3.2 RETRY STATE LOOPS

• Reflect Reverse achieves the lowest mean retry percentage in all 16 settings, while lower temperatures yield higher mean retry percentages in 21 of 24 pairwise comparisons.Retry percentage is the total number of retries divided by the interaction budget.Source: §4.3.2 · p. 8
• Fewer retries do not necessarily translate into better task performance, which also depends on generating correct, task-appropriate actions.For example, on ALFWorld at T = 0.9, iLLaDA with Reflect Forward has fewer retries than One Pass but lower success and progress rates.Source: §4.3.2 · p. 8

• Adaptive decoding can defer high-entropy reasoning forks until committed context has already resolved them, a trap the paper characterizes as sharper in retry loops.At a failure state, the failed action is already present in context and can be copied when the deferred action is filled.Source: §5 · p. 8
• Recent evaluations report that parallel-decoding latency gains do not transfer to dLLM agent competence, with retry loops a dominant failure mode.The paper attributes this failure to adaptive decoding bridging failure feedback rather than confronting it.Source: §5 · p. 8
• Unlike reranking methods that use external verifiers, voting, or domain-conditional baselines, Reflect Reverse scores candidates with the model’s own reverse conditional.The paper distinguishes this training-free decoding-time correction from work using a conditional PMI objective as a training objective.Source: §5 · p. 9

6 CONCLUSION AND FUTURE WORKS

• The conclusion traces retry loops to adaptive decoding postponing a failure-state decision until the committed context favors repeating the failed action.It formalizes the resulting action-distribution distortion as a task-blind state–action factor and reports verifying it through goal-swap interventions.Source: §6 · p. 9
• Reflect Reverse is training-free, needs no learned critic, value function, or environment rollouts, and adds a few parallel denoising passes per candidate.The conclusion reports improved success and progress over forward-scoring baselines on four embodied benchmarks and two backbones.Source: §6 · p. 9
• Because proposals still come from the corrupted forward conditional, the faithful model’s support may go uncovered; future work proposes extending the invariance to proposal selection.The paper also suggests fine-tuning at failure states or regularizing forward–reverse consistency to address the training-artifact pathology.Source: §6 · p. 10

A TASK CORRUPTION VERIFICATION

• For each retry state, the evaluation held the repeated action and an admissible alternative fixed while replacing the original task with 10 random descriptions.LLaDA action scores used a Monte Carlo diffusion variational estimate; Qwen scores used summed autoregressive token log likelihood.Source: A TASK CORRUPTION VERIFICATION · p. 14
• Smaller σ indicates less variation in action preference across replacement goals.The original task is excluded when computing σ.Source: A TASK CORRUPTION VERIFICATION · p. 14
• σ was 1.01 nats for LLaDA and 4.71 nats for Qwen, with preference flip rates of 6.6% and 44.4%, respectively.The evaluation used 2,080 task-replacement comparisons per model.Source: A TASK CORRUPTION VERIFICATION · p. 14

B REFLECT REVERSE ALGORITHM

• Reflect Reverse proposes and deduplicates ReAct action candidates, then reflects by masking task-token positions and conditioning the dLLM on each candidate.The procedure accumulates reflection scores across M rollouts.Source: B REFLECT REVERSE ALGORITHM · p. 15

C PROOF AND THEORY

• Assumption C.1 requires the model’s forward and reverse conditionals to satisfy Bayes’ rule with respect to the task prior p(u | s).The condition applies for every action and task in the prior’s support.Source: C PROOF AND THEORY · p. 15
• The paper treats this conditional coherence as an idealization that licenses interpreting the reverse score as a posterior.The text compares it to idealizations used when composing a single model’s conditionals.Source: C PROOF AND THEORY · p. 15
• A stronger coherence assumption with the task posterior removes the task prior from the proposition entirely.Under that variant, the proposition uses a task-dependent reweighting instead.Source: C PROOF AND THEORY · p. 15

C.1 PROOF OF THE RECOVERY PROPOSITION

• The state–action corruption factor cancels exactly from the reverse conditional under Assumptions 2.1 and C.1.The resulting score is a faithful task posterior under a positive task weight that does not depend on the candidate action.Source: §C.1 · p. 16
• The recovered posterior preserves the faithful task-evidence ratio’s candidate ordering, ties, and top-k set.The margins are those of the ratio, not necessarily those of the faithful conditional itself.Source: §C.1 · p. 16
• The corrected policy is the softmax of the faithful task-evidence ratio at any positive temperature; its limits are argmax selection and uniform sampling.These limits correspond to β → ∞ and β → 0, respectively.Source: §C.1 · p. 17

C.2 RESIDUAL BIAS OF THE CORRECTED POLICY

• The corrected policy is not generally the softmax of the faithful conditional because candidate-dependent task-marginal likelihood remains after normalization.The policies coincide when that marginal is constant across candidates or in the argmax limit.Source: §C.1 · p. 17
• The appendix bounds residual policy bias using the spread of the faithful conditional and task-marginal likelihood across candidates.It also states that Monte Carlo estimation error adds to the bound and that this contribution vanishes as the estimator becomes consistent.Source: §C.2 · p. 18

D HYPERPARAMETERS

• ScienceWorld and Jericho use more Monte Carlo sampling steps for Reflect Reverse because task descriptions are longer than candidate actions.The longer descriptions require more tokens to evaluate during reverse reflection.Source: D HYPERPARAMETERS · p. 19

E SUPPLEMENTARY EXPERIMENT RESULTS

• Tables 3 and 4 provide performance results across benchmarks, model variants, methods, and sampling temperatures.The appendix describes these tables as the detailed results omitted from the main paper for space.Source: E SUPPLEMENTARY EXPERIMENT RESULTS · p. 20
• Table 3 reports LLaDA-8B-Instruct success rate and progress rate as mean ± sample standard deviation.Boldface marks the best method within each benchmark-temperature category, including Mean, with ties included.Source: Table 3 · p. 20
• Table 4 reports iLLaDA-8B-Instruct success rate and progress rate as mean ± sample standard deviation.Boldface marks the best method within each benchmark-temperature category, including Mean, with ties included.Source: Table 4 · p. 21

F RETRY STATE EXAMPLES

• The two ALFWorld examples represent retry patterns also frequently observed across other multi-turn benchmarks.Source: F RETRY STATE EXAMPLES · p. 22, Table 5 · p. 22, Table 6 · p. 22
• Table 5 presents a single-action repetition loop, marked in red including its initial action.The table identifies A as the action and O as its resulting observation.Source: Table 5 · p. 22
• Table 6 presents an alternating-action loop, marked in red including the pair that starts it.Source: Table 6 · p. 22
• For the task “Put a clean egg in microwave,” the example shows the agent putting the egg back in the fridge.Source: F RETRY STATE EXAMPLES · p. 23
• One illustrated sequence repeatedly closes and opens the fridge, forming an alternating-action pattern.Source: F RETRY STATE EXAMPLES · p. 23