PaperlayerView exact revision on arXiv

Source-linked AI summary

Potential-Guided Particle Steering for Negation-Constrained Dexterous Grasping

Geonho Kim, SooGon Kim, Jongmin Lee

arXiv:2609.00555v1cs.ROcs.CV

TL;DR

Language-driven dexterous grasping is trained and evaluated mostly with positive instructions, leaving negative constraints such as avoiding a part under-addressed. This paper steers grasp generation at inference time without negation-specific training examples; on NegGrasp, it lowers the strongest baseline’s violation rate from 57.9% to 17.2%.

  • Problem

    Language-driven dexterous grasping is trained and evaluated almost exclusively on positive instructions, leaving performance under negative constraints such as avoiding a part insufficiently addressed.

    Source: §1 · pp. 1–2

  • Method

    The inference-time framework combines classifier-free guidance and Sequential Monte Carlo to steer toward the instructed part and away from a forbidden region without negative training examples.

    Source: §1 · p. 2

  • Results

    17.2% violation rate, down from 57.9% for the strongest baseline on NegGrasp.

    Source: Constraint compliance · p. 6

  • Takeaways & Limitations

    The results show that inference-time steering can reduce forbidden-part violations without negative training examples.

    Source: §1 · p. 2, Constraint compliance · p. 6

    The method derives steering signals at inference time without access to ground-truth part labels.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Language-driven dexterous grasp models, such as DextER, perform well when instructions specify where to grasp, but we find they fail systematically when an instruction also specifies where not to grasp (e.g., "grasp the handle but avoid the body"). Existing training corpora, DexGYSNet among them, contain virtually no avoidance instructions, and collecting examples for every possible constraint is impractical. Moreover, because every part mentioned during training denotes a contact target, models may interpret a forbidden part as another region to grasp rather than one to avoid. We therefore introduce an inference-time framework for negation-constrained dexterous grasping that requires no negation-specific training examples. Combining Sequential Monte Carlo with classifier-free guidance, our method guides sampling toward the instructed part while pruning candidates headed for the forbidden region, without any negation examples during training. A frozen 3D part-grounding model localizes the forbidden region from the language instruction. To evaluate this setting, we construct NegGrasp, a benchmark of paired positive/negative instructions with constraint-aware metrics that credit a grasp only if it both accomplishes the task and respects the stated constraint. On NegGrasp, our method reduces the violation rate of the strongest baseline from 57.9% to 17.2% while improving both constraint-aware and physical success.

1 Introduction

Language-driven dexterous grasping models are trained mainly on positive instructions and often fail to avoid forbidden regions. The paper proposes inference-time steering without negative training examples and introduces a benchmark for evaluating constraint-aware grasps.

1 Introduction

  • DexGYSNet (Wei et al. 2024) contains essentially no avoidance phrasing; given instructions pairing a positive target with an explicit negative clause, DextER (Lee, Park, and Cho 2026), the strongest baseline, frequently ignores the clause and contacts the forbidden region.Source: §1 · p. 2
  • Diffusion provides a revisable continuous grasp candidate at each sampling step, unlike DextER’s decoder, which freezes emitted tokens and only approximately determines the decoded grasp.This makes candidates spatially checkable during generation, enabling sequential Monte Carlo to prune predicted violations; annotating every task- and user-specific avoidance constraint in advance is impractical.Source: §1 · p. 2
  • The method retains DextER’s frozen Qwen (Qwen Team 2024) and PartField (Liu et al. 2025) conditioning backbones, trains only the diffusion decoder, and uses frozen Find3D (Ma, Yue, and Gkioxari 2025) to ground forbidden parts.Find3D is used as an open-vocabulary 3D part-grounding plug-in during steering.Source: §1 · p. 2
  • The framework steers a flow-matching grasp diffusion transformer with classifier-free guidance and sequential Monte Carlo, using a marginal-preserving stochastic sampler to maintain particle diversity without negative training examples.Classifier-free guidance nudges candidates away from forbidden regions, while sequential Monte Carlo prunes candidates predicted to violate constraints.Source: §1 · p. 2
  • NegGrasp pairs positive and negative instructions and provides constraint-aware metrics that credit grasps only when they accomplish the task and respect the stated constraint.The benchmark evaluates the setting where a grasp must both perform the instructed task and avoid the forbidden region.Source: §1 · p. 2

2 Related Work

Prior work establishes language-conditioned dexterous grasping and inference-time steering, but does not address freely specified avoidance constraints on finished dexterous grasps. This work adapts negative-prompt guidance and particle steering to fill that gap.

2.1 Dexterous Grasp Generation

  • Existing language-conditioned grasp models target desired contact only, leaving avoidance-style instructions unaddressed.DexGYS introduced DexGYSNet, while DextER uses an autoregressive contact-token formulation to predict hand-link contacts before decoding a grasp (Wei et al. 2024; Lee, Park, and Cho 2026).Source: §2.1 · p. 2

Negative prompting in text-to-image diffusion.

Negation in dexterous grasping.

  • OVAL-Grasp and AffordDex address grasp avoidance, but neither provides freely specified, test-time constraints for full dexterous grasp generation (Tong et al. 2025; Zhao et al. 2026).OVAL-Grasp uses image heatmaps for parallel-jaw grasping, whereas AffordDex learns negative affordances during reinforcement learning; this work instead changes the constraint per instruction at test time.Source: Negation in dexterous grasping · pp. 2–3

2.3 Steering Diffusion with SMC and CFG

  • The steering stack combines SMC resampling with a max-potential Feynman-Kac potential, CFG-guided proposals, a corrector, and marginal-preserving churn.The max-potential variant retains credit for earlier safe checkpoints (Singhal et al. 2025); correctors connect drift tilting and particle reweighting (Skreta et al. 2025), while Feynman-Kac-Flow derives the sampler underlying churn (Mark et al. 2025).Source: §2.3 · p. 3
  • Unlike twisted proposals or repulsion without resampling, the method keeps CFG-guided proposals separate and uses marginal-preserving churn alongside resampling.Twisted Diffusion Sampler incorporates the potential into transitions (Wu et al. 2023), while Particle Guidance uses particle-dependent repulsion without resampling.Source: §2.3 · p. 3
  • Although SMC can steer autoregressive models, partial contact tokens cannot commit to or revise the finished-grasp constraint, motivating diffusion-transformer steering.DextER's contact tokens make potentials spatially meaningful, but the constraint applies to the completed grasp (Lew et al. 2023).Source: §2.3 · p. 3

3 Method

The method generates grasps that reach a positive target while avoiding a forbidden region, despite training without avoidance supervision. It keeps the trained weights unchanged and handles negation during inference through two-stage generation and steering.

3.1 Grasp Generator: Flow-Matching DiT

  • A 12-block flow-matching DiT jointly refines grasp poses and eight contact slots, conditioned through frozen Qwen and PartField encoders.The pose includes translation, rotation, and 22 ShadowHand joint angles; only the DiT and two conditioning-projection layers are trained.Source: §3.1 · p. 3

3.2 Instruction Parsing and Region Grounding

  • Gemma parses instructions into a positive-only rewrite and a part vocabulary flagging forbidden names; Find3D labels the point cloud to define the forbidden region.The parser also forms an affirmative prompt naming the forbidden part for guidance; grounding uses the full vocabulary and requires no ground-truth part labels.Source: §3.2 · p. 3

3.3 Steering by Classifier-Free Guidance

  • When a look-ahead contact estimate heads toward the forbidden region, guidance contrasts positive and affirmative forbidden-part prompts using only the orthogonal component of their contact-velocity difference.Guidance is gated off once the candidate clears the region and can be limited to a step window, avoiding cancellation of the positive target.Source: §3.3 · p. 4

A look-ahead potential.

  • The look-ahead potential scores contacts higher when they are clear of the forbidden region and already touching the object, ranking candidates rather than imposing a hard constraint.Its distances are measured from estimated contact coordinates to the forbidden region and object surface.Source: A look-ahead potential · p. 5

3.4 Steering by Sequential Monte Carlo

  • At scheduled checkpoints, sequential Monte Carlo resamples N trajectories by importance weights, pruning those predicted to violate the constraint and replicating safer candidates.The implementation uses two resampling steps; running-maximum potential scores preserve credit for earlier safe estimates.Source: A look-ahead potential · p. 5

A look-ahead potential.

  • Stochastic churn replaces deterministic Euler updates at every denoising step so resampled duplicate particles can diverge, while preserving each particle’s marginal target distribution.The churn strength η ≥ 0 recovers the deterministic ODE at η = 0.Source: A look-ahead potential · p. 5
  • Final selection checks reconstructed hand meshes against a fixed forbidden-region margin, returns the qualifying grasp closest to the object surface, and relaxes the margin if none qualifies.The geometric check is reserved for final selection because the contact-based potential is only a proxy.Source: A look-ahead potential · p. 5

4 Experiments

The experiments introduce NegGrasp, a benchmark for negation-constrained grasping, and evaluate a training-free inference-time steering method. The full method achieves the best constraint-aware success and reduces violations, while ablations show complementary roles for guidance and particle resampling.

4.1 NegGrasp dataset

  • NegGrasp contains 6,928 instructions over 672 scenes and 20 categories, with 2,793 test instructions and object-level separation from training.Contested parts are paired with avoiding grasps, and instructions vary tone, clause order, and avoidance expressions; methods must ground the forbidden part from language at inference.Source: §4.1 · p. 6

4.2 Experimental Setup

  • The 130M-parameter generator is trained by rectified-flow matching on DexGYSNet positive instructions alone, using frozen Qwen2.5-0.5B and PartField encoders.Inference uses 16 particles over 25 denoising steps, guidance weight w=4, a 50 mm contact gate, resampling at steps 10 and 20, and fixed steering hyperparameters.Source: Implementation Details · p. 6

Implementation Details.

  • CSR requires forbidden-part avoidance, instructed-part contact, and object contact; TSR additionally requires Isaac Gym stability and no violation.The study also reports violation rate, contamination, and median clearance, using a 4 mm contact threshold and macro averages over 27 category-part cells.Source: Evaluation Metrics · p. 6

4.3 Main Results

4.4 Ablation Study

Deterministic SMC trades particle collapse for physics.

4.5 Analysis

  • At n=1, early resampling {5} reaches 36.2 TSR with the sweep's lowest CSR, while late {20} reaches 63.1 CSR with the worst TSR of 28.5; n=2 is near the top of both.Churn preserves all 16 unique particles, allowing resampling schedules to vary without risking collapse; violations stay near CFG levels through n=2 and then drift upward.Source: Table 3 · p. 7, Resampling schedule · p. 7
  • At w=4, violation is lowest and TSR peaks; at w=6 and w=16, TSR falls to 34.8 and 31.1 as stronger guidance pushes samples away from the learned distribution.Clearance and CSR continue rising through w=16 while violation drifts upward; the contact gate fires only near the forbidden region.Source: Table 3 · p. 7, CFG strength · p. 7

5 Conclusion

Language-driven dexterous grasping fails when instructions specify forbidden contact, and this work addresses the problem entirely at inference time. On NegGrasp, the method reduces violations while improving constraint-aware and physical success.

5 Conclusion

  • On NegGrasp, the method cuts the strongest baseline’s violation rate to less than a third while improving constraint-aware and physical success.NegGrasp is a benchmark for grasping with forbidden-contact instructions.Source: §5 · p. 7
  • The inference-time method steers a flow-matching grasp DiT with contact-gated CFG and churn-diversified SMC resampling, without negative training examples.It addresses the systematic failure of language-driven dexterous grasping when instructions say what not to touch.Source: §5 · p. 7

Supplementary Material … B.2 Aggregation: Macro and Micro

The supplementary material records Section 3 notation and formalizes the evaluation metrics and aggregation used for the imbalanced test split. It distinguishes macro scores, which the main paper reports, from appendix-only micro scores.

A Notation

  • Table 4 summarizes notation used in Section 3 and Algorithm 1.The table is identified as the notation reference for Section 3.Source: A Notation · p. 9, Table 4 · p. 9

B Metrics

  • The appendix formalizes Section 4.2 metrics and specifies aggregation across the imbalanced test split.Source: B Metrics · p. 9

B.1 Formal Definitions

  • Contact tests use a 4 mm threshold; violation means any hand point reaches the forbidden part, while contamination is the fraction of contacted object points on it.Contamination is defined as 0 when the hand contacts no object points.Source: §B.1 · p. 9
  • Clearance is the minimum distance from the hand to the forbidden part and is reported as a median to limit the influence of a single distant hand.Source: §B.1 · p. 9
  • Constraint-aware success uses target parts contacted by the reference grasp, excluding the forbidden part, and defines indicators for target hit and object contact.Instructions are generated from a reference compliant grasp, so its contacted parts geometrically realize the instructed part.Source: §B.1 · p. 9
  • Task success is the product of stability and one minus violation, with stability measured under DextER’s Isaac Gym simulation protocol.Source: §B.1 · p. 9

B.2 Aggregation: Macro and Micro

  • Macro averages scores within each of 27 category–forbidden-part cells before averaging cells; micro averages cases directly, over the shared 2,793 grounded cases.For clearance, the inner aggregation is a median—within each cell for macro and over all cases for micro—while macro still averages cells by their mean.Source: §B.2 · p. 9
  • The test split is heavily imbalanced, so the paper reports macro scores; micro scores appear only in this appendix and are labeled as such.Mugs account for 1,794 of 2,793 evaluated cases (64%), all in the same category–forbidden-part cell.Source: §B.2 · p. 10

B.3 Micro-Averaged Results

Under micro averaging, the full method substantially reduces violations versus DextER + BoN, while the Ours variants retain their ranking and widen their margins over baselines.

B.3 Micro-Averaged Results

  • 84% lower violation under micro averaging for the full method versus DextER + BoN (43.0% to 6.9%).The reduction is larger than the 42% reduction under macro averaging (29.9% to 17.2%).Source: §B.3 · p. 10
  • The Ours variants’ ranking is unchanged under micro averaging, and their margins over baselines widen because mug cases dominate the pooled set.Source: §B.3 · p. 10

C Datasets … D LLM Instruction Parser

DexGYSNet contains almost no part-directed avoidance language, motivating NegGrasp, a benchmark built from existing compliant grasps paired with new prohibitive instructions. The appendix also describes how an instruction parser separates positive grasp requests from explicitly forbidden parts.

C Datasets

  • The appendix quantifies DexGYSNet’s negation gap and documents NegGrasp’s construction.The corpus scan found only five part-directed avoidance instructions, while NegGrasp adds newly written prohibitive instructions to existing scenes and compliant grasps.Source: C Datasets · p. 10, §C.1 · p. 10, §C.2 · p. 10

C.1 Negation is Essentially Absent from DexGYSNet

  • Only 5 of DexGYSNet’s 49,433 instructions (0.010%) forbid touching a named part, leaving too few examples to teach avoidance behavior.Although 23 instructions contain a negation cue, 18 negate an outcome rather than a contact region.Source: §C.1 · p. 10

C.2 NegGrasp Construction

  • NegGrasp pairs each scene and forbidden part with a newly written grasp instruction and an existing ground-truth grasp that already respects the prohibition.This compliant reference grasp demonstrates that the instruction is satisfiable, so benchmark failures reflect failure to honor the constraint rather than inability to produce a suitable grasp.Source: §C.2 · p. 10
  • NegGrasp forbids only contested parts, touched by some grasps and clearly avoided by others, using hand-to-part distance thresholds below 4 mm and above 10 mm.The gap excludes borderline grasps, and compliant grasps have a median 26.4 mm clearance from the forbidden part.Source: §C.2 · p. 10
  • Gemma 4 writes one command from measured finger-link contacts, the forbidden part, and clearance, with randomized tone, clause order, and avoidance wording.Some instructions include a reason, and roughly three quarters also specify where to grasp; every instruction forbids the part.Source: §C.2 · p. 10
  • NegGrasp filters for explicit negative expressions and deduplicates instructions, yielding 6,928 cases across 672 scenes and 20 object categories.The 4,085-train/2,843-test split inherits DexGYSNet’s object-level split, so test instances are unseen; experiments use the test split only.Source: §C.2 · pp. 10–11

D LLM Instruction Parser

  • The parser returns a positive-instruction rewrite and forbidden-part list as validated JSON, retrying invalid outputs up to three times.On test, it flags multiple parts in 80 of 2,843 cases and none in 6; the latter are among 50 cases excluded because no forbidden part can be grounded.Source: D LLM Instruction Parser · p. 11
  • Parser rules standardize part names, prefer fewer larger parts, split bilateral parts, and flag only explicitly negated parts—not grasp targets.The parser is shared by compared methods, so prompt effects on grounding apply equally to methods marked decomp. in Table 1.Source: D LLM Instruction Parser · p. 11

E Hyperparameter Sensitivity Analysis

The appendix varies one inference-time setting at a time on the full NegGrasp test split around the final configuration. Churn strength is the most sensitive axis, while the contact gate, guidance window, and potential saturation distances are comparatively insensitive.

Number of particles N

  • Violation falls from 32.8% to 17.2% and contamination from 15.5% to 6.6% as N increases from 2 to 16; CSR saturates between N=8 and 16.At N=32, inference cost doubles without improving any metric, placing the default N=16 where compliance has converged.Source: E Hyperparameter Sensitivity Analysis · pp. 11–12

Weight sharpness λ

  • Violation increases mildly from 15.8% at λ=5 to 19.5% at λ=50, while CSR stays within a 2.6-point band across the sweep.The default is the only setting near the top of both compliance and physical columns; sharper resampling favors the potential, which rewards object proximity as well as avoidance.Source: E Hyperparameter Sensitivity Analysis · p. 12

Churn strength η

  • At η=0, violation rises to 28.3%; η=0.05 and 0.1 pass the distribution-preservation check, with violation rates of 19.3% and 17.2%, respectively.Within the admissible interval, η=0.05 has CSR of 65.9%; at η=0.1, 0.2, and 0.3, TSR falls to 38.7%, 29.2%, and 27.0%, with no compliance gain beyond η=0.1.Source: E Hyperparameter Sensitivity Analysis · p. 12

Contact gate and guidance window

  • Across a fivefold gate-distance change from 20 to 100 mm, violation and CSR move by at most 2.5 and 1.3 points; early-only or late-only guidance changes them by about one point.Neither gate distance nor guidance window improves on the default or substantially degrades the method.Source: E Hyperparameter Sensitivity Analysis · p. 12

Potential saturation distances τ and δ

  • Across τ=8 to 80 mm and δ=15 to 60 mm, no setting improves on the default in compliance or success, and CSR stays within 2.2 points of the default.These potential saturation distances therefore require no per-task tuning.Source: E Hyperparameter Sensitivity Analysis · p. 12

F Compute Cost and Reproducibility · G Model

Appendix F reports the steering stack’s compute cost, run-to-run stability, and statistical validation. Appendix G details the frozen encoders, conditioning, and DiT implementation used for inference-time steering.

F Compute Cost and Reproducibility

  • Generating, scoring, and simulating 2,843 test instances takes about 1.6 hours per configuration on 4×NVIDIA TI-TAN RTX (24 GB), scaling linearly with particle count N.The DiT forward pass uses bfloat16 autocast.Source: F Compute Cost and Reproducibility · p. 12
  • Three final-configuration runs yielded micro-averaged CSR of 70.2%, 70.1%, and 70.1% (σ=0.1 pp), while macro-averaged violation varied with σ=2.3 pp.Experiments use seed 0; deterministic scoring reproduces tables from released predictions, but generation is not bit-reproducible, so differences below these run-to-run floors are not interpreted.Source: F Compute Cost and Reproducibility · p. 12
  • Against DextER with best-of-N selection, paired macro differences were −12.6 [−17.8, −7.4] on violation, −7.8 [−11.1, −4.3] on contamination, and +10.0 [+4.3, +15.7] on CSR.A stratified paired bootstrap with B=10,000 found that all three 95% intervals excluded zero; churn reduced violation against deterministic SMC by −11.1 [−16.1, −6.0].Source: F Compute Cost and Reproducibility · p. 12

G Model

  • Frozen PartField encodes 10,000 XYZ points into 768 patch tokens of width 1024 and is never fine-tuned.Its PVCNN and triplane transformer produce 3×32×32 tokens, average-pooled 2×2 per plane; other input channels use a constant placeholder.Source: G Model · p. 12
  • Frozen Qwen2.5-0.5B-Instruct encodes the templated positive instruction as per-token 896-dimensional hidden states for cross-attention.The base transformer, not its language-modeling head, supplies all tokens of the last hidden state.Source: G Model · pp. 12–13
  • The DiT cross-attends to separately projected language and geometry streams; training drops text with probability 0.15, while inference substitutes a second real prompt B.The geometry stream is never dropped, and the two modalities are concatenated rather than summed or interleaved.Source: G Model · p. 13
  • The 12-block, width-768 DiT uses adaLN-zero and fully bidirectional attention over nine state tokens and the conditioning memory.Self-attention operates among state tokens, while cross-attention queries the conditioning memory; the model uses no causal masking.Source: G Model · p. 13

H Qualitative Results

Across ten NegGrasp cases spanning seven object categories, baselines often contact forbidden parts, while the proposed method avoids them and contacts the instructed part.

H Qualitative Results

  • In Figure 4, the method avoids forbidden parts in every case shown, often finding compliant grasps distinct from the reference GT.Baselines frequently contact the forbidden part, and sometimes seize it as though it were the instructed target.Source: H Qualitative Results · p. 13

I Limitation and Future Direction

The framework depends on its frozen grounding stack and has only been evaluated in simulated single-object scenes with a ShadowHand. Future work can improve grounding, support multiple constraints, and validate the inference-time steering stack on physical robots in clutter.

I Limitation and Future Direction

  • Grounding fails to localize the forbidden part in 50 of 2,843 generated test cases, leaving steering without a region to act on.The framework inherits the quality of its frozen parser and 3D part-grounding model.Source: I Limitation and Future Direction · p. 13
  • The parser keeps only the first flagged part, so instructions forbidding several regions at once are only partially enforced.Supporting multiple simultaneous constraints is a natural extension.Source: I Limitation and Future Direction · p. 13
  • Evaluation is limited to simulated single-object scenes with a ShadowHand, and physical success is judged by the Isaac Gym protocol rather than real hardware.A natural extension is validating the steering stack on a physical robot in cluttered scenes.Source: I Limitation and Future Direction · p. 13
  • Improving open-vocabulary part grounding is a natural extension, and the inference-time design leaves the trained generator untouched.The proposed extensions can be made without modifying the trained generator.Source: I Limitation and Future Direction · p. 13

Related papers

Loading 2609.00555v1…