PaperlayerView exact revision on arXiv

Source-linked AI summary

Potential-Guided Particle Steering for Negation-Constrained Dexterous Grasping

Geonho Kim, SooGon Kim, Jongmin Lee

arXiv:2609.00555v1cs.ROcs.CV

TL;DR

Positive-only dexterous grasp models struggle with instructions that specify both where to grasp and where not to grasp, while collecting negation-specific supervision for all constraints is impractical. The paper uses inference-time classifier-free guidance and Sequential Monte Carlo with frozen 3D part grounding, without negative training examples. On NegGrasp, it reduces the strongest baseline’s violation rate from 57.9% to 17.2% while improving constraint-aware and physical success.

  • Problem

    Positive-only grasp training provides little evidence for avoidance semantics, while enumerating negative examples for task- and user-specific constraints is impractical.

  • Method

    The method trains a flow-matching DiT on positive instructions and handles negation at inference by combining CFG, SMC, and frozen 3D part grounding.

  • Results

    17.2% violation rate, compared with 57.9% for the strongest baseline, with improvements in constraint-aware and physical success on NegGrasp.

  • Takeaways & Limitations

    Inference-time steering enables constraint-aware dexterous grasping without negative training examples.

  • Takeaways & Limitations

    The approach assumes frozen pretrained language and geometric encoders provide useful linguistic and part-geometry priors; fine-tuning can erode these priors.

Abstract

from arXiv · show

Language-driven dexterous grasp models, such as DextER, perform well when instructions specify where to grasp, but we find they fail systematically when an instruction also specifies where not to grasp (e.g., "grasp the handle but avoid the body"). Existing training corpora, DexGYSNet among them, contain virtually no avoidance instructions, and collecting examples for every possible constraint is impractical. Moreover, because every part mentioned during training denotes a contact target, models may interpret a forbidden part as another region to grasp rather than one to avoid. We therefore introduce an inference-time framework for negation-constrained dexterous grasping that requires no negation-specific training examples. Combining Sequential Monte Carlo with classifier-free guidance, our method guides sampling toward the instructed part while pruning candidates headed for the forbidden region, without any negation examples during training. A frozen 3D part-grounding model localizes the forbidden region from the language instruction. To evaluate this setting, we construct NegGrasp, a benchmark of paired positive/negative instructions with constraint-aware metrics that credit a grasp only if it both accomplishes the task and respects the stated constraint. On NegGrasp, our method reduces the violation rate of the strongest baseline from 57.9% to 17.2% while improving both constraint-aware and physical success.

1 Introduction

Language-driven dexterous grasping models are trained mainly on positive instructions, leaving them unreliable when users specify regions to avoid. The paper proposes inference-time steering and introduces NegGrasp to evaluate target completion together with constraint compliance.

  • Motivation: Positive-only training corpora provide little evidence for associating negation with avoidance behavior.DexGYSNet contains essentially no avoidance phrasing, while every mentioned part typically denotes a contact target.
  • Motivation: Models frequently contact forbidden regions when instructions combine a grasp target with an explicit prohibition.The strongest baseline, DextER, often ignores clauses such as avoiding a mug’s body while grasping its handle.
  • Motivation: Collecting negative instructions for every user- and task-specific constraint is impractical.The paper therefore asks whether negation can be handled without additional supervision by steering generation during inference.
  • Approach: The framework combines classifier-free guidance and Sequential Monte Carlo on a flow-matching grasp diffusion model, using frozen encoders and inference-time region grounding.It retains frozen Qwen and PartField conditioning backbones and uses Find3D to ground the forbidden part.
  • Evaluation: NegGrasp pairs positive and negative instructions and evaluates grasps with constraint-aware metrics.The benchmark credits a grasp only when it accomplishes the task and respects the stated constraint.

2 Related Work

Prior work established language-conditioned dexterous grasping and negative-prompt steering, but existing avoidance methods and sampling techniques leave room for freely specified 3D constraints. This paper combines these ideas for inference-time negation-constrained grasping.

  • Language-conditioned grasping: DexGYSNet introduced large-scale language-guided dexterous grasp annotations, and DextER uses an autoregressive contact-token formulation trained on them.These approaches support language-specified grasping but are not designed for negative instructions.
  • Negative guidance: Perpendicular classifier-free guidance reduces interference between positive and negative scores in diffusion models.The paper transfers this formulation from image scores to contact-coordinate velocities in a grasp generator.
  • Avoidance grasping: OVAL-Grasp identifies grasp and avoidance regions for parallel-jaw, task-oriented grasping using image heatmaps without training.Unlike that dataset-level notion of usually wrong-to-touch regions, this paper targets freely specified constraints that change per instruction.
  • Sampling-based steering: Sequential Monte Carlo steers sampling by resampling particles according to importance weights without retraining.The paper uses a max-potential variant that preserves credit for earlier safe checkpoints.
  • Sampling-based steering: The proposed stack keeps the potential as a separate corrector and uses churn for diversity rather than twisted transitions or particle-dependent repulsion.These choices distinguish its implementation from Twisted Diffusion Sampler and Particle Guidance.

3 Method

The method trains a flow-matching diffusion transformer on positive instructions, then handles negation entirely during inference by grounding forbidden regions and steering, resampling, and checking candidate grasps.

  • Grasp generator: The generator denoises grasp poses and contact coordinates with a flow-matching diffusion transformer trained only on positive DexGYSNet instructions.Its state jointly refines pose and eight contact slots, so neither is decoded before the other.
  • Grasp generator: Frozen Qwen and PartField encoders provide linguistic and geometric conditioning while only the diffusion decoder and projection layers are trained.Freezing preserves priors that the paper reports can erode under narrow downstream fine-tuning.
  • Instruction parsing and grounding: An LLM rewrites the instruction into a positive prompt and flags negatively mentioned parts for a second affirmative prompt.The affirmative forbidden-part prompt avoids teaching a positive-only generator to interpret negation as avoidance.
  • Instruction parsing and grounding: Find3D labels the object point cloud, and the forbidden region is the set of points receiving a flagged part name.This region constrains sampling externally rather than entering the generator.
  • Classifier-free guidance: Each denoising step estimates finished contacts and applies gated perpendicular guidance only when they head toward the forbidden region.Contact coordinates permit a direct spatial check, while steering their velocity reshapes the co-evolving grasp pose.
  • Look-ahead potential: A potential scores candidates by both clearance from the forbidden region and proximity to the object, then ranks and prunes them.The potential is a soft score rather than a hard constraint.
  • Particle steering: Sequential Monte Carlo resamples trajectories by constraint compliance during denoising instead of selecting only at the end.Churn adds marginal-preserving stochasticity so duplicated particles can diverge after resampling.
  • Final selection: Final selection reconstructs hand meshes, rejects candidates too close to the forbidden region, and returns the closest qualifying grasp.If none qualifies, the margin is progressively relaxed and the least-violating candidate is selected.

4 Experiments

Experiments on NegGrasp evaluate the proposed inference-time steering framework, its components, and sampling hyperparameters using constraint-aware and physical grasp metrics. The full method combines CFG, SMC resampling, and marginal-preserving stochasticity to improve avoidance while retaining physical success.

  • Benchmark and setup: NegGrasp augments DexGYSNet with prohibitive constraints by pairing contested parts with avoiding grasps, producing scenes where compliant grasps remain available.The benchmark uses paired positive and negative instructions and evaluates cases in which forbidding a contested part rules out some natural grasps but not all compliant grasps.
  • Benchmark and setup: Constraint-aware evaluation measures violation rate, contamination, and clearance alongside physical grasp success, using a single 4 mm contact threshold.Standard stability and intention metrics do not penalize contacting the forbidden part, motivating complementary compliance metrics.
  • Main results: 57.9%: the strongest baseline violates the forbidden part on raw mixed instructions, while the full method reduces violation to 17.2% and contamination to 6.6%.The scaffolding alone lowers violations to 29.9%, and the full method achieves 21.1 mm clearance, comparable to the best 22.3 mm.
  • Main results: 61.5% CSR: the full method achieves the best constraint-aware success, 5.5 points above the strongest baseline, while TSR reaches 38.7%.CSR jointly requires forbidden-region avoidance, instructed-part contact, and object contact; TSR measures physical success.
  • Ablation study: Negative-prompt CFG improves constraint metrics but lowers TSR from 38.3 to 31.0, whereas SMC raises TSR to 40.5 while increasing violation to 28.3 through particle collapse.With deterministic ODE integration, resampling duplicates clones and reduces 16 particles to between 2 and 5 unique samples.
  • Ablation study: Marginal-preserving SDE churn keeps all 16 particles unique and combines CFG avoidance with SMC's physical success, while two resampling events provide the best joint trade-off.The n=2 schedule is near the top of both CSR and TSR, whereas additional events provide no further benefit and can increase violations.
  • Hyperparameter analysis: w=4 is selected because stronger guidance improves some compliance measures but makes TSR fall to 34.8 at w=6 and 31.1 at w=16.The guidance sweep shows that compliance and physical success peak at different weights.

5 Conclusion

The paper shows that language-driven dexterous grasping fails systematically on instructions specifying what not to touch, and addresses this entirely at inference time without negative training examples. On NegGrasp, the resulting steering framework reduces violations while improving both constraint-aware and physical success.

  • Conclusion: The inference-time framework steers a flow-matching grasp DiT with contact-gated CFG and churn-diversified SMC resampling, requiring no negative training examples.It addresses negation-constrained grasping without adding negation supervision.
  • Conclusion: On NegGrasp, the method cuts the strongest baseline's violation rate to less than a third while improving constraint-aware and physical success.The conclusion reports the benchmark-level outcome without replacing it with a single secondary metric.

Supplementary Material

The supplementary material defines the constraint-aware metrics and explains macro/micro aggregation for NegGrasp, including the shared evaluation set and imbalance motivating macro reporting.

  • Formal metrics: All contact tests use a single 4 mm threshold, while clearance is reported as a median distance from the forbidden part.Median reporting limits the influence of a single distant hand.
  • Formal metrics: CSRi combines violation avoidance, instructed-part contact, and object contact into one constraint-aware success measure.The metric is defined as (1 − Violi) · Hiti · Contacti.
  • Formal metrics: TSRi combines simulated grasp stability with non-violation under the DextER protocol.TSRi = Stablei · (1 − Violi).
  • Aggregation: Macro averaging first averages within each category–forbidden-part cell and then averages the 27 cell scores; micro averaging pools cases directly.Clearance uses medians for the inner case aggregation while retaining a mean across macro cells.
  • Aggregation: The evaluated set contains 2,793 of 2,843 cases, and mug instances account for 64%, so the main paper reports macro scores to reduce category imbalance.All methods share the same grounded case set.

B.3 Micro-Averaged Results

Under micro averaging, the full method retains its advantage and widens its violation-rate improvement over the strongest baseline configuration.

  • Micro-averaged results: 84% reduction in violation under micro averaging lowers DextER + BoN from 43.0% to 6.9%.The corresponding macro reduction is 42%, from 29.9% to 17.2%.

C Datasets

The paper quantifies the scarcity of contact-region negation in DexGYSNet and constructs NegGrasp from solvable contested-part constraints without using its train split in experiments.

  • Corpus analysis: Only 5 of 49,433 DexGYSNet instructions, or 0.010%, explicitly forbid touching a named object part.Among 23 instructions containing any negation cue, 18 negate outcomes rather than contact regions.
  • Benchmark construction: NegGrasp pairs each scene with a forbidden object part and a newly written instruction requesting a grasp while prohibiting that part.The benchmark changes the language while retaining a ground-truth grasp that satisfies the prohibition.
  • Benchmark construction: Contested parts are selected because at least one ground-truth grasp contacts them while another avoids them, making constraints non-vacuous and solvable.The construction uses below-4 mm contact and above-10 mm avoidance thresholds, with compliant grasps having 26.4 mm median clearance.
  • Experimental protocol: The NegGrasp train split is unused: the generator trains on original positive DexGYSNet instructions, and baselines receive no NegGrasp fine-tuning.Reported numbers are computed on the test split alone.
  • Instruction parsing: The shared parser converts each raw instruction into a positive rewrite and a complete part vocabulary with explicit forbidden flags.Outputs are constrained to validated JSON with snake_case part names.
  • Sensitivity analysis: Hyperparameter sensitivity is evaluated on NegGrasp using macro averages, with one steering-stack axis varied per block.All other settings remain fixed at the final configuration within each sweep.

E Hyperparameter Sensitivity Analysis

The sensitivity analysis finds particle count and churn strength most consequential, while CFG gate, guidance window, and potential saturation distances are comparatively robust.

  • Number of particles N: Violation falls from 32.8% to 17.2% and contamination from 15.5% to 6.6% as particles increase from N=2 to 16.Constraint compliance improves monotonically with particle count; N=32 doubles inference cost without improving any metric.
  • Weight sharpness λ: Violation rises from 15.8% at λ=5 to 19.5% at λ=50 as sharper weighting favors the potential’s object-proximity and avoidance terms.CSR remains within a 2.6-point band, and the default is near the top of both compliance and physical-success columns.
  • Churn strength η: At η=0, violation reaches 28.3%, whereas nonzero churn restores compliance to the 17–19% range.The marginal-preservation check passes for η ∈ {0.05, 0.1} but fails from η=0.3 upward; the default η=0.1 yields 17.2% violation.
  • Contact gate and guidance window: Changing the contact gate from 20 to 100 mm moves violation and CSR by at most 2.5 and 1.3 points, respectively.Restricting guidance to either trajectory half changes compliance and CSR by about one point, with no substantial degradation.
  • Potential saturation distances τ and δ: Across tenfold changes in τ and fourfold changes in δ, no setting improves compliance or success, while CSR stays within 2.2 points of default.The potential saturation distances are therefore low-sensitivity axes in the tested range.

F Compute Cost and Reproducibility

The appendix reports inference cost, reproducibility, statistical validation, and implementation details for the frozen-conditioned flow-matching DiT used by the steering framework.

  • Inference cost: Generating, scoring, and simulating 2,843 test instances takes about 1.6 hours per configuration, with cost scaling linearly in particle count N.Experiments use 4× NVIDIA TITAN RTX GPUs and bfloat16 autocast for the DiT forward pass.
  • Reproducibility: Three independent final-configuration runs yield micro-averaged CSR values of 70.2%, 70.1%, and 70.1% (σ=0.1 pp).Macro-averaged violation varies with σ=2.3 pp, so differences below these run-to-run floors are not interpreted.
  • Statistical validation: A stratified paired bootstrap with B=10,000 tests whether headline Table 1 comparisons are artifacts of the finite test set.Resampling occurs within each of 27 category-part cells using identical indices across methods.
  • Implementation: The flow-matching DiT is trained once on DexGYSNet’s positive instructions and left untouched during inference-time steering.Its architecture jointly refines grasp pose and eight contact-slot tokens through a 12-block DiT conditioned on frozen PartField and Qwen encoders.
  • Implementation: The model uses nine state tokens: one grasp token and eight contact-slot tokens, with bidirectional self- and cross-attention.The grasp token represents translation, axis-angle rotation, and 22 ShadowHand joint angles; contact slots encode coordinates and occupancy.
  • Implementation: Three output heads predict grasp velocity, contact velocity, and 18-way link-classification logits.The regression heads produce 28-dimensional grasp velocities and 8×4 contact velocities.

H Qualitative Results

Qualitative NegGrasp examples show baselines contacting or grasping forbidden parts, while the proposed steering avoids them and still reaches instructed parts.

  • Qualitative comparison: In ten test cases spanning seven object categories, baselines frequently contact forbidden parts, whereas the method stays clear while contacting the instructed part.The forbidden region is rendered in red; several baseline grasps seize it as though it were the target.

I Limitation and Future Direction

The framework is limited by single-part grounding and simulated single-object evaluation, while its prompts and parser explicitly separate positive grasp instructions from forbidden-part information.

  • Limitations: The parser or frozen 3D grounding model fails to localize the forbidden part in 50 of 2,843 generated test cases.When localization fails, steering has no region to constrain.
  • Future direction: Instructions forbidding several regions are only partially enforced because the parser retains only the first flagged part.The authors identify multiple-constraint support as a natural extension.
  • Limitations: Evaluation is conducted in simulation on single-object scenes with a ShadowHand, using the Isaac Gym protocol for physical success.The paper proposes validation on physical robots in cluttered scenes as a future direction.
  • Instruction parsing: The instruction parser rewrites commands into a positive grasp instruction and a complete list of physically distinct object parts.Its prompt removes avoidance clauses from the grasp instruction and discourages inventing small, non-load-bearing subparts.
  • Instruction generation: The instruction-generation prompt specifies an object, reference command, used contacts, and a part that must remain untouched.The command generator is instructed to output only a natural-language command sentence.
Loading 2609.00555v1…