PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM2R: From Large Language Models to 3D Reassemblers

Zhouhan Zhong, Qiuhong Shen, Xingyi Yang, Jinyi Xu, Xinchao Wang

arXiv:2610.18604v1cs.CVcs.AIcs.RO

TL;DR

Part placement depends on the whole set, but existing assembly methods can struggle with global constraints and scale poorly as part counts grow. LLM2R processes geometric part tokens with a pretrained decoder and whole-assembly summary tokens, predicting all poses in one pass. It leads on several benchmark metrics and its advantage grows with part count.

  • Problem

    Part placement depends on the complete part set, while pairwise relations and iterative generation and verification scale poorly for objects with many parts.

    Source: §1 · pp. 1–2

  • Method

    LLM2R concatenates geometric tokens for all parts and uses a LoRA-adapted decoder-only LLM with appended summary tokens to predict all poses in one pass.

    Source: §1 · p. 2

  • Results

    LLM2R leads PartNet and PartNeXt in part accuracy and Breaking Bad in translation error and inference time; at 100 parts, it gains 33.5 part-accuracy points over GARF on Breaking Bad.

    Source: §1 · p. 2

  • Takeaways & Limitations

    One point-cloud-only architecture handles semantic and geometric assembly without images, templates, or category-specific priors.

    Source: §1 · p. 2

    LLM2R is order-sensitive, and its decoder context window limits processing to about 125 parts at K = 64 with an 8,192-token limit.

    Source: §5 · p. 10

Abstract

from arXiv · show

3D part assembly aims to reconstruct a complete shape from a set of unassembled parts by predicting a rigid SE(3) transformation for each part. Because the placement of a part is rarely determined by its own geometry, the task demands reasoning over the entire part set, and existing methods, which rely on pairwise relations or iterative generation and verification, scale poorly to objects with many parts. We present LLM2R, which casts part assembly as sequence modeling with a pretrained decoder-only large language model (LLM): a Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens, the blocks of all parts are concatenated, and a causal LLM adapted with Low-Rank Adaptation (LoRA) processes the sequence in one forward pass. Causal attention, however, lets the hidden states of a part observe only the parts that precede it. LLM2R therefore appends a block of learnable summary tokens after the last part: these tokens follow every part, so their states aggregate the whole assembly at a constant token cost, and a part-aware pose head fuses each part's states with them before predicting its rotation and translation. One architecture thus handles semantic and geometric assembly and predicts all poses in a single non-autoregressive pass. LLM2R achieves the highest part accuracy and success rate on all three PartNet categories, the best rotation error, translation error, and part accuracy on PartNeXt, and the lowest translation error and fastest inference on Breaking Bad, where only a fracture-specialized method is more accurate. Its advantage grows with the number of parts: at 100 parts it exceeds GARF by 33.5 points in part accuracy on Breaking Bad and Assembler by 29.1 points on PartNet, and it transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.

1 INTRODUCTION

3D part assembly requires globally consistent poses, yet existing approaches struggle with higher-order constraints, unavailable auxiliary cues, and growing part counts. LLM2R models parts as a geometric-token sequence and uses a pretrained decoder-only LLM with summary tokens to predict all poses in one pass.

1 INTRODUCTION

  • Part placement depends on the whole set, while pairwise or local matching struggles with global consistency, auxiliary cues may be unavailable, and iterative verification costs grow with part count.The task also arises in settings without a CAD model or template, where configuration must be inferred from part geometry.Source: §1 · pp. 1–2
  • LLM2R encodes each part as geometric tokens, concatenates the parts, and processes them with a LoRA-adapted pretrained decoder-only LLM using point clouds alone.The geometric tokens are produced by a Point-BERT encoder and a Q-Former; the same architecture handles semantic and geometric assembly.Source: §1 · p. 2
  • Learnable summary tokens follow every part and aggregate the full assembly; a part-aware pose head fuses those states with each part’s states to predict all poses in parallel.The summary block has constant token cost and preserves the pretrained decoder’s causal attention.Source: §1 · p. 2
  • LLM2R achieves the best part accuracy on PartNet and PartNeXt, and lowest translation error and inference time on Breaking Bad; its advantage reaches 33.5 and 29.1 part-accuracy points over GARF and Assembler at 100 parts.It transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.Source: §1 · p. 2
  • To the authors’ knowledge, LLM2R is the first method to adapt a pretrained decoder-only LLM to direct SE(3) pose prediction for 3D part assembly.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses semantic assembly, fractured fragments, point-cloud understanding, continuous outputs, auxiliary-token aggregation, and point-cloud encoding. LLM2R differs by predicting per-part poses from a whole-sequence summary in one pass.

3D shape assembly

  • Semantic assembly methods model inter-part relations, while fractured reassembly emphasizes fracture complementarity; prior approaches include iterative, hierarchical, diffusion-based, matching, and generation-assisted methods.Source: §2 · p. 2
  • BrickGPT applies autoregressive language modeling to brick construction, whereas LLM2R reassembles given parts in one long-context LLM pass using a learned whole-assembly summary.Source: §2 · p. 3

LLMs for point cloud understanding

  • Point-cloud LLMs address object or scene description, question answering, and interaction through free-form text; LLM2R instead predicts per-part SE(3) transformations conditioned on the full part set.Source: §2 · p. 3

Pretrained language models for continuous outputs

  • Prior continuous-output adaptations include sequence tasks, tokenized robot actions, and human-pose regression; LLM2R predicts continuous poses for a variable number of parts in one pass.Source: §2 · p. 3

Auxiliary tokens in Transformers

  • Transformer aggregation mechanisms include global or memory tokens, vision registers, and latent queries; LLM2R appends summary tokens after all inputs and fuses their states with each part’s inside the pose head.Source: §2 · p. 3

Point cloud encoders

  • Point-cloud encoders use varied set-processing and pretraining strategies; LLM2R uses pretrained Point-BERT features and a learned Q-Former to compress patch features into fixed-count tokens.Source: §2 · p. 3

3 METHOD

LLM2R converts unassembled part point clouds into geometric-token sequences and predicts each part’s rigid pose. Trailing summary states provide assembly-wide context for a single-pass pose head.

3 METHOD

  • Given canonical-frame point clouds, LLM2R predicts each part’s rigid SE(3) pose to place it in the assembled object’s global frame.Canonical frames and pose targets come from part geometry; images, templates, and semantic labels are not used.Source: §3 · p. 3

3.1 GEOMETRIC TOKENIZATION AND SEQUENCE CONSTRUCTION

  • A Q-Former compresses each part into 64 shared geometric tokens; a learned 64-token summary block follows all part blocks.Point-BERT supplies patch-level descriptors, while the summary block is shared across objects and appended once per object.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 CAUSAL DECODING WITH ASSEMBLY-WIDE SUMMARY STATES

  • Because causal part states see only a prefix, trailing summary states aggregate the full part set and give every pose assembly-wide context at constant token cost.The summary block provides this context without modifying the decoder’s causal attention mask.Source: §3.2 · p. 5

3.3 PART-AWARE POSE PREDICTION

  • The pose head fuses each part state with its matching summary state, then predicts rotation from anchor quaternions and translation directly.The anchors form a continuous rotation parameterization rather than a discrete rotation classification.Source: §3.3 · pp. 5–6
  • Training combines pose losses with Chamfer supervision on transformed point sets and repulsion for interchangeable-part groups; inference predicts all poses in one pass.The Chamfer term uses 1,024 random points per part and compares predicted- and target-pose geometry.Source: §3.3 · p. 6

4 EXPERIMENTS

LLM2R is evaluated on three assembly benchmarks using their established protocols, with comparisons spanning semantic and geometric assembly. Results cover benchmark performance, zero-shot transfer, scaling with part count, and ablations of the model design.

4.1 EXPERIMENTAL SETUP

  • The three benchmarks use established protocols: Breaking Bad tests fracture reassembly, PartNet semantic parts, and PartNeXt hierarchical annotations.Models are trained separately for each benchmark using the training and evaluation protocols of the strongest published baseline.Source: §4.1 · pp. 6–7

4.2 BENCHMARK COMPARISONS

  • LLM2R leads pose-centric metrics on PartNeXt and PA/SR on PartNet, while Breaking Bad favors it in translation error and inference time.On PartNeXt, rotation error is 41.12 against CRAG's 45.12, translation RMSE is 5.24 against 9.13, and PA is 67.93% against 65.89%; CD remains higher. On PartNet, Chair SR is 41.40% against Assembler's 22.59%; on Breaking Bad, RMSE(T) is 0.46 and 0.53 against GARF's 1.34 and 1.77, and inference time is 0.092 s per instance against GARF's 0.225 s.Source: Table 1 · p. 7, §4.2 · pp. 7–8, Table 3 · p. 8

4.4 ABLATION STUDIES

  • Removing the summary block causes the largest PA drop, while removing instruction tokens produces the largest SR drop in the PartNet Chair ablation.Without the summary block, PA falls 7.7 points and CD rises 2.31; without instruction tokens, PA falls 6.1 points and SR falls 6.6 points.Source: Table 4 · p. 8, §4.4 · p. 9

4.3 GENERALIZATION AND SCALABILITY

  • Zero-shot on unseen PartNet categories, LLM2R's PA is at least 16.7 points above the highest baseline on Bed, Door, and Storage Furniture.It also attains the lowest CD and RMSE(R) on each category after training only on Chair, Table, and Lamp.Source: §4.3 · p. 8, Table 5 · p. 9
  • At 100 parts, LLM2R exceeds GARF by 33.5 PA points on Breaking Bad and Assembler by 29.1 on PartNet.At 100 parts, PA is 57.85% versus GARF's 24.33% on Breaking Bad, and 36.83% versus Assembler's 7.78% on PartNet.Source: Figure 2 · p. 9, §4.3 · p. 9

4.4 ABLATION STUDIES

  • Increasing K improves CD, PA, and SR monotonically, but gains saturate beyond K = 64 as the geometric-token budget doubles.From K = 64 to 128, PA improves by only 0.13 points and CD by 0.08; CA stays within 0.2 points across the tested budgets.Source: §4.4 · pp. 9–10, Table 6 · p. 10
  • On 20-part Chair instances, PA declines gradually by about 24 points across the input sequence, without a tail collapse.The first two positions are most accurate at 83.4% and 80.9%, while positions 18 to 20 reach 56.4% to 59.6%; later parts may more often be small or repeated and intrinsically ambiguous.Source: Table 6 · p. 10, §4.4 · p. 10

5 CONCLUSION

LLM2R uses whole-assembly summary tokens and a part–summary fusion head to predict poses in one pass, with strong benchmark results and documented scope limits.

5 CONCLUSION

  • LLM2R achieves the best part accuracy on PartNet and PartNeXt and the lowest translation error and inference time on Breaking Bad, where GARF remains more accurate.Its advantage over baselines grows with part count, and it transfers zero-shot to unseen categories.Source: §5 · p. 10
  • A learnable summary block aggregates the complete assembly in one forward pass, and a part–summary fusion head conditions each pose on that context at constant token cost.The same architecture handles semantic and geometric assembly without autoregressive decoding.Source: §5 · p. 10
  • The summary block contributes most to part accuracy, while instruction ablations show that the decoder uses the language instruction as well as its capacity.The paper interprets this as evidence that the pretrained backbone is used for more than its capacity.Source: §5 · p. 10
  • The method is order-sensitive, limited by the decoder context window, predicts only one pose per part, and faces ambiguity in PCA-derived targets for symmetric parts.At K = 64, the context window permits about 125 parts; the Chamfer objective only partly compensates for pose-label ambiguity.Source: §5 · p. 10

A.1 ASSEMBLY INSTANCES FROM PARTNEXT

The appendix describes how PartNeXt assemblies are constructed and how each part receives a geometry-derived canonical frame and pose target.

A.1 ASSEMBLY INSTANCES FROM PARTNEXT

  • PartNeXt shapes are converted from textured meshes and face-based part masks into per-part point sets, with one part per scene primitive used when masks are unusable.The adopted CRAG split contains 15,563 training and 3,903 evaluation shapes, each with 2–20 parts.Source: §A.1 · p. 15

A.2 CANONICAL FRAMES AND POSE TARGETS

  • Each part receives a PCA-derived canonical reference frame, and its learning target is the rigid transform from that frame to its placement in the assembled object.The PCA axes are ordered by eigenvalue, with the third axis flipped when needed to make the frame right-handed.Source: §A.2 · p. 15
  • This target makes pose prediction the placement of a geometry-consistent canonical part rather than an absolute pose.The canonical frame can be ambiguous for symmetric parts because principal-axis signs and the ordering of similar eigenvalues are arbitrary.Source: §A.2 · p. 15

A.3 INPUT COORDINATES AND DATA SPLITS

Input preprocessing expresses each part in a canonical frame while retaining its coordinates in the normalized object frame for pose-target construction.

A.3 INPUT COORDINATES AND DATA SPLITS

  • Before tokenization, each part’s vertices are centered at its centroid and rotated into its principal axes so the encoder observes intrinsic geometry.The model predicts the transform that returns the part to the object frame.Source: §A.3 · p. 15
  • Parts with too few points for grouping are padded by repeating randomly selected points.Source: §A.3 · p. 15
  • PartNet and Breaking Bad use their respective authors’ official data splits and the same canonical-frame construction.Source: §A.3 · p. 15

B.1 MODEL AND TRAINING SETTINGS

The configuration combines Point-BERT features, Q-Former geometric tokens, a summary-based pose head, and LoRA adaptation; the quaternion head uses a fixed anchor set.

B.1 MODEL AND TRAINING SETTINGS

  • The main configuration uses a 12-layer Point-BERT encoder with 64 neighborhoods of 32 points and a Q-Former with 64 learnable queries and two layers.The encoder is frozen for the first 80 epochs and fine-tuned for the last 20; Table 7 lists the main settings.Source: §B.1 · p. 16, Table 7 · p. 16
  • The pose head uses 64 summary tokens, fuses each part state with summary context, and predicts rotation and translation through separate branches.LoRA adapts the query, key, value, and output projections of the backbone’s six full-attention layers.Source: §B.1 · p. 16

B.2 QUATERNION ANCHOR SET

  • The rotation head mixes 32 fixed unit-quaternion anchors, including signed coordinate-basis anchors and anchors with two nonzero entries.The set represents 16 distinct octahedral-group rotations twice and positively spans R4, allowing normalized convex combinations to express any rotation.Source: §B.2 · pp. 16–17

B.3 EVALUATION PROCEDURE

The evaluation records per-part pose and shape errors, thresholded part accuracy, and inference time, then aggregates metrics using dataset-specific protocols.

B.3 EVALUATION PROCEDURE

  • Per-part evaluation records geodesic rotation error, Euclidean translation error, squared Chamfer distance, and whether that distance is below the 0.01 PA threshold.Chamfer distance uses at most 2,048 randomly selected points.Source: §B.3 · p. 17
  • Breaking Bad metrics are RMSE(R), RMSE(T), PA, and whole-object CD; PartNet metrics are CD, PA, CA, and SR.The protocols follow GARF for Breaking Bad and Assembler for PartNet.Source: §B.3 · p. 17
  • Inference time is measured with device synchronization around the model call and divided by the number of evaluated instances.Source: §B.3 · p. 17

C ADDITIONAL QUALITATIVE RESULTS

Qualitative comparisons show LLM2R producing coherent, well-aligned assemblies on Breaking Bad and PartNet, including instances with many parts.

DGL Jigsaw PF++ GARF GT

  • On Breaking Bad, LLM2R produces globally coherent assemblies with fragments near ground-truth locations, while baselines more often leave them displaced or scattered.Figure 3 compares LLM2R with DGL, Jigsaw, PF++, and GARF.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 3 · p. 18
  • LLM2R places parts with accurate poses across PartNet Table, Lamp, and Chair instances, while several baselines show misaligned or detached parts.Figure 4 compares RGL, DGL, Score-PA, SPAFormer, and Assembler with LLM2R and ground truth.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 4 · p. 19
  • For PartNet instances with many thin or repeated parts, LLM2R places parts consistently, while baselines often produce collapsed or misaligned structures.Figure 5 includes wireframe chairs and storage furniture with many drawers.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 5 · p. 20

Related papers

Loading 2610.18604v1…