PaperlayerView exact revision on arXiv

Source-linked AI summary

ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking

Oliver Groth, Fabian B. Fuchs, Ingmar Posner, Andrea Vedaldi

arXiv:1804.08018v2cs.CV

TL;DR

The paper asks whether physical intuition and object stackability can be learned from visual observations for generalised stacking. It introduces ShapeStacks and trains a visual stability predictor, then uses its learned stackability in simulated construction. The model performs commensurately with state of the art on simulated and real-world images, and the approach builds stable towers, including through counterbalancing.

  • Problem

    The paper investigates whether intuitive physical principles and task-relevant object affordances can be acquired from visual observations rather than specified in advance.

    Source: §1 · p. 1

  • Method

    The authors construct ShapeStacks and train a visual stability predictor, then use the learned stackability of object geometries to guide simulated stacking.

    Source: §1 · pp. 2–3

  • Results

    The predictor performs commensurately with state of the art on simulated and real-world images, and the approach builds stable towers and stabilises unstable structures with counterweights.

    Source: §1 · p. 3, §6 · pp. 13–14

  • Takeaways & Limitations

    The results suggest that physical principles and geometric affordances can be acquired from visual observation and used in manipulation tasks.

    Source: §6 · p. 14

    The mechanics analysis is restricted to single-stranded stacks of convex objects, excluding structures such as arches, multiple columns, and forks.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Physical intuition is pivotal for intelligent agents to perform complex tasks. In this paper we investigate the passive acquisition of an intuitive understanding of physical principles as well as the active utilisation of this intuition in the context of generalised object stacking. To this end, we provide: a simulation-based dataset featuring 20,000 stack configurations composed of a variety of elementary geometric primitives richly annotated regarding semantics and structural stability. We train visual classifiers for binary stability prediction on the ShapeStacks data and scrutinise their learned physical intuition. Due to the richness of the training data our approach also generalises favourably to real-world scenarios achieving state-of-the-art stability prediction on a publicly available benchmark of block towers. We then leverage the physical intuition learned by our model to actively construct stable stacks and observe the emergence of an intuitive notion of stackability - an inherent object affordance - induced by the active stacking task. Our approach performs well even in challenging conditions where it considerably exceeds the stack height observed during training or in cases where initially unstable structures must be stabilised via counterbalancing.

1 Introduction

The paper studies learning physical intuition from visual observations and using it in generalised object stacking. It introduces ShapeStacks, evaluates stability prediction, and uses learned stackability to guide active stacking.

1 Introduction

  • ShapeStacks provides diverse geometric primitives and annotations of mechanical failure points inferred by formally analysing the underlying physics.The dataset is introduced to support exploration of stackability across object types.Source: §1 · p. 2
  • The model based on ShapeStacks outperforms Lerer et al. and performs commensurately with on real-world images without a physics engine at test time.The evaluation extends stability prediction to stacks containing multiple object geometries and quantifies localization of stability violations.Source: §1 · p. 2
  • Passive stability prediction yields stackability scores that rank block geometries for piece selection during active stacking.The authors report that adding noise to the stacking process produces a more intuitive notion of stackability.Source: §1 · p. 3
  • The approach builds stable towers and stabilises previously unstable structures by adding counterweights.The authors suggest this may reflect an intuitive understanding of counterbalancing.Source: §1 · p. 3

2 Related Work

Related work spans human physical intuition, visual learning of physical properties, stability prediction, and robotic stacking. ShapeStacks combines visual stability prediction with learned stackability across varied geometries.

2 Related Work

  • Prior intuitive-physics work includes predicting motion on inclined surfaces and ball-collision dynamics using deep learning and physics simulators.Cognitive science work also investigates human predictions of stability, relative mass, and trajectories.Source: §2 · p. 3
  • Unlike approaches that specify affordances a priori, this work learns stackability by passively predicting stack stability.Related visual-learning work includes support relations and geometric affordances, and inferring forces in Newtonian image understanding.Source: §2 · p. 3
  • The approach matches on real-world data while retaining efficient feed-forward prediction, unlike 's scene re-rendering and physics-engine stability computation.Lerer et al. introduced feed-forward stability prediction from simulated and real images.Source: §2 · p. 3
  • Prior work trained a stability predictor for simulated Kappla blocks to guide robotic stacking [20], stacked irregular stones using 3D perception and simulation, or learned to stack two blocks in simulation and reality.The cited comparison says [20] do not consider varied geometries or object affordances; is complementary, focusing on end-effector actuation while this work concentrates on visual feedback and emerging object affordances.Source: §2 · p. 4

3 The ShapeStacks Dataset

This section presents ShapeStacks as a diverse simulated stacking dataset and establishes the mechanical criteria and scope used to label stability.

3 The ShapeStacks Dataset

  • ShapeStacks provides 20,000 simulated stacking scenarios across cubes, cuboids, cylinders, and spheres, and the dataset is publicly available.The scenarios vary in geometry, structural complexity, and types of stability violation.Source: §3 · p. 4, §3.1 · p. 5

3.1 Dataset Content

  • Scenarios are single-stranded stacks two to six objects high, with stable and unstable cases balanced at each height and instability types evenly represented.The majority of stacks have height three; scenarios are split roughly evenly between cube-only and cuboid, cylinder, and sphere (CCS) subsets.Source: §3.1 · p. 5
  • Images use randomized appearance and 16 camera angles at 224 x 224 resolution, with binary stability labels and semantic maps identifying the violating object, first falling object, base, and top.Scenarios are split into approximately 70% train, 15% validation, and 15% test; cases where simulation outcomes conflict with physical derivation are discarded.Source: §3.1 · pp. 5–6

3.2 The Mechanics of Stacking

  • The analysis is limited to convex, single-stranded stacks, excluding structures such as arches, multiple columns, and forks.The stability condition assumes objects initially at rest and support planes parallel to the xy plane, orthogonal to gravity.Source: §3.2 · pp. 6–7
  • A stack is stable when, at every interface, the projection of the CoM of the blocks above that interface lies within its contact surface.The criterion is applied iteratively from the top of the tower downward.Source: §3.2 · pp. 6–7
  • Considering the CoM of each upper sub-stack permits stable configurations with apparent violations for individual blocks, including structures not constructible by adding one object at a time.Counterbalancing by blocks higher in the stack can keep the overall structure stable.Source: §3.2 · p. 7
  • The dataset distinguishes shape-dependent VPSF violations from positioning-dependent VCOM violations and uses Lemma 1 to control and annotate the violated contact area.Each unstable scenario has exactly one VPSF or VCOM violation, and the image marks the object to which it is attributable.Source: §3.2 · p. 7

4 Stability Prediction

The section trains RGB-image classifiers to predict stack stability and tests whether their attention identifies mechanical failure regions. Inception v4 performs strongly on real-world towers, while its attention maps often focus on annotated violations.

4.1 Training the Stability Predictor

  • The classifiers learn stable-versus-unstable labels from initial stack images, using AlexNet or Inception v4 and balanced training subsets with both violation types represented.Training images are augmented while preserving physically plausible, upright towers; no collapse images are used.Source: §4.1 · pp. 7–8
  • Inception v4 generalises from cube-only training to other structures, suggesting it learns a CoM principle applicable to more complex shapes.Performance is best on the all-shapes simulated test data when training includes multiple object types.Source: §4.1 · p. 8
  • On real images, Inception v4 trained from scratch on ShapeStacks outperforms Lerer et al. and is on par with Wu et al..Table 2 reports stability prediction accuracy across models, training data, and real and simulated test sets; Inception v4 is consistently better than AlexNet.Source: §4.1 · p. 8, Table 2 · p. 9

4.2 Instability Localisation

  • An occlusion study blurs sliding image patches and aggregates the classifier’s predicted stability scores into attention maps.The analysis compares attention-map maxima with annotated failure regions across 1,500 randomly sampled test images.Source: §4.2 · p. 9
  • In 79.9% of unstable cases, attention focuses on the violation region; Figure 4 compares attention heatmaps with ground-truth segmentation maps.The violation region is the smallest rectangle enclosing the violating object and the first object to fall.Source: §4.2 · p. 9, Figure 4 · p. 10
  • For VPSF instabilities, attention targets the violating curved object in 52.1% of cases; for VCOM, it also covers the unsupported upper tower in 38.1%.The VCOM attention pattern is consistent with the physics governing those instabilities.Source: §4.2 · p. 9, Table 3 · p. 10

5 Stacking and Stackability

The section tests whether passive stability prediction yields actionable geometric affordances by applying the learned model to stackability estimation, tower construction, and counterbalancing. Across these tasks, the model ranks objects sensibly, builds towers with varied shapes, and can add counterweights to unstable structures.

5 Stacking and Stackability

  • Three active tasks test whether stability predictors acquire geometric affordances for building new stacks: ranking objects, positioning blocks, and counterbalancing instability.The section frames these tasks as a test of whether models learned from passive stability prediction acquire actionable physical knowledge.Source: §5 · p. 11

5.1 Stackability

  • The learned stackability ranking favors large cuboid faces and upright cylinders while rejecting spheres as generally unsuitable supports.Stackability estimates how well an object supports the other objects in the set, using stability scores across placements and perturbations.Source: §5.1 · pp. 11–12

5.2 Stacking Shapes in Simulation

  • The CCS-trained predictor outperforms the cubes-only predictor in all three stacking scenarios; the cubes-only model largely fails on varied shapes.Each episode starts with a pool of 12 objects, and CCS scenarios include two spheres, limiting their maximum height to 11.Source: Figure 7 · p. 13, §5.2 · p. 13

5.3 Balancing Unstable Structures

  • The model solves the unstable-T counterbalancing task with high probability by placing an additional object to counterbalance the frozen stack.The figure distinguishes the frozen unstable stack, its collapse, and successful counterweight placement, and reports success rates by counterweight type over 50 episodes.Source: §5.3 · p. 13

6 Conclusions

The conclusion presents ShapeStacks as a basis for learning physical intuition and geometric affordances from visual observations. The model performs in line with state of the art, localizes instability, and uses learned knowledge to guide stacking and counterbalancing.

6 Conclusions

  • The ShapeStacks-trained visual predictor performs commensurately with state of the art on simulated and real-world images and correctly localizes structural instabilities.The conclusion describes ShapeStacks as a public dataset with detailed annotations of mechanical stability violations.Source: §6 · p. 13
  • The conclusion links learned physical intuition to discovering object stackability and stabilizing previously unstable structures with counterweights.The authors describe the counterbalancing interpretation as arguable rather than definitive.Source: §6 · p. 13, Figure 9 · p. 14
  • The model guides simulated stacking solely from visual cues, and the results suggest physical principles and geometric affordances can be acquired from visual observation and used in manipulation.Examples include successful cube and cuboid towers reaching height 12 and counterweight placement to prevent collapse.Source: §6 · p. 14, Figure 8 · p. 14, Figure 9 · p. 14
Loading 1804.08018v2…