Articulation-aware Canonical Surface Mapping

• The paper predicts canonical surface mappings from pixels to a 3D template and the template’s articulation and pose for an input image.Source: §1 · p. 1
• Prior approaches learn these tasks independently and rely on large-scale annotations, limiting their scalability to generic classes.Examples include hundreds of thousands of annotations for human-mesh mappings and keypoint annotations for articulation inference.Source: §1 · pp. 1–2
• Geometric relations between dense pixel-to-template mappings and global articulation and camera pose let the predictions provide supervisory signal for each other.The paper enforces consistency between them to learn without direct supervision for either task.Source: §1 · p. 2
• The approach learns articulation and CSM across diverse animal classes from image collections, including ImageNet images with approximate off-the-shelf foreground segmentations.The authors report that allowing articulation improves CSM inference and enables unsupervised articulation learning.Source: §1 · p. 2

Pose and Articulation Prediction

• Pose and articulation methods for animate objects use fitting or prediction, but rely on often dense 2D keypoint annotations for learning and sometimes inference.The paper instead seeks to learn these predictions without that supervision.Source: §2 · p. 2

Dense Mappings and Correspondences

• Prior pixel-to-surface and pixel-to-volume mapping approaches rely on direct supervision, while learned camera-invariant embeddings do not enforce geometric structure.Kulkarni et al. learn mappings without direct supervision but ignore articulation, which this work models to obtain more accurate results.Source: §2 · p. 2

Reconstructing Objects in 3D

• More expressive 3D prediction approaches include volumetric, point-cloud, and morphable-model representations, but typically require 3D, multi-view, or keypoint supervision.The paper frames its articulated-template-and-pose representation as a restricted form of 3D reconstruction.Source: §2 · p. 2

3. Approach

• A-CSM learns CSM, articulation, and camera pose jointly, using their geometric consistency for supervision and foreground masks to constrain learning.The method can also incorporate sparse keypoint supervision when available.Source: §3 · pp. 2–3, §3.2 · pp. 4–5

3.1. Articulation-aware Geometric Consistency

• Softened per-vertex part assignments enable smooth template-mesh deformations across part boundaries under articulation.The mesh is rigged into functional parts, with each part transformed relative to its parent.Source: Figure 2 · p. 3, §3.1 · p. 3
• The mapping φ sends a 2D coordinate to a point on the canonical template, and applying articulation δ yields its point on the articulated shape.Thus, a pixel's CSM coordinates can be converted into its corresponding point on the articulated surface.Source: §3.1 · p. 3, §3 · p. 3
• Geometric cycle consistency penalizes the distance between a foreground pixel and the reprojection of its CSM-mapped, articulated template point.This closes the cycle from pixel to template surface, through articulation and camera projection, back to the image.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2. Learning CSM and Articulation Prediction

• Visibility loss discourages mappings to occluded regions, while mask supervision counters degenerate solutions and uses mask distance fields to encourage precise articulation.The overall objective combines the losses; the camera and deformation predictor also outputs eight pose-and-deformation hypotheses with associated probabilities.Source: §3.2 · pp. 4–5

4. Experiments

• The experiments evaluate both pixel-to-template CSM prediction and inference of articulation and pose that transform the template into the image frame.The study tests whether articulation improves CSM prediction and whether CSM consistency supports meaningful articulation.Source: §4 · p. 5

4.1. Datasets and Inputs

• The dataset combines CUB-200-2011, PASCAL VOC, and ImageNet, with quantitative results for birds, cows, horses, and sheep and qualitative results for other animals.Images are filtered to retain one large, untruncated, largely unoccluded animal.Source: §4.1 · p. 5

4.2. Evaluating CSM via Correspondence Transfer

• Across both keypoint-supervision settings, articulation-aware geometric consistency improves CSM transfer over articulation-agnostic consistency and Dense Equivariance.The authors also report accurate transfers across differing articulations, including bent horse heads.Source: §4.2 · p. 6, Table 1 · p. 8

4.3. Articulation Evaluation via Keypoint Reprojection.

• Articulation-aware predictions yield more accurate keypoint reprojections than models without articulation, except for unsupervised birds, where template-keypoint ambiguity affects evaluation.Qualitative results depict moving legs, elephant trunks, and animal heads.Source: §4.3 · p. 8, Table 2 · p. 8
• Without keypoint supervision, enforcing consistency among CSM, articulation, and pose is critical; with keypoint supervision, it gives modest improvements.The comparison is against a variant trained without the geometric cycle loss.Source: §4.3 · p. 8, Table 3 · p. 8

4.4. Learning from Imagenet

• Using category-level ImageNet images and automatically obtained instance masks, the method learns meaningful articulation and consistent CSM prediction, although some classes use other training images.The stated ImageNet-only training exception is birds, horses, sheep, and cows.Source: §4.4 · p. 8

• The allowed transformations may not capture intrinsic shape variation, constraining the method when within-category shapes differ.The discussion identifies categories without a well-defined template, such as chairs, as a more challenging case where the approach is not directly applicable.Source: §5 · p. 8
• Categories without a well-defined template, such as chairs, are a stated boundary for direct application.Source: §5 · p. 8
• The techniques may also serve as regularization or incorporate more unlabeled data when large-scale annotations are available.Source: §5 · p. 8

• The surface map φ composes a mapping from the 2D square to a spherical mesh with a mapping from the sphere to the template; the square-to-sphere map is analogous to latitude and longitude.Source: A. Constructing φ and δ · p. 11
• Template shapes are genus-0 meshes with 642 vertices and 1280 faces, and preprocessing deforms a sphere so corresponding faces have similar areas by minimizing squared differences of log triangle areas.The area-matching optimization uses Adam and is performed offline for each category.Source: A. Constructing φ and δ · p. 11
• Articulation assigns each part a local rigid transform, and each node’s global transform composes its local transform with its parent’s transform.Each part hierarchy has a parent for every node except the root.Source: A. Constructing φ and δ · p. 11
• Barycentric interpolation maps a square coordinate to a spherical-mesh face, then uses that face’s barycentric coordinates with the corresponding articulated-template face vertices to compute a 3D point.Source: A. Constructing φ and δ · p. 11

B.1. Network Details

• A ResNet-18 encoder with ImageNet-pretrained features and a five-layer decoder outputs a 3D unit vector per pixel, converted to a 2D square coordinate.The encoder features are extracted after four blocks.Source: §B.1 · p. 11

B.2. Optimization

• Part transforms use a category-shared learned axis and network-predicted angle, while camera pose is orthographic and predicts rotation, scale, and translation.The model predicts eight camera-pose and part-transform hypotheses.Source: §B.2 · p. 11
• Training initially learns camera pose and CSM for 10,000 iterations, then enables articulation for the remaining iterations using Adam at a learning rate of 10^-4.Source: §B.2 · p. 11

B.3. Losses

• Mask consistency penalizes rendered mesh pixels using the ground-truth mask’s Euclidean distance field, keeping the rendered mask inside the object mask.The distance field is summed over pixels in the rendered mask.Source: §B.3 · p. 11
• Mask coverage complements consistency by requiring every foreground object pixel to lie near a reprojected mesh vertex.The coverage objective competes with the consistency loss.Source: §B.3 · p. 11
• The mask objective sums the competing consistency and coverage losses.Source: §B.3 · p. 11
• The visibility loss uses the rendered depth map and each mapped pixel’s camera-frame depth.Source: §B.3 · p. 11
• Additional regularizers act on part-transform translations and encourage diversity in the multipose predictor.Source: §B.3 · p. 12

C. Evaluation

• A source keypoint is mapped to the non-articulated template, then matched to the target pixel whose CSM lies closest to that template point.Source: §C.1 · p. 12

C.2. Importance of Ground Truth Masks

• Using Mask-RCNN segmentations instead of precise ground-truth masks causes no significant performance drop, and the method remains competitive.The comparison uses PCK-Transfer and Key-Point Projection on CUB-2011.Source: §C.2 · p. 12
• On birds, evaluating PCK transfer with ground-truth rather than predicted masks changes average performance by 0.1 points.The passage reports no significant disadvantage from using predicted masks for evaluation.Source: §C.2 · p. 12

D. Qualitative Sampled Results on 11 Categories

• Across 11 categories and a wide range of articulations, sampled results show consistent CSMs for functional regions such as quadruped heads.The head pixels map to the corresponding green region on the template.Source: D. Qualitative Sampled Results on 11… · p. 12
• Internet-video examples are processed frame by frame without temporal smoothing.Source: D. Qualitative Sampled Results on 11… · p. 12, Figure 13 · p. 17
• The sampled visualizations cover bisons, cows, birds, elephants, sheep, rhinos, horses, tapirs, hippos, giraffes, and kangaroos.Source: Figure 9 · p. 13, Figure 10 · p. 14, Figure 11 · p. 15