PaperlayerView exact revision on arXiv

Source-linked AI summary

MeshAdv: Adversarial Meshes for Visual Recognition

Chaowei Xiao, Dawei Yang, Bo Li, Jia Deng, Mingyan Liu

arXiv:1810.05206v2cs.CRcs.CVcs.LGstat.ML

TL;DR

Pixel-level attacks may not be physically realizable, and texture-based attacks may leave 3D shape features intact. MeshAdv optimizes adversarial 3D meshes through differentiable rendering and also proposes a pipeline for black-box photorealistic rendering. The paper reports attacks on classifiers and detectors, including across viewpoints.

  • Problem

    Pixel manipulation may not be physically realizable, while texture-based physical attacks can leave an object's 3D shape features intact.

    Source: §1 · p. 1

  • Method

    MeshAdv uses a differentiable renderer to propagate model gradients to mesh shape or texture, and proposes a pipeline for black-box photorealistic rendering.

    Source: §1 · pp. 1–2

  • Results

    MeshAdv attacks classifiers and a detector with slight shape or texture perturbations, and multiple-view optimization increases attack success across viewpoints.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that these evaluations provide a better understanding of adversarial behaviors of 3D meshes in practice and may motivate future defenses.

    Source: §7 · p. 8

    The meshAdv method assumes a known renderer that is differentiable with respect to the mesh, requiring assumptions about material, lighting, and interreflection.

    Source: §4 · p. 3

Abstract

from arXiv · show

Highly expressive models such as deep neural networks (DNNs) have been widely applied to various applications. However, recent studies show that DNNs are vulnerable to adversarial examples, which are carefully crafted inputs aiming to mislead the predictions. Currently, the majority of these studies have focused on perturbation added to image pixels, while such manipulation is not physically realistic. Some works have tried to overcome this limitation by attaching printable 2D patches or painting patterns onto surfaces, but can be potentially defended because 3D shape features are intact. In this paper, we propose meshAdv to generate "adversarial 3D meshes" from objects that have rich shape features but minimal textural variation. To manipulate the shape or texture of the objects, we make use of a differentiable renderer to compute accurate shading on the shape and propagate the gradient. Extensive experiments show that the generated 3D meshes are effective in attacking both classifiers and object detectors. We evaluate the attack under different viewpoints. In addition, we design a pipeline to perform black-box attack on a photorealistic renderer with unknown rendering parameters.

1. Introduction

Prior physical attacks alter surface texture, but may be unsuitable for textureless objects; meshAdv instead perturbs 3D shape or texture. The paper reports attacks on classifiers and a detector, including across viewpoints, and a pipeline for black-box photorealistic rendering.

1. Introduction

  • A differentiable renderer propagates prediction gradients to mesh shape or texture, enabling end-to-end optimization of adversarial meshes with negligible perturbations.The approach targets objects with rich shape features but minimal texture variation.Source: §1 · pp. 1–2
  • MeshAdv consistently tricks classifiers and a detector with slight shape or texture perturbations, and multiple-view optimization increases attack success across viewpoints.The authors also report user studies finding the perturbations negligible to human perception.Source: §1 · pp. 1–2
  • Estimating rendering parameters and improving perturbation robustness lets meshAdv attack a photorealistic renderer with unknown parameters.The pipeline addresses rendering that is non-differentiable.Source: §1 · p. 2
  • Visualized shape perturbations identify potentially vulnerable mesh regions, which may inform robustness work on models trained with 3D meshes.The paper names viewpoint estimation and indoor scene understanding among possible tasks.Source: §1 · p. 2
  • User studies find the generated 3D perturbations subtle enough not to affect user recognition.This is one of the paper's stated contributions.Source: §1 · p. 2

2. Related Work

Prior physical adversarial attacks primarily manipulate texture, while meshAdv emphasizes shape perturbations and direct optimization in mesh representation. The related work also distinguishes its rendering setup, targets, and attack scope from prior approaches.

Adversarial Attacks

  • Texture-based physical attacks can be robust to camera transformations, but may not apply to arbitrary 3D shapes; meshAdv emphasizes distinctive shape features even with constant reflectance.The paper contrasts this focus with using 3D objects to carry information-rich textures.Source: §2 · p. 2
  • Zeng et al. [60] perturb physical parameters and use derivative-free optimization for non-differentiable rendering, whereas meshAdv optimizes plausible mesh shapes with gradients.The paper notes that their differentiable-renderer setup assumes known camera parameters and perturbs normal maps under fixed projection.Source: §2 · p. 2
  • Compared with concurrent work [32], meshAdv targets all other classes, reports almost 100% targeted success, and attacks both classifiers and object detectors.The paper also contrasts perturbation visibility, renderer accessibility, and the practicality of manipulating shape and texture rather than lighting.Source: §2 · p. 2

Watermarking for Meshes

  • Mesh watermarking hides secret data under strict vertex and edge properties, while meshAdv seeks perceptually realistic meshes whose rendered images fool a learning model.The paper distinguishes their goals despite both subtly manipulating meshes.Source: §2 · p. 3

3. Problem Definition and Challenges

The problem is to perturb a 3D object so its rendered image misleads a model, rather than manipulating image pixels directly. The task is constrained by coupled rendering variables and the perceptual consequences of changing 3D shape or texture.

3. Problem Definition and Challenges

  • In 2D, an adversarial attack generates an image that makes the model predict incorrectly or a specified target label.The ground-truth label is y, and the specified malicious target is y′.Source: §3 · p. 3
  • For a 3D attack, the image is rendered from object S using camera P and illumination L, so the attack must perturb S rather than image pixels.The perturbed object must render an image that causes incorrect predictions.Source: §3 · p. 3
  • Shape, texture, and illumination jointly determine rendered pixels, while 3D geometry and human perception make imperceptible mesh perturbations challenging.The paper notes that 3D constraints are not directly reflected in 2D.Source: §3 · p. 3

4. Methodology

meshAdv uses a differentiable renderer to optimize an adversarial mesh end to end, balancing targeted misclassification or detection loss against perceptual realism. Its perceptual loss smooths rendered texture perturbations and, for shape perturbations, smooths neighboring vertex displacements.

4. Methodology

  • meshAdv optimizes the adversarial mesh end to end through a known, differentiable renderer, balancing adversarial and perceptual losses with a hyperparameter.The renderer models camera and lighting parameters; the method assumes it is known and differentiable with respect to the mesh.Source: §4 · p. 3, §4.1 · p. 3

4.1.1 Adversarial Loss

  • For classification, meshAdv uses cross entropy toward a specified target; for Yolo-v3 detection, it uses disappearance attack loss.The detection model predicts bounding boxes and class probabilities, and the attack aims to make the victim object disappear.Source: §4.1.1 · p. 3

4.1.2 Perceptual Loss

  • For texture perturbations, a Laplacian loss smooths rendered images using differences between each pixel and its 4-connected neighbors.The loss is introduced to keep the adversarial mesh perceptually realistic.Source: §4.1.2 · p. 3
  • For shape perturbations, meshAdv instead smooths vertex displacement flow, encouraging neighboring vertices to move similarly and avoiding unwanted topology changes.The neighbors are connected by mesh triangles.Source: §4.1.2 · p. 4

5. Transferability to Black-Box Renderers

The black-box setting uses a non-differentiable photorealistic renderer with unknown parameters, where direct transfer from a differentiable renderer may fail under complex conditions. The proposed pipeline estimates rendering parameters, optimizes a robust adversarial mesh, and tests it in the original scene.

5. Transferability to Black-Box Renderers

  • The black-box attack targets a non-differentiable photorealistic renderer with unknown parameters and limited access, avoiding its expensive rendering during mesh generation.The renderer may include interreflection, occlusion, and rich illumination models.Source: §5 · p. 4
  • With camera and lighting held fixed, the controlled test replaces the differentiable renderer with the photorealistic renderer to assess whether adversarial meshes transfer.This tests the same mesh under the same rendering configuration, changing only the renderer.Source: §5 · p. 4
  • For unknown parameters, the pipeline first estimates the camera from object-mask silhouette error, then estimates lighting from masked rendered-image error.The mask rendering does not require lighting; the image comparison uses a Hadamard product with the object mask.Source: §5 · p. 4
  • After parameter estimation, meshAdv generates an adversarial mesh with random perturbations to camera and lighting parameters, then tests it in the original photorealistic scene.The random parameter perturbations are added to improve robustness.Source: §5 · p. 4

6. Experimental Results

MeshAdv is evaluated against classifiers, object detectors, and a photorealistic renderer, with experiments examining perturbation visibility and viewpoint robustness. Results include nearly 100% classifier attack success for shape and texture perturbations, attacks in indoor and outdoor detection scenes, and transfer to Mitsuba under unknown rendering parameters.

6.1. Experimental Setup

  • The classifier experiments used PASCAL3D+ meshes, rendered with NMR under camera and lighting configurations selected for perfect pristine-mesh accuracy.Seven classes had 72 rendering configurations each; the models were DenseNet and Inception-v3.Source: §6.1 · pp. 4–5

6.2. MeshAdv on Classification

  • MeshAdv achieved almost 100% attack success rate for either adversarial perturbation type; PASCAL3D+ rendering configurations were chosen so models had 100% test accuracy on pristine meshes.Average distance was measured using 3D Laplacian loss for shape perturbations and root-mean-squared face-color change for texture perturbations.Source: §6.2 · p. 5, Table 1 · p. 5
  • Vertex-flow visualizations associate greater vulnerability with high-curvature regions near the camera, where displacement changes surface normals and shading.Adjacent vertices also tend to flow in similar directions, consistent with the 3D Laplacian loss applied to vertex flows.Source: §6.2 · p. 5, Figure 3 · p. 6
  • In 99.29 ± 1.96% of trials, participants correctly recognized adversarial meshes, despite the classifier attack setting.The study collected 3,820 annotations from 49 participants.Source: §6.2 · pp. 5–6
  • Training on more victim views can improve attack robustness across unseen viewpoints, although a wider azimuth range makes high success harder to achieve.The study trained with 5, 10, or 15 views and tested on 20 unseen views in the same range.Source: §6.2 · p. 6, Table 2 · p. 6

6.3. MeshAdv on Object Detection

  • MeshAdv removed target detections in a synthetic indoor scene and an outdoor photo using shape or texture manipulation indoors and shape manipulation outdoors.The indoor scene used a Stanford Bunny and targeted the table or all detections; the outdoor scene targeted a dog or bicycle.Source: §6.3 · pp. 6–7

6.4. Transferability to Black-Box Renderers

  • Targeted attacks barely transferred to Mitsuba when meshes optimized for a differentiable renderer were directly rendered with matched camera and lighting parameters.In the same controlled setting, untargeted attacks transferred with relatively high success; Figure 6 shows targeted-attack confusion matrices.Source: §6.4 · p. 7, Figure 6 · p. 7
  • Attacks transferred to Mitsuba under unknown rendering parameters, hiding a targeted chair and misclassifying an aeroplane as “hammerhead.”The pipeline estimates camera and lighting parameters and adds small random perturbations to them during optimization; the lighting estimate need not be exact.Source: §6.4 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

7. Conclusion

MeshAdv generates adversarial meshes by manipulating shape or texture, and the paper evaluates these meshes on CAD models and black-box renderers.

7. Conclusion

  • The paper evaluates meshAdv quantitatively and qualitatively using CAD models from PASCAL3D+ and reports that adversarial behavior transfers to black-box renderers.The authors say this provides a better understanding of adversarial behavior of 3D meshes in practice and may motivate future defenses.Source: §7 · p. 8
  • MeshAdv manipulates mesh shape or texture to create meshes that can be rendered into 2D images to mislead machine-learning models.Source: §7 · p. 8
  • The paper presents its evaluation as contributing to understanding adversarial behavior of 3D meshes in practice and motivating potential defenses.Source: §7 · p. 8

A. Formulation of Differentiable Rendering

The differentiable-rendering formulation represents objects as triangular meshes with face colors and defines lighting and camera parameters for producing 2D images.

A. Formulation of Differentiable Rendering

  • The renderer computes a 2D image from a 3D object, camera parameters, and lighting parameters by approximating physical rendering.The formulation uses a differentiable renderer that makes computation differentiable with respect to the object, camera, and lighting inputs.Source: A. Formulation of Differentiable… · p. 12
  • The mesh representation uses 3D vertices and indexed triangular faces, with one RGB reflectance color attached to each face.Source: A. Formulation of Differentiable… · p. 12
  • Illumination uses directional light sources and ambient light, while the camera is described by distance, azimuth, elevation, and tilt.Camera intrinsics are fixed, and gradients are needed only for the extrinsic camera parameters.Source: A. Formulation of Differentiable… · p. 12
  • The rendering pipeline computes face normals, uses them to calculate shading, combines shading with face reflectance, and rasterizes face colors onto the 2D camera plane.Rasterization uses raycasting and depth testing, and color values are capped to [0, 1].Source: A. Formulation of Differentiable… · p. 12
  • The implementation uses the off-the-shelf PyTorch implementation of Neural Mesh Renderer.Source: A. Formulation of Differentiable… · p. 12

B. MeshAdv on Classification

The classification experiments create synthetic PASCAL3D+ renderings, retain samples correctly classified by both evaluated networks, and illustrate shape- and texture-based adversarial meshes.

B. MeshAdv on Classification

  • PASCAL3D+ renderings use CAD meshes with random azimuths from [0°, 360°), elevations from [0°, 90°], and sampled directional lighting within 60° of the view.Meshes are scaled to [−1, 1], and the scene uses directional and ambient lighting.Source: B. MeshAdv on Classification · p. 13
  • The experiment maps PASCAL3D+ classes to ImageNet classes and filters renderings until DenseNet and Inception-v3 both classify them correctly.The retained rendering configurations are saved for meshAdv evaluation.Source: B. MeshAdv on Classification · p. 13
  • Figure A presents benign renderings and corresponding DenseNet-targeted adversarial meshes, with shape perturbation in panel (a) and texture perturbation in panel (b).Source: B. MeshAdv on Classification · p. 13

C. Human Perceptual Study Procedures

The user study measures whether people perceive adversarial images as the ground-truth class or the target class.

C. Human Perceptual Study Procedures

Related papers

Loading 1810.05206v2…