Source-linked AI summary
Beyond Classification: Task-Dependent Learnability under Privacy-Motivated Image Transformations
Leon Ranke, Wolfgang Hübner, Ronny Hug, Michael Arens, Jürgen Beyerer
TL;DR
Classification-only evaluation can miss task-relevant geometric and spatial degradation in representation-altering PETs, while exhaustive downstream evaluation is expensive. The paper introduces a compute-aware protocol combining lightweight proxy tasks and shows that learnability varies by task, motivating joint evaluation beyond classification.
Problem
Classification mainly tests semantic separability and provides limited evidence about geometric, spatial, boundary, and orientation structure, while broad downstream evaluation is computationally expensive.
Method
The paper evaluates irreversible transformations, key-based block primitives, and learnable image-encryption schemes using classification, relative-angle prediction, and jigsaw solving as lightweight complementary probes.
Results
Learnability is strongly task-dependent: transformations with similar classification performance can differ substantially on geometric and spatial tasks, while fixed-key transforms can produce shortcut cues.
Takeaways & Limitations
PET evaluation should jointly assess task-dependent learnability, shortcut leakage, and image-domain obfuscation rather than rely on classification performance alone.
Takeaways & Limitations
Image-domain obfuscation and leakage metrics are indicators, not formal privacy guarantees, and do not by themselves establish resistance to attacks.
Abstract
from arXiv · showhide
Privacy-Enhancing Technologies (PETs) in computer vision often rely on noise or image perturbations to protect visual data while securely processing it, creating a trade-off between task performance and protection. This trade-off is commonly evaluated using image classification, which primarily captures semantic separability and remains robust despite significant geometric, spatial layout or local boundary alterations. As a result, it is too simplistic as a proxy for generic vision tasks. Exhaustive downstream-task evaluation, however, is computationally expensive because models must often be trained for each PET transformation and parameter setting. We therefore propose a compute-aware multi-task protocol for evaluating PETs in model training. It combines lightweight proxy tasks that target complementary aspects of visual structure while remaining simple and fast to compute. Across irreversible privacy transformations, key-based block primitives, and learnable image encryption schemes, we demonstrate that PETs with similar classification accuracy can differ substantially on other tasks. The outcomes highlight the need for PET evaluation protocols that move beyond classification-only reporting.
1 Introduction
Representation-altering PETs create a tension between obscuring sensitive visual content and retaining information usable for learning. Because classification mainly tests semantic separability, the paper proposes a compute-aware multi-task protocol using lightweight structural probes.
- Motivation: Classification accuracy is a narrow utility measure because it may remain high despite degraded geometry, spatial layout, boundaries, and orientation-dependent structure.Such transformations may still perform poorly on detection, segmentation, tracking, pose estimation, or geometric matching.
- Motivation: Exhaustive downstream evaluation is expensive because each transformation, parameterisation, encryption key, and scale may require training from scratch.The paper therefore motivates lightweight proxy tasks before expensive downstream pipelines.
- Operational framing: Learnability is the task performance achieved by a model trained from scratch on transformed samples under a specified transformation, model, and protocol.It characterises information accessible to a particular learner under finite data and computational resources, rather than an intrinsic property of the transformation.
- Contributions: Learnability is strongly task-dependent: similar classification performance can conceal severe impairment of geometric and spatial tasks or shortcut cues in key-based transformations.The findings motivate jointly assessing task-specific learnability, shortcut leakage, and image-domain obfuscation.
- Contributions: The protocol combines classification, relative-angle prediction, and jigsaw solving to probe complementary visual structures.Its contributions include analysing multi-scale obfuscation, block transforms, and learnable image encryption while separating classification, proxy-task utility, and image-domain metrics.
2 Related Work
Prior PPML work covers protected computation, training protocols, and image transformations, but systematic joint evaluation of image-domain obfuscation and downstream utility remains limited. The paper frames classification as incomplete and supplements it with structural probes and security indicators.
- Research gap: PPML research has comparatively little systematic methodology jointly assessing image-domain obfuscation and downstream task utility.
- PET taxonomy: PET utility is defined as the predictive performance preserved by a protected training pipeline relative to an equivalent plaintext pipeline.The paper distinguishes functionality-preserving, training-oriented, and representation-altering PETs.
- Representation-altering PETs: Representation-altering PETs transform images before training, so their utility depends on retaining sufficient task-relevant information and is predominantly evaluated through classification.
- Utility evaluation: Classification indicates semantic-label predictability but provides limited evidence about orientation, spatial compatibility, boundary continuity, and position-dependent cues.These properties matter for many downstream vision tasks, yet full downstream training is computationally expensive.
- Security evaluation: Image-security metrics quantify perceptual leakage, encrypted-domain statistics, and differential or key sensitivity, but they are indicators rather than formal privacy guarantees.They do not by themselves establish resistance to de-obfuscation, reconstruction, recognition, inference, or membership attacks.
3 Methodology
The methodology treats transformations as operators that alter visual-signal components and uses inexpensive proxy tasks as diagnostic probes of what remains accessible to learning algorithms. It evaluates controlled obfuscations, keyed block primitives, and learnable image-encryption schemes across structural requirements.
- Protocol: Proxy tasks diagnose whether visual-signal components removed, rearranged, or statistically altered by transformations remain accessible to a learning algorithm.
- Transformation families: The transformation families include controlled irreversible transforms, key-based block primitives, and learnable image-encryption schemes for systematic structural analysis.The first group removes selected structures, while the latter groups represent learnable image-encryption transformations.
- Transformation families: Gaussian blurring removes high-frequency components while preserving spatial topology and global intensity relationships, with increasing σ suppressing progressively finer details.Its kernel size is k = 2⌊3σ⌋+1.
- Transformation families: Locally Orderless Images discard spatial ordering within B × B windows while approximately preserving local marginal intensity distributions.Increasing B removes local correlations at progressively larger scales.
- Transformation families: Block-based transformations partition images into B × B blocks and apply deterministic key-based operations that disrupt global arrangement or local adjacency while preserving other structure.Learnable image-encryption schemes combine such primitives and remain invertible given the secret key.
- Evaluation tasks: Classification tests class separability, angle prediction tests orientation consistency across transformed views, and jigsaw solving tests spatial compatibility and local-to-global consistency.The probes are intended as a combined structural-sufficiency test rather than independent benchmarks.
4 Experimental Setup
Experiments train models from scratch on transformed data under consistent scales and fixed keys, using a 15-class animal-species dataset and task-specific evaluation protocols. Puzzle experiments align transformation-block and puzzle-piece scales.
- Training protocol: All models are trained from scratch exclusively on transformed data without paired plaintext images, secret keys, or transformation-specific adaptation modules.
- Training protocol: Scale parameters are fixed across training and testing, while key-based transformations use one sampled key per configuration for both phases.This fixed-key setting may expose deterministic key-dependent regularities.
- Datasets and metrics: Classification uses a 15-class Animal Species Classification dataset with images resized to 416×416 and top-1 accuracy as the metric.
- Datasets and metrics: Angle prediction reports mean angular error after rotation and cropping, with uniformly random prediction having a 90° chance level.
- Datasets and metrics: Jigsaw experiments resize images to 384×384, use k ∈ {4, 6, 8}, and choose block size relative to each puzzle piece to avoid scale misalignment.Performance is the percentage of correctly placed pieces, with chance level 1/k^2.
5 Results & Discussion
Across classification, angle prediction, and jigsaw solving, privacy-motivated transformations preserve different kinds of visual information. These task-dependent outcomes, related to perceptual obfuscation, show why no single utility measure adequately evaluates representation-altering PETs.
- Evaluation protocol: The evaluation compares classification, relative-angle prediction, jigsaw solving, and image-domain obfuscation metrics as complementary probes.The protocol first reports classification, then geometric and spatial tasks, and finally relates task performance to obfuscation metrics.
- Image Classification: 87.00% top-1 accuracy for the plain baseline and 57.10% at σ = 70 for Gaussian blur show that classification remains above chance despite substantial alteration.Chance performance is 6.67%; LOIs instead fall to 27.75% when correlations are removed globally.
- Angle Prediction: Classification can remain high while geometric learnability differs substantially across transformations.Angle prediction stays robust when coarse global structure survives but degrades when spatial order is removed at larger scales or local and global disruptions combine; key-based transformations show scale-dependent effects.
- Jigsaw Puzzle Solving: LOI approaches chance on jigsaw solving when transformation blocks are as large as or larger than puzzle pieces, whereas fixed-key transformations can remain highly solvable through deterministic cues.Changing the evaluation key causes key-dependent puzzle performance to collapse toward chance, indicating that high utility need not reflect natural spatial compatibility.
- Obfuscation-Utility Trade-off: No single point on the obfuscation axis predicts task utility across transformations because the same LPIPS level corresponds to different learnability depending on task requirements.Some transformations preserve classification while offering limited perceptual obfuscation, while others increase perceptual distance but degrade geometric or spatial learnability.
6 Conclusion
The conclusion treats learnability under image obfuscation as task-dependent and argues that classification alone provides only a partial evaluation. It recommends jointly interpreting complementary proxy tasks, shortcut leakage, and image-domain obfuscation while noting that proxy-task predictive value for full downstream tasks remains unconfirmed.
- Classification accuracy provides only a partial view of learnability because semantic separability does not fully capture geometric consistency or spatial compatibility.
- The protocol combines classification, relative-angle prediction, and jigsaw solving as lightweight probes of complementary visual structures.
- Fixed-key transformations can introduce deterministic shortcut cues that some tasks exploit.
- Proxy-task performance should be interpreted jointly rather than in isolation alongside task-dependent learnability, shortcut leakage, and image-domain obfuscation.
- The three tasks capture distinct structural properties, but their ability to predict full downstream tasks remains unestablished.
Hyperparameter Settings
The experiments use task-specific hyperparameter tables, PyTorch defaults unless otherwise stated, exact block tiling, and one random seed per configuration.
- Tables 4 to 6 list hyperparameters for classification, relative-angle prediction, and jigsaw puzzle solving, respectively.
- Unless stated otherwise, the experiments use PyTorch defaults.
- Block sizes B are chosen as divisors of each input size so blocks tile every image exactly.
- All experiments use a single random seed per configuration.
Obfuscation-Utility Trade-off
Figures 7 to 16 plot normalized task utility against normalized image-domain obfuscation metrics across transformation families and individual transformation–scale configurations.
- Figures 7 to 16 show task utility against image-domain obfuscation metrics, complementing the LPIPS trade-off in the main paper.
- Colour identifies the transformation family, marker shape identifies the individual transformation, and each point represents one transformation–scale configuration.
- Task utility is normalized per task as performance relative to plaintext baseline and chance performance.
- For angle prediction, the error is inverted before normalization so higher utility consistently indicates better performance.
- A normalized utility of u = 1 corresponds to plaintext-level performance, while u = 0 corresponds to chance.
- Obfuscation metrics are min–max normalized per metric with direction-aware inversion, while NPCR, UACI, and |K| are omitted because they measure sensitivity rather than per-configuration obfuscation.