Distilling Knowledge From a Deep Pose Regressor Network

1. Introduction

• Deep networks’ large computational and space requirements, together with near-real-time demands, constrain their deployment in resource-limited settings.Examples given include mobile phones and quadcopters.Source: §1 · p. 1
• Unlike classification, pose regression lacks softened-label dark knowledge, and real-valued teacher predictions can provide highly erroneous guidance.The paper proposes teacher loss as a confidence score for deciding when to trust the teacher.Source: §1 · p. 1
• The work targets visual odometry camera-pose regression and presents itself as the first attempt to distill a deep pose regressor.The paper describes DNN-based methods that learn features for estimating 6-DoF camera poses.Source: §1 · p. 2
• The approach applies teacher-loss confidence through Attentive Imitation Loss (AIL) and Attentive Hint Training (AHT).AIL weights imitation according to teacher reliability; AHT uses teacher loss when learning the teacher’s intermediate representation.Source: §1 · p. 2
• Up to 92.95% fewer student parameters and 2.12× faster computation were achieved while keeping prediction accuracy very close to the teacher.The experiments were conducted on KITTI and Malaga.Source: §1 · p. 2

2. Related Work

• Quantization reduces the bits used for network weights, with binary representations offering memory and computation reductions at the cost of significant accuracy loss.The passage mentions 16-bit, 8-bit, and 1-bit representations.Source: §2 · p. 2
• Pruning removes redundant weights and can reduce parameters without greatly affecting accuracy, but typically needs extra training stages and sparse-multiplication support.That support may require additional resources and specialized hardware or software.Source: §2 · p. 2
• Low-rank decomposition reduces matrix multiplications, but its compression ratio is generally lower than pruning’s and performance may depend on careful rank selection.The passage describes factorizing fully connected or convolutional layers using SVD or CP decomposition.Source: §2 · p. 2

3. Background: Distillation for Classification

• Knowledge distillation transfers a large teacher’s learned structure to a small student rather than training only from true labels.The student is trained to mimic the teacher output.Source: §3 · pp. 2–3
• The classification objective combines hard-label and soft-label cross-entropy terms, balanced by α.The hard labels are one-hot encodings; the student uses the same softened-output temperature as the teacher during training.Source: §3 · p. 3
• Softened teacher outputs convey relative similarities among incorrect classes, but regression lacks this dark-knowledge signal and training only on ground truth gave poor results in the authors’ experiments.The passage contrasts class probabilities with regression predictions that share the ground truth’s characteristics but have an unknown error distribution.Source: §3 · p. 3

4. Blending Teacher, Student, and Imitation Loss

• The minimum-loss formulation assumes the teacher is accurate, while the upper-bound formulation avoids extra imitation penalties when the student already outperforms it.A separate formulation adds imitation loss as a regularizer to student loss.Source: §4 · p. 3
• Probabilistic imitation models teacher uncertainty with a parametric distribution, such as Laplace, and has the student predict σ for the negative log-likelihood objective.The paper notes that Gaussian distributions could also be used.Source: §4 · p. 3
• AIL weights the imitation loss by normalized teacher loss, giving less relative importance to examples where the teacher is unreliable.The weighting is computed per sample and lets the transfer rely more on examples the teacher predicts well.Source: §4 · p. 4

5. Learning Intermediate Representations

• Standard Hint Training matches a student guided layer to a teacher hint layer, but does not account for incorrect teacher guidance.The paper presents intermediate-representation training as another way to regularize the student and mimic the teacher’s generalization capability.Source: §5 · p. 4
• Attentive Hint Training weights feature matching by normalized teacher loss to down-weight unreliable teacher guidance.The weighted objective matches the teacher and student representations up to their respective hint and guided layers.Source: §5 · p. 4
• The authors train AHT and the main objective in two stages, first up to the guided layer with AHT and then through the remaining student layers with the main objective.They report that separate training yielded superior performance, especially in absolute pose error.Source: §5 · p. 4

6.1. Camera Pose Regression with DNNs

• DNN-based VO regresses 6-DoF camera poses from consecutive images, with translation and rotation losses balanced separately and the rotation represented as Euler angles.The distillation losses are decomposed across translation and rotation, with separate normalization because the teacher-error distributions differ.Source: §6.1 · pp. 4–5, Figure 2 · p. 5

6.2. Network Architecture

• The student retains the teacher’s feature-extractor and regressor building blocks, but the 92.95% distillation-rate design removes five CNN layers and two LSTM layers.The CNN remains initialized with FlowNet weights; added fully connected layers contribute fewer than 1% additional parameters.Source: §6.2 · p. 5, Figure 3 · p. 5

6.3. Training Details

• Training first uses AHT between the teacher’s first fully connected layer and the student’s selected guided layer, then freezes those student weights and trains the remaining layers with the main objective.The student’s third fully connected layer is guided, or its second when the distillation rate is below 75%.Source: §6.3 · p. 5

7.1. Experiment Environments

• The experiments use KITTI Seq 00–08 for training and Seq 09–10 for testing, and use Malaga only for qualitative testing after training on KITTI.Malaga lacks ground truth, so its results are compared qualitatively against GPS data.Source: §7.1 · p. 6

7.2. Metrics

• Accuracy is assessed with RMS translation and rotation RPE and RMS ATE, alongside student-to-teacher parameter percentage, computation time, and model size.Source: §7.2 · p. 6

7.3. Sensitivity Analysis

• AIL achieves the best accuracy in RPE and ATE among the compared loss-blending approaches, while approaches assuming accurate teacher predictions perform worse than the student-loss baseline.PIL has good RPE but larger overall drift; the upper-bound objective balances RPE and ATE but remains inferior to AIL.Source: §7.3 · p. 6, Figure 4 · p. 6
• Hint training sharply reduces the student's latent-representation reconstruction error, and AHT reduces it further by emphasizing teacher representations associated with accurate predictions.AHT's average reconstruction error is lower than HT's even though AHT does not blindly imitate the teacher representation.Source: §7.3 · p. 7, Figure 5 · p. 7

7.4. Trade-off between Accuracy, Model Size, and Execution Time

• At drate = 92.95%, the student stays close to the teacher; computation time falls from 87ms to 41ms (2.12×), while performance degrades more rapidly above that reduction rate.The model size falls from 286.9MB to 7.3MB (2.5%).Source: Table 2 · p. 7, §7.4 · pp. 7–8, Figure 6 · p. 7

7.5. Comparison with Related Works

• On KITTI Seq 09, the proposed distillation gives a trajectory closer to the teacher than competing approaches, and on untrained Malaga sequences its trajectory is closer to GPS than the teacher's.The Malaga evaluation is qualitative because the dataset has no ground truth; all models were trained on KITTI Seq 00–08.Source: Figure 7 · p. 7, §7.5 · p. 8, Figure 9 · p. 8

• Teacher loss serves as an effective attentive mechanism for transferring knowledge from a deep pose regressor to a much smaller network with a small loss of accuracy.Source: §8 · p. 8