Loss Functions and Metrics in Deep Learning

1 Introduction

• The review surveys commonly used loss functions and performance metrics, explaining their strengths, weaknesses, and applications across deep learning tasks.Coverage includes regression and classification, computer vision, natural language processing, and retrieval-augmented generation.Source: §1 · p. 6

1.1 Loss Functions vs. Performance Metrics

• Loss functions optimize model parameters during training, whereas performance metrics evaluate post-training generalization and compare models or configurations.Loss selection depends more on the architecture and task, while metrics are generally more interpretable and less architecture-dependent.Source: §1.1 · pp. 6–7

1.2 Properties of Loss Functions

• Loss-function selection should consider convexity, differentiability, robustness, smoothness, sparsity, and monotonicity to support effective optimization and task-specific behavior.These properties address gradient-based optimization, outliers, sparse outputs, and movement toward the correct solution.Source: §1.2 · p. 7

1.3 Implementation Considerations and Software Libraries

• Modern frameworks such as PyTorch, TensorFlow/Keras, and MATLAB provide standard losses and metrics while supporting custom objectives, and scikit-learn enables standardized assessment of deep-network predictions.These tools support flexible experimentation, metric computation, and comparisons with classical machine-learning methods.Source: §1.3 · pp. 7, 8, 11

2 Regression

• Regression predicts continuous output values from input features and supports applications including finance, healthcare, energy, sports, and engineering.Examples include house-price prediction, energy-consumption forecasting, disease prediction, stock-price forecasting, and customer-lifetime-value prediction.Source: §2 · p. 11

2.1 Regression Loss Functions

• Regression loss selection balances data nature, outlier robustness, interpretability, and optimization smoothness, with model parameters trained by minimizing loss over prediction-target pairs.The reviewed losses include MSE, MAE, Huber, Log-Cosh, quantile, and Poisson losses.Source: §2.1 · p. 11

2.1.1 Mean Squared Error (MSE)

• MSE is smooth, differentiable, and convex in predictions, enabling gradient-based optimization, but its squared-error emphasis makes it sensitive to outliers and target scale.For linear models MSE is also convex in parameters; RMSE or MSPE may improve interpretability across scales, while MAE or Huber may be preferable with outliers [29].Source: §2.1.1 · pp. 11–13

2.1.2 Mean Absolute Error (MAE)

• MAE penalizes errors linearly, making it more robust to outliers than MSE, but its nondifferentiability at zero and lack of smoothness can slow optimization.Subgradient methods can address the nondifferentiability, and scale-invariant metrics such as MAPE or NMAE support inter-problem comparisons.Source: §2.1.2 · p. 13

2.1.3 Huber Loss

• Huber loss combines MSE-like quadratic penalties for errors below δ with MAE-like linear penalties for larger errors, providing controlled robustness and smoothness.The threshold δ can be selected by cross-validation; smaller δ makes the loss more MAE-like, while larger δ makes it more MSE-like, but δ requires tuning.Source: §2.1.3 · pp. 13–14

2.1.4 Log-Cosh Loss

• Log-Cosh offers a fully smooth, continuously differentiable loss that is less sensitive to extreme errors than MSE while retaining stronger emphasis on small deviations than Huber.Its smooth gradient landscape makes it attractive for complex models such as deep networks.Source: §2.1.4 · p. 14

2.1.5 Quantile Loss

• Quantile loss applies asymmetric penalties controlled by q, coincides with MAE at q = 0.5, yields conditional q-th quantiles, and supports interval predictions for uncertainty-aware decision making.When q > 0.5, it penalizes underestimates more strongly, which is useful in safety-critical applications where under-forecasting is risky.Source: §2.1.5 · p. 15

2.1.6 Poisson Loss

• Poisson loss models nonnegative count targets through the Poisson negative log-likelihood, commonly using an exponential link to ensure positive predicted rates.It aligns the objective with discrete count responses while retaining interpretability through expected count rates, with applications including traffic, healthcare, insurance, customer service, internet usage, manufacturing, and crime analysis [46, 47, 48, 49, 50, 51, 52].Source: §2.1.6 · pp. 15–16

2.2.2 Mean Absolute Percentage Error (MAPE)

• MAPE offers an interpretable, scale-independent percentage error for relative-error applications but is sensitive to zero or near-zero targets, outliers, and asymmetric error effects.Finance, retail, and forecasting applications may use epsilon-adjusted MAPE or sMAPE when targets approach zero.Source: §2.2.2 · p. 18

2.2.3 Symmetric Mean Absolute Percentage Error (SMAPE)

• SMAPE balances over- and under-prediction through a symmetric denominator, making it suitable when both error directions have comparable costs, despite continued zero and outlier sensitivities.Applications include inventory, energy, finance, sales, and transportation forecasting.Source: §2.2.3 · pp. 18–19

2.2.4 Coefficient of Determination R2

• R2 measures explained target variation, with values near 1 indicating strong explanatory power, values near 0 resembling mean prediction, and negative values indicating worse-than-mean performance.Interpretation can be less straightforward for nonlinear models, while outliers, small samples, bias, and added predictors can mislead assessment.Source: §2.2.4 · p. 19

2.2.5 Adjusted R2

• Adjusted R2 penalizes additional predictors, supporting fairer comparisons of nested models and reducing overfitting that raw R2 can mask.It may fluctuate substantially for small samples and remains less naturally interpretable for nonlinear or complex models.Source: §2.2.5 · p. 20

3 Classification

• Classification learns a decision function mapping input features to discrete labels in binary, multi-class, or multi-label settings.The section distinguishes binary labels, C-class labels, and assignments of multiple classes to one instance.Source: §3 · p. 20

3.1.1 Binary Cross-Entropy Loss (BCE)

• Binary Cross-Entropy is differentiable, penalizes confident mistakes, and can be weighted to reduce bias toward majority classes.Weighted BCE assigns class-specific weights, often inversely proportional to class frequencies.Source: §3.1.1 · p. 22

3.1.2 Categorical Cross-Entropy Loss (CCE)

• Categorical Cross-Entropy handles one-hot multi-class targets, while Sparse CCE is optimization-equivalent but indexes integer labels more efficiently.CCE is typically paired with softmax outputs and heavily penalizes assigning high probability to an incorrect class; Sparse CCE avoids memory-intensive one-hot encoding.Source: §3.1.2 · p. 22, §3.1.3 · p. 22, Usage · p. 23

3.1.4 Weighted Cross-Entropy (WCE)

• Weighted Cross-Entropy emphasizes minority or costly classes, but excessive weights can destabilize training and fixed weights may degrade under distribution shifts.Weights may be chosen by inverse frequency or application-specific error costs, with normalization used to stabilize gradients.Source: §3.1.4 · pp. 23–24

3.1.5 Cross-Entropy Loss with Label Smoothing

• Label smoothing reduces overconfidence and can improve generalization, although large smoothing factors may blur class distinctions.Typical smoothing factors range from 0.05 to 0.2 and are tuned using validation performance.Source: §3.1.5 · p. 24

3.1.6 Negative Log-Likelihood

• Negative Log-Likelihood is equivalent to cross-entropy for one-hot targets and encourages high probability on the correct class.With softmax outputs, gradient descent adjusts parameters so the probability of the true class approaches 1.Source: §3.1.6 · p. 25

3.1.7 Polyloss

• PolyLoss unifies Cross-Entropy and Focal Loss through polynomial terms, enabling task-specific emphasis on deviations, hard examples, or minority classes.Poly-1 adds one parameterized linear term to Cross-Entropy; setting ϵ = 0 recovers standard Cross-Entropy, while ϵ > 0 increases penalties for confidently wrong predictions.Source: §3.1.7 · p. 25

3.1.8 Hinge Loss

• Hinge Loss promotes maximum margins, while confusion matrices and derived metrics expose correct classifications and error patterns for binary and multi-class models.Accuracy, Precision, Recall, and F1-score derive from TP, TN, FP, and FN; multi-class matrices use diagonal entries for correct predictions and off-diagonal entries for misclassifications.Source: §3.1.8 · pp. 25–26, §3.2.1 · p. 26, §3.2.2 · p. 27

3.2.2 Confusion Matrix in Multi-Class Classification

• A multi-class confusion matrix separates correct diagonal predictions from off-diagonal errors and can reveal targeted confusions, such as Class C being misclassified as Class B more often than A.Such patterns can guide feature engineering or model-capacity adjustments.Source: Table 10 · p. 29, §3.2.2 · p. 29

3.2.3 Accuracy

• Accuracy measures the proportion of correct predictions, but severe or multi-class class imbalance can make it misleading; Balanced Accuracy instead averages per-class Recall equally.A classifier predicting the negative class for every example can achieve 99% accuracy when 99% of samples are negative while missing all positives.Source: §3.2.3 · p. 29

3.2.4 Precision, PPV, or TDR

• Precision measures the correctness of predicted positives, while macro-averaging weights classes equally and micro-averaging aggregates global counts, favoring classes with greater support.Precision is especially relevant when false alarms are costly, but it should be considered alongside Recall because maximizing it can miss actual positives.Source: §3.2.4 · pp. 30–31

3.2.5 Recall, Sensitivity, or True Positive Rate (TPR)

• Recall measures the proportion of actual positives identified, and multi-class reporting requires per-class averaging whose macro or micro choice can materially change results under severe imbalance.Increasing Recall can raise false positives and reduce Precision, so the preferred balance depends on application-specific false-positive and false-negative costs.Source: §3.2.5 · p. 31

3.2.6 F1-Score

• F1 combines Precision and Recall through a harmonic mean that penalizes disparity, while Fβ weights Recall more when β > 1 and Precision more when β < 1.F1 is useful for imbalanced datasets but does not explicitly distinguish the costs of false positives and false negatives.Source: §3.2.6 · pp. 31–32

3.2.7 Specificity or True Negative Rate (TNR)

• Specificity measures correctly identified negatives and complements Recall by characterizing false-positive avoidance, making their joint use informative for balancing error types in high-stakes domains.High Recall with low Specificity produces many false alarms, whereas high Specificity with low Recall misses actual positives.Source: §3.2.7 · p. 32

3.2.8 False Positive Rate (FPR)

• False Positive Rate measures incorrectly positive predictions among actual negatives, varies with the decision threshold, and is plotted against TPR in ROC analysis, where AUC provides a threshold-independent summary.Lower thresholds typically increase false positives and FPR, while higher thresholds tend to reduce them.Source: §3.2.8 · pp. 32–33

3.2.9 Negative Predictive Value (NPV) / 3.2.10 False Discovery Rate (FDR)

• NPV evaluates the correctness of negative predictions and is prevalence-sensitive, whereas FDR evaluates false positives among predicted positives and equals 1 − Precision; both support domain-specific reliability assessment.NPV is important when missed positives are costly, while controlling FDR is important when false alarms or erroneous positive declarations consume resources.Source: §3.2.9 · p. 33, §3.2.10 · p. 34

3.2.11 Precision-Recall Curve

• Precision-recall curves compare models across thresholds, with curves consistently above others indicating better performance and Average Precision (AP) summarizing each curve.Figure 2 illustrates PR curves for three models, with a dashed ideal-performance line.Source: §3.2.11 · p. 35, Figure 2 · p. 35

3.2.12 Precision-Recall Curve in Multi-Class Classification

• Multi-class PR curves use a One-vs-All formulation to evaluate each class against all others, supporting per-class AUC-PR and macro- or micro-averaged summaries.Macro-averaging weights classes equally, whereas micro-averaging weights them by frequency; these curves are especially useful for imbalanced or critical classes but become cumbersome when C is large.Source: §3.2.12 · pp. 35–36

3.2.13 Area Under the Receiver Operating Characteristic Curve (AUC-ROC)

• ROC curves plot TPR against FPR across thresholds, with AUC values ranging from random guessing at 0.5 to perfect separability at 1.0.For rare positive classes, ROC curves can appear overly optimistic because large numbers of true negatives keep FPR low, making precision-recall curves more informative.Source: §3.2.13 · p. 36

3.2.14 Multi-Class Receiver Operating Characteristic Curve (AUC-ROC)

• Multi-class ROC evaluation applies One-vs-All or One-vs-One comparisons, then summarizes per-class or pairwise AUCs using macro, weighted, or micro averaging.One-vs-One produces C(C−1)/2 curves and can become computationally expensive for large C; macro- and micro-averaged AUCs enable simpler model comparisons.Source: §3.2.14 · pp. 37–38

4 Image Classification

• Image classification categorizes an image as a whole into a specific label and supports applications spanning medicine, agriculture, retail, security, environmental monitoring, and manufacturing.Applications include abnormality detection, crop monitoring, product categorization, surveillance, environmental assessment, and automated quality control.Source: §4 · p. 38
• AlexNet achieved substantially better state-of-the-art performance than previous results in 2012, demonstrating deep learning’s ability to outperform hand-engineered machine-learning methods.This breakthrough spurred research and helped establish image classification as a foundation for wider AI advances.Source: §4 · p. 38

4.1 Image Classification Loss Functions

• Image classification uses standard classification losses, while label smoothing, focal loss, weighting, augmentation, and regularization address large-scale, imbalanced, and overconfident predictions.Binary cross-entropy commonly treats labels independently in multi-label image classification, and CNNs or Vision Transformers serve as backbones without changing the fundamental loss computation.Source: §4.1 · pp. 38–39

4.2 Image Classification Metrics

• Image classification retains core metrics but adds top-k accuracy, balanced or per-class measures, precision-recall analysis, confusion matrices, robustness, interpretability, and multi-label precision, recall, and F1.These additions address many similar classes, class imbalance, rare high-impact categories, visual confusions, adversarial or noisy inputs, and images with multiple labels.Source: §4.2 · p. 39

5.1 Object Detection Loss Functions

• Detection models combine classification and bounding-box regression losses to jointly penalize class mistakes and localization errors across diverse data distributions.Typical objectives sum a classification term with an L1/L2-style regression term, weighted by a factor λ.Source: §5.1 · pp. 39–40

5.1.1 Smooth L1 Loss

• Smooth L1 uses quadratic penalties for small deviations and linear penalties for large ones, providing stable gradients and reducing sensitivity to bounding-box outliers.It is widely used in two-stage detectors including Fast R-CNN and Faster R-CNN, with β controlling the transition.Source: §5.1.1 · pp. 40–41

5.1.2 Balanced L1 Loss

• Balanced L1 adaptively applies quadratic-like penalties to small errors and logarithmic penalties to large errors, improving gradient stability when object scales and outliers vary widely.It is particularly useful for heterogeneous scales, unusual aspect ratios, slow convergence, or instability from extreme bounding-box errors.Source: §5.1.2 · pp. 41–42

5.1.3 Intersection Over Union (IoU) Loss

• IoU-based losses optimize box overlap, but standard IoU provides no gradient for non-overlapping boxes and can be insensitive to partial-overlap differences.GIoU addresses non-overlap with more continuous gradients, while DIoU and CIoU additionally model centroid distance and aspect-ratio alignment.Source: §5.1.3 · p. 42, §5.1.4 · p. 42

5.1.6 Focal Loss

• Focal Loss mitigates severe foreground–background imbalance by down-weighting well-classified examples and emphasizing hard or minority-class instances.Its focusing parameter γ typically ranges from 1 to 5, while excessive values may destabilize training; RetinaNet uses it for dense detection.Source: §5.1.6 · p. 44

5.1.7 YOLO Loss

• YOLO Loss combines localization, objectness, and classification terms, with weighting factors balancing box precision, confidence, and class accuracy in grid-based detection.Later YOLO variants replaced scale-sensitive sum-of-squares box terms with IoU- or GIoU-based regression losses.Source: §5.1.7 · p. 45, YOLO Loss · p. 45

5.1.8 Wing Loss

• Wing Loss emphasizes near-accurate predictions through a logarithmic small-error term while dampening large errors, making it applicable to bounding-box regression.It is useful when small positional deviations matter and absolute outliers should not dominate training, including potential medical-imaging applications.Source: §5.1.8 · p. 46

5.2 Object Detection Metrics

• Object detection metrics evaluate localization and classification using TP, FP, and FN definitions based on whether predicted and ground-truth boxes meet an IoU threshold.False positives reduce Precision, false negatives reduce Recall, and true negatives are often omitted because the candidate search space is very large.Source: §5.2 · p. 46

Common IoU Thresholds in Object Detection

• IoU thresholds encode evaluation stringency: IoU ≥0.25 supports high recall, IoU ≥0.5 is widely used, and IoU ≥0.75 demands precise localization.These choices correspond respectively to settings such as medical imaging, Pascal VOC, and safety-critical applications such as autonomous driving.Source: Common IoU Thresholds in Object… · p. 46

5.2.1 Average Precision (AP)

• Average Precision measures per-category detection performance and averages categories into mAP, preventing frequent classes from obscuring weaker categories.AP uses an IoU threshold τ, while COCO evaluates multiple thresholds from 0.5 through 0.95 to assess lenient and strict localization.Source: §5.2.1 · p. 47

Pascal VOC AP Computation

• Pascal VOC computes AP by matching detections using IoU, constructing confidence-ranked precision-recall curves, applying 11-point interpolation, and summing the interpolated area.The VOC protocol uses recall levels from 0 to 1 in increments of 0.1.Source: Pascal VOC AP Computation · p. 47

Microsoft COCO AP Computation

• For COCO, AP uses 101-point interpolation over recall levels, averages AP across IoU thresholds 0.50–0.95 and categories, and can be reported separately by object size.Source: Microsoft COCO AP Computation · p. 47

5.2.2 Average Recall (AR)

• Average Recall (AR) summarizes how well a model detects all objects, measuring the proportion of ground-truth objects found across possible per-image detection limits and averaged over multiple IoU thresholds.Source: §5.2.2 · pp. 47–48

6 Image Segmentation

• Segmentation labels each pixel and includes semantic, instance, and panoptic variants used in scene understanding, medical imaging, robotics, autonomous vehicles, surveillance, and augmented reality.Semantic segmentation assigns shared class labels, instance segmentation separates object occurrences, and panoptic segmentation combines both.Source: §6 · pp. 48–49

6.1.1 Cross-Entropy Loss for Segmentation

• Cross-Entropy and Binary Cross-Entropy provide efficient per-pixel probabilistic classification, but standard forms can underweight rare foreground classes under severe imbalance.Weighting schemes, Focal Loss, and region-based losses such as Dice Loss are identified as remedies.Source: §6.1.1 · pp. 49–50, §6.1 · p. 49

6.1.2 Dice Loss

• Dice Loss directly optimizes predicted-to-ground-truth region overlap, making it effective for rare small objects and improving sensitivity to boundaries and shape consistency.Its overlap formulation de-emphasizes large background areas and is particularly relevant when mask overlap matters more than independent pixel classification [177].Source: §6.1.2 · pp. 50–51

6.1.3 Intersection Over Union (IoU) Loss for Segmentation

• IoU Loss aligns training with a widely used segmentation benchmark metric [180] by optimizing overlap, but weak gradients near zero overlap can motivate combinations with pixel-wise or Dice losses.Multi-class or large-image extensions can also be computationally involved when per-class IoU is aggregated naively.Source: §6.1.3 · p. 51

6.1.4 Tversky Loss

• Tversky Loss generalizes Dice Loss through α and β, allowing practitioners to customize false-positive and false-negative penalties for imbalanced or high-stakes segmentation.Setting α = β = 0.5 recovers Dice Loss, while changing the weights can emphasize missed positives.Source: §6.1.4 · p. 52

6.1.5 Lovász Loss

• Lovász Loss provides a smooth, differentiable surrogate that directly optimizes IoU and often outperforms purely pixel-level losses when class imbalance or small-object segmentation is important.Its convex Lovász extension approximates the discrete Jaccard index for gradient-based optimization.Source: §6.1.5 · pp. 52–53

6.1.6 Focal Loss for Segmentation

• Focal Loss down-weights easy pixels and uses α and γ to emphasize minority, difficult, or ambiguous regions, making it useful for highly imbalanced segmentation.It has been adopted from object detection for semantic segmentation, including cases with small foreground structures such as tumors.Source: §6.1.6 · p. 53

6.2 Segmentation Metrics

• Segmentation evaluation combines pixel accuracy with overlap and boundary measures: pixel accuracy is fast but can hide minority-class failures, whereas Boundary F1 evaluates predicted-boundary alignment using thresholded matching.IoU or Dice are commonly paired with pixel accuracy for more robust assessment under class imbalance, while Boundary F1 computes precision and recall from matched boundary points.Source: §6.2.1 · p. 54, §6.2.2 · pp. 54–55

6.2.2 Boundary F1 Score (BF)

• Boundary F1 Score (BF) emphasizes predicted-boundary alignment, with the distance threshold δ controlling matching strictness, but should accompany region metrics such as IoU or Dice.BF is suited to applications where accurate object outlines are critical, including medical image analysis of organs or lesions.Source: §6.2.2 · p. 55

6.2.3 Masked Average Precision (Mask AP)

• Masked Average Precision (Mask AP) evaluates instance segmentation by matching confidence-ranked predicted masks to category-matched ground-truth masks above IoU thresholds and averaging precision across thresholds.Common reports include AP50, AP75, and mean AP over IoU thresholds from 0.50 to 0.95; strict thresholds, confidence calibration, and crowded scenes can affect scores.Source: §6.2.3 · pp. 55–56

6.2.4 Panoptic Quality (PQ)

• Panoptic Quality (PQ) unifies semantic and instance segmentation by balancing pixel-level segmentation accuracy with instance recognition, making it useful for overlapping objects and background classes.PQ combines Segmentation Quality, which averages IoU over matched segments, with Recognition Quality, which penalizes false and missed instances; higher PQ indicates better performance.Source: §6.2.4 · pp. 56–57

7 Face Recognition

• Face recognition matches an individual’s face in an image or video to a corresponding identity in a face database using learned discriminative features.CNNs and Transformers are trained on large face-image datasets, supporting applications in security, social media, and biometric identification.Source: §7 · p. 57

7.1 Face Recognition Loss Functions and Metrics

• Face-recognition objectives include classification-based losses such as Softmax, A-Softmax, Center, Large-Margin Cosine, and Additive Angular Margin losses, alongside representation-based Triplet, Contrastive, Circle, and Barlow Twins losses.These losses focus on preserving fine-grained relational structure among face embeddings.Source: §7.1 · p. 57
• Face-recognition evaluation uses accuracy, precision, recall, F1-score, and ROC curves, supplemented by verification and identification protocols.Table 15 summarizes loss-function selection according to usage, data characteristics, advantages, and limitations.Source: §7.1 · p. 57, Table 15 · p. 58

7.1.1 Softmax Loss

• Softmax Loss classifies identity logits but does not explicitly enforce inter-class margins or compress intra-class variation, limiting discrimination among visually similar faces.Margin-based or representation-learning objectives are therefore often used to improve embedding-space separation.Source: §7.1.1 · pp. 57–58

7.1.2 A-Softmax (SphereFace) Loss

• A-Softmax (SphereFace) Loss normalizes weights and features so classification depends on angular relationships, then replaces cos(θ_y) with cos(mθ_y) for the true class.The angular margin imposes a stricter correct-classification requirement by forcing a smaller angle to the true-class vector.Source: §7.1.2 · pp. 58–59

4. Scaled Logits: The final logits become

• A-Softmax uses cos(mθ) to tighten intra-class clustering and enlarge inter-class angular separation, reducing confusion among visually similar faces and improving recognition metrics.Larger m strengthens separation but can hinder optimization, while an additional scaling factor s can stabilize training.Source: §4 · p. 59

7.1.3 Center Loss

• Center Loss learns identity-specific centers alongside classification loss, reducing intra-class variation and producing tightly clustered, discriminative face embeddings.The balance parameter λ controls classification versus clustering, while exponential-moving-average center updates avoid abrupt fluctuations.Source: §7.1.3 · pp. 59–60

7.1.4 CosFace: Large-Margin Cosine Loss

• CosFace subtracts a margin from the correct-class cosine similarity after normalization, often improving generalization and discriminability over standard softmax or SphereFace.Direct cosine-margin adjustment simplifies optimization and prevents trivial norm increases from replacing angular discrimination.Source: §7.1.4 · p. 60

7.1.5 ArcFace: Additive Angular Margin Loss

• ArcFace [193] adds an angular margin that improves hyperspherical separation, interpretability, and face verification and identification performance under challenging conditions.Its additive angular boundary is monotonic and typically outperforms earlier margin-based losses such as SphereFace and CosFace on tasks including LFW and MegaFace.Source: §7.1.5 · pp. 60–61

7.1.6 Triplet Loss / 7.1.7 Contrastive Loss

• Triplet Loss [195, 196] and Contrastive Loss [199] learn shared embedding spaces by pulling same-identity samples together and pushing different identities apart beyond margins.Triplet effectiveness depends on informative negative mining, whereas Contrastive Loss supports adding new identities without retraining a classifier but treats negatives uniformly.Source: §7.1.6 · p. 61, §7.1.7 · p. 62

7.1.8 Circle Loss

• Circle Loss jointly optimizes positive and negative similarities with adaptive slack and scaling, supporting robust separation under complex inter-class relationships or skewed distributions.Its circle-like similarity boundary focuses penalties on margin-violating pairs, but overly large margins or scaling can destabilize training.Source: §7.1.8 · pp. 62–63

7.1.9 Barlow Twins Loss / 7.1.10 SimSiam Loss

• Barlow Twins and SimSiam learn self-supervised facial representations from augmented views without negative pairs, but require suitable augmentations and careful training choices.Barlow Twins reduces feature redundancy and may need large embeddings, while SimSiam uses stop-gradient to prevent collapse and can be refined with ArcFace or CosFace.Source: §7.1.9 · p. 63, §7.1.10 · pp. 63–64

8 Monocular Depth Estimation (MDE)

• Deep learning has improved monocular depth estimation, while real-world systems still must handle varied lighting, occlusions, computational demands, and scalability.Supervised, self-supervised, and unsupervised approaches use image sequences or stereo pairs, and current research also incorporates geometric and physics-based constraints.Source: §8 · p. 64

8.1 Depth Estimation Loss Functions

• Modern MDE losses balance global depth accuracy with fine-grained structural detail through composite objectives incorporating scale, photometric, smoothness, semantic, and boundary-aware priors.This evolution targets depth accuracy, structural consistency, robustness to noisy data, generalization across datasets, and computational efficiency.Source: §8.1 · p. 64

8.1.1 Point-wise Error

• Point-wise MAE is relatively robust to moderate outliers, whereas MSE emphasizes large errors and can rapidly correct major depth misestimates during early training.Point-wise errors minimize pixel-level residuals but miss structural and relative-depth relationships, so they are often combined with SSIM or smoothness constraints; this approach was used in and subsequent MDE works.Source: §8.1.1 · p. 66

8.1.2 Scale Invariant Error

• Scale-invariant error prioritizes relative depth consistency in log space and mitigates monocular scale ambiguity, but absolute metric scale requires additional constraints or multi-view geometry.Eigen et al. proposed this formulation; λ controls the global log-offset penalty, with λ = 1 reproducing the original formulation.Source: §8.1.2 · p. 66

8.1.3 Structural Similarity Index Measure (SSIM)

• SSIM encourages predicted depth maps to exhibit structurally consistent edges and gradients relative to reference depth, and is often combined with ℓ1, MSE, or smoothness losses.SSIM ranges from −1 to 1, with 1 indicating perfect structural alignment.Source: §8.1.3 · p. 67

8.1.4 Photometric Loss

• Photometric and appearance-matching losses enable self-supervised depth learning by comparing reference images with views warped using predicted depth and camera geometry.Photometric consistency combines intensity differences with SSIM, while appearance matching commonly uses a weighted ℓ1–SSIM objective with α = 0.85; occlusions, motion, and reflectance changes require masking, minimum reprojection, or auto-masking.Source: §8.1.4 · pp. 67–68, §8.1.6 · p. 68

8.1.5 Disparity Smoothness Loss

• Disparity smoothness regularizes depth to be piecewise smooth in homogeneous regions while reducing penalties near image edges, and is combined with photometric, point-wise, or scale-invariant terms.Log-scaled and multi-scale variants address wide depth ranges and coherence across resolutions.Source: §8.1.5 · p. 68

8.1.7 Left-Right Consistency Loss / 8.1.8 BerHu Loss (Reverse Huber)

• Left-right consistency removes dependence on labeled depth by enforcing agreement between stereo disparities, while BerHu balances linear penalties for smaller errors with quadratic penalties for large ones.Left-right consistency complements appearance matching and smoothness, whereas BerHu can suit wide depth ranges but depends on threshold tuning and lacks inherent structural priors; introduces the combined stereo pipeline and [208] describe BerHu.Source: §8.1.7 · p. 69, §8.1.8 · pp. 69–70

8.1.10 Minimum Reprojection Loss

• Minimum Reprojection Loss selects the lowest photometric error across source views, reducing penalties from occlusions, dynamic objects, and invalid reprojections.It is commonly paired with smoothness, edge-aware, or geometric-consistency constraints, although reliance on minima can oversmooth regions with multiple valid views.Source: §8.1.10 · p. 71

8.1.11 Scale-and-shift-invariant Loss (SSI)

• SSI Loss aligns predicted and ground-truth disparities by scale and shift, improving robustness, cross-dataset generalization, and numerical stability across heterogeneous depth annotations.Its trade-offs include per-sample alignment overhead, sensitivity to poor estimates, and possible degradation from extreme ground-truth errors.Source: §8.1.11 · pp. 71–72
• Combining SSI Loss with multi-scale gradient matching preserves global alignment while sharpening object boundaries and local details, often improving zero-shot performance on unseen datasets.The parameter α controls the balance between global alignment and high-frequency refinement.Source: §8.1.11 · p. 73

8.2 Depth Estimation Metrics

• Depth-estimation evaluation should combine metrics because each captures distinct dimensions of absolute error, relative error, pixel accuracy, ordinal agreement, or depth-shape consistency.Metric choice depends on the application and dataset, including absolute-depth, ordinal, varying-scale, and relative-depth settings.Source: §8.2 · p. 73

8.2.1 Mean Absolute Relative Error (AbsRel)

• AbsRel emphasizes errors at shallow depths, whereas RMSE penalizes large residuals more strongly but can be unstable near zero or across wide depth ranges.AbsRel may therefore be complemented by RMSE when depth ranges are broad or values approach zero.Source: §8.2.1 · p. 73, §8.2.2 · p. 73

8.2.3 Logarithmic RMSE (RMSE(log))

• RMSE(log) and Mean Log10 Error emphasize relative or multiplicative discrepancies, supporting mixed near- and far-field data but complicating interpretation near zero.Log-space measures can be less intuitive than linear errors and may suffer numerical issues for depths near zero.Source: §8.2.3 · p. 74, §8.2.5 · p. 75

8.2.4 Threshold Accuracy (δ)

• Threshold Accuracy δ reports the fraction of pixels within multiplicative bounds, while high-error percentages identify pixels exceeding chosen relative-error thresholds for stricter or safety-oriented assessment.Common δ thresholds are δ1 = 1.25, δ2 = 1.252, and δ3 = 1.253; high-error thresholds may be τ = 0.1, 0.2, or 0.5.Source: §8.2.4 · pp. 74–75, §8.2.6 · p. 75

8.2.7 Weighted Human Disagreement Rate (WHDR)

• WHDR evaluates ordinal closer, same, or farther judgments when metric depths are unavailable, whereas Scale-Invariant Error emphasizes relative shape but omits uniform global-scale accuracy.WHDR depends on human annotation quality and does not assess metric scale, motivating complementary absolute-based metrics.Source: §8.2.7 · pp. 75–76, §8.2.8 · p. 76

9 Image Generation

• Image generation includes VAEs, GANs, Normalizing Flows, EBMs, and Diffusion Models, supporting applications such as super-resolution, denoising, inpainting, and style transfer.VAEs learn latent representations; GANs use generator–discriminator competition; flows use invertible transformations; EBMs learn energy functions; diffusion models transform noise into target data.Source: §9 · pp. 76–77

9.1 Image Generation Loss Functions

• Image-generation losses pair reconstruction and KL terms for VAEs, adversarial or Wasserstein objectives for GANs, likelihood for flows, energy-based criteria for EBMs, and denoising errors for diffusion models.VAE reconstruction preserves input fidelity while KL regularizes the latent distribution; WGAN uses Wasserstein distance to improve stability and avoid mode collapse.Source: §9.1 · p. 77

9.1.1 Reconstruction Loss

• Reconstruction loss measures input–output fidelity, using MSE for continuous images or BCE for binary and probabilistic outputs, but MSE can blur details and miss structural semantics.In VAEs it is combined with KL divergence; restoration tasks use it to recover detail, while perceptual or adversarial terms can improve realism and structural fidelity.Source: §9.1.1 · pp. 77–79

9.1.2 Kullback–Leibler Divergence Loss

• KL divergence aligns a learned distribution q with a target p, but its direction affects optimization and zero probabilities can cause undefined or infinite terms.A small ε, such as 10^-7, is often added to both distributions; VAEs use KL to match latent representations to a prior.Source: §9.1.2 · p. 79

9.1.3 Perceptual Loss

• Perceptual loss compares deep-network feature activations rather than pixels, improving sharpness, semantic structure, and visual coherence while sacrificing pixel-perfect alignment and inheriting pretrained-network bias.It helps super-resolution and deblurring recover sharp edges and fine structures, and can be combined with reconstruction or adversarial objectives.Source: §9.1.3 · p. 80

9.1.4 Adversarial Loss

• Adversarial loss trains a generator and discriminator through alternating minimax updates, producing realistic samples but risking mode collapse, instability, and non-guaranteed convergence.Under ideal conditions, training reaches a Nash equilibrium where generated outputs are indistinguishable from real data; gradient penalties or spectral normalization can stabilize training.Source: §9.1.4 · p. 81

9.1.5 Wasserstein Loss

• Wasserstein loss replaces KL or JS distribution matching with Earth Mover’s Distance, providing smoother, more stable gradients for moving the generated distribution toward real data.Its optimal-transport formulation measures the minimum cost of morphing one distribution into another.Source: §9.1.5 · p. 82
• Wasserstein loss provides smoother gradients, mitigates mode collapse, and supports more stable GAN convergence, although Lipschitz enforcement and hyperparameter tuning add overhead.WGAN-GP replaces discrete weight clipping with a gradient penalty enforcing the 1-Lipschitz condition, improving training stability.Source: §9.1.5 · p. 82

9.1.6 Negative Log-Likelihood in Normalizing Flows

• Negative log-likelihood in normalizing flows enables exact density estimation through invertible transformations and tractable Jacobian determinants, but requires greater computational and memory cost.Architectures such as RealNVP [226] and Glow simplify determinant computation using invertible layers, while coupling transformations can represent complex multimodal distributions.Source: §9.1.6 · p. 83

9.1.7 Contrastive Divergence

• Contrastive Divergence makes energy-based image generation practical by estimating the negative phase with short MCMC chains, trading exactness for efficiency and biased gradients.Increasing the chain length k can reduce bias but increases computational cost; PCD and mean-field methods provide alternative approximations.Source: §9.1.7 · p. 84

9.2 Image Generation Metrics

• Image-generation evaluation combines pixel-level fidelity, perceptual similarity, recognizability, diversity, and feature-distribution alignment through PSNR, SSIM, IS, and FID.These metrics target different aspects of generated-image quality rather than a single universal evaluation objective.Source: §9.2 · p. 85

9.2.1 Peak Signal-to-Noise Ratio (PSNR)

• PSNR is a simple reference-based measure where higher values indicate fewer pixel-wise deviations, but it is less informative for unconstrained generation than IS [273] or FID [274].It remains useful for super-resolution, denoising, and other tasks with known ground-truth images.Source: §9.2.1 · p. 85

9.2.2 Structural Similarity Index (SSIM)

• SSIM better aligns with human perception than pixel-wise measures by modeling luminance, contrast, and structure, but requires references and cannot capture diversity or distribution coverage.It is therefore useful for reference-based restoration tasks and often paired with IS or FID for generative evaluation.Source: §9.2.2 · p. 86

9.2.3 Inception Score (IS)

• Inception Score rewards recognizable samples and class diversity, but depends on ImageNet-trained features, may diverge from human realism judgments, and can overlook within-class mode collapse.Higher IS values indicate better generative performance under its classifier-based formulation.Source: §9.2.3 · pp. 86–87
• Inception Score remains a straightforward heuristic for image quality and diversity, but it does not penalize class imbalance when coverage is broad and requires many samples for stable, low-variance estimates.Practitioners often complement it with FID or user studies, especially when labels align with ImageNet concepts, to assess perceptual fidelity and distribution coverage more thoroughly.Source: §9.2.3 · p. 87

9.2.4 Fréchet Inception Distance (FID)

• FID’s reliability is limited by dependence on ImageNet-trained features, the Gaussian assumption for activations, and the need for large real and generated sample sets.These conditions can be problematic for domains far from natural images, non-Gaussian feature distributions, or small datasets.Source: §9.2.4 · p. 88
• FID compares real and generated feature-distribution means and covariances to assess both image realism and diversity, capturing shifts that Inception Score may miss.It extracts features with a fixed Inception network and models them as multivariate Gaussians; mean distance reflects distribution-center shifts, while covariance terms reflect spread.Source: §9.2.4 · p. 88
• FID has become a standard vision metric because it more robustly assesses distributional fidelity than single-sample or classification-based metrics and is less sensitive to recognizable-image noise.Its robustness comes from penalizing discrepancies in feature-space location and spread, while still allowing subtle style or texture shifts to affect the score.Source: §9.2.4 · p. 88

10 Natural Language Processing (NLP)

• NLP covers text classification, language modeling, machine translation, named entity recognition, part-of-speech tagging, summarization, and question answering.Source: §10 · p. 89

10.1 Loss Functions Used in NLP

• NLP tasks require specialized losses and metrics because language objectives involve discriminative classification, sequential dependencies, large vocabularies, and exposure bias.Source: §10 · p. 89, §10.1 · p. 89
• Selecting among NLP losses requires balancing training efficiency, interpretability, sequential or geometric objectives, and alignment with the desired linguistic or functional outcome.Table 19 organizes these choices by usage, data characteristics, advantages, and limitations.Source: §10.1 · p. 89

10.1.1 Cross-Entropy Loss (Token-Level)

• Token-level Cross-Entropy remains the primary supervised objective, increasing probability on correct tokens and supporting language modeling, translation, summarization, and token labeling.It provides direct per-token supervision and stable training, but may miss global sequence properties, diverge from BLEU or ROUGE, and create exposure bias under teacher forcing.Source: §10.1.1 · pp. 90–91

10.1.2 Hinge Loss

• Hinge Loss uses margin-based separation for binary classification, structured sequence labeling, sparse lexical models, and adaptive precision–recall or adversarial objectives.Its strengths include robust max-margin classification and sparse structured models, while multiclass extensions, absent probability estimates, and margin tuning complicate use.Source: §10.1.2 · pp. 91–92

10.1.3 Cosine Similarity Loss

• Cosine Similarity Loss aligns embedding geometry with semantic closeness for semantic textual similarity, paraphrase identification, retrieval, ranking, and clustering.It requires continuous embeddings and careful handling of negative pairs, margins, and zero-similarity ambiguity.Source: §10.1.3 · pp. 92–93

10.1.4 Marginal Ranking Loss

• Marginal Ranking Loss trains models to score relevant text above irrelevant text in retrieval, question answering, recommendation, and paraphrase detection.Its pairwise margin formulation faces margin-tuning, negative-sampling or computational-cost, subtle-relevance, and global-ordering challenges.Source: §10.1.4 · pp. 93–94

10.2 Losses for Sequence Generation

• Sequence-generation tasks predict token sequences for translation, summarization, and dialogue while balancing fluency, coherence, and faithful reflection of intended meaning.The section examines losses designed for these sequence-level challenges.Source: §10.2 · p. 94

10.2.1 Connectionist Temporal Classification (CTC) Loss

• CTC learns sequence mappings without manual alignments by summing over valid blank-inclusive paths, enabling tractable training for monotonic tasks such as speech and character recognition [296].Dynamic programming computes the alignment probability and gradient efficiently, but strict monotonicity limits tasks involving input-output reordering.Source: §10.2.1 · pp. 94–95

10.2.2 Minimum Risk Training (MRT)

• MRT directly optimizes sequence-level metrics such as BLEU and ROUGE, often improving over token-level cross-entropy but requiring careful sampling, metric design, and hyperparameter tuning.It minimizes expected sequence-level risk, assigning higher probability to outputs with lower metric-based discrepancy.Source: §10.2.2 · pp. 95–96

10.2.3 REINFORCE Algorithm

• REINFORCE treats sequence generation as a stochastic policy, directly optimizing non-differentiable rewards such as BLEU or ROUGE while requiring variance reduction, efficient sampling, and delayed-credit handling.Higher-reward sequences receive increased probability, but noisy gradients, large action spaces, and sequence-level rewards can hinder stable training.Source: §10.2.3 · pp. 96–97

10.3.1 Accuracy

• Accuracy measures correct predictions in NLP classification, sequence labeling, and language detection, but can mislead on imbalanced data and ignore contextual or fine-grained quality.It is simple and interpretable, yet practitioners often complement it with precision, recall, F1-score, or specialized measures.Source: §10.3.1 · pp. 97–99

10.3.2 Precision, Recall, and F1 Score

• Precision, recall, and F1 score evaluate classification and labeling by distinguishing false-positive and false-negative costs, with particular value under class imbalance and cost-sensitive conditions.F1 combines precision and recall, while semantic near-misses and generative sequence coherence may require domain-specific or complementary metrics.Source: §10.3.2 · pp. 99–100

10.3.3 AUC-ROC

• AUC-ROC summarizes binary NLP classifier discrimination across thresholds, supporting probabilistic tasks such as spam detection, sentiment analysis, and topic filtering.It is threshold-independent and useful with skewed classes, but rare-positive or cost-sensitive settings may also require precision-recall AUC.Source: §10.3.3 · p. 100

10.3.4 BLEU Score

• BLEU evaluates generated text through reference-based n-gram overlap and brevity control, but lexical and single-reference dependence can undervalue valid paraphrases or semantic adequacy.It remains a standard baseline for translation and related generation tasks, often supplemented by METEOR, ROUGE, or human evaluation.Source: §10.3.4 · p. 101

10.3.5 METEOR

• METEOR evaluates generation using aligned exact, stem, synonym, and paraphrase matches, combining unigram precision and recall with a fragmentation penalty.Its broader matching and ordering considerations are intended to better reflect nuanced outputs and correlate more closely with human judgments than BLEU.Source: §10.3.5 · p. 102

METEOR =

• METEOR improves on simple n-gram matching by balancing precision, recall, and fragmentation while rewarding synonym and morphological matches, often correlating better with human evaluations than BLEU.It supports machine translation, summarization, dialogue, paraphrase, and other reference-based generation tasks, but remains limited in global coherence and deep semantics.Source: METEOR = · p. 102

10.3.6 ROUGE Score

• ROUGE measures reference-content recall through n-gram, longest-common-subsequence, weighted, and skip-bigram overlaps, making it a mainstay for summarization and coverage-oriented generation evaluation.Its surface-level recall can miss paraphrases, depend on reference diversity, and over-reward lengthy candidates containing extraneous material.Source: §10.3.6 · p. 103

10.3.7 Perplexity

• Perplexity evaluates language-model predictive uncertainty, with lower values indicating more confident token prediction and typically stronger capture of linguistic regularities.Scores are meaningful only across comparable domains, test sets, and vocabulary settings, and should complement sequence-level or semantic metrics because calibration can mislead.Source: §10.3.7 · p. 104

10.3.8 Exact Match (EM)

• Exact Match assigns credit only when the prediction exactly equals the reference, making it reliable for extractive QA, immutable labels, code, and other strictly formatted outputs [316].Its binary, semantically insensitive scoring provides no partial credit, so paraphrase-tolerant or partially correct tasks often require additional metrics or human evaluation.Source: §10.3.8 · pp. 104–105

10.3.9 Word Error Rate (WER)

• WER quantifies substitutions, deletions, and insertions needed to transform a hypothesis into a reference, serving as an interpretable benchmark for speech recognition and related transcription systems.It does not recognize semantic equivalence, weights error types identically, and can be inflated by uncommon or domain-specific vocabulary.Source: §10.3.9 · pp. 105–106

10.3.10 Character Error Rate (CER)

• CER applies normalized edit-distance evaluation at the character or subword level, providing finer and more language-agnostic assessment when word segmentation is ambiguous or artificial.It is useful for character-based languages, low-resource scripts, and handwriting recognition, but treats all character differences equally and ignores semantic severity.Source: §10.3.10 · pp. 106–107

11 Retrieval-Augmented Generation (RAG)

• RAG retrieves external passages, augments the language-model input, and generates context-aware responses, reducing hallucinations and outdated knowledge while supporting specialized and evolving information needs [318] [320].Its workflow consists of retrieval, generation, and augmentation; applications include open-domain question answering, legal drafting, and specialized advisory systems.Source: §11 · p. 107

11.1 Loss Functions Used in RAG

• Fine-tuning is optional for many RAG systems but can improve domain adaptation, retrieval ranking, token generation, and coordination through cross-entropy or NLL, contrastive or marginal ranking, and KL-divergence losses.Cross-entropy and NLL target generation; contrastive and marginal ranking losses target retrieval; KL divergence aligns teacher-student or multi-component distributions.Source: §11.1 · pp. 107–108

11.2 Performance Metrics Used in RAG

• RAG evaluation must separately assess retrieval quality and generation quality because references may be unavailable, motivating reference-free or partially reference-based frameworks such as RAGAS [323] and ARES [324].Important considerations include reference-free evaluation, faithfulness to retrieved context, and focused retrieval that avoids irrelevant or noisy passages [323].Source: §11.2 · p. 108

11.2.1 Answer Semantic Similarity / 11.2.2 Answer Correctness

• Answer Semantic Similarity compares generated and reference answers semantically, while Answer Correctness combines semantic similarity with token-level F1 to reward accurate rephrasings and penalize missing or incorrect tokens [323].Answer Correctness uses a weighting factor α to balance semantic similarity and lexical overlap.Source: §11.2.1 · pp. 108, 110, §11.2.2 · p. 110

11.2.3 Answer Relevance / 11.2.4 Context Precision / 11.2.5 Context Recall

• Answer Relevance measures topicality and completeness independently of factual accuracy, whereas Context Precision and Context Recall assess whether relevant evidence is ranked early and sufficiently covers ground-truth statements [323, 326, 327, 328].Answer Relevance can score factually incorrect outputs highly; Context Precision emphasizes top-ranked chunks but lacks recall, while Context Recall evaluates retrieval completeness rather than generation correctness.Source: §11.2.3 · p. 110, §11.2.4 · p. 111, §11.2.5 · p. 111

11.2.6 Faithfulness / 11.2.7 Summarization Score / 11.2.8 Context Entities Recall / 11.2.9 Aspect Critique

• Faithfulness Score ranges from 0 to 1; the summarization metric balances completeness and brevity, entity recall tracks reappearing significant entities, and Aspect Critique gives binary compliance feedback.A Faithfulness Score of 1.0 indicates every response statement is context-grounded. Entity identification can be inaccurate, and synonyms or paraphrases may not match exactly.Source: §11.2.6 · p. 112, §11.2.7 · p. 112, §11.2.8 · p. 113

11.2.9 Aspect Critique

• Aspect critiques can be customized for domain-specific concerns, but their LLM-based judgments may be biased or variable and complex aspects may require deeper validation.Applications include automatically identifying harmful content and checking policy compliance in finance or healthcare.Source: §11.2.9 · pp. 113–114

11.2.10 Context Relevance

• Context relevance measures whether retrieved passages pertain to a query using a contrastively trained DeBERTa-v3-Large [332] judge and PPI-based confidence intervals.ARES trains on synthetic positive and negative query-passage pairs, including weak random negatives and strong same-document or BM25 negatives, then scores query-passage-answer triples.Source: §11.2.10 · p. 114

11.2.11 Answer faithfulness

• Answer faithfulness evaluates whether responses are grounded in retrieved context by classifying triples as faithful or unfaithful with a fine-tuned DeBERTa-v3-Large [332] judge and PPI.Synthetic training data includes grounded and hallucinated or contradictory answers, with weak negatives sampled from other passages and strong contradictory negatives generated by prompting an LLM.Source: §11.2.11 · pp. 114–115

11.2.12 Answer Relevance

• Answer relevance assesses whether a response addresses the original query using retrieved information through a contrastively trained LM judge and PPI-derived confidence intervals.ARES uses synthetic relevant and irrelevant examples, weak unrelated answers, and strong answers that fail to address or contradict the query and passage.Source: §11.2.12 · p. 115

12 Combining Multiple Loss Functions

• Weighted combinations of losses optimize multiple objectives simultaneously, but auxiliary terms can overshadow the primary objective or provide little benefit when underweighted.A common formulation is Ltotal = λ1 L1 + λ2 L2 + · · · + λk Lk, with nonnegative λi weights tuned empirically or using GradNorm and uncertainty weighting.Source: §12 · pp. 115–116
• Multi-loss designs pair complementary objectives across applications, including localization and classification in object detection, adversarial and auxiliary losses in GANs, and token- and sequence-level losses in language modeling.These combinations target spatial and semantic accuracy, image realism, content or style control, and coherent sequence-level quality while reducing exposure bias [335].Source: §12 · p. 116
• Reconstruction and perceptual losses jointly preserve low-level fidelity and high-level feature consistency in image-to-image tasks such as super-resolution and style transfer.Pixel-level MSE supports sharp edges and stable colors, while pretrained-CNN features promote texture consistency and semantic alignment.Source: §12 · p. 116
• Combining cross-entropy with Dice or IoU losses addresses class imbalance and overlap in segmentation, while retrieval and generation losses help RAG systems use external knowledge for high-fidelity responses.The RAG setup pairs contrastive or ranking loss for retrieval with sequence modeling loss for text generation.Source: §12 · p. 116
• Conflicting gradients can destabilize training and create loss competition, motivating adaptive reweighting, multi-objective optimization, and automated discovery or tuning of loss weights.As tasks become more multifaceted, optimal loss combinations must incorporate task-specific constraints, domain knowledge, and trade-offs among objectives.Source: §12 · p. 116

Data Limitations and Imbalances; Addressing Class Imbalance and Data Scarcity

• Weighted Binary Cross-Entropy and Focal Loss address expensive annotation and class imbalance by emphasizing harder or minority samples, while retrieval systems use contrastive or ranking-based losses for relevant documents.Traditional cross-entropy and MSE treat instances equally and can overfit majority classes; specialized weighting focuses learning on underrepresented examples and relevant retrieval targets.Source: Data Limitations and Imbalances · p. 117, Addressing Class Imbalance and Data… · p. 117

Sensitivity to Outliers and Noise; Robustness to Outliers

• Huber, Smooth L1, and Log-Cosh reduce sensitivity to outliers and noisy data by limiting extreme-error influence, supporting stability in heavy-tailed forecasting, sensor, and healthcare applications.MSE disproportionately penalizes large errors, whereas robust alternatives transition or behave less aggressively around extreme values to reduce overfitting.Source: Sensitivity to Outliers and Noise · p. 117, Robustness to Outliers · p. 117

Exposure Bias and Sequence-Level Learning; Sequence-Level Training and Global Metrics

• Minimum Risk Training and reinforcement learning optimize BLEU, ROUGE, or METEOR directly, addressing exposure bias and improving sequence-level fluency, semantic alignment, and global properties beyond token-level prediction.Next-token training uses ground-truth prefixes, whereas inference uses generated prefixes, allowing errors to accumulate across sequences.Source: Exposure Bias and Sequence-Level… · p. 117, Sequence-Level Training and Global… · p. 117

Complex Evaluation Requirements; Multi-Task and Multi-Loss Objectives

• Sophisticated tasks require multi-faceted evaluation and multi-loss objectives, combining localization and classification losses or adversarial, reconstruction, and perceptual losses to balance distinct quality dimensions.Examples include faithfulness versus relevance in retrieval-augmented generation and bounding-box precision versus recall in detection.Source: Complex Evaluation Requirements · p. 117, Multi-Task and Multi-Loss Objectives · p. 117

Automated Loss Function Search; Continual and Online Learning Scenarios; Robustness and Safety in Deployment

• Emerging research explores automated loss-function search, online adaptation to distribution shifts, and safety-oriented metrics that assess fairness, interpretability, reliability, and harmful behavior.Approaches include gradient-based meta-learning or evolutionary search, online surrogates and incremental evaluation, and deployment-specific measures for medical, autonomous, legal, and retrieval systems.Source: Automated Loss Function Search · p. 118, Continual and Online Learning Scenarios · p. 118, Robustness and Safety in Deployment · p. 118

Task-Adaptive and Data-Adaptive Metrics

• Task-adaptive metrics align evaluation with real-world goals, including censored-data metrics for survival analysis and Answer Faithfulness or Context Relevance for retrieval-augmented generation.Generic MSE and cross-entropy can be suboptimal when they do not match application objectives, factual correctness, or utility.Source: Task-Adaptive and Data-Adaptive Metrics · p. 118

Enhanced Interpretability and Explainability; Large Language Models and Prompt-Engineering Paradigms

• Future objectives increasingly incorporate interpretability, traceable evidence paths, and penalties for factual inconsistencies or hallucinations as large language models and retrieval-augmented systems expand.Evaluation is expected to cover context usage, reasoning chains, prompt understanding, and transparent links to external documents.Source: Enhanced Interpretability and… · p. 118, Large Language Models and… · p. 118

• Loss functions and metrics must be tailored to each task and its data characteristics, especially under class imbalance or noisy labels.The review spans regression, classification, computer vision, and NLP applications.Source: §14 · p. 118
• Specialized losses and metrics address domain-specific challenges, while multi-loss setups provide richer learning signals when objective weights are carefully tuned or adapted.Examples include focal loss, smooth L1, adversarial objectives, AP/AR, IoU-based measures, BLEU, and ROUGE.Source: §14 · p. 118
• Automated loss and metric selection through search algorithms or meta-learning could reduce human trial and error.Source: §14 · p. 118
• Robust, interpretable, and task-adaptive objectives should remain stable under noise or domain shifts while aligning with fairness, explainability, and retrieval faithfulness.Source: §14 · p. 119
• Hybrid approaches must extend to retrieval-augmented generation by jointly addressing retrieval accuracy and generative fidelity.Improved understanding of losses and metrics is intended to support stronger, more reliable, high-performing, and robust deep learning systems.Source: §14 · p. 119