PaperlayerView exact revision on arXiv

Source-linked AI summary

Loss Functions and Metrics in Deep Learning

Juan Terven, Diana M. Cordova-Esparza, Alfonso Ramirez-Pedraza, Edgar A. Chavez-Urbiola, Julio A. Romero-Gonzalez

arXiv:2307.02694v5cs.LGcs.AIcs.CV

TL;DR

Deep learning practitioners need to choose losses and metrics that match varied tasks and challenges, but many alternatives make selection difficult. This review synthesizes these choices across general tasks, computer vision, NLP, and RAG, examining specialized and combined objectives. It concludes that no single approach applies universally and that effective evaluation requires task-specific tailoring, including careful balancing of multi-loss setups.

  • Problem

    Choosing suitable loss functions and performance metrics is difficult because deep learning tasks involve different objectives, data characteristics, class imbalance, outliers, and hybrid retrieval-generation structures.

  • Method

    The paper reviews commonly used losses and metrics across regression, classification, computer vision, NLP, and RAG, including specialized objectives and multi-loss setups.

  • Results

    The review finds that no single loss or metric applies universally, while specialized losses, advanced metrics, and carefully tuned multi-loss combinations address domain-specific challenges.

  • Takeaways & Limitations

    Losses and metrics should be selected or designed to align with the task, data characteristics, and practical evaluation objectives such as retrieval faithfulness.

  • Takeaways & Limitations

    The review notes that imbalanced or expensive-to-label data can limit traditional losses such as cross-entropy and MSE, which treat instances equally.

Abstract

from arXiv · show

This paper presents a comprehensive review of loss functions and performance metrics in deep learning, highlighting key developments and practical insights across diverse application areas. We begin by outlining fundamental considerations in classic tasks such as regression and classification, then extend our analysis to specialized domains like computer vision and natural language processing including retrieval-augmented generation. In each setting, we systematically examine how different loss functions and evaluation metrics can be paired to address task-specific challenges such as class imbalance, outliers, and sequence-level optimization. Key contributions of this work include: (1) a unified framework for understanding how losses and metrics align with different learning objectives, (2) an in-depth discussion of multi-loss setups that balance competing goals, and (3) new insights into specialized metrics used to evaluate modern applications like retrieval-augmented generation, where faithfulness and context relevance are pivotal. Along the way, we highlight best practices for selecting or combining losses and metrics based on empirical behaviors and domain constraints. Finally, we identify open problems and promising directions, including the automation of loss-function search and the development of robust, interpretable evaluation measures for increasingly complex deep learning tasks. Our review aims to equip researchers and practitioners with clearer guidance in designing effective training pipelines and reliable model assessments for a wide spectrum of real-world applications.

1 Introduction

The introduction frames loss functions and performance metrics as distinct but complementary components of deep learning, requiring task-specific selection. The review organizes common choices across general learning tasks, computer vision, and natural language processing, while also discussing implementation tools.

  • Selecting appropriate losses and metrics is challenging because their suitability depends on the specific task and its data characteristics.
  • The review covers regression and classification losses before surveying computer vision tasks and NLP, including retrieval-augmented generation.
  • Loss functions optimize model parameters during training, whereas performance metrics evaluate generalization and compare models after training.
  • The review summarizes general-task, computer-vision, and NLP losses and metrics in three tables.
  • Common frameworks provide standard losses, metrics, automatic differentiation, and customization options for implementing deep learning pipelines.

2 Regression

Regression selects loss functions to optimize continuous predictions and metrics to assess them, balancing error sensitivity, robustness, interpretability, and optimization behavior. The reviewed choices span MSE, MAE, Huber, Log-Cosh, quantile loss, and SMAPE, each matching different data and task conditions.

  • Regression foundations: Regression predicts continuous outputs from input features, with training framed as minimizing a loss over model parameters.The model prediction is defined as ˆy_i = f_θ(x_i), and the objective is to minimize the overall loss across n samples.
  • Regression loss selection: Table 4 organizes regression losses by usage, data characteristics, advantages, and limitations to guide selection.The comparison explicitly considers outlier robustness and the balance between interpretability and optimization smoothness.
  • MSE: MSE is smooth, differentiable, and convex in predictions, but its squared-error penalty makes it sensitive to outliers and target scale.For neural networks, nonlinear activations can still produce a non-convex overall error surface; MSE may therefore be suboptimal with outliers.
  • MAE: MAE penalizes errors linearly and is more robust to outliers than MSE, but its non-differentiability at zero can slow optimization.Subgradient methods can address the zero-residual corner, while MAE remains affected by the target scale.
  • Huber loss: Huber loss combines quadratic penalties for errors below δ with linear penalties for larger errors, providing controlled robustness while retaining smoothness for smaller residuals.A smaller δ makes Huber behave more like MAE, whereas a larger δ makes it closer to MSE; δ requires tuning.
  • Specialized losses and metrics: Quantile loss supports predictive intervals and asymmetric penalties, while SMAPE is suitable when over- and under-prediction have comparably important costs.For quantile level q > 0.5, underestimates receive stronger penalties; SMAPE requires care with zeros and outliers.

3 Classification

Classification learns mappings from input features to discrete labels, with loss functions guiding training and metrics assessing model performance. The review covers common classification losses and metrics, including approaches for class imbalance and balanced evaluation.

  • Classification maps input features to discrete labels through a parameterized decision function.
  • Appropriate classification losses depend on the data, number of classes, and model architecture.
  • Classification Loss Functions: Binary Cross-Entropy is a differentiable binary-classification loss that penalizes confident misclassifications and supports gradient-based optimization.
  • Classification Loss Functions: Categorical and Sparse Categorical Cross-Entropy handle multiclass prediction, with Sparse CCE using integer labels instead of one-hot vectors.
  • Classification Loss Functions: Weighted Cross-Entropy increases minority-class influence during training, but its weights require tuning and may become suboptimal after distribution shifts.
  • Classification Metrics: Balanced Accuracy gives each class equal influence, while Macro-average Precision treats classes equally and Micro-average Precision weights them by support.

4 Image Classification

Image classification applies whole-image categorization across domains including medical imaging and agriculture. Its standard losses and metrics remain applicable, but large-scale, multilabel, and imbalanced datasets motivate additional techniques and measures.

  • Image classification categorizes an image as a whole into a specific label and has broad applications including medical imaging and agriculture.
  • AlexNet achieved substantially better performance than previous best results in 2012, accelerating deep-learning research and applications.
  • Loss Functions: Image classification uses standard losses such as cross-entropy, weighted cross-entropy, focal loss, and hinge loss.
  • Loss Functions: Large-scale or imbalanced visual datasets may require label smoothing, focal loss, data augmentation, or other regularization and hyperparameter adjustments.
  • Loss Functions: Multilabel image classification commonly uses binary cross-entropy by treating each label as an independent positive/negative prediction.
  • Performance Metrics: Image classification evaluation may combine core metrics with per-class or balanced measures, multilabel evaluation, top-k accuracy, and confusion-matrix heatmaps.
  • Performance Metrics: Top-5 Accuracy counts a prediction as correct when the true class appears among the five highest-probability predictions.

5 Object Detection

Object detection jointly localizes objects and assigns their classes, so training commonly combines classification and bounding-box regression losses. The review contrasts coordinate-based and overlap-based localization losses, including their robustness and gradient limitations.

  • Object detection requires both bounding-box localization and object classification, typically using a composite loss with classification and regression components.
  • Detection models often sum classification loss with weighted regression loss over predicted classes and bounding-box coordinates.
  • Bounding-Box Regression Losses: Smooth L1 uses quadratic penalties for minor deviations and linear penalties for large deviations, reducing outlier effects while retaining differentiability.
  • Bounding-Box Regression Losses: Smooth L1 is widely used in two-stage detectors and provides stable gradients with less sensitivity to large bounding-box errors than MSE.
  • Bounding-Box Regression Losses: Balanced L1 combines quadratic-like treatment for small errors with logarithmic growth for large errors, balancing precision and robustness.
  • Bounding-Box Regression Losses: Balanced L1 can yield more stable gradient updates and faster convergence, particularly when object scales vary or extreme outliers affect training.
  • IoU-Based Losses: IoU Loss encourages predicted and ground-truth boxes to overlap, but non-overlapping boxes produce zero intersection and no gradient signal.
  • IoU-Based Losses: GIoU extends IoU by considering the smallest enclosing box, providing continuous gradients for non-overlapping or partially overlapping boxes.

6 Image Segmentation

Image segmentation assigns labels pixel by pixel and includes semantic, instance, and panoptic formulations. Its losses and metrics address class imbalance, region overlap, boundary quality, and instance-aware scene understanding.

  • Segmentation assigns a label to each pixel using local features or full-image context.
  • Segmentation types: Semantic segmentation labels shared “stuff” categories, instance segmentation separates individual “things,” and panoptic segmentation combines both.
  • Applications: Segmentation supports scene understanding, medical imaging, robotic perception, autonomous vehicles, video surveillance, and augmented reality.
  • Loss functions: Segmentation losses are designed for heavy class imbalance and coherent output shapes, with choices varying by application and desired overlap behavior.
  • Overlap-based losses: Dice Loss directly optimizes region overlap and emphasizes small target regions, making it useful for rare foregrounds and fine structures.
  • Overlap-based losses: Lovász Loss is valuable when IoU is the primary metric and often outperforms purely pixel-level losses under class imbalance or small-object segmentation.
  • Segmentation metrics: Panoptic Quality balances region accuracy with instance identification in scenes containing overlapping objects and background classes.

7 Face Recognition

Face recognition learns discriminative features for matching faces to identities, using classification-based and representation-based losses. Evaluation combines general classification metrics with verification and identification protocols.

  • Face recognition matches an input face in an image or video to an identity in a face database.
  • Loss categories: Face-recognition losses preserve relational structure among face embeddings through classification-based and representation-based objectives.
  • Evaluation: Evaluation uses accuracy, precision, recall, F1-score, and ROC curves alongside specialized verification and identification protocols.
  • Classification losses: Standard softmax classifies identities but does not explicitly enforce large inter-class margins or compact intra-class representations.
  • Margin-based losses: A-Softmax normalizes weights and features and applies an angular margin to make correct classification more discriminative.

4. Scaled Logits: The final logits become

Margin-based and metric-learning losses shape face embeddings through angular, cosine, class-center, or distance-based constraints. These objectives trade stronger separation and discriminability against optimization and sampling considerations.

  • A-Softmax: A-Softmax encourages tighter intra-class clustering and larger inter-class separation, improving verification and identification metrics over plain softmax.
  • A-Softmax: Larger A-Softmax margins strengthen separation but can make optimization harder, while scaling factors may stabilize training.
  • Center Loss: Center Loss adds a class-center clustering term to classification loss, with λ balancing classification and compactness objectives.
  • CosFace: CosFace subtracts a margin from the correct class’s normalized cosine similarity, producing a direct cosine-space separation constraint.
  • CosFace: CosFace often outperforms standard softmax or SphereFace on challenging face benchmarks, improving generalization and discriminability.
  • ArcFace: ArcFace adds an angular offset to the correct-class logit, yielding a geometrically interpretable boundary and typically outperforming earlier margin-based losses.
  • Metric learning: Triplet Loss separates anchor-positive and anchor-negative distances by a margin, with effectiveness depending strongly on negative sampling.

8 Monocular Depth Estimation (MDE)

Monocular depth estimation infers 3D depth from a single image, motivating losses and metrics that balance absolute accuracy, relative scale, structural detail, and multi-view consistency. The review describes point-wise, scale-invariant, structural, photometric, and composite objectives, along with practical limitations and combined-loss benefits.

  • Task Overview: Monocular depth estimation infers depth from a single 2D image, unlike stereo methods that use multiple images.The task is challenging because it must recover 3D depth from a 2D projection using cues such as object size and scene context.
  • Point-wise Losses: Point-wise MAE is relatively robust to moderate outliers, whereas MSE penalizes large errors more heavily and can correct major misestimates early.Point-wise errors are straightforward but do not capture structural relationships or relative depth cues, which can impair boundaries and texture-less regions.
  • Scale-invariant Loss: Scale-invariant error emphasizes relative depth consistency and tolerates global scaling errors, making it valuable when only relative depth matters.Its log-space formulation prioritizes multiplicative errors and includes a term that counteracts mean log-space bias; absolute metric scale may require additional constraints or multi-view geometry.
  • Structural Losses: SSIM loss encourages structurally consistent edges and gradients, and is commonly combined with numeric errors or smoothness priors.SSIM compares luminance, contrast, and local structure; its index ranges from −1 to 1, with 1 indicating perfect structural alignment.
  • Photometric Loss: Photometric loss enables self-supervised depth estimation by comparing a reference image with a source image warped using predicted depth and camera pose.Occlusions, motion, and reflectance changes violate brightness constancy, so practical methods use masking, minimum reprojection, or auto-masking; smoothness and geometric constraints can improve robustness.
  • Composite Objectives: Combining complementary losses can improve zero-shot performance on unseen datasets while balancing global alignment against local detail.The reviewed scale-and-shift-invariant combination uses one component for global alignment and another for sharper object boundaries and local details, with α controlling the trade-off.

9 Image Generation

Image generation spans VAEs, GANs, normalizing flows, EBMs, and diffusion models, each using distinct training objectives and exhibiting different practical trade-offs. The reviewed losses and metrics balance realism, fidelity, distribution matching, stability, and computational feasibility.

  • Generative models: VAEs combine reconstruction loss with KL divergence to preserve input fidelity while keeping latent distributions near a standard Gaussian.The decoder reconstructs inputs, while KL divergence regularizes the encoder distribution toward a unit Gaussian.
  • Generative models: GANs train a generator and discriminator adversarially, with adversarial loss commonly based on cross-entropy against real or fake labels.The generator improves sample realism while the discriminator refines its ability to distinguish real from generated data.
  • Generative models: Wasserstein loss provides non-saturating gradients, mitigates mode collapse, and can support more stable convergence than classic GAN losses.The review notes that hyperparameters and Lipschitz constraints remain important, including gradient-penalty enforcement in WGAN-GP.
  • Generative models: Normalizing flows maximize likelihood by minimizing negative log-likelihood, providing exact density estimation and generative sampling at higher computational cost.Their invertible transformations yield tractable likelihoods, making them attractive when precise likelihood evaluation is required.
  • Generative models: Contrastive Divergence makes energy-based learning computationally feasible by using fewer MCMC steps, but its short negative-phase chain introduces biased gradients.The bias can decrease as k increases, while computational cost rises; the method is especially prominent in RBMs, DBMs, and related EBMs.
  • Evaluation metrics: Image-generation evaluation should match the task: SSIM and perceptual losses emphasize human-aligned structure, whereas FID assesses feature-space distribution fidelity and PSNR suits known references.Perceptual loss can recover sharp edges and fine structures, while FID jointly penalizes differences in feature-space location and spread.

10 Natural Language Processing (NLP)

NLP uses task-specific losses and metrics for classification, sequence labeling, language modeling, and generation. The section emphasizes token-level supervision while noting mismatches with sequence-level quality and the roles of ranking and semantic objectives.

  • NLP tasks: NLP spans text classification, language modeling, translation, and other tasks requiring specialized optimization and evaluation choices.These tasks include categorization, next-token prediction, translation, and language generation.
  • Cross-Entropy Loss: Cross-entropy provides token-level supervision by comparing predicted vocabulary distributions with one-hot ground-truth tokens.It supports language modeling, translation, summarization, and token labeling.
  • Cross-Entropy Loss: Cross-entropy independently penalizes token mismatches, which can overlook global sequence properties, semantic coherence, and inference-time exposure bias.Teacher forcing exposes models to ground-truth histories during training but not necessarily during inference.
  • Cross-Entropy Loss: Token-level cross-entropy remains primary in supervised and sequence-to-sequence frameworks, while sequence-level objectives can address holistic output fidelity.Sequence-level reinforcement learning and minimum risk training are cited as additional objectives.
  • Specialized NLP losses: Cosine similarity loss targets semantic closeness, whereas marginal ranking loss separates relevant from irrelevant text through margin-based score differences.Ranking loss is used in retrieval, question answering, recommendation, and paraphrase detection.

10.2 Losses for Sequence Generation

Sequence-generation losses address alignment uncertainty, sequence-level evaluation, and nondifferentiable objectives. CTC handles unobserved monotonic alignments, while MRT and REINFORCE connect training more directly to final sequence quality.

  • CTC: CTC learns mappings from longer inputs to shorter targets without manually specified alignments by summing over valid alignment paths.Blank symbols and dynamic programming make variable-length alignment training tractable.
  • CTC: CTC is constrained by a strict monotonic alignment assumption, limiting its suitability for tasks requiring substantial reordering.The blank symbol also adds output-space complexity and requires careful training.
  • Minimum Risk Training: Minimum Risk Training minimizes expected sequence-level risk under a chosen metric such as BLEU or ROUGE instead of relying solely on token-level cross-entropy.Sampling approximates the otherwise intractable expectation over possible output sequences.
  • Minimum Risk Training: MRT can improve alignment with evaluation metrics and end-task performance, but stability depends on sampling, metric design, scaling, and hyperparameter choices.Large scaling factors can produce overly peaked distributions and reduce exploration.
  • REINFORCE: REINFORCE treats generated sequences as stochastic-policy outcomes and uses final rewards to optimize nondifferentiable metrics.Large vocabularies and long sequences create sample-efficiency and variance challenges.

10.3 Performance Metrics Used in NLP

NLP evaluation uses classification metrics, sequence-overlap measures, and predictive metrics according to task and data characteristics. The section repeatedly emphasizes complementing simple metrics when imbalance, semantic variation, or sequence coherence matters.

  • Generation and language-model metrics: BLEU and ROUGE compare generated text with references, while perplexity measures language-model token prediction and Exact Match supports question answering.Metric choice depends on the output type and evaluation objective.
  • Accuracy: Accuracy measures the proportion of correctly classified inputs and applies broadly to text classification, sequence labeling, and language detection.Its simplicity makes it interpretable across discrete-label tasks.
  • Accuracy: Accuracy can mislead on highly imbalanced data and does not capture confidence, semantic closeness, or contextual dependencies among sequence labels.Practitioners often complement it with precision, recall, F1-score, or specialized measures.
  • Precision, Recall, and F1: Precision, recall, and F1 distinguish false-positive and false-negative effects, making them important for imbalanced or cost-sensitive classification and labeling tasks.F1 combines precision and recall into a balanced score.

METEOR =

NLP metrics balance lexical overlap, recall, predictive confidence, and semantic flexibility. METEOR broadens matching beyond exact n-grams, ROUGE emphasizes content recall, and perplexity evaluates token-sequence prediction with domain-sensitive interpretation.

  • METEOR: METEOR incorporates synonyms and morphological variants while balancing precision, recall, and fragmentation in reference-based generation evaluation.It is used for translation, summarization, dialogue, and paraphrase generation.
  • METEOR: METEOR can correlate better with human evaluations than BLEU, but remains limited by lexical resources, unigram focus, and reference coverage.A single reference may penalize valid alternate translations.
  • ROUGE: ROUGE primarily measures recall of n-gram or sequence overlaps, making it a standard content-coverage metric for summarization.ROUGE-L uses longest common subsequence structure, while ROUGE-W emphasizes contiguous matches.
  • ROUGE: ROUGE has limited semantic sensitivity, depends on reference coverage, and may over-reward longer candidates because of its recall orientation.Multiple diverse references can accommodate more valid expression alternatives.
  • Perplexity: Perplexity is the geometric mean of inverse token probabilities, with lower values indicating greater confidence in predicting observed sequences.Comparisons require consistent domains, test sets, and vocabulary conditions.
  • Perplexity: Perplexity varies across corpora and vocabularies and may not reflect generative quality when probability calibration is poor.Sequence-level metrics can complement perplexity for semantic or global-coherence evaluation.

11 Retrieval-Augmented Generation (RAG)

RAG combines retrieval and generation to incorporate external knowledge, but its hybrid structure requires evaluating both retrieved context and generated responses. The review surveys fine-tuning losses and specialized metrics for retrieval quality, generation quality, relevance, correctness, and faithfulness.

  • RAG architecture: RAG retrieves relevant external passages and uses a language model to synthesize coherent, context-aware text.
  • Fine-tuning objectives: Fine-tuning can improve domain adaptation and retrieval or generation quality when prompting alone is insufficient for specialized or high-stakes applications.
  • Fine-tuning objectives: Contrastive and marginal ranking losses refine retrieval, while cross-entropy, NLL, and KL divergence losses target generation or distribution alignment.
  • Evaluation requirements: RAG evaluation must separately assess retrieval quality and generation quality because the system combines external-context retrieval with response generation.
  • Evaluation requirements: Reference-free or partially reference-based frameworks such as RAGAS and ARES address settings where conventional BLEU- or ROUGE-style references are unavailable.
  • RAG metrics: Answer semantic similarity, answer correctness, answer relevance, and faithfulness capture complementary aspects of meaning, factuality, topicality, and context grounding.

12 Combining Multiple Loss Functions

Multi-loss training combines objectives to address complementary task requirements, often through weighted sums. Its benefits depend on balancing objectives carefully because conflicting gradients and poorly chosen weights can destabilize training or reduce one objective’s gains.

  • Rationale: Multi-loss setups combine two or more objectives, commonly in a weighted sum, to optimize different facets of complex tasks simultaneously.
  • Applications: Object detection combines regression for bounding-box coordinates with classification loss for object labels, balancing spatial accuracy and semantic correctness.
  • Applications: GANs pair adversarial loss with pixel-level or perceptual terms to control content, style, detail, and visual fidelity.
  • Applications: Sequence-level objectives can complement token-level cross-entropy in summarization or translation, improving coherence and context awareness while reducing exposure bias.
  • Challenges and future directions: Conflicting gradients and loss competition can cause instability, motivating adaptive reweighting, multi-objective optimization, and automated weight discovery.

13 Challenges and Trends

Deep learning losses and metrics must adapt to imbalanced data, outliers, sequence-level objectives, and increasingly multifaceted tasks. Current trends emphasize specialized, adaptive, robust, interpretable, and automatically designed objectives.

  • Data and objective challenges: Class imbalance and expensive annotation can make standard cross-entropy or MSE overfit majority classes, motivating weighted cross-entropy and focal loss.
  • Data and objective challenges: MSE is sensitive to outliers and noisy data, whereas Huber Loss and Smooth L1 transition between squared- and absolute-error behavior to improve robustness.
  • Data and objective challenges: Exposure bias arises when sequential inference differs from ground-truth-prefix training, motivating sequence-level optimization with BLEU or ROUGE.
  • Metric design: Complex tasks require task-specific or multifaceted metrics because strong performance on one metric may coexist with failure on another.
  • Current responses: Specialized losses address minority classes, outliers, and global sequence properties, while ranking losses focus retrieval on relevant documents.
  • Emerging directions: Loss Function Search uses gradient-based meta-learning or evolutionary algorithms to discover effective differentiable loss operators for particular datasets or tasks.
  • Emerging directions: Metrics increasingly target application objectives such as retrieval faithfulness, fairness, interpretability, reliability, and robustness to changing data distributions.

14 Conclusion

The review concludes that losses and metrics must be tailored to task characteristics rather than selected universally. It highlights carefully balanced multi-loss designs, automated selection, and robust, interpretable, task-adaptive evaluation as future priorities.

  • Main conclusion: No single loss or metric applies universally across regression, classification, computer vision, and NLP tasks.
  • Main conclusion: Specialized losses and metrics address domain-specific challenges, while multi-loss setups provide richer learning signals when their weights are tuned or adapted carefully.
  • Future directions: Future work includes automating loss and metric selection through search algorithms or meta-learning to reduce human trial and error.
  • Future directions: Robust, interpretable, and task-adaptive objectives should remain stable under noise or domain shifts while aligning with practical goals such as fairness and retrieval faithfulness.
  • Future directions: Advancing these directions is expected to support stronger and more reliable deep learning models for complex tasks.
Loading 2307.02694v5…