PaperlayerView exact revision on arXiv

Source-linked AI summary

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang

arXiv:2609.33076v2cs.CVcs.AI

TL;DR

Manual food logging is time-consuming and susceptible to reporting errors, motivating image-based nutrition estimation. NutriVision combines RGB imagery, estimated depth, and ingredient information in an ingredient-conditioned prediction framework. On Nutrition5k it reports 13.60 ± 0.10% mean PMAE, while its authors identify ingredient-aware prediction as the largest architectural contribution.

Abstract

from arXiv · show

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

I. INTRODUCTION

NutriVision addresses the partial integration of geometric and ingredient-level cues by conditioning RGB–depth fusion and nutrient prediction on ingredient semantics. It estimates five nutritional quantities using estimated depth, RGB imagery, and ingredient information.

I. INTRODUCTION

  • Nutrition estimation predicts energy, mass, fat, carbohydrate, and protein from food images, addressing the time and reporting-error burdens of manual food logging.The work targets settings with ingredient information available; fully autonomous image-only estimation would require an upstream ingredient-recognition component and is outside its scope.Source: §I · p. 1, Contributions · p. 2
  • Existing approaches only partially integrate geometric and ingredient information: frequency fusion is ingredient-independent, while ingredient-aware methods add semantics after visual fusion.The proposed formulation uses monocular depth estimation to retain single-image deployability.Source: §I · pp. 1–2

Contributions

  • IC-FAFM conditions multi-scale FFT-based RGB–depth fusion on CLIP ingredient representations, allowing food composition to influence cross-modal spectral alignment.It operates across four ConvNeXt feature scales.Source: Contributions · p. 2
  • IA-MPH combines an ingredient-conditioned semantic gate, magnitude-based top-k channel selection, and ingredient-to-visual cross-attention for nutrient regression.Its channels can vary with dish composition rather than depending solely on visual activations.Source: Contributions · p. 2
  • Modality-specific ISM applies DPE- and FGM-based blocks independently to RGB and estimated-depth feature hierarchies before fusion.This supports localized semantic representations while preserving modality-specific information.Source: Contributions · p. 2
  • 13.60 ± 0.10% mean PMAE on Nutrition5k improves on the IGSMNet reimplementation by 0.89 percentage points and OmniFood8k by 2.90 percentage points.Module-level retraining ablations identify IA-MPH as the largest individual architectural contributor, increasing PMAE by 1.50 ± 0.17 percentage points when removed.Source: Contributions · p. 2

II. RELATED WORK

Prior nutrition-estimation research has progressed from image-only regression to methods using depth geometry and ingredient semantics. NutriVision builds on these lines by conditioning both frequency-domain RGB–depth interaction and prediction on ingredient information.

A. Image-Based Food Nutrition Regression

  • The Nutrition5k Inception baseline achieved 18.9% mean PMAE, while Swin-Nutrition reached 17.2% using RGB images alone.The task predicts calories, mass, fat, carbohydrates, and protein from RGB or RGB-D input.Source: A. Image-Based Food Nutrition Regression · p. 2

B. Depth-Assisted Nutrition Estimation

  • RGB-D methods such as FBFPN and NuNet reported 17.3% and 15.65% mean PMAE, respectively, but assume calibrated depth hardware.DPF-Nutrition and OmniFood8k instead estimate depth from RGB; NutriVision likewise uses DepthAnything-V3 and SSRA to adapt relative geometry.Source: B. Depth-Assisted Nutrition Estimation · p. 3

C. Frequency-Domain RGB-Depth Fusion

  • OmniFood8k’s FAFM fuses RGB and adapted-depth features in the frequency domain, but its alignment and channel selection rely on visual features alone.NutriVision conditions frequency-domain interaction and final channel selection on ingredient semantics.Source: C. Frequency-Domain RGB-Depth Fusion · p. 3

D. Ingredient-Guided Nutrition Estimation

  • IGSMNet uses CLIP ingredient embeddings to guide fused RGB-D features, while NutriVision also conditions frequency alignment and its prediction head on ingredients.NutriVision additionally applies modality-specific semantic modeling before fusion.Source: D. Ingredient-Guided Nutrition… · p. 3

III. METHODOLOGY

NutriVision combines RGB appearance, estimated-depth geometry, and ingredient information to predict five nutritional quantities. Its pipeline adapts monocular depth, processes modality-specific feature hierarchies, fuses them under ingredient guidance, and applies an ingredient-aware prediction head.

III. METHODOLOGY

  • The forward pass applies an ingredient gate and a training-only channel mask before IA-MPH cross-attention and MLP regression.The ingredient string is encoded offline using CLIP ViT-B/32.Source: §III · p. 4
  • Separate RGB and adapted-depth encoders refine features with modality-specific ISM blocks before ingredient-guided frequency fusion and multi-scale aggregation.The complete model uses a normalized nutrition-regression objective with an auxiliary RGB-depth alignment loss.Source: §III · p. 4

A. Problem Formulation and Notation

  • The model maps an RGB image, monocular depth, and ingredient embedding to calories, total mass, fat, carbohydrates, and protein.Experiments use Nutrition5k ground-truth ingredient strings as an inference-time input, not as a target predicted by NutriVision.Source: A. Problem Formulation and Notation · p. 4

B. Scale-Shift Residual Adapter (SSRA)

NutriVision adapts monocular depth and uses dual ConvNeXt feature hierarchies with modality-specific semantic modeling. Its multi-scale frequency fusion is ingredient-conditioned, though the reported component effects are small and diffuse.

B. Scale-Shift Residual Adapter (SSRA)

  • SSRA applies a global affine correction followed by a local CNN residual to adapt monocular depth, contributing about 100 K parameters.The depth map is described as defined up to an unknown affine transformation.Source: B. Scale-Shift Residual Adapter (SSRA) · p. 4

C. Dual ConvNeXt-Base Backbone

  • The RGB and depth branches use pretrained ConvNeXt-Base encoders, each producing a four-stage feature pyramid projected to 256 channels.The first two stages are frozen, leaving about 44 M trainable parameters per branch.Source: C. Dual ConvNeXt-Base Backbone · p. 4

D. Internal Semantic Modelling (ISM)

  • ISM applies dynamic positional encoding and window-based fine-grained self-attention independently to both modalities at each scale.Two blocks are used per scale per modality.Source: D. Internal Semantic Modelling (ISM) · p. 4

E. Ingredient-Conditioned Frequency-Aligned Fusion (IC-FAFM)

  • IC-FAFM extends FAFM with multi-scale operation and ingredient-conditioned cross-attention, but its individual module-level deltas sit at the per-run noise floor.Its FFT split separates layout and footprint cues in the low band from texture and edge cues in the high band; τ=0.20 was inherited without a sweep.Source: E. Ingredient-Conditioned… · pp. 4–5

F. Ingredient-Aware Mask-based Prediction Head (IA-MPH)

IA-MPH combines the fused representation with both modalities’ CLS embeddings, then uses ingredient-aware channel selection and cross-attention for nutrient regression.

F. Ingredient-Aware Mask-based Prediction Head (IA-MPH)

  • The fused vector is concatenated with both modalities’ CLS embeddings before ingredient-conditioned prediction.This forms the head’s joint representation.Source: F. Ingredient-Aware Mask-based… · p. 5
  • During training, a dynamic mask retains the top 75% of channels by magnitude as a regulariser.The regulariser encourages informative channels to be associated with each food type.Source: F. Ingredient-Aware Mask-based… · p. 5
  • Ingredient cross-attention and gated fusion feed a three-layer MLP that outputs a five-dimensional nutrition vector.The MLP dimensions are 1280→512→256→5, with dropouts of 0.1 and 0.05.Source: F. Ingredient-Aware Mask-based… · p. 5

G. Loss Function

The training objective combines nutrient prediction loss with alignment loss across four scales, and the nutrient loss directly optimizes PMAE.

G. Loss Function

  • Lalign is the average of Eq. (15) over four scales, and Lnutri is the normalized ℓ1 loss of.Source: G. Loss Function · p. 5
  • The alignment component is averaged across four scales before being combined with the nutrient loss.Source: G. Loss Function · p. 5
  • The normalized ℓ1 loss directly optimizes PMAE and is scale-invariant across the five tasks.Source: G. Loss Function · p. 5

H. Parameter Budget

The model has approximately 196 M parameters, including approximately 105 M trainable parameters; baseline comparisons are not parameter-matched.

H. Parameter Budget

  • The full model has approximately 196 M parameters, of which approximately 105 M are trainable.Table I presents the per-component parameter budget.Source: H. Parameter Budget · p. 5, Table I · p. 6
  • Compute reporting estimates forward-pass FLOPs and measures training and inference times on a single RTX A6000.End-to-end inference includes DepthAnything-V3 and CLIP ingredient-string encoding.Source: Table II · p. 6
  • The 0.89 pp input-matched comparison against IGSMNet is not parameter-matched, and a size-controlled comparison is left for future work.The reported gap therefore combines architecture, ingredient input, and backbone scale rather than isolating architecture.Source: H. Parameter Budget · p. 6

IV. EXPERIMENTS

Experiments evaluate NutriVision on a dish-level Nutrition5k split using mean PMAE across five nutrient tasks. NutriVision achieves 13.60 ± 0.10% mean PMAE, improving on the input-matched IGSMNet reimplementation.

A. Dataset and Protocol

  • The 5,006-dish Nutrition5k split is dish-level, with approximately 15,000 training and 3,000 testing images and no dish shared across the sets.The test set is also used for early stopping and reporting, without a separate validation split, making the headline optimistically biased relative to held-out generalization.Source: A. Dataset and Protocol · p. 6

A. Comparison with State-of-the-Art

  • NutriVision consumes RGB, estimated depth, and a ground-truth ingredient string; removing ingredients raises its mean PMAE to 27.44%.Source: A. Dataset and Protocol · p. 6

B. Evaluation Metric

  • PMAE is reported for each nutrient task and averaged across calories, mass, fat, carbohydrates, and protein.Source: B. Evaluation Metric · p. 6

C. Implementation Details

  • Training uses Adam for 150 epochs, with learning rates of 1e−5 for unfrozen ConvNeXt stages and 5e−5 for the remaining parameters.The reported checkpoint is selected using the same test split.Source: C. Implementation Details · p. 6

A. Comparison with State-of-the-Art

B. Comparison with Real-Depth Methods

On Nutrition5k, NutriVision’s estimated-depth configuration numerically outperforms NuNet’s real-depth configuration, but the comparison uses different input modalities. The reported ablations attribute the dominant module-level gain to the ingredient-aware prediction head.

B. Comparison with Real-Depth Methods

A. Input-Level Ablation

Input ablations identify ingredient text as the largest non-RGB signal, while module retraining identifies IA-MPH as the largest tested architectural contributor. Synthetic pretraining fits its source data but transfers worse to Nutrition5k than training from scratch.

A. Input-Level Ablation

  • Removing ingredient text costs 13.92 mean-PMAE points, with the largest nutrient hits in fat (+18.98) and protein (+21.24).The input ablations zero modalities at test time without retraining.Source: A. Input-Level Ablation · p. 8
  • Depth and SSRA zeroing change mean PMAE by only +0.13 and 0.00, respectively, but these inference-time tests measure routing-around, not causal contribution.The model was trained with both modules; from-scratch retraining without depth or SSRA remains future work.Source: A. Input-Level Ablation · p. 8
  • Zeroing RGB raises PMAE to 66.34%, showing that this trained architecture needs its visual stream to exploit ingredient information.The result does not establish that ingredients need vision in a dedicated text-only or text-plus-area model.Source: A. Input-Level Ablation · p. 8

B. Module-Level Retraining Ablation

  • +1.50 ± 0.17 pp mean PMAE when IA-MPH is removed, the largest tested architectural degradation, with paired p<0.001 across 3 seeds.The comparison replaces IA-MPH with a plain MLP; the module-level table reports the retraining protocol.Source: B. Module-Level Retraining Ablation · p. 9, Table VI · p. 9
  • Fusion-module changes are small or within the per-run noise floor; removing ISM changes PMAE by −0.20 in a single-seed test.The authors describe frequency fusion and ISM as scaffolding below the noise floor on Nutrition5k.Source: B. Module-Level Retraining Ablation · p. 9

C. Synthetic Pretraining (Negative Result)

  • Synthetic pretraining reaches 8.83% mean PMAE on synthetic validation, but Nutrition5k transfer plateaus at 14.80% versus 13.60 ± 0.10% from scratch.Transfer regresses most on Fat (+1.35) and Mass (+1.53).Source: C. Synthetic Pretraining (Negative… · p. 9

VII. DISCUSSION

On Nutrition5k, ingredient-conditioned frequency fusion has little measured effect, while the ingredient-aware prediction head carries the dominant tested architectural gain. The paper also limits its claims to a controlled Western-cuisine dataset and identifies unresolved depth, evaluation, and deployment boundaries.

A. Frequency-Domain Fusion and Ingredient Conditioning

  • IC-FAFM’s frequency split and ingredient cross-attention show little measured contribution on Nutrition5k; the larger paired-tested gain concentrates in IA-MPH (+1.50 ± 0.17 pp, p < 0.001).The A7 and A8 deltas are at the noise floor; the authors retain fusion as scaffolding and for synergy with the head.Source: A. Frequency-Domain Fusion and… · p. 10

B. Cross-Cuisine Generalisation

  • The model is evaluated only on controlled Western-cuisine Nutrition5k, so the authors make no cross-cuisine generalisation claim for OmniFood8k’s predominantly Chinese-cuisine samples.They note the ingredient-conditioning CLIP embeddings have a Western bias and have not validated on OmniFood8k’s test split.Source: B. Cross-Cuisine Generalisation · p. 10

C. Keeping the SSRA and the depth branch

  • Depth and SSRA input zeroing tests measure test-time routing-around, not causal contribution; retraining without either module is identified as the main methodological gap.The authors cite concentrated changes in calories and protein as a pattern consistent with a working depth signal, not proof of contribution.Source: C. Keeping the SSRA and the depth branch · p. 10

D. Limitations

  • Ground-truth ingredient strings preclude photo-only use, and the approximately 196 M-parameter backbone lacks a parameter-matched baseline; best-on-test selection is also described as optimistic.The comparison to IGSMNet uses a best-effort reimplementation because official code was unavailable.Source: D. Limitations · p. 10

VIII. CONCLUSION AND FUTURE WORK

  • The conclusion reports 13.60 ± 0.10% 3-seed mean PMAE and IA-MPH as the dominant architectural contribution; planned work includes no-depth retraining, stronger controls, and cross-validation improvements.Mass remains behind Kwan, OmniFood8k, and IGSMNet, consistent with their real depth or larger-backbone advantage.Source: §VIII · p. 10

Related papers

Loading 2609.33076v2…