Optimization Encoders: Rethinking Second-Order Meta-Learning for Neural Fields

• Few-step meta-learning adapts a sample-specific latent from a shared initialization while keeping the decoder fixed, then updates shared parameters after adaptation.Source: §1 · pp. 1–2
• MetaLF adds equivariant self-attention to localized latent representations, allowing local observations to inform coherent non-local structure while retaining fine detail.Source: §1 · p. 2, Figure 2 · p. 3
• Second-order differentiation trains the encoding procedure alongside the decoder, while first-order approximations discard the pathway through which the outer objective steers encoding.Source: §1 · p. 2
• On controlled polynomial fields, latent coordination yields more compact, function-aligned representations and improves reconstruction.Source: §1 · p. 2
• MetaLF improves reconstruction over functa-based baselines on images and 3D shapes within three to five gradient updates.Source: §1 · p. 3
• The framework separates inner encoding and outer task objectives, supporting end-to-end classification and segmentation through reconstruction-based encoding and semantic supervision.Source: §1 · p. 3

• ENF latent point sets pair geometric poses with appearance features, and transformations of the poses correspond to transformations of the represented field.Source: §2 · p. 3
• Relative-geometry interactions share a local mapping across latent elements and transformed signals, biasing each latent toward information associated with a signal region.Source: §2 · p. 3
• The inner loop applies K gradient updates to a latent from a shared initialization while the decoder remains fixed; the outer loop updates decoder parameters after adaptation.Source: §2 · p. 3
• Figure 3 depicts globally contextualized latent updates and task-specific outer objectives, with the same inner-loop encoder serving diverse downstream objectives.Source: Figure 3 · p. 4
• MAML differentiates the outer objective through inner updates, including mixed second-order derivatives, whereas FOMAML omits those terms to reduce meta-training cost.Source: §2 · p. 4

3.1 Unrolling latent adaptation

• The inner loop applies K gradient updates to a sample latent from shared initialization z0 while holding decoder parameters θ fixed, then decodes the adapted latent.The adaptation procedure maps observations to a representation through gradient updates rather than a separate feed-forward network.Source: §3.1 · p. 4
• The optimization encoder depends on initialization, latent parameterization, inner objective, learning rate, update count, and decoder because updates use decoder gradients.This makes the procedure a finite-step bilevel optimization problem whose outer objective depends on the inner-update trajectory.Source: §3.1 · p. 4

3.2 Training through the optimization encoder

• The meta-gradient separates direct decoder updates from gradients through adapted zK, with mixed derivatives injecting decoder dependence at each inner step.The gradient through the optimization encoder transports these contributions across the remaining updates.Source: §3.2 · p. 5
• Full MAML trains how inner gradient dynamics construct representations, whereas first-order training removes second derivatives and truncates this encoder path.The distinction is structural: MAML backpropagates through an optimization encoder, while conventional encoder-decoder training backpropagates through a feed-forward encoder.Source: §3.2 · p. 5

3.3 Inner and outer objectives

• The inner objective encodes observations into zK and must be computable at test time, while the outer objective specifies what the representation should support and may use meta-training-only supervision.Second-order differentiation lets the outer objective train representation construction even when the inner and outer losses express different tasks.Source: §3.3 · p. 5

4 Attentive Latent Fields

• MetaLF uses equivariant self-attention to coordinate a latent point set before evaluating the output field at query coordinates.Each sample-specific latent has a pose and content, and both are adapted by the optimization encoder.Source: §4 · pp. 5–6
• Self-attention updates each latent's content from its neighborhood using poses as the geometric reference frame, with layer-normalized attention, a GeGLU feed-forward network, and residual connections.Figure 4 shows a receiving latent attending to neighboring source latents using bi-invariant geometric attributes; geometry and source content form keys and values.Source: §4 · p. 6, Figure 4 · p. 6
• Attention aggregates values over the KSA nearest latent poses with Gaussian windowing to promote local interaction, while stacking blocks propagates information over longer distances..Source: §4 · p. 6
• After self-attention contextualizes the latents, ENF-style cross-attention maps them to query coordinates, and bi-invariant pose interactions preserve equivariance under G.Because predictions depend on interactions among latent variables, updates can incorporate information represented by neighboring latents.Source: §4 · p. 6

5.1 Polynomial fields

• Increasing self-attention depth on polynomial fields yields lower effective rank, stronger polynomial alignment, and higher decoded-field PSNR.The analysis uses degree-six polynomial fields, N = 25 latents, and reports MSE, effective rank, and polynomial tangent fraction; Table 1 averages three seeds.Source: Table 1 · p. 7, §5.1 · p. 7

5 Experiments

• The experiments evaluate reconstruction and SDF fitting alongside classification and segmentation, where semantic outer objectives differ from the reconstruction inner objective.The study covers image, shape, and volume tasks, with per-task training details deferred to Appendix A.5.Source: §5.2 · p. 7, §5 · p. 7

5.2 End-to-end meta-learning

• At K = 3 inner steps, MetaLF outperforms Functa and ENF on CIFAR10, CelebA, and ImageNet-1K under both tested bi-invariant settings.The authors report near-lossless CIFAR10 reconstruction, substantial gains on the higher-resolution datasets, and little sensitivity to geometric-group choice.Source: §5.2 · p. 7, Table 2 · p. 7
• Table 3 reports MAML-based shape-reconstruction IoU on ShapeNet-Part, ShapeNet-Core, and ACDC hold-out test sets, averaged across three seeds.The supplied passage identifies the table's scope but does not state its comparative outcomes.Source: Table 3 · p. 8
• Table 4 reports classification and reconstruction results for CIFAR10, ImageNet-1K, and ShapeNet-Core, with task-specific metrics and aggregation schemes.The supplied passage identifies the table's scope but does not state its comparative outcomes.Source: Table 4 · p. 8
• MetaLF consistently outperforms evaluated few-step segmentation baselines on OMBRIA, ShapeNet-Part, and OASIS, using reconstruction-only adaptation at test time.On OASIS it reaches 86.9% foreground Dice, versus 81.0% for NISF and 91.0% for MetaSeg; the published protocols differ, and MetaSeg uses 100 fitting steps versus five for MetaLF.Source: §5.2 · p. 8
• MetaLF benefits most from datasets with shared underlying structure, while gains are more modest on heterogeneous data.The authors suggest latent attention is most effective when the decoder can learn common structure across samples.Source: §5.2 · pp. 8–9
• Global attention produced overly smooth reconstructions, whereas localized attention improved reconstruction quality and training stability.Increasing the attention neighbourhood also requires additional attention matrices for neighbours' relative geometry; second-order training and iterative test-time optimization remain computational costs.Source: §5.2 · p. 9

• Second-order meta-learning trains latent representation construction alongside the decoder, making decoder architecture part of inference design.The conclusion frames few-step latent adaptation as an optimization encoder.Source: §6 · p. 9
• Equivariant self-attention coordinates spatially grounded latents, shaping gradient updates so local observations inform coherent representations.The controlled analysis links this coordination to more compact, function-aligned latent geometry.Source: §6 · p. 9
• Separating inner encoding from outer task supervision supports end-to-end classification and segmentation through reconstruction-only adaptation at test time.The conclusion also reports improved few-step image and shape reconstruction.Source: §6 · p. 9

• MAML differentiates through the full sequence of inner updates, whereas FOMAML computes the outer gradient at the final adapted parameters while ignoring the optimization trajectory.The two methods use the same forward adaptation process but differ in whether the outer gradient passes through it.Source: §A.1 · p. 12
• For conditional neural fields, the inner loop updates each sample's latent code from a shared initialization, while the outer loop updates shared decoder parameters.The decoder stays fixed during adaptation but determines the gradients that update the latent code.Source: §A.1 · p. 12
• The algorithm specifies inner and outer learning rates, an inner-step count, and a choice between MAML and FOMAML.It initializes a shared decoder and latent point set, then samples signals and inner and outer observations.Source: §A.1 · p. 12

• Each inner update introduces decoder dependence and transports dependencies accumulated at earlier steps through subsequent updates.The backward recursion computes the same decoder gradient as the expanded meta-gradient; a learned shared initialization receives a gradient through the same recursion.Source: §A.2 · pp. 12–13
• Full MAML's decoder gradient combines a direct outer-loss term with a term that backpropagates through the optimization encoder.The latter carries the outer objective's dependence on how latent updates constructed the representation.Source: §A.2 · p. 13
• First-order training detaches the adaptation trajectory, eliminating the gradient-through-encoder term while retaining the direct decoder gradient.It runs the same forward optimization encoder but does not backpropagate the outer objective through the computation that constructed the representation.Source: §A.2 · p. 13

A.3.1 Cross-attention

• MetaLF builds on ENF's equivariant cross-attention, with field coordinates querying contextualized latent features through position relative to each latent pose.The coordinate query has no content feature; the latent feature is contextualized by the self-attention stack.Source: §A.3.1 · p. 13
• Cross-attention aggregates over the K_CA nearest latents with a Gaussian spatial window and normalized attention weights.A single cross-attention head aggregates their contributions.Source: §A.3.1 · p. 14
• A residual GeGLU block refines the aggregated feature pointwise before output projection while preserving equivariance.The block adds nonlinear refinement after spatial aggregation.Source: §A.3.1 · p. 14

A.3.2 Equivariance

• Self-attention preserves aggregated features because its attributes, spatial windows, and Euclidean nearest-neighbor sets remain unchanged under translations and rotations.Feature-only operations such as layer normalization, GeGLU, and residual additions preserve this property through the stack.Source: §A.3.2 · p. 15
• Joint transformations preserve the relative attributes used in cross-attention, leaving its positional queries, keys, values, and attention weights unchanged.The same bi-invariance applies to both attention operations.Source: §A.3.2 · p. 15
• For decoded signed-distance fields, rigidly transforming latent poses moves the represented shape while preserving corresponding field values.The value at gx after transformation equals the value at x before transformation.Source: §A.3.2 · p. 15

A.3.3 General implementation details

• MetaLF initializes latent poses on a noisy grid, uses Meta-SGD and tanh stabilization, and finds geometric-feature-based FiLM modulation converges more stably than content-feature modulation.The geometric feature is more consistent during training, while content-feature influence starts at zero and can grow as needed; implementation uses JAX for MAML.Source: §A.3.3 · p. 15

A.4.1 Polynomial-field implementation and geometry metrics

• Polynomial-field diagnostics use degree-six polynomials on a 32 × 32 grid to measure update-scaled content effective rank and the fraction of local variation within the polynomial subspace.Training uses 500 epochs, 256 observations per inner loop, and three full second-order MAML updates; evaluation fits each field from a deterministic 256-coordinate mask.Source: §A.4.1 · p. 16

A.4.2 FOMAML and MAML training behaviour

• On CIFAR10, full MAML’s latent effective rank rises as reconstruction improves, whereas FOMAML’s rank declines and its reconstruction quality plateaus.MAML generally has a lower upper response scale; a fourth update slightly improves MAML, but further updates worsen reconstruction for both methods beyond the three-step training horizon.Source: §A.4.2 · p. 17, Figure 6 · p. 17
• Mean off-diagonal encoding coupling rises from 0.36 at NSA = 0 to 0.67 at NSA = 4, while decoding routing mass and row entropy increase from 0.26 to 0.69 and 0.28 to 0.63.Increasing self-attention spreads token influence across larger neighborhoods, making the optimization encoder’s inner problem more strongly coupled.Source: §A.4.2 · p. 17, Figure 7 · p. 18

A.5.1 Image reconstruction

• Image reconstruction baselines show notably higher Functa PSNR on CelebA and ImageNet-1K, with CIFAR10 values of 38.1 dB versus 36.2 dB.The baselines were rerun with slightly different parameterization and longer optimization time, which may account for some differences.Source: §A.5.1 · p. 19

A.5.2 Shape reconstruction

• Shape reconstruction represents ShapeNet objects with per-object SDFs and ACDC frames with three jointly decoded SDF channels, using full second-order MAML with five latent updates.ShapeNet uses a tanh-transformed SDF data term; ACDC adds surface, eikonal, and balanced inside/outside penalties to its clipped SDF loss.Source: §A.5.2 · p. 19

A.5.3 Meta-classification

• MetaLF reaches 42.6% ImageNet-1K top-1 accuracy, 18.5 points above MWT’s 24.1%, and 83.3% CIFAR10 accuracy versus two-step ENF’s 82.1%.MetaLF uses only random horizontal flips, while ENF uses 50 random crops with flips per image; reconstruction quality is typically lower when classification is also an outer objective, reflecting a trade-off between the tasks.Source: §A.5.3 · pp. 20–21

A.5.4 Meta-segmentation

• For meta-segmentation, the inner loop adapts latents using reconstruction supervision, while segmentation loss trains the outer objective; test-time latent adaptation requires no segmentation labels.The decoder predicts signal values and segmentation logits as separate output channels of the same network.Source: §A.5.4 · p. 21
• Some baseline segmentation entries are incomplete: Functa collapsed and SpatialFuncta diverged late, while global Functa failed to fit or diverged on ShapeNet-Part and OASIS.These numeric entries do not represent completed 200- and 100-epoch runs, respectively; the final OASIS attempt failed before non-finite gradients stopped training at step 12,542.Source: §A.5.4 · pp. 22–23
• MetaLF often performs similarly to SpatialFuncta and ENF when local information suffices, but suppresses false-positive flood predictions around cloudy regions in the median OMBRIA example.The authors attribute this qualitative advantage to non-local latent interactions that let spatially separated evidence inform local predictions.Source: §A.5.4 · p. 23, Figure 8 · p. 23

A.6.1 Parameterization and computational costs

• Spatial Functa has the lowest evaluated costs; within MetaLF, added self-attention layers increase parameters and latency but comparatively modestly increase FLOPs.ENF shows that fewer parameters do not necessarily mean less computation because its wide query-dependent transformations are repeatedly evaluated across the image.Source: §A.6.1 · p. 23, Table 11 · p. 24
• Table 10 reports CIFAR-10 validation-set reconstruction PSNR for MetaLF capacity variations, with rows varying latent count N and grouped columns varying latent channels D and self-attention depth NSA.Source: Table 10 · p. 24

A.6.2 Generative modeling

• On CelebA, MetaLF improves over ENF under the same DiT backbone: FID decreases from 18.82 to 13.05 and coverage increases from 70.51% to 86.43%; CIFAR10 results are similar.The authors relate the CelebA gains to shared spatial regularities in centered faces that latent self-attention can exploit.Source: Table 12 · p. 25, §A.6.2 · p. 25
• With a parameter-matched, 12-layer, width-384 MetaLF-Diffuser, CIFAR10 FID improves from 20.08 to 18.83 and coverage from 90.56% to 92.87%; CelebA FID is similar at 13.38 versus 13.05 for DiT.Both backbones are trained with flow matching for 500 epochs.Source: Table 12 · p. 25, §A.6.2 · p. 25