PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Aligned Local Composition of Discrete Experts for Sequence Restoration

Mohammad Panahazari, Usman A. Khan, Shuchin Aeron

arXiv:2609.05801v1cs.AI

TL;DR

Restoring corrupted mixed-domain documents requires recovering missing symbols while deciding which frozen expert composition to trust at each position without labels or a trained router. The paper infers a smoothed, soft local composition from corrupted-observation evidence estimated with denoising losses, and finds that field recovery and restoration improve over global weighting when experts are distinct. The method is approximate in important parts of the optimization and relies on sequence-level targets that may not be exactly solved for contextual experts.

  • Problem

    Restoration of corrupted mixed-domain sequences requires jointly recovering missing content and position-wise expert composition without region labels or a trained router.

  • Method

    Evidence-aligned local composition infers a soft simplex-valued expert field from corrupted-observation marginal evidence, estimated with denoising losses and smoothed across positions.

  • Results

    The inferred field tracks true regions and improves restoration over an independently optimized global weight on constructed mixtures; gains on natural documents track expert separation and vanish when experts converge.

  • Takeaways & Limitations

    Soft local weighting can represent mixed regions while concentrating on one expert when appropriate, matching the paper’s position-wise recovery objective.

  • Takeaways & Limitations

    The practical composition algorithm only approximates the target evidence objective, and contextual-expert sampling is not known to draw from the defined composed sequence model.

Abstract

from arXiv · show

A document modeled as a discrete sequence of tokens can be thought of as being generated from a composition of texts from different domains; a README file, for example, moves between prose, code, and configuration. When such a document is corrupted and only frozen domain experts are available, restoring it requires deciding both what is missing and which expert to trust at each position, at test time and without region labels or a trained router. We introduce evidence-aligned local composition, which infers a soft, position-wise weighting over the experts from the marginal evidence of the corrupted observation under a given corruption model, estimating the evidence from the experts' own denoising losses and smoothing the weights across positions. Because the weighting is soft, it recovers a mixture when the true composition is mixed and concentrates on one expert when that suffices. Across a categorical simulator, byte-level experts, and experts fine-tuned from a $1.3$B discrete flow-matching model, the inferred weights track the true regions at $0.85$ field accuracy on naturally mixed scientific documents, and at $0.98$ on constructed mixtures whose regions are lexically disjoint. Restoration improves over a single global weight when the experts are genuinely distinct and reduces to it when they converge, tracking a measure of expert separation.

1 Introduction

The paper addresses restoration of mixed-content documents by inferring both missing symbols and which frozen experts to trust at each position, without region labels or a trained router. Its evidence-aligned method uses corrupted-observation evidence to infer a smoothed local composition field and improves restoration when experts are distinct.

  • Mixed documents interleave domains such as prose, code, configuration, or mathematics, so restoration must recover missing symbols across changing content types.
  • The task couples missing-content recovery with selecting an expert composition at every position, without position labels or a trained router.
  • Existing global expert weights suit roughly uniform competence but fail when the preferred composition changes across positions.
  • The method infers a soft simplex-valued weight field from corrupted-observation marginal evidence, using an exponentiated-gradient update and positional smoothing.
  • Denoising losses provide local expert energies for estimating the evidence, while the deployed unweighted score remains a surrogate.
  • Without region labels, the inferred field tracks true regions and improves restoration over an independently optimized global weight on constructed mixtures; gains on natural documents track expert distinctness.

2 Related Work

Prior work combines frozen experts or diffusion priors, but commonly applies one weight to an entire input or relies on supervision. This paper instead infers position-dependent weights at test time from the corrupted observation and known corruption channel.

  • Products of experts, compositional diffusion, and model fusion commonly combine models through multiplied densities, summed log scores, or parameter-level merging.
  • Most compared approaches assign one weighting to the whole input, while position-dependent weights are otherwise supplied by supervision or confidence-based composition.
  • The proposed setting varies weights by position and infers them at test time from the corrupted observation and a given channel, without router training or region labels.
  • Discrete diffusion denoising objectives can upper-bound sequence negative log likelihood, while the paper targets local composition fields rather than single guided reconstructions.

3 Problem Formulation

The formulation assigns each position a simplex-valued expert composition and infers the field from corrupted-observation evidence under a composed sequence prior. Contextual experts require a normalized sequence-level energy model because their local conditionals do not generally define a valid joint distribution.

  • Problem setting: The problem gives contextual generative experts over sequences and seeks to recover a clean sequence from its corrupted observation under a known channel.
  • Local composition: Each position receives a simplex-valued weight: a simplex vertex routes to one expert, while an interior point composes several experts.
  • Local composition: Region structure means contiguous blocks generated by one expert, with near-vertex weights inside blocks and mixed weights near boundaries; labels are used only for evaluation.
  • Sequence-level composition: Because contextual experts depend on the rest of the sequence, multiplying position-wise conditionals need not yield a normalized joint distribution, motivating a direct sequence-level energy model.
  • Objective: The field is inferred from the corrupted observation by maximizing log marginal likelihood with a total-variation penalty that favors piecewise-constant weights.
  • Objective: For position-factorized experts and memoryless channels, the evidence separates across positions, whereas contextual experts retain the sequence-level objective.

4 Method

The method estimates local expert energies from denoising losses, compares posterior and prior evidence, and updates a position-wise simplex field with mirror ascent and smoothing. Its deployed estimator is evidence-aligned rather than an exact likelihood maximizer because it uses finite-sample scores and an approximate sampler.

  • Evidence gradient: The evidence gradient compares each expert’s local energy under sequences consistent with the observation against sequences drawn from the composed prior.An expert gains relative weight when its posterior-minus-prior energy gap is larger.
  • Evidence gradient: At masked positions, factorized models have zero local evidence gradients, so contextual dependence and neighbor smoothing must transmit evidence.This limitation motivates relying on sequence context or smoothing in single-observation experiments.
  • Local expert energies: Denoising losses provide tractable surrogates for local expert energies, with lower loss indicating greater compatibility with a candidate sequence.For factorized experts, the surrogate plays the role of token log density.
  • Practical estimator: The algorithm estimates prior and posterior expectations with particle sets, then uses their empirical averages to form the deployed gradient.The practical estimator substitutes finite-sample denoising scores for exact energies.
  • Optimization and smoothing: Mirror ascent uses an entropic geometry that keeps each iterate on the simplex, while a proximal smoothing step regularizes the position-wise field.The normalization is a Bregman projection, and the smoothing penalty can return a piecewise-constant field.
  • Optimization and smoothing: The forward mirror step and subsequent smoothing step are each exact for stated subproblems, but their composition is not the full joint optimization problem.This scope boundary is explicit in the derivation.

5 Experiments

The experiments evaluate evidence-aligned local composition across a categorical simulator, byte-level experts, pretrained experts, and naturally mixed documents. Results show that local evidence improves restoration and recovers composition fields, especially when experts are distinct, while gains disappear when experts converge.

  • Experimental setup: The evaluation spans an exact categorical simulator, learned byte-level experts, pretrained 1.3B experts, and naturally mixed documents.Methods share the frozen experts, corrupted input, channel, decoder, and source initialization within each experiment; only composition weights vary.
  • 5.1 Recovering a known composition field: 0.013 mean absolute error recovers the categorical simulator’s field, versus 0.333 for equal weights and 0.500 for a single expert.Replacing exact densities with trained flow-matching experts and a denoising surrogate raises error only to 0.031.
  • 5.2 Byte-level experts: 0.660 document-weighted accuracy is achieved by local evidence across 256 byte-level windows, above the best single expert, global weight, equal weights, and adapted baselines.The inferred field recovers three regimes at macro one-versus-rest AUROC 0.993, while shuffling the field reduces restoration.
  • 5.3 Pretrained experts at scale: 0.653 restoration accuracy exceeds the independently optimized global weight at 0.611 by +0.042 in the fine-tuned 1.3B setting.The field agrees with true regions at 0.995, while the privileged hard-label router reaches 0.655.
  • 5.5 Naturally mixed documents: 0.691 accuracy exceeds the global weight by 0.010 and recovers regions at field accuracy 0.852 on naturally mixed scientific documents.Local evidence is reported as the only method that both improves restoration over the global family and attributes regions faithfully.
  • 5.5 Naturally mixed documents: When prose and code experts converge on matched README corpora, restoration is indistinguishable for local and global weighting: 0.709 versus 0.708.The local field still provides regional attribution, with field accuracy 0.765.

6 Conclusion

The paper introduces evidence-aligned local composition for restoring corrupted discrete sequences with frozen experts and no region supervision or trained router. Its inferred position-wise field tracks regions, improves over global weighting when experts differ, and reduces to attribution when experts converge.

  • Evidence-aligned local composition infers a simplex-valued expert-weight field from corrupted observations rather than assigning one expert to the whole sequence.The method estimates an evidence gradient from experts’ denoising losses and smooths the resulting weights across positions.
  • 0.85 field accuracy is achieved on naturally mixed scientific documents, while 0.98 is achieved on constructed mixtures with lexically disjoint regions.
  • Local composition outperforms independently optimized global weighting on constructed windows and distinct natural-document regions.
  • The method reduces to attribution when README prose and code experts converge, while gains track expert separation in distinct domains.

Appendices

The appendices establish the evidence-gradient identities underlying local composition, derive the factorized position-wise form, and characterize when local evidence is informative or absent.

  • A.1 The sequence-level evidence gradient: The appendix defines a finite-state composed prior and an arbitrary corruption channel, then proves smoothness and the sequence-level evidence-gradient identity.The channel need not be memoryless, and expert energies may depend on the whole sequence.
  • A.1 The sequence-level evidence gradient: The gradient is a posterior-minus-prior difference of the same expert statistic, so an expert gains weight when it better scores observation-compatible tokens.
  • A.2 The factorized special case: The factorized special case makes the marginal objective separable across positions, yielding local objectives and gradients for each expert-weight vector.
  • A.2 The factorized special case: The local derivative requires full-support expert probabilities and uses the exponential-family score identity, whose score has zero mean under the current pool.
  • A.2 The factorized special case: A masked position contributes nothing to the factorized local gradient because its channel posterior equals the current prior.Visible observations can still contribute, and contextual experts may retain sequence-level information through cross-position dependence.

A.3 Contextual 2 × 2 validation

The contextual validation separates energy and sampling approximations in an enumerable model, then documents the experimental settings and expert configurations used for broader evaluations.

  • Experimental settings: The study evaluates factorized and contextual models, byte-level experts, pretrained experts, and naturally mixed README and scientific documents under mask or replacement channels.
  • A.3 Contextual 2 × 2 validation: Cosine 1.000 agreement with a finite-difference gradient validates the exact contextual gradient implementation.
  • A.3 Contextual 2 × 2 validation: The energy surrogate aligns at cosine 0.65, mean-field sampling at 0.92, and their deployed combination at 0.62.The energy approximation perturbs the gradient direction more than the sampler, but the experiment does not establish tolerance at scale.
  • A.3 Contextual 2 × 2 validation: Field recovery improves monotonically with the number of shared corrupted observations and closes about three quarters of the uniform initialization error.
  • Expert configurations: The prose–mathematics 1.3B pair shows balanced specialization margins of 0.47 and 0.46 nats, whereas the README-matched pair separates weakly.The scientific pair therefore favors local composition over global weighting, while the README setting ties.

B.3 The specialization check

The specialization check tests whether each expert assigns the lowest denoising energy to its own domain, identifying when local composition is operating in an accepted regime.

  • The byte-level experts pass the specialization check with diagonal margins of 1.03, 0.40, and 0.51 nats across prose, code, and configuration.
  • The 7B pair fails diagonal dominance because the general expert assigns lower energy to code than to prose.
  • Despite the failed diagnostic, the 7B stress test reaches 0.95 field accuracy and is reported outside the accepted regime.
  • A complementary byte-level pair that fails to specialize also fails to produce a usable field, supporting the diagnostic’s practical role.

B.4 Composition hyperparameters

The experiments use scale-specific recovery, particle, sampling, step-size, and smoothing settings, with smoothing operators varying by study. Only smoothing strength and corruption rate were searched, while most settings were fixed in advance; evaluation protocols and decoder choices were controlled but composition remains inference-costly.

  • Scale-specific settings: R, P, K, η, and τ differ across scales: byte uses R = 30, 48, P = 4, η = 1, τ = 0.6; 1.3B uses R = 12, 24, P = 2, η = 0.5, and τ = 0.4 or 0.3; 7B uses R = 6, eight nominal particles, and τ = 0.2.The field starts uniformly on the simplex, and the seed is 0.
  • Smoothing implementation: The smoothing operator is study-specific: fused-lasso or proximal-map smoothing is used for constructed mixtures, while natural documents use a width-five moving average.The three-expert study applies the map column-wise with clipping and row renormalization; natural-document smoothing uses fixed 1/5 taps and zero padding.
  • Corruption sensitivity: Method ordering is unchanged across corruption rates, although the byte-scale advantage over equal weights narrows from +0.101 at r = 0.1 to +0.078 at r = 0.3.At 1.3B, ordering persists through r = 0.7 with a margin near 0.05 over equal weights; these sweeps are descriptive because subsets and sample sizes differ.

C.2 Independently trained expert pairs

Independent retraining shows that local composition depends on expert specialization: a pair that fails the specialization diagnostic loses its advantage over a best single expert. Finer-grained code/comment mixtures remain harder, while decoder changes preserve method ordering.

  • Retrained pairs: 0.51 field accuracy and 0.449 local-evidence accuracy occur for the second retrained pair, whose local evidence is near the 0.447 best single expert.The failed composition coincides with a failed specialization diagnostic.
  • Retrained pairs: Expert specialization is an empirical prerequisite for the observed composition advantage, although the retraining study does not isolate it as the sole cause.Only two retrainings were run, so seed stability is not characterized.
  • Retrained pairs: 0.326 hard-router accuracy falls below 0.410 equal-weight accuracy for the nonspecializing pair, because hard labels no longer identify the better expert.Equal weighting retains averaging that hard routing discards when the pair does not specialize.
  • Finer-grained domains: 0.227 local-evidence accuracy matches the hard label router and exceeds the code expert, global weight, and equal weights on interleaved Python code and comments.The corresponding accuracies are 0.222, 0.212, and 0.188; lower absolute accuracy is consistent with greater vocabulary overlap.
  • Decoder sensitivity: Method ordering is unchanged under iterative decoding, and at 1.3B iterative decoding brings local evidence level with the privileged label router.The iterative analysis is secondary and uses window-weighted aggregation; decoding remains nonlinear in the composition field.

C.5 The smoothing prior

Smoothing helps at byte scale and improves both restoration and field accuracy at 1.3B, but its effect depends on scale and is not a universal scaling law. Field evaluation shows strong interior recovery but weaker boundary localization, while off-assumption and replacement-channel tests delimit the evidence.

  • Smoothing ablation: 0.348 unsmoothed byte-level accuracy falls below the 0.413 best single expert, while τ = 0.2 raises accuracy to 0.430 above single-expert and global baselines.Figure S2 likewise reports that unsmoothed local evidence falls below the best single expert.
  • Smoothing ablation: 0.679 restoration accuracy and 0.993 field accuracy are reached at 1.3B with τ = 0.4, versus 0.659 unsmoothed field accuracy and 0.636 for the field-average global reference.Accuracy varies by only 0.0003 across τ ∈ {0.2, 0.4, 0.6}, indicating a plateau rather than a sharp optimum.
  • Scope and assumptions: Two ablations are insufficient to establish a general scaling law for smoothing.The observed contrast is consistent with reduced reliance on smoothing when expert signals are stronger, but does not establish that relationship generally.
  • Field recovery: 0.987 interior accuracy contrasts with 0.687 accuracy near transitions and 0.720 boundary F1 within four positions of a regime change.Macro F1 is 0.972, IoU is 0.948, and segment F1 is 0.979; interiors are recovered more reliably than exact transitions.
  • Field recovery: 0 boundary F1 for constant global and equal-weight references means these fields cannot test boundary recovery.Their field accuracies are 0.669 and 0.334, reflecting the prevalence of the favored region.
  • Off-assumption stress test: 0.605 local-evidence accuracy in the 7B off-the-shelf stress test is not detectably different from the hard label router at 0.606, while exceeding the general model by 0.024.The pair fails specialization, the composed fill is deterministic, and the study contains only 64 windows.
  • Replacement corruption: 0.015 is the maximum accuracy gap from 1 − r under replacement corruption, where strong copying makes composition weights rarely change the decoder mode.This setting therefore does not test the field; a decoder able to overturn observed tokens is required for prior composition to affect output.
  • Scope and assumptions: The 1.3B and natural-document results use heuristic denoising energies outside the mask-source identity’s assumptions, leaving the uniform-source mismatch uncontrolled.A uniform-source analogue or mask-source fine-tuning is identified as a way to address the gap.

C.10 Independent baseline implementations

The baselines separate global, single-expert, local-router, and marginal-evidence selection strategies while holding experts, corruption, decoding, and initialization fixed. Local evidence outperforms the independent global baseline at byte, 1.3B, and arXiv scales, whereas several local references provide distinct comparison points.

  • Global baseline: 0.660 byte-level accuracy for local evidence exceeds 0.618 for the independently optimized global baseline by +0.042.The difference is statistically supported with p < 10^-3 and dz = 1.18.
  • Global baseline: 0.653 accuracy at 1.3B for local evidence exceeds the independent global baseline’s 0.611 by +0.042 over 256 shared windows.The 95% CI is [+0.039, +0.046], with p < 10^-3 and dz = 1.47.
  • Global baseline: 0.691 arXiv accuracy for local evidence exceeds the independent global baseline’s 0.682 by +0.010, while matching the jointly optimized weight within an insignificant −0.0006 difference.The jointly optimized reference is 0.682, with p = 0.61 for the difference.
  • Single-expert selection: 0.613 byte-scale and 0.599 1.3B accuracies are obtained by the best-single-expert selector, which uses posterior-score heuristics rather than marginal evidence.These heuristics omit free-energy and posterior-entropy terms.
  • Comparison protocol: All methods use the same experts, corrupted input, channel, decoder, and matched source initialization; only composition weights differ.Comparisons are paired over the declared statistical unit, with document- or window-level aggregation as specified.
  • Adapted local routers: DEMix routing uses per-position weights from visible-token scores and expands nearest-visible scores into corrupted positions before applying the shared smoothing.This adaptation is parameter-free and uses β = 1.
  • Adapted local routers: 0.684, 0.644, and 0.568 are HMM-router accuracies on README, arXiv, and 1.3B respectively, trailing local evidence at every reported scale.The HMM uses a two-state model over per-position visible-token scores with a symmetric transition prior and was not run at byte scale.
  • Evidence-based references: The marginal-evidence selector chooses the expert with the lowest mean visible-token score, estimating single-expert marginal evidence under the mask channel.It differs from the own-posterior selector, which scores each expert on its own reconstructions.
Loading 2609.05801v1…