RefRoute: Decoupling Conditioning Cost from References via Compact Residual Conditioning and Spatial Routing

• Dense concatenation makes conditioning grow with reference count and resolution; a 1024×1024 reference contributes 4096 visual tokens.Lowering resolution can discard high-frequency cues needed for faithful reconstruction, while unrestricted attention leaves reference-to-region assignment implicit.Source: §1 · p. 2
• RefRoute combines compact residual conditioning with condition and attention routing to address representation and interaction scaling together.The introduction argues that compact representations alone leave global attention intact, while spatial masking alone leaves resolution-dependent token counts.Source: §1 · p. 2
• A 1024×1024 reference is reduced from 4096 latent tokens to 256, while pixel-derived residual features retain fine-grained appearance cues.This fixed token budget makes representation cost independent of reference resolution and changes reference-dependent attention cost by roughly 70× at K = 16, ignoring text tokens.Source: §1 · p. 2
• Rotary-coordinate alignment and structured attention link references to assigned target regions, block direct interactions between distinct references, and retain selective access outside those regions.Outside assigned regions, target queries can access the highest-ranked reference, allowing information such as shadows and reflections to influence the surrounding scene.Source: §1 · p. 2
• 36.06 overall Weighted-Ref-VIEScore on ManyRef100 after many-reference fine-tuning, versus 8.88 for FLUX.2-Klein-9B; at 16 references, speedups are 18.3× and 14.2× over corresponding FLUX baselines.The introduction also notes a quality trade-off on the predominantly few-reference MICo-Bench.Source: §1 · p. 2
• RefRoute-Data and ManyRef100 support training and evaluation, with ManyRef100 covering human, object, and mixed compositions at 10–17 references per case.The data include aligned reference–instance–spatial annotations.Source: §1 · p. 3

• Multi-reference methods commonly concatenate reference-token blocks into a shared attention context, leaving the model to learn which reference controls each target instance.StructGen assigns explicit identifiers to references and relations, but retains a shared attention context; RefRoute binds references to designated target regions and restricts unnecessary interactions.Source: §2 · p. 3
• Existing efficiency methods address condition processing or sampling cost, but not conditioning-cost scaling with large numbers of appearance references.RefRoute focuses on this scaling problem with fixed-size per-reference representations and spatially restricted interactions.Source: §2 · p. 3
• Existing benchmarks assess reference fidelity, composition, and association, but do not combine large reference counts with instance-level spatial assignment and computational scaling.RefRoute-Data and ManyRef100 target this setting with aligned reference–instance–spatial conditions and evaluation of generation quality and scaling.Source: §2 · p. 3

3.1 PRELIMINARIES OF MULTI-REFERENCE CONDITIONED GENERATION

• The formulation encodes target and reference images with a VAE, concatenates their embeddings with text for joint DiT processing, and uses a flow-matching objective.The unified sequence grows with reference token counts, and unrestricted joint attention permits information from different references to mix across target regions.Source: §3.1 · p. 4

3.2 COMPACT RESIDUAL REFERENCE REPRESENTATION

• Downsampling reduces each 1024×1024 reference from 4096 to 256 tokens, while full-resolution pixel residual features supplement appearance cues without increasing sequence length.The residual is added to the corresponding projected reference tokens before the first DiT block.Source: §3.2 · pp. 4–5
• The Pixel Residual Compressor is pretrained on reference reconstruction and same-subject generation under changed poses, viewpoints, or contexts.Reconstruction encourages faithful appearance encoding, while subject-driven generation encourages transferable identity and appearance cues.Source: §3.2 · p. 5

3.3 CONDITION AND ATTENTION ROUTING

• Separate reference and spatial LoRA adapters operate in the specified streams, while rotary-coordinate remapping expresses compact-reference positions at target-latent scale.The reference adapter operates in text and visual streams; the spatial adapter operates in the visual stream and both are used in the combined stream.Source: §3.3 · p. 5
• Compact-reference coordinates are remapped to target-latent scale, and target queries access assigned references within their regions or only the top-ranked reference outside them.This preserves reference–region correspondence while allowing limited influence beyond assigned regions.Source: §3.3 · pp. 5–6
• The routing mask blocks direct exchange between reference streams while retaining shared text and spatial guidance for the other token groups.Reference queries access their own reference tokens, text, and spatial conditions, but not noisy target tokens or other references.Source: §3.3 · p. 5

3.4 DATASET CREATION

• RefRoute-Data provides per-instance references, boxes, and masks for 20k object scenes and 1k human scenes; its mean reference–target ArcFace similarity is 0.53 versus 0.19 for MICo-150K’s human subset.The cited MICo-Bench limitations include few-reference cases, missing instance-level spatial annotations, and human-reference identity mismatches.Source: §3.4 · p. 6

Experimental Setup

• Evaluations use 1024×1024 outputs in BF16 across MICo-Bench and ManyRef100, which contains 100 human, object, and mixed cases with 10–17 references.MICo-Bench has 897 cases, 89.2% of which contain two to five references.Source: §4 · pp. 6–7

Multi-Reference Composition

• Before additional many-reference fine-tuning, the undistilled and distilled MICo variants score 52.81 and 49.93, below their corresponding FLUX baselines of 55.67 and 55.18.The undistilled variant exceeds the reported Gemini-3-Pro-Image-Preview and GPT-Image-1.5 scores.Source: §4 · p. 6

Qualitative Comparison

• On MICo-Bench, RefRoute better preserves facial identity, clothing structure, and background details in a qualitative comparison with several systems.Figure 3 compares few-reference MICo-Bench examples and 10–17-reference ManyRef100 examples, with assigned regions overlaid in the latter.Source: §4 · pp. 7–8, Figure 3 · p. 7

Scaling with the Number of References

• At 16 references, RefRoute achieves 18.3× and 14.2× speedups over corresponding FLUX baselines at 50 and four sampling steps, respectively.Across K = 1 to 16, peak allocated memory at 50 steps rises 0.12 GiB, versus 9.05 GiB for the dense baseline.Source: §4 · p. 8, Figure 4 · p. 8

Multi-Reference Composition

• 36.06 overall Weighted-Ref-VIEScore on ManyRef100 after many-reference fine-tuning, versus 8.88 for FLUX.2-Klein-9B; Ours-ManyRef leads overall and per-category scores.Before fine-tuning, Ours-MICo scores 17.32, exceeding both FLUX.2 baselines.Source: Table 2 · p. 7, §4 · p. 8

5 ABLATION STUDY

• The ablation study tests compact residual representation and dynamic attention routing.Training configurations are provided in Appendix A.6.Source: §5 · p. 8

5.1 RESIDUAL REPRESENTATION DESIGN

• 24.78 dB PSNR for pixel-to-DiT on Stage 1 reconstruction, versus 23.57 dB for pixel-to-VAE and 24.22 dB for VAE-latent-to-DiT.With a matched Stage 2 budget of 10k steps and routing enabled, pixel-to-DiT also performs best across ManyRef100 categories and metrics; Human face similarity rises from 0.264 to 0.300, Object DINO similarity from 0.701 to 0.731, and Human RRBA from 0.390 to 0.517 versus concatenation without residuals.Source: §5.1 · p. 8

5.2 CONDITION ROUTING DESIGN

• With strict masking, the man appears to float above the ground; dynamic top-K routing preserves contact between his feet and the ground.The comparison uses the same checkpoint, seed, and prompt; dynamic routing allows queries outside assigned regions to access the highest-ranked reference.Source: §5.2 · p. 9, Figure 5 · p. 9
• For scenes requesting ten people, Stage 1 pretraining reduces person-count MAE from 1.643 to 1.057 after a matched 10k-step Stage 2 budget.On mixed scenes, MAE falls from 1.900 to 1.075.Source: §5.2 · p. 9, Table 4 · p. 9
• The study covers visual-reference conditioning, not long text conditioning or combinations of long prompts with many visual references.Representing and routing long-range semantic information in those settings is beyond the current framework’s scope.Source: §5.2 · p. 9

6 CONCLUSION

• RefRoute combines compact residual conditioning with spatially aligned condition and attention routing to address representation cost and attention overhead.The conclusion describes these mechanisms as complementary and reports improved ManyRef100 composition and more favorable inference-cost scaling.Source: §6 · p. 9
• Lower performance on the predominantly few-reference MICo-Bench highlights a challenge in maintaining generation quality across reference-count regimes.Source: §6 · p. 9
• The authors report using generative AI for synthetic data, data filtering, evaluation scoring, and drafting and editing parts of the paper.They state that research questions, method design, implementation, and result interpretation are their own, and that AI-assisted text was checked against experimental records.Source: AI USE STATEMENT · p. 10
• The work does not involve human-subject experiments or personal-information collection; applications involving sensitive visual content are outside its scope and may require additional safeguards.Source: ETHICS STATEMENT · p. 10

A APPENDIX

• The paper and supplement describe architecture, training, configurations, cost measurements, evaluation protocols, and benchmark construction; code and related materials are slated for release upon publication subject to licensing requirements.Source: REPRODUCIBILITY STATEMENT · p. 10
• The appendix follows the experimental claims with data and benchmark construction, evaluation and baseline configurations, implementation and training details, and cost measurements.Source: A APPENDIX · p. 14

Data Construction

• Each RefRoute-Data training record pairs a target image and scene description with per-instance references, bounding boxes, and masks.These annotations provide explicit correspondence between references and instances.Source: §A.1 · p. 14

Data scale

• The construction manifests contain 20,000 object scenes with 143,249 retained instances, plus 1,000 human scenes with 4,123 annotated people and 4,067 exported references.These totals describe the constructed data, while the training subset depends on checkpoint configuration; retained objects cover requested inventories, and exported human references may contain identity uncertainty.Source: §A.1 · p. 14

Training data construction

• Object and human data use separate branches in a shared four-stage construction pipeline.The stages include prompt generation, target synthesis, instance annotation, and verification and reference construction.Source: §A.1 · p. 14

Many-reference benchmark

• ManyRef100 contains 100 cases—30 Human, 30 Object, and 40 Mixed—with 10–17 references and explicit reference-to-instance spatial assignments.The benchmark supports controlled evaluation of many-subject composition; generalization from independently photographed references remains outside its scope.Source: §A.1 · p. 14

A.2 WORKED EXAMPLES AND CONSTRUCTION FLOW

• An audited object scene retains all eight requested instances, while a four-person identity-conditioned scene retains four matched original references.These examples illustrate the construction flow, not corpus-level annotation accuracy.Source: §A.2 · p. 15
• An eight-person text-conditioned scene exports seven generated references, with one flagged as identity-uncertain.The audited cases distinguish spatial annotations, reference availability, and reference quality.Source: §A.2 · p. 15
• Figure 6 depicts target synthesis, decomposition into boxes, masks, and references, followed by verification; the person branch also includes poses.Corpus counts and quality limits are specified in Appendix A.1.Source: Figure 6 · p. 15

A.3 EVALUATION SETUP AND BASELINES

• Published MICo-Bench scores follow each source’s inference settings and are benchmark comparisons, not controlled inference ablations.The four-step and 50-step comparisons use step-distilled and undistilled FLUX.2-Klein-9B checkpoints, respectively.Source: §A.3 · p. 15

A.4 EFFECT OF STAGE 1 PRETRAINING

• Stage 1 reproduces facial identity and garment details from compact references, including skirt pleats, a hat brim, and boot shape.The ground-truth target itself differs from the references in the face and some details.Source: §A.4 · p. 16

A.5 PIXEL RESIDUAL COMPRESSOR DETAILS

• The pixel residual encoder uses six stride-2 convolutions, then two spatial self-attention blocks, and projects features into the DiT hidden space for addition to low-resolution latent features.The convolutions use GroupNorm and SiLU; the projection is zero-initialized.Source: §A.5 · p. 16

A.6 TRAINING CONFIGURATION

• At K = 16, the recorded EMA36k student takes 7.84 s, compared with 111.10 s for the four-step FLUX.2-Klein-9B baseline.The measurement uses four sampling steps, CFG 1, and disabled reference KV caching; it does not establish generation quality.Source: §A.6 · p. 16
• Stage 2 uses 40k steps for the main experiments, while the high-K variant continues training on RefRoute-Data for 20k additional steps.The high-K variant totals 60k steps; Stage 3 follows TBSM for distillation.Source: Table 6 · p. 17
• The OminiControl2 reference-branch comparison uses an untrained condition adapter and differs in reference-branch topology, so the timing gap is not attributed solely to token count.The comparison characterizes mechanism cost rather than the complete trained system.Source: §A.6 · p. 17

B.1 RESULTS ON MICO-BENCH

• Figure 8 presents eight additional MICo-Bench cases, with references on the left, outputs on the right, and K indicating the number of references.The accompanying figure caption identifies the material as additional qualitative results on MICo-Bench.Source: §B.1 · p. 17, Figure 8 · p. 18
• Figure 8’s layout supports comparing each case’s references with RefRoute’s output; the passage does not state a comparative outcome.Source: §B.1 · p. 17

B.2 ADDITIONAL MANY-REFERENCE EXAMPLES

• Figure 9 shows four additional many-reference compositions, each using 11 to 14 references.Each case displays the reference images on the left and the output on the right.Source: §B.2 · p. 19