PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Beyond the Question: Preserve-Focus-Fuse for Whole-Slide Image Question Answering

Xiangyu Li, Jincheng Li, Tianle Du, Yanheng Li, Haoxuan Che, Yuxin Sun, Long Bai, Liangliang Liu, Ran Su

arXiv:2610.29431v1cs.CV

TL;DR

Whole-slide questions may omit morphology needed for an answer, while broad compression can dilute focal evidence. BeyondPath preserves morphology across questions, retrieves from all foreground patches, and fuses both streams; it reports 0.769 Avg-9 on WSI-Bench and leads report generation.

  • Problem

    Whole-slide questions may leave supporting morphology implicit, while broad compression can dilute focal evidence and question-guided selection can omit it.

    Source: §1 · p. 2

  • Method

    BeyondPath learns a question-agnostic Morphology Bank through cross-question answer supervision, retrieves question-specific evidence from all foreground patches, and fuses both at evidence positions in a 96-token prefix.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    0.769 Avg-9 versus 0.754 for WSI-LLaVA; BeyondPath also leads report generation and reaches 0.639 zero-shot accuracy on SlideBench-BCNB.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support retaining broader slide morphology while still seeking evidence for each question, rather than relying only on explicit question cues.

    Source: §5 · p. 12

    Evaluation uses retrospective public cohorts, with external transfer assessed only on SlideBench-BCNB; the implementation covers one CONCH–Qwen configuration and one 20× scale.

    Source: F. Limitations · p. 24

Abstract

from arXiv · show

A question about a whole-slide image rarely names every morphological finding needed to answer it. Question-guided patch selection can therefore overlook supporting morphology, while slide-level compression can dilute focal evidence. Inspired by how pathologists combine broad slide review, question-directed examination, and integration of their findings, we introduce BEYONDPATH, a Preserve-Focus-Fuse framework for whole-slide image question answering. Preserve learns a question-agnostic Morphology Bank through answer supervision across multiple questions about each training slide. Focus retrieves questionspecific evidence from all foreground patches, including regions outside the Bank. Fuse integrates the preserved morphology at the retrieved evidence positions through Morphology-on-Visual-Evidence Fusion, producing a fixed 96-token visual prefix. On WSI-Bench, BEYONDPATH achieves an Avg-9 of 0.769 and leads all eight reported report-generation metrics using the same checkpoint. It also achieves 0.639 zero-shot overall accuracy on SlideBench-BCNB. Controlled ablations show complementary benefits from morphology preservation and question-directed retrieval, and support cross-question supervision and fusion at the evidence positions. These results support looking beyond the question's explicit cues: preserving broader slide morphology while retaining the ability to seek evidence for each question.

1. Introduction

Whole-slide image question answering must find evidence across gigapixel tissue, but a question may not name every morphology needed to answer it. BeyondPath preserves morphology across questions, retrieves evidence from the full foreground set, and fuses both for answering.

1. Introduction

  • Question-guided selection can omit supporting morphology, while broad compression can dilute focal evidence and prevent retrieval from recovering excluded regions.The paper describes this as the Relevance–Sufficiency Gap: question-matched patches may not provide all findings needed to answer.Source: §1 · p. 2
  • Inspired by pathological slide review, Preserve–Focus–Fuse keeps a broad morphological impression, seeks question-relevant findings, and integrates them for interpretation.BeyondPath learns a shared Morphology Bank from answers to multiple questions per slide; Focus searches all foreground patches, and MoV Fusion combines both at retrieved evidence positions in a 96-token visual prefix.Source: §1 · p. 2
  • 0.769 Avg-9 versus 0.754 for WSI-LLaVA; BeyondPath also leads report generation and reaches 0.639 zero-shot accuracy on SlideBench-BCNB.Controlled comparisons show benefits from the two evidence streams, cross-question supervision, and evidence-position fusion.Source: §1 · pp. 2–3

2. Related Work

Prior pathology vision–language work improves image–text representations and language interaction, while WSI-VQA must organize slide-level morphology for each question. BeyondPath distinguishes itself by preserving cross-question morphology and retrieving from the full foreground set before fusion.

Pathology vision–language modeling

  • Pathology vision–language pretraining and instruction-following models improve image–text alignment and language interaction, but WSI-VQA also requires organizing slide-level evidence per question.Examples include PLIP and CONCH, LLaVA-Med, and Quilt-LLaVA.Source: §2 · p. 3

Whole-slide multimodal reasoning

  • WSI-VQA methods use patch–word co-attention, long patch sequences, global slide representations, regional pooling, or question-guided patch selection.BeyondPath instead learns a separate Morphology Bank from cross-question answers, retrieves over the full foreground set, and fuses at evidence positions.Source: §2 · p. 3, Figure 2 · p. 4

Visual token compression and selection

  • Visual-token methods select, prune, merge, or diversify tokens, often through one reduced stream or a staged candidate set.BeyondPath organizes morphology under a fixed language-side token budget by combining cross-question morphology with question-specific findings in the evidence sequence.Source: §2 · pp. 3–4

3. Method

BEYONDPATH separates slide-wide morphology preservation from question-specific evidence retrieval, then fuses both into a fixed-length visual prefix. A two-stage schedule first learns a shared Morphology Bank from multiple questions per slide, then trains retrieval and fusion while keeping Preserve fixed.

3. Method

  • The method builds a fixed-length visual prefix by fusing question-specific evidence with a question-agnostic Morphology Bank, with only the evidence sequence passed to the language model.The default uses six 16-token experts for a 96-token Bank and selects 96 evidence tokens.Source: §3 · pp. 4–5

3.1. Preserve: Question-Agnostic Morphology Selection

  • Preserve learns one shared Bank per slide from answer losses across its training questions, while its patch-scoring router receives no question embeddings.Four semantic experts target tumor, stroma, immune infiltration, and necrosis; two free experts can retain features beyond those anchors.Source: §3.1 · pp. 5–6

3.2. Focus: Question-Conditioned Evidence Retrieval

  • Focus scores all retained foreground patches for the current question, including patches outside the Bank, using pretrained image–text similarity and learned question–patch compatibility.At inference, it selects 96 evidence anchors using hard Top-K.Source: §3.2 · pp. 6–7

3.3. Fuse: Morphology-on-Visual-Evidence Fusion

  • MoV uses evidence tokens as queries over expert banks and applies position-specific gates, integrating morphology at the evidence positions while retaining 96 outputs.This avoids exposing 192 concatenated Bank and evidence tokens to the language model.Source: §3.3 · pp. 7–8

3.4. Two-Stage Learning and Inference

  • Training first learns Preserve through cross-question answer supervision, then freezes the Preserve path while optimizing Focus, MoV, the projector, and rank-32 Qwen LoRA parameters.At inference, the Bank is rebuilt for each slide-question request, and two MoV blocks update the 96 selected anchors before decoding.Source: §3.4 · p. 8

4. Experiments

Experiments evaluate BeyondPath on WSI-Bench and SlideBench-BCNB, then test its component streams, Bank supervision, fusion, and qualitative behavior. The model leads key WSI-Bench measures, all eight reported report-generation metrics, and several zero-shot BCNB endpoints.

4.1. Experimental Setup

  • Evaluation covers WSI-Bench performance and design choices, then zero-shot transfer to SlideBench-BCNB without target-cohort adaptation.The controlled comparisons use the official WSI-Bench test split and share cached CONCH features, the Qwen backbone, training annotations and schedule, and evaluator.Source: §4.1 · p. 9

4.2. Main Results

  • 0.769 Avg-9, versus 0.754 for WSI-LLaVA, with BeyondPath leading the WSI-P and WSI-R measures for morphological analysis and diagnosis.The evaluation uses a 96-token visual prefix.Source: §4.2 · p. 9
  • 0.639 question-count-weighted Overall accuracy on zero-shot SlideBench-BCNB accompanies leads on HER2 expression, HER2 status, and molecular subtype.The best-performing model varies by endpoint; Table 2 reports zero-shot accuracy with Overall weighted by task question counts.Source: §4.2 · p. 10, Table 2 · p. 10
  • The same Stage-II checkpoint leads all eight reported WSI-Bench report-generation metrics, spanning lexical and pathology-oriented measures.The checkpoint transfers the visual interface from focused question answering to longer-form slide description.Source: §4.2 · p. 10, Table 3 · p. 10

4.3. Controlled Analyses

  • The full model scores 0.769 Avg-9, above Focus only (0.721), Preserve only (0.713), Bank-restricted Focus (0.718), and single-stream scoring (0.739).Table 4 reports WSI-Bench Avg-9 ablations, generally with 96-token prefixes; Bank-restricted Focus limits retrieval to Preserve-selected patches.Source: §4.3 · p. 11, Table 4 · p. 11
  • All-QA Preserve scores 0.769 versus 0.736 for Single-QA Preserve; MoV scores 0.769 versus 0.727 for resampling, 0.739 for 192-token concatenation, and 0.756 for uniform gates.Random same-slide and wrong-slide Banks score 0.713 and 0.677, respectively.Source: §4.3 · p. 11
  • Increasing both budgets to 192 yields 0.772; four-block MoV and four semantic-only experts score 0.765 and 0.759, respectively.Because both selectors scan all patches, these are interface rather than retrieval-efficiency comparisons.Source: §4.3 · p. 11

4.4. Qualitative Analysis

  • Two pathologist-reviewed cases show Focus selecting diagnostic patterns, Preserve supplying broader morphology, and MoV combining both at evidence positions.The examples cover FIGO grading and lung classification.Source: §4.4 · p. 11, Figure 4 · p. 11, Figure 5 · p. 11

5. Conclusion

The conclusion presents BeyondPath as a Preserve–Focus–Fuse framework, with cross-question supervision for the Morphology Bank and MoV fusion in a 96-token prefix. The accompanying protocol describes WSI-Bench partitions and question-count weighting for BCNB Overall accuracy.

5. Conclusion

  • BeyondPath combines a cross-question Morphology Bank, Focus retrieval from all foreground patches, and MoV fusion in a 96-token prefix.The conclusion frames this as preserving broad morphology while seeking findings for each question.Source: §5 · p. 12

A.1. Data Partitions and WSI Processing

  • WSI-Bench uses an official split of 9,642 training WSIs with 175,450 question–answer pairs and 208 test WSIs with 4,119 pairs.Fifteen percent of training patients are reserved for checkpoint selection, and patient slides and questions stay within one partition.Source: §A.1 · p. 14

A. Reproducibility and Evaluation Protocol

  • Main comparisons and controlled ablations use the official test split; no test WSI or question–answer pair updates model parameters.The WSI-Bench-trained VQA model is evaluated directly on SlideBench-BCNB without target-cohort training or checkpoint selection.Source: §A.1 · p. 14

A.2. BCNB Score Aggregation

  • BCNB Overall accuracy weights task accuracies by question counts because the seven tasks have unequal question totals.HER2 expression and HER2 status remain separate tasks, each weighted by 1,058 of 7,274 questions.Source: §A.2 · p. 14

A.3. Baseline Provenance and BCNB Evaluation Join

The appendix distinguishes locally reproduced general-model BCNB evaluations from values reported by the original model papers. It also retains HER2 expression and status as separate task scores and describes tissue processing and cached patch features.

A.3. Baseline Provenance and BCNB Evaluation Join

  • General-model BCNB rows are local or reproduced evaluations under the stated normalization and endpoint protocol, not verbatim scores from the model papers.Outputs are normalized and mapped to the official seven-label vocabulary; ambiguous or unmappable outputs count as incorrect.Source: §A.3 · p. 14
  • Unlike the merged HER2 score reported by PathNavigate [Yang et al., 2026], this evaluation keeps HER2 expression and status separate and computes Overall from all seven task counts before rounding.The merged value is not used to reconstruct either component score.Source: §A.3 · pp. 14–15
  • At 20×, tissue processing retains patches with more than 10% overlap with the tissue mask and extracts non-overlapping 224 × 224 patches.Background tiles are discarded, and patch coordinates are retained for recovery and visualization.Source: §A.3 · p. 15
  • Each retained patch is encoded once with frozen CONCH V1; controlled experiments reuse cached features so ablations change evidence organization while holding visual input fixed.The pipeline caches both 512-dimensional pre-normalization features and their L2-normalized copies.Source: §A.3 · p. 15

A.4. Architecture and Optimization

The architecture uses a shared 512-dimensional visual workspace, a frozen Qwen2.5-7B-Instruct base with Stage-II LoRA, and a continuous visual prefix. The appendix also describes optimization and prompt-derived semantic morphology experts.

A.4. Architecture and Optimization

  • Preserve uses four semantic and two free experts with 16 patch anchors each; Focus selects 96 evidence anchors, and two MoV blocks fuse the views.The visual projector maps the fused representation into the hidden space of the Qwen2.5-7B-Instruct continuous visual prefix.Source: §A.4 · p. 15, Table 6 · p. 16
  • Stage I runs five epochs and Stage II three; both use AdamW, bfloat16 mixed precision, gradient clipping, warm-up, cosine decay, and weight decay.Stage I averages question-micro-batch gradients for each sampled WSI before a Preserve update; the selected checkpoint has the highest validation Avg-9.Source: §A.4 · p. 15
  • VQA uses greedy decoding up to 128 new tokens, while report generation allows up to 512; closed-ended outputs that cannot map unambiguously to an allowed label count as incorrect.Report-generation lexical metrics use the official WSI-Bench scripts, and WSI-P and WSI-R use released GPT-4o claim-scoring prompts with deterministic decoding.Source: §A.4 · p. 16

A.5. Morphology Prompt Ensembles

  • Semantic experts use averaged, re-normalized CONCH text embeddings from prompt ensembles, while the two free experts have no text anchors.Table 7 is identified as the prompt ensembles for the four semantic morphology experts.Source: §A.5 · p. 16, Table 7 · p. 16

B. Exact Method Details

The method builds a question-independent Morphology Bank, retrieves question-specific evidence from all foreground patches, and fuses the two streams at fixed evidence positions. Its implementation uses hard-forward selection with relaxed gradients, expert banks, and a two-stage training and inference flow.

B.2. Hard-Forward, Relaxed-Backward Top-K

  • The straight-through Top-K operator selects the hard subset forward while sending gradients through the relaxed selection matrix.Only hard selection is evaluated after Preserve is frozen and at test time.Source: §B.2 · p. 17

B.3. Semantic Presence and Expert Diversity

  • Preserve’s six experts reserve Bank capacity for tumor, stroma, immune infiltration, necrosis, and two unanchored categories, with semantic presence weighting the anchored experts.The reported configuration selects the 50th-percentile threshold for all four semantic experts; free experts use ρs,e = 1.Source: §B.3 · pp. 17–18

B.4. MoV and Visual-Prefix Implementation

  • MoV uses evidence tokens as queries to read expert-bank keys and values, updates only evidence positions, and keeps the output length at Kq.The projected 512-dimensional tokens enter Qwen’s 3584-dimensional hidden space between learned visual boundary embeddings.Source: §B.4 · p. 18

B.5. Training and Inference Data Flow

  • Stage I trains a shared Bank from cross-question answer losses while Stage II freezes Preserve and trains Focus, MoV, the projector, and Qwen LoRA.The same slide’s Bank is used during Stage II fusion, while inference reconstructs it for each request.Source: §B.1 · pp. 16–17, §B.5 · p. 19
  • At inference, Preserve constructs a Bank for each request, Focus retrieves 96 anchors from all foreground patches, and MoV produces a 96-token prefix for Qwen.Preserve is frozen in Stage II; no Bank is cached across questions about the same slide.Source: §B.5 · p. 19

C.1. Complete Ablation Breakdown

The section tests how evidence selection, morphology preservation, fusion, and supervision affect WSI-VQA, then characterizes parameter sensitivity and visual-interface cost. The reported 96-token setting captures most of the observed gain over 192 tokens with lower computational cost and nearly unchanged memory use.

C.1. Complete Ablation Breakdown

  • The ablations compare Focus-only, Preserve-only, single-stream selection, Bank-restricted Focus, random interventions, and alternative fusion interfaces against the full model.Most variants use 96 language-side tokens; direct concatenation uses 192, and intervention variants include random same-slide or wrong-slide Banks and Random Focus.Source: §C.1 · p. 19, Table 9 · p. 20
  • The question-count comparison varies how many distinct QA answers supervise each slide while holding five Stage-I passes and answer-loss normalization fixed.Smaller settings use nested prefixes of a fixed random question order; All-QA uses every available question.Source: §C.1 · p. 20

C.2. Parameter Sensitivity

  • Increasing each stream from 32 to 96 tokens is associated with a 0.027 Avg-9 increase, while doubling the default budget to 192 changes Avg-9 by 0.003.These sensitivity sweeps characterize the operating region and were not used for checkpoint selection.Source: §C.2 · p. 20, Table 10 · p. 21

D.1. Profiling Protocol

  • Profiling times visual selection, fusion, language-model prefill, and generation under a fixed protocol, excluding offline image and feature extraction.Measurements use pre-extracted CONCH features, batch size one, bfloat16 inference, and at most 128 generated answer tokens.Source: §D.1 · pp. 21–22, Table 11 · p. 22, Table 12 · p. 22

D.2. Visual Budget and Cost

  • From 96 to 192 tokens per stream, Avg-9 increases by only 0.003 while peak memory rises by 2.0% and TFLOPs by 25.6%.The authors describe 96 tokens as capturing most of the observed gain at lower computational cost and nearly unchanged memory use.Source: §D.2 · p. 22

D.3. Model-Level Visual Interfaces

The model processes retained foreground features during Preserve and Focus but sends only 96 fused evidence anchors to Qwen. Its measured latency rises gradually with patch count in the tested range.

D.3. Model-Level Visual Interfaces

  • 15.95 GB peak memory and 3.304 TFLOPs are reported for BeyondPath, which uses fewer language-side visual tokens and lower measured TFLOPs than SlideChat under the stated profiling setup.All retained foreground features are examined, while Qwen receives only 96 fused evidence anchors.Source: §D.3 · p. 22

D.4. Scaling with Foreground Patch Count

  • At 32,768 patches, BeyondPath requires 2.65 seconds, compared with 10.72 for SlideChat.The downstream visual sequence remains fixed while slide-wide scoring is linear in the foreground patch count.Source: §D.4 · p. 23

E. Qualitative Analysis

Qualitative cases illustrate how preservation and question-directed retrieval contribute different evidence to grading and classification answers. The displayed candidate scores are normalized scoring outputs, not calibrated class probabilities.

Case 3

  • The full model predicts the reference Grade 2, scoring it 0.78, while Focus-only, Preserve-only, and random-patch variants favor Grades 3, 1, and 3; its NLL is lowest at 0.29.The other displayed NLL values are 0.91, 1.12, and 0.98, respectively.Source: E. Qualitative Analysis · pp. 23–24

Case 4

  • For invasive ductal carcinoma, the full answer combines invasive glands in desmoplastic stroma with a cribriform DCIS component, while single-stream variants omit either invasion or subtype specificity.The full model’s displayed NLL is 0.56 versus 1.11, 1.34, and 1.42 for the alternatives.Source: E. Qualitative Analysis · p. 24

Scoring

  • The grading candidate scores are normalized from exponentiated negative NLL values and are not calibrated class probabilities.The displayed NLL is teacher-forced and length-normalized, excluding visual, question, boundary, and EOS tokens from the average.Source: E. Qualitative Analysis · p. 24

F. Limitations

The evaluation and implementation have several scope and validation limits: testing uses retrospective public cohorts, and external transfer is assessed only on SlideBench-BCNB. The selected patches have not been systematically checked against expert annotations across the test set, and Focus scans all retained foreground patches for each question.

F. Limitations

  • External transfer is assessed only on SlideBench-BCNB, and the evaluation uses retrospective public cohorts.Source: F. Limitations · p. 24
  • The implementation is limited to one CONCH–Qwen configuration and a single 20× scale.Source: F. Limitations · p. 24
  • All reported runs use random seed 42, without multi-seed performance variance or confidence intervals for predictive metrics.Source: F. Limitations · p. 24
  • Selected patches are not pathologist-verified localizations, and selections across the test set lack systematic validation against expert-annotated regions.The reported correspondence is functional, not a claim of clinical equivalence to expert review.Source: F. Limitations · p. 24
  • Focus scans all retained foreground patches per question, so the 96-token fused prefix reduces decoder context rather than slide-wide retrieval cost.Source: F. Limitations · p. 24

Related papers

Loading 2610.29431v1…