PaperlayerView exact revision on arXiv

Source-linked AI summary

DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression

Yiheng Su, Junyi Jessy Li, Matthew Lease

arXiv:2610.19206v1cs.CL

TL;DR

Multi-output text regression remains less explored than scalar prediction. DUALVIEW extracts broad and output-specific representations from a frozen LLM and predicts the full score vector, outperforming supervised baselines across tasks and backbones with substantially fewer fitted parameters.

  • Problem

    LLM-based text regression has largely focused on scalar targets, leaving prediction of multiple numerical targets for the same text less explored.

    Source: §1 · p. 1

  • Method

    DUALVIEW extracts broad and output-specific representations from a frozen LLM and uses a dual-branch model with shared predictors to predict all output dimensions.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    Across five tasks and eight LLM backbones, DUALVIEW beats seven supervised baselines; versus GAT-AES, aggregate Pearson correlation improves 18.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The broad and output-specific evidence views provide complementary predictive benefits, while shared predictors retain strong performance with fewer trainable parameters.

    Source: §8 · p. 8

    DUALVIEW requires access to model internals, and its frozen backbone may yield weaker predictions on unfamiliar domains when pretrained representations are inadequate.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While work on LLM-based regression has largely focused on scalar prediction, multi-output text regression remains less explored. We present DualView, a parameter-efficient approach to leverage LLMs for multi-output text regression. \method{} first extracts broad and output-specific representations from a frozen LLM, then uses a dual-branch architecture to produce the full score vector. Across five heterogeneous regression tasks and eight LLMs (up to 122B), DualView outperforms seven supervised regression baselines while requiring 82--1842 times fewer fitted parameters than the three strongest baselines. These improvements extend across individual output dimensions and persist even with limited training data. Notably, DualView using a smaller open-weight LLM can outperform unsupervised prediction with Gemini 3.1 Pro. Controlled ablations show that broad and output-specific evidence provide complementary predictive benefits, while lightweight shared predictors achieve strong performance with fewer trainable parameters.

1 Introduction

Multi-output text regression predicts several numerical targets for one text input, but remains less explored than scalar text regression. DUALVIEW addresses this gap with complementary representations and shared predictors, outperforming supervised baselines across diverse tasks and models.

1 Introduction

  • Fine-grained evaluation scores free-form outputs across dimensions such as fluency, factuality, coherence, and relevance, while specialized tasks assess code and research ideas along multiple criteria.Examples include code correctness and efficiency, and research-idea novelty, relevance, and feasibility.Source: §1 · p. 1
  • Existing methods can require costly LLM fine-tuning or separate predictors per output, while specialized architectures may not suit heterogeneous domains and targets.Essay-specific features may not transfer to code quality estimation, and code correctness and peak memory may be weakly correlated.Source: §1 · pp. 1–2
  • DUALVIEW shifts specialization from domain-specific prediction architectures to evidence extraction, using complementary broad and output-specific LLM representations with shared predictors.The LLM backbone is frozen, and lightweight modules are trained on its representations.Source: §1 · p. 2
  • 18.4% higher aggregate Pearson correlation and 2.8% lower NRMSE than GAT-AES, with DUALVIEW outperforming seven supervised baselines across five tasks and eight LLM backbones.The evaluated backbones range from 8B to 122B parameters.Source: §1 · p. 2
  • 5.1K–57.4K fitted parameters—82–1842× fewer than the strongest supervised baselines—while retaining an advantage with limited supervision and over zero- and few-shot Gemini 3.1 Pro.The limited-supervision result is reported for training-data efficiency experiments.Source: §1 · p. 2

2 Related Work

Prior work includes scalar text-regression methods and general multi-task learning, while fine-grained evaluation motivates assessing multiple dimensions of the same text. DUALVIEW uses complementary evidence extraction to address this structured prediction setting.

2.1 LLM-Based Text Regression

  • Prior LLM text-regression work includes autoregressive numerical decoding, regression-aware inference, and predictive heads; DUALVIEW instead targets multi-output prediction with evidence specialization and experiments up to 122B parameters.Earlier supervised evaluations cited in the paper used LLMs up to 32B parameters.Source: §2.1 · p. 2

2.2 Fine-Grained Text Evaluation

  • Fine-grained evaluation assesses one text input along multiple numerical axes, supporting identification of weaknesses and trade-offs; DUALVIEW reportedly outperforms supervised baselines with as few as 64 training examples.The paper also describes human annotations and execution-based metrics across dimensions as potentially expensive to obtain.Source: §2.2 · p. 2

2.3 Multi-Task Learning

  • Unlike general multi-task learning across heterogeneous tasks, multi-output text regression predicts the same continuous targets from the same input; DUALVIEW uses complementary evidence in broad and focused branches.The broad branch predicts all outputs jointly, while the focused branch predicts each dimension from target-specific representations.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

3 DUALVIEW

DUALVIEW formulates multi-output text regression as predicting described continuous targets from shared text inputs. It extracts broad and target-specific evidence from a frozen LLM, then combines shared predictors over both views.

3.1 Problem Formulation

  • Given target descriptions, multi-output text regression learns to predict all D continuous targets from each text input.Source: §3.1 · p. 3

3.2 Evidence Extraction

  • DUALVIEW extracts broad evidence for all targets and target-specific evidence using a shared task prompt plus one target-focused prompt per output.It mean-pools hidden states for the broad view and takes the last-token hidden state for each focused prompt; shared-prefix states are reused across prompts.Source: §3.2 · p. 3

3.3 Broad Vector Predictor

  • The broad predictor uses reduced-rank factorization to share k linear projections across targets while allowing each target different projection weights.Source: §3.3 · p. 4

3.4 Shared Focused Predictor

  • A shared focused coefficient vector scores target-specific evidence, allowing distinct target predictions while pooling supervision across outputs.The focused and broad branches serve complementary purposes: common predictive directions versus specialized evidence.Source: §3.4 · p. 4

3.5 Training

  • Training combines both branches’ predictions and minimizes regularized squared error subject to a rank constraint on the broad predictor.Alternating minimization fits each branch to the other branch’s residuals until convergence, with rank and regularization selected on validation data.Source: §3.5 · p. 4

4 Experimental Setup

The experiments evaluate DUALVIEW with frozen LLM backbones on five fine-grained quality-estimation tasks. They compare it with supervised regression methods and prompting baselines using correlation and normalized error metrics.

Models

  • Eight LLM backbones are evaluated with their parameters frozen, while only lightweight regression predictors are fitted.Source: §4 · p. 4

Datasets

  • Evaluation covers five fine-grained quality-estimation tasks in summarization, translation, research ideation, code generation, and dialogue.Source: §4 · p. 4

Baselines

  • DUALVIEW is compared with seven supervised baselines and zero- and four-shot LLM-as-a-judge prompting baselines.Source: §4 · p. 4

Metrics

  • The study reports Pearson correlation and NRMSE, with NRMSE normalized by training-set standard deviation for comparisons across different scales.Spearman correlation is additionally reported in the appendix.Source: §4 · p. 4

5 Key Results

DUALVIEW delivers strong multi-output prediction across datasets and targets, with a small fitted footprint and gains that persist under limited supervision. It also outperforms Gemini 3.1 Pro prompting in the reported comparison.

5.1 Predictive Performance

  • 18.4% higher Pearson correlation and 2.8% better NRMSE than GAT-AES, the strongest supervised baseline, across five datasets and eight LLM backbones.DUALVIEW has the strongest aggregate performance across the evaluated datasets and backbones.Source: §5.1 · p. 4, Table 1 · p. 5
  • DUALVIEW is strongest on 20 of 21 output targets, with MSumBench faithfulness effectively tied at 0.675.The comparison macro-averages Pearson correlation over eight LLMs; Figure 3 compares DUALVIEW with the three strongest supervised baselines.Source: §5.1 · p. 5, Figure 3 · p. 5
  • DUALVIEW has higher strict win rates than GAT-AES—65.9% versus 8.9% for Pearson and 71.4% versus 7.7% for NRMSE—and smaller mean worst-output gaps.Mean worst-output gaps are 0.039 versus 0.149 for Pearson and 0.016 versus 0.055 for NRMSE; DUALVIEW also outperforms supervised baselines under aggregation schemes that weight poorly predicted targets more heavily.Source: §5.1 · p. 5

5.2 Trainable Footprint

  • 5.1K–57.4K fitted parameters for DUALVIEW versus 1.84M–20.98M for the three strongest supervised baselines, or 82–1842× more for the baselines.DUALVIEW’s trainable footprint scales with representation width rather than backbone parameter count.Source: §5.2 · pp. 5–6, Table 2 · p. 5

5.3 Training-Data Efficiency

  • DUALVIEW consistently outperforms the three strongest supervised baselines at every evaluated training budget across five datasets.Its lead narrows on MSumBench and remains stable or widens on the other four datasets; Figure 4 uses nested training subsets from 64 examples to each full training split.Source: §5.3 · p. 6, Figure 4 · p. 6

5.4 Comparison with Gemini 3.1 Pro

  • 30.7% higher Pearson correlation and 19.8% better NRMSE than Gemini 3.1 Pro under its stronger few-shot condition.The comparison uses DUALVIEW with Gemma 4 31B against Gemini 3.1 Pro under zero-shot and few-shot prompting.Source: §5.4 · p. 6, Table 3 · p. 6

6 Ablations

The ablations test the evidence views, fitting strategy, target alignment, and parameter sharing. Their results favor combining both views, alternating joint fitting, matched target evidence, and shared predictors.

6.1 Evidence Complementarity

  • Combining broad and focused evidence improves Pearson correlation by 11.3% and reduces NRMSE by 2.1% over the strongest single-view configuration, full-rank broad ridge.The combination outperforms either evidence view alone even when the single-view predictors have more capacity.Source: §6.1 · p. 7, Table 4 · p. 7

6.2 Training Strategy Ablations

  • Alternating joint fitting improves Pearson correlation by 3.1% over separate fitting and 4.3% over fixed-basis joint fitting, while reducing NRMSE by 0.8% and 0.9%, respectively.Fixed-basis joint fitting does not improve over separate fitting; the results suggest adapting broad predictive directions to focused evidence helps the branches contribute more complementarily.Source: §6.2 · p. 7, Table 5 · p. 7

6.3 Target-Specific Evidence Alignment

  • Mismatched focused representations lower performance in both configurations, with larger losses when combined with broad evidence.Pearson drops by 10.0% versus 5.3%, and NRMSE increases by 1.8% versus 1.2%, respectively.Source: §6.3 · p. 7, Table 6 · p. 7

6.4 Sharing Across Outputs

  • DUALVIEW achieves comparable predictive performance to less constrained alternatives while fitting the fewest parameters.Its reduced-rank broad and shared focused predictors require (k+1)(H + D) parameters, compared with 2HD + D for full-rank broad and independent focused predictors.Source: §6.4 · pp. 7–8, Table 7 · p. 8

7 Discussion and Future Work

The discussion weighs supervised fitting against prompting costs, outlines extensions to distribution prediction and multi-objective optimization, and notes a boundary on transfer to new targets.

When does supervised adaptation pay off?

  • Whether supervised adaptation pays off depends on the deployment context, including upfront fitting, marginal inference cost, request volume, and number of outputs.Processing N inputs requires N broad and ND focused representations, while shared-prefix reuse avoids redundant backbone computation.Source: When does supervised adaptation pay off? · p. 8
  • DUALVIEW could extend to text-to-distribution prediction by treating quantile levels as separate targets, but the authors leave this extension to future work.The proposed setup uses pinball loss with a monotonicity constraint to prevent quantile crossing.Source: When does supervised adaptation pay off? · p. 8
  • Multi-objective optimization could complement DUALVIEW by reweighting or modifying output-wise gradients during shared-parameter training, but integration is left for future work.The proposed methods would affect how supervision is balanced across outputs, not feature extraction from the LLM backbone.Source: When does supervised adaptation pay off? · p. 8
  • The shared focused predictor may generalize to unseen target descriptions without retraining, but the broad predictor is learned for a fixed output set and requires adaptation for new targets.Source: When does supervised adaptation pay off? · p. 8

8 Conclusion

DUALVIEW combines broad and output-specific representations from a frozen LLM in a lightweight multi-output regression approach. Across five tasks and eight backbones, it outperforms supervised baselines with fewer fitted parameters, and its advantages persist across outputs and with limited supervision.

8 Conclusion

  • Across five tasks and eight LLM backbones, DUALVIEW outperforms seven supervised baselines while fitting substantially fewer parameters.The advantages extend across nearly all output dimensions and persist with limited supervision.Source: §8 · p. 8
  • DUALVIEW uses a rank-constrained multivariate predictor for shared output structure and a shared scalar predictor over target-specific evidence.Both branches use representations extracted from a frozen LLM.Source: §8 · p. 8
  • Ablations show that broad and output-specific evidence are complementary, while parameter sharing preserves performance with fewer trainable parameters.Source: §8 · p. 8

Limitations

The limitations concern prompt sensitivity, dependence on pretrained LLM representations, nonconvex optimization, and inference costs that grow with the number of targets. The appendix also records related work and supplementary material organization.

Limitations

  • Alternative prompt formulations were not evaluated, and prompt-format changes may affect extracted representations and predictions.The experiments use fixed prompt templates for each dataset.Source: Limitations · p. 9
  • DUALVIEW requires model internals and may not work with black-box models; its frozen-backbone performance may degrade on unfamiliar domains.Performance therefore depends on the quality of pretrained representations.Source: Limitations · p. 9
  • Alternating updates optimize each predictor with the other fixed, but the nonconvex joint objective has no guarantee of a globally optimal solution.Source: Limitations · p. 9
  • Inference computation grows approximately linearly with the number of targets because each input needs one broad and D focused representations.Fewer fitted parameters do not necessarily mean lower inference latency or memory use.Source: Limitations · p. 9

A Appendix Organization

  • The appendix covers related work, method and experimental details, additional results and ablations, discussion, prompt templates, and output-level tables.Source: A Appendix Organization · p. 13

B.1 LLM-based Text Regression

  • DUALVIEW shifts specialization from domain-specific prediction architectures to evidence extraction and simple predictors over broad and target-specific representations.The cited discussion contrasts this approach with prior methods that fine-tune LLMs or use specialized architectures.Source: §B.1 · p. 13

B.2 Fine-Grained Text Evaluation

DUALVIEW uses simple predictors over broad and target-specific LLM representations to handle fine-grained evaluation and share information across outputs. Its extraction and fitting procedures combine a reduced-rank broad branch with a shared focused branch.

B.2 Fine-Grained Text Evaluation

  • DUALVIEW uses ridge predictors over complementary broad and target-specific representations, shifting specialization from prediction architecture to evidence extraction.The discussion contrasts its simple predictors with specialized approaches for essay scoring.Source: §B.2 · p. 13

B.3 Sharing Across Tasks and Outputs

  • Multi-output regression shares a fixed set of continuous targets, but sharing supervision while retaining target-specific information remains an open design question.DUALVIEW addresses this with a reduced-rank broad predictor and a shared predictor over focused representations.Source: §B.3 · p. 14

C.1 Feature Extraction

  • For each input, DUALVIEW extracts a broad representation and target-conditioned focused representations from penultimate-layer hidden states without generating numerical answers.The broad view mean-pools prompt tokens; the focused view uses the last-token state after a target instruction, with shared-prefix activations reusable across prompts.Source: §C.1 · p. 14

C.2 Evidence and Label Standardization

  • Representations and labels are standardized using training-split statistics, which remain fixed at evaluation time.Each target's labels use their training mean and population standard deviation; positive constants prevent division by zero in representation scaling.Source: §C.2 · p. 14

C.3 Rank-Constrained Broad Predictor

  • The broad branch learns k common predictive directions and target-specific weights through a rank-constrained predictor.Its factorization stores k(H + D) coefficients rather than HD for a full-rank broad predictor.Source: §C.3 · pp. 14–15

C.4 Shared Focused Predictor

  • The focused branch applies one shared linear coefficient vector across targets, using target-conditioned representations to produce different scores.It has H fitted coefficients rather than HD coefficients for separate target-specific linear predictors.Source: §C.4 · p. 15

C.5 Training and Inference

  • The two branches add their standardized predictions, and alternating ridge updates fit each branch to residuals with the other held fixed.The procedure repeats until convergence and selects rank and regularization penalties on validation data.Source: §C.5 · p. 15

D.1 Models

  • The evaluation uses eight frozen LLM backbones, ranging from Gemma 4 E4B to Qwen3.5 122B-A10B.Source: §D.1 · p. 15

D.2 Datasets

The experimental setup evaluates multiple-output quality-estimation tasks with source-group partitions, adapted supervised baselines, and equally weighted aggregation across datasets and LLMs. Supplementary material expands the benchmark results and reports the compute budget and analysis topics.

D.2 Datasets

  • The five datasets cover summarization, translation, research ideation, code generation, and dialogue, with source-group splits keeping related examples together.Partitions are made by source document, segment, conversation, programming problem, or paper; three seeds use matched partitions across methods.Source: §D.2 · p. 16

D.3 Baseline Adaptations

  • The supervised baselines are adapted to frozen LLM backbones and multi-output targets, including methods originally designed for scalar prediction or essay scoring.For example, RELISH receives a D-dimensional readout, while GenRe2 uses one shared decoder conditioned on output identity.Source: §D.3 · pp. 17–18

D.4 Metrics and Aggregation

  • The study reports NRMSE, Pearson r, and Spearman ρ; aggregate results weight datasets and LLMs equally.NRMSE is normalized by training standard deviation, and output dimensions receive equal weight within each dataset.Source: §D.4 · p. 18

D.5 Hyperparameters and Model Selection

  • DUALVIEW selects its broad rank and ridge penalties on validation data, while baseline methods use their respective objectives and model-selection criteria.Source: §D.5 · p. 18

D.6 Computational Budget

  • The supervised experiments were completed within one thousand GPU hours on NVIDIA Grace Hopper GH200 nodes.Source: §D.6 · p. 19

E Additional Key Results

  • The supplementary analyses cover predictive performance, output-level results, parameter footprint, training-data efficiency, and Gemini 3.1 Pro comparisons.Source: E Additional Key Results · p. 19

E.1 Predictive Performance

  • Tables 11–13 provide Pearson, Spearman, and NRMSE results for all seven supervised baselines across dataset–LLM combinations, with equal weighting of datasets and LLMs.Source: §E.1 · p. 19

E.1.1 Fine-grained Performance Analyses

DUALVIEW wins most output-level comparisons and remains strongest when aggregation penalizes weaker outputs more heavily. Its shared predictors use a compact fitted footprint, while the supplied comparisons also examine training budgets, Gemini 3.1 Pro, and linear readouts.

E.1.1 Fine-grained Performance Analyses

  • DUALVIEW leads on 19 of 21 outputs for Spearman and 20 of 21 for NRMSE; its two Spearman deficits are DSCodeBench correctness and runtime.GenRe2 leads those two outputs by 0.008 and 0.018; MSumBench faithfulness is the sole NRMSE exception, effectively tied with GAT-AES (< 0.01 difference).Source: §E.1.1 · p. 20, Figure 5 · p. 20, Figure 6 · p. 20
  • DUALVIEW has higher strict win rates and smaller mean worst-output gaps than the three strongest supervised baselines.For Pearson and NRMSE, its win rates are 65.9% and 71.4%, versus 8.9% and 7.7% for GAT-AES; its corresponding gaps are 0.039 and 0.016, versus 0.149 and 0.055.Source: §E.1.1 · p. 20, Dataset LLM SAJA T-MES RELISH MT-ICL… · p. 24, Table 14 · p. 24
  • DUALVIEW remains strongest under harmonic means for correlation and RMS for NRMSE, and under the stronger q = −2 penalty on weak correlations.Harmonic means penalize small correlations more strongly than arithmetic means, while RMS assigns greater weight to large errors.Source: Dataset LLM SAJA T-MES RELISH MT-ICL… · p. 24, Table 15 · p. 26, Table 16 · p. 27

E.2 Trainable Footprint

  • The shared-predictor parameterization depends on hidden size H, target count D, and selected rank k, but not directly on LLM-backbone size.Its fitted parameters comprise the reduced-rank broad predictor, shared focused predictor, and target-specific intercepts.Source: §E.2 · p. 24

E.3 Training-Data Efficiency

  • Training-data efficiency figures plot Pearson, Spearman, and NRMSE against training-set size, with validation and test splits held fixed.Each point macro-averages eight LLMs within a seed, and whiskers show standard deviation across three seeds.Source: §E.3 · p. 24, Figure 7 · p. 25, Figure 8 · p. 25

E.4 Comparison with Gemini 3.1 Pro

  • The Gemini comparison uses zero-shot and four-shot Gemini 3.1 Pro against DUALVIEW with Gemma 4 31B on each dataset.Gemma 4 31B was selected for DUALVIEW using validation results to keep the comparison apples-to-apples.Source: §E.4 · p. 24, Table 18 · p. 28

F.1 Strong Linear Baselines

  • Broad-only ridge already outperforms the adapted supervised baselines, while combining broad and target-specific representations improves Pearson by 11.3% and NRMSE by 2.1%.Less constrained two-view predictors perform similarly, making parameter sharing primarily an efficiency mechanism; these comparisons do not measure total computational savings.Source: §F.1 · p. 26

F.2 Statistical Reliability of Ablation Improvements

Paired, source-group bootstrap intervals support improvements over both broad-only ridge and separate fitting with α fusion. The section also describes supplementary output-level metric tables across datasets and LLM backbones.

F.2 Statistical Reliability of Ablation Improvements

  • The paired source-group bootstrap resamples groups within each dataset using identical resamples for compared methods across 5,000 replicates.The analysis recomputes Pearson correlation and NRMSE and aggregates across outputs, LLM backbones, datasets, and seeds.Source: §F.2 · p. 28
  • DUALVIEW improves aggregate Pearson correlation from 0.3876 to 0.4322 (+0.0447) and reduces NRMSE from 0.8919 to 0.8732 (+0.0186) over full-rank broad-only ridge.The corresponding 95% confidence intervals are [0.0328, 0.0497] and [0.0134, 0.0229].Source: §F.2 · p. 28, Table 20 · p. 28
  • Compared with separate fitting followed by α fusion, DUALVIEW improves Pearson correlation from 0.419 to 0.432 (+0.013) and reduces NRMSE from 0.880 to 0.873 (a reduction of 0.007).The corresponding 95% confidence intervals are [0.0053, 0.0177] and [0.0039, 0.0109].Source: §F.2 · p. 28, Table 20 · p. 28

I Output-Level Performance Tables

Related papers

Loading 2610.19206v1…