PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training

Jian Gao, Xiao Zhang, Xun Zhu, Miao Li, Ji Wu

arXiv:2609.02244v1cs.AI

TL;DR

Low-resource ambiguity can leave LLM training unable to distinguish intended rules from plausible spurious alternatives. The paper introduces PGT and CPS, which add positive and negative prior-conditioned learning signals to supervised training; CPS improves results across AmbiMath, Jigsaw, and MNLI/HANS.

  • Problem

    Low-resource or ambiguous data may not distinguish intended task rules from spurious alternatives, while prompting treats natural-language priors as context rather than learning objectives.

  • Method

    PGT incorporates task-level natural-language priors as auxiliary learning signals, and CPS combines the original supervised loss with positive and negative prior-conditioned objectives.

  • Results

    CPS improves over plain and prompt fine-tuning across AmbiMath, Jigsaw, and MNLI/HANS, with early-training alignment margin increasing from −0.094 to 0.966 from 0.2 to 1.0 epoch.

  • Takeaways & Limitations

    Task-level natural-language priors can provide useful guidance as auxiliary learning signals for low-resource LLM training.

Abstract

from arXiv · show

Large language models (LLMs) often struggle when low-resource training data are ambiguous or incomplete. Task-level natural-language priors can provide useful guidance in such settings, but existing approaches usually treat these priors as input context rather than as learning signals during training. We propose Prior-Guided Tuning (PGT), a training perspective that incorporates natural-language priors as auxiliary learning signals for low-resource LLM training. Under this perspective, we introduce Contrastive Prior Steering (CPS), which keeps the original supervised objective intact while adding positive and negative prior-conditioned auxiliary losses to encourage task-consistent learning and discourage plausible but misleading alternatives. Experiments on AmbiMath, Jigsaw, and MNLI/HANS show that CPS consistently improves over plain and prompt fine-tuning. On AmbiMath, CPS achieves 97.6% average exact-match accuracy. On Jigsaw, CPS improves average Macro F1 by 9.5 percentage points over standard fine-tuning, and with 1/10 of the experimental training data slightly exceeds full-data plain fine-tuning. On HANS, CPS improves non-entailment accuracy by 8.3 and 5.2 percentage points for LLaMA 3.1 8B and Qwen 2.5 7B, respectively, while maintaining comparable in-domain MNLI accuracy. These results support our central claim: task-level natural-language priors can provide useful guidance as auxiliary learning signals for low-resource LLM training. Our code and data will be publicly available.

1 Introduction

Low-resource and ambiguous training data can leave LLMs unable to distinguish intended task rules from spurious alternatives. The paper proposes treating task-level natural-language priors as auxiliary learning signals through PGT and CPS.

  • Low-resource, incomplete data can encourage LLMs to rely on spurious correlations or superficial cues when the intended task rule is underdetermined.
  • Prompt-based fine-tuning appends priors as context, but does not explicitly make them part of the training objective or control ambiguous supervised updates.
  • Prior-Guided Tuning (PGT) treats task-level natural-language priors as explicit auxiliary learning signals rather than contextual inputs.
  • Contrastive Prior Steering (CPS) combines positive and negative prior-conditioned objectives with the standard supervised loss to encourage task-consistent learning and discourage misleading alternatives.
  • 97.6% average exact-match accuracy is achieved by CPS on AmbiMath, while Jigsaw Macro F1 and HANS non-entailment robustness also improve over standard approaches.

2 Related Work

The paper distinguishes CPS from prompting, instruction tuning, knowledge-aware learning, and preference optimization by using task-level priors as supervised-training signals.

  • Instruction tuning and prompting provide task information through inputs, learned prompts, or instruction-formatted training examples.
  • Preference optimization learns response preferences from instance-level preferred and rejected outputs, whereas CPS uses task-level positive and negative priors shared across examples.
  • CPS guides low-resource supervised training toward an intended task principle rather than modeling human preferences.
  • Knowledge-aware methods inject structured or external knowledge into representations, inputs, prompts, or fine-tuning examples, forming a related but distinct line of work.

3 Motivation: The Ambiguity of Data

Low-resource datasets may leave intended and spurious task rules empirically indistinguishable. AmbiMath isolates this ambiguity by constructing tied hypotheses and testing whether natural-language priors resolve it without replacing data-driven operator learning.

  • Maximum-likelihood training depends on data identifying the underlying task function, but low-resource datasets can suffer from specification ambiguity.
  • When a spurious feature correlates perfectly with labels, intended and misleading models can achieve the same empirical loss, leaving statistical patterns unable to distinguish them.
  • AmbiMath uses synthetic function-calculation tasks in which only one of two parameters determines the answer, separating feature selection from operator learning.
  • Its training construction makes a target rule and a spurious rule exactly consistent with every observed label, creating an empirical tie resolved by a natural-language prior.
  • Inference removes priors and uses disentangled samples where competing rules disagree, while positional and semantic prior forms test robustness to task-level wording.

4 Method

PGT augments supervised learning with task-level natural-language guidance, and CPS operationalizes this through positive and negative prior-conditioned views while preserving the original data signal.

  • Prior-Guided Tuning: PGT uses a task-level natural-language statement shared across examples to define an auxiliary training objective.
  • Prior-Guided Tuning: The general PGT objective combines standard supervised loss with a prior-induced auxiliary objective weighted by λ.Ldata is the supervised loss, Rprior is the prior-induced objective, and λ controls its strength.
  • Prior-Guided Tuning: PGT complements data-driven learning: priors guide the solution principle while examples still determine remaining task structure such as the arithmetic operation.
  • Contrastive Prior Steering: CPS constructs original, positive-prior, and negative-prior views for each labeled example and computes target-token cross-entropy losses.
  • Contrastive Prior Steering: The positive loss encourages target prediction under task-consistent guidance, while the negative loss discourages fitting the target under a misleading prior.λ balances the prior objective with the supervised objective, and γ controls the negative-prior term.

5 Results

Experiments across AmbiMath, Jigsaw, and MNLI/HANS test whether task-level priors can act as auxiliary learning signals in diverse low-resource settings. CPS is compared with plain and prompt fine-tuning and improves ambiguity resolution, toxicity classification, and HANS non-entailment robustness.

  • AmbiMath: AmbiMath tests whether priors resolve controlled ambiguity when low-resource training data fit multiple solution rules.CPS reaches 96.3–100.0 EM across positional and semantic priors, whereas plain and prompt fine-tuning are lower and less stable.
  • Jigsaw: Jigsaw evaluates low-resource toxicity classification on identity-related comments where models may learn unreliable associations between identity terms and toxicity labels.The evaluation reports accuracy, toxic-label F1+, non-toxic-label F1-, and Macro F1 by gender group.
  • Jigsaw: CPS consistently improves F1+ and Macro F1 across Jigsaw identity groups for both LLaMA and Qwen relative to plain and prompt fine-tuning.Prompt fine-tuning provides limited and inconsistent gains, while a DPO-style baseline improves some groups less consistently than CPS.
  • MNLI/HANS: MNLI/HANS evaluates in-domain MNLI accuracy alongside diagnostic robustness to shallow lexical and syntactic heuristics.HANS groups contradiction and neutral cases as non-entailment and is used only for diagnostic evaluation, not model selection.
  • MNLI/HANS: CPS improves HANS non-entailment accuracy from 39.2% to 47.5% on LLaMA and from 45.1% to 50.3% on Qwen while maintaining comparable MNLI accuracy.The gains are concentrated on HANS non-entailment examples, the key robustness metric for shallow heuristic reliance.

6 Discussion

The discussion examines whether CPS improves data efficiency, affects optimization early, and depends on both contrastive components or a particular prior wording. Results show consistent gains across data scales, rapid early alignment with the intended rule, and stronger reliability for full CPS.

  • Data efficiency: CPS consistently improves both accuracy and Macro F1 over plain fine-tuning across Jigsaw data scales.The scaling analysis uses 1/30, 1/10, 1/3, and all of the experimental training set.
  • Data efficiency: 1/10 of the experimental training data with CPS slightly exceeds full-data plain fine-tuning on Jigsaw.
  • Early-training optimization diagnostic: From 0.2 to 1.0 epoch, CPS raises alignment margin from −0.094 to 0.966 and exact-match accuracy from 0.0% to 94.5% on AmbiMath.The alignment margin measures closeness to the intended rule relative to the competing rule.
  • Early-training optimization diagnostic: At 1.0 epoch, plain fine-tuning reaches 1.5% exact match with margin −0.278, while prompt fine-tuning reaches 2.5% with margin 0.192.
  • Prior formulation and component ablations: Positive-Only and Negative-Only CPS improve average scores over plain and prompt fine-tuning, but full CPS is substantially more reliable across all four prior types.The ablation removes either the negative-prior or positive-prior term, respectively.
  • Prior formulation and component ablations: Additional experiments indicate that gains are not explained by one handcrafted wording, test-time prior reintroduction, or generated rationale supervision.

7 Conclusion

The paper concludes that task-level natural-language priors can guide low-resource LLM training as auxiliary learning signals. Across AmbiMath, Jigsaw, and MNLI/HANS, CPS improves over plain and prompt fine-tuning in settings involving ambiguity, toxicity classification, and shallow NLI heuristics.

  • PGT treats task-level natural-language priors as explicit learning signals rather than contextual inputs for low-resource LLM training.CPS instantiates this perspective.
  • CPS combines the original supervised objective with positive and negative prior-conditioned losses.The method encourages task-consistent learning and discourages misleading alternatives while preserving data-driven learning.
  • CPS improves over plain and prompt fine-tuning across AmbiMath, Jigsaw, and MNLI/HANS.These settings involve synthetic rule ambiguity, identity-related toxicity classification, and shallow NLI heuristics.
  • Limited supervision becomes substantially more useful with CPS, while diagnostics suggest it affects optimization when models first commit to a solution rule.Ablations also indicate contributions from both positive and negative priors.
  • Task-level natural-language priors provide useful guidance as auxiliary learning signals for low-resource LLM training.The paper presents CPS as one lightweight way to operationalize this idea without instance-level rationales or preference pairs.

A.1 Training Procedure and Implementation Details

CPS trains on original, positive-prior, and negative-prior views while retaining the standard supervised loss. Its objective combines these losses with weighting and negative-prior-strength hyperparameters, followed by gradient clipping and optimizer updates.

  • Each training example produces original, positive-prior, and negative-prior views with corresponding target-token losses.The original view uses the unmodified input, while the other views prepend positive or negative priors.
  • The CPS objective is L_CPS = L_0 + λ(L_+ − γL_−), combining the original loss with positive and negative prior-conditioned losses.λ balances the prior-induced objective with the supervised objective, while γ controls the negative-prior term.
  • Global gradient-norm clipping is applied after differentiating the combined objective, using τ = 1.0 in the experiments.The clipped gradient is then used for the optimizer update.
  • Experiments train LoRA adapters with AdamW and search learning rates in {1×10^-4, 3×10^-4, 5×10^-4}.Main CPS experiments use γ = 0.1 and search λ in {0.3, 0.5, 1.0}.

A.2 Dataset and Prior Details

The benchmarks construct ambiguity or heuristic-confounded tasks and pair each with positive guidance describing the intended decision principle and negative guidance describing a misleading alternative. MNLI/HANS uses low-resource MNLI training and evaluates both in-domain accuracy and heuristic robustness.

  • AmbiMath: AmbiMath training examples make x1 + 2 and x2 −2 equally consistent with observed labels, while disentangled tests separate the alternatives.Exact match is computed on the final numeric answer.
  • AmbiMath: AmbiMath positive and negative priors respectively select the first or second input parameter as the basis for the output.Other variants replace “first” with “second”, “Chinese”, or “English” as appropriate.
  • Jigsaw: Jigsaw uses gender-associated toxicity examples and evaluates gender-related subsets with guidance focused on contextual toxicity evidence versus identity-term heuristics.The negative prior predicts toxic when an identity group is mentioned and non-toxic otherwise.
  • MNLI/HANS: MNLI/HANS trains on 1,000 MNLI examples and evaluates MNLI accuracy alongside HANS robustness, grouping neutral and contradiction as non-entailment.The positive prior rejects entailment based solely on lexical overlap, subsequence matching, or constituent matching.
  • Jigsaw: The DPO-style Jigsaw baseline forms preferred gold-label and rejected opposite-label responses, but CPS instead uses task-level priors shared across examples.CPS is intended to guide supervised training rather than model human preference.

A.3 Additional AmbiMath Prior Variants

An additional AmbiMath variant reverses the division of labor between data and priors: the prior supplies the arithmetic operation, while the data identify the consistent input parameter. CPS remains substantially stronger than plain and prompt fine-tuning in this setting.

  • The additional AmbiMath setting gives the arithmetic operation in the prior and requires the model to infer the consistent parameter from the training distribution.The training examples remain ambiguous and are resolved on disentangled test examples.
  • For func(13, 17)=15, a +2 prior supports the first parameter, while a −2 prior supports the second parameter.The prior supplies the operation and the data distribution supplies the parameter choice.
  • Disentangled test examples measure whether the model combines the prior-specified operation with the learned parameter choice.Exact-match accuracy reflects whether the two information sources are combined correctly.
  • CPS remains substantially stronger than plain and prompt fine-tuning when the prior supplies a different task component.This setting tests whether CPS is limited to priors about parameter selection.

A.4 CoT-SFT Diagnostics

The diagnostics compare CPS with CoT-SFT, bounded surrogate objectives, and alternative prior formulations. They show CPS is more consistent than the bounded surrogate, remains robust across negative-prior wordings, and can reach a higher Jigsaw Macro F1 with fewer examples and optimization steps despite greater per-step cost.

  • CoT-SFT Diagnostics: 79.9% of Jigsaw training examples produced a correct rationale/label pair for CoT-SFT without gold labels, even with up to three attempts.The filtered correct CoT data were then used for training; a gold-label-revealed setting was also evaluated.
  • CoT-SFT Diagnostics: CoT-SFT did not recover the intended first-input rule on the harder AmbiMath construction, despite gold-answer access during rationale generation.The construction satisfies 2a + 5 = 3b −2 = c during training but is disentangled at test time.
  • Bounded Surrogate Diagnostic: The bounded surrogate improves some AmbiMath settings but is much less consistent than CPS, especially for first-parameter and Chinese-prior conditions.The paper attributes the weaker corrective signal to sigmoid saturation after positive and negative views begin separating.
  • Prior Formulation Diagnostics: Both alternative Jigsaw negative-prior formulations retain strong performance, suggesting CPS is not tied to one handcrafted wording.The alternatives replace the direct identity shortcut with subjective associations or conservative toxicity detection.
  • Sensitivity and Cost Diagnostics: λ = 0.3 and λ = 0.5 are nearly identical, while λ = 1.0 remains close; very small prior weights are less effective.This indicates a useful operating range rather than a single narrow setting, so CPS is not claimed to be hyperparameter-free.
  • Sensitivity and Cost Diagnostics: CPS reaches higher Jigsaw Macro F1 than the larger plain-fine-tuning run with fewer optimization steps, although it is slower per step because it evaluates three views.The comparison is framed as cost-to-target in the low-resource regime rather than per-step cost.
Loading 2609.02244v1…