PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

Tong Ye, Hang Yu, Tengfei Ma, Xuhong Zhang, Jianguo Li, Peng Di, Peiyu Liu, Jianwei Yin, Wenhai Wang

arXiv:2605.30039v2cs.AI

TL;DR

Domain-specific data synthesis is difficult when a target domain is implicit in reference examples rather than expressible through natural-language descriptions. DOMINO learns a minimal sufficient representation with prompt tuning and contrastive disentanglement, then uses it to guide generation; theoretically it expands distributional support, and empirically it improves coding Pass@1 by up to 4.63% over strong instruction-tuned models. The approach supports scalable synthesis without manual prompt design or explicit domain specifications.

  • Problem

    Existing synthesis methods rely on explicit natural-language domain definitions, leaving synthesis from implicitly defined reference examples underexplored.

  • Method

    DOMINO combines prompt tuning with contrastive disentanglement to learn a minimal sufficient domain representation from reference samples and guide synthetic-data generation.

  • Results

    Pass@1 improves by up to 4.63% over strong instruction-tuned models on coding tasks with implicit domains, while DOMINO consistently outperforms baselines across coding and instruction-following tasks.

  • Takeaways & Limitations

    DOMINO enables scalable generation of diverse, domain-aligned samples without explicit domain definitions, prior domain knowledge, or manual prompt engineering.

  • Takeaways & Limitations

    With reference sets ranging from tens to hundreds of examples, learned representations can overfit idiosyncratic training details and produce overly similar synthetic data.

Abstract

from arXiv · show

Large Language Models have demonstrated remarkable progress in general-purpose capabilities and can achieve strong performance in specific domains through fine-tuning on domain-specific data. However, acquiring high-quality data for target domains remains a significant challenge. Existing data synthesis approaches follow a deductive paradigm, heavily relying on explicit domain descriptions expressed in natural language and careful prompt engineering, limiting their applicability in real-world scenarios where domains are difficult to describe or formally articulate. In this work, we tackle the underexplored problem of domain-specific data synthesis through an inductive paradigm, where the target domain is defined only through a set of reference examples, particularly when domain characteristics are difficult to articulate in natural language. We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data. DOMINO integrates prompt tuning with a contrastive disentanglement objective to separate domain-level patterns from sample-specific noise, mitigating overfitting while preserving core domain characteristics. Theoretically, we prove that DOMINO expands the support of the synthetic data distribution, ensuring greater diversity. Empirically, on challenging coding benchmarks where domain definitions are implicit, fine-tuning on data synthesized by DOMINO improves Pass@1 accuracy by up to 4.63\% over strong, instruction-tuned backbones, demonstrating its effectiveness and robustness. This work establishes a new paradigm for domain-specific data synthesis, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications.

1. Introduction

DOMINO addresses domain-specific data synthesis when target domains are defined by reference examples rather than explicit natural-language descriptions. It learns a minimal sufficient representation from those examples and uses it to guide diverse, domain-aligned generation.

  • Motivation: Existing synthesis methods require explicit, human-articulable domain definitions translated into textual instructions.This deductive paradigm is limited when domain conventions and patterns resist verbalization.
  • Motivation: Reference examples can reveal domain patterns, but limited-data supervised fine-tuning risks memorizing superficial sample features.The challenge is learning quality rather than merely generating more data.
  • DOMINO: DOMINO learns a minimal sufficient domain representation from reference samples and uses it to guide an LLM in generating novel samples.The representation captures essential domain characteristics while discarding sample-specific noise.
  • DOMINO: DOMINO combines prompt tuning with contrastive disentanglement to separate shared domain patterns from unique sample-level features.The combined objective enforces reconstruction fidelity and information minimality.
  • Contributions: DOMINO theoretically expands the support of the synthetic data distribution compared with baselines, ensuring greater diversity.The framework is intended to preserve shared structural patterns while varying specific content.
  • Contributions: Pass@1 improves by up to 4.63% over strong instruction-tuned models on coding tasks with implicit domains.DOMINO also consistently outperforms baselines across coding and instruction-following tasks.

2. Related Work

Prior synthetic-data methods mainly guide generation through prompts, extracted concepts, or system instructions. These approaches depend on explicit natural-language domain descriptions, whereas DOMINO uses implicit supervision from reference examples.

  • Existing Approaches: Existing LLM-driven synthetic-data approaches include Instruction Evolution, Key-Point-Driven methods, and System Prompt Guided methods.The categories differ in whether they expand instructions, extract domain concepts, or use predefined query templates and system prompts.
  • Instruction Evolution: Instruction Evolution methods use iterative prompt engineering to expand instruction sets from curated seeds or existing instructions.Self-Instruct and Evol-Instruct exemplify this category.
  • Key-Point-Driven: Key-Point-Driven methods extract concepts, taxonomies, or topic–key point pairs to guide synthesis.The cited methods differ in how they construct and sample domain knowledge, and some produce distributions that deviate from real data.
  • System Prompt Guided: System Prompt Guided methods synthesize domain data with predefined query templates and domain-specific system prompts.MAGPIE requires deliberate system-prompt design to steer generation toward the target domain.
  • Limitations and Contrast: All three categories break down when the target domain cannot be explicitly described in natural language and no prior domain knowledge is available.DOMINO instead synthesizes target-domain data from implicit supervision without manual prompt engineering.

3. Methodology

DOMINO learns a minimal sufficient domain representation from limited reference examples by combining likelihood-based prompt tuning with contrastive disentanglement. The learned domain representation is then used alone to synthesize diverse domain-specific data while separating shared domain patterns from sample-specific information.

  • Minimal Sufficient Domain Representation Learning: Small reference sets create overfitting risk because prompt tuning can tailor D to individual examples and reproduce their limited variation.The paper considers reference sets ranging from tens to hundreds of examples and motivates minimality as a response to this risk.
  • Implicit Domain Representation Learning via Prompt Tuning: The method learns domain soft tokens D from reference examples by maximizing their likelihood under a fixed pretrained LLM.The representation contains k soft tokens of dimension d and is optimized to explain the reference data.
  • Minimal Sufficient Domain Representation Learning: Minimal sufficiency retains domain-wide patterns while discarding sample-specific details that are irrelevant to the domain.The framework targets shared topic, task, or style information rather than specific facts, word choices, or constraints.
  • Contrastive Disentanglement: DOMINO jointly optimizes D* with sample-level tokens S(i), using contrastive learning to assign unique example information to S(i) rather than D*.The numerator reconstructs the matching example, while the denominator penalizes D* for helping reconstruct unrelated examples.
  • Overall Objective: The overall objective combines domain-level likelihood and contrastive loss, with λ controlling the trade-off between fidelity and representation disentanglement.The paper characterizes this objective as a tractable proxy for minimal sufficiency, jointly encouraging sufficiency and minimality.
  • Data Synthesis: After training, only D* guides synthesis from p(X|D*), and Proposition 4 states that its ε-support is strictly larger than vanilla prompt tuning’s support.The training and synthesis stages keep the LLM fixed; the support result formalizes the intended increase in synthetic-data diversity.

4. Experiments

DOMINO is evaluated on implicit-domain coding and NLP benchmarks through controlled comparisons, distribution analyses, and ablations of synthesis and representation-learning choices. Across these experiments, DOMINO shows stronger domain adaptation, broader synthetic distributions, and robustness to temperature, representation capacity, reference-data quantity, and disentanglement weighting.

  • Experimental Setup: LiveCodeBench provides implicit, evolving coding domains, evaluated with OPENCODER-7B and QWEN2.5-CODER backbones across Live Code Generation and Live Code Execution.The study compares direct prompting, reference-data fine-tuning, MAGPIE variants, DOMINO-Direct Domain, and DOMINO.
  • Main Results: DOMINO and DOMINO-Direct Domain achieve the best overall performance, while DOMINO-generated data fine-tunes Base models beyond the original Instruction versions.Reference SFT can overfit, particularly on Live Code Execution, whereas DOMINO benefits from synthetic domain-aligned data and contrastive disentanglement.
  • Synthetic Sample Distribution: DOMINO’s synthetic samples are more dispersed and more consistent with reference distributions than MAGPIE-Few Shot samples in both coding domains.The comparison uses CODET5-EMBEDDING representations visualized with t-SNE in Figures 3 and 4.
  • Interpretability of D*: The contrastive disentanglement objective pushes DOMINO beyond sample mimicry by separating shared domain principles from sample-specific details.With only a sufficiency objective, the model overfits superficial details; adding disentanglement produces more diverse and novel samples.
  • Generalization to More Task Domains: DOMINO improves the average NLP benchmark score by 3.48 points over the instruction-tuned backbone and by 1.61 points over DOMINO-Direct Domain.It consistently outperforms all baselines across four subtasks, extending the findings beyond coding.
  • Ablation Studies: DOMINO remains relatively stable across temperatures from 0.2 to 1.0, while larger domain-soft-token counts improve performance and reduced reference-data proportions degrade it.The reported trends connect generation control, representation capacity, and the amount of reference data used for domain representation learning.

5. Conclusion

DOMINO enables domain-specific data synthesis from reference data under implicit supervision, without manual prompt engineering or explicit domain definitions.

  • DOMINO learns minimal sufficient domain representations from reference data for domain-specific synthetic data generation.
  • The framework is designed to generate virtually unlimited domain-aligned samples without manual prompt engineering or prior domain knowledge.
  • Theoretical and empirical results show that DOMINO captures essential domain characteristics while producing diverse, domain-aligned samples.

A.1.1. The Proof of Proposition 1.

The proof shows that minimizing the contrastive loss L2 maximizes conditional mutual information between each sample-specific representation and its sample given the domain representation.

  • Minimizing L2 directly maximizes I(S(i); X(i)|D*), encouraging S(i) to retain sample-specific information.
  • The proof rewrites the relevant mutual-information expression using Bayes’ theorem and expands its denominator.
  • The resulting inequality follows because p(X(i)|D*) is at most 1 and p(X(i)|D*, S(i)) is non-negative.

A.1.2. The Proof of Proposition 2.

The proof establishes that L2 minimizes mutual information between the sample-specific representation S(i) and the domain representation D*, promoting their disentanglement under a bounded-information assumption.

  • L2 directly minimizes I(S(i); D*), promoting disentanglement between sample-specific and domain-level representations.
  • The proof assumes that the combined domain-level and sample-level prompts are shorter than the observed data, bounding I(S(i); X(i)).
  • Because L2 maximizes conditional mutual information while total mutual information is bounded, it necessitates minimizing I(S(i); D*).

A.1.3. The Proof of Proposition 3.

The proof interprets L1 + λL2 as a tractable proxy for a minimal sufficient domain representation: L1 promotes sufficiency, while L2 promotes minimality by separating domain and sample information.

  • The combined objective L = L1 + λL2 jointly encourages sufficiency and minimality in the learned domain representation D*.
  • The proof assumes that empirical loss computed on reference samples reasonably estimates expected loss over the true data distribution.
  • Minimizing L1 reduces H(X|D*), thereby maximizing I(D*; X) and encouraging D* to capture relevant data information.
  • Minimizing L2 maximizes I(S; X|D*) while the sample-specific representation has a fixed information capacity.
  • The fixed information budget is partitioned between information shared with D* and information unique to S given D*.

A.1.4. The Proof of Proposition 4.

Proposition 4 establishes that minimal sufficient, disentangled prompt tuning produces a synthetic distribution with strictly larger ε-support than vanilla prompt tuning.

  • The proof compares distributions generated by vanilla prompt tuning and minimal sufficient, disentangled prompt tuning.The proposition defines p_D and p_D* as p(X|D) and p(X|D*), respectively.
  • The argument uses an ε-support definition based on outcomes whose probability exceeds ε.For ε ∈ (0, 1/e), supp_ε(p) contains x such that p(x) > ε, with cardinality S_ε.
  • The ε-support of p(X|D*) is strictly larger than that of p(X|D).Here, D* denotes the minimal sufficient, disentangled representation trained with L1 + λL2, while D denotes vanilla prompt tuning.
  • An entropy-support bound underpins the comparison between the distributions’ support sizes.The proof invokes H(p) ≤ log S_ε, alongside the decomposition p = H(p) + δ_p.

A.2. Experimental Details

The experiments evaluate DOMINO on temporally separated coding and instruction-following benchmarks, combining synthetic-data case studies with performance and sensitivity analyses. Results show diverse, domain-consistent synthesis, strong generalization, near-reference-data performance, and dependence on representative reference data.

  • Experimental setup: DOMINO uses pre-cutoff samples as references and final-update samples as tests for Live Code Generation and Live Code Execution.The evaluation uses LiveCodeBench’s official scripts and reports Pass@k metrics.
  • Case study: DOMINO’s D* samples are selected outside D’s dense t-SNE cluster, providing a visual indication of greater synthetic diversity.Figure 7 contrasts standard synthetic samples from D with diverse samples from D*.
  • Case study: D samples mimic reference instances, copying structures, variable names, and naming conventions such as n, k, and k-xxx.This pattern is interpreted as sample-level overfitting.
  • Case study: D* samples abstract competitive-programming structure while introducing new graph-theory, dynamic-programming, and string-algorithm problem types.These problem types are distinct from the reference sample.
  • Generalization: DOMINO consistently outperforms baselines across four LiveBench subtasks, raising the average score by 3.48 points over the instruction-tuned backbone.It also exceeds standard prompt tuning by 1.61 points, using 40K synthesized samples and a Qwen2.5-7B-Instruct backbone.
  • Data quality: DOMINO’s equal-sized synthetic data performs very close to original reference data, supporting high per-sample quality.The authors attribute the remaining small gap to the reference samples’ ground-truth status.
  • Sensitivity analysis: Reducing reference data from 100% to 20% lowers performance, while the improvement plateaus between 80% and 100%.The result suggests sufficient reference quantity matters, but additional data yields diminishing gains beyond a threshold.
  • Baseline analysis: MAGPIE remains comparable with 3 or 5 reference samples, but using 10 samples severely compromises performance.The reported explanation is that the larger context challenges extraction of domain-specific information.
Loading 2605.30039v2…