PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations

Guy Rotman, Adi Kopilov, Danit Berger Zalmanson, Omri Allouche

arXiv:2606.15641v1cs.CL

TL;DR

Specialized B2B conversation classification remains insufficiently examined despite limited labels, evolving intents, and the need for interpretable outputs. This paper distills examples into task instructions and reports stronger performance than example-based methods, with Criteria reaching 76.3% AUC versus 69.3% for Examples.

  • Problem

    ICL remains insufficiently examined for specialized B2B conversations, where evolving intents, limited labels, and interpretability requirements complicate classification.

  • Method

    The paper introduces a five-concept B2B benchmark and distills verbose examples into structured classification criteria and precise task descriptions.

  • Results

    Advanced prompting methods consistently outperform basic example-based methods, with Criteria achieving 76.3% AUC versus 69.3% for Examples.

  • Takeaways & Limitations

    The framework provides interpretable task instructions that support human-in-the-loop enhancement and user guidance.

  • Takeaways & Limitations

    The framework depends on the underlying LLM to extract accurate criteria, making it susceptible to hallucinated rules, missed nuances, and error propagation.

Abstract

from arXiv · show

In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the \texttt{Call Playbook} dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99\% reduction in token usage and improves macro-averaged AUC by up to 7\% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.

1 Introduction

The introduction identifies severe limitations of standard in-context learning for specialized B2B sales conversations under limited-data and low-annotation constraints. It presents a knowledge-extraction framework and five real-world B2B classification tasks to replace concatenated examples with transparent classification criteria and task descriptions.

  • Specialized B2B sales classification remains underexamined despite the broader success of in-context learning across NLP tasks.
  • Organizations need automated classification of diverse, evolving conversational intents despite limited labeled data, minimal annotation overhead, and impractical task-specific fine-tuning.
  • Standard in-context learning degrades with far fewer tokens than modern context windows can support because concatenating multiple few-shot examples undermines the assumption that more examples improve performance.
  • The proposed framework extracts generalizable task knowledge from a small labeled example set and converts it into natural-language classification criteria and task descriptions that substitute for few-shot examples.
  • Unlike retrieval, prompt-optimization, and token-level compression methods, the approach eliminates example concatenation and produces transparent, interpretable representations for automated processing and human oversight.
  • The Call Playbook dataset introduces five annotated B2B classification tasks derived from real-world sales conversations and targeting fundamental sales concepts.

2 Related Work

Related work examines how demonstrations, context length, and compression shape in-context learning, while B2B NLP research has largely overlooked extended, multi-stakeholder sales dialogues. This paper addresses both gaps with concise task-knowledge demonstrations and a dataset of authentic B2B conversations that supports user-guiding task instructions.

  • In-context learning: ICL research studies prompt design and the selection, ordering, retrieval, and active-learning-based subset selection of demonstrations.This work instead derives task-specific knowledge directly from a small number of examples.
  • Context compression: Long-context ICL struggles when many examples are used, motivating token-level and model-based compression methods.Token-level methods remove redundant content or iteratively compress examples, while model-based approaches target contexts such as retrieval-augmented generation and question answering.
  • Context compression: The proposed method generates concise, interpretable demonstrations that preserve coherence and essential task knowledge without additional model training.This contrasts with model-based compression approaches that require labeled datasets and specialized training.
  • B2B NLP: B2B conversations challenge NLP through multiple stakeholders, long sales cycles, domain-specific language, and implicit business signals.These properties make B2B conversations a rigorous benchmark for interpreting nuanced, context-specific language.
  • B2B NLP: Existing B2B NLP work addresses business goals, sales, customer segmentation, and forecasting but overlooks fluid multi-stakeholder dynamics across deal progression.The paper introduces authentic, extended multi-party sales dialogues annotated for interconnected prospect-focused concepts and demonstrates automatically generated user-guiding task instructions for ICL.

3 The Call Playbook Dataset

The Call Playbook dataset comprises annotated English B2B sales-call transcripts organized around five core sales concepts. Its construction combines expert-supervised span annotation, overlapping conversational snippets, and balanced, leakage-controlled evaluation sets.

  • Dataset composition: 50 English B2B sales calls lasting 30–90 minutes form the dataset, with transcripts represented as speaker-attributed monologues from sellers or prospects.The dataset includes annotations for five key sales concepts.
  • Annotation scheme: The five concepts are Business Goals, Decision Criteria, Decision Makers, Decision Making Process, and Pain Points.They capture distinct aspects of prospect intent and the purchasing process, from desired outcomes and evaluation standards to decision roles, steps, and obstacles.
  • Annotation and segmentation: Three trained in-house annotators, supervised by a domain expert, labeled spans expressing each concept in overlapping snippets of five consecutive monologues.Adjacent snippets overlap by one monologue, preserving conversational flow while enabling targeted classification.
  • Dataset splits: 200 samples per concept populate each class-balanced train and test set.When positive examples were limited, positives were split evenly and remaining slots filled with randomly sampled negatives.
  • Dataset splits: Calls did not overlap between train and test sets, preventing data leakage and reducing the risk that majority-class prevalence would mask evaluation results.The resulting benchmark maintains balanced or nearly balanced distributions across most categories.

4 Methodology

The methodology frames B2B conversational classification as binary prediction from minimal task intent and few labeled examples, then replaces raw few-shot demonstrations with distilled criteria or task descriptions. These representations reduce token demands, make classification logic interpretable, and support iterative refinement for human-in-the-loop workflows.

  • Problem setup: Given a task intent and labeled conversational segments, the classifier predicts whether an unseen segment contains the target concept, using binary labels 1 and 0.The setup defines f: I × X → C, with C = {0, 1}; 1 denotes presence and 0 absence of the concept specified by the intent.
  • General framework: The framework samples N labeled examples while preserving class distribution, then either prompts with raw examples or extracts distilled knowledge for classification.Sample selection is shared across all proposed methods, and knowledge extraction transforms examples into criteria or detailed task descriptions.
  • Baseline: Raw Examples consume token budget as N grows and require models to infer classification rules implicitly, which can impair generalization on difficult or underspecified tasks.Conversational snippets may be hundreds of words long, making longer few-shot contexts costly and potentially less reliable.
  • Knowledge extraction methods: Summary-Ex condenses each example into 3-5 sentences, reducing token usage while retaining the example-based format, but still depends on implicit pattern inference.The method preserves speaker affiliations and business content while removing redundancy and filler words, though contextual nuances may be lost.
  • Knowledge extraction methods: Criteria-Ex generates positive and negative criteria from examples, making classification logic explicit while reducing tokens and improving generalization beyond specific demonstrations.The extracted criteria indicate concept presence or absence and require fewer tokens than original conversation snippets.
  • Knowledge extraction methods: Description-Ex creates a cohesive task description explaining concept boundaries, while Criteria-De and Description-Cr iteratively convert descriptions and criteria to support progressive refinement.These representations are intended to capture context-dependent relationships and enable successive refinement in dynamic human-in-the-loop workflows.

5 Experiments

The experiments evaluate methods across five Call Playbook concepts, varying few-shot context size and testing five language models under controlled conditions. Results are averaged over repeated random samples with preserved class distributions.

  • Evaluation setup: Five Call Playbook concepts are evaluated across ICL settings containing 0, 10, 25, 50, 75, or 100 examples.Examples are varied systematically to assess performance across different few-shot levels.
  • Evaluation setup: Five repeated runs use randomly sampled examples while preserving the original class distribution, with averaged metrics reported for statistical reliability.Each configuration is repeated with different random samples to mitigate potential biases.
  • Evaluation setup: The evaluation covers GPT-4o, Claude Sonnet 3.7, Claude Haiku 3, Mistral Large, and Mistral Small.The models include proprietary and open-weight systems with diverse architectures and parameter scales.
  • Evaluation setup: The same examples are used across models, and temperature is set to 0 to ensure fair comparisons and deterministic outputs.This controls the evaluation conditions across the five language models.

6 Results

Across five models and concepts, structured criteria- and description-based prompting outperformed example-based methods while remaining robust to longer contexts. The approach also reduced token usage and processing time, resisted extractive-compression degradation, and supported effective human refinement.

  • Method Performance Overview: 76.3% AUC was achieved by Criteria-De, narrowly exceeding Criteria-Ex at 76.2%, while Examples trailed at 69.3%.Description variants reached 75.9% and 75.6%, whereas Summary-Ex reached 72.0%.
  • Few-shot Scaling Patterns: Examples declined from 71.5% macro-F1 at 0-shot to 60.7% at 100-shot, while Summary-Ex fell to 64.7%.The degradation was observed consistently as few-shot examples increased.
  • Few-shot Scaling Patterns: Description-Ex improved from 71.5% to 72.6% macro-F1 across 0 to 100 shots, while Criteria-De rose from 69.3% to 72.2%.Description-Cr and Criteria-Ex peaked at 25 shots before declining, indicating performance saturation.
  • Model and Concept Patterns: Sonnet 3.7 achieved the highest overall model performance at 77% macro-average F1, while Decision Makers reached 80% across methods.Decision Makers performed best across concepts, and concept characteristics determined whether descriptive context or structured criteria were more suitable.
  • Compression Baselines: SC dropped from 71.4% to 62.1% macro-F1 under increasing example counts, whereas LLMLingua-2 remained below 50% throughout.The extractive baselines pruned task instructions or produced fragmented examples, while structured distillation generated coherent, interpretable patterns without training.

7 Conclusions

The work improves in-context learning for professional B2B classification by distilling examples into precise task instructions and introducing a five-concept benchmark. Across models and few-shot settings, the framework delivers strong performance, efficiency, adaptability, interpretability, and scalability without domain-specific customization.

  • Contributions: The study introduces knowledge-extraction methods that distill examples into precise task instructions for B2B classification.It also contributes a benchmark spanning five essential B2B concepts.
  • Empirical results: The methods outperform standard ICL and token-level compression techniques across varied LLMs and few-shot settings while improving time efficiency and cost-effectiveness.The experiments cover multiple models and few-shot configurations.
  • Interpretability and scalability: The framework adapts across business concepts without domain-specific customization and produces interpretable artifacts that support human-in-the-loop collaboration.Its flexible architecture also enables distillation from powerful models to efficient models, supporting transparent and scalable NLP applications.

Limitations … A.3 Annotator Profile and Recruitment

The study is limited by its small expert-annotated dataset, dependence on LLM reasoning, and focus on binary classification. The paper also documents licensing, governance, and annotator recruitment practices.

  • Limitations: The evaluation uses 50 B2B sales calls lasting 30–90 minutes, so broader validation across additional B2B corpora remains necessary.Three independent professional annotators labeled the calls under expert supervision, reflecting the resource-intensive nature of expert human annotation.
  • Limitations: The knowledge extraction framework can propagate errors when the underlying LLM hallucinates rules or misses subtle semantic nuances.The quality of distilled artifacts is bounded by the distillation model’s capabilities despite observed performance gains.
  • Limitations: The work focuses on binary classification, while extending the representation to more classes would increase its size linearly and potentially reduce efficiency.Underrepresented classes may also weaken generated criteria, descriptions, generalization, and overall performance.
  • A.1 Data Licensing and Intended Use: The dataset is released for non-commercial NLP research, and derivatives must remain within research contexts.The complete license agreement accompanies distribution, and existing artifacts were used according to their intended purposes and licensing terms.
  • A.2 Ethical Approval and Governance: The data collection protocol underwent internal legal and scientific governance review for compliance with ethical guidelines and data protection standards.
  • A.3 Annotator Profile and Recruitment: The annotation task was completed by female native speakers of American English recruited through established professional networks.
  • A.3 Annotator Profile and Recruitment: Annotators were compensated at rates consistent with United States industry standards for linguistic annotation work.

B Data Anonymization … E.3 Description Generation

The paper anonymizes and simplifies B2B conversation snippets while preserving meaning, then classifies five sales concepts through structured prompts populated with examples or distilled summaries, criteria, and descriptions. These representations are generated from labeled examples or user intents to support concise, consistent concept recognition.

  • B Data Anonymization: Sensitive entities were detected through trained-annotator review, automated heuristics, and Claude Sonnet 3.7, then replaced with fictional alternatives while preserving semantic coherence.Names, organizations, products, locations, contact information, and numeric identifiers were covered; roles, titles, and percentages remained unchanged.
  • B Data Anonymization: Sentence-level rewriting with Claude Sonnet 3.7 altered syntax and wording while preserving conversational structure, semantic content, and essential meaning.Experiments confirmed no performance degradation between original and anonymized data.
  • C Representative Examples of the Dataset: Representative positive snippets cover five concepts and illustrate diverse, complex B2B dialogues with bolded relevant spans and color-coded prospect and seller utterances.The examples were processed and anonymized according to the described procedures.
  • D Classification Methodology: The classification method uses a structured prompt containing a user-provided objective, labeled-example knowledge, and a desired structured output format.The model analyzes a snippet for evidence of a specified B2B concept, while the knowledge section varies across examples, summaries, criteria, and descriptions.
  • D.2 User Intents: Five high-level user intents target Business Goals, Decision Criteria, Decision Makers, Decision Making Process, and Pain Points in purchasing conversations.Their abstraction reflects typical real-world user inputs and motivates augmenting intents with knowledge from labeled examples.
  • D.3 Knowledge Section Formats: Positive examples or criteria precede negative ones in every knowledge format because experiments found this ordering produced superior performance.The formats are Examples, Criteria, and Description, with Summary-Ex also using the Examples format.
  • E Knowledge Extraction Process: LLMs transform labeled examples into condensed summaries, explicit criteria, and task descriptions that are inserted into the classification prompt’s knowledge section.The extraction process defines prompt designs for each representation and connects them to the knowledge-section formats.
  • E.1 Summary Generation; E.2 Criteria Generation; E.3 Description Generation: Summary-Ex compresses labeled snippets while preserving meaning, speaker structure, and conversational flow; Criteria-Ex and Criteria-De generate positive and negative criteria, while Description-Ex and Description-Cr generate balanced task descriptions.Criteria can derive from examples or an existing description, and descriptions can derive from examples or Criteria-Ex; zero-shot variants use only the user intent.

F Comparative Analysis of the Criteria and Description Methods · G Abstraction vs. Coverage in LLM-Generated Descriptions

The comparison shows that criteria-based instructions provide systematic, granular guidance, whereas narrative descriptions offer contextual and holistic guidance. Description abstraction and example coverage trade off differently across models, with Sonnet 3.7 favoring coverage, Mistral abstraction, and GPT-4o balance.

  • F Comparative Analysis of the Criteria and Description Methods: Criteria-Ex and Criteria-De give discrete categorical guidelines, whereas Description-Ex and Description-Cr provide more contextual guidance accessible to non-experts.The criteria-based approaches support systematic application; narrative descriptions flow more naturally.
  • F Comparative Analysis of the Criteria and Description Methods: Criteria-Ex emphasizes product or service connections to business context, while Criteria-De emphasizes objectives; Description-Ex highlights conversational indicators, whereas Description-Cr emphasizes measurable outcomes.
  • F Comparative Analysis of the Criteria and Description Methods: All four approaches identify positive examples as containing business objectives, using explicit goal statements in Criteria-Ex and measurable metrics in Criteria-De.The approaches therefore provide complementary analytical lenses for the same positive cases.
  • F Comparative Analysis of the Criteria and Description Methods: Each approach flags negative examples as lacking business-objective articulation, distinguishing small talk in Negative 1 from administrative details without business context in Negative 2.
  • F Comparative Analysis of the Criteria and Description Methods: Criteria-based approaches provide granular analytical points that may support consistent annotation, whereas narrative descriptions offer holistic frameworks for contextual nuances.Together, the methods balance analytical precision with contextual sensitivity.
  • F Comparative Analysis of the Criteria and Description Methods: Criteria-based methods systematically decompose conversational elements, while narrative descriptions encourage comprehensive evaluation of speaker intent.These distinct pathways inform robust annotation protocols.
  • G Abstraction vs. Coverage in LLM-Generated Descriptions: Sonnet 3.7 achieves the highest coverage, especially with larger example sets, whereas both Mistral models achieve superior abstraction with reduced source-example coverage.Coverage uses cosine similarity between descriptions and source examples; abstraction uses concreteness ratings.
  • G Abstraction vs. Coverage in LLM-Generated Descriptions: GPT-4o balances good abstraction with reasonable coverage, while Mistral follows the expected example-count trade-off, Sonnet 3.7 shows the opposite trend, and Haiku 3 shows no clear correlation.Descriptions that mirror examples tend to be more concrete, whereas abstract descriptions capture fewer specific details.

H Qualitative Analysis of Generated Classification Criteria

The qualitative analysis of more than 3,000 generated criteria reveals distinct linguistic patterns between example-derived and description-derived criteria, while both favor abstract, declarative guidance over quantitative or conditional reasoning.

  • Linguistic differences: Both approaches achieve an abstraction score of 0.69 and produce comparable numbers of positive and negative criteria.Criteria-De generates slightly more criteria per sample.
  • Reasoning patterns: Quantifiable metrics are nearly absent across all criteria, indicating a strong preference for qualitative rather than quantitative classification reasoning.This pattern was observed across the analyzed criteria.
  • Reasoning patterns: 38.1% of Criteria-De criteria use conditional statements, compared with 33.9% for Criteria-Ex.Both approaches use conditional logic relatively infrequently, favoring declarative statements instead.

I Human-in-the-Loop Modification Example … K.2 Token-Level Compression Comparison Grid

The paper shows that targeted human refinement can substantially improve distilled classification criteria, while the proposed criteria and description methods remain efficient and robust as example counts increase. Across comprehensive evaluations, these methods outperform or remain more stable than token-level compression baselines across B2B concepts and models.

  • I Human-in-the-Loop Modification Example: Human modifications to generated criteria produced contrasting results on Pain Points classification, with the original model achieving 73.2% F1.The comparison examines Sonnet 3.7’s original criteria and revisions by two annotators.
  • I Human-in-the-Loop Modification Example: 83.0% F1 (+9.8% improvement) followed Annotator 1’s strategic simplification, semantic generalization, and scope expansion.These changes enhanced the criteria’s applicability and clarity.
  • I Human-in-the-Loop Modification Example: 71.9% F1 (-1.3% degradation) followed Annotator 2’s excessive paraphrasing, increased linguistic complexity, and inconsistent terminology.The revisions changed wording without semantic improvement and introduced unnecessary complexity.
  • I Human-in-the-Loop Modification Example: Effective refinement requires strategic simplification, semantic broadening, and principled scope adjustments rather than superficial rewording or unnecessary complexity.The resulting interpretable artifacts can be refined through targeted human expertise to improve classification performance.
  • J Average Processing Time: Criteria and description methods scale more efficiently than the traditional Examples method as few-shot example counts increase, maintaining stable processing times.Summary-Ex provides moderate improvements, while knowledge extraction requires only one LLM call per few-shot example set.
  • K.1 Main Results Grid: Figure 16 evaluates macro-average F1 across five B2B concepts, five models, and increasing numbers of few-shot examples.The grid enables direct comparison of few-shot learning methods across Business Goals, Decision Criteria, Decision Makers, Decision Making Process, and Pain Points.
  • K.2 Token-Level Compression Comparison Grid: Criteria-Ex and Description-Ex remain robust as examples increase, whereas LLMLingua-2 stays under 50% F1 across concepts and SC degrades consistently.The token-level compression comparison is averaged across all five models; SC shows steep declines across every concept.
Loading 2606.15641v1…