Source-linked AI summary
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Shubhra Kanti Karmaker Santu, Dongji Feng
TL;DR
Research on diverse prompting for ill-defined complex tasks remains limited, and prompt variation makes benchmarking comparisons difficult. The paper proposes TELeR, a taxonomy organized around four prompt dimensions. It argues that reporting prompt categories against a common standard can support more meaningful comparisons and accurate conclusions.
Problem
Diverse prompt studies have focused mainly on well-defined NLP tasks, while complex-task performance can vary with prompt type and detail, complicating comparisons.
Source: §1 · pp. 1–2Method
TELeR categorizes complex-task prompts by turn, expression, role, and level of detail.
Source: §3 · p. 4Results
The authors propose that TELeR’s common standard will facilitate meaningful comparisons across studies and more accurate conclusions about LLM performance on complex tasks.
Source: §1 · p. 2Takeaways & Limitations
The paper urges researchers to report the TELeR prompt categories used in their studies to support comparison across independent work.
Source: §5 · p. 5TELeR is intended for LLM prompts targeting complex tasks, and its seven detail levels do not apply to simple tasks.
Source: §6 · p. 6
Abstract
from arXiv · showhide
While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied. Indeed, we are yet to conduct comprehensive benchmarking studies with multiple LLMs that are exclusively focused on a complex task. However, conducting such benchmarking studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, the paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs' performance on a specific complex task.
1 Introduction
Prompt effectiveness depends on prompt type and level of detail, making complex-task benchmarking difficult when studies use different prompts. The paper proposes TELeR as a shared taxonomy to make comparisons across studies more meaningful.
- Studies of diverse prompts have mostly addressed well-defined NLP tasks, while diverse prompting for ill-defined complex tasks remains rare.Source: §1 · p. 1
- Prompt choices and detail can substantially change performance on complex tasks, complicating comparisons across models and independent studies.These tasks may be ill-defined, goal-oriented, subjective, difficult to evaluate quantitatively, and composed of multiple subtasks.Source: §1 · pp. 1–2
- Accuracy reports without finer prompt details make comparisons across LLMs meaningless.Source: §1 · p. 2
- TELeR offers a common standard for categorizing prompts, intended to support meaningful comparisons and more accurate conclusions about LLM performance on complex tasks.Source: §1 · p. 2
2 Prompt Engineering for Complex Tasks
Prompt engineering is especially challenging for complex tasks, which require multiple steps and can respond differently to variations in prompt design. The section identifies prompt factors that affect how such tasks are specified and carried out.
- Prompt engineering iteratively crafts and revises queries or context to elicit desired LLM responses, often through trial and error.Source: §2 · p. 2
- Complex tasks often involve multiple subtasks and require semantic understanding, planning, reasoning, and natural-language generation, making prompt design more challenging.Source: §2 · p. 2
- Task specifications can vary in detail through clear goals, distinctions between data and directives, explicit subtasks, evaluation criteria, and few-shot examples.Examples can guide the model toward desired input-output patterns.Source: §2 · p. 2
- Other prompt choices include providing context, using question or instruction style, and delivering instructions in one turn or across multiple turns.The passage states these choices can affect model accuracy or performance.Source: §2 · p. 3
3 Proposed TELeR Taxonomy
TELeR categorizes complex-task prompts along four dimensions, treating directive details as the source of differences when the task data is fixed.
- With task data fixed, prompts differ in the specifics and details of their directives.Source: §3 · pp. 3–4
- TELeR classifies prompts by turn count, directive expression, whether a system role is defined, and level of detail.Turn is single or multi-turn; expression is question- or instruction-style; role is defined or undefined.Source: §3 · p. 4
- The detail dimension has seven levels, from Level 0 with no directive or aspects to Level 6 with the highest detail.Level 6 may include clear goals, subtasks, explanations or justifications, evaluation criteria, retrieved information, and/or few-shot examples.Source: §3 · p. 4
4 Two Example Use Cases
The paper illustrates TELeR with prompt examples for meta-review generation and narrative braiding. Both examples show how task prompts can be expanded across levels of detail.
4 Two Example Use Cases
- The examples apply TELeR to generating meta-reviews from peer reviews and combining alternative narratives into a braided narrative.Source: §4 · p. 4
4.1 Use-Case 1: Meta-Review Generation
- For meta-review generation, the examples progress from reviewer comments alone to prompts that specify desired content, questions, output constraints, retrieved information, and justification.The demonstration uses single-turn question-style prompts with no defined system role and assumes three reviewers.Source: §4.1 · pp. 4–5
4.2 Use Case 2: Narrative Braiding
- For narrative braiding, the examples progress from two input narratives to prompts that specify shared, unique, and conflicting information and subtasks for extracting, resolving, ordering, and condensing it.Higher levels add output constraints, retrieved information, and a request to justify included or discarded information.Source: §4.2 · p. 5
5 Final Words
TELeR is proposed as a unified standard for comparing LLM performance across independent studies. The authors urge researchers to report the prompt categories they use, supporting more meaningful comparisons and more accurate conclusions.
- TELeR is proposed as a unified standard for comparing and benchmarking LLM performance across independent studies.Source: §5 · p. 5
- The authors urge future work to report the specific TELeR prompt categories used in prompt design.Source: §5 · p. 5
- Such reporting is intended to support more meaningful comparisons, more accurate conclusions, and faster consensus on state-of-the-art performance.Source: §5 · p. 5
6 Limitations
TELeR is limited to prompts for complex tasks, and its seven levels do not apply to simple tasks. The taxonomy is not presented as final: the authors welcome extensions, including application-specific dimensions.
- TELeR applies exclusively to prompts for complex tasks, and its seven levels do not apply to simple tasks.Source: §6 · p. 6
- The authors state that TELeR should not be considered an ultimate taxonomy and that further extensions are possible and desired.Source: §6 · p. 6
- The taxonomy can be extended with dimensions added as needed by the target application.Source: §6 · p. 6