Source-linked AI summary
Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior
Minjae Lee, Minsuk Kahng
TL;DR
LLM application developers can edit prompts, but encoding nuanced local policies is challenging and test data is often developed separately from prompt instructions. The paper introduces an interactive co-evolution workflow in which users grow test sets and refine prompts together; a controlled study reports more systematic refinement and better alignment with intended policies than a baseline.
Problem
Prompt instructions and test data are often developed separately, despite the challenge of encoding nuanced, application-specific policies in LLM prompts.
Source: §1 · p. 2Method
The interactive workflow generates edge cases, elicits user rationales, and iteratively evaluates revised prompts against a growing test set.
Source: §1 · p. 2, §4.1.5 · p. 6, §4.3 · p. 7Results
The controlled study found that the workflow produced more concrete policy refinement, richer test sets, and more consistent alignment between participants’ intentions and model behavior than a baseline prompt-editing tool.
Source: §1 · p. 2, §9 · p. 15Takeaways & Limitations
The findings suggest that iterative, evidence-grounded test-set growth in the development loop can support continuous, human-in-the-loop LLM refinement.
Source: §9 · p. 15The study focuses on prompt refinement; applying co-evolution to model-parameter updates remains more technically challenging.
Source: §8.2.7 · p. 15
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly embedded in applications, and people can shape model behavior by editing prompt instructions. Yet encoding subtle, domain-specific policies into prompts is challenging. Although this process often benefits from concrete test cases, test data and prompt instructions are typically developed as separate artifacts, reflecting traditional machine learning practices in which model tuning was slow and test sets were static. We argue that the fast, iterative nature of prompt engineering calls for removing this separation and enabling a new workflow: data-prompt co-evolution, where a living test set and prompt instructions evolve in tandem. We present an interactive system that operationalizes this workflow. It guides application developers to discover edge cases, articulate rationales for desired behavior, and iteratively evaluate revised prompts against a growing test set. A user study shows our workflow helps people refine prompts systematically, better aligning them with their intended policies. This work points toward more robust and responsible LLM applications through human-in-the-loop development.
1 Introduction
Prompt engineering lets application developers shape LLM behavior, but general-purpose models lack local context and nuanced policies, while test-data creation and prompt refinement remain separated. The paper proposes data-prompt co-evolution and an interactive workflow, reporting more systematic prompt and test-set refinement and better alignment than a baseline.
1 Introduction
- General-purpose LLMs lack application-specific context and nuanced policies, prompting reactive edits and spot-checking against a few examples.Comprehensive test data for the long tail of LLM applications is difficult to author, and existing practices leave feedback between failure discovery and behavior refinement broken.Source: §1 · p. 2
- Static test sets and prompt refinement remain separate despite prompt iteration taking seconds, creating friction in interactive development.The separation reflects traditional machine-learning practices in which slow, costly fine-tuning encouraged distinct data-curation and model-refinement stages.Source: §1 · p. 2
- Data-prompt co-evolution treats test data and prompt instructions as interdependent artifacts that evolve together through discovered failures and prompt revisions.The system proposes candidate inputs, elicits user labels and rationales, generates similar cases, and retains labeled data in a growing test set.Source: §1 · p. 2
- The study’s workflow produced more detailed instructions, larger and richer test sets, more consistent model-behavior changes, and higher satisfaction than a baseline.The controlled study included 16 participants.Source: §1 · p. 2
- The paper contributes a co-evolution workflow, an interactive system, and controlled-study insights on prompt specificity, test-set growth, behavioral alignment, and satisfaction.The work frames these contributions as supporting more responsible, policy-aware LLM application development.Source: §1 · p. 2
2 Related Work
Prior work addresses prompt design, interactive annotation, test-driven evaluation, alignment, and red teaming, but this paper positions its workflow around coupling generated failure cases with policy refinement. It emphasizes human participation in defining application-specific policies and iterating on them.
2.1 Human-Centered Prompt and Policy Design
- Human-centered prompt tools support policy authoring and prompt comparison, while automated prompt optimization treats design as black-box search and can struggle with implicit human intentions [36] [32] [28] [42] [52] [24] [2] [17] [9] [8] [65].The paper argues for bringing people into the loop to define and refine governing policies.Source: §2.1 · pp. 2–3
2.2 Interactive Machine Learning and Labeling
- Interactive machine-learning and labeling work supports iterative human-model adaptation, but this workflow additionally elicits failure rationales for prompt revisions and test-set improvements [48] [23] [31].Data-centric methods such as weak supervision prioritize throughput and coverage, potentially missing interactive elicitation [41].Source: §2.2 · p. 3
2.3 Data Iteration and Test-Driven Development
- Behavioral test frameworks turn errors into executable checks, but often use static datasets or require substantial manual test formulation [44] [4].The paper’s comparison with prior iterative tools distinguishes a setting where new examples must be generated and failures translated into policies [57].Source: §2.3 · p. 3
- Unlike fixed-corpus example selection, data-prompt co-evolution generates cases when a corpus may not exist and turns failure rationales into prompt-policy updates [57].The workflow couples synthesized failure-grounded slices with iterative policy updates rather than stopping at test generation or harm awareness.Source: §2.3 · p. 3
2.4 Bidirectional Human-AI Alignment and Criteria Drift
- Prior work describes criteria drift as users’ standards changing during interaction, whereas this work also focuses on refining the underlying policy specifications [24] [46].Source: §2.4 · p. 3
2.5 Human Roles in Red Teaming
- Red-teaming research often separates dataset curators from developers, while this workflow supports domain experts and smaller teams in probing their own systems and capturing local policies [15, 38, 22, 64, 13].It shifts red teaming beyond benchmark collection toward a process accessible without deep technical expertise.Source: §2.5 · p. 4
3 Formative Study
Five practitioners’ interviews surfaced four findings about iterative development, prompt use, application-specific testing, and evaluation trade-offs. Together, they motivated an integrated workflow that treats prompt specifications and test inputs as evolving artifacts.
3.1 Participants and Procedure
- The formative study used 60-minute semi-structured interviews with five practitioners from varied roles and LLM application domains, analyzed through thematic analysis.Interviews were conducted in Korean and approved by the institution’s IRB; one participant’s excerpts were later excluded, while the four themes remained consistent across five interviews.Source: §3.1 · p. 4
3.2 Findings
- The findings identify a gap between current practices and the need for more systematic tooling.Source: §3.2 · p. 4
3.2.1 Development is a Process of Constant, Unending Iteration.
- Practitioners described LLM application development as continuous iteration, with unexpected cases prompting policy edits and quick checks of whether changes generalize.Early deployments often resembled extended beta tests as colleagues, testers, or users surfaced unanticipated cases.Source: §3.2.1 · p. 4
3.2.2 Prompt Engineering is the Preferred Modality for Rapid Iter-
- Most practitioners preferred prompt engineering to fine-tuning for its speed and cost, as application needs increasingly centered on encoding nuanced policies and constraints.Source: §3.2.2 · p. 4
3.2.3 Test Sets are Diverse, Application-Specific, and Often Ad-Hoc.
- Application-specific test sets were often informal and diverse, spanning difficulty levels, out-of-scope inputs, and sensitive or problematic cases.Some collections were simple shared-document lists without systematic evaluation structure.Source: §3.2.3 · pp. 4–5
3.2.4 Evaluation Dichotomy Between Rapid Gut-Checks and Costly Metrics.
- Practitioners balanced fast, low-confidence gut-checks against costly formal evaluation, which they used when greater confidence or stakeholder justification was needed.Manual review costs limited some teams to formal evaluation about once a month, leaving no continuous clear signal of whether iterative changes improved the system.Source: §3.2.4 · p. 5
3.3 Design Implications
- The design implications call for structured iteration, a durable living test set, and faster side-by-side evaluation with human oversight.The proposed co-evolution workflow treats prompts and test inputs as intertwined: failures should affect the specification, and specification changes should be checked against accumulated data.Source: §3.3 · p. 5
4 Interactive Co-Evolution Workflow
The paper presents an interactive system to operationalize data-prompt co-evolution for refining LLM behavior. This section introduces its design principles, interface, and workflow.
4.1 Design Principles
Data-prompt co-evolution treats the prompt instruction and test set as linked artifacts: cases expose weaknesses, while user rationales ground revisions. A growing, human-governed test set supports generalizing policy rules and tracking progress.
4.1 Design Principles
- The prompt is the editable system instruction encoding intended policy, while data is the test set of end-user inputs to the model.These terms distinguish the policy specification from the examples used to validate it.Source: §4.1 · p. 5
4.1.1 From Prompt to Data: Using the Specification to Discover Its Own Blind Spots.
- The current instruction generates challenging inputs that surface potential failure modes and help users anticipate problematic cases.The workflow uses data synthesis to probe weaknesses beyond average-case behavior.Source: §4.1.1 · pp. 5–6
4.1.2 From Data to Prompt: Grounding Specification Refinements in Concrete Evidence.
- Users label surfaced cases and explain failures with rationales that clarify policy and ground prompt updates in concrete evidence.The rationale guides the instruction revision rather than leaving changes to vague intuition.Source: §4.1.2 · p. 6
4.1.3 A Living Test Set.
- Discovered cases persist in a growing test set that reflects the current prompt and supports progress assessment and regression detection.The set serves as an evolving benchmark for systematic improvement.Source: §4.1.3 · p. 6
4.1.4 LLM-Assisted, Human-Governed Workflow.
- The LLM suggests edge cases, rationales, similar examples, and prompt revisions, while users retain final authority because suggestions remain editable.Assistance accelerates the process without removing human control.Source: §4.1.4 · p. 6
4.1.5 Scaffolding Generalization: From a Single Case to a General Rule.
- Users test a failure rationale against similar cases before revising prompts, helping turn a one-off correction into a broader policy rule.The workflow aims to build confidence and support more robust results through this generalization step.Source: §4.1.5 · p. 6
4.2 System Interface
The interface separates prompt editing from test-data discovery while linking them through version comparisons. Its panels support prompt tracking, test-set evaluation, and review of generated examples.
4.2 System Interface
- The two-column interface separates prompt specification from data, with prompt editing and version tracking on the left and test-data discovery on the right.The panels keep the artifacts distinct while supporting their joint refinement.Source: §4.2 · p. 6
4.2.1 The Prompt Instruction Panel .
- The prompt panel combines an instruction editor with version history that charts each version’s accuracy on the current test set and allows comparison or reversion.The chart distinguishes correct and incorrect responses and user labels from LLM-as-judge labels.Source: §4.2.1 · pp. 6–7
4.2.2 The Data Panel: Discovering and Accumulating Evidence.
- The data panel places labeled test cases beside staged, unlabeled generated examples; labeling a generated case promotes it into the test set.The test-set table shows old and new prompt responses side by side for comparison.Source: §4.2.2 · p. 7
4.3 Co-Evolution Workflow in Action
The workflow moves from hard-case discovery to rationale articulation, neighborhood validation, prompt revision, and test-set evaluation. Each cycle preserves labeled cases and compares responses across prompt versions.
4.3 Co-Evolution Workflow in Action
- The system generates hard or ambiguous candidate inputs, and users inspect responses and label failures against the application’s policy.In the banking example, a credit-score question mentioning job applications is labeled incorrect when the chatbot refuses to answer.Source: §4.3 · p. 7
- For an incorrect response, users provide or edit a rationale that turns a judgment about the example into an explicit policy principle.The interface offers LLM-suggested rationale options while allowing users to write their own.Source: §4.3 · p. 7
- Neighborhood probing presents semantically similar inputs with varied contexts so users can check whether a rationale predicts failures beyond the original case.Users can refine a rationale when it does not hold across the neighborhood; validated examples join the test set.Source: §4.3 · p. 7, Figure 4 · p. 8
- After neighborhood labels are collected, the system synthesizes a prompt revision from the failure, rationale, and labels, which users may accept, edit, or reject.Suggestions may add a policy line or clarification.Source: §4.3 · p. 8
- Applying a revision adds the failed case and neighborhood examples to the test set, runs the new prompt across all cases, and supports LLM-judge labels that users can override.The test-set view compares old and new responses, while version history updates its accuracy metric.Source: §4.3 · p. 8
- The cycle of failure discovery, prompt revision, and test-set evaluation repeats as users probe for further edge cases.The workflow aims to strengthen test sets and models in tandem.Source: §4.3 · p. 8
4.4 Implementation Details
The system uses a browser-based React interface and a Flask server to support LLM-driven workflow steps. Parallel judge calls reduce evaluation time as test sets grow.
4.4 Implementation Details
- A browser-based React interface connects to a Flask server that handles LLM calls through external APIs.The server supports the workflow’s LLM-dependent stages.Source: §4.4 · p. 8
- Gemini 2.5 Flash [11] powers most workflow steps, while Gemini 2.5 Flash Lite handles end-user model responses.The implementation avoided reasoning-heavy thinking mode in most cases to keep interactions fast.Source: §4.4 · p. 8
- Evaluating 50 examples took 5.6 seconds with 10 workers (9.4x speedup) and 3.4 seconds with 20 workers (15.6x speedup), versus 52.8 seconds without parallelization.The experiment reports near-linear speedup; API response time still bounds latency.Source: §4.4 · p. 8
5 Technical Experiments
Controlled simulations test whether failure discovery produces useful edge cases and whether rationale-informed prompt refinement improves behavior across iterations. Results show stronger failure coverage, better single-iteration accuracy than an ablated workflow, and later gains that plateau under fixed personas.
5.1 Setup
- The simulation used two moderation domains and deterministic personas as ground-truth oracles, with a classifier LLM mapping inputs to policy categories at 89% accuracy.The classifier was the pipeline’s only LLM; persona moderation decisions were deterministic.Source: §5 · p. 9, §5.1 · p. 9
5 Technical Experiments
- The experiments test whether the workflow’s failure-discovery and instruction-refinement mechanisms operate as intended in a controlled setting.The authors note that real-world policy design involves evolving human intent not captured by fixed personas.Source: §5 · p. 9, §5.3.2 · p. 10
5.2 Experiment 1: Validity and Coverage of Failure Discovery
- Failure discovery was evaluated for validity and semantic coverage across generated cases, comparing the workflow with simple and adversarial generation baselines.Five failure candidates per persona were assessed using two metrics averaged over 10 runs.Source: §5.2 · p. 9
- Failure discovery achieved validity 0.635 versus 0.497 for adversarial generation and 0.031 for simple generation, and coverage 0.917 versus 0.677 for adversarial generation.Coverage captures examples that trigger failures for at least one persona, indicating policy grey areas.Source: §5.2 · p. 9
5.3 Experiment 2: Effectiveness of Instruction Refinement
- Instruction refinement was assessed both by removing rationale generation and neighborhood verification and by simulating eight consecutive discovery-and-refinement cycles.Each of two domains paired with four personas was run ten times for the sequential experiment.Source: §5.3 · p. 9, §5.3.1 · p. 9, §5.3.2 · p. 9
5.3.1 Ablation Study.
- The full workflow reached holdout accuracy 0.502 after one iteration, versus 0.428 for the ablated baseline and 0.403 initially.The ablation removed rationale generation and neighborhood verification; the holdout contained 20 real-user examples.Source: §5.3.1 · p. 9
5.3.2 Sequential Iterations.
- Accuracy rose from 0.408 to 0.517 after the first iteration, peaked at 0.591 at step 7, then stabilized at 0.588 by step 8.The authors attribute the later plateau partly to fixed personas with deterministic, well-scoped policies; real users may revise policies over time.Source: §5.3.2 · p. 10
6 User Study Setup
The user study compared Co-Evolution with a conventional prompt-editing interface in a counterbalanced, within-subjects design. Participants refined content-moderation instructions, and separate real-world holdout sets were used to evaluate their prompts against their own policy labels.
6 User Study Setup
- A controlled lab study compared the workflow with a conventional prompt-editing interface for refining application-specific LLM behavioral policies.Each participant completed both conditions.Source: §6 · p. 10, §6.1 · p. 10
6.1 Study Design
- Condition order and two comparable moderation domains were counterbalanced; participants also completed a tutorial in a third domain before each block.The within-subjects study used a balanced Latin square to mitigate learning and carryover effects.Source: §6.1 · p. 10
6.2 Task: Refining a Content Moderation Policy
- Participants acted as policy designers, crafting moderation prompts that allow critical commentary while filtering harmful content and reflecting their intended boundaries.The task covered campus-community and restaurant-review contexts.Source: §6.2 · p. 10
6.3 Conditions: Co-Evolution vs. Baseline
- Co-Evolution offered generated examples, neighborhood exploration, and prompt suggestions; the baseline allowed manual test inputs and output inspection but lacked those tools.Both conditions began with the same three test examples to reduce cold-start differences.Source: §6.3 · p. 10
6.4 Participants
- The 16 participants were university students in South Korea, familiar with generative AI and with AI/ML coursework or equivalent experience.The study targeted practitioners who need not be professional software engineers but understand core AI concepts.Source: §6.4 · p. 10
6.5 Session Protocol
- Each remote session included two task blocks, holdout labeling, and post-questionnaires; Co-Evolution instructions were longer and included more policy examples than baseline instructions.The questionnaire had 10 Likert-scale and three open-ended questions; the figures compare word and sentence counts and policy-specificity features.Source: §6.5 · p. 10, Figure 5 · p. 11, Figure 6 · p. 11
6.6 Holdout Evaluation Sets
- Separate 20-item holdout sets from online communities balanced eight borderline cases with 12 clear-cut cases; participants labeled them and their prompts were evaluated on them.The holdouts were distinct from interface-created test sets and used solely for post-task evaluation.Source: §6.6 · p. 11
7 Study Results
In the user study, Co-Evolution improved alignment between participant intentions and model outputs while producing larger, more effective test sets and more detailed prompts. Participants also reported stronger perceived support for policy refinement than with the baseline interface.
7.1 RQ 1: How do prompt instructions refined by our workflow differ from the baseline?
- Co-Evolution prompts averaged 111.1 words versus 81.2 for baseline; they included explicit constraints for 13 versus 9 participants and examples for 9 versus 2.Policy-statement counts did not differ significantly: 3.31 for Co-Evolution versus 3.00 for baseline.Source: §7.1 · pp. 11–12
7.2 RQ 2: How does the workflow shape the prompt refinement process?
- Test sets were larger with Co-Evolution (14.19 versus 7.94 items), while revision counts were comparable (4.31 versus 3.88 versions).The test-set difference was significant; the revision-count difference was not.Source: Figure 7 · p. 12, §7.2 · p. 12
7.3 RQ 3: Do instruction revisions lead to changes in model behavior?
- Participants induced 4.25 output changes on average with Co-Evolution versus 3.25 with baseline, a nonsignificant count difference; every Co-Evolution participant changed at least one output.Four baseline participants produced no changes.Source: Figure 8 · p. 12, §7.3 · p. 12
7.4 RQ 4: Does the workflow improve alignment between instructions and model behavior?
- Co-Evolution yielded F1-score 0.69 versus 0.56 for baseline, a significant alignment improvement (paired t-test p=0.0241).Additional analyses found no significant task-order or interface-order effects.Source: §7.4 · pp. 12–13, Figure 9 · p. 13
7.5 RQ 5: What are the efficacy and coverage of the test sets created?
- Co-Evolution test sets exposed failures in 51.9% of examples (93/179), versus 26.7% (23/86) for baseline.Coverage was 58% (average 13.8 of 24 categories) versus 49% (average 11.7 categories); the difference partly reflects more generated examples.Source: §7.5 · p. 13
7.6 RQ 6: How do participants perceive the effectiveness of the workflow?
- Participants rated both conditions similarly on basic usability, but more strongly endorsed Co-Evolution for adapting preferences, confidence, long-tail coverage, policy clarity, satisfaction, and reuse.For example, 13 of 16 rated preference adaptation 6 or 7, versus four under baseline.Source: Figure 10 · p. 13, §7.6 · pp. 13–14
8 Discussion
Participants reported that concrete examples helped them articulate nuanced policies, while rapid feedback and side-by-side comparisons supported prompt refinement. The discussion also identifies limits in binary judgments, example diversity, test-set scale, automated judging, and the workflow’s scope.
8.1.1 Enabling Fine-Grained Specification through Concrete Ex-
- Concrete, diverse examples helped participants articulate subtle policy boundaries, especially for ambiguous tasks such as content moderation.Participants said reviewing examples made the abstract work of setting policy more manageable.Source: §8.1.1 · p. 14
8.1.2 Facilitating Nuanced and Rapid Co-evolution.
- Rapid feedback supported prompt iteration, while side-by-side comparisons helped participants assess whether revisions produced desired changes.Participants valued seeing old and new prompt outputs together when refining community guidelines.Source: §8.1.2 · p. 14
8.1.3 Scaffolding Human Judgment with Just-in-Time AI Assistance.
- AI-suggested rationales helped participants verbalize judgments, leaving more attention for nuanced decisions.The authors characterize this just-in-time assistance as a promising direction for human-AI collaboration.Source: §8.1.3 · p. 14
8.2.1 Beyond Binary Judgments: The Need for Handling Ambiguity.
- Binary correct/incorrect labels made ambiguous, borderline cases difficult to judge, despite their importance for refining nuance.The discussion points to uncertainty expression, such as a “Maybe” option, as a possible design direction.Source: §8.2.1 · p. 14
8.2.2 The Challenge of Diverse Example Generation for Creative Exploration.
- Generated examples sometimes lacked diversity, limiting their usefulness for creative edge-case exploration.The authors suggest more sophisticated diversity strategies and user guidance as future directions.Source: §8.2.2 · p. 14
8.2.3 Managing Policy and Test Set Complexity.
- The workflow does not fully resolve policy conflicts in complex applications, and a growing test set creates management and cognitive-load challenges.Suggested future interfaces include clustering related failures, filtering, summarization, and visualizing how policy changes affect subsets.Source: §8.2.3 · p. 15
8.2.4 Navigating the Double-Edged Sword of Automated Judgments.
- LLM-as-judge assistance can mislead users when judgments are wrong and may foster over-reliance even when judgments are often correct.The authors suggest communicating confidence and designing for continued human oversight.Source: §8.2.4 · p. 15
8.2.5 Extending to Open-Ended Generative Tasks.
- The study focused on solo developers and binary content moderation; open-ended tasks may need richer feedback, teams raise collaboration challenges, and model-weight updates are technically harder than prompt edits.The authors identify rubrics or demonstrations, collaborative workflows, and parameter-efficient fine-tuning as future avenues.Source: §8.2.5 · p. 15, §8.2.6 · p. 15, §8.2.7 · p. 15
9 Conclusion
The conclusion presents data-prompt co-evolution as a workflow that treats prompts and test data as jointly evolving artifacts. In a controlled study, it supported more concrete policy refinement, richer test sets, and more consistent alignment than a baseline prompt-editing tool.
9 Conclusion
- Data-prompt co-evolution treats prompt instructions and test data as jointly evolving rather than separate, static resources.The authors describe an interactive workflow evaluated in a controlled user study.Source: §9 · p. 15
- The study found more concrete policy refinement, richer test sets, and more consistent alignment between participant intentions and model behavior than with a baseline prompt-editing tool.Source: §9 · p. 15
- The authors conclude that iterative, evidence-grounded test-set growth can support continuous, human-in-the-loop LLM refinement.They identify larger test sets, collaborative settings, and model-weight updates as future opportunities.Source: §9 · p. 15