Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation

1 Introduction

• Regression oracles record current behavior rather than intended specifications, while specification-based Java oracles require semantic reasoning and strict adherence to type constraints and language-specific syntax.Source: §1 · p. 2
• TOGLL depends on EvoSuite scaffolds, inheriting its test-readability and complex-program limitations, requires costly version-specific fine-tuning, and is less robust when method signatures are unavailable.Source: §1 · pp. 2–3
• CANDOR is presented as the first multi-agent framework for Java testing using only off-the-shelf LLMs; Initializer, Planner, Tester, and Inspector iteratively scaffold, plan for coverage, produce, and inspect tests.Source: §1 · p. 3
• Reasoning-LLM Panelists evaluate tentative oracles against description-derived requirements; an Interpreter structures their outputs, and a Curator determines corrections and produces consensus oracles to cross-validate assertions against intended behavior.Source: §1 · p. 3
• CANDOR matches EvoSuite on high-coverage test prefixes, achieves a higher mutation score, and exceeds TOGLL oracle correctness by at least 21.1 percentage points.The paper notes that the TOGLL comparison favors the fine-tuned approach because CANDOR uses off-the-shelf LLMs while TOGLL uses extra data.Source: §1 · p. 4
• Removing Planner lowers line coverage by 0.050, branch coverage by 0.046, and mutation score by 0.070; removing Requirement Engineer and panel discussion lowers oracle correctness by at least 0.007 and 0.067, respectively.Source: §1 · p. 4

• CANDOR uses specialized agents to iteratively generate, validate, and refine test cases without expensive LLM fine-tuning or external tools such as EvoSuite.Source: §1 · p. 4
• A panel discussion-inspired strategy is introduced to mitigate LLM hallucinations during test-oracle generation.Source: §1 · p. 4
• A dual-LLM pipeline extracts concise oracle evaluations from verbose reasoning-model outputs to address overthinking.Source: §1 · p. 4
• On HumanEvalJava and LeetCode, CANDOR achieves comparable line/branch coverage and higher mutation scores than EvoSuite, while oracle accuracy exceeds a SOTA baseline by at least 21.1 percentage points.Source: §1 · p. 4

• The description specifies transforming each list element and returning the sum of the transformed list.It assumes the description covers the input, core logic, and expected output.Source: §2 · pp. 4–5
• The Java source implements the described logic, but its condition is faulty: it uses i%3!=0 where the example identifies an error.Source: §2 · pp. 5–6
• Because faulty or incomplete code can make its behavior an unreliable oracle, the example motivates combining source code and description to align generated tests with intended functionality.The example is adapted from HumanEvalJava and is described as a realistic, challenging test-generation scenario.Source: §2 · pp. 4, 6

3.2 Step II: Test Prefix Generation

• Because the first two steps use source code to generate tests, faulty code can yield incorrect tentative oracles that the Inspector may not flag when it exhibits the same buggy behavior.In the example, the Inspector identifies missing imports but misses the incorrect oracle because it evaluates the test against the faulty implementation.Source: §3 · p. 6, §3.2 · p. 9

3.1 Step I: Initialization

• Initialization iteratively validates and refines an LLM-generated test file until it is syntactically correct and error-free or reaches the maximum attempts, producing a valid but typically low-coverage template.This step is optional when an initial test case is already provided.Source: §3.1 · pp. 6–7

3.2 Step II: Test Prefix Generation

• The Planner identifies coverage gaps, the Tester generates test code, and the Inspector provides execution feedback in an iterative loop that ends at an attempt limit or coverage threshold.The Planner uses uncovered lines and the current test file to propose additional cases.Source: §3.2 · pp. 7–9

3.3 Step III: Oracle Fixing

• Step III extracts natural-language requirements and, when possible, predicate-logic specifications, then uses Panelist–Interpreter pipelines and a Curator to revise test oracles against intended behavior.Panelists assess each test case and correct its oracle when needed; formal specifications may be left empty when formalization is inapplicable or ambiguous.Source: §3.3 · pp. 9–10
• Each Interpreter extracts relevant insights from its Panelist’s thoughts and produces concise, structured evaluations of tentative oracles.Source: §3.3 · p. 10
• Parallel dual-LLM pipelines make independent oracle judgments, which the Curator aggregates to issue a final judgment.In the example, the Curator identifies the tentative oracle as incorrect and revises it to assertEquals(147, SumSquares1.sumSquares(input)).Source: §3.3 · p. 10
• Step III yields a final test file with high code coverage and oracles corrected to match the natural-language description.Source: §3.3 · p. 10

4 Experimental Setup

• CANDOR is evaluated on test-prefix coverage, oracle correctness for correct and faulty code, and the contributions of the Planner, Requirement Engineer, and panel discussion.Faulty scenarios are created by injecting three PiTest mutants per system under test; each experiment is repeated three times.Source: §4 · pp. 10–11

4.1 Datasets

• CANDOR was evaluated on HumanEvalJava [8] and a more challenging LeetCodeJava dataset, using a random sample of 50 problems from medium and hard categories.LeetCode-Medium and LeetCode-Hard CC values were 6.30 and 9.46, respectively, compared with 4.90 for HumanEvalJava.Source: §4.1 · pp. 11–12

4.2 Baselines

• CANDOR is positioned as the first end-to-end multi-agent Java test-generation approach jointly handling test prefixes and specification-based oracles, compared with EvoSuite, LLM-Empirical, TOGLL, and EVO-CANDOR.LLM-Empirical uses a single off-the-shelf LLM; EvoSuite represents test-prefix generation, while TOGLL represents specification-based oracle generation.Source: §4.2 · p. 12
• The TOGLL comparison omits method signatures and uses EvoSuite-generated prefixes, and is described as biased against CANDOR because TOGLL is fine-tuned on external data.The study implements LLM-Empirical with GPT-3.5-Turbo because code-davinci-002 is unavailable.Source: §4.2 · p. 12
• EVO-CANDOR uses the same EvoSuite test prefixes as TOGLL to enable a fair comparison of oracle quality.The passages note that oracles generated for different prefixes are not directly comparable.Source: §4.2 · p. 13

4.3 Evaluation Metrics and Statistical Testing

• Coverage, mutation score, and oracle correctness assess executed code, detected mutants, and accurate assertions, respectively.Coverage and mutation scores are calculated only for test prefixes with regression oracles; PiTest’s standard mutation operators generate mutants, and oracle correctness is judged against the natural-language specification or reference implementation.Source: §4.3 · p. 13
• Experiments are repeated three times and averaged, with CANDOR-baseline differences tested using paired Wilcoxon signed-rank tests at a 0.05 significance level.For RQ2, the study also reports A12 effect size for oracle correctness differences between CANDOR and LLM-Empirical.Source: §4.3 · p. 13

4.4 Implementation

• Experiments run on a Precision 7960 workstation with dual NVIDIA RTX 6000 Ada GPUs; EvoSuite uses its default assertion configuration.The implementation is in Python with LangChain, and PiTest uses its default mutation-operator group.Source: §4.4 · p. 14
• The implementation uses Llama 3.1 70B as the basic LLM and DeepSeek R1 Llama-distilled 70B as the reasoning LLM, with three Panelist + Interpreter pipelines.Initialization and test-prefix generation allow three attempts, and DeepSeek outputs are capped at 2,000 tokens; one to five pipelines were tested, with little improvement beyond three and longer generation time.Source: §4.4 · p. 14

5.1 RQ1 Results: Test Prefix Quality

• Across all datasets, CANDOR and EvoSuite achieve comparable high-quality test prefixes in line and branch coverage, while CANDOR significantly outperforms EvoSuite in mutation scores.CANDOR outperforms EvoSuite by at least 0.049 (0.937 −0.888 on Leetcode-Hard).Source: §5.1 · p. 15

5.2 RQ2 Results: Test Oracle Quality

• On correct code, CANDOR’s oracle correctness is 0.910, 0.930, and 0.894 across the datasets, while EVO-CANDOR beats TOGLL by at least 0.255 (0.857 −0.602) on Leetcode-Hard.EVO-CANDOR is compared with TOGLL because both use the same EvoSuite test prefixes.Source: §5.2 · p. 15, Figure 4 · p. 16

5.3 RQ3 Results: Ablation study

• Removing the Planner significantly reduces test-prefix quality across all datasets, with minimum decreases of 0.050 (0.990 − 0.940), 0.046 (0.949 − 0.903), and 0.070 (0.939 − 0.869) (p-value < 1e−4).Source: §5.3 · p. 17
• Removing the Requirement Engineer lowers oracle correctness by 0.028 (0.910 −0.882), 0.007 (0.930 −0.923), and 0.021 (0.894 −0.873); panel removal also primarily affects oracle correctness.The values are for HumanEvalJava, Leetcode-Medium, and Leetcode-Hard, respectively.Source: §5.3 · p. 18
• Requirement Engineer ablation changes were significant on HumanEvalJava and Leetcode-Hard (p-value < 1e−3), but the Leetcode-Medium decline was not (p-value = 0.17).Its benefits are expected to be greater when natural-language documentation is less clear.Source: §5.3 · p. 18
• Removing the panel discussion substantially and significantly reduces oracle correctness, consistent with its role in generating accurate oracles.Without the panel, LLMs are more prone to uncertainty and hallucination.Source: §5.3 · p. 18
• 0.014 (0.910 −0.896) was the minimum oracle-correctness decrease from replacing panel discussion with majority voting, on HumanEvalJava; decreases were significant across datasets (p-value < 1e-2).Majority voting reduced oracle correctness consistently across all datasets.Source: §5.3 · p. 18
• Panelists disagreed in over 70% of observed discussions, while the Curator cross-checks their reasoning, inputs, and outputs to correct errors rather than relying only on majority voting.This design makes panel discussion more robust than simple majority voting for producing accurate oracles.Source: §5.3 · p. 18
• Ablations confirm the Planner’s essential role in producing high-quality test prefixes and the panel discussion’s role in generating accurate test oracles.The Requirement Engineer generally improves oracle accuracy.Source: §5.3 · p. 18

6.1 Key Insights

• Dividing test planning and Java test generation among specialized agents prevents interference between subtasks and improves CANDOR’s unit-test generation effectiveness.The authors describe cases where an LLM combining planning with Java test generation produced incorrect syntax.Source: §6.1 · p. 19
• Multiple LLMs independently generate oracles, then discuss their reasoning to select an answer based on the most consistent logic.The authors report that incorrect oracles can occur even with clear instructions and that hallucinations are usually inconsistent across models.Source: §6.1 · p. 19
• A dual-LLM pipeline truncates DeepSeek’s reasoning at 2,000 tokens and uses a basic LLM to extract the oracle, addressing verbose outputs that could exceed 10,000 tokens.The authors observed that DeepSeek sometimes identified the correct oracle early but continued reasoning.Source: §6.1 · p. 19

6.2 Threats to validity

• Results may depend on the selected LLMs, and the authors did not exhaustively evaluate alternatives; possible pretraining-data leakage is also noted.The framework uses LLaMA 3.1 70B and DeepSeek R1, selected in preliminary experiments.Source: §6.2 · p. 19
• Because LLM outputs are random, experiments were repeated three times, averaged, and statistically tested; mutation score supplements coverage as an evaluation measure.The authors describe mutation score as a way to evaluate behavioral changes in synthetically modified programs.Source: §6.2 · pp. 19–20
• The evaluation’s external validity is limited to HumanEvalJava and LeetCodeJava, leaving generalization to other datasets, especially programs with complex dependencies, unclear.Project-level bug detection was not included; handling external or user-defined class dependencies is left for future work.Source: §6.2 · pp. 19–20

7.1 Test Prefix Generation

• Traditional test-generation approaches include fuzzing, feedback-directed random generation, symbolic execution, and search-based methods such as EvoSuite, which can achieve high coverage but may produce unreadable tests.The cited discussion attributes readability problems to limited understanding of focal-method semantics.Source: §7.1 · p. 20
• LLM prefix-generation work includes fine-tuned systems such as AthenaTest and prompt-based generation-and-refinement approaches using execution feedback and other contextual information.The approaches incorporate signals such as coverage reports, failure information, mutation testing, and program analysis.Source: §7.1 · pp. 20–21
• CANDOR is presented as a multi-agent prompt-engineering test generator with code coverage comparable to EvoSuite and better mutation rates.The cited contribution describes it as a generic framework with carefully designed agents, including Planner, Tester, and Inspector.Source: §7.1 · p. 21

7.2 Test Oracle Generation

• Specification-based oracle generation is challenging because it requires understanding functional requirements, edge cases, and expected outcomes; regression oracles instead reflect current implementation behavior.The cited discussion says regression oracles can be limited for checking correctness against intended specifications.Source: §7.2 · p. 21
• Prior oracle-generation methods include fine-tuned TOGA and prompt-based LLM-Empirical, while TOGLL combines EvoSuite prefixes with fine-tuned models and requires extensive fine-tuning.CANDOR is distinguished by specialized agents, a panel discussion to reduce hallucination, and a dual-LLM pipeline for extracting structured information from verbose reasoning.Source: §7.2 · p. 21

8 Conclusion

• CANDOR achieves line and branch coverage comparable to EvoSuite while substantially outperforming it in mutation score.The evaluation uses HumanEvalJava and LeetCode.Source: §8 · p. 22
• CANDOR produces specification-based oracles that outperform TOGLL by at least 21.1 percentage points.The conclusion also attributes improved oracle correctness to the Requirement Engineer and panel discussion.Source: §8 · p. 22
• The Planner is identified as important for test-prefix quality, while the Requirement Engineer and panel discussion are identified as important for oracle correctness.The authors plan to extend CANDOR to project-level tests and explore panel strategies to further reduce hallucinations.Source: §8 · p. 22

CANDOR with Alternative LLMs

• Mistral 22B has the lowest effectiveness, with substantially lower line coverage, branch coverage, mutation scores, and oracle correctness than Llama 70B; differences are significant at p < 10^-4.The authors report that Mistral’s differences from Llama 70B are statistically significant across these measures.Source: CANDOR with Alternative LLMs · p. 26
• Llama 70B and CodeLlama 70B produce comparable test-prefix results, and none of their differences is statistically significant.The largest reported prefix-generation gap is 0.09 in mutation score on LeetCode-Medium.Source: CANDOR with Alternative LLMs · p. 26
• For oracle generation, Llama 70B has slightly higher correctness than CodeLlama 70B on LeetCode-Medium (0.011) and LeetCode-Hard (0.025).The supplied results do not state whether these oracle-correctness differences are statistically significant.Source: CANDOR with Alternative LLMs · p. 26