Source-linked AI summary
A Multi-Domain Study on the Effectiveness of LLMs at Writing Property-Based Tests
M M Abid Naziri, Feiran, Qin, Khush Patel, Marcelo d'Amorim
TL;DR
LLM-based PBT writers had been evaluated only on Python libraries, leaving their generalization to programs with language-specific text inputs open. This study compares two writers across ten programs in five domains. Agentic-PBT often covers more branches than Proptest-AI, while mature fuzzers lead on some targets and maintainers confirm bugs across all five domains.
Problem
LLM-based PBT writers had been evaluated only on Python libraries, leaving open how well they generalize to programs that accept language-specific text inputs and run through a process or server.
Source: §1 · p. 2Method
The study compares Proptest-AI and Agentic-PBT on ten programs in five domains, using domain harnesses where needed and measuring coverage, failures, test properties, oracles, bugs, and comparisons with domain-specific tools.
Source: §1 · p. 2, §3.1 · p. 6, §3.2 · p. 8Results
Agentic-PBT covered more branches than Proptest-AI on seven of ten programs with Opus 5 and all ten with GLM-5.3-Flash; maintainers confirmed 65 reports in all five domains.
Source: §1 · pp. 2–3Takeaways & Limitations
Low observability hurts PBT writers, while mature fuzzers and LLM-written properties offer complementary ways to test whether features behave as documented.
Source: §1 · p. 3, §7 · p. 19The study covers two programs in each of five domains, two writers, and two LLMs, so its results may not hold for other programs, domains, writers, or models.
Source: §5.2 · p. 18
Abstract
from arXiv · showhide
A property-based test (PBT) checks a property of a program on many generated inputs, but writing one takes a property, a generator of valid inputs, and an oracle. LLM-based PBT writers produce all three, yet they have been evaluated only on Python libraries. We study how two of them, Proptest-AI and Agentic-PBT, generalize to ten programs in five domains: deep learning libraries, databases, system software, compilers, and LLM serving engines. Most of these programs take text in their own language, such as queries, configurations, and programs. Agentic-PBT covers more branches than Proptest-AI on seven of the ten programs with Opus 5 and on all ten with GLM-5.3-Flash. Domain-specific fuzzers cover more of the compilers and of PostgreSQL and less of PyTorch than Agentic-PBT. We filed 180 bug reports, 44 of which duplicate reports that others filed after the models' knowledge cutoffs; maintainers have confirmed 65 of the others, in all five domains, and fixed 23. We draw lessons for building and evaluating such writers: low observability hurts them, and mature fuzzers paired with LLM-written properties could check that each feature behaves as documented.
1 Introduction
LLM-based property-based test writers had been evaluated only on Python libraries, leaving their performance on programs with language-specific text inputs untested. This study evaluates two writers across five domains and finds that Agentic-PBT generalizes broadly, while exposing coverage and observability limits.
Introduction
- Prior evaluations covered Python libraries, leaving open whether writers can test programs whose valid inputs are text in the program’s own language and whose execution requires a process or server.For these programs, generators must produce semantically valid inputs and oracles must predict results or compare against another computation.Source: §1 · pp. 1–2
Our Work
- The study evaluates Proptest-AI and Agentic-PBT on ten programs across deep learning, databases, system software, compilers, and LLM serving engines.It supplies a run-and-return-output harness for programs reached through a process or server, while the LLM writes generators, properties, and oracles.Source: §1 · p. 2
- The study measures branch coverage and failures, examines Agentic-PBT’s properties and oracles, counts confirmed bugs, and compares it with domain-specific tools.A previously reported bug counts only if its first public report postdates the discovering model’s knowledge cutoff.Source: §1 · p. 2
Results
- Agentic-PBT covered more branches than Proptest-AI on seven of ten programs with Opus 5 and all ten with GLM-5.3-Flash; domain-specific tools led on compilers and PostgreSQL, while Agentic-PBT led on PyTorch.MetaMut and GrayC found 23 and 4 compiler crashes, respectively, versus 4 for Agentic-PBT, with none in common.Source: §1 · p. 2
- Maintainers confirmed 65 of the 180 bug reports in all five domains, and fixed 23; 44 reports duplicated reports filed after the models’ knowledge cutoffs.The other 136 reports included the 65 confirmations, 23 fixes, and 3 rejections; the rest awaited a response.Source: §1 · pp. 2–3
Key Lessons
- Low observability makes documented behavior harder to test, while mature fuzzers and LLM-based writers contribute different strengths.Fuzzers tend to check crashes or fixed relations; documentation-derived properties can detect wrong results, motivating future work pairing fuzzers with LLM-written properties.Source: §1 · p. 3
Our Work
- The study contributes lessons for building and evaluating writers, 65 maintainer-confirmed bugs, and an artifact containing materials to repeat the study with other models and writers.Source: §1 · p. 3
2 Background
Property-based testing checks a program property on generated inputs using a generator and an oracle. The section then introduces two LLM-based test writers, which differ in how they obtain properties and whether they execute and revise tests.
2.1 Property-Based Testing
- A property-based test samples valid inputs with a generator and uses an oracle to check the program’s output against a property; a rejected input is a counterexample.The property is approximated over sampled inputs because the input space is usually too large to enumerate.Source: §2.1 · p. 3
- Hypothesis strategies generate inputs for a test function, while its runner executes the test repeatedly and shrinks a failing input.The quicksort example checks that each output is ordered, but this partial oracle would miss a sort that drops elements.Source: §2.1 · pp. 3–4
2.2 LLM-Based PBT Writers
- Proptest-AI prompts an LLM with one Python API method’s documentation, either requesting properties and tests together or in two sequential prompts.In the two-prompt mode, the first prompt requests output properties and the second asks for a Hypothesis test suite to check them.Source: §2.2 · p. 4
- Agentic-PBT directs a coding agent to inspect source code and documentation, propose properties, write and run tests, triage failures, and report remaining failures.When it judges a failure a false alarm, it revises the test, strategy, or property and reruns it.Source: §2.2 · pp. 4–5
- Unlike Proptest-AI, Agentic-PBT runs its tests and revises those it judges to be false alarms; it also searches code and callers for properties rather than relying only on supplied documentation.Source: §2.2 · p. 5
3 Testing the Domains with PBTs
The section distinguishes domains by input type and how tests reach and observe programs, then illustrates Agentic-PBT tests across five domains. These examples show how tests build valid inputs and check documented behavior, including through custom generators and different oracle designs.
3.1 Test Characteristics
- Domains group programs by input kind and how tests control and observe them; tests also vary by level, input language, harness, relational constraints, and oracle family.API-level tests call functions directly, while system-level tests run programs or send input to running ones.Source: §3 · p. 5, §3.1 · pp. 5–6
3.2 Studied Domains
- The study selects five popular domains that vary in test level, input language, and harness, illustrating tests for query, crontab, and C++ inputs.The three shown tests use custom generators for documented constraints and fail on the program.Source: §3.2 · p. 6
- Deep-learning tests can generate valid device groups and compare tensor results with NumPy, while serving-engine tests compare responses across runs, modes, or weights.The serving properties include response independence from other requests and from streaming mode.Source: §3.2 · pp. 6, 9
- PostgreSQL’s nth_value test generates mutually constrained frame ends, then checks its result against array_agg; tied rows expose a failure.The test discards queries the server rejects, so it cannot detect wrongful rejection of a valid query.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
- The cronie test generates bounded crontab fields and compares cronnext’s schedule with a Python model; it finds an extra Monday midnight for */2.The test runs cronnext through a harness on a generated crontab line.Source: Figure 4 · p. 7, §3.2 · p. 8
- Clang’s alignment-builtin test generates defined-behavior C++ inputs and checks compile-time results against run-time results and a Python calculation.It discards programs that fail to compile and draws alignments as powers of two.Source: Figure 5 · p. 8, §3.2 · pp. 8–9
4 Evaluation
The evaluation compares two LLM-based PBT writers across ten programs in five domains, then characterizes Agentic-PBT’s tests, confirmed bugs, and performance against domain-specific tools. Agentic-PBT generally covers more branches, while the other findings show both the reach and boundaries of its tests.
4.1 RQ1: Performance of PBT Writers
- On all ten targets with GLM-5.3-Flash and seven with Opus 5, Agentic-PBT covers more branches than Proptest-AI; Proptest-AI leads by 5–9% on GCC, Clang, and cronie.With Opus 5, Agentic-PBT writes about a third as many tests as Proptest-AI at similar cost; sampled true-positive rates were 30% versus 13%, but the small samples cannot confirm a difference.Source: Figure 6 · p. 11, §4.1 · pp. 11–12, Table 5 · p. 12
4.2 RQ2: Characteristics of the Property-Based Tests
- Documentation states 78% of Agentic-PBT’s properties, and 54% of its primary oracles check a condition on one outcome.Metamorphic and differential oracles together account for 41% of tests; documentation states 56% of LLM-serving properties, where code states 38%.Source: §4.2 · p. 13, Figure 7 · p. 14
4.3 RQ3: Developer-Confirmed Bugs
- Of 180 reported bugs, maintainers confirmed 109, including 65 new bugs, and fixed 23; only 24 reports were crashes, uncaught exceptions, or hangs.The other 156 reports were wrong results detectable through an oracle; new confirmed bugs per focal point ranged from 0.02 in databases to 0.18 in system software. Examples include an ArangoDB string-replacement bug and a confirmed Clang vectorcall crash.Source: §4.3 · pp. 14–15
4.4 RQ4: Comparison with Domain-Specific Tools
- Domain-specific fuzzers cover more of GCC, Clang, and PostgreSQL, where Agentic-PBT reaches 73–85% of the leader’s coverage; Agentic-PBT covers more of the sampled PyTorch APIs.MetaMut and GrayC found 23 and 4 compiler crashes, respectively, versus Agentic-PBT’s 4, with no matching crash signatures; this comparison covers crashes, not oracle-detected bugs.Source: §4.4 · pp. 16–17, Table 7 · p. 16
5 Discussion
The discussion identifies low observability and writers’ handling of failures as evaluation and test-design concerns, and proposes combining fuzzer-generated inputs with LLM-written properties. The study’s validity is bounded by its evaluation procedures and selected programs, writers, and models.
5.1 Lessons Learned
- Low observability makes documented properties hard to check when program outputs do not reveal the behavior they concern, especially in system-level tests.The cronie test uses the 1,000-setting limit to make its interpretation of a crontab line observable; the authors did not try prompts for supplying such knowledge or probes exposing internal decisions.Source: §5.1 · p. 17
- Writer evaluations should include failure reports and record excluded inputs, because a writer may stop generating inputs that exposed a failure.Agentic-PBT reported a failure for torch.fft.ifftshift calls with no dimensions to rearrange, but its final test no longer generated those calls.Source: §5.1 · p. 17
- Fuzzers reach more compiler and PostgreSQL coverage, while LLM-based writers can supply documented properties that check behavior beyond crashes or fixed query relations.The authors suggest that fuzzers could check what each feature should compute using properties mined from documentation and code.Source: §5.1 · p. 17
5.2 Threats to Validity
- Each writer generated tests once per configuration, so repeated fuzzing measured test variability rather than variability in test generation.RQ4 ran each tool only once; disagreements in low-agreement labels and failures discarded by the RQ3 triage agent remain threats.Source: §5.2 · p. 17
- Results may not generalize beyond two programs per domain, two writers, two LLMs, and the selected target versions and focal points.PyTorch’s RQ4 sample included only APIs supported by Centaur and FlashFuzz; pending maintainer reports make confirmed-bug rates lower bounds.Source: §5.2 · p. 18
6 Related Work
Prior tools mined properties from particular artifacts or domains, while other work generated assertions, tests, or fuzz inputs. LLM-based PBT writers combine property, generator, and oracle generation from artifacts that state the property.
LLM-based PBT writers
- The two general-purpose LLM-based PBT writers identified by the authors had both been evaluated only on Python libraries.Related work also includes LLM-written PBTs for validating generated code and coding agents that write generators without oracles.Source: §6 · p. 18
Mining properties and oracles
- Earlier property-mining tools targeted particular sources and domains, including execution traces, Javadoc, existing tests, and deep-learning API relations.Other tools generated assertions or postconditions, whereas an LLM-based PBT writer produces the generator, property, and oracle together from an artifact stating the property.Source: §6 · p. 18
Test generation and fuzzing
- Before LLMs, JQF and Zest, FuzzChick, and RLCheck combined property-based testing with feedback or reinforcement learning, but users supplied the property and generator.LLM-based fuzzing work generates unit tests or input seeds, and such generators can discard tests that reveal bugs.Source: §6 · p. 18
Domain-specific testing
- Domain-specific tools include mature compiler and database testers, deep-learning-library fuzzers, and early tools for LLM serving engines.The study’s RQ4 comparison included five domain-specific tools.Source: §6 · p. 18
7 Conclusion
The study finds that Agentic-PBT transfers across five domains, but coverage and observability remain limits. Its results also motivate pairing fuzzer inputs with LLM-mined properties to check documented behavior.
7 Conclusion
- Agentic-PBT wrote tests across five domains and yielded 65 maintainer-confirmed new bugs; it covered more branches than Proptest-AI on seven of ten programs with Opus 5 and all ten with GLM-5.3-Flash.Most reported bugs were wrong results detected only by an oracle.Source: §7 · p. 18
- Agentic-PBT’s generators reached 73% to 85% of a mature fuzzer’s coverage on compilers and PostgreSQL, while on PyTorch Agentic-PBT covered more than its fuzzers.MetaMut found 23 compiler crashes against Agentic-PBT’s 4, with none in common.Source: §7 · p. 18
- Fuzzer generators steered toward an LLM-mined property’s focal point could reach inputs the writer’s generator misses and check documented behavior.The discussed domain-specific fuzzers check crashes or, for SQLancer, one fixed relation between queries.Source: §7 · p. 19