Source-linked AI summary
Technical Challenges in Maintaining Tax Prep Software with Large Language Models
Sina Gogani-Khiabani, Varsha Dewangan, Nina Olson, Ashutosh Trivedi, Saeid Tizpaz-Niari
TL;DR
Tax-law changes make tax-preparation software difficult to maintain, and the paper asks whether LLMs can assist with updates. It studies candidate-code ranking and reports that combining CodeBertScore with MajorityVote outperforms either metric alone, while noting that generated candidates can still contain subtle errors.
Problem
Tax-preparation software updates are time-consuming and error-prone, and the paper asks whether LLMs can help developers update implementations of tax law.
Source: §1 · pp. 1–2Method
The paper studies ranking LLM-generated tax-code candidates and situates this work in a framework combining metamorphic specifications, testing and debugging, and LLM code generation.
Source: §1 · p. 2, §7 · p. 16Results
CodeBertScore combined with MajorityVote outperformed either metric alone; without prior-year code, GPT-3.5 top-ranked candidates achieved 0%-2% accuracy versus GPT4.0 candidates at 43%-100%.
Source: §1 · p. 3Takeaways & Limitations
The authors’ framework brings formal methods, automated testing and debugging, and LLM code generation together to support maintaining tax-preparation software.
Source: §7 · p. 16High MajorityVote scores can coexist with mismatches to the ground truth, so comprehensive testing is needed to detect subtle errors.
Source: §5.2 · p. 15
Abstract
from arXiv · showhide
As the US tax law evolves to adapt to ever-changing politico-economic realities, tax preparation software plays a significant role in helping taxpayers navigate these complexities. The dynamic nature of tax regulations poses a significant challenge to accurately and timely maintaining tax software artifacts. The state-of-the-art in maintaining tax prep software is time-consuming and error-prone as it involves manual code analysis combined with an expert interpretation of tax law amendments. We posit that the rigor and formality of tax amendment language, as expressed in IRS publications, makes it amenable to automatic translation to executable specifications (code). Our research efforts focus on identifying, understanding, and tackling technical challenges in leveraging Large Language Models (LLMs), such as ChatGPT and Llama, to faithfully extract code differentials from IRS publications and automatically integrate them with the prior version of the code to automate tax prep software maintenance.
1 Introduction
Tax-law changes make maintaining tax-preparation software difficult and error-prone, while the formal language of IRS amendments may support LLM-generated code. The paper studies candidate ranking and reports results across models, prior-year-code contexts, and ranking metrics.
1 Introduction
- Changing tax regulations make manual software maintenance time-consuming and error-prone, motivating the question of whether LLMs can help developers update tax-law implementations.The introduction notes that tax preparation software is increasingly used by taxpayers.Source: §1 · pp. 1–2
- The authors argue that the formal language of tax amendments in IRS publications may be translated automatically into executable code using LLMs.Their work explores opportunities and challenges in using LLMs to maintain tax-preparation software as tax law changes.Source: §1 · p. 2
- The paper focuses on ranking the varied candidate implementations that LLMs generate, since identifying the fittest candidate is a key challenge for correct tax software.The study generates 10 implementations for tax-bracket, deduction, and EITC calculations from tax-year 2021 descriptions, with and without 2020 reference code; existing ranking metrics often fail to put lower-error candidates first.Source: §1 · p. 2
- CodeBertScore combined with MajorityVote outperformed either metric alone when ranking generated tax-code implementations.MajorityVote ranks candidates by how many other candidates they agree with.Source: §1 · p. 3
- Without prior-year code, top-ranked GPT-3.5 candidates achieved 0%-2% accuracy, versus 43%-100% for GPT4.0; with prior-year code, accuracy was 21%-100% and 48%-100%, respectively.Under δ=10%, GPT3.0 without prior-year code context barely achieved 10% accuracy; GPT4.0 without context reached 100% when δ was at least 7%.Source: §1 · p. 3
- The authors share prompting practices and plan to validate and refine top-ranked code using metamorphic specifications and testing, with targeted feedback prompts for failures.The stated next phase is to assess how far LLMs can update and maintain tax-preparation software automatically.Source: §1 · p. 3
2 Maintainability Challenges in Tax Prep Software
Tax-software correctness is difficult to test because labeled individual tax returns are hard to obtain and determining a correct return requires expertise. The authors use metamorphic relations and TenForty to express compliance requirements, generate tests, and identify failure circumstances.
Challenges
- Prior work used metamorphic testing to find that open-source tax software had not correctly incorporated new tax legislation, underscoring the need for automated update methods.The cited work is [18].Source: §2 · p. 3
- The oracle problem makes correctness difficult to establish: labeled optimal tax returns are hard to obtain, and judging an individual decision is costly and error-prone.The passage attributes these difficulties to privacy and legal concerns around labeled data and the subjective, expertise-dependent nature of tax decisions.Source: §2 · p. 4
Metamorphic Relations
- The authors express compliance properties as metamorphic relations between an individual and a counterfactual individual.They characterize 33 specifications across five tax domains, including disability credits, EITC, child and education credits, and itemized deductions.Source: §2 · p. 4
TenForty Framework
- TenForty generates tens of thousands of random tests from compliance requirements and uses CART decision trees to explain circumstances associated with software failures.The framework was used to examine OpenTaxSolver across tax years 2018–2021 and domains including disability, credits, and deductions.Source: §2 · pp. 4–5
- TenForty revealed missing eligibility conditions, failures near zero-valued returns, and corner-case failures in updated software.The examples include married people filing separately who are incorrectly treated as eligible for EITC and users explicitly opting for an option in an updated version.Source: §2 · p. 5
3 Overview: Generating Software Code from Tax Code via LLM
The examples show that LLM-generated tax updates can contain syntax or logic errors, even when they resemble reference code. The authors motivate combining semantic similarity with functional checks to rank candidates.
Updating Tax Brackets without Prior Software Code
- Without prior code, the faulty generation leaves TAX BRACKETS incomplete, while the corrected snippet defines TAX BRACKETS and DEDUCTIONS and is syntactically valid.The corrected snippet has a CodeBertScore of 0.871, while its ground truth or validation score is 0.45.Source: §3 · pp. 5–6
Updating Tax Brackets with Prior Software Code
- 0.972 vs 0.977 CodeBertScore: the faulty and correct prior-code updates score closely despite logical errors in the faulty version.The faulty snippet misaligns brackets and rates and may double-count the blindness deduction; the correct snippet avoids these errors.Source: Figure 3 · p. 6, §3 · p. 6
Ranking generated code
- The ranking approach combines CodeBertScore with MajorityVoteScore because semantic similarity alone does not guarantee functional correctness.The combined ranking is intended to distinguish candidate quality and select promising code for further validation and refinement.Source: §3 · pp. 6–7
4 Methodology
The framework uses current IRS policy updates and prior-year tax code to generate and rank updated-code candidates. It proposes metamorphic testing and feedback-driven refinement, while this paper focuses on ranking and leaves integration work for later.
Input and Code Generation
- The LLM receives prior-year tax code and current IRS policy updates, then generates multiple candidates using prompts; the experiments use ChatGPT3.5 and GPT4.0.The authors also experiment without previous-year code.Source: §4 · p. 7
Ranking and Selection
- CodeBertScore measures semantic alignment, MajorityVoteScore compares outputs across random profiles, and their weighted score uses weights of 0.6 and 0.4, respectively.The authors note that reliance on majority voting may be misleading for less capable LLMs such as GPT-3.5.Source: §4 · pp. 7–8
Metamorphic Testing and Refinement
- After ranking, metamorphic testing checks the top candidate; failures guide feedback prompts and iterative refinement, while passing candidates are returned to developers as correct updates.The framework may repeat generation, ranking, testing, and refinement until the candidate passes all metamorphic tests.Source: §4 · p. 8
Scope and Limitations
- This paper focuses on ranking generated code; integrating metamorphic testing as validation and developing feedback-prompt generators remain future work.The authors say prompt generation may require extensive future work.Source: §4 · pp. 8–9
5 Experiments and Results
The experiments assess LLM-generated tax software updates with and without prior-year code and examine how model performance varies across error tolerances and refinement prompts. They also identify complex tax logic and imperfect ground-truth matches as challenges.
5.1 Updating tax prep software without prior code context via LLMs
- Without prior-year code, both GPT-4 and GPT-3.5 have lower MajorityVoteScore and CodeBertScore than with reference code; adding deductions and EITC reduces both models’ accuracy, particularly GPT-3.5.Source: §5.1 · p. 11
- Without prior code, GPT-3.5 barely reaches 10% accuracy at δ=10%, while GPT-4 reaches 100% at δ≥7%, though that tolerance still permits errors.Source: Figure 5 · p. 11, §5.1 · p. 12
5.2 Updating tax prep software with prior code context via LLMs
- Providing prior-year code significantly enhances both models’ tax-software update performance compared with generating code from scratch.The comparison covers the top-ranked candidates in Tables 1 and 2.Source: Table 1 · p. 10, §5.2 · p. 13, Table 2 · p. 14
- With prior code, both models excel in the Brackets Only scenario, but ground-truth matches fall as complexity rises; EITC remains difficult even for GPT-4.High MajorityVoteScore can coexist with lower ground-truth matching, indicating subtle errors and a need for comprehensive testing.Source: §5.2 · p. 15
- At tolerances of 5% or higher, GPT-3.5 matches or exceeds GPT-4.0, while GPT-4.0 remains preferable when precision is paramount.GPT-3.5 does not produce high-quality code as often, but its outputs can be more robust to larger error margins when guided by prior code.Source: §5.2 · p. 15
- The ranking system identifies promising candidates, especially GPT-4.0 generations, for refinement into fully correct implementations.This holds even when generated code is not perfectly accurate.Source: §5.2 · pp. 15–16
- Scatter plots across error margins show generated-code robustness and the need for validation and feedback prompts to reach the desired accuracy.Source: §5.2 · p. 16
6 Discussion
Since the initial study on ranking code, the work has shifted toward automating tax-software updates. Preliminary results suggest smaller LLMs can update and extend Python code accurately, while broader testing is underway.
6 Discussion
- Preliminary results indicate that smaller LLMs can accurately update and extend Python code, including changes to tax policy.The new framework is being tested with symbolic execution across more LLMs and complex tax scenarios.Source: §6 · p. 16
- The initial study, conducted in Fall 2023 and Spring 2024, focused on ranking code, primarily in C.Source: §6 · p. 16
- The ongoing work is expected to help tax-preparation developers update code as tax law changes each year.Source: §6 · p. 16
7 Conclusion
The paper aims to develop principled tools for maintaining tax-preparation software as laws change. Its proposed framework combines formal methods, software engineering, and LLM code generation to support compliance and maintainability.
7 Conclusion
- The proposed framework combines metamorphic specifications, automated testing and debugging, and LLM code generation to support tax-software maintenance.Source: §7 · p. 16
- The framework aims to reduce update effort, improve tax-calculation accuracy, and strengthen software reliability and user trust.Source: §7 · p. 16
- The project aims to help programmers respond to legislative changes and meet taxpayers’ needs efficiently.Source: §7 · p. 16