Source-linked AI summary
Building Mathematical Software -- Lessons from the Development of GAP
Alexander Hulpke
TL;DR
Mathematical computation often pushes beyond realistic size bounds, making implementation efficiency consequential. Drawing on GAP, the paper discusses software-development needs and considers AI coding systems; it reports that LLMs handle some structured tasks well but are uneven across others.
Problem
Mathematical problems can exceed realistic size bounds, while researchers often push computation as far as feasible, making implementation efficiency important.
Source: §1.2 · p. 3Method
The paper draws on GAP’s development experience to discuss software needs and the potential impact of LLMs on programming.
Source: §1.1 · p. 3, §8 · p. 15Results
LLMs can succeed on clearly structured routines relying on documented information, but their skills vary across tasks.
Source: §8 · p. 13Takeaways & Limitations
Users will still need to describe algorithms, not just tasks, when working with future LLM-assisted programming.
Source: §8 · p. 15LLM-generated code attempts tasks straightforwardly without considering speed, memory use, feasibility, or search-space shortcuts.
Source: §8 · p. 15
Abstract
from arXiv · showhide
Mathematical software has particular needs that are not all covered by the existing literature on software engineering or algorithm design. This paper looks at some of these aspects, in the concrete context of the computer algebra system GAP. It also studies the potential impact AI systems could have on the development process for mathematical software.
1 Robustness and speed as necessities
Mathematical software must serve users beyond its original academic setting, including people with different setups and unanticipated inputs. The GAP experience treats robustness and speed as requirements to build in, using systematic tests and efficient implementations.
1 Robustness and speed as necessities
- A broader audience brings different computer setups and unforeseen inputs, so documentation, installation instructions, and robustness must account for users beyond the authors’ initial expectations.Some users may lack a system administrator and still use the software nontrivially.Source: §1 · p. 2
1.1 Ensuring robustness
- Testing groups of small order exposed base-case bugs, and fixing them made GAP more robust.The sanity check calculated properties including subgroups, character tables, conjugacy classes, and automorphisms for groups up to about order 100.Source: §1.1 · pp. 2–3
- Web-based automated testing made successful tests a requirement for integrating new code, while the test suite combines author-selected examples with past user-reported errors.The user-error collection helps prevent recurrence of the same or related bugs.Source: §1.1 · p. 3
- Assertions check conditions the code assumes, but GAP uses a hierarchy because costly checks can make large collections of higher-level operations infeasible.Assertions can be disabled by default to avoid normal-use runtime costs; more thorough checks can be enabled selectively.Source: §1.1 · p. 3
1.2 Making algorithms run fast
- Because mathematical problems can exceed realistic size bounds and researchers push computation as far as feasible, implementations need both effective representations and real-world efficiency.The paper argues robustness and speed should be considered during system development, not added as final polish.Source: §1.2 · pp. 3–4
- At x = 27, 28, 29, and 30, matrix-product times were 35, 101, 189, and 252 ms as scaling pushed entries beyond the small-integer range.The resulting intermediate values increasingly required long-integer calculations.Source: §1.2 · p. 4
2 Implicit assumptions and general usability
Mathematical software must handle legitimate structures that may fall outside assumptions mathematicians make implicitly, such as rings without a one. Accommodating broader domains can require either case distinctions or a fundamental rethinking of concepts.
2 Implicit assumptions and general usability
- Implicit assumptions about structures, such as rings having a one, can make software unsuitable for researchers studying other legitimate cases.The paper notes that assumptions often left implicit in mathematical writing are harder to leave unstated in software for a broad user community.Source: §2 · p. 5
- Supporting rings without a one may require adding case distinctions or rethinking concepts built around a more restrictive structure.The paper notes that researchers studying such rings also want to use existing software.Source: §2 · p. 5
- Generalizing familiar textbook concepts can reveal subtleties: in the ring of even integers, 6 and 4 have no GCD.The passage connects domain distinctions to the difficulty of anticipating every needed generalization in advance.Source: §2 · p. 5
3 Types
GAP’s types record the system’s current knowledge about objects and primarily select effective algorithms, rather than represent mathematical truth. The section traces how this approach supports method selection while balancing flexibility, efficiency, and practical constraints.
3 Types
- GAP types encode current system knowledge about objects and select task-specific routines rather than formalize mathematical truth.Their information may change as more is learned, but GAP does not generally seek unknown properties just to make a better method available.Source: §3 · pp. 5–6, §3.2 · p. 8
3.1 Using types to solve problems
- Methods are selected from installed routines using bits that record known properties; when several apply, the routine with the most requirements is typically chosen.Only known information is used, and tests for additional properties may be costly enough to require an explicit user request.Source: §3.1 · p. 6
- The type model can leave calculations unavailable for a newly created finitely presented group until its finiteness is established.GAP does not test finiteness automatically because such tests may be costly.Source: §3.1 · p. 6
- For some groups, GAP routes calculations through an isomorphic image using a NiceMonomorphism because other methods are unavailable.The section also notes that finding a better representation and evaluating an isomorphism take time, so GAP typically avoids that search.Source: §3.1 · p. 7
3.2 Types, the computer science view
- GAP combines a flexible, not strictly typed language with more specific internal object types and flags that can support efficient routines.For example, immutability lets lists of vectors retain sorted status for binary search; the paper contrasts this flexibility with Magma’s stricter typing.Source: §3.2 · pp. 7–8
- The authors do not claim soft typing with internal checks is optimal, and changing the design now would require a major rewrite.They describe a possible preprocessor that inserts type conversions as a way to combine flexible user code with fixed-type efficiencies.Source: §3.2 · p. 8
- The GAP-to-C compiler did not deliver the expected speedup because untyped variables still required runtime operation dispatch, while hand-edited C could use typed variables and direct calls.Generated code instead served as guidance for accessing GAP objects and interfacing C functions with the system.Source: §3.2 · p. 8
4 Memory, but not too much
GAP’s automatic fallback to generic methods can consume huge amounts of memory, especially when useful object properties are unknown. A hard allocation limit protects other processes but can interrupt calculations that need more memory.
4 Memory, but not too much
- GAP’s generic fallback can build explicit element lists and rapidly allocate huge amounts of memory when no better method is available or object properties are insufficiently known.This is especially consequential on multiuser systems or computers running other applications.Source: §4 · p. 9
- After jobs inconvenienced other users, GAP adopted a hard memory-allocation limit, but users could then hit the limit during calculations that needed more memory.GAP allocates more than the minimum needed to avoid frequent expansion steps.Source: §4 · p. 9
- The current compromise prompts the user to continue or stop when allocation exceeds the initial limit, treating the calculation as if interrupted.The prompt also provides an opportunity to stop a runaway memory-grabbing job.Source: §4 · p. 9
5 Groups via generators
GAP represents groups through generators, and orbit/stabilizer calculations can produce many Schreier generators for subgroups. Redundancy is difficult to predict, making generator reduction a heuristic trade-off between later orbit cost and the cost of testing redundancy.
5 Groups via generators
- For an orbit of length n with m generators, computing a transversal costs generators times orbit length in edges; there are n·m arrows, n−1 spanning-tree arrows, and n(m−1)+1 dashed arrows.Source: §5 · p. 10
- Schreier generators from dashed edges generate the point stabilizer, but their redundancy varies by case and cannot be predicted generally.For a free group, all are needed; detecting redundancy therefore requires case-by-case work.Source: §5 · p. 10
- With two generators and an orbit of length 1000, the graph yields 1001 dashed edges and potentially 1001 Schreier generators, far from an optimal generating set.Removing identities and duplicates may help, but is not expected to eliminate most redundancy.Source: §5 · p. 10
- The issue arises in iterative orbit/stabilizer calculations and other subgroup construction routes, including preimages under homomorphisms and combinations of smaller subgroups.Sims’s stabilizer-chain approach uses membership tests on Schreier generators, but the large number of tests can itself be expensive.Source: §5 · pp. 10–11
- Redundant subgroup generators can make subsequent orbit algorithms unnecessarily expensive, while testing and reducing them also costs time.GAP heuristically forces reduction when a potentially expensive orbit calculation is expected, rather than reducing every subgroup.Source: §5 · p. 11
6 Heuristics
Algorithm heuristics depend on which mathematical objects and cases are expected, but researchers can disagree about what is typical. GAP’s choices reflect motivating applications and examples, while user feedback can expose cases where an algorithm performs poorly.
6 Heuristics
- Algorithm choices such as testing properties or trading memory for speed depend on the objects researchers expect, and researchers may answer that question differently.The paper notes that leaving the choice to the general user seems equally difficult.Source: §6 · p. 11
- There is no settled choice of assumptions or stopping thresholds for searches such as finding intermediate subgroups or suitable-degree faithful permutation representations.An intermediate subgroup may not exist, and the degree considered good enough for a representation is unspecified.Source: §6 · p. 11
- Heuristics often reflect the applications and standard examples that motivated an implementation, so they may not be optimal for every case.User feedback can reveal relevant cases where an algorithm performs poorly, while constructive feedback can stretch choices based on authors’ interests.Source: §6 · pp. 11–12
7 Transitioning
Long-term software development raises career, succession, and legacy-code challenges beyond writing the software itself. GAP’s longevity depends on its community’s willingness to maintain and adapt existing code.
7 Transitioning
- Academic recognition of useful software remains unresolved, despite initiatives to count it appropriately alongside papers.The author frames this as a community and hiring-and-promotion challenge.Source: §7 · p. 12
- Open-source contributions help bring in developers, but most contributions are short-term and few contributors take on maintenance.GAP has undergone several responsibility handovers, though the author attributes their success partly to luck and shared priorities rather than a deliberate strategy.Source: §7 · p. 12
- For code whose authors no longer maintain it, the author favors rewriting over adoption as the ultimately less painful option.A rewrite can reuse working code while generalizing routines or cleaning up interfaces; AI might make improving documentation and structure easier and adoption more plausible.Source: §7 · p. 12
- System longevity depends at least as much on community willingness to keep existing code relevant as on code quality.The author identifies no good answer to ensuring software work receives appropriate recognition in academic hiring and promotion.Source: §7 · p. 12
8 Vibe-coding GAP?
LLMs can help with structured, well-described GAP tasks and code review, but their uneven skills leave algorithm design, performance, and correctness dependent on human judgment. Examples show both substantial code generation and debugging assistance, alongside limits that make inspection and testing essential.
8 Vibe-coding GAP?
- LLM impact is uneven: they can help substantially on some tasks, but their performance varies across task types.The author argues that this variation reflects uneven skill distribution rather than simply hallucination.Source: §8 · p. 13
- Reviewing and annotating code works well, and can improve existing code retroactively even after its original author has left.Examples include adding comments, documenting functions, renaming variables, and checking for uses of a function, often faster and more completely than a person.Source: §8 · p. 13
- LLMs produce GAP code well from very clear descriptions, including translations from other programming languages.The author expects his Magma-to-GAP translator may become obsolete.Source: §8 · p. 13
- Claude’s subgroup-lattice routine handled groups of order 64 in about two minutes, while delegating the isomorphism test to the Digraphs package.For groups with modular subgroup lattices, results agreed with the literature; the author notes that the code was easy because existing GAP functions did the hard work.Source: §8 · pp. 13–14
- Claude generated and tested an 800-line GAP routine for reading .xlsx files in about 15 minutes by unpacking the archive and parsing its XML.The task required little design ingenuity but depended on file-format knowledge, Unix command syntax, and working through implementation details.Source: §8 · p. 14
- Claude can identify potential bugs, but may misdiagnose them or suggest fixes that mask rather than resolve the underlying problem.In a review of old polynomial code, it flagged 20 places, including one clear error in calculating polynomial-root bounds.Source: §8 · p. 14
- Generated code typically attempts the specified task straightforwardly, without considering efficiency, alternative representations, or ways to reduce search.The author says LLMs treat GAP functions as black boxes and do not design sophisticated new algorithms.Source: §8 · p. 15
- A generated implementation of a published permutation-group algorithm was well structured but sluggish, and an earlier version missed an implicit case that a newer Claude version caught.The author concludes that careful inspection against the original description remains unavoidable, and that humans remain crucial to testing, correctness, and performance.Source: §8 · p. 16
AI disclaimer
The author used Claude for language checking, occasional thesaurus help, and proofreading earlier drafts. The author inspected its suggestions individually and states that the writing and opinions remain entirely their own.
AI disclaimer
- Claude served as a spell and grammar checker, occasional thesaurus, and proofreader of earlier drafts.It indicated language problems for the author to inspect one by one.Source: AI disclaimer · pp. 16–17
- The author inspected Claude’s language-problem indications individually.Source: AI disclaimer · p. 17
- The author states that the paper’s writing and opinions are entirely their own.Source: AI disclaimer · p. 17