Source-linked AI summary
Building a research-software catalog with a coding agent: from hackathon prototype to public deployment
Kazuyoshi Yoshimi, Satoshi Terasaki, Gotai Yamada
TL;DR
The paper examines whether AI-assisted development can help maintain research-software portals as software ecosystems expand. It documents a hackathon catalog, its validation and deployment engineering, and an exploratory retrieval agent for MateriApps. The central result is that reliable publication required explicit validation, monitoring, and repeated review because silent failures could yield plausible but incorrect outputs, while curated metadata and maintained documentation remained necessary.
Problem
As research software and updates multiply, manually maintaining discoverable, current portal information becomes increasingly difficult.
Method
The study reports on a coding-agent-built repository catalog and an exploratory MateriApps retrieval workflow combining curated metadata, external documentation, and language-model generation.
Results
Reliable publication required repeated review and validation because the most consequential defects were silent failures producing plausible yet incomplete or incorrect outputs.
Takeaways & Limitations
Automated retrieval extends rather than removes the need for editorial curation and maintained documentation in research-software portals.
Takeaways & Limitations
The study is an experience report without a controlled comparison between agent-assisted and conventional development, and its MateriApps evaluation uses generated questions and a local-model judge.
Abstract
from arXiv · showhide
Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackathon and subsequently examined the engineering required to make it suitable for public deployment, including adversarial review, data-quality checks, browser-level validation, and publication safeguards. We then explored whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal, through a retrieval agent under development for MateriApps that combines curated portal metadata, external documentation, vector search, and local language-model generation. Implementation with coding agents was rapid, but achieving reliable operation required substantial additional engineering: the most consequential problems were not crashes but silent failures that produced plausible yet incomplete or incorrect outputs, arising from incomplete data acquisition, misleading assessments, and retrieval or preprocessing failures. These observations suggest that AI-assisted software portals require explicit validation, monitoring, and repeated review, and that curated metadata and maintained documentation remain essential. The MateriApps work is exploratory and remains under active development, so the observations reported for it are preliminary; a comparable combination of curated metadata, automatically collected documentation, and retrieval-based assistance may nevertheless be useful for extending other research-software portals.
GRAPHICAL ABSTRACT
The project moved from a three-day, 68-repository hackathon prototype toward a publishable catalog and a larger MateriApps retrieval application. Explicit validation was needed because silent failures could produce plausible but incomplete or incorrect results.
- The system could continue running while producing plausible but incomplete or incorrect output rather than raising crashes.
- The prototype was built during a 3-day hackathon and initially covered 68 repositories.
- Incomplete data acquisition and retrieval or preprocessing failures were identified as central failure modes.
- The lessons were applied to a larger, human-curated portal through a retrieval workflow covering 336 applications and approximately 45,000 passages.
- The resulting deployments were considered publishable only with explicit validation and monitoring.
1. Introduction
The paper asks whether generative AI can help maintain research-software portals as software ecosystems grow, while documenting a prototype and its exploratory transfer to MateriApps. It argues that automation can complement, but not replace, editorial curation and maintained documentation.
- Maintaining portal information becomes increasingly difficult as software packages and updates increase.
- The study asks whether technologies that accelerate software development can reduce recurring portal-maintenance effort.
- The hackathon produced a catalog with 68 automatically collected repository records and a local-language-model interface.
- The MateriApps case extended the investigation to a much larger, more heterogeneous, human-curated documentation corpus.
- Automation can build upon existing editorial work because manually assigned categories and tags remain important for narrowing broad queries.
- The paper identifies silent plausible errors and complementary roles for automated collection, curated metadata, retrieval, and language-model generation.
2. What we built first: the catalog
The first system was a staged repository catalog with independently testable data interfaces, readiness assessments, publication filtering, and exploratory local retrieval. Its simple token-overlap search exposed why retrieval quality matters for capability-specific questions.
- The catalog uses JSON interfaces between four pipeline stages so each stage can be executed, inspected, and tested independently.
- Fetching collects repository metadata, while data building normalizes records, merges curated fields, applies publication policy, and computes readiness.
- The PASUMS deployment reuses harvesting, readiness assessment, publication filtering, and site generation but requires institution-specific configuration and taxonomy.
- The public site supports English and Japanese browsing, purpose-based entry points, category counts, and keyword search.
- Each repository receives a 0–100 readiness score based on seven binary signals and a checklist showing missing artifacts.
- The exploratory question-answering feature was disabled in public deployments and used a locally hosted model after token-overlap retrieval from the public dataset.
- Single-stage token overlap returned broader Monte Carlo software for variational Monte Carlo queries, revealing a limitation for specific capability searches.
3. Making the catalog publishable
Turning the catalog prototype into a public system required an adversarial review process organized around data integrity, reliability, and security rather than feature construction alone. The review issues provide the paper’s framework for examining deployment readiness.
- The coding-agent development history comprised construction and adversarial-review phases, with most later commits addressing numbered issues.
- The review filed 25 issues across eight days from the first commit through the initial development and review cycle.
- The issues were grouped into development process, published-data integrity, and published-site reliability and security.
3.1. Working with the coding agent
Coding-agent implementation was rapid, but reliable development required iterative adversarial review, browser-level verification, and repeated correction. Review and validation exposed assumptions that code inspection alone missed.
- Claude Code implemented features, while Codex performed adversarial review using structured failure scenarios and proposed corrections.
- A second review after corrections found additional data-pipeline and front-end problems, establishing review as an iterative cycle rather than a terminal step.
- Separating implementation and review across agents and fresh contexts exposed assumptions that had not been examined during construction.
- Rapid implementation accounted for only part of the effort, while repeated failure-mode analysis constituted a substantial fraction of the engineering work.
- Browser-level verification checked deployed behavior rather than accepting code inspection alone, including language switching and relative-date calculation.
3.2. Maintaining the integrity of published data
Published-data integrity depended on distinguishing unavailable data from negative evidence and preventing degraded or private information from reaching the public catalog. Validation therefore had to cover acquisition, assessment rules, publication boundaries, and operational safeguards.
- Acquisition failures could silently become negative assessments, causing a successful pipeline run to publish incorrect readiness scores.README failures from rate limiting or server errors were recorded as missing README files.
- Authentication failures could leave partial output valid-looking, overwrite the last known-good dataset, and occupy continuous integration through unbounded requests or waits.
- Strict deployment runs fail when at least five requests are attempted and more than 20% fail, while local runs mark degraded data and may continue.This threshold is an operational choice for the present system, not a general standard.
- The revised operating principles reject silent negative values, stop deployment beyond an explicit degradation threshold, and preserve the last known-good dataset.
- Publication checks combine leak detection, empty-output guards, and generated-site inspection, and public-build mode excludes local-only routes.
- Shared record objects allowed internal operations such as slug deduplication to alter already-validated public records, changing public URLs indirectly.
- Privacy requirements must be verified at the publication boundary, and the deployment checks themselves must be tested and documented.
- Rule-based readiness indicators produced false positives and negatives, so test, license, and citation detection required repository-specific validation and rendered-catalog inspection.
3.3. Maintaining the reliability of the published site
Reliable site publication required testing browser behavior, accessibility, client-side data exposure, and configuration values in addition to source-code inspection. These checks addressed failures that could leave a functioning site incomplete, misleading, or unsafe.
- Browser testing found duplicate identifiers from Japanese labels, incorrect label activation, Safari history-update limits, and temporary blank pages during client-side rendering.
- Source-level correctness did not guarantee compatibility with browser-specific constraints or accessibility requirements.
- Separating filter helpers from catalog data reduced the relevant JavaScript bundle from 455 kB to 16 kB.
- Configuration values inserted into CSS were validated and image paths were percent-encoded, with pure-function checks covered by automated tests.
- Configuration values must be treated as inputs requiring validation rather than as inherently trusted data.
- Reliable publication required browser-level testing, control of client-sent data, and configuration validation because silent incomplete, misleading, or unsafe results were more consequential than crashes.
4. Applying the approach to a larger existing portal
The MateriApps experiment transferred the catalog lessons to a larger, heterogeneous corpus using curated metadata, external documentation, staged retrieval, and vector search. Scaling exposed silent retrieval and corpus-construction failures, so the findings remain preliminary and require continued validation and monitoring.
- The exploratory corpus covered 336 applications and approximately 49,500 indexed passages from portal pages and linked external documentation.The corpus included Japanese and English portal pages.
- Retrieval first uses curated metadata to identify candidate applications, then combines candidate passages with unrestricted multilingual vector search across applications.A cross-encoder re-ranks retrieved material.
- Dictionary routing helps when a question names an application, but contributes nothing when no application name is detected.
- As the corpus grew, default approximate-nearest-neighbor search lost passages; Japanese top-5 hit rate fell from 1.000 to 0.950 and recovered to 0.990 when ef search increased from 100 to 500.
- Crawler duplication and aggressive deduplication could respectively overrepresent one site or remove valid documentation with boilerplate.
- A cleaning rule reduced one package’s 22 installation pages to none and its indexed passages to 17 instead of 183, while aggregate benchmarks failed to reveal the defect.The loss became visible only when a user asked how to install the package.
- The MateriApps observations are early findings from an agent under development, with ongoing work focused on retrieval quality, corpus validation, application coverage, and monitoring.
5. Discussion
Automated retrieval can extend editorial curation, but its usefulness remains bounded by curated terminology, source-corpus completeness, and sustained human oversight. The systems studied also require validation because silent failures can produce plausible but incomplete or misleading outputs.
- Editorial curation: Curated metadata guides retrieval effectively when users’ terminology matches the assigned application names, but contributes little when wording differs.The same pattern appears in the repository catalog, where manually prepared names, taglines, and use cases receive greater retrieval weight than harvested README text.
- Documentation limits: Automated retrieval cannot recover information absent from the source corpus.The examined crawl lacked external documentation for 62 of 336 MateriApps applications.
- Documentation limits: Machine-assisted discovery reinforces established documentation practices, including clear descriptions, licensing, examples, citation guidance, and identifiable releases.These materials support both human understanding and automated processing.
- Scope and limitations: Whether institutional maintenance can preserve accuracy without the original developers’ continuous supervision remains unresolved.PASUMS is an initial institutional deployment, but it has not operated long enough to show whether automation reduces maintenance effort or shifts it toward review.
- Operational safeguards: Long-running services need automated checks for coverage changes, acquisition failures, and degraded outputs because silent failures can persist without execution errors.Routine inspection alone may miss plausible but incomplete or misleading results.
- Scope and limitations: The evidence is limited by small-team development, exploratory MateriApps work, non-gold-standard evaluation, and the absence of a controlled comparison with conventional development.The study cannot determine which observed defects were specific to coding-agent use.
6. Conclusion
The study moved from a rapidly built coding-agent prototype to deployment-oriented validation and then explored transfer to a larger curated portal. It finds that reliable research-software infrastructure depends on repeated validation, editorial metadata, maintained documentation, and explicit safeguards against silent failures.
- 6. Conclusion: Coding agents enabled rapid implementation, but reliable publication required adversarial review, data-quality checks, browser validation, and publication safeguards.Testing the running application, rather than source code alone, was part of the required validation.
- 6. Conclusion: The most consequential defects were silent failures that produced plausible yet incomplete or incorrect outputs.These failures arose despite continued system operation.
- 6. Conclusion: Automated retrieval builds upon rather than removes the need for editorial curation and maintained documentation.Curated metadata helps guide known terminology, while missing documentation cannot be recovered by retrieval architecture alone.
- 6. Conclusion: Generative AI is most useful in research-software infrastructure when it extends existing technical and editorial work.The supported goal is to keep software, metadata, documentation, and automated outputs verifiable, maintainable, and useful.
Data availability
The paper’s data are archived, while live services continue to change beyond the dated snapshots analyzed. The catalog and retrieval-agent source code are not publicly released but are available from the corresponding author on reasonable request.
- Data availability: The generated catalog, its institutional deployment, and MateriApps are publicly accessible live services that do not themselves reproduce the dated analytical snapshots.The analyzed systems continue to change.
- Data availability: The underlying article data are archived in the ISSP Data Repository, including catalog exports, readiness signals, provenance metadata, validation scripts, and the MateriApps corpus.The archive contains the data underlying the reported analyses.
- Data availability: The catalog-generator and retrieval-agent source code, question sets, and per-question retrieval results are not publicly released but are available from the corresponding author on reasonable request.This limits fully public reproduction of the implementation and evaluation workflow.