The Other Half of Workflow Portability: Evidence-Backed HPC Site Profiles with Agentic Discovery

• Site-specific scheduler, storage, and network decisions still require manual tuning because portable workflows do not provide those values or permissions.Incorrect defaults may be discovered only after queue time and software or data preparation have already been spent.Source: §I · p. 1
• Site knowledge is fragmented across login-node commands, documentation, and job behavior, and incomplete documentation can impede independent verification across facilities, [8].Source: §I · p. 2
• Construction combines login-node probes, typed documentation extraction, and eligible pilot jobs; deterministic checks verify proposed fields before preflight returns a plan or explainable failure.Source: §I · p. 2
• The evaluation builds profiles for three sites and preflights a real workflow packaged as a backpack.It covers 108 profile builds across three sites, four models, three retrieval strategies, and three repetitions.Source: §I · p. 2
• The proposed HPC site profile records how each field was established, with evidence-backed values and unresolved fields left unset for further action.A Stampede3 example uses the documented GPU count when Slurm reports no value, alongside measured scratch storage and pilot-confirmed compute visibility.Source: §I · p. 2, Figure 2 · p. 2

A. What Deployment Needs from a Site

• Manager-worker deployment depends on site-specific filesystems, quotas, scheduler settings, submission options, and network permissions.Source: A. What Deployment Needs from a Site · p. 2

B. Where That Knowledge Lives

• Login-node reports can conflict with enforced policy: Stampede3 reports no GPU resource while its guide lists four GPUs per node, and Anvil's reported unlimited walltime contrasts with a 96-hour documented cap.Source: B. Where That Knowledge Lives · p. 3
• Some constraints, such as outbound internet or storage access from compute nodes, may require pilot jobs to establish them.Figure 3 depicts documentation proposals being checked against evidence, unresolved eligible fields becoming pilot work orders, and reconciliation applying source priorities.Source: B. Where That Knowledge Lives · p. 3, Figure 3 · p. 3

C. The HPC Site Profile

• The profile records each value's provenance and observation time, leaves unresolved fields unset, and retains conflicting evidence for reconciliation.For Stampede3, the documented GPU count is retained over the scheduler's null report.Source: C. The HPC Site Profile · p. 3

A. Overview and Trust Boundary

• Each field has permitted evidence sources and precedence; construction combines login-node measurement, typed documentation extraction, and approved pilot jobs.The documentation agent can propose fields with citations, but rule-based checks accept or discard them.Source: A. Overview and Trust Boundary · p. 3

B. Probing the Login Node

• Reviewed login-node commands record scheduler details, reported resources, and visible storage paths, but cannot establish compute-node visibility or reachability.Measurements seed documentation queries, while measured values such as core counts and time limits are withheld so conflicts can surface.Source: B. Probing the Login Node · p. 3

C. Discovering Documentation

• A bounded agent searches and fetches target-site documentation from allowed domains, can request additional checked queries, and stops at a fixed step bound.Final selection is checked against a closed schema with one correction attempt and a fallback to eligible pages.Source: C. Discovering Documentation · pp. 3–4, Figure 4 · p. 4
• Discovery excludes documentation for sibling systems at the same center through a scope check.Source: C. Discovering Documentation · p. 4

D. Corpus Construction and Retrieval

• The selected pages become a persistent corpus with document provenance, bounded chunks, and sentence- or row-level citation spans.One frozen corpus is shared across retrieval comparisons to isolate retrieval from discovery.Source: D. Corpus Construction and Retrieval · p. 4
• Retrieval compares full-corpus batches, fixed-query BM25, and LLM-expanded BM25 over the same corpus; identical lexical queries return identical chunks.LLM-expanded BM25 adds at most two model-generated query variants per field.Source: D. Corpus Construction and Retrieval · p. 4

E. Typed Extraction and Evidence Validation

• The model proposes typed field candidates tied to stable documentation spans, and validation checks each span's retrieval, site scope, and compatibility with the profile contract.Candidates that still fail after one constrained repair call are discarded; citation validity alone does not establish that a value is correct.Source: E. Typed Extraction and Evidence… · p. 4

F. Pilot Jobs and Reconciliation

• Only unresolved fields with registered pilots receive fixed, reviewed jobs to test observable compute-node behavior.Pilots test storage access, outbound networking, manager connections, or worker-to-worker communication; live submission requires explicit approval.Source: F. Pilot Jobs and Reconciliation · p. 4
• Reconciliation applies predefined source precedence: it records conflicts and selects the highest-precedence value rather than the first value found.If the highest-precedence source is silent, a lower-precedence source may populate the field with its own provenance; unresolved fields remain unset.Source: F. Pilot Jobs and Reconciliation · p. 4

IV. WORKFLOW PREFLIGHT

• A normalized, extensible contract expresses workflow requirements independently of workflow system, with adapters translating native formats into that contract.The implemented Floability Backpacks adapter combines the backpack specification with command-line overrides to recover effective requirements.Source: §IV · p. 4
• Preflight checks the target site's profile schema and freshness; missing or stale profiles invoke construction, while valid profiles proceed to comparison.Users configure when a field becomes stale and needs rechecking.Source: §IV · p. 4
• A workflow exceeding a known limit is blocked, while unresolved required behavior is returned as unknown rather than assumed safe or rejected.If requirements are satisfied, preflight returns selected resources, scheduler arguments, network settings, and any modified system command; otherwise it names issues and remediation.Source: §IV · pp. 4–5, Figure 5 · p. 5
• The structured result determines status and plan without a language model, which may only translate that result into a human-readable explanation.The optional model cannot change the status, selected resources, command, or remediation.Source: §IV · p. 5

V. EVALUATION

• The evaluation asks whether construction fills fields accurately and conservatively, how model and retrieval choices affect quality and cost, and whether preflight catches incompatibilities before submission..Source: §V · p. 5

A. Experimental Setup

• Profiles cover two Slurm sites, Purdue Anvil and TACC Stampede3, and the Notre Dame CRC, which uses HTCondor.Source: A. Experimental Setup · p. 5
• Each site has a hand-built reference profile based on documentation, read-only login measurements, and pilot results; uncertain fields are excluded.Source: A. Experimental Setup · p. 5
• The study crosses four language models with three retrieval modes and repeats each site-model-retrieval configuration three times, yielding 108 profile builds.Discovery returned the same authoritative documentation across 36 preliminary runs, so the resulting corpus and other inputs were reused to isolate extraction choices.Source: A. Experimental Setup · p. 5, Table I · p. 5
• Field accuracy counts both populated values and expected absences; precision, recall, abstention accuracy, and mechanical citation integrity assess additional profile properties.The evaluation also measures source disagreement, extraction tokens, latency, and run-to-run variation.Source: A. Experimental Setup · p. 5

B. RQ1: Profile Construction Quality

• Field accuracy ranges from 0.783 to 0.910 and precision from 0.957 to 0.992; Notre Dame CRC's recall is 0.771, mainly because fields remain unresolved.All 5,592 accepted citations pass the mechanical integrity checks.Source: B. RQ1: Profile Construction Quality · p. 5
• Correct abstention brings field accuracy to 89% for Anvil, 91% for Stampede3, and 78% for Notre Dame CRC, with remaining gaps of 9–22%.Login measurement resolves 30–38% of fields, documentation adds 22–39%, and pilot jobs add 4–10%; the paths are complementary.Source: B. RQ1: Profile Construction Quality · p. 5, Figure 6 · p. 6
• The cited Section V-D case identifies PI storage as a documented field that went unrecovered, and says this missing field is what causes the false failure there.Source: B. RQ1: Profile Construction Quality · p. 6
• Measurement and documentation conflict on up to eight Anvil fields and six Stampede3 fields, with none at Notre Dame CRC.Nearly all conflicts concern partition walltime caps: scheduler reports show no limit while documentation states one, and measurement is more permissive in every case.Source: B. RQ1: Profile Construction Quality · p. 6

C. RQ2: Model and Retrieval Effects

• Field accuracy ranged from 0.841 to 0.875; Gemini 3.6 Flash reached 0.875, while GPT-5.6 Terra reached 0.866 with the fewest tokens and fastest completion.The three strongest models differed by less than one percentage point; GPT-5 mini had lower accuracy, higher latency, and the largest variance.Source: C. RQ2: Model and Retrieval Effects · p. 6
• BM25 achieved the highest retrieval-level accuracy, 0.868, using 57,590 tokens; full-corpus retrieval used 2.18 times the tokens and 2.43 times the latency yet scored lower.LLM-expanded BM25 used nearly the same tokens as BM25 but had slightly lower accuracy, higher latency, and greater variance.Source: Table III · p. 6, C. RQ2: Model and Retrieval Effects · p. 6
• Figure 7 shows Terra with BM25 as the lowest-cost high-accuracy configuration and Flash with BM25 as the most accurate; both lie on the observed Pareto frontier, while every full-corpus point falls below and to its right.Source: C. RQ2: Model and Retrieval Effects · p. 6

D. RQ3: Preflight Case Study on a Real Workflow

• On Anvil, preflight matched the request to the GPU partition, selected scratch storage for the 108 GB stage, and produced a profile-grounded Floability command.The partition provides 128 CPU cores and four GPUs per node, and the command included required options and pilot-verified manager ports.Source: D. RQ3: Preflight Case Study on a Real… · p. 6
• On Stampede3, preflight blocked the request before submission because GPU partitions provide only 96 CPU cores per node, below the requested 112.Those partitions provide enough GPUs and memory, but not the requested CPU cores.Source: D. RQ3: Preflight Case Study on a Real… · p. 6
• On CRC, missing per-PI storage information caused a false failure: the profile exposed only home storage, which lacked space for the 108 GB stage.The site could have run the workflow; the failure was in the profile, not the preflight logic.Source: D. RQ3: Preflight Case Study on a Real… · p. 6

E. Qualitative Observations and Discussion

• Pilot jobs contribute the fewest fields but establish compute-node behaviors that neither login-node measurement nor documentation can resolve.The authors use this to motivate combining evidence paths and applying deterministic verification before fields enter the profile.Source: E. Qualitative Observations and… · p. 6
• Figure 7 compares field accuracy with provider-reported tokens and extraction latency; color marks model, shape marks retrieval, and the dashed line marks the observed Pareto frontier.The accuracy axis is truncated to expose differences.Source: Figure 7 · p. 7
• Standard BM25 with fixed queries achieved the highest retrieval-level accuracy and lowest cost, while full-corpus retrieval and query expansion did not improve the verified profile.The authors suggest that irrelevant or competing context may obscure evidence for a closed, typed schema; broader retrieval may help open-ended profile construction.Source: E. Qualitative Observations and… · p. 7
• The CRC case shows that conservative preflight can reject a feasible deployment when important site knowledge, such as per-PI storage, is missing from the profile.The authors propose using deployment outcomes to guide later measurements, documentation searches, and pilot jobs.Source: E. Qualitative Observations and… · p. 7

VI. RELATED WORK

• Package managers, containers, workflow standards, submission APIs, and backpacks address portability but still leave users to supply site-specific submission values.The site profile is proposed to fill that remaining gap.Source: §VI · p. 7
• Compared with Pegasus Site Catalog and GLUE, the profile includes additional deployment requirements, evidence and provenance for each field, and reconciliation across measurement, documentation, and pilot sources.Examples of additional requirements include network reachability and node-local storage survival.Source: §VI · p. 7
• HPC documentation-answering systems produce answers or scripts for individual requests, whereas this work creates a reusable verified artifact and checks documentation against the running system.The extraction process also verifies citations against the frozen corpus and reconciles accepted fields with measurement and pilot evidence.Source: §VI · p. 7

VII. CONCLUSIONS AND FUTURE WORK

• The conclusion reports that profiles supported site-specific preflight, rejected an incompatible resource request, and also exposed a false failure caused by missing profile evidence.After verification, model choice had little effect, while standard BM25 achieved comparable quality at lower cost than more expensive retrieval strategies.Source: §VII · p. 8
• Preflight was evaluated only with a Floability Backpack, and the current profile cannot guarantee coverage of all information needed by other workflow systems.The authors plan to extend schema, measurement, and pilot coverage to additional workflow systems and scheduler environments.Source: §VII · p. 8