PaperlayerView exact revision on arXiv

Source-linked AI summary

IOSage: Detecting HPC I/O Bottleneck Patterns to Guide LLM Recommendations

Mahdi Banisharif, Tabassum Mahmud, Ahmad Tarraf, Suren Byna, Ali Jannesari

arXiv:2610.36370v1cs.DCcs.LGcs.PF

TL;DR

HPC teams need I/O expertise to turn Darshan logs into fixes, and a single log gives an LLM no baseline for deciding which patterns are abnormal. IOSage uses a classifier to select patterns and supplies an LLM with cards, log evidence, and configuration files to recommend changes. Applied changes improved wall time in evaluated applications, while the classifier achieved 0.929 Micro-F1 on labeled benchmark test runs.

  • Problem

    Turning Darshan logs into concrete I/O fixes requires scarce expertise, and a single log gives an LLM no baseline for what is abnormal.

    Source: §I · p. 1

  • Method

    A classifier selects registered patterns, then an LLM uses their benchmark-run cards, per-file Darshan evidence, and job configuration files to recommend changes.

    Source: §I · pp. 1–2

  • Results

    IOSage names a helpful change in 5 of 6 evaluated scenarios with one; applied changes yield 4.38× for an openPMD kernel and 1.42× to 1.93× for WRF restarts, while detection reaches 0.929 Micro-F1 on 488 labeled benchmark test runs.

    Source: §I · p. 2

  • Takeaways & Limitations

    On 78 benchmark test runs without a pattern, Claude Sonnet 4 proposes changes for 64 when deciding alone and for 9 within IOSage.

    Source: §I · p. 2

    IOSage detects only seven registered patterns and recommends known changes; adding a pattern requires a new benchmark sweep, change template, and retraining.

    Source: §VI · p. 10

Abstract

from arXiv · show

HPC applications often leave much of their parallel file system's bandwidth unused, and turning their Darshan logs into concrete I/O fixes still requires a scarce I/O expert. Rule-based tools flag counters against thresholds set by experts, and in tools built on large language models (LLMs), such as ION and IOAgent, the LLM decides which bottlenecks a log contains. We present IOSage, in which a classifier trained on many labeled logs, not the LLM, makes that decision, since a single log gives an LLM no baseline for what is abnormal. The LLM receives the I/O bottleneck patterns the classifier detects, the job's configuration files, per-file Darshan evidence, and for each pattern a card built from benchmark training runs, and writes recommended changes. Applied by hand and measured in repeated paired runs, the changes IOSage names speed up median total wall time 4.38x for an openPMD kernel (4.02x on a second storage tier) and 1.42x for hourly WRF restarts. For 5 of the 6 application scenarios with a helpful change, IOSage with Claude Sonnet 4 names it, including the change for 10-minute WRF restarts (up to 1.93x faster); with GPT-4o and Llama 3.1 as well, the classifier and cards name more helpful changes than the LLM alone. On 78 benchmark test runs without a pattern, the LLM deciding alone proposes changes for 64; with the classifier, for 9. The classifier reaches 0.929 Micro-F1 on 488 labeled benchmark test runs (0.916 as the gate; 0.832 on the 158 from configurations and jobs outside training), above Drishti's rules, WisIO, and IOAgent's pipeline without its literature retrieval. On TraceBench's nine expert-labeled traces from production systems, it has the highest precision of the tools compared (0.82), with 2 false positives against 12 for the keyword-mapped LLM diagnoses shipped with TraceBench and 5 for Drishti, on the five issue labels that match registered patterns.

I. INTRODUCTION

HPC teams need I/O expertise to turn Darshan logs into fixes, while existing tools either rely on expert rules or let an LLM decide bottlenecks from a single log. IOSage uses a trained classifier to select patterns before an LLM recommends changes, with measured speedups on evaluated applications.

I. INTRODUCTION

  • Existing tools use expert-set thresholds or rules, predict throughput rather than patterns, or let an LLM diagnose a log without a learned baseline.Recognizing Darshan patterns otherwise requires scarce expertise in counters, Lustre striping, MPI-IO hints, and application code.Source: §I · p. 1
  • The classifier decides which patterns a log shows, and the LLM turns those detections into changes using pattern cards, per-file evidence, and job configuration files.The cards come from benchmark training runs; recommended changes are applied by hand and judged by total wall time over repeated runs.Source: §I · pp. 1–2
  • On 78 benchmark test runs without a pattern, Claude Sonnet 4 proposes changes for 64 when deciding alone and for 9 within IOSage.The classifier selects the patterns and their cards that appear in the LLM prompt.Source: §I · p. 2
  • With benchmark runs labeled by inducing configuration and rule-labeled production logs weighted lower, the classifier reaches 0.929 Micro-F1 on 488 labeled test runs, above the 0.916 gate, and 0.832 on 158 runs from configurations and jobs outside training.Source: §I · p. 2
  • IOSage names a helpful change in 5 of 6 evaluated scenarios with one; applied changes yield 4.38× for an openPMD kernel and 1.42× to 1.93× for WRF restarts.A change proposed by the LLM alone slows Nek5000.Source: §I · p. 2

A. What Darshan Logs Record

Darshan records per-file I/O counters that show how applications issue I/O, but not why a bottleneck occurs. On Polaris, the cleaned log cohort is large, and detected patterns occur in logs that account for most recorded I/O time.

A. What Darshan Logs Record

  • Per-file POSIX counters include operation counts, bytes, access-size histograms, frequent sizes and strides, timestamps, and cumulative read, write, and metadata time.MPI-IO records similar counters and separates independent from collective operations; STDIO lacks histograms and access sizes.Source: A. What Darshan Logs Record · p. 2
  • Darshan counters show I/O behavior, but not its cause, application semantics, storage state, other jobs, or fine-grained timing.IOSage does not use HEATMAP data, so time-resolved behavior such as periodic I/O phases is largely invisible; it treats patterns as counter-visible behavior, not diagnosed causes.Source: A. What Darshan Logs Record · pp. 2–3
  • Fig. 2 measures recorded POSIX and STDIO read, write, and metadata time per process relative to run time across the cleaned Polaris logs.Values above 1 occur in 397 logs because calls overlap.Source: Figure 2 · p. 3

B. Production I/O on Polaris

  • The Polaris cohort contains 1,397,216 anonymized logs, but cleaning leaves 131,151 and excludes STDIO-only logs from the cohort and production training data.Among flagged logs, the median share of rank time spent in I/O is 0.83%, versus 0.025% for other logs.Source: B. Production I/O on Polaris · p. 3

C. Challenges

The paper identifies three challenges: production logs lack ground-truth pattern labels, a single log offers no baseline for abnormality, and a change’s effect requires repeated measurement. IOSage addresses these through a classifier, pattern cards, LLM recommendations, and manual validation.

C. Challenges

  • Production logs have no independent pattern labels, so configuration-labeled benchmark runs provide independent labels while production labels add scale.The configuration labels describe benchmark behavior, and generalization to applications must be tested separately.Source: C. Challenges · p. 3
  • A change’s benefit cannot be established by a citation alone; shared-file-system I/O noise makes a single run with and one without the change insufficient.The paper therefore measures changes over repeated runs.Source: C. Challenges · p. 3

III. DESIGN

  • IOSage detects patterns with a classifier, retrieves a card for each detected pattern, and prompts the LLM with those cards, log evidence, and job configuration files.Recommended changes are applied by hand and evaluated with runs both with and without the change.Source: §III · p. 3

A. Bottleneck Patterns

  • The seven registered patterns are behaviors that known changes can remove, and a log may contain several at once.Patterns are defined by inducing configurations rather than fixed thresholds; detection is advisory, and users decide whether a suggested change fits the application.Source: A. Bottleneck Patterns · p. 3

B. Labels and Training

IOSage combines benchmark runs labeled by inducing configuration with lower-weight, rule-labeled production logs to train its pattern detector. The training runs also provide pattern cards, while the classifier uses counter-derived features to predict registered patterns.

B. Labels and Training

  • Table I covers registered patterns, the benchmark configurations used to induce them, each card’s change template, and when that change does not apply.The cards are built from benchmark training runs.Source: Table I · p. 4, B. Labels and Training · p. 4
  • 689 configuration-labeled benchmark runs span six benchmarks, with metadata intensity labeled from mdtest, parallelism efficiency from mpi4py programs, and every other label from at least two benchmarks.Runs are labeled by inducing configuration, including efficient configurations labeled as having no registered pattern.Source: B. Labels and Training · p. 4
  • Configuration-labeled training runs receive weight 100 and production logs weight 1, so one benchmark run counts as much as 100 production logs.Production labels come from Drishti rules applied to the earliest 70% of the cleaned Polaris logs; configuration labels define the patterns and production labels add scale.Source: B. Labels and Training · pp. 4–5
  • The classifier uses one XGBoost model per pattern, trained on 157 features computed from POSIX, MPI-IO, and STDIO counters.An eighth model predicts no registered pattern; the seven pattern models supply the gate used at inference.Source: B. Labels and Training · p. 5

C. Precision Gate

IOSage uses a thresholded classifier to select which registered I/O patterns and pattern cards enter the LLM prompt. The cards describe patterns and generic fixes; the LLM uses them with log evidence and configuration files to recommend application-specific changes.

C. Precision Gate

  • Each of seven pattern models assigns a probability, and patterns above τ = 0.3 determine which patterns and cards the prompt contains.When none is detected, the prompt says so and the LLM may recommend no change.Source: C. Precision Gate · p. 5

D. Knowledge Base of Pattern Cards

  • Each pattern card summarizes its counter behavior and a generic change template, which the LLM adapts to the application rather than treating as an application-specific fix.Cards are built from configuration-labeled training runs and include counter medians, an inducing I/O call location, and before-and-after code.Source: D. Knowledge Base of Pattern Cards · p. 5

E. Recommendation

  • Application prompts combine detected patterns and cards with per-file Darshan evidence and supplied configuration files to request the smallest correct setting, library, or code change.For the 10-minute WRF restart example, Claude ranks switching serial netCDF to parallel netCDF first, based on the evidence and io_form_restart = 2.Source: E. Recommendation · p. 5

F. Validation by Measured Wall Time

The evaluation tests whether recommendations improve real applications, whether the classifier makes LLM recommendations more precise, and how well the classifier detects patterns against independent labels. Application changes are assessed with repeated paired runs, while benchmark test runs and system details define the evaluation setting.

F. Validation by Measured Wall Time

  • Application changes are evaluated by 11 paired runs with and without each change, randomized in round-robin order and audited for equivalent work and outputs.Speedup is the median of 11 pair ratios; the 3rd-to-9th sorted ratios form a nominal 93.5% interval.Source: F. Validation by Measured Wall Time · p. 5

IV. EVALUATION

  • The evaluation asks whether recommendations name changes that speed up real applications, improve LLM precision, and detect patterns against independent labels.It also applies the classifier to Polaris logs.Source: §IV · pp. 5–6

A. Setup and Metrics

  • The 689 configuration-labeled benchmark runs are split into 201 training and 488 test runs; 158 test runs come from configurations and jobs outside training.Tests cover six benchmarks; the listed setup uses NCSA Delta CPU nodes and specifies the software and storage tiers.Source: A. Setup and Metrics · p. 6

B. Recommendations on Real Applications

Measured application changes improve total wall time in several cases, and IOSage with Claude Sonnet 4 names the helpful change in five of six scenarios. Its recommendations are compared with LLM-only prompts and other tools, with no-change behavior also tested.

B. Recommendations on Real Applications

  • 4.38× speedup for the openPMD kernel, 1.95× for WaComM++, and 1.42× to 1.93× for WRF are reported for measured changes; the WRF 10-minute restart change reaches 1.93× on the main setting.The 10-minute restart change also helps on four nodes and the flash tier, by 1.19× and 1.34×; other measured WRF history changes yield smaller total-wall-time gains.Source: B. Recommendations on Real Applications · p. 6, Table II · p. 6
  • The measured restart write is faster in all six WRF restart series, while repeated testing finds a 1.17× total-wall-time gain for 10-minute restarts and no significant change for hourly restarts.Darshan time inside I/O calls does not fall for the WRF changes because serial netCDF gathers fields to rank 0 through MPI outside those calls.Source: B. Recommendations on Real Applications · p. 6
  • Claude names the helpful change in 5 of 6 application scenarios and ranks 4 first; IOSage names more helpful changes than LLM alone with every tested model.For Claude, the comparison is 5 helpful changes with IOSage against 2 deciding alone; removing per-file evidence reduces first-ranked helpful changes from 4 scenarios to 2.Source: B. Recommendations on Real Applications · p. 7, Table III · p. 7
  • On logs without a detected pattern, IOSage recommends no change for Nek5000, LAMMPS, and GROMACS; a measured shared-file change for Nek5000 is slower at 0.95×.The LAMMPS one-dump-file-per-rank option speeds up its output step 2.22× but shows no significant total-wall-time change.Source: B. Recommendations on Real Applications · p. 7
  • Drishti and IOAgent without retrieval do not name most of the measured WRF and WaComM++ changes, and IOAgent without retrieval makes 42.5 calls per log versus one for IOSage.With Claude Sonnet 4, the reported cost is about $0.55 per log for IOAgent without retrieval versus $0.016 for IOSage with the benchmark prompt.Source: B. Recommendations on Real Applications · p. 7

C. Recommendations on Benchmark Runs

On benchmark runs without a pattern, the classifier sharply reduces how often Claude proposes a change. On runs with patterns, recommendations generally agree with detected patterns, while asserting every pattern instead produces more recommendations and poorer agreement.

C. Recommendations on Benchmark Runs

  • 64 of 78 no-pattern test runs prompt a Claude change when it decides alone, compared with 9 when the classifier selects patterns and cards.The classifier wrongly flags 7 of those 9 runs, all DLIO runs; on the other 2, no pattern is detected.Source: C. Recommendations on Benchmark Runs · pp. 7–8
  • At τ = 0.3, the classifier has 91.8% recall and 0.914 precision across the eight labels on 488 benchmark test runs.The prompt-setting comparison differs only in whether detected patterns and cards are supplied.Source: C. Recommendations on Benchmark Runs · p. 8
  • With the benchmark prompt, 0.895 to 0.906 of each model’s recommendations name a pattern present in the run; when all seven patterns are asserted, Claude’s share falls to 0.203.In that all-pattern setting, Claude proposes changes on all 77 no-pattern runs whose output parses.Source: C. Recommendations on Benchmark Runs · p. 8

D. Detection

IOSage’s classifier outperforms the compared rule-based and LLM tools on benchmark pattern detection, while performance varies with training coverage and pattern. Its counter features also provide insight into the patterns it identifies.

D. Detection

  • 0.929 Micro-F1 on 488 benchmark test runs exceeds Drishti’s 0.364, WisIO’s 0.311, and IOAgent without retrieval’s 0.331.The gate scores 0.916; on the four patterns covered by every tool, IOSage reaches 0.931 Micro-F1 against at most 0.447.Source: D. Detection · p. 8, Table IV · p. 8
  • 0.832 Micro-F1 on 158 runs from configurations and jobs outside training compares with 0.972 on 330 runs sharing a configuration or job with training.Leaving one benchmark out lowers scores on its runs to 0–0.64, depending on the benchmark.Source: D. Detection · pp. 8–9
  • Biquality weighting raises benchmark-only training from 0.909 to 0.929 Micro-F1; a random forest reaches 0.916 and an MLP 0.881 with the same weighting.With one seed, weights from 10 to 500 reach 0.918–0.932; the gate scores 0.911–0.917 for thresholds from 0.15 to 0.5.Source: D. Detection · p. 8, Table IV · p. 8
  • Per-pattern F1 ranges from 0.81 for throughput utilization to 1.00 for file strategy, with counter features varying by pattern.Fig. 5 shows the top counter features; for example, metadata time supports metadata intensity, while sequential-access ratios support access pattern.Source: D. Detection · p. 8, Figure 5 · p. 9
  • Leaving each benchmark out of training lowers performance on its runs, reaching 0 for the mpi4py programs.The reported scores range from 0.26 to 0.64 for HACC-IO, mdtest, h5bench, and IOR; the authors conclude the classifier needs labeled runs of each workload kind.Source: D. Detection · p. 9

E. Real-Application Traces

On expert-labeled production traces, the classifier has the highest precision among the compared tools, though recall and misses vary by pattern. The facility study reports registered-pattern prevalence in cleaned Polaris logs, while related work distinguishes IOSage’s pattern-selection role from other approaches.

E. Real-Application Traces

  • 0.82 precision on nine expert-labeled production traces is highest among the three tools, with 2 false positives versus 12 for LLM diagnoses and 5 for Drishti.The classifier finds 9 true issues; its F1 is 0.67, matching Drishti’s 0.65 within one detection, and its recall is 0.56.Source: E. Real-Application Traces · p. 9

F. Facility Study and Overhead

  • 45.8% of 131,151 cleaned Polaris logs show at least one registered pattern, and those logs contain 97.8% of recorded I/O time summed over ranks.The most common detected patterns are access granularity (30.8%), throughput utilization (20.5%), and access pattern (5.2%).Source: F. Facility Study and Overhead · p. 9

V. RELATED WORK

  • Unlike tools where an LLM chooses the diagnosis, IOSage uses a trained classifier to select patterns for the prompt and has the LLM turn them into changes.Its cards come from benchmark runs, and rule-labeled production logs are added at lower weight; STELLAR instead tunes Lustre parameters by rerunning applications.Source: §V · pp. 9–10

VI. LIMITATIONS

IOSage covers only registered patterns and its generalization and application measurements have defined scope boundaries. The conclusion identifies further work in automation, source mapping, richer features, and facility-specific training.

VI. LIMITATIONS

  • IOSage detects only seven registered patterns; adding one requires a benchmark sweep, a change template, and retraining.It cannot identify unregistered patterns, locate a pattern by phase or rank from whole-log features, or find a slowdown’s root cause inside application code.Source: §VI · p. 10
  • Micro-F1 falls to 0.832 on configurations and jobs outside training and to 0–0.64 on a benchmark left out of training.Training uses two Lustre-based systems; other parallel file systems remain untested, and rank-imbalance detection is not claimed generally.Source: §VI · p. 10
  • Application results cover eight scenarios, six with a helpful change, and measurements from one Lustre file system on Delta at one to four nodes.Changes were applied by hand, two first-ranked source edits were not run, and the study did not measure saved expert effort.Source: §VI · p. 10

VII. CONCLUSION

  • The authors plan to apply recommendations automatically, run remaining recommendations, add source-line mapping and time-resolved features, and retrain on each facility’s logs.They also plan to compare Darshan counters with richer traces such as DFTracer.Source: §VII · p. 10

Related papers

Loading 2610.36370v1…