PaperlayerView exact revision on arXiv

Source-linked AI summary

PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?

Shengchao Chen, Ting Shu, Sufen Ren

arXiv:2610.32485v1cs.CV

TL;DR

Photonic crystal fiber design is a manual, expert-intensive process that requires linking visual structure and mode fields to waveguide physics. PCFBench evaluates VLMs on this perception-to-design loop using a large expert-annotated FDTD dataset and single-model and multi-agent evaluations. The results show a persistent gap between geometry perception and physics-aware design decisions, which does not disappear with scaling or multi-agent coordination.

  • Problem

    PCF design couples visual inspection with waveguide-physics reasoning, but it remains unclear whether VLMs can translate photonic images into precise, physics-aware design actions.

    Source: §1 · pp. 1–2

  • Method

    PCFBench evaluates 26 perception-to-design tasks using over 220K FDTD samples from 18 fiber families, expert annotations, and single-VLM and multi-agent tracks.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    A persistent perception–design gap remains across model scales and coordination strategies: geometry recognition does not reliably translate into physically grounded design decisions.

    Source: §4.3 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    PCFBench provides a testbed for simulation-in-the-loop training, VLM-based agentic design, and solver-coupled decision making.

    Source: §5 · p. 10

    PCFBench is a controlled benchmark for relative geometry-to-design evaluation and complements, rather than replaces, full-fidelity design and fabrication verification.

    Source: E Limitations and Broader Impacts · p. 65

Abstract

from arXiv · show

Photonic crystal fibers (PCFs) underpin critical applications in telecommunications, sensing, and high-power laser delivery, yet their design remains a manual, expert-intensive process that couples geometric intuition with deep waveguide physics. Vision-language models (VLMs) offer a promising path toward automating PCF design, but can they truly reason about photonic physics from microscopy images, or do they merely recognize visual patterns? We introduce PCFBench, the first comprehensive benchmark for evaluating VLMs on the perception, reasoning, and decision steps of the photonic inverse-design loop. PCFBench spans 26 tasks organized into six categories of increasing cognitive demand (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse-Design Decision, and Code Generation), built from over 220K FDTD simulation samples across 18 structurally diverse fiber families. We evaluate both individual VLMs on isolated tasks and multi-agent systems (MAS) on the full sequential pipeline from perception to design. Our results reveal a striking perception-design gap: current models can identify fiber geometry with moderate success, yet consistently fail to translate that visual understanding into physics-aware design actions. Moreover, cascading agents through the pipeline exposes significant error propagation, where upstream perception mistakes compound into downstream design failures. These findings suggest that neither model scaling nor naive agent chaining suffices, and that closing the loop demands explicit physical grounding. PCFBench provides the community with a rigorous, multi-dimensional testbed to measure progress toward AI-assisted photonic engineering. The project is available at https://pcfbench.github.io/.

1 Introduction

PCF design is a manual, expert-intensive loop that links cross-section inspection and mode-field interpretation to geometric changes and electromagnetic simulation. PCFBench addresses the open question of whether VLMs can reason about photonic design from images with a benchmark spanning perception through design.

1 Introduction

  • PCF design requires engineers to infer optical properties from structure and mode-field images, diagnose bottlenecks, propose geometric changes, and validate them with FDTD simulation.Each iteration takes hours of simulation and trial-and-error parameter tuning.Source: §1 · p. 1
  • PCFBench evaluates whether VLMs can handle the precision and physics-awareness required for photonic inverse design, an open question given prior visual-reasoning work in other domains.Prior work has deployed VLMs as agents in scientific workflows, but the photonic-design question remains unexplored.Source: §1 · p. 2
  • Existing VLM benchmarks emphasize natural images or general science, while machine-learning PCF design often uses parameter vectors rather than visual reasoning.No benchmark evaluates the step from seeing a fiber cross-section to proposing a physically sound design modification.Source: §1 · p. 2
  • PCFBench organizes 26 tasks across six evaluation dimensions, progressing from geometry perception through physics understanding and multimodal reasoning to design decisions and code generation.The benchmark uses over 220K FDTD samples across 18 fiber families, with structure and mode-field images and expert annotations.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The benchmark evaluates both individual models and multi-agent systems to analyze inter-stage dependencies and error propagation across the design pipeline.Its dataset pairs FDTD samples from diverse fiber families with expert annotations, and its evaluations reveal a persistent perception–design gap.Source: §1 · p. 3

2 Related Work

Prior photonic inverse-design work includes solver-based optimization, learning-based methods, and foundation-model or agentic approaches, while VLM benchmarks largely target natural images or general scientific reasoning. PCFBench addresses the remaining gap by assessing photonic visual perception, physics reasoning, and action in a sequential design workflow.

Photonic Inverse Design

  • Photonic inverse design spans costly Maxwell-equation optimization, learned property prediction and design, physics-informed networks, and newer foundation-model and agentic approaches.Most approaches operate on structured parameters rather than visual inspection of structures and mode fields.Source: §2 · p. 3

Vision-Language Models

  • VLM benchmarks show spatial and quantitative reasoning on natural-image domains, but transfer to fine-grained, physics-governed photonic patterns remains unclear.PCFBench evaluates photonic geometry perception together with underlying physics reasoning and inverse-design action.Source: §2 · p. 3

Scientific Problem Solving Benchmarks

  • Scientific-agent benchmarks cover discovery, analysis, and simulation, but mostly evaluate isolated tasks and leave error propagation in sequential workflows underexplored.PCFBench evaluates VLMs and multi-agent systems along a photonic perception-to-design pipeline.Source: §2 · p. 3

3 PCFBench

PCFBench combines FDTD simulations and expert involvement to evaluate VLMs across a perception-to-design chain rather than isolated visual tasks. Its corpus, task hierarchy, and evaluation suite cover diverse fibers, expert-annotated design information, and both single-model and multi-agent assessment.

3 PCFBench

  • PCFBench is a physics-grounded benchmark for evaluating VLMs across the photonic inverse-design chain, built through FDTD simulation and sustained expert involvement.Its pipeline curates 18 fiber families, simulates two telecom wavelengths, aggregates metadata, layers expert annotations, verifies them, and defines 26 tasks.Source: §3 · pp. 3–4

3.1 Dataset Construction

  • The 220K-sample corpus pairs geometric parameters and structure images with FDTD mode fields at two telecom wavelengths and four levels of expert annotation.Annotations range from geometry descriptors and physics QA to mechanism hypotheses, actionable suggestions, and executable Meep scripts.Source: §3.1 · p. 5

3.2 Task Taxonomy

  • The task taxonomy comprises 26 tasks organized along a perception–understanding–reasoning–design hierarchy, with structure and mode-field images identified as S and F.The overview figure presents this hierarchy as an increasing-difficulty workflow.Source: Figure 2 · p. 4, Table 1 · p. 5

3.3 Multi-dimensional Evaluation Suite

  • The evaluation suite assesses models individually on tasks and benchmarks multi-agent strategies on a sequential design workflow, with physics-grounded decision targets and expert scoring.Direction and closed-loop ground truth use family-specific expert rules, including non-monotonic regimes; two photonics experts assess correctness and physical plausibility.Source: §3.3 · p. 6

4 Experiments

PCFBench evaluations reveal that geometry perception does not reliably translate into physics-aware design, and that adding scale or agents alone does not close the gap. Results also show limits on real-fiber and propagation tasks, while closed-loop edits work best on simpler, single-axis changes.

4.1 Can Individual VLMs Bridge Perception to Design?

  • Qwen3.5-122B-A10B reaches 35.5% closed-loop success versus 63.8% for Qwen2.5-VL-7B despite leading topology accuracy, exposing weak transfer from perception to design.Source: §4.1 · p. 7

4.3 Analysis and Discussion

  • Scaling from 7B to 235B in Qwen-VL raises topology recognition 25 pp but lowers closed-loop success 47 pp; across open VLMs, geometry improves while inverse-design decisions stay flat or decline.Source: §4.1 · p. 7, §4.3 · p. 8

4.1 Can Individual VLMs Bridge Perception to Design?

  • Geometry-description BERTScore-F1 stays in a 0.27–0.36 band and code syntax and modification completeness nearly saturate, while paired prompts expose majority-class shortcuts in mechanism diagnosis.Source: §4.1 · p. 7

4.2 Does Multi-Agent Coordination Close the Gap?

  • MAD leads on GPT-4o-mini at CL = .731 but falls to .411 on Qwen-VL-Max, where Vanilla reaches .580; EvoMAC shifts from .232 on GPT-4o-mini to .732 on Qwen-VL-Max.Source: §4.2 · p. 7
  • Two-agent AutoGen beats several larger MAS on GPT-4o-mini, while domain decomposition improves PSel but upstream perception errors cascade at rates exceeding 60%.Source: §4.2 · p. 8

4.3 Analysis and Discussion

  • On real SEMs, six models retain their simulated family-classification order (Kendall τ = 1) but lose accuracy; across 600 fibers, no model exceeds .69 on dispersion or .59 on pulse broadening.Source: §4.3 · p. 9, Table 6 · p. 10
  • Geometry tasks cluster but correlate weakly with inverse-design decisions, code generation is largely decoupled, and extended-reasoning models dominate the global ranking, led by Qwen2.5-VL-72B among open-weight models.Source: Figure 7 · p. 9, §4.3 · p. 9

4.4 What Do Models Actually Design?

  • Closed-loop edits improve 9/11 ARF and 10/11 dual-core cases, but nested and quasi-crystalline designs requiring coordinated changes remain unsolved.Three feedback rounds add two to five improved cases, whereas joint two-parameter edits change outcomes by at most one case.Source: Figure 8 · p. 10, §4.4 · p. 10

5 Conclusion

PCFBench evaluates VLMs across photonic inverse-design tasks and reveals a persistent gap between geometry recognition and physically grounded design. Closed-loop results show that success largely reduces to single-axis edits, while multi-parameter geometries remain challenging.

5 Conclusion

  • Accurate geometry recognition does not translate into physically grounded design decisions, even with scaling or multi-agent coordination.Closed-loop re-simulation shows success largely reduces to single-axis edits, while multi-parameter geometries remain challenging.Source: §5 · p. 10
  • PCFBench covers 26 tasks across 220K FDTD-simulated samples from 18 fiber families, evaluating 41 VLMs and 13 multi-agent systems.The tasks span perception through design, and the benchmark is proposed as a testbed for simulation-in-the-loop training, agentic design, and solver-coupled decision making.Source: §5 · p. 10
  • Closed-loop re-simulation indicates that successful design changes largely involve single-axis edits; multi-parameter geometries remain challenging.Source: §5 · p. 10

A Dataset

The dataset section describes PCFBench’s corpus, fiber-family taxonomy, annotation levels, and sample-quality checks. Its examples connect structural and mode-field images with progressively richer annotations.

A.1 Statistics and Composition

  • 220,000 samples span 18 families grouped by guidance mechanism, with distributions reported for pitch, hole diameter, ring count, and confinement loss.The corpus is organized into four guidance-mechanism groups.Source: §A.1 · p. 18, Figure 9 · p. 18, Figure 10 · p. 18

A.2 Family Catalog

  • The family catalog pairs structural exemplars with simulated mode fields and describes each family’s structure and principal application domain.It includes configurations such as single-ring anti-resonant fibers, nested anti-resonant fibers, and hollow-core photonic-bandgap fibers.Source: §A.2 · p. 19, Figure 11 · p. 20

A.3 Annotation Showcase

  • Four annotation levels progress from caption to quantitative, diagnostic, and executable content, illustrated with a single-ring, seven-tube anti-resonant fiber.For that sample, annotations describe geometry and guidance, interpret mode-field evidence, and propose wall-thickness or nested-tube design changes; cross-level consistency is checked.Source: §A.3 · p. 21

A.4 Quality Control

  • Every released sample passes five automated checks before expert review; rejected samples are regenerated rather than corrected post hoc.Regeneration may use resampled parameters, longer simulation time, a finer grid, or a new diagnostic annotation.Source: §A.4 · p. 22, Table 8 · p. 22

A.5 Comparison with Prior Benchmarks

PCFBench differs from representative multimodal, scientific, and agent-oriented benchmarks through simulator-derived ground truth, actionable inverse-design tasks, and native multi-agent evaluation.

A.5 Comparison with Prior Benchmarks

  • PCFBench is the only compared benchmark with ground truth fully from an in-the-loop FDTD simulator; it also offers actionable inverse design and native multi-agent evaluation.PICBench is the other benchmark identified as requiring actionable inverse design; PCFBench includes a four-stage sequential pipeline and 12 MASLab baselines.Source: Table 9 · p. 22, §A.5 · p. 22

A.6 Unified Simulation Configuration

The benchmark holds a shared 2D FDTD configuration across fiber families to make outputs comparable. Its evaluation suite specifies shared call protocols, task prompts, evaluated models, and an expert rubric.

A.6 Unified Simulation Configuration

  • A shared 2D FDTD configuration holds the grid, source, run time, and monitors constant across all 18 families, making loss and mode-field outputs comparable.The loss proxy is flux-based and read from four near-PML monitors; the plasmonic family models metal as a lossy effective medium.Source: §A.6 · p. 23, Table 10 · p. 23
  • Spearman ρ = 0.97: replacing fixed-silica refractive index with a Malitson Sellmeier index leaves model leaderboard ranking essentially unchanged.The sensitivity check uses a stratified 200-sample subset; the reported rankings are therefore robust to the silica dispersion model.Source: Table 11 · p. 23, §A.6 · p. 23

B Evaluation Suite

  • The evaluation suite defines shared model-call protocols, category-specific input-output prompts, evaluated models, and an expert rubric for inverse-design Human Scores.Source: B Evaluation Suite · p. 23

B.1 Shared Evaluation Protocol

This section defines a shared evaluation protocol and tasks spanning geometry perception, physics understanding, text generation, multimodal reasoning, inverse-design decisions, and code generation, with task-specific inputs, answer formats, extraction rules, and metrics. It also describes the evaluated models and human-scoring protocol and reports findings on geometry perception and physics understanding.

B.1 Shared Evaluation Protocol

  • Most tasks use a fixed prompt, decoding configuration, answer extractor, and metric; results use 200 stratified samples per task, except mechanism diagnosis, with full-pool spot checks preserving rankings and scores within 1%.All tasks use temperature T = 0.1 and task-specific output-token budgets; mechanism diagnosis uses 15 reliably annotated samples.Source: §B.1 · pp. 23–24, Table 12 · p. 24

B.2.1 Geometry Perception

  • Geometry Perception tests family and lattice classification, pitch and hole-diameter regression, and cladding-ring counting from structure images.The 17-family classification excludes Kagome; pitch spans 0.8–50 µm and hole diameter 0.2–7.5 µm, with the family supplied for parameter prediction.Source: §B.2.1 · p. 25

B.2.2 Physics Understanding

  • Physics Understanding evaluates loss prediction from mode fields, application and mode classification, and free-text explanation of the light-guidance mechanism.Loss values are clamped at 10^-9 dB/km before log-space scoring, with Log-MAE primary; mechanism explanations use keyword recall.Source: §B.2.2 · pp. 26–27

B.2.3 Text Generation

  • Text Generation assesses geometry descriptions, answers to annotation-derived questions using a mode-field image, and structured reports combining structure and field images.Reports cover geometry, guidance mechanism, performance, and applications; generated text is scored against expert annotations or curated term lists.Source: §B.2.3 · pp. 27–28

B.2.4 Multi-modal Reasoning

  • Multi-modal Reasoning compares wavelength losses, diagnoses loss mechanisms, and generates evidence-based explanations in paired structure-and-field or field-only settings.The paired versus field-only inputs isolate the contribution of the structure image across these three reasoning tasks.Source: §B.2.4 · pp. 28–29

B.2.5 Inverse-Design Decision

  • Inverse-Design Decision tests parameter selection and increase-or-decrease choices for pitch or hole diameter, using family-aware physics rules for directional ground truth.The direction tasks report accuracy and positive-class binary F1, precision, and recall.Source: §B.2.5 · p. 29
  • Closed-loop improvement succeeds only when a proposed parameter change follows the physics-consistent direction and has a relative step between 2% and 50%.Source: §B.2.5 · p. 29

B.2.6 Code Generation

  • Code-generation evaluation covers complete Meep simulation scripts, applying expert parameter changes, and refactoring scripts for error handling, readability, documentation, and performance.Simulation scripts receive a weighted score for syntax, imports, and required components; parameter modifications are checked for syntax and completeness, while refactoring uses five binary checks.Source: §B.2.6 · pp. 30–31

B.3 Evaluated Models

  • The evaluated cohort spans ten vendors and roughly two orders of magnitude in parameter count, with “-T” variants enabling extended test-time deliberation on a shared backbone.Source: §B.3 · p. 31, Table 13 · p. 32

B.4 Human Evaluation Rubric

  • Human evaluation rates proposals on six dimensions using a stratified 120-sample subset, dual expert ratings, and Cohen’s κ to assess reliability.Experts adjudicated 12.5% of items; leaderboard Kendall τ was 0.97 before versus after adjudication, indicating conclusions were insensitive to adjudication.Source: §B.4 · p. 31

C.1 Geometry Perception and Physics Understanding

  • Models diverge more on physics than perception; none accurately recovers pitch or hole diameter, while near-perfect mode classification reflects majority-class collapse.Loss prediction shows the widest proprietary/open-weight gap, linking physics understanding to reasoning depth rather than visual capacity. Tables 16–18 provide per-task metrics across categories.Source: §C.1 · pp. 31–32, C Detailed Per-Category Results · p. 31

C.2 Text Generation and Multi-modal Reasoning

Text-generation similarity scores saturate, while paired-image reasoning is stricter; design and code results distinguish nominal correctness from physically usable outputs.

C.2 Text Generation and Multi-modal Reasoning

  • Geometry-description similarity saturates, making technical-report completeness more discriminative; paired-image reasoning is stricter because it requires evidence from both images.Perfect Field-only Mechanism Diagnosis scores can reflect majority-class prediction, so a large Field-only/Paired gap signals a label shortcut.Source: §C.2 · pp. 32–33

C.3 Inverse-Design Decision and Code Generation

  • Correct design answers do not guarantee workable designs or code: direction accuracy can overstate competence, while closed-loop and human scores measure the perception–design gap.Closed-loop scoring requires a physically reasonable step within family constraints; the Human Score averages expert ratings across six dimensions and two raters.Source: §C.3 · p. 33, Table 14 · p. 33
  • Syntax and completeness are near ceiling for most models, but far fewer scripts run and reproduce the reference physics.Physical fidelity, rather than syntax, remains the open challenge in code generation.Source: §C.3 · p. 34

C.4 Robustness Analyses

This section examines robustness to prompt wording and unfamiliar imagery, extends analyses to pulse propagation, pipeline coordination, and cost, and uses diagnostics and physics-aware rules to locate challenges in physical reasoning. It also presents design rules linking geometry adjustments and parameter selection to fiber type and loss conditions.

C.4 Robustness Analyses

  • Log-MAE provides reliable loss rankings because NRMSE is dominated by a few high-loss samples across targets spanning about 16 orders of magnitude; Self-Consistency gains plateau by N = 5 without closing the perception–design gap.The Self-Consistency sweep uses GPT-4o-mini at T = 0.1; the authors attribute the remaining bottleneck to physical reasoning rather than decoding variance.Source: §C.4 · pp. 34–35, Table 20 · p. 35, Table 21 · p. 36
  • Model rankings survive prompt rewrites, and leaderboard gaps above about .03 exceed paraphrase variation; unfamiliar imagery partly explains perception errors but not incorrect design moves.Families farther from natural images are recognized less accurately (Spearman ρ ≈−.6 for Gemini-3-Flash and GPT-5), while closed-loop success is unrelated to image distance.Source: §C.4 · p. 37, Table 22 · p. 37, Table 23 · p. 37

C.5 Propagation Extension

  • The propagation extension adds two image-grounded binary tasks on 600 solid-core samples: classifying dispersion regime and whether a 100 fs pulse broadens through soliton dynamics or quasi-linear propagation.Ground truth combines an empirical effective-index model with scalar GNLSE propagation at 1550 nm; the labels are balanced, making chance accuracy .500.Source: §C.5 · p. 37

D.1 MAS Baselines and Coordination Patterns

  • The Sequential Pipeline divides inverse design into perception, understanding, reasoning, and design stages, comparing ground-truth-context Oracle and predicted-context Cascaded modes to diagnose error propagation.Its diagnostics report stage accuracies, parameter-selection accuracy, closed-loop success, error cascade, consistency, and design accuracy conditioned on correct perception; stage prompts pass prior-stage answers forward.Source: §D.1 · pp. 38–39, §D.4 · p. 40

D.3 Cost-Performance and Token Budget

  • Higher cost, token use, and stronger generalist models do not yield better designs; inexpensive models define the Pareto frontier, while top rankings reflect geometry and design-decision rather than physics competence.AutoGen stays near the best debate method with fewer tokens, while EvoMAC uses the most tokens and produces the worst designs.Source: §D.3 · p. 40, Figure 12 · p. 40

D.5.1 Rule Structure and Coverage

  • Inverse-design rules depend on fiber family, target parameter, and loss regime: Design Direction uses a 0.01 dB/km threshold, Parameter Selection 0.1 dB/km, and Closed-Loop Improvement requires a 2–50% change.Source: §D.5 · p. 41, §D.5.1 · p. 42

D.5.2 Rule 1: Non-Monotonic Strut Thickness for Anti-Resonant Fibers

  • For anti-resonant fibers, thinning a strut away from a resonance thickness reduces loss, while in the low-loss regime the rule instead increases thickness; enlarging the hollow core also suppresses loss.The strut rule reflects resonance peaks in glass-tube cladding, while core enlargement follows hollow-waveguide loss scaling.Source: §D.5.2 · p. 42

D.5.3 Rule 2: Bandgap-Width Ranking for Hollow-Core PBG Fibers

  • For hollow-core photonic-bandgap fibers at 1550 nm, increasing d/Λ widens the bandgap and increasing pitch shifts it toward longer wavelengths; both moves reduce loss.A robust triangular-lattice bandgap opens for d/Λ ≥0.9.Source: §D.5.3 · pp. 42–43

D.5.4 Rule 3: Monotonic Rule for Solid-Core Index-Guiding Fibers

  • Across thirteen mTIR-guiding fiber families, larger pitch, holes, cores, or ring radii strengthen confinement, while thinner struts and gaps reduce scattering in a monotonic rule.Larger pitch or d/Λ increases Veff and tightens confinement; adding rings is beneficial when loss is excessive because finite-cladding confinement loss decays geometrically with ring count.Source: §D.5.4 · p. 43

D.5.5 Rule 4: Loss-Regime Switch for Parameter Selection

  • Parameter selection switches between fine-tuning near optimal loss and rescue above a confinement-loss threshold, with the first-adjusted parameter depending on fiber family and limiting physics.Near-optimal adjustments use each family's canonical handle; at higher loss, HC-PBG switches from pitch to hole size and multi-core fibers from core diameter to pitch. The selection threshold separates two regimes and is tenfold coarser than the design-direction threshold.Source: §D.5.5 · p. 43

D.5.6 Magnitude Constraint and Complementary Tasks

Closed-loop rules agree with re-simulated outcomes on most samples, with disagreements in non-monotonic regimes excluded from ranking. Successful actions also require a bounded change magnitude.

D.5.6 Magnitude Constraint and Complementary Tasks

  • Closed-loop success requires a 2–50% relative change, so extreme answers fail even when their direction is correct.Free-text tasks separately probe birefringence tuning, mode selection, and dispersion shaping through mechanism reasoning and evidence-based explanation.Source: §D.5.6 · p. 43

D.5.7 Rule Agreement with Re-Simulation

  • 94.1% of expert rules agree with re-simulated outcomes; disagreements in non-monotonic regimes are excluded from the closed-loop ranking.In those regimes, the first-order direction depends on the local loss-band position.Source: §D.5.7 · p. 44
  • Table 27 reports non-monotonic cases, rule conflicts, and re-simulation exclusions for each cluster.Source: Table 27 · p. 44

D.6 Per-Case Inverse-Design Findings

The 20-case sweep compares single-parameter design proposals across fiber families and separates outcomes into success, near-floor regression, and failure requiring coordinated changes.

D.6 Per-Case Inverse-Design Findings

  • Models succeed with monotonic single-parameter edits, regress from near-floor baselines, and fail most severely when improvements require coordinated changes.In the coordinated-change cases, every system regresses by +12 to +30 dB.Source: §D.6 · p. 44, Figure 13 · p. 46, Figure 14 · p. 47
  • In anti-resonant fibers, 9/11 systems improve on the ARF, but all regress on nested anti-resonant and single-ring hollow-core ARF cases.The nested case requires two gap widths to change together; the single-ring failure implicates multi-resonance-band reasoning rather than geometric complexity.Source: §D.6 · p. 44, Figure 13 · p. 46, Figure 14 · p. 47
  • Lower-loss nested anti-resonant and hybrid plasmonic cases each show improvement in only 2/11 systems.The lower-loss nested case partially recovers, while success on the hybrid plasmonic case is described as sporadic and unreliable.Source: §D.6 · p. 44, Figure 15 · p. 48, Figure 16 · p. 49
  • Kagome is the strongest success, with 9/10 systems reaching the numerical floor, while multi-core and suspended cobweb cases remain difficult.Only 1/11 systems marginally improve on multi-core, and none improve on suspended cobweb despite its largest baseline loss.Source: §D.6 · p. 44, Figure 17 · p. 50, Figure 18 · p. 51
  • Near-floor cases 9–12 mostly regress, with 0/11 improvements on PBG, all-solid bandgap, and graded-index multi-ring designs.The solid-core hexagonal case has one anomalous improvement to an alternative near-floor configuration.Source: §D.6 · p. 44, Figure 20 · p. 53, Figure 21 · p. 54
  • Vortex ring-core has no improvements; high-birefringence PM has one, while elliptical core is evenly contested, with 6/11 improving.The vortex ring-core baseline is already near the simulator floor; elliptical-core changes are modest in either direction.Source: §D.6 · p. 45, Figure 24 · p. 57, Figure 25 · p. 58
  • Square/rectangular designs show one improvement and octagonal designs three, while twisted hexagonal and fractal quasi-crystal cases regress uniformly.The best octagonal result is −96 dB; fractal quasi-crystal regressions are attributed to the lack of lattice periodicity to exploit.Source: §D.6 · p. 45, Figure 27 · p. 60, Figure 28 · p. 61

D.6.1 Case Study Prompts and Model Responses

  • Six representative cases share one prompt template and show per-case fiber parameters alongside verbatim proposals from three models.The examples include different single-parameter proposals for Kagome, dual-core, and fractal quasi-crystal fibers.Source: §D.6.1 · pp. 64–65

E Limitations and Broader Impacts

PCFBench is a controlled benchmark for relative evaluation that complements, rather than replaces, full-fidelity design and fabrication verification. Its scope centers on transverse structure-to-field problems, and its broader impacts include reproducible comparisons as well as a risk of overinterpreting results as fabrication readiness.

Limitations

  • PCFBench supports relative, physics-aware evaluation and complements downstream full-fidelity design and fabrication verification.The 18 covered fiber families reflect principal guidance mechanisms in routine practical use.Source: E Limitations and Broader Impacts · p. 65
  • The benchmark covers transverse structure-to-field problems; Appendix C.5 extends this to propagation in index-guiding fibers, with hollow-core propagation identified as a next target..Source: E Limitations and Broader Impacts · p. 66

Broader Impacts

  • Benchmark performance may be overinterpreted as fabrication readiness, so the authors recommend simulation-in-the-loop verification before fabrication.The benchmark releases curated geometric parameters, simulated fields, and expert annotations; the paper characterizes direct misuse risk as low.Source: E Limitations and Broader Impacts · p. 66

Related papers

Loading 2610.32485v1…