PaperlayerView exact revision on arXiv

Source-linked AI summary

HALO: A Physics-Aware LLM Agent Framework for Nanophotonic Design

Yubo Zhang, Jinlin Xiang, Zijun Zhao, Yang Zhao, Eli Shlizerman, Arka Majumdar

arXiv:2608.28877v1physics.opticscs.MA

TL;DR

Nanophotonic design agents must reliably convert natural-language objectives into simulation-ready designs, execute electromagnetic analysis, and revise from numerical feedback. HALO addresses this with typed specifications, simulation, diagnostics, iterative revision, and optional failure-memory reuse, evaluated on HALO-Bench. Across the benchmark, fixed structured workflows are most token-efficient and avoid observed code- and path-level failures, while autonomous coding can achieve higher success with stronger models; retrieved failures accelerate targeted delayed-success cases.

  • Problem

    It remains unclear whether language models can reliably translate optical objectives into simulation-ready designs, execute electromagnetic analysis, and revise decisions from numerical feedback.

  • Method

    HALO couples language-model planners with typed design specifications, electromagnetic simulation, diagnostic evaluation, iterative revision, and optional reuse of prior failed trajectories.

  • Results

    Across HALO-Bench, fixed structured workflows are most token-efficient and avoid observed code- and path-level failures, while autonomous coding reaches 79.8% macro-average success with Claude Sonnet 4.6.

  • Takeaways & Limitations

    Retrieved failure history reduces iterations to first success and token use on targeted delayed-success cases, clarifying tradeoffs among structured interfaces, autonomous execution, and reusable experience.

  • Takeaways & Limitations

    HALO-Bench primarily covers periodic nanophotonic structures, evaluates each model–configuration–case once, and the memory study measures convergence on targeted delayed-success cases rather than benchmark-wide gains.

Abstract

from arXiv · show

Language models have recently been applied to nanophotonic design, but it remains unclear whether they can reliably translate optical objectives into simulation-ready designs, execute electromagnetic analysis, and revise decisions from numerical feedback. We introduce HALO, a physics-aware framework that couples language-model planners with typed design specifications, electromagnetic simulation, diagnostic evaluation, and optional reuse of prior failure trajectories in an iterative design loop. We further introduce HALO-Bench, a 52-task benchmark spanning lab-derived, paper-derived, and open-ended nanophotonic design tasks under a shared evaluation protocol. We compare three planner configurations: a Fixed Structured Workflow, an Autonomous Structured Agent using the same simulation interface, and an Autonomous Coding Agent that directly writes and executes simulation code. The Fixed Structured Workflow is the most token-efficient and exhibits no observed code- or path-level failures, while autonomous coding can achieve higher task success with stronger models at the cost of additional operational failures. We also study reuse of prior failed trajectories. On targeted multi-round tasks, retrieved failure feedback reduces both iterations to first success and total token use. These results clarify the tradeoffs between explicit interfaces, autonomous execution, and reusable design experience in scientific agents.

1 Department of Electrical & Computer Engineering, University of Washington, Seattle, WA, USA

HALO addresses the bottleneck of converting underspecified optical objectives into simulation-ready nanophotonic designs. It combines structured formulation, electromagnetic evaluation, iterative revision, and benchmarked planner comparisons.

  • Motivation: Nanophotonic objectives often omit geometry, materials, illumination, solver settings, design variables, and evaluation metrics needed for electromagnetic simulation.This makes the upstream conversion from natural-language intent to a simulation-ready problem a substantial bottleneck.
  • HALO Framework: HALO standardizes conversion from natural-language design intent into simulation-ready formulations and supports execution, diagnostics, revision, and failed-trajectory reuse.Design trajectories store prompts, artifacts, simulation outputs, scores, evaluator notes, and failure records for auditability.
  • Benchmark: HALO-Bench spans Lab-Derived Simulation, Paper-Derived, and Open-Ended tasks testing recovery, reconstruction, and iterative improvement.The regimes assess distinct forms of nanophotonic design capability under a shared framework.
  • Study Design: The study compares fixed versus autonomous workflow control and typed interfaces versus self-generated simulation code.It examines how much workflow and interface structure scientific planners need.
  • Contributions: The contributions are HALO, HALO-Bench, and a study of schema-assisted retrieval of failed trajectories for delayed-success cases.The retrieval study evaluates whether prior failure information accelerates convergence.

LLM Agents for Scientific and Optical Design

Prior work applies language models to scientific and optical design, but systems differ in representation, execution control, memory, and evaluation practices.

  • Optical Design: Language models have been applied to multilayer design, nanophotonic generation, metasurface optimization, and tool-assisted simulation.These applications span several optical-design problem types.
  • System Comparison: Representative systems vary in language-model role, execution control, explicit representations, trajectory memory, and benchmark evaluation.Table 1 specifically tracks geometry, material binding, validation, and executable multi-task evaluation.
  • HALO Positioning: HALO evaluates fixed and autonomous planners within a shared executable framework while retaining complete trajectories for failure analysis.It also studies whether prior failed trajectories accelerate later convergence.

Structured Spatial and Physical Reasoning

HALO focuses on jointly consistent spatial and physical reasoning for simulation-ready optical design. Its typed design schema makes key device and simulation decisions explicit before execution.

  • Structured Spatial and Physical Reasoning: Executable optical design requires consistent spatial arrangement, material assignment, solver conditions, and links between design variables and measurable objectives.These capabilities are often evaluated separately in related work.
  • Scope: The scope concerns simulation-ready physical consistency rather than general materials discovery.This boundary narrows the claims about the reasoning capability being evaluated.
  • HALO Workflow: HALO represents each candidate device with a typed, machine-readable schema covering geometry, spatial relations, materials, illumination, solver settings, variables, and objectives.Structured proposals are validated before RCWA execution.
  • Design History: HALO distinguishes within-run trajectory history from retrieval memory drawn from prior design trajectories.Retrieval memory is enabled only in the memory ablation, whereas within-run history is available in default closed-loop experiments.

HALO-Bench

HALO-Bench evaluates nanophotonic design across three task regimes that test recovery, reconstruction, and open-ended improvement. The benchmark includes lab-derived, paper-derived, and relaxed-constraint tasks.

  • Lab-Derived Simulation Tasks: HALO-Bench contains 20 Lab-Derived Simulation Tasks with previously verified workable reference configurations.These provide feasible simulation targets but are not experimentally fabricated or measured devices.
  • Paper-Derived Tasks: The 21 Paper-Derived Tasks ask planners to produce simulation-compatible designs capturing structures and target behaviors from published descriptions.They do not require reproducing every experimental curve, fabrication detail, or reported numerical value.
  • Open-Ended Tasks: The 11 Open-Ended Tasks relax selected numerical or structural constraints while preserving the broad physical objective.They require exploration and iterative improvement rather than exact reconstruction.

Planner Configurations

HALO compares three planner configurations that progressively increase autonomy and simulation-access freedom while sharing core task constraints. Fixed and autonomous structured agents use HALO’s structured interface, whereas the autonomous coding agent directly writes and executes simulation code.

  • Configuration progression: The three configurations increase autonomy from a predefined workflow to autonomous orchestration, with coding additionally removing the predefined simulation-interface constraint.Fixed Structured Workflow uses fixed execution and structured simulation; Autonomous Structured Agent adds workspace-based agentic control; Autonomous Coding Agent also writes and executes simulation code.
  • Fixed Structured Workflow: Fixed Structured Workflow follows a predefined execution path with one planner call per outer iteration and HALO validation plus RCWA simulation tools.It stops after evaluator-confirmed success or continues within the outer-iteration loop.
  • Autonomous Structured Agent: Autonomous Structured Agent can analyze, manipulate workspace files, maintain notes, inspect artifacts, and choose intermediate actions while retaining structured final designs.Formal RCWA simulations still use HALO’s predefined structured simulation interface.
  • Autonomous Coding Agent: Autonomous Coding Agent directly writes, executes, inspects, and revises Python simulation code, increasing implementation freedom but reducing structural auditability.This configuration also introduces code, API, path, artifact, and operational failure modes.

Common Evaluation Protocol

HALO evaluates heterogeneous planner outputs through a shared evaluator-generation protocol and benchmarks three task regimes. It separately studies whether retrieved failed trajectories accelerate later convergence.

  • Shared evaluator protocol: The evaluator uses the same task objective, metric requirements, execution environment, and pass/fail contract across planner configurations.A case-specific executable script converts saved artifacts and simulation outputs into numerical metrics and a score/pass decision, then remains fixed during that run.
  • Shared evaluator protocol: The evaluator-generated script supports comparable scoring despite schema-based and code-based planners producing heterogeneous artifacts.The evaluator inspects the available design artifact and simulation output before generating the case-specific script.
  • Benchmark regimes: HALO-Bench contains Lab-Derived, Paper-Derived, and Open-Ended tasks spanning seven nanophotonic mechanisms and multiple publication venues.The benchmark composition is 20 Lab-Derived, 21 Paper-Derived, and 11 Open-Ended tasks.
  • Evaluation scope: Reported comparisons are system-level evaluations of complete model–configuration pipelines rather than isolated measurements of planner quality.Planner and evaluator use the same underlying model in the evaluator-generation process.
  • Trajectory memory: Retrieval memory is analyzed separately from default within-run history and can provide prior geometry choices, parameter ranges, evaluation signals, and recovery strategies.The ablation compares no retrieved memory, schema-assisted retrieval, and a same-task failure-memory control.

Experimental Setup

The benchmark uses 52 bounded nanophotonic design tasks with a common evaluator, iteration limit, and wall-clock limit. Token efficiency is reported separately because configurations have different internal workloads, while first-step schema results use 80 prompt–schema pairs per model.

  • Benchmark protocol: 52 tasks comprise 20 Lab-Derived Simulation, 21 Paper-Derived, and 11 Open-Ended cases evaluated with at most five iterations and a one-hour wall-clock limit.Runs stop early after confirmed success, and each model–configuration–case is run once.
  • Token accounting: Token efficiency is measured as successful cases per million recorded language-model tokens because configurations differ in tool use, code execution, and debugging workload.This metric is most reliable within the same model family, and planner and evaluator token usage are recorded separately.
  • First-step evaluation: First-step structured-design accuracy is evaluated on 80 prompt–schema pairs per model using human-annotated reference schemas and four reported dimensions.Overall is the unweighted mean of the four dimensions, with bold values marking the best result within each model family.

Structured Design Understanding

The first-step study isolates whether models can formulate simulation-ready structured designs before feedback and revision. More than 93% overall accuracy on perturbed Lab-Derived tasks indicates that later closed-loop failures involve execution and revision as well as initial formulation.

  • Evaluation design: The first-step study evaluates layout representation, numerical binding, material assignment, and sweep-variable specification before validation, simulation, or iterative feedback.The comparison uses paired human-annotated reference schemas and is restricted to Lab-Derived tasks because other regimes may allow multiple compatible formulations.
  • Interpretation: The high first-step accuracy indicates that closed-loop failures cannot be explained solely by inability to produce an initially plausible structured design.Reliability also depends on preserving, executing, and revising decisions from simulation feedback.

Closed-Loop Success and Efficiency

Success depends on both the underlying model and planner configuration, while autonomy does not uniformly improve performance. Fixed structured execution is especially efficient in token usage, whereas autonomous coding can achieve the highest observed success with stronger models.

  • 79.8% macro-average success is the highest observed result, achieved by the Autonomous Coding Agent with Claude Sonnet 4.6.
  • Autonomous Coding Agent success rises from 32.3% with Gemini 2.5 Flash to 68.8% with Gemini 3.1 Pro Preview.
  • With Gemini 2.5 Flash and Gemini 2.5 Pro, Fixed Structured Workflow achieves higher macro-average success than either autonomous configuration.
  • 2.327 successes per million tokens is Fixed Structured Workflow’s Claude Sonnet 4.6 efficiency, versus 0.474 for Autonomous Structured Agent and 0.604 for Autonomous Coding Agent.

Memory Retrieval Ablation

The memory ablation tests whether retrieving prior failed trajectories accelerates convergence on cases that eventually succeed but require multiple feedback rounds. Schema-assisted retrieval substantially improves first-iteration success and reduces token use.

  • Schema-assisted retrieval augments text similarity with schema-level similarity to find physically related failed trajectories despite differing descriptions or ordering.
  • The ablation evaluates 16 cases where the No Retrieved Memory baseline eventually succeeded but required more than one iteration.
  • 10 of 16 cases become first-iteration successes with Schema-Assisted Top-3 Retrieval.
  • Schema-Assisted Top-3 Retrieval reduces mean first-success iteration from 2.44 to 1.44 and aggregate token use from 41.38M to 14.79M, a 64.3% reduction.

Failure Analysis

Failure patterns differ across planner configurations: fixed workflows avoid observed code- and path-level failures, while autonomous coding introduces additional software and artifact-management failures. HALO’s efficiency and failure conclusions are bounded by the benchmark’s scope and evaluation design.

  • 100 Fixed Structured Workflow, 133 Autonomous Structured Agent, and 115 Autonomous Coding Agent failures are aggregated across five evaluated models.
  • Fixed Structured Workflow failures occur mainly at simulator/evaluation or scientific-target stages and show no code/path errors.
  • Autonomous Structured Agent has more missing or invalid results and setup/sweep-management errors, while Autonomous Coding Agent adds code/tool/API and artifact/path failures.
  • The Fixed Structured Workflow is most token-efficient and directly inspectable, whereas Autonomous Coding can achieve higher success with a strong model but incurs additional software, artifact, and output-format failures.
  • The memory study measures convergence on baseline-delayed-success cases rather than benchmark-wide success gains, and final designs may require higher-fidelity validation.
Loading 2608.28877v1…