PaperlayerView exact revision on arXiv

Source-linked AI summary

CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?

Damien Sileo, Dimitri Kachler

arXiv:2609.01600v1cs.CLcs.AI

TL;DR

Dynamic agent harnesses create a need to reason about lifecycle effects that propagate through dependencies and cleanup. CordisBench measures this ability across formal and executable settings as relevant interactions scale, finding declining reliability on consequences and cross-order reasoning while finite reference semantics agrees exactly with executable scoring outcomes.

  • Problem

    The paper asks how reliably models can anticipate lifecycle consequences in mutable agent harnesses without symbolic assistance or execution feedback.

  • Method

    CordisBench evaluates 1,200 structured-output questions in formal and executable Cordis settings while scaling relevant interactions from 2 to 32.

  • Results

    Models usually handle small systems well but become less reliable with more relevant interactions, especially for final-state prediction and reasoning across teardown orders.

  • Takeaways & Limitations

    Explicit finite semantics can compute or verify the mechanical lifecycle consequences for these controlled instances, reducing the need for unaided model reasoning.

  • Takeaways & Limitations

    The benchmark isolates lifecycle reasoning and its largest 24- and 32-interaction instances are stress tests rather than estimates of typical deployed configurations.

Abstract

from arXiv · show

Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a local plugin change can propagate through dependencies and cleanup. We introduce CordisBench, a 1,200-question benchmark of this lifecycle reasoning. It combines a controlled formal setting with programs executed against Cordis, a runtime that manages component dependencies and cleanup, and asks models to identify affected components, predict state after a specified teardown order, determine which conditions hold under all or some orders, and choose reconfigurations that succeed when executed. Across these tasks, we evaluate three efficiency-oriented models at low reasoning effort with 2, 4, 8, 16, 24, or 32 relevant interactions, using deterministic task-specific scoring. Models usually handle small systems well but grow less reliable as more interactions become relevant, especially when predicting final state and when reasoning across teardown orders. Additional inference effort recovers marked gains for some models. The cost is nontrivial: on our 16-interaction subset, GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question at medium effort. For these controlled instances, that cost is avoidable: an independent finite reference semantics agrees with Cordis execution on every observation and action outcome used for scoring across all 528 executable questions.

1 Introduction

CordisBench studies whether models can anticipate how runtime lifecycle changes propagate through dependencies and order-sensitive cleanup. It evaluates this reasoning in formal and executable settings while scaling relevant interactions without changing task format.

  • Lifecycle challenge: Different teardown orders can leave different final states when plugins restore values captured at activation.This occurs even when each plugin’s cleanup is reasonable in isolation.
  • Evaluation design: The benchmark evaluates both a compact formal setting and executable Cordis programs.Cordis manages component dependencies, lifetimes, and cleanup, while model predictions and proposed reconfigurations are checked through execution.
  • Scaling design: 2 to 32 relevant interactions are introduced while keeping each task’s question form and answer format fixed.Larger instances require tracking more interacting effects rather than processing unrelated context.
  • Benchmark scope: CordisBench is a 1,200-question benchmark covering localization, schedule prediction, guaranteed conditions, and reachable conditions.The benchmark also includes a Cordis-native setting where proposed reconfiguration sets are executed.
  • Main result: Performance on small systems does not reliably carry over to larger compositions.Localization often remains strong as interaction count grows, while final-state prediction and reasoning across teardown orders become less reliable.

2 Lifecycle Reasoning in Cordis

Cordis lifecycle reasoning tracks dependency-triggered departures and cleanup interactions, including cases where legal teardown orders produce different final states. CordisBench uses finite reference semantics to enumerate these consequences exactly.

  • Lifecycle model: Removing a provider can force dependent components to leave, triggering their cleanup effects in an order that affects lifecycle outcomes.Cordis components declare dependencies, and lifecycle changes determine both which cleanups run and their relative ordering.
  • Lifecycle model: Effects that restore state captured at activation can interfere when overlapping components stop in different legal orders.CordisBench focuses on cleanup behavior that is individually reasonable but jointly order-sensitive.
  • Reference semantics: Interaction count denotes relevant effect groups in the formal setting and queried dependents in the Cordis-native setting, while task form and scoring remain fixed.The benchmark design therefore measures simultaneously relevant interactions rather than total harness size.
  • Reference semantics: Finite reference semantics enumerates legal lifecycle continuations and executes them to produce exact answers for observations, schedules, conditions, and outcome counts.Each formal instance specifies components, dependencies, state, effects, and the lifecycle change under consideration.

3 Benchmark

CordisBench contains structured lifecycle-reasoning tasks in formal and executable Cordis settings, spanning dependency localization, teardown-state prediction, order-based conditions, reconfiguration, and outcome counting.

  • Benchmark design: 1,200 questions derive from 240 independently generated systems across formal and executable Cordis settings at six interaction counts.The benchmark includes 1,056 primary tasks and 144 outcome-count diagnostics with deterministic scoring.
  • Task families: Localization asks which components or application slots a lifecycle change can affect, testing dependency tracking before final-state computation.Schedule prediction instead supplies a teardown order and asks for the resulting application-visible observation.
  • Task families: Guaranteed and reachable conditions test whether named properties hold under all or at least one considered teardown schedule.The Cordis-native setting additionally evaluates executed reconfiguration plans that must use the smallest sufficient disposal set.
  • Task families: A representative reconfiguration item requires tracking captured startup values and selecting dependent disposals so a target holds under both teardown orders.The example uses an initial slot value of 488151, with plugins writing 36000 and 934261 before cleanup restores captured values.
  • Scaling: Interaction counts preserve question form, answer type, and scoring while adding effects or dependents whose consequences must be tracked.Formal interactions are effect groups touching the same adjacent state entries; native interactions are dependents whose cleanup can alter an observed slot.
  • Scaling: Successful reconfiguration must reach the target, preserve unrelated state, and use the minimum number of prior disposals.Each independent interfering pair requires one prior disposal by construction.
  • Validation: Shortcut controls reach only 7.3% whole-answer exact-match accuracy across the full benchmark.The controls use task identity, interaction count, prompt length, position, or lexical similarity rather than system reasoning.

4 Experimental Setup

The experiments evaluate three efficiency-oriented models under fixed, tool-free conditions using task-specific metrics designed to remain informative as answer size grows.

  • Models and inference: Gemini 3.7 Flash, GPT-5.6 Luna, and DeepSeek V4 Flash are evaluated at temperature zero with low reasoning effort.Each question receives one completion without tools or execution feedback, with an 8,192-token output limit.
  • Metrics: Primary metrics are Jaccard similarity for localization and condition sets, per-observable accuracy for schedule prediction, and executed success for reconfiguration.Malformed or missing outputs score zero, and outcome-count diagnostics are excluded from primary metrics.
  • Metrics: Figure 3 uses cluster-bootstrap percentile intervals over generated systems because questions from the same system are correlated.The benchmark was generated independently three times, producing three replicates per task and interaction count.

5 Results

Performance generally degrades as relevant interactions increase, most sharply for final-state and teardown-order reasoning, while additional inference effort can recover performance for some models. Exact finite semantics agrees with Cordis execution on all scored executable outcomes.

  • Scaling results: Localization often remains strong with increasing interaction count, while final-state prediction and teardown-order condition reasoning become less reliable.The decline appears in both per-observable accuracy and set Jaccard, not only whole-answer exact match.
  • Scaling results: GPT-5.6 Luna’s reachable-condition Jaccard falls from 91.7% to 14.1%, while executed reconfiguration success falls from 62.5% to 25.0%.Its localization remains near ceiling, and all primary responses parse, so the declines reflect incorrect answers rather than formatting failures.
  • Scaling results: DeepSeek returns all labels on nearly every parsed condition question from size 8 onward, making its flat curves reflect a return-all strategy.Because each condition question balances true and false labels, returning every label yields exactly 50% Jaccard.
  • Diagnostics: At size 32, raising Gemini’s output limit increases guaranteed-condition Jaccard from 20.2% to 71.2%, reachable-condition Jaccard from 31.1% to 45.0%, and prediction accuracy from 79.8% to 84.0%.Twenty-nine responses had reached the 8,192-token limit; all completed and parsed after rerunning with 32,768 tokens.
  • Diagnostics: GPT-5.6 Luna’s scaling decline persists with exactly two teardown schedules: guaranteed-condition Jaccard drops from 81.2% at size 8 to 64.4% at size 32.Reachable-condition Jaccard also falls from 91.1% to 69.7% in the fixed-schedule diagnostic.
  • Reasoning effort: From no reasoning to medium effort at 16 interactions, GPT-5.6 Luna’s prediction rises from 31.2% to 85.4% and executed reconfiguration from 0% to 50%.At medium effort, it uses 2,967 reasoning tokens per question on average.
  • Execution validation: Finite reference semantics agrees with Cordis execution on every observation and action outcome across all 528 executable questions.The agreement indicates that these controlled lifecycle consequences can be computed exactly from dependency and effect structure.
  • Execution results: GPT-5.6 Luna reaches the target on 67 of 96 reconfiguration questions but has 56 minimal successes after excluding unnecessary prior disposals.DeepSeek reaches the target on 33 of 96 but has only one minimal success; the target-attainment gap is caused by nonminimal plans.

6 Related Work

CordisBench is positioned among work on formal semantics, schedule-sensitive reasoning, composition and recovery, and evolving agent harnesses. Unlike these related efforts, it focuses on predicting concrete lifecycle effects from dependency-driven cleanup in a running configuration.

  • Formal semantics and schedule-sensitive reasoning: Formal-semantics benchmarks evaluate models as interpreters of supplied operational semantics or reactive systems with temporal-causal structure.PLSemanticsBench includes mutated rules, while TempoBench separates forward simulation from harder temporal-causal reasoning.
  • Formal semantics and schedule-sensitive reasoning: CordisBench focuses on the interaction between cleanup effects and dependency-driven removal.
  • Formal semantics and schedule-sensitive reasoning: Concurrent-program studies evaluate comprehension, verification, and generation under possible interleavings, whereas CordisBench predicts concrete lifecycle changes in running configurations.ScratchLens instead studies behavioral equivalence of concurrent Scratch programs with LLM equivalence judges as baselines.
  • Composition and recovery: Related formal-methods work verifies or synthesizes systems, including reversible actions, temporal strategies, and recovery for composed long-running actions.CordisBench instead requires the model to predict the consequences of lifecycle changes.
  • Harness evolution and Cordis: Agent-harness research modifies harness structure or studies iterative harness improvement, evaluation protocols, and downstream task performance.These efforts include Agentic Harness Engineering, Self-Harness, Hierarchical Self-Improvement, Evo-Bench, and related evaluation studies.

7 Conclusion

CordisBench finds that lifecycle reasoning becomes less reliable as more interactions matter, particularly for final-state prediction and teardown-order reasoning, while localization remains stronger. The conclusion argues that explicit finite semantics can compute controlled lifecycle consequences exactly, reducing the need for unaided model inference.

  • Findings: As more interactions become relevant, final-state prediction and reasoning across teardown orders become less reliable even when localization remains strong.
  • Findings: Reconfiguration plans can miss the target or reach it with more prior disposals than necessary.
  • Implications: For GPT-5.6 Luna, additional inference substantially improves reliability on the 16-interaction subset, while finite semantics solves the controlled instances exactly.The paper therefore recommends computing or verifying explicitly represented dependency and cleanup consequences, leaving higher-level goals to the model.

Limitations

CordisBench measures a controlled form of lifecycle reasoning rather than the full complexity of deployed agent harnesses. Its scope is limited by the restricted behaviors, isolated evaluation, model coverage, and stress-test scale of its largest instances.

  • Scope and scale: 24- and 32-interaction instances are stress tests, not estimates of typical deployed configurations, although degradation appears at smaller sizes.The benchmark scales interacting effect groups or dependents as a controlled difficulty measure.
  • Behavioral coverage: The benchmark omits failures, irreversible external actions, hot module replacement, and other production concerns.Its native tasks focus on dependency-driven removal and restore-oncleanup effects.
  • Evaluation setting: Evaluation isolates lifecycle reasoning with three efficiency-oriented models, whereas full agent harnesses may add tools, execution feedback, retries, or other difficulty-changing mechanisms.Some large formal instances also require long structured outputs, and increasing the output limit improves Gemini’s guaranteed-condition results while declines remain on other metrics.
  • Model coverage: Model coverage limits the scaling picture because Gemini 3.7 Flash is near ceiling on several native tasks and DeepSeek V4 Flash uses a near return-all strategy on larger condition tasks.GPT-5.6 Luna consequently provides the clearest dynamic range for several comparisons.

C Decomposed Structured-Output Metrics

The benchmark decomposes structured-output performance into parsing, exact-match, per-observable, and set-overlap diagnostics, while excluding executable reconfiguration from these metrics. These diagnostics reveal declining reliability with increasing interaction count.

  • Parse rate is reported alongside parsed exact match, parsed per-observable accuracy, and set Jaccard.Parsed metrics condition on successful parsing, while missing or malformed set predictions score zero.
  • The main per-observable metric assigns zero to every missing value.Set-valued tasks use Jaccard similarity, which gives partial credit for overlap while penalizing omissions and spurious elements.
  • Executable reconfiguration is excluded because execution success is the relevant measure.
  • 94.7% to 50.5%: GPT-5.6 Luna’s formal-task Jaccard falls from size 2 to size 32.Its parsed per-observable accuracy also falls from 93.8% to 56.4%; Gemini 3.7 Flash remains stronger per observable, but declines at size 32.
  • Table 4 reruns 28 length-limited primary responses with a 32,768-token limit, leaving all other predictions unchanged.

E Fixed-Schedule Diagnostic

The fixed-schedule diagnostic holds teardown schedules constant while increasing the number of simultaneously relevant interactions. Its table reports GPT-5.6 Luna’s Jaccard across task-size combinations.

  • The diagnostic samples ten Cordis-native systems per interaction count in each of three benchmark replicates.
  • Each question lists two opposite teardown orders, keeping the number of schedules fixed as relevant dependents grow.
  • Table 5 reports GPT-5.6 Luna’s Jaccard (%) for two controlled schedules at every size, with 30 questions per task-size combination.Parse rate is 100%.

F Outcome-Count Diagnostic

The outcome-count diagnostic evaluates whether models correctly count outcomes across larger interaction counts, separately reporting formal and Cordis-native performance. Performance is low overall and differs substantially by model and setting.

  • 38/144 (26.4%): Gemini 3.7 Flash answers the larger-interaction outcome-count questions correctly, versus 20/144 (13.9%) for GPT-5.6 Luna and 6/144 (4.2%) for DeepSeek V4 Flash.
  • In the formal setting, the models score 11/96, 0/96, and 0/96, while in the Cordis-native setting they score 27/48, 20/48, and 6/48, respectively.
Loading 2609.01600v1…