PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows

Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo

arXiv:2610.11482v1cs.SEcs.AIcs.DBcs.LG

TL;DR

Evidence about locally deployable LLM agents producing complete, valid engineering artifacts remains limited. This paper evaluates local configurations on reproducible mobility-workflow tasks using deterministic checkers, and finds higher pass rates under the closed-loop workspace condition than one-shot generation.

  • Problem

    Evidence remains limited on whether locally deployable agents can produce valid complete engineering artifacts rather than plausible text or isolated code.

    Source: §1 · p. 2

  • Method

    The study evaluates local models on a fixed-fixture mobility benchmark with deterministic artifact checkers and repeated one-shot and workspace-condition attempts.

    Source: §1 · p. 2, §3.3 · p. 8, §4 · p. 9

  • Results

    Among non-2B models, pass rates are 26.7–52.0 percentage points higher in the workspace condition than under one-shot generation.

    Source: §7 · p. 21

  • Takeaways & Limitations

    Complete agent configurations can be evaluated reproducibly through their produced artifacts, and workflows with explicit specifications and executable checks offer a setting for validation-gated assistance.

    Source: §7 · p. 22

    Results may vary with other model families, agent designs, hardware, domains, and production-scale pipelines.

    Source: §6.5 · p. 21

Abstract

from arXiv · show

Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure. Methods: We introduce a benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU. Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB memory footprint. Gains are largest when intermediate artifacts expose errors the agent can inspect and repair. Conclusion: Local open-weight agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The benchmark provides a reproducible method for evaluating complete agent configurations before adoption in engineering workflows.

1 Introduction

Evaluating local LLM agents requires checking complete engineering artifacts, not just plausible text or isolated code. This study introduces a reproducible mobility-workflow benchmark to assess artifact validity and how local agents perform across workspace conditions and model configurations.

1 Introduction

  • Evidence remains limited for complete, locally deployable agent workflows evaluated through heterogeneous artifacts rather than a single code-generation task.Engineering value depends on whether scripts execute, structured files meet schemas, and derived results remain consistent with their inputs.Source: §1 · p. 2
  • Local open-weight models offer organizational control over data boundaries, model versions, inference settings, availability, and cost, but scale and quantization may affect workspace reliability.The study situates mobility workflows as a testbed combining public-transport feeds, statistical indicators, geospatial data, transformations, and analytical artifacts.Source: §1 · p. 2
  • The study evaluates local agents with a reproducible benchmark of 15 mobility-workflow tasks and deterministic artifact checkers.The checkers assess scripts, tables, structured files, figures, and reports; the harness also records success, failure category, runtime, and tool interaction.Source: §1 · p. 2
  • The work contributes a fixed-fixture benchmark with deterministic checkers and machine-readable traces, a controlled study of ten local configurations, and evidence on correctness and engineering trade-offs.The benchmark is intended for reproducible evaluation and does not claim that mobility tasks cover all software or data-engineering practice.Source: §1 · pp. 2–3
  • The evaluation asks how one-shot and closed-loop workspace conditions, model scale and architecture, and quantization relate to artifact success, runtime, tool interactions, and efficiency.The experimental scope is limited to models servable on a single 24 GB consumer GPU.Source: §1 · p. 3

2 Related work

Prior work examines language-model agents, executable code benchmarks, tool use, local inference, and reproducibility, but often focuses on narrower tasks or evaluation targets. This study applies artifact-level evaluation to complete, locally deployable data-engineering workflows.

2 Related work

  • Unlike general-purpose interactive-agent benchmarks, this work narrows evaluation to reproducible engineering tasks and submitted-artifact validity.AgentBench evaluates language models across open-ended multi-turn environments [11].Source: §2 · pp. 3–4
  • The study extends software-engineering evaluations to complete data-engineering artifacts, heterogeneous task families, local serving constraints, repeated execution, and failure analysis.Earlier work includes code review and semantic understanding [4] [5], multi-agent requirements generation, and comparisons of artifact validity under agent configurations.Source: §2 · p. 4
  • Executable benchmarks assess functional correctness or repository-level fixes, whereas this benchmark checks connected workflow outputs beyond source code alone.Relevant examples include HumanEval [12], MBPP [13], CodeXGLUE [14], DS-1000 [15], and SWE-bench.Source: §2 · p. 4
  • The paper combines mobility data-engineering tasks with a controlled local-model sweep measuring workspace-condition gain, runtime, tool interaction, quantization sensitivity, and failure categories.The authors state that, to their knowledge, no prior work combines these elements in one evaluation.Source: §2 · p. 5

3 Benchmark design

The benchmark turns mobility data engineering into independently checked tasks, using fixed local fixtures and a bounded workspace. It compares one-shot and tool-mediated work and scores reproducible artifact correctness alongside failure behavior.

3 Benchmark design

  • Tasks transform natural-language specifications and bounded workspaces into verifiable artifacts checked against schemas, numerical constraints, domain rules, and report requirements.Fixed local fixture snapshots give models the same inputs and avoid variability from live APIs, outages, or schema changes.Source: §3 · pp. 5–6

3.1 Task suite

  • The 15 tasks span five mobility-workflow families, from data discovery and feed processing through semantic enrichment, analytics, validation, and reporting.Tasks are independently scored to diagnose which workflow stages are within reach for a model and which remain difficult.Source: §3.1 · pp. 6, 8, Table 1 · p. 6

3.2 Interaction modes

  • Chat-only models submit artifacts in one response, while agent-with-tools models can inspect files, run commands, and revise outputs in an isolated workspace.Both modes receive the same task definition and checker; the reported agentic gain measures the complete workspace condition, not tools in isolation.Source: §3.2 · p. 8

3.3 Evaluation and failure categories

  • Deterministic checkers score artifact correctness and classify failures, while structural compliance and consistency—not subjective prose quality—are assessed for natural-language artifacts.Failure categories distinguish issues such as logic, protocol, format, schema, wrong data source, tool loops, and domain errors; trace logging records metrics automatically.Source: §3.3 · pp. 8–9

4 Experimental setup

The experiments use a controlled Qwen-family sweep on a single consumer GPU, with model precision choices shaped by memory limits. They hold inputs and generation policy fixed while permitting bounded local workspace interaction.

4 Experimental setup

  • Ten local Qwen-family configurations are evaluated through llama.cpp on one RTX 4090, spanning 2B to 35B parameters and dense and MoE architectures.The study is a controlled within-family sweep, not a broad comparison across unrelated model families.Source: §4 · pp. 9–10
  • The 27B dense and 35B-A3B MoE models use UD-Q4_K_XL because they exceed the GPU's 24 GB BF16 memory budget; 2B, 4B, and 9B models also have BF16 variants.Using the same quantization across evaluated sizes reduces precision level as a source of variation in scale comparisons.Source: §4 · p. 9
  • All runs use one fixed stochastic generation policy, selected after preliminary greedy-decoding runs produced repetitive outputs and non-converging tool loops.The preliminary runs are not included in the 1,500 scored attempts, and decoding-parameter effects are not independently estimated.Source: §4 · p. 10
  • Agents operate in isolated workspaces with fixed fixtures and bounded local tools; checker code and reference expectations remain hidden until submission.The harness records execution caps and stops attempts that exceed the defined interaction, tool-call, time, or shell-command limits.Source: §4 · pp. 11–12

5 Results

Across the evaluated local configurations, artifact pass rates rose with model scale and generally improved in the workspace condition, while task-level results varied substantially. Performance also traded off against latency, and repeated runs exposed reliability differences.

5.1 Main results

  • 85.3% for Qwen 3.6 27B was the highest agent-mode pass rate, while quantized Qwen 3.5 9B reached 69.3% and completed a majority of attempts.Beyond 9B, pass rates continued to increase with size; the Qwen 3.5 27B dense and 35B-A3B MoE configurations reached 74.7% and 77.3%, respectively.Source: Table 3 · p. 13, §5.1 · p. 13

5.3 Performance analysis

  • 85.3% for Qwen 3.6 27B came with a 31.3 s median submission time, versus 14.7 s for Qwen 3.6 35B-A3B, which reached 81.3%.Both MoE configurations occupy a favorable region relative to dense 27B models, though architecture is not isolated from other configuration differences.Source: §5.1 · p. 13, §5.3 · p. 17

5.1 Main results

  • +52.0 percentage points for Qwen 3.5 9B Q4 was the largest observed workspace gain; middle-scale models gained most when they could follow the protocol but remained weak at one-shot generation.The comparison describes the complete workspace condition, not the isolated causal effect of tool access.Source: Table 4 · p. 14, §5.1 · p. 14
  • At 4B, BF16 had a 20.0-point higher agent-mode pass rate than Q4; at 9B, Q4 reached 69.3% and BF16 66.7%.Task-level contrasts were mixed, including Q4 passing 0/5 T07 attempts while BF16 passed all five; with five repetitions, these contrasts identify follow-up tasks rather than stable effects.Source: §5.1 · pp. 14–15, Table 5 · p. 15, §5.2 · p. 15

5.2 Per-task analysis

  • 80% represents four successes in five fixed-condition attempts, while 40% represents two; divergent plans and repair trajectories make a single successful run weaker evidence.Wilson intervals summarize pooled attempts descriptively, not population-level uncertainty across all possible data-engineering tasks.Source: §5 · p. 13, §5.2 · p. 15
  • T08 rose from 0/50 chat-only passes to 37/50 agent passes, while T01 rose only from 4/50 to 9/50, consistent with greater gains when artifacts are locally inspectable.T01 requires semantic selection without an equally direct local oracle for adequacy, whereas T08 permits inspecting and repairing a generated CSV.Source: §5.2 · pp. 15, 17, Figure 3 · p. 16

5.3 Performance analysis

  • Smaller quantized models were dominated by protocol, schema, and loop failures, while logic errors became the main residual category for larger models.Qwen 3.5 35B-A3B showed more format errors than its Qwen 3.6 counterpart, and the 9B model still had several tool-error loops.Source: §5.3 · p. 17

6 Discussion

Local agents produce valid artifacts for a substantial subset of the benchmark, but performance varies with model capability, task verifiability, precision, and runtime. The discussion considers these results through four research questions and describes the study’s validity boundaries.

6.3 RQ3: Artifact success, runtime, and tool-use trade-offs

  • The artifact-success–latency plot compares median submission time with pass rate, while marker size and color encode model size.The dense Qwen 3.6 27B leads in pass rate at 85.3% but takes 31.3s; the MoE configurations are faster, though architecture alone is not isolated as the cause.Source: Figure 4 · p. 18, §6.3 · p. 20

6.1 RQ1: Effect of the workspace interface

  • 26.7–52.0 percentage points: agent mode raises pass rates over chat-only for non-2B models, with larger gains when intermediate work is self-verifiable.T08 passes in 37/50 agent-mode attempts versus 0/50 chat-only attempts; T01 improves from 4/50 to 9/50, but semantic selection remains difficult.Source: §6.1 · p. 19

6.2 RQ2: Capability of locally deployable open-weight models

  • 85.3%: Qwen 3.6 27B dense has the highest pass rate; Qwen 3.5 9B Q4 reaches 69.3%.The 9B Q4 checkpoint uses approximately 6.5 GB, while larger models require workstation-class GPU budgets.Source: §6.2 · p. 19
  • Task performance is uneven: procedural, inspectable tasks often succeed, while semantic selection, exact clustering, and complete validation remain harder.Stronger agents also fail less often through interface breakdowns and more often through task-specific reasoning, validation, or fixture-grounding errors; plausible files can still violate parser or report requirements.Source: §6.2 · pp. 19–20

6.4 RQ4: Effect of quantization and precision

  • At 4B, BF16 raises pass rate from 37.3% to 57.3%; Q4 remains viable for sufficiently capable models, but broader claims need replication.The matched-size comparison favors Qwen 3.6 point estimates, while the 35B-A3B configuration has the lowest latency among higher-performing configurations.Source: §6.4 · p. 20

6.5 Threats to validity

  • Findings are bounded by the tested constructs and configurations: checks omit maintainability and security, the modes bundle components, and broader generalization and population inference remain limited.The study covers a controlled Qwen-family sweep on one GPU; five repetitions may miss rare trajectories, and aggregate rates combine tasks of different difficulty.Source: §6.5 · p. 21

7 Conclusion

The workspace condition substantially improves artifact pass rates, while model choice involves a correctness–resource trade-off. The benchmark supports validation-gated assistance and further reproducible evaluation, but not unvalidated autonomous execution.

7 Conclusion

  • Across 1,500 attempts, configurations differed substantially in artifact correctness, interaction reliability, runtime, and failure behavior; local agents support parts of the work, not unvalidated autonomous execution.The benchmark and replication package support extension to other model families, agent scaffolds, hardware, and task suites; explicit specifications and executable checks offer the strongest setting for validation-gated assistance.Source: §7 · pp. 21–22
  • 26.7–52.0 percentage points higher pass rates than one-shot generation are observed for non-2B models in the complete workspace condition.Gains are greatest when intermediate states are locally verifiable through parsers, executable scripts, numerical checks, or inspectable files.Source: §7 · p. 21
  • 85.3% pass rate is reached by Qwen 3.6 27B dense, while Qwen 3.5 9B Q4 reaches 69.3% with an approximately 6.5 GB checkpoint.The smaller configuration may suit validation-gated assistance on constrained hardware; higher memory or latency costs accompany larger models.Source: §7 · p. 21
  • MoE configurations occupy a favorable artifact-success–latency region, while the exploratory precision comparison suggests BF16 matters more at 4B than at 9B in this study.These observations are configuration-specific, not general claims about architecture or quantization.Source: §7 · p. 21
  • Agent selection requires evaluating artifact correctness together with latency, memory footprint, protocol reliability, and deterministic validation.None of the measured pass rates removes the need for task-specific checkers.Source: §7 · pp. 21–22

CRediT authorship contribution statement

The contribution statement assigns research, implementation, validation, writing, visualization, supervision, and project-management roles across five authors.

CRediT authorship contribution statement

  • García-Carrasco led methodology, software, validation, investigation, data curation, drafting, editing, and visualization; Sanchis, Reina-Reina, Maté, and Trujillo contributed to conceptualization, editing, supervision, or project administration.Maté and Trujillo are listed for supervision and project administration; Sanchis, Maté, and Trujillo for conceptualization; all five for review and editing.Source: CRediT authorship contribution statement · p. 22

Funding

The study lists support from European, Spanish national, and Valencian public funding programs and projects.

Funding

  • Funding came from NextGenerationEU through SOPHIA and the ENIA Chair, KOSMOS-UA, Generalitat Valenciana’s BALIDA-AA and IAEAV, and MCIN/AEI’s RED2022-134656-T.The funding bodies had no role in study design, data collection, analysis, interpretation, manuscript preparation, or the decision to submit.Source: Funding · p. 22
Loading 2610.11482v1…