PaperlayerView exact revision on arXiv

Source-linked AI summary

Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation

Bhawani Shankar Leelar, Pawan Chorasiya, Davin Hill, Robert E. Tillman, Tamer Soliman

arXiv:2610.10639v1cs.LGcs.AIcs.SE

TL;DR

Whether visible reasoning should match SQL or pandas remains insufficiently tested under controlled, execution-based comparisons. This paper evaluates matched analytics requests across personas, targets, reasoning formats, and internal-reasoning configurations. It finds no universal visible-CoT or target-matching advantage, with effects depending on the configuration.

  • Problem

    Whether visible-CoT representation should match SQL or pandas remains insufficiently tested in execution-controlled analytics generation, despite the same request admitting both targets.

    Source: §1 · p. 1

  • Method

    The study evaluates query-matched SQL and pandas generation while crossing persona phrasing, visible-CoT format, direct generation, controls, and internal-reasoning configurations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Visible CoT and target-language matching provide no universal accuracy advantage; effects vary with persona, target, model configuration, and internal-reasoning setting.

    Source: §1 · p. 2, §5 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Reasoning strategies should be selected for the model-persona-target configuration rather than adopted as a global prompting default.

    Source: §1 · p. 2, §6 · p. 8

    Each matched cell has one temperature-zero evaluation, so the analysis does not quantify variance across repeated decoding seeds or service replicas.

    Source: §7 · p. 8

Abstract

from arXiv · show

Visible Chain-of-Thought (CoT) is often treated as a broadly useful reasoning instruction, yet analytics code generation combines natural-language ambiguity, schema grounding, target-language constraints, and model-specific inference behavior. Because the same analytics request can be expressed in two distinct target languages-SQL and Python (pandas)-this setting provides a natural test of a common but under-examined assumption: that visible reasoning is more effective when its representation matches the requested target, as in "think in SQL" or "think in Python." Together with generic instructions such as "think step-by-step," such recommendations remain insufficiently evaluated under controlled, execution-based comparisons. We study a query matched SQL-pandas benchmark that crosses persona phrasing, target language, visible-CoT format, control prefixes, direct generation, and internal-reasoning configurations. The results do not support either a universal accuracy advantage from visible CoT or a consistent benefit from matching the reasoning representation to the target language. Instead, the effects depend on the persona, target, model configuration, and internal-reasoning setting. The control ablations further distinguish effects of reasoning content from those of prompt format. These findings indicate that reasoning strategies should be selected jointly for the model, persona, target, and internal-reasoning configuration rather than adopted as universal defaults. More broadly, the study provides a controlled framework for identifying when visible reasoning improves executable generation, when it primarily perturbs model behavior, and when the internal-reasoning configuration is the more consequential factor.

1 Introduction

The paper tests whether visible reasoning should match the target language in analytics code generation, while distinguishing visible text from internal-reasoning configurations. It finds no universal advantage and argues that effects depend on the configuration.

1 Introduction

  • Visible CoT is externally observable prompt-response text, whereas internal reasoning denotes unobserved model- or service-level inference-time computation.The study measures visible outputs and executable code, not provider-side internal traces.Source: §1 · pp. 1–2
  • The query-matched evaluation fixes the database task and executable reference while varying target language, persona phrasing, visible-CoT format, and model configuration.It tests the assumption that reasoning matching the requested target preferentially benefits procedural pandas generation.Source: §1 · pp. 1–2
  • Three controls—length filler, scrambled tokens, and wrong content—are compared with both coherent NL-CoT and direct generation.The dual comparison separates loss of task-aligned reasoning from effects of adding a long, structured context.Source: §1 · p. 2
  • Neither visible CoT nor target-language matching provides a universal accuracy advantage; effects vary with persona, target, and model configuration.A localized scrambled-prefix result motivates a prompt-format explanation rather than a new default.Source: §1 · p. 2
  • The analyses examine visible CoT and internal-reasoning configurations alongside difficulty, token use, and latency, retaining direct generation as an essential reference.The comparison supports selecting a reasoning configuration rather than assuming added reasoning is beneficial.Source: §1 · p. 2

2 Related Work

Prior work examines visible rationales, code-generation representations, text-to-SQL systems, internal computation, and benchmark grounding. The paper uses these strands to motivate a matched evaluation that separates reasoning representation, phrasing, and inference configuration.

Visible CoT and causal interpretation

  • Prior work cautions that fluent visible rationales may not faithfully represent the computation behind an answer (Turpin et al., 2023; Lanham et al., 2023).The paper's length-filler, scrambled, and wrong-content conditions apply these concerns to visible NL-CoT for executable analytics generation.Source: §2 · p. 2

Reasoning representations for code

  • Code-generation research uses plans, programs, and structured intermediate representations, whereas this study diagnoses visible representations with task, reasoning-slot structure, and evaluator held fixed.Related approaches include program-aided computation (Gao et al., 2023) and comparisons of natural-language, pseudocode, and code-like representations (Deng et al., 2024; Sun et al., 2024; Puerto et al., 2024).Source: §2 · p. 2

Text-to-SQL planning and analytics generation

  • Agentic text-to-SQL systems combine ingredients such as multiple calls, feedback, and selection, while this factorial design isolates visible-prefix effects under bounded recovery.It does not estimate the value of full agentic systems with extended planning, tool use, or open-ended multi-turn revision.Source: §2 · pp. 2–3

Internal computation and adaptive inference

  • Quiet and latent reasoning work distinguishes model-side computation from public tokens, motivating separate analysis of internal-reasoning configurations and visible CoT.The paper observes behavioral consequences, not internal computation content; test-time-compute work also motivates difficulty- and resource-conditioned analyses.Source: §2 · p. 3

Benchmarks, schema grounding, and persona

  • The matched setting combines database-grounded BIRD requests and paired SQL-pandas targets with Original, Novice, and Expert phrasings.The same underlying intent can leave different amounts of procedural and schema detail implicit across phrasings.Source: §2 · p. 3

2 Related Work

  • The literature motivates a matched evaluation that separates reasoning representation, task phrasing, and inference configuration.The following design specifies the factors and execution-based estimands.Source: §2 · p. 3

3 Experimental Design

The design holds the query and reference result fixed while crossing persona, target language, and reasoning condition. It evaluates executable correctness with paired comparisons and keeps resource summaries descriptive and configuration-specific.

3.1 Matched Panel and Conditions

  • For each of 304 questions, the database schema and executable gold answer are fixed while persona, target, and reasoning condition vary.The paired formulation preserves the same analytic intent across SQL and pandas, where the result may be declarative or require DataFrame operations.Source: §3.1 · pp. 3–4
  • Three controls remove task-relevant semantics while retaining approximate volume, permute reasoning-like words, or substitute a trace from another query.The controls target different properties of the visible prefix.Source: §3.1 · p. 3
  • A shared schema-grounded JSON contract requests a visible thinking list, target-language code, and code-type field; explicit-CoT prompts require eight labeled steps.Direct generation requires an empty thinking list, and internal-reasoning configurations are toggled separately from the visible response contract.Source: §3.1 · p. 3

3.2 Evaluation and Inference

  • The primary estimand is paired within-query accuracy change against direct generation, with exact execution determining correctness.Parse failures, runtime failures, and missing outputs count as incorrect; the primary tests use 252 model-persona-target-condition contrasts.Source: §3.2 · p. 4
  • Exact McNemar tests, query-cluster bootstrap intervals, and Benjamini-Hochberg correction support paired inference across the declared primary test family.A complementary query-clustered GEE checks associations adjusted for annotated query properties, without treating correlated features as randomized causes.Source: §3.2 · p. 4
  • Token and latency summaries are descriptive, retain individual CoT formats, condition on difficulty, and compare values within a serving configuration.They do not assert that latency is directly comparable across infrastructure.Source: §3.2 · p. 5

4 Results

Across the tested panel, corrected tests show no universal visible-CoT or target-matching advantage, and direct generation remains an essential reference. Outcomes vary by persona, target, model configuration, and resource setting.

4.1 Direct Generation as the Reference

  • Direct SQL accuracy exceeds direct Python accuracy by 8.22–16.45 percentage points across six configurations, while explicit formats have no common ranking across targets.Direct generation ties the strongest direct-or-explicit-CoT value for Gemma4-off on Python.Source: §4.1 · p. 5, Table 3 · p. 6

4.2 No Universal Visible-CoT Advantage

  • No condition survives the 252-test primary family, and no target-matched format has a BH-corrected advantage.The direct-reference matrix is descriptive evidence of heterogeneity, not a replacement for the corrected paired tests.Source: §4.2 · p. 5
  • Mode-by-persona evidence survives BH correction, while the complete persona matrices caution against generalizing Original-focused results to every phrasing regime.SCS has no distinct adjusted association; ambiguity and phrasing specificity do, but these covariates describe the benchmark rather than causal persona mechanisms.Source: §4.2 · p. 5
  • Visible CoT generally occupies larger token and latency budgets, but its accuracy return is small and configuration-specific rather than systematic.Harder queries shift direct and prompted generations toward larger budgets, and internal reasoning changes the resource range occupied by paired evaluations.Source: §4.2 · p. 5

4.3 Controls as NL-CoT Ablations

  • Length filler and wrong-content controls show no corrected difference from either comparator; scrambling shows no corrected difference from NL-CoT but is modestly higher than direct generation.The localized scrambled-prefix effect is strongest under Novice-SQL; low exact-step adherence and no compliance-correctness association argue against attributing it to recovery of the eight-step contract.Source: §4.3 · p. 5, Figure 1 · p. 6, Table 4 · p. 6

and Resource-Bounded Accuracy

  • Internal-reasoning settings redistribute evaluations across resource budgets, with accuracy changes differing between Gemma4 and Gemini Flash and across targets and difficulty levels.Token-budget plots are descriptive rather than causal dose-response plots; harder queries shift both target languages toward larger budgets, while visible CoT generally extends the spread.Source: and Resource-Bounded Accuracy · p. 6

5 Discussion

The discussion finds no universal visible-CoT or target-matching advantage in the tested panel; outcomes depend on persona, configuration, and resource budgets.

5 Discussion

  • Direct generation can match the strongest direct-or-explicit-CoT option, with no universal accuracy or target-language-matching advantage in the tested panel.The discussion describes the broader pattern as conditional on persona and operating regime.Source: §5 · pp. 6–7
  • Persona phrasing changes the operating regime in which prompts are evaluated.The discussion treats persona as a substantive factor, not an ornamental one.Source: §5 · pp. 6–7
  • The scrambled-prefix result is consistent with an interaction between task-related tokens and visible output structure, not a longer-prefix-only effect.The proposed mechanism remains a falsifiable hypothesis for future frozen-trace tests.Source: §5 · p. 7
  • Visible CoT generally uses larger token and latency budgets, while accuracy returns are small and configuration-specific.Harder queries shift generations toward larger budgets, and the figures condition comparisons on resource buckets or token bins.Source: §5 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7

6 Conclusion

In matched analytics-code generation trials, the paper finds no universal visible-CoT or target-matching advantage. It concludes that effects are conditional and that prompt-format effects and internal-reasoning settings matter alongside coherent visible reasoning.

6 Conclusion

  • Visible CoT is neither a universal optimizer nor a reliably target-language-matched one; effects vary across persona, target, model, and internal-reasoning configuration.Direct generation remains a competitive, configuration-dependent reference.Source: §6 · p. 8
  • Dual-baseline ablations indicate that prompt-format and lexical effects can perturb generation independently of coherent visible CoT.Source: §6 · p. 8
  • The findings provide a controlled basis for studying when visible reasoning improves executable generation, perturbs behavior, or is outweighed by internal-reasoning configuration.Source: §6 · p. 8

7 Limitations

The limitations bound interpretation of the control ablations, evaluation loop, model comparisons, resource measures, and task scope.

7 Limitations

  • The diagnostic controls are not counterfactual edits of one frozen rationale and cannot isolate the causal roles of content, sequence, length, and output structure.Scrambled-control findings are mechanism hypotheses, not evidence that incoherent reasoning is generally beneficial.Source: §7 · p. 8
  • One temperature-zero evaluation per cell under bounded recovery leaves stochastic and operational variance unquantified.The authors also caution that correlated benchmark descriptors make adjusted GEE associations non-causal.Source: §7 · p. 8
  • The six model configurations are structured coverage rather than a randomized sample, and model-family contrasts remain confounded by implementation differences.These include architecture, training data, weights, access path, and serving stack.Source: §7 · p. 8
  • Resource analyses support within-configuration comparisons, not cross-provider cost benchmarks, and conclusions may not transfer beyond the tested SQL-pandas setting.Latency is client-observed rather than hardware-normalized; the task scope excludes several broader software and agent settings.Source: §7 · p. 8

A Factorial and Artifact Audit

The appendix audits a complete factorial frame, defines inherited query descriptors, and documents how controls and response compliance are operationalized.

A Factorial and Artifact Audit

  • 87,552 unique factorial cells cover six model configurations, 304 queries, three personas, two targets, and eight reasoning conditions, with no duplicated keys or missing correctness labels or SCS values.Internal-reasoning-enabled configurations use the same cached native gold executions and question set as their paired disabled or budget-zero configurations.Source: A Factorial and Artifact Audit · p. 8

A.1 Annotation score definitions

  • Phrasing specificity, ambiguity, and SQL structural complexity are fixed benchmark-derived descriptors, not experimental treatments.The ambiguity score is study-specific, ranges from 0 to 1, and combines detected ambiguity coverage and volume; SCS describes reference SQL structure, which generated code need not reproduce.Source: §A.1 · pp. 8–9

A.2 Prompt construction and compliance protocol

  • The three controls replace coherent NL-CoT while retaining the schema, target-code, and JSON contracts, and differ in semantic relevance, token order, or task alignment.Length filler approximates prefix volume with unrelated text; scrambling permutes template tokens; wrong content supplies another query’s reasoning-like trace or a deterministic non-gold trace.Source: §A.2 · p. 9
  • Exact eight-step compliance is measured from parsed responses, but the eight-string requirement applies to explicit CoT and not uniformly to controls.Execution correctness remains the primary outcome; control step-count behavior is observed rather than required.Source: §A.2 · p. 9

A.3 Reproducibility protocol

The protocol fixes query exclusions and execution-based correctness, records runtime and serving details, and cautions against reading descriptive persona-wise cells as condition effects.

A.3 Reproducibility protocol

  • Five non-executable SQL gold answers and one unsupported-function query were excluded before inference, yielding an audited 304-query common frame.Outputs count as correct only when execution matches the normalized gold result; parse, runtime, and missing-output failures count as incorrect, with no semantic judge used for the primary label.Source: §A.3 · p. 9

A.4 Serving and runtime configuration

  • Latency is client-observed wall-clock time and supports within-configuration runtime descriptions, not hardware-normalized comparisons across serving paths.Gemma4-on/off share a checkpoint and differ in the request-time internal-reasoning switch; Gemini Flash was queried through an API, with the Mac serving as client and orchestrator.Source: §A.3 · p. 10, §A.4 · p. 10

B Detailed Persona-Wise Accuracy

  • Table 7 expands the Original-phrasing matrix to every persona, target, and visible-CoT condition while retaining direct generation as comparator.The matrix is descriptive and does not establish a condition effect; direct generation remains competitive in multiple rows.Source: B Detailed Persona-Wise Accuracy · p. 11

C.1 Notation and analysis unit

This section defines paired query-level comparisons and complementary uncertainty and adjustment procedures; adjusted covariate associations do not replace the paired condition tests.

C.1 Notation and analysis unit

  • Correctness is an executed program matching the normalized gold result, and paired contrasts hold query, model, persona, and target fixed while varying reasoning condition.Direct generation is the comparator for the primary family.Source: §C.1 · p. 11

C.2 McNemar Test

  • Exact McNemar/binomial tests use discordant matched outcomes; concordant pairs provide no evidence of a difference.The discordant counts represent queries correct only under one of the paired conditions.Source: §C.2 · p. 11

C.3 Effect Sizes and Confidence Intervals

  • Paired accuracy effects are reported in percentage points, with confidence intervals bootstrapped over 304 whole-query cluster means.Resampling individual rows would understate uncertainty because each query appears across multiple conditions.Source: §C.3 · p. 11

C.4 BH False-Discovery Control

  • Benjamini-Hochberg correction controls the expected false-discovery fraction among rejections, and the 252 mode-versus-direct contrasts form the primary family.Internal-reasoning contrasts, target matching, persona moderation, and control screens are separate declared families.Source: §C.4 · p. 11

C.5 Token Comparisons

  • Token differences are paired within query-persona-target and tested over query-level differences; token use cannot show whether a prefix was attended to or ignored.The signed-rank test and query-cluster bootstrap target different summaries of those differences.Source: §C.5 · p. 12

C.6 Clustered Logistic GEE

  • The clustered logistic GEE models 87,552 Bernoulli outcomes across 304 query clusters, adjusting for model, condition, persona, target, and query descriptors.The reported model uses independence working correlation with query-clustered robust covariance; the exchangeable fit failed to converge and was discarded.Source: §C.6 · p. 12

C.7 Structural Complexity and Other Query Covariates

  • SCS has no distinct adjusted association with correctness (BH-adjusted omnibus value 0.362); difficulty is also non-significant, while ambiguity and phrasing specificity remain associated.These benchmark features are correlated, not independently manipulated causal treatments; paired condition tests remain primary.Source: §C.7 · p. 12

D Original and Novice Focused Families

Focused and configuration-specific analyses find localized effects rather than a general visible-CoT or target-matching rule, with the corrected Novice-SQL result arising from a scrambled context.

D Original and Novice Focused Families

  • Original phrasing shows positive raw SQL-side movement but no corrected discovery; the smallest adjusted value is 0.481 across 48 explicit-CoT cells.An exploratory aggregate explicit-CoT comparison gives Original-SQL +1.56 points (95% query-bootstrap interval [0.23, 2.93]) but does not survive correction (pBH = 0.0954); these analyses were post hoc.Source: D Original and Novice Focused Families · pp. 12–13
  • Scrambled context improves Novice-SQL accuracy by 5.15 points after correction, the only corrected result in the focused Novice analyses; explicit CoT remains non-significant.The narrowed Original-and-Novice GEE does not make the omnibus mode family significant (pBH = 0.270).Source: D Original and Novice Focused Families · p. 12

D.1 Focused Control Mechanism Contrasts

  • The Novice-SQL scrambled result is not a generic length effect: no corrected length-filler difference from direct is detected, while scrambled exceeds filler and wrong content.The supported interpretation is sensitivity to a scrambled reasoning-like context, not a benefit from incoherent reasoning.Source: §D.1 · p. 13
  • Three Novice-SQL model gains survive correction within the 18-control screen but not the wider 72-cell focused-control correction; Novice scrambled contrasts are positive across all difficulty levels.The corresponding Original contrasts do not survive their correction family.Source: §D.1 · p. 13

E Internal-Reasoning Configuration Detail

  • Internal-reasoning configuration effects are localized: 16 of 96 contrasts survive BH, with Gemma4 accounting for 15 and Gemini Flash for one.The distribution makes a single pooled “internal reasoning helps” coefficient misleading; the analyses observe outcomes and visible traces, not internal reasoning contents or amount.Source: E Internal-Reasoning Configuration… · p. 13

F Target Matching, Controls, and Best Formats

  • No target-matched comparison survives BH across 36 cells, and no selected best CoT beats every alternative in its corrected family.Interaction tests show effects vary by persona and model, so the result is evidence against a single global target-matching rule, not interchangeability of representations.Source: F Target Matching, Controls, and Best… · p. 13
  • Six localized cells survive the narrower seven-tests-within-cell correction, but none survives the 252-test primary family.These include three scrambled Novice-SQL gains, two GLM4 Expert-Python losses, and one Gemma4-on Expert-Python gain.Source: F Target Matching, Controls, and Best… · p. 13

G Efficiency Detail

Efficiency comparisons treat each matched model-persona-target-condition cell as an accuracy-resource point. The results show control-prefix overhead and variable accuracy returns, while bucket figures are descriptive rather than causal.

G Efficiency Detail

  • The efficiency analysis compares direct generation and four explicit-CoT formats as accuracy-resource points, without assigning controls a cost/latency comparison.Individual formats are reported separately rather than collapsed into a single accuracy-per-token ratio across providers.Source: G Efficiency Detail · p. 13
  • Harder tasks move direct and CoT conditions toward larger resource budgets, while explicit-CoT accuracy returns vary by model and target.Each plotted point is mean accuracy within a model-adaptive bucket; reasoning condition, target language, and difficulty are encoded by color, marker shape, and point size.Source: G Efficiency Detail · p. 14
  • Every control increases completion and total tokens versus direct generation in all six configurations; 14 of 18 model-by-control cells have shorter completions than mean explicit CoT.All 14 shorter-completion differences are corrected.Source: G Efficiency Detail · p. 14
  • Controls average 8.6–10.2 thinking steps; exact eight-step adherence is 58.5% for length filler, 15.6% for scrambled, and 39.2% for wrong content, versus 99.5–99.9% for explicit CoT.The control signature is shorter but more fragmented reasoning plus positive token overhead relative to direct generation.Source: G Efficiency Detail · p. 14
  • Exact-step compliance is positively associated with correctness for length filler and wrong content, but not scrambled context; these observed associations do not establish causation.The analysis adjusts for model configuration, persona, target, and difficulty.Source: G Efficiency Detail · p. 14
  • Bucket figures are descriptive budget-stratified comparisons, and formal equivalence or non-inferiority tests with predeclared margins are needed before recommending cheaper substitutes.The figures should not be interpreted as causal dose-response curves.Source: G Efficiency Detail · pp. 14–15
Loading 2610.10639v1…