PaperlayerView exact revision on arXiv

Source-linked AI summary

AlgoPilot: Cross-Paradigm Reasoning in Language Models via Strategy Selection and Guidance

Yilun Hao, Ruixiao Yang, Hudson Hilal, Yongchao Chen, Kaizhi Qian, Yada Zhu, Yang Zhang, Chuchu Fan

arXiv:2611.04079v1cs.AIcs.CLcs.LG

TL;DR

Heterogeneous reasoning problems require choosing among different solving paradigms and formulating each problem appropriately. AlgoPilot learns to select a paradigm and guidance mode, provide paradigm-specific guidance when needed, and execute the solution; it raises same-model accuracy from 38.4% to 72.9% and outperforms the best large-model prompting baseline by 11.7%.

  • Problem

    Real-world reasoning problems differ in which computational approach suits them, and LLM reasoning must both choose a paradigm and formulate the problem for it.

    Source: §1 · p. 1

  • Method

    AlgoPilot trains SteerLM to select a paradigm and guidance mode, GuideLMs to produce structured paradigm-specific formulations, and an ExecuteLM to solve using that strategy and optional guidance.

    Source: §1 · pp. 2–3

  • Results

    38.4% to 72.9% average accuracy for the same small model; AlgoPilot also outperforms the best large-model prompting baseline GLM-5 by 11.7%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The learned steering and guidance modules transfer to improve another execution model, suggesting strategy selection and formulation information is partially reusable across models.

    Source: §1 · p. 3, §4.5 · p. 10

    Execution-derived feedback can be computationally expensive and may limit scalability, while training from one execution model's behavior can make transfer imperfect.

    Source: §5 · p. 10

Abstract

from arXiv · show

Real-world problems span diverse domains, including program synthesis, symbolic reasoning, planning, and optimization, and often require fundamentally different solution paradigms. A central challenge for LLM-based reasoning is twofold: identifying an appropriate problem-solving approach and formulating it correctly. In this paper, we propose AlgoPilot, a cross-paradigm reasoning framework that enables adaptive selection of the solving approaches and structured problem formulation prior to execution. The AlgoPilot framework contains three components: 1) a SteerLM, trained via supervised fine-tuning (SFT) and reinforcement learning (RL), that adaptively selects the solving strategy (paradigm and modes) based on input problems, 2) the corresponding expert GuideLM, trained via SFT, that generates structured formulation guidance tailored to the selected strategy, and 3) an ExecutionLM that generates solutions conditioned on both the selected strategy and its formulation. Unlike prompting-based methods that rely on fixed reasoning templates, our approach learns to adaptively choose and structure problem-solving strategies based on the input. We build AlgoPilot on Qwen3-8B and evaluate across 5 problem categories and 13 benchmarks with 66 unique domains, comparing against both the latest similarly scaled small models and large LLMs. AlgoPilot significantly improves average accuracy for the same model from 38.4% to 72.9%, outperforming the best large LLM prompting baseline GLM-5 by 11.7%. Ablation studies confirm the effectiveness of each component. Notably, we show that the learned steering and formulation modules can transfer across execution models, suggesting their potential to improve other LLMs. Project page: https://sites.google.com/view/algopilot.

1 Introduction

Heterogeneous reasoning problems require different computational paradigms, so a solver must choose an approach suited to each input. AlgoPilot learns to select a paradigm and guidance mode, generate formulation guidance when needed, and execute a solution; it improves benchmark accuracy over both similarly scaled models and a large-model prompting baseline.

1 Introduction

  • Real-world problems differ in what makes them difficult: some suit natural-language reasoning, others code execution, and others formal solvers such as SMT, PDDL, or MILP.This diversity makes choosing among fundamentally different computational paradigms part of effective reasoning.Source: §1 · p. 1
  • AlgoPilot separates paradigm selection from paradigm-specific formulation: SteerLM chooses a paradigm and whether guidance is needed, GuideLM supplies it when selected, and ExecuteLM produces the solution.The framework is evaluated using Qwen3-8B, with SteerLM trained by SFT and GRPO and GuideLMs trained by SFT.Source: §1 · pp. 2–3
  • Accuracy rises from 38.4% to 72.9% for the same small model, and AlgoPilot outperforms the best large-model prompting baseline by 11.7%.Evaluation spans 13 benchmarks and 66 domains.Source: §1 · p. 3

2 Related Works

Prior work adds structure, tools, and adaptive choices to LLM reasoning, but often focuses on one executor or the text–code choice. AlgoPilot extends adaptive selection across multiple computational paradigms and adds paradigm-specific formulation guidance.

LLM Reasoning

  • Reasoning methods use intermediate natural-language steps, executable code, external tools, or formal executors, while code-augmented work also studies choosing between text and code.AlgoPilot covers CoT, Code, SMT, PDDL, and MILP rather than focusing on a single executor or only the text–code choice.Source: §2 · p. 3

Adaptive Selection

  • Adaptive inference methods select among models, prompts, budgets, decomposition levels, or model–tool combinations, while AlgoPilot learns paradigm–mode selection and explicitly generates formulation guidance before execution.Classical algorithm-selection and portfolio methods also choose a solver for each problem instance.Source: §2 · p. 3

3 AlgoPilot

AlgoPilot addresses two challenges: inferring a suitable solving paradigm and formulating the problem for it. Its pipeline separates steering, optional paradigm-specific guidance, and execution; its training procedure constructs guidance and steering supervision from execution outcomes.

3.1 Problem Formulation and Challenges

  • A problem's paradigm is not specified in its natural-language input, and general-purpose LLMs can struggle to create the paradigm-specific formulations many approaches require.Formulations may require variables, constraints, or intermediate reasoning structures.Source: §3.1 · p. 4

3.2 AlgoPilot Pipeline

  • SteerLM selects a paradigm and Direct or Guided mode; when Guided is selected, the matching GuideLM creates structured guidance, and ExecuteLM produces the answer using the problem, strategy, and optional guidance.The five paradigms are CoT, Code, SMT, PDDL, and MILP.Source: §3.2 · p. 4

3.3 AlgoPilot Training

  • Training separates the search for valid guidance from the search for useful paradigms, avoiding a joint combinatorial search.The process trains GuideLMs and then uses cross-paradigm evaluation to construct SteerLM supervision and rewards.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.3.1 Initial Dataset Construction

  • Initial data pairs each question with one paradigm, then retains candidate guidance that achieves at least three correct executions across five runs.Up to three useful guidance pairs are kept per problem.Source: §3.3 · p. 4, §3.3.1 · p. 5

3.3.2 Dataset Expansion

  • Dataset expansion evaluates paradigms in Guided and Direct modes, ranks useful combinations as supervision, and balances the data across paradigms.Direct combinations require at least three successful executions out of five; Guided combinations require at least three guidances meeting that threshold.Source: §3.3.2 · pp. 5–6

3.3.3 SteerLM Training

  • SteerLM is trained with SFT on the expanded dataset followed by GRPO using execution-derived rewards for its ranked paradigm–mode predictions.The reward also favors higher-scoring candidates earlier and penalizes invalid formatting and duplicate predictions.Source: §3.3.3 · p. 6

4 Experimental Results

Across five reasoning paradigms, AlgoPilot is evaluated on heterogeneous benchmark tasks and achieves its strongest reported average accuracy against prompting baselines. Ablations and generalization tests examine the roles of steering and guidance and their performance beyond the original evaluation setting.

4.1 Experimental Setup

  • The suite spans 66 domains across 13 benchmarks and five reasoning categories, with 2,220 held-out test instances evaluated using the same test split across baselines.The full dataset contains approximately 22k instances, split 80%/10%/10% for training, validation, and test.Source: §4.1 · p. 6

4.2 Cross-Paradigm Evaluation Analysis

  • Many examples are best addressed by paradigms other than their original task category, supporting instance-level steering across heterogeneous problems.Figure 3 summarizes empirically best paradigm distributions among examples with at least one useful paradigm–mode label.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 AlgoPilot Performance

  • 72.9% average accuracy for AlgoPilot SFT+GRPO exceeds GLM-5DIRECT by an average of 11.7%.This is the best average among the methods and baselines, excluding the non-comparable Top-3 analysis variant.Source: §4.3 · p. 7, Table 1 · p. 8
  • GRPO raises AlgoPilot’s average accuracy from 67.2% to 72.9%, with the largest gains on Code and SMT.The authors suggest execution-derived rewards help SteerLM rank and prioritize effective paradigm–mode choices.Source: §4.3 · p. 8

4.4 Ablation Studies

  • Ablations attribute gains to both SteerLM and GuideLM, while oracle results indicate that choosing the best paradigm–mode pair alone can still leave formulation guidance necessary.The authors report 55.5% for ORACLE1 and 84.5% for ORACLE2, and find the combined system outperforms ablated variants at 72.9%.Source: §4.4 · p. 9

4.5 Robustness and Generalization

  • On paraphrased inputs, AlgoPilot reaches 68.6% accuracy versus 72.9% on original inputs, while SteerLM’s top-1 routing agrees with the original in 98.0% of cases.Paraphrases vary lexical choice, syntax, and natural formulation while preserving the underlying problem.Source: §4.5 · p. 9, Table 3 · p. 9
  • On 432 problems from 10 held-out domains, AlgoPilot reaches 73.3% instance-weighted accuracy, compared with 53.0% for DIRECT and 66.2% for REASON prompting.The held-out domains span all five paradigms and include new domains within seen benchmarks and tasks from unseen benchmarks; formulation transfer is identified as the main bottleneck.Source: §4.5 · p. 9, Table 4 · p. 10
  • With Gemma-4-E4B-it as ExecuteLM and the same trained SteerLM and GuideLM, transferred AlgoPilot reaches 60.5%, versus 39.0% for DIRECT and 50.8% for REASON prompting.The transfer gain is smaller than with Qwen3-8B because training data filtering, cross-paradigm evaluation, and GRPO rewards derive from the original execution model.Source: §4.5 · p. 10, Table 5 · p. 10

5 Conclusion

AlgoPilot decomposes LLM problem solving into adaptive paradigm selection, structured formulation guidance, and execution, and experiments show improvements over a small base model and strong large-model prompting baselines. The framework’s feedback and model-specific training create scalability and transferability constraints, while deployment calls for oversight and validation.

5 Conclusion

  • AlgoPilot combines adaptive selection among reasoning paradigms with structured formulation guidance and execution, improving a small base model and outperforming strong large-model prompting baselines across diverse reasoning categories.The learned steering and guidance modules also transfer to another execution model, though transfer may be imperfect.Source: §5 · p. 10
  • Execution-derived feedback makes guidance-data construction and steering training computationally expensive, potentially limiting scalability to very large datasets or newly added paradigms.Because the modules are trained from one execution model’s behavior, transfer to others may be imperfect.Source: §5 · p. 10

A Broader Impact Statement

  • The authors frame AlgoPilot as reasoning support rather than autonomous decision-making, recommending human oversight, downstream validation, and task-specific safety checks before deployment.Incorrect paradigm selection or faulty solver formulations could produce misleading outputs without verification.Source: A Broader Impact Statement · p. 16

B.1 SteerLM Training Details

SteerLM learns to rank paradigm–mode choices using execution-derived supervision and feedback. Its training targets, output format, and reward encourage a useful first choice alongside viable alternatives and valid structured output.

B.1 SteerLM Training Details

  • The SteerLM training set contains 5,858 datapoints, each targeting up to three ranked paradigm–mode pairs and excluding examples without a useful pair.Pairs are ranked by execution-derived success score.Source: §B.1 · p. 16
  • SteerLM outputs an ordered JSON-style list of paradigm–mode labels, with earlier labels representing more effective choices according to execution-derived scores.Training examples can include up to three pairs; labels combine a mode and paradigm category, such as DIRECT_CODING or GUIDED_MILP.Source: §B.1 · p. 16
  • The GRPO reward uses offline execution scores to favor a strong top-ranked strategy while assigning smaller weights to viable lower-ranked alternatives.A ranking bonus rewards score-consistent ordering; formatting and duplicate-label penalties encourage valid, distinct outputs.Source: §B.1 · p. 17
  • The reward-design ablation holds the Qwen3-8B SFT initialization, data, optimizer, schedule, sampling, and training budget constant while changing only the reward function.The full reward combines top-ranked and lower-ranked execution scores, ranking bonus, formatting penalty, and duplicate-label penalty.Source: §B.1 · p. 17

B.2 Routing Quality and Structured Generation

Reward-design ablations show that lower-ranked candidate rewards, ranking bonuses, and structural penalties improve candidate-set quality and structured generation. Padding controls indicate that the gains are not explained by list length alone.

B.2 Routing Quality and Structured Generation

  • Adding weighted lower-ranked rewards raises top-3 accuracy from 77.13% to 83.67%, NDCG@3 from 57.81% to 75.25%, and valid three-label generation from 10.7% to 84.7%.Top-1 accuracy alone does not capture the candidate-set quality and ordering gains.Source: §B.2 · p. 18
  • Adding Brank as the A2-to-A3 ranking bonus raises three-label completeness to 94.8% and improves pairwise ordering agreement, defined by predicted pairs matching precomputed execution-score order, from 82.0% to 83.6%.With structural penalties, validness improves from 94.8% to 100.0%.Source: §B.2 · p. 18
  • 100.0% of outputs are valid with the full reward, which also gives the best overall NDCG@3 and top-3 oracle result.The full reward produces a valid, complete list for every test instance.Source: §B.2 · p. 18, Table 6 · p. 18
  • Despite producing fewer than three candidates on average, A2 exceeds the strongest source-category padding control in Recall@3 (91.72% versus 89.19%), NDCG@3 (75.25% versus 72.89%), and best candidate score (78.99% versus 77.08%).The authors conclude the gain is not explained by list length alone.Source: §B.2 · p. 19, Table 7 · p. 19
  • Weighted rewards for lower-ranked candidates help SteerLM form a task-relevant set that covers high-performing strategies, rather than merely encouraging three-item output.The study reports top-choice benefits from graded execution feedback and broader viable-method coverage from lower-ranked rewards.Source: §B.2 · p. 19

B.3 Analysis of Guided Mode Selection

The analysis finds that unnecessary Guided selections are uncommon overall, while paradigm-specific guidance templates structure inputs for execution and are trained as separate specialized models.

B.3 Analysis of Guided Mode Selection

  • Same-paradigm Direct execution outperforms a selected Guided route for 11.70% of Guided selections, or 9.91% of the test set; any-paradigm Direct is better for 18.54% of selections.Mode-selection errors concentrate in easier Code and CoT categories; unnecessary Guided selection is almost absent for MILP and PDDL.Source: §B.3 · p. 20, Table 8 · p. 20

C.1 GuideLM Guidance Templates

  • GuideLM templates differ by paradigm and provide standardized training supervision plus explicit problem-solving structure for execution.Code guidance covers algorithm and implementation details; CoT guidance lays out facts and a reasoning plan, while optimization, planning, and constraint-solving use their own formulations.Source: §C.1 · pp. 21–23

C.2 GuideLM Training Details

  • Separate paradigm-specific GuideLMs are fine-tuned on filtered guidance data, with training sets of around 12k, 8.7k, 3k, 4,5k, and 6k datapoints for Code, CoT, MILP, PDDL, and SMT, respectively.The models use full-parameter supervised fine-tuning to learn each paradigm’s formulation patterns and important contents.Source: §C.2 · p. 23

C.3 GuideLM Effectiveness

AlgoPilot’s structured guidance is tested against direct QA fine-tuning and a unified GuideLM, while additional routing baselines probe whether its gains require learned instance-specific steering and formulation.

C.3 GuideLM Effectiveness

  • 72.9% average accuracy with specialist guidance exceeds the matched-data QA-SFT baseline’s 59.3%, although QA SFT reaches 87.6% on PDDL and 67.4% on MILP.The QA baseline uses the same problems and training scale, replacing formulation targets with verified correct answers.Source: §C.3 · p. 23, Table 9 · p. 23
  • GuideLM separates formulation expertise from the executor, allowing the same trained guidance models to benefit a different ExecuteLM without fine-tuning it again.The authors contrast this modularity with direct executor SFT, which generally must be repeated when the executor changes.Source: §C.3 · p. 23
  • 72.9% average accuracy for paradigm-specific GuideLMs exceeds 60.5% for a unified GuideLM, whose shared capacity must model several distinct formulation languages.The authors identify possible cross-paradigm interference in the unified model and say specialization is particularly beneficial for distinct symbolic representations.Source: §C.3 · pp. 23–24, Table 10 · p. 24

D.1 Additional Prompting and Routing Baselines

  • 72.9% average accuracy makes AlgoPilot the strongest evaluated baseline, versus 71.5% for GLM-5 with heuristic routing and 44.4% for matched Qwen3-8B.The comparison supports the benefit of combining learned instance-level steering with problem-specific guidance.Source: §D.1 · p. 25, Table 11 · p. 25
  • Exposing paradigm labels alone remains unreliable for precise PDDL and MILP formalization, while nominal task categories and solver-specific templates alone also fall short of AlgoPilot.The authors attribute the distinction to instance-level steering and guidance that supplies problem-specific variables, constraints, actions, and relations.Source: §D.1 · p. 25

D.2 AlgoPilot Generalization

AlgoPilot is evaluated for robustness to paraphrasing and generalization beyond its training domains. The supplied results report held-out-domain performance, while prompt passages describe execution and baseline configurations.

D.2 AlgoPilot Generalization

  • 98.04% of SteerLM top-1 routing decisions agreed with the original-input decision across paraphrases, and 93.55% retained the same ordered top-3 prediction.For 96.88% of original problems, all three paraphrases preserved the original top-1 decision.Source: §D.2 · p. 26, Table 12 · p. 26
  • 68.62% end-to-end average accuracy on paraphrased inputs compares with 72.90% on corresponding originals, a decrease of 4.28 percentage points.The authors characterize these preliminary results as suggesting end-to-end robustness to natural-language reformulation.Source: §D.2 · p. 26, Table 13 · p. 27
  • 73.27% instance-weighted accuracy on held-out domains compares with 53.01% for Qwen3 8B Direct and 66.20% for Qwen3 8B Reason; AlgoPilot beats both on 5 of 10 domains.The accompanying discussion describes cross-domain generalization as meaningful but incomplete, with downstream formulation or execution often implicated in failures.Source: §D.2 · pp. 26–27, Table 14 · p. 28

D.3 AlgoPilot main result performance with standard deviation

  • Table 15 reports accuracy across five categories as mean ± standard deviation across three samples or runs.The supplied passage states the table’s scope but not its cell values.Source: Table 15 · p. 28

E.1 ExecuteLM prompts

  • ExecuteLM receives the problem and selected algorithm family, with GuideLM’s structured guidance included only in Guided mode.Direct mode removes the Guidance field from the otherwise shared prompt template.Source: §E.1 · p. 28

E.2 Baseline prompts

  • DIRECT and REASON baselines receive the raw problem without algorithm-specific guidance; REASON additionally asks the model to select an approach before solving.The baseline prompt leaves paradigm choice to the model and permits several output formats, which the authors say can mismatch evaluation requirements.Source: §E.2 · pp. 31–32

F.1 Baselines

The baseline analysis compares token consumption across model–prompt configurations and task types, highlighting differences in completion-token use.

F.1 Baselines

  • 20.15M completion tokens for Qwen3-Coder under REASON compares with 3.17M for GLM-5 under DIRECT, a 6.4× difference between the most and least verbose configurations.Prompt tokens remain roughly 3.1–3.4M; the authors attribute completion-token differences to model verbosity and additional algorithm-selection reasoning under REASON.Source: §F.1 · p. 33, Table 16 · p. 33
  • PDDL and MILP are the most token-intensive tasks; CoT averages 19 completion tokens per call for DIRECT + GLM-5 versus about 522–792 under REASON.The authors link formal-specification generation to PDDL and MILP token use, and note that REASON substantially increases CoT output length.Source: §F.1 · p. 33, Table 17 · p. 33

F.2 AlgoPilot

AlgoPilot’s evaluation-time cost is concentrated in guidance and execution, while its baseline failure analysis shows how output-format mismatches and paradigm-specific errors limit different approaches. Across the baseline families, failure causes range from pipeline incompatibility to runtime and formulation errors.

F.2 AlgoPilot

  • 7,514 GPU-hours and 124.5 wall-clock hours were required for data construction, whose cached execution outcomes can be reused in training and transfer experiments.The guidance-search stage used 1,503 GPU-hours; the cross-paradigm stage used 6,011 GPU-hours.Source: §F.2 · pp. 33–34
  • 16.17M tokens across 14,460 calls make up AlgoPilot’s evaluation cost, with ExecuteLM using 9.49M tokens (58.7%), GuideLM 5.88M (36.3%), and SteerLM 0.81M (5.0%).The three stages are strategy selection, optional guidance generation, and final solution generation; cost is comparable to stronger verbose baselines but exceeds economical direct prompting.Source: §F.2 · p. 34, Table 18 · p. 34

G.1 Baselines

  • Baseline failures are grouped into format mismatch, execution error, timeout, generation mismatch, and incorrect answer.The analysis covers DIRECT and REASON configurations across large models and five algorithm families.Source: §G.1 · p. 36

G.1.1 CoT: Prompt–Pipeline Interaction

  • 84–87% accuracy is comparable across prompt types when CoT models do not produce code blocks; the reported gap is driven by format mismatch.The REASON prompt permits code, but the CoT pipeline extracts an answer from raw text without executing code.Source: §G.1.1 · p. 36, Table 20 · p. 36
  • 76.4% of Qwen3-Coder runs produce code even under DIRECT, illustrating how output-format choices can conflict with the CoT pipeline’s expectations.When CoT emits code, the pipeline’s answer-extraction regex can capture a template literal instead of a computed value.Source: §G.1.1 · p. 36

G.1.2 PDDL: Domain Generation Failures

  • 47–55% PDDL accuracy for GLM-5 contrasts with below 8% for Qwen3-Coder, whose failures often stem from generating Python instead of PDDL specifications.For Qwen3-Coder, 70–90% of failures are generation mismatches; no failures are attributed to a valid but incorrect plan.Source: §G.1.2 · p. 36, Table 21 · p. 37

G.1.3 SMT: Runtime Errors and Namespace Issues

  • 88.5% of Qwen3-Coder’s DIRECT SMT failures and 83.1% under REASON are runtime crashes, often involving NameError.The recurrent issue is referencing Z3’s unknown without importing or qualifying it; GLM-5’s REASON failures are mostly incorrect logical models.Source: §G.1.3 · p. 37

G.1.4 MILP: Timeout and Solver Errors

  • 60–76% of MILP failures are incorrect answers, reflecting difficulty formalizing optimization constraints; GLM-5 also times out on 34/373 DIRECT failures (9.1%).Both models also show syntax and type errors, often from incorrect PySCIPOpt or PuLP API usage.Source: §G.1.4 · p. 37

G.2 AlgoPilot

AlgoPilot yields 4,942 successful runs among 6,660 executions, and its failures differ from baseline failures. Format and generation mismatches are largely absent, while incorrect answers and guidance or formulation problems remain.

G.2 AlgoPilot

  • AlgoPilot’s failures largely avoid baseline pipeline mismatches: generation mismatch is absent and format mismatch accounts for only 11 failed runs.When selecting PDDL, it consistently produces domain and problem files, though some resulting specifications still fail semantically.Source: §G.2 · pp. 37–38
  • AlgoPilot has 157 execution-error failures and 15 timeouts; PDDL failures include 112 runs where the planner returns no valid plan.Timeouts are associated with overly expensive algorithms, while PDDL failures can reflect over-constrained or incomplete encodings.Source: §G.2 · p. 38
  • Guidance can introduce errors when its formulation is wrong: Code-Guided reaches 69.3% micro success versus 91.2% for Code-Direct, and CoT-Guided 64.2% versus 88.3% for CoT-Direct.The passage notes that this does not mean guidance is intrinsically harmful; flawed or unnecessary guidance can over-constrain execution on easier problems.Source: §G.2 · p. 38
  • 1,535 of AlgoPilot’s 1,718 failed runs are incorrect answers, the largest remaining failure category for the 8B model.Across 2,220 evaluated instances, AlgoPilot produced 6,660 execution runs and 4,942 successful runs.Source: §G.2 · pp. 37–38

H Task Descriptions

This section describes evaluation tasks across 66 domains, spanning reasoning, coding, optimization, planning, and SMT, and summarizes the checklist’s reported materials and disclosures. It covers the availability of code, data, and experimental details, along with author responses on ethics, assets, human subjects, and LLM use.

H Task Descriptions

  • The 66 domains cover five categories, with tasks drawn from established reasoning, coding, optimization, planning, and SMT benchmarks.Sources include Big-Bench-Hard, GSM8K, CLUTRR, LeetCode, OptiGuide, Chain-of-Experts, OptiBench, NL4Opt, NLP4LP, LLMFP, the International Planning Competition, SAT-LM, and Logic-LM.Source: H Task Descriptions · p. 39

CoT

  • CoT tasks span causal, kinship, formal-fallacy, mathematical, temporal, and object-tracking reasoning, among other forms.The 17 tasks also include navigation, recommendations, counting, and Boolean reasoning.Source: CoT · p. 39

Coding

  • Coding domains range from Boolean evaluation, string and array manipulation, and data structures to backtracking, dynamic programming, and greedy algorithms.The 27 tasks are organized by algorithmic topic and include problems such as range queries, bit manipulation, and connectivity.Source: Coding · pp. 40–41

Mixed Integer Programming

  • Optimization tasks require modeling and solving problems such as facility location, linear and nonlinear programming, network flow, and workforce scheduling.Other examples include supply-chain planning, traveling-salesperson tours, and allocating tasks across heterogeneous robots.Source: Mixed Integer Programming · pp. 41–42
  • Planning tasks include Blocksworld and logistics scenarios that require generating, checking, reusing, or revising action plans.Mystery Blocksworld replaces familiar objects and actions with abstract symbols, requiring the model to infer action semantics from preconditions and effects.Source: Mixed Integer Programming · p. 42
  • SMT tasks include constraint-based assignments, boardgame rule application at increasing depths, spatial ordering, and first-order proof writing.Proof-writing tasks chain up to five inference steps.Source: Mixed Integer Programming · p. 42

NeurIPS Paper Checklist

  • The authors provide code and data in supplementary materials and report that details needed to reproduce the main results appear in the Methods, Experiment Results, and Appendix sections.They also state that detailed experimental settings are included in the experiment section and appendix.Source: NeurIPS Paper Checklist · pp. 44–45
  • Error bars and compute-resource information are reported in the appendix.The checklist marks both the reporting of error bars and discussion of required resources as addressed.Source: NeurIPS Paper Checklist · pp. 45–46
  • The authors state that the paper follows the NeurIPS Code of Ethics, includes a broader impact statement in the appendix, and poses no risks requiring release safeguards.The safeguards response is marked yes on the basis that the paper poses no such risks.Source: NeurIPS Paper Checklist · pp. 46–47
  • The authors say existing assets are publicly available and properly cited, and that appendix details and a code-submission README document the dataset and model.The checklist answers yes to proper credit and licensing for existing assets and documentation of new assets.Source: NeurIPS Paper Checklist · pp. 47–48
  • The paper reports no crowdsourcing or human-subject research and declares that LLMs are used as baselines and are fine-tuned, with discussion in the Methods and Appendix.The human-subject and crowdsourcing checklist items are marked not applicable.Source: NeurIPS Paper Checklist · pp. 48–49
Loading 2611.04079v1…