PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking

1 Introduction

• Biomaterials research requires models to integrate multiscale, cross-disciplinary knowledge and move beyond conventional predictions toward reflection, error correction, and exploration.The introduction frames these capabilities as important for predicting actionable outcomes and solving cross-disciplinary problems.Source: §1 · p. 2

1.1 Modeling reasoning, thinking and more

• PRefLexOR addresses limits in single-pass reasoning with iterative thought refinement and learning from preferred and rejected outputs.Its dynamic knowledge graph connects questions generated from random text chunks with relevant information retrieved from the corpus.Source: §1.1 · pp. 2–3

1.2 Motivation and Challenges

• Static supervised datasets and existing preference methods leave gaps in dynamic adaptation and recursive refinement, while inference-time reasoning strategies often require architectural changes.The section situates PRefLexOR among recursive reasoning approaches, including STaR and Quiet-STaR [29] [30], and methods using silent tokens such as X-LoRA [12].Source: §1.2 · pp. 3–5

1.3 PRefLexOR Framework

• PRefLexOR uses thinking tokens, ORPO alignment, rejection-sampling preference refinement, and on-the-fly task generation to support recursive reasoning.The paper describes three components: Structured Thought Integration Training, Independent Reasoning Development, and a Recursive Reasoning Algorithm.Source: §1.3 · pp. 5–6

1.4 Outline of this paper

• The paper outlines training strategies, inference examples, iterative reflection-based response improvement, and a discussion of strengths and future work.Source: §1.4 · p. 6

2 Results and Discussion

• Structured Thought Integration Training uses supervised fine-tuning and ORPO to teach special thinking tokens and favor well-structured, step-by-step reasoning.By the end of this phase, the model generates outputs adhering to explicit thought structures.Source: §2 · p. 7
• Independent Reasoning Development masks thinking tokens so the model learns to produce reasoning strategies and correct answers without relying on explicit markers.The phase targets challenging or ambiguous tasks and unseen question-answer pairs with unknown reasoning steps.Source: §2 · p. 7
• The trained model improves on reasoning tasks and makes decisions without explicit guidance, with the section presenting this as preparation for diverse real-world scenarios.Source: §2 · p. 7
• On-the-fly question-answer generation uses retrieved context to create a thinking section, a correct answer, and a rejected answer that is updated during preference alignment.The revised rejected answers are generated by the model in its current training state.Source: §2 · p. 7
• The first phase applies ORPO to align preferred and disfavored outputs, while the second applies EXO with intermediate reasoning masked to refine final answers.The passage characterizes EXO as minimizing reverse KL divergence and focusing on dominant modes in preference data.Source: §2 · p. 7
• During EXO training, rewards and margins rise, loss declines, and rewards/accuracy stabilize after approximately 200 steps.Source: Figure 4 · p. 8

2.1 Sample Results

• The inference examples are presented as tests of whether the model generalizes its learned reasoning across training-domain, intersecting, and unseen tasks.Prompting can include a draft thinking section, and the paper describes recursive reasoning and reflection as ways to scale inference compute.Source: §2.1 · pp. 8–9

2.1.1 Properties of Hierarchical Structures

• For hierarchical structures, the model connects general principles such as energy dissipation and size adaptation to material performance and examples including nacre.The example omits an explicit materials prompt and reports a structured final answer.Source: §2.1.1 · p. 9

2.1.2 Biological Materials Failure Mechanism

• For graceful failure, the model reasons through mechanisms including viscoelasticity, crack bridging, and fiber sliding, then synthesizes them into an explanation with examples such as bone and nacre.The account also identifies hierarchical structure as relevant to stress redistribution and energy absorption.Source: §2.1.2 · p. 9

2.1.3 Intersection between Literature, Philosophy and Materials Science

• In an out-of-training example, the model proposes that proteins and Hesse’s Glass Bead Game share hierarchical, interconnected, and dynamic features despite their different domains.The task was selected to examine generalization beyond materials science.Source: §2.1.3 · pp. 9–10

2.1.4 Analysis of Research Abstract and Proposal of new Hypotheses

• The model identifies a 15-fold increase in strength and modulus in the myco-composites study, then frames a hypothesis about scalable, tunable composites with improved properties.It also highlights high-resolution additive manufacturing, robust mycelium colonization, and hierarchical design.Source: §2.1.4 · p. 11
• The model proposes eight future research directions for structural myco-composites, including scale-up, property enhancement, multifunctionality, biodegradability, and hybrid-living materials.Other listed directions include complex geometries, inoculation strategies, and additive-manufacturing improvements.Source: §2.1.4 · p. 11
• The myco-composites proposals additionally include complex-geometry studies, inoculation strategies, and improved additive-manufacturing techniques.The suggestions address geometry’s mechanical effects, colonization, and production resolution and speed.Source: §2.1.4 · p. 11

2.1.5 Overall Analysis of Inference Examples

• Across the examples, the model is described as connecting abstract and applied domains and extrapolating research directions from initial findings.The myco-composites analysis also cites a modulus of 160 MPa and tensile strength of 0.72 MPa.Source: §2.1.5 · p. 12

2.2 Expanding the Analysis to incorporate Thinking and Reflection for Recursive Improvement in Agentic Modeling

• For the hierarchical-structures question, the initial response identifies mechanical properties and anisotropy, while reflection clarifies that anisotropy follows from organized material changes.The reflection also recommends emphasizing hierarchical organization and the interaction of structure, composition, and architecture.Source: §2.2 · pp. 12–14
• Thinking outlines reasoning, references relevant material, and forms hypotheses; reflection critiques and refines those ideas before the final answer.In the hierarchical-structures example, reflection adds the interplay of structure, composition, and architecture, along with design and manufacturing costs.Source: §2.2 · pp. 12–13, Figure 5 · p. 13

2.2.1 Recursive Reasoning Algorithm

• Each recursive cycle extracts a reflection, uses the critic to improve the thinking process, and regenerates an answer from that revised thinking.The algorithm pairs a fine-tuned reasoning model with a general-purpose critic and repeats for a preset number of iterations or until quality criteria are met.Source: §2.2.1 · p. 14
• The algorithm can return the final iteration or integrate the generated responses into a broader answer.The latter option may capture a wider range of insights developed across iterations.Source: §2.2.1 · p. 14, Figure 6 · p. 15
• 8.75/10 average score made iteration i=2 the strongest of three, with accuracy (9/10) and depth (9/10) its highest-scoring criteria.Table 1 also describes i=2 as the most comprehensive response, including greater depth, quantitative information, and loading-condition effects.Source: §2.2.1 · p. 14, Table 1 · p. 16

2.2.2 Comparison with Non-fine-tuned Model

• The reasoning model gives a specialized biological-materials analysis, while the non-fine-tuned model offers a broad organizational view and lacks thinking and reflection sections.The reasoning model discusses multiscale organization, energy distribution, and progressive damage in biological and engineered materials.Source: §2.2.2 · pp. 16–17

3 Conclusions

• PRefLexOR integrates preference optimization with recursive reasoning, guided by thinking tokens, active task generation, and iterative feedback to support domain-specific scientific problem solving.The work also describes preference alignment using ORPO and DPO/EXO and reports LoRA training as efficient on local GPU hardware.Source: §3 · p. 18
• The model connects Hermann Hesse’s Glass Bead Game with protein hierarchy and transfers its reasoning capability to summarization and research proposal development.The analogy frames iterative reflection as a way to connect ideas across disciplines.Source: §3 · p. 18, §3.1 · p. 19

3.1 Enhancing the Algorithm by Invoking Multidisciplinary Concepts from the Glass Bead Game

• The proposed Glass Bead Game extension replaces a single Reasoning Model with specialized Collaborative Agents and replaces the Critic with a shared Interdisciplinary Knowledge Base Model.The knowledge base emphasizes cross-domain synthesis, while the agents contribute distinct perspectives through iterative refinement.Source: §3.1 · p. 19, Figure 9 · p. 21
• The extension adds symbolic representation, interdisciplinary synthesis, contemplative reflection, collaborative refinement, and evolutionary memory updates as stages in an iterative response-generation process.Explicit abstraction is also proposed as a way to direct abstraction toward specific reasoning goals.Source: §3.1 · pp. 20, 22, 23

3.2 Future Work, Challenges and Opportunities

• Recursive phases may constrain real-time use through increased computational cost, while the current domain focus motivates testing broader multidisciplinary training.Suggested future directions include larger models, agentic strategies, and multiple thinking sections with partial masking.Source: §3.2 · p. 23

4.1 Special Tokens for Reasoning

• The work uses <|thinking|>/<|/thinking|> for reasoning steps and <|reflect|>/<|/reflect|> for reflection during training.Reflection marks evaluation and improvement of initial reasoning.Source: §4.1 · p. 24, Table 3 · p. 24
• The tokenizer also includes <|response|>/<|/response|> markers for final answers and optional <|scratchpad|>/<|/scratchpad|> markers for interim calculations or thoughts.The table notes that these tokens are available for extending the approach, but only thinking and reflection tokens are used in this work.Source: §4.1 · p. 24, Table 3 · p. 24
• The token scheme separates reasoning phases to provide more precise control over inference and answer generation.The customized tokenizer is available with the models developed in the work or separately.Source: §4.1 · p. 24

4.2 On-the-fly dataset generation via in-situ knowledge extraction

• The pipeline builds training examples in situ by selecting context nodes, generating a domain-specific question, and producing structured information and answer responses.The question is intended to be challenging and answerable by a domain expert without seeing the context.Source: §4.2 · p. 24, §4.2.3 · p. 25, §4.2.4 · p. 25

4.2.1 Context Enhancement with Retrieval-Augmented Generation during Dataset Generation

• Retrieval-augmented generation embeds text chunks and queries, ranks them by cosine similarity, and appends the most relevant chunks to the original context.The implementation uses BAAI/bge-large-en-v1.5 embeddings with Llama Index.Source: §4.2.1 · p. 25

4.2.2 Raw data used for training

• The training corpus comprises 500 scientific papers on biological and bio-inspired materials, chunked at 1,024 tokens with 20-token overlap.The PDFs are converted to markup before text chunking.Source: §4.2.2 · p. 25

4.2.3 Context Retrieval

• For question generation, the system can retrieve three random nodes when no topic is specified, then asks a model to formulate a standalone question from their combined context.The prompt requests an important, challenging fact without naming a specific study, paper, or author.Source: §4.2.3 · p. 25, §4.2.4 · p. 25

4.2.5 Category-Based Information Extraction

• Category-based extraction organizes context into predefined fields, while the paper’s single thinking section embeds multiple categories as separate analysis streams.The structured approach can be adapted with special tokens or LoRA adapters to elicit domain-specific thought processes.Source: §4.2.5 · pp. 25–26, Table 4 · p. 26

4.2.6 Thinking Section for Reasoning

• The extracted categories are assembled into a Thinking Section that supplies structured insights for constructing the answer.The section is designed to guide a logical, comprehensive response to the generated question.Source: §4.2.6 · p. 26

4.2.7 Correct and Incorrect Answer Generation

• The pipeline pairs a context-grounded correct answer with a rejected answer, which is generated as incorrect in the first stage or from the current trained model in the second.The correct response uses the context and Thinking Section; the rejected response is designed to lack logical reasoning and contextual support in the first stage.Source: §4.2.7 · pp. 26–27, §4.2.8 · p. 27

4.2.8 Final Output

• The final training output includes a question, a correct answer enhanced by the Thinking Section, and a rejected answer; an optional reflection section critiques the reasoning and suggests corrections.The reflection prompt asks for improvements or missing considerations without answering the question.Source: §4.2.8 · p. 27, §4.2.9 · p. 27

4.3.1 Masking of Thinking Tokens in Multiple Sections

• The loss excludes inner thinking tokens while retaining <|thinking|> and <|/thinking|> markers so the model learns to produce them.A masking fraction α selects ⌊α · n⌋ inner tokens; α = 0 masks none and α = 1 masks all.Source: §4.3.1 · p. 28
• Partial masking randomly selects tokens within valid thinking-token bounds, leaving the overall reasoning structure intact and skipping segments with unmatched markers.The valid range lies between the start and end markers; the start and end markers themselves may be excluded from masking.Source: §4.3.1 · p. 28, Figure 12 · p. 30

4.3.2 Dynamic Final Answer Detection via Masked Thinking Tokens

• Dynamic detection starts the compared span after the last <|/thinking|>, allowing variable-length reasoning; study experiments used one thinking section, equivalent to masking before the final answer.The last end-thinking token may optionally be excluded from comparison.Source: §4.3.2 · pp. 28–29, Figure 11 · p. 29

4.3.3 Comparison of Masking Approaches

• The approaches are complementary: masking retains learning of reasoning markers, while dynamic detection ignores intermediate tokens and focuses on the final output.The implementation toggles these strategies with the dynamic_answer_comparison flag.Source: §4.3.3 · p. 30

4.4 Optimizing Final Answers with Masked Reasoning Using Mode-Seeking Preference Alignment

• For K = 2, EXO trains on the ground-truth answer and a rejected answer predicted by the current model, using β = 0.1 to control preference-alignment strength.The described setup also uses label smoothing of 5 × 10^-3, learning rate 5 × 10^-7, maximum gradient norm 0.3, and one epoch per generated 50-sample dataset.Source: §4.4 · p. 31
• With reasoning tokens masked, EXO uses the preferred versus rejected final answers to infer and prioritize reasoning patterns associated with successful answers.The authors describe label smoothing as regularizing predictions and discouraging excessive confidence.Source: §4.4 · p. 31

4.4.1 EXO vs. DPO: Impact on Final Answers

• The authors contrast EXO's focus on dominant reasoning patterns with DPO's mean-seeking distribution across multiple paths, which they say can dilute performance.Their account emphasizes final-answer optimization even when intermediate reasoning is not visible.Source: §4.4.1 · p. 31
• EXO with label smoothing and β = 0.1 led to superior ground-truth alignment, better long-term training performance, and less overfitting when final-answer accuracy was prioritized.The authors report significant performance improvements with EXO compared with DPO while masking reasoning tokens.Source: §4.4.1 · p. 31, §4.4 · p. 31

4.5 Iterative Response Improvement using Thinking and Reflection model

• A reasoning model generates responses with thinking and reflection, while a critic improves the thinking and integrates the generated responses into a comprehensive answer.The algorithm repeats critic-guided thinking improvement and response generation for a specified number of iterations.Source: §4.5 · pp. 31–32
• Separating response generation from improvement and integration lets the fine-tuned reasoning model and general-purpose critic contribute distinct capabilities.The authors note either model could potentially be replaced or improved independently.Source: §4.5 · p. 32
• Thinking and reflection tokens delineate response components so the algorithm can extract and target the thinking process for improvement in each iteration.The reasoning model is specified as fine-tuned; the critic is a non-fine-tuned general-purpose model.Source: §4.5 · p. 32

4.5.1 Analysis of Iterative Result Quality

• GPT-4o analyzed text quality across iterations i = 0–2, assessing each response independently and requesting scores for coherency and accuracy.Source: §4.5.1 · p. 33

4.6 Training Stages

• Stage one uses rank-64, α = 64 LoRA adapters on attention, feed-forward, embedding, and language-model-head layers, then merges the adapter into the base model.The specified layers are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, and lm_head.Source: §4.6 · p. 33
• Stage two adds a rank-64, α = 64 adapter to the stage-one model and prioritizes accurate final answers over explicit guidance through intermediate reasoning.This phase excludes embedding and head layers from the adapter and uses EXO-based preference alignment to prioritize the most likely correct answer.Source: §4.6 · p. 33
• The authors characterize the combination of first-stage ORPO and second-stage EXO preference optimization as improving reasoning abilities and final outputs.The first stage targets basic reasoning, while the second refines answer accuracy.Source: §4.6 · p. 33

4.7 Brief recap of Key Concepts in Hesse’s novel The Glass Bead Game

• The Glass Bead Game symbolically combines disciplines such as music, mathematics, literature, and philosophy into a cohesive whole.Players use symbolic language to explore connections between fields.Source: §4.7 · pp. 33–34
• Within Castalia, the Game is a formalized practice of composing and interpreting symbolic sequences, overseen by the Magister Ludi.The Magister Ludi guides the players and maintains the Game’s integrity and ideals.Source: §4.7 · p. 34
• The section parallels the Game’s synthesis of knowledge with AI and complex reasoning engines expanding the scope of human intellect.It frames people as interpreters, ethical stewards, and collaborators with intelligent systems.Source: §4.7 · pp. 33–34

4.8 Relation between Preference Optimization and Reinforcement Learning Strategies

• Generating random training data and reference reasoning at each step exposes the model to a distribution of reasoning tasks rather than a fixed task.The section relates this varied task distribution to fast adaptation and generalization across reasoning challenges.Source: §4.8 · pp. 34–35
• Dynamic reference reasoning prompts exploration of different reasoning paths, while partial masking trains the model to produce preferred answers despite incomplete reasoning.The section distinguishes this preference signal from RL’s cumulative rewards over state-action sequences.Source: §4.8 · pp. 34–35
• The authors report that dynamic data generation and reasoning masking create a meta-learning-like environment for adapting across diverse reasoning tasks without overfitting to a specific pattern.They state that their experiments confirmed this behavior.Source: §4.8 · p. 35

4.8.1 Modification to Further Align with Reinforcement Learning

• The proposed RL-aligned modification treats reasoning steps as actions in a trajectory and would let a learned policy select paths or data using preference-based rewards.Past reasoning could also shape the data encountered next, creating an interactive feedback loop.Source: §4.8.1 · p. 35

• Reflection corrections add design qualifications: hierarchy involves structure, composition, and architecture, may increase design and manufacturing costs, and does not make anisotropy a sole design principle.The corrections also connect anisotropy with adaptation to functional requirements in different directions.Source: Integrated thinking and reflection model · p. 46
• The showcased recursive algorithm makes three response-improvement iterations using thinking and reflection tokens, with detailed analyses referenced in Table 1 and Figure 7.Source: Integrated thinking and reflection model · p. 48
• The biological-materials response names helical fibers and sacrificial bonds or mineral bridges as toughening features that dissipate energy across scales.The described range extends from molecular to macroscopic scales.Source: Integrated thinking and reflection model · p. 48
• The response attributes graceful failure to hierarchical and periodic structures plus multiscale energy dissipation, especially under sudden and impact loading.The account says these mechanisms help prevent catastrophic failure.Source: Integrated thinking and reflection model · p. 48
• The non-fine-tuned Llama-3.2-3B-Instruct instead interprets hierarchy organizationally, citing authority, specialization, communication, and scalability.The caption identifies this as the base model’s response.Source: Integrated thinking and reflection model · p. 49