Source-linked AI summary
PA-CoT: Profile-Adaptive Chain-of-Thought for Personalized Nutritional Consulting
Evgenii Garmashov, Nikita Kulin, Artur Khairullin, Viktor Zhuravlev, Daniil Sukhorukov, Mikhail Mozikov, Ilya Makarov, Sergey Muravyov
TL;DR
Nutritional consulting prompts often pass structuredly relevant user information as an unstructured block without a dedicated profile-analysis step. The paper introduces PA-CoT, which analyzes profiles explicitly before generation, and QPA, a 200-sample benchmark for evaluating this setting. PA-CoT ranks first overall and on Personalization and Safety, while the authors caution that safety still requires qualified human validation before deployment.
Problem
Widely used nutritional-consulting prompts pass user profiles as unstructured text without a dedicated analysis step, limiting structured personalization.
Method
PA-CoT is a multi-stage prompting method that treats profile analysis as a standalone stage before response generation, evaluated with QPA structured profiles.
Results
PA-CoT ranks first on Avg (4.21), Personalization (4.71), and Safety (4.68) among 12 approaches, with non-overlapping 95% CIs over Self-Refine on the latter two criteria.
Takeaways & Limitations
The results support dedicated profile analysis as a distinguishing design choice for personalization and safety in this nutritional-consulting evaluation.
Takeaways & Limitations
Safety is judged by an LLM rather than clinicians or registered dietitians, so qualified human validation is required before real-world deployment.
Abstract
from arXiv · showhide
In health and nutrition consulting, widely used prompting methods pass the user profile as an unstructured block without a dedicated analysis step, leaving personalization as a critical structural gap. We introduce PA-CoT (Profile-Adaptive Chain-of-Thought), a multi-stage prompting method that treats profile interpretation as an explicit, standalone reasoning step prior to response generation. To enable systematic evaluation, we introduce the QPA (Question--Profile--Answer) benchmark -- 200 nutritional consulting samples with structured user profiles scored on four criteria. In a comparative study against 11 comparison methods (CoT, Few-Shot, Role Prompting, DSPy, TextGrad, Self-Refine, and others, plus a Zero-Shot Baseline; 12 total including PA-CoT), PA-CoT achieves the best average score (4.21 on the G-Eval 1--5 scale) and leads on both Personalization (4.71 vs. 4.39) and Safety (4.68 vs. 4.52) with non-overlapping 95\% confidence intervals over the nearest competitor -- the only method to simultaneously top both criteria. The results confirm that an explicit profile-analysis step is the key driver of personalization gains over widely used prompting approaches.
1. Introduction
The paper addresses a structural personalization gap in nutritional consulting: common prompting methods pass profiles as unstructured text without explicitly analyzing them. It introduces QPA and PA-CoT to evaluate and address this gap through structured profiles and dedicated profile analysis.
- Ignoring concrete attributes such as age, weight, goals, preferences, and nutritional statistics produces recommendations that differ fundamentally from personalized advice.
- Traditional methods often omit a dedicated profile-analysis step, passing user information as one unstructured text block.
- QPA provides a benchmark format with structured user context for nutritional consulting, unlike benchmarks focused on macronutrient estimation or unstructured histories.
- PA-CoT treats profile analysis as a dedicated stage within a multi-stage prompting pipeline and evaluates the three applicable stages offline.
- Evaluation on 200 samples reports the best average score among 12 compared approaches, with advantages in personalization and safety.
- PA-CoT differs from agent frameworks, weight-optimization methods, and task-decomposition prompts by focusing on reasoning over a given structured profile at the prompting level.
2. The QPA Format and Benchmark
QPA extends standard question answering with structured profiles and specialist references, using a filtered and partially populated benchmark of 200 nutritional consulting samples. Its construction is designed to reflect real product conditions while preserving a reference answer for evaluation.
- 2.1. QPA Format: Each QPA sample contains a freeform question, a structured user profile, and a specialist reference answer.
- 2.1. QPA Format: The profile includes demographic, dietary, nutritional-statistics, sleep, and stress fields, with unknown values allowed when data is untracked or undisclosed.
- 2.2. Benchmark Construction: The source corpus is Medical Alpaca, approximately 23,000 medical question-answer pairs drawn from open forums.
- 2.2. Benchmark Construction: A binary LLM classifier excludes predominantly medical questions and retains questions where nutrition is primary or co-equal.
- 2.2. Benchmark Construction: An LLM extracts available QPA schema fields from each selected question.
- 2.2. Benchmark Construction: Approximately 20% of profile fields are populated on average, mirroring partial data availability in real-world products.
- 2.2. Benchmark Construction: The final benchmark contains 200 samples, each paired with a reference answer from the source corpus.
3. The PA-CoT Method
PA-CoT makes profile interpretation an explicit reasoning stage before response generation. Its architecture also specifies interactive profile completion and independent safety verification, although fixed benchmark experiments omit the interactive stage.
- 3. The PA-CoT Method: PA-CoT is a multi-stage pipeline whose core design treats user profile analysis as a standalone stage before response generation.
- 3.1. Stage 1: Profile Analysis: Stage 1 identifies critical issues grounded in profile values and constructs a relevance matrix linking each issue to necessary profile fields.
- 3.1. Stage 1: Profile Analysis: The relevance matrix shifts profile interpretation to a dedicated auxiliary model, giving the generator ready-made critical points.
- 3.2. Stage 2: Profile Completeness Check with Interactive Clarification: Stage 2 checks whether necessary fields are populated and can trigger follow-up questions, profile updates, and renewed Stage 1 analysis.
- 3.2. Stage 2: Profile Completeness Check with Interactive Clarification: In the sample relevance matrix, height, activity, and kcal/day are required but absent, so Stage 2 would request those fields.
- 3.3. Stage 3: Response Generation: Stage 3 generates a chain-of-thought response augmented with Stage 1’s critical points rather than independently selecting profile aspects.
- 3.3. Stage 3: Response Generation: Stages 1 and 4 use temperature 0.3, while Stage 3 uses 0.7, matching baseline methods for experimental fairness.
- 3.4. Stage 4: Safety Verification: Stage 4 detects harmful recommendations, supports corrective escalation or referral, blocks unresolved dangers, and appends additional safety caveats when needed.
4. Experiments
The study compares PA-CoT with 11 alternatives on the 200-sample QPA benchmark using G-Eval across four criteria. PA-CoT leads overall and particularly on personalization and safety, while optimizer complexity does not predict personalization.
- 4.1. Compared Methods: PA-CoT is compared with 10 named prompt-engineering methods plus a Zero-Shot Baseline, for 12 methods total.
- 4.1. Compared Methods: All methods receive the question together with the user profile.
- 4.2. Infrastructure and Metrics: The experiments run all methods on GPT-4o Mini, while Qwen3-235B-A22B-Instruct-2507 serves as the deterministic G-Eval judge.
- 4.3. Results: PA-CoT ranks first on Avg (4.21), Personalization (4.71), and Safety (4.68).
- 4.3. Results: Personalization improves by 0.32 points over Self-Refine (4.39), with non-overlapping 95% confidence intervals.
- 4.3. Results: Relative to CoT, PA-CoT gains 0.26 on Personalization and 0.58 on Safety through dedicated profile analysis and separate safety verification.
- 4.3. Results: Personalization spans 2.34–4.71 across methods and shows the widest spread, while CoT’s 4.45 interval overlaps with PA-CoT’s.
- 4.3. Results: DSPy (3.14), PhaseEvo (3.20), and AMPO (3.81) score below CoT (4.45) on Personalization despite using automatic optimization.
5. Conclusion and Future Directions
PA-CoT separates profile analysis into a dedicated reasoning step and ranks first across average performance, Personalization, and Safety on 200 QPA samples.
- PA-CoT ranked first on Avg, Personalization, and Safety among 12 compared approaches.The method adds a dedicated profile-analysis step, while also incorporating safety verification.
Impact Statement
PA-CoT is a prompting-level method for reasoning over structured profiles in health and nutrition consulting. Its prototype is intended to operate under professional oversight because LLM-judged safety may miss context-specific risks.
- PA-CoT is not a substitute for professional medical or dietetic advice because safety is evaluated by an LLM judge rather than clinicians or registered dietitians.The paper recommends qualified human validation before real-world deployment, especially for vulnerable populations.
- PA-CoT treats an available structured profile as input and focuses on reasoning over it, without prescribing profile collection or maintenance.This distinguishes it from system-level frameworks that manage external data collection and retrieval.
- QPA uses explicit fields such as age, weight, goals, and nutritional statistics rather than unstructured textual user histories.The structured format is intended to match real health-product architectures and support PA-CoT.
- PA-CoT belongs to decomposed-prompting approaches that stage reasoning instead of generating an answer in a single pass.Related methods include plan-and-solve, least-to-most, and skeleton-of-thought prompting.
B. QPA Sample Examples
The QPA examples span demographic groups, profile completeness, question types, and safety scenarios. They include sparse, medically constrained, statistically rich, and safety-critical profiles.
- Benchmark diversity: The QPA examples vary across demographics, profile completeness, question type, and safety scenarios.Populated fields are distinguished from unknown fields in the sample presentations.
- Profile completeness: Only sex, age, weight, and goal are known for the adolescent male weight-loss example, illustrating a sparse profile.Approximately 20% of profile fields are populated on average across the benchmark.
- Profile completeness: The 47-year-old female example combines disability, obesity, hypothyroidism, and Hashimoto’s disease without nutritional statistics.It tests personalization under a cold profile while respecting medical constraints.
- Profile completeness: The 43-year-old female example provides detailed intake, activity, sleep, stress, expenditure, and low_energy_intake data for a weight-loss goal.Its 7-day statistics include 1350 kcal/day intake and 2300 kcal/day expenditure.
- Safety scenarios: The 9-month-old infant example has an empty statistics profile and concerns nighttime feeding and water supplementation.The scenario is safety-critical because harmful advice could carry significant risk.
C. Sample Walkthrough
A fatigue case illustrates progressive gains from Baseline to Self-Refine to PA-CoT, with PA-CoT using profile data more specifically and achieving the strongest reported personalization and safety scores.
- Baseline: Baseline scored Personalization 1/5 and Safety 3/5 by giving generic fatigue causes while ignoring age, goals, and testosterone data.The evaluator described the response as entirely generic.
- Self-Refine: Self-Refine scored Personalization 3/5 and Safety 4/5 but remained a broad, reusable checklist that did not explicitly use age or testosterone.The evaluator characterized it as partially adapted but still templated.
- PA-CoT: PA-CoT scored Personalization 4/5 and Safety 5/5 by identifying and referencing the user’s age, testosterone level, and chronic fatigue.Its stages produced profile-linked recommendations and added safety caveats against self-treatment.
- Comparison: Baseline, Self-Refine, and PA-CoT improved stepwise from pers=1, safe=3 to pers=3, safe=4 and then pers=4, safe=5.The sample summary attributes the simultaneous optimization to separating profile analysis from safety verification.
- Evaluation: The G-Eval rubrics score four criteria on integer scales from 1 to 5 using criterion-specific reasoning and token-probability weighting.Qwen3-235B-A22B-Instruct-2507 performs the evaluations at temperature 0.
E. PA-CoT Version Ablation
The ablation shows that separating personalization from safety verification resolves the inverse relationship observed in earlier PA-CoT versions. V4 achieves strong scores on both criteria through dedicated architectural stages.
- Earlier versions: 4.44 Personalization and 4.43 Safety characterize V1 after adding an expert nutritionist role directive to Stage 3.The role grounded responses in a professional persona and provided implicit safety guardrails.
- Earlier versions: Personalization rises from 4.44 to 4.48 in V2 after removing the role directive, while Safety falls from 4.43 to 4.21.The role directive also suppressed unsafe content, producing the observed trade-off.
- V4 architectural separation: V4 resolves the inverse correlation between Personalization and Safety observed in V1–V3 through architectural separation.The intermediate-version table evaluates 200 samples using the G-Eval 1–5 scale with 95% confidence intervals.
- V4 architectural separation: 4.71 Personalization and 4.68 Safety are achieved when Stage 3 focuses on personalization and Stage 4 handles safety independently.Removing the Stage 3 remark and adding dedicated safety verification breaks the earlier trade-off.
- Comparison methods: DSPy, Self-Refine, PhaseEvo, TextGrad, AMPO, Mixture of Prompts, and Meta-Prompting provide the comparison methods used alongside PA-CoT.The implementations span prompting, meta-prompting, automatic optimization, and evolutionary prompt optimization approaches.
G. Limitations and Future Work
The study’s main limitations concern benchmark scale, profile quality, reference-answer bias, and experimental scope. Future work includes larger benchmarks, broader domains, interactive clarification, and stronger generators.
- Benchmark scale: 200 samples detect large effects but limit power for small differences, especially on Correctness and Completeness.The reported 95% confidence-interval widths range from ±0.05–0.18, and expansion to 500–1000 samples is planned.
- Profile quality: Profiles are LLM-extracted from question text, limiting completeness compared with real product data that accumulates user history automatically.Reference answers may bias Correctness scores, while Personalization is unaffected because it measures profile use rather than reference alignment.
- Scope and future work: Experiments are limited to nutritional consulting and GPT-4o Mini, while temperature differences across PA-CoT stages confound architectural effects.Future work proposes uniform-temperature ablations, additional structured-context domains, interactive clarification, and larger generator models.
H. Generator Robustness: GLM-4.7 Results
Using GLM-4.7, PA-CoT remains first on Personalization and preserves the method ranking seen with GPT-4o Mini. Its typical three-call path costs less than maximum-cycle Self-Refine while retaining a personalization advantage.
- Generator robustness: 4.39 Personalization places PA-CoT first on GLM-4.7, followed by TextGrad at 4.19, Self-Refine at 4.14, and Zero-Shot Baseline at 2.21.The ranking is preserved across generators, although absolute scores are slightly lower on GLM-4.7.
- Generator robustness: The stable relative ordering indicates that PA-CoT’s profile-adaptive advantage is not an artifact of GPT-4o Mini’s instruction-following characteristics.The robustness rerun used GLM-4.7 as generator and the same Qwen3 judge.
- Call count and cost: 3 calls define the typical PA-CoT path, while DANGER escalation can add Stage 3 and Stage 4 calls per retry, up to 2 retries.ADDITIONS appends safety caveats without an extra call but increases output tokens.
- Call count and cost: 0.32 points of personalization gain over Self-Refine comes at lower inference cost for the typical PA-CoT path.The typical path costs approximately 3× a single-call method and 3× less than Self-Refine at maximum cycles.