Source-linked AI summary
A Human-Centred Approach to Benchmarking LLMs for Parenting Advice
Yunke Zhao, Isobel Voysey, Alastair van Heerden, Rob Hughes, Jun Zhao
TL;DR
Parenting advice from LLMs is subjective and difficult to evaluate with small, accuracy-focused benchmarks. This study benchmarks 15 models across multilingual scenarios using an expert-informed, multidimensional rubric and LLM judges, finding that aggregate scores can obscure rubric-specific weaknesses while models and languages differ in parenting styles.
Problem
Existing evaluations of parenting-oriented LLM advice often use small samples and accuracy-focused criteria, although general parenting questions are subjective and lack universally correct answers.
Method
The study evaluates 15 LLMs across multilingual parenting scenarios with an expert-informed multidimensional rubric, LLM-based judging, and auditable analysis.
Results
Aggregate scores obscure rubric-specific weaknesses, while models encourage different parenting styles and respond differently in English and Chinese.
Takeaways & Limitations
LLM parenting advice should be evaluated across models, languages, rubric dimensions, parenting styles, and auditable generation and judging traces.
Takeaways & Limitations
Scores relied on LLM judges rather than human annotators, with agreement varying by rubric and caution warranted for safety, accuracy, and bias-avoidance dimensions.
Abstract
from arXiv · showhide
People are increasingly using large language models (LLMs) to seek advice, including for parenting. Parenting is a critical and socially sensitive domain. Thus, evaluating advice provided by LLMs requires indicators beyond aggregated information quality benchmarks to consider relational and behavioural elements of the responses. With a multi-dimensional rubric created by parenting experts, this paper evaluates 15 LLMs across 100 parenting scenarios in 2 languages (English and Chinese), using an LLM-as-a-judge method. Results show that aggregate scores can hide rubric item-specific weaknesses, models implicitly encourage different parenting styles, and language influences responses. We highlight the importance of evaluation output auditability and challenges involved in evaluating LLM-generated advice in domains like parenting. Our findings provide important insights for selecting LLMs for direct user engagement and the development of user-facing parenting advice applications.
1 Introduction
Parenting advice from LLMs requires evaluation beyond factual accuracy because it is relational, behavioural, culturally situated, and often lacks a single objectively correct answer. This paper introduces a multilingual, auditable evaluation pipeline and finds multidimensional differences in advice quality and parenting style across models and languages.
- Motivation: Parents increasingly seek advice online, including from LLMs, to support complex decisions about caring for their children.
- Evaluation challenge: Parenting advice quality depends on relational and behavioural qualities, including actionability, tone, and cultural relatability, not factual accuracy alone.
- Study design: The study evaluates 15 LLMs across 100 parenting scenarios in English and Mandarin using rubric-based evaluation and parenting-style analysis.
- Contributions: The scalable pipeline combines scenario generation, response generation, LLM-based judging, automated analysis, failure repair, auditable score traces, and multilingual evaluation.
- Findings: Models vary widely in advice quality and rubric-specific strengths; in English they lean more authoritative, while in Chinese they lean more authoritarian.
2 Background · 3 Methods
The paper motivates human-centred evaluation of LLM parenting advice because parenting support is increasingly sought online and involves subjective, relational, behavioural, and culturally situated guidance. It develops a multi-stage, expert-informed pipeline to assess advice quality and characteristics across models and languages.
- 2.1 LLMs and Parenting: Parents increasingly seek parenting information and social support online, creating a potential role for LLMs in parenting support.This shift contrasts with earlier reliance on family, friends, and childcare professionals.
- 2.1 LLMs and Parenting: Although real-world parental use of LLMs is not established, general public usage and existing online behaviour make LLM-based parenting support worth studying.One cited survey found 21% of adult participants used LLMs to seek health information.
- 2.2 Human-Centered Evaluation of LLMs: Automated benchmarks and human-in-the-loop evaluations are limited by subjective performance criteria and the time and resources required for human assessment.Traditional benchmarks typically target narrowly specified tasks, while human-centred evaluation can be costly to scale.
- 2.2 Human-Centered Evaluation of LLMs: Parenting advice evaluation must address subjective and relational dimensions, including empathy, understandability, actionability, and language framing beyond medical accuracy.Existing parenting work has largely focused on medical information, while relational characteristics have received less attention.
- 2.2 Human-Centered Evaluation of LLMs: The evaluation uses multiple LLM judges and an expert-informed multidimensional rubric to scale assessment while addressing potential judge biases.Identified biases include preferences for longer answers and models from the same family.
- 2.3 Culturally-Sensitive Parenting Advice: Parenting advice is socially and culturally situated, shaping both the LLM–parent relationship and recommended parent–child behaviours, attitudes, and approaches.The paper treats advice as normative guidance rather than purely informational content.
- 2.3 Culturally-Sensitive Parenting Advice: The pipeline evaluates context-specified scenarios, classifies parenting style using responsiveness–demandingness theory, and compares English and Chinese responses.This design examines models’ normative stances on how parents should react to and relate to children.
- 3.1 Overview of Evaluation Pipeline: The methods comprise a four-stage evaluation pipeline assessing the quality and characteristics of LLM parenting advice across multiple models and languages.The stages are presented in Figure 1.
1. Expert-guided parenting scenario generation
The study constructed 100 parenting scenarios spanning developmental stages, family contexts, and diverse parenting topics, then refined them through expert review and translated them into Chinese for cross-language comparison.
- Parenting Scenario Creation: 100 scenarios represented common parenting situations across developmental stages, family contexts, and topics including health, safety, development, discipline, education, sleep, and parent wellbeing.The scenario set also covered nutrition, social-emotional support, special needs, and low-resource contexts.
- Expert Review and Refinement: Expert reviewers with parenting and child-development expertise refined the initial scenarios to improve realism, diversity, and clarity.
- Scenario Translation: Finalised scenarios were translated from English into Chinese to enable cross-language comparison while preserving semantic content, risk level, tone, and key facts.GPT-5 performed the translation under prompt constraints, and a bilingual author reviewed a subset for verification.
Response Validation and Repair
The pipeline validated, repaired, and cleaned generated responses before judging to prevent malformed outputs and formatting differences from distorting advice-quality comparisons. Manual review, bounded regeneration, metadata logging, and preservation of original and cleaned outputs supported auditability.
- Validation: Validation targeted incomplete, malformed, or failed generations that could distort later scores for reasons unrelated to advice quality.Automatic checks used generation metadata and heuristics for API errors, truncation, missing terminal punctuation, cut-off endings, and unusually short fragments.
- Repair: Flagged cases were manually reviewed because surface rules can over-flag complete responses across languages and punctuation systems.Confirmed defects were regenerated for a maximum of three attempts before being merged into canonical answer files.
- Auditability: Repair manifests, rerun logs, and generation metadata recorded changed rows, retry use, suspected truncation, finish reasons, and API errors.These records helped distinguish complete but weak answers from backend failure, empty generation, or length-limited stopping.
- Cleaning: Reasoning-like segments, wrapper formats, and explicit final-answer markers were stripped to reduce output-shape confounding during judging.Original outputs, cleaned outputs, and metadata recording whether and how many characters were removed were retained.
Rubric-Based Judging
The study uses an expert-informed, eight-dimensional rubric with LLM-as-a-judge scoring to evaluate open-ended parenting advice beyond a single reference answer. Each rubric receives repeated 0–100 judgments in a fixed JSON format, enabling consistent parsing and multidimensional comparison.
- Rubric-Based Judging: LLM-as-a-judge was selected because parenting advice is open-ended and cannot be evaluated against a single reference answer.The rubric was inspired by HealthBench and informed by parenting experts.
- Rubric-Based Judging: The evaluation uses eight rubrics: accuracy, safety, helpfulness, empathy, completeness, bias avoidance, limitation awareness, and communication.Together, these dimensions cover factual grounding, harm avoidance, practical actionability, supportive tone, coverage, stereotypes, professional referral, clarity, and clarifying questions.
- Rubric-Based Judging: The multidimensional rubrics distinguish advice that is practical but weak in safety from advice that is safe but too vague to be useful.This enables response quality to be analysed across separate dimensions rather than only through an aggregate assessment.
- Rubric-Based Judging: 0 to 100 scores and one short comment were assigned for each rubric, with outputs constrained to a fixed JSON schema for consistent parsing and merging.The primary LLM judge evaluated each response three times, and each rubric’s final score was computed as the mean across repeats.
Parenting Style Classification
Parenting style was evaluated separately from response quality using responsiveness and demandingness, capturing how advice balances warmth, structure, reassurance, and boundaries. Structured scores were converted into probabilities for four parenting styles to compare variation across models and languages.
- Framework: The framework defines responsiveness as warmth and emotional support, and demandingness as structure, expectations, behavioural guidance, and boundary-setting.Together, these dimensions define authoritative, authoritarian, permissive, and uninvolved parenting styles.
- Classification method: Each answer receives Responsiveness (R) and Demandingness (D) scores from 0 to 1, which are converted into a four-style probability distribution.The formulas are authoritative = R × D, authoritarian = (1 − R) × D, permissive = R × (1 − D), and neglectful = (1 − R) × (1 − D), with probabilities normalised to sum to one.
- Interpretation: Separating style classification from rubric scoring preserves a distinction between advice quality and interpersonal stance.Rubric scores alone may not capture whether advice differs in warmth, structure, reassurance, or boundary-setting.
- Analysis: The resulting scores were used to analyse parenting-style variation across models and languages.The analysis matrix retained parenting style classification probabilities alongside generated answers, judge scores, rubric means, and comments.
- Comparisons: The analysis compared models within each language, languages within each model, and robustness across GPT and Deepseek judges.These comparisons were supported by summary statistics and visualisations.
4 Results
The benchmark reveals clear and stable performance differences across models, while rubric profiles, language, parenting style, and judge agreement expose important variation hidden by overall scores. Language changes both advice quality and advisory stance, and some rubric dimensions are more judge-sensitive than others.
- Overall performance: GPT-5.2 ranks strongest overall, followed by GPT-5 Nano and Kimi K2 Thinking, while GLM-4 9B and Llama 3.1 8B rank lower across most scores.Pairwise win-rate comparisons similarly show GPT-5.2 dominating almost all comparisons, with strong positions for GPT-5 Nano and Kimi K2 Thinking.
- Rubric profiles: Strong models have distinct rubric profiles: Kimi K2 Thinking excels in Helpfulness, Empathy, and Communication, whereas DeepSeek V3.1 is relatively stronger in Accuracy, Safety, and Bias Avoidance.GPT-5.2 and GPT-5 Nano are strong and balanced, while GPT-4o Mini shows larger deficits, especially in Completeness and Limitation Awareness.
- Language differences: Cross-language score differences are heterogeneous and model-dependent: GLM-4 9B rises most in Chinese, whereas Llama 3.1 8B and GLM-4.6 show the clearest declines.Chinese answers often increase Completeness and Limitation Awareness while declining in Accuracy and Empathy; high-difference safety and health scenarios vary in direction.
- Parenting styles: Higher-performing models are more likely to provide structured advice, while Chinese responses generally become more authoritarian or neglectful and less warm than English responses.GPT-5.2 and Kimi K2 Thinking retain the strongest authoritative profiles, whereas GPT-4o Mini and GLM-4 9B show weaker authoritative cores and larger neglectful shares.
- Judge robustness: Judge agreement is strongest for Completeness, Empathy, Helpfulness, and Communication, while Accuracy, Safety, and Bias Avoidance are more judge-sensitive.Limitation Awareness remains relatively aligned but has a larger spread between judges.
5 Discussion
The discussion argues that parenting advice from LLMs requires multidimensional, multilingual, and auditable evaluation because aggregate scores can obscure rubric-specific weaknesses, language-dependent behaviour, and implicit parenting values. It also frames these systems as relational technologies that must balance safety with culturally and contextually responsive support.
- Evaluation implications: LLM parenting advice varies across models, languages, and evaluation dimensions, so a single aggregate quality score is insufficient.Models with similar total scores can have different strengths, including practical guidance, communication, and other rubric dimensions.
- Multilingual evaluation: Cross-language evaluation is necessary because models may improve or worsen in Chinese, with uneven effects across rubrics and scenarios.Language can alter both practical advice quality and relational aspects of model behaviour, rather than functioning as a simple translation condition.
- Parenting styles: Higher-performing English models more often produced authoritative advice, whereas many models shifted toward authoritarian or neglectful profiles in Chinese.The findings do not establish one language as better, but indicate that language may change the advice’s normative stance and implied parent-child relationship.
- System design: Parenting advice systems should support pluralistic values and context-sensitive interaction within safe boundaries, making assumptions explicit and offering options when appropriate.Safety remains essential, including avoiding advice that normalises violations of children’s rights; systems should not simply follow any user preference.
- Normative dimensions: LLMs do not provide value-neutral parenting guidance, implicitly recommending different balances of warmth and structure in parent-child interaction.Parenting-style analysis makes these normative dimensions visible without claiming that authoritative advice is always best or that style descriptors fully determine quality.
- Auditability: Robust auditability is essential because response validation and repair prevent generation failures, truncation, wrapper text, and reasoning-like segments from distorting quality judgments.LLM-as-a-judge evaluation provides a meaningful but non-definitive signal: judges substantially agreed overall, while agreement varied by rubric and was lower for Accuracy, Safety, and Bias Avoidance.
6 Limitations and Future Work
The study’s limitations concern reliance on LLM judges, translated rather than culturally originating scenarios, and remaining manual and prompt-design challenges. Future work should validate evaluations with diverse human experts, develop culture-specific scenarios, and improve and scale the pipeline.
- Limitations of evaluation: LLM judges replaced human annotators, with variable agreement across rubrics and caution warranted for safety, accuracy, and bias-avoidance dimensions.Future validation should compare rubric scores and parenting-style classifications with parenting experts, parents, and culturally diverse annotators.
- Limitations of evaluation: The English and Chinese evaluations translated the same 100 scenarios, enabling controlled comparison but incompletely representing culturally and linguistically originating parenting situations.Future work should develop language-specific and culture-specific scenario sets rather than relying only on translation.
- Future work: The pipeline still contains manual steps, including review of flagged generations, and requires iterative refinement of rubrics and judging prompts.Improvements are particularly important for culturally sensitive cases and high-risk scenarios where small framing differences may substantially affect advice.
7 Conclusion
The work argues that evaluating LLM parenting advice requires accounting for variation across models, languages, rubric dimensions, parenting styles, and judges. It presents a multilingual, auditable evaluation pipeline combining expert-informed scenarios, response validation and repair, rubric-based judging, parenting-style classification, and automated analysis.
- Evaluation of LLM parenting advice must address variation across models, languages, rubric dimensions, parenting styles, and judges.
- The multilingual pipeline combines expert-informed scenarios, responses from 15 models, validation and repair, rubric-based LLM judging, parenting-style classification, and automated analysis.
- The evaluation pipeline produces auditable traces, while aggregate scores can obscure findings.
A Extended Results · A.1 Rubric-Level Means
The extended results include rubric-item means reported separately by model.
- A.1 Rubric-Level Means: Table 2 reports means for each rubric item by model.