Source-linked AI summary
METIS: Mentoring Engine for Thoughtful Inquiry & Solutions
Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra
TL;DR
Students often lack expert research mentorship, motivating the question of whether AI can guide undergraduates from initial ideas toward publishable papers. METIS combines stage-aware routing with literature, guideline, methodology, and memory tools, and evaluations found stronger guidance than strong chat baselines, especially in document-grounded stages, while retaining known failure modes.
Problem
Students often lack access to expert research mentorship, raising whether an AI system can guide undergraduates from an initial idea toward a publishable paper.
Method
METIS is a tool-augmented, stage-aware research-writing assistant combining literature search, curated guideline retrieval, methodology checks, session memory, and evaluation across six writing stages.
Results
METIS outperforms Claude Sonnet 4.5 on pairwise preferences, yields higher student-perspective scores across stages, and slightly improves multi-turn outcomes versus GPT-5, with strongest gains in document-grounded stages D–F.
Takeaways & Limitations
A focused, tool-aware mentoring workflow can raise guidance quality over strong chat baselines, while stage-aware routing and grounding align with the largest gains.
Takeaways & Limitations
The evaluation covers six text-centric writing stages, uses rubric-based proxies rather than longitudinal learning outcomes, and compares only GPT-5 and Claude Sonnet 4.5.
Abstract
from arXiv · showhide
Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware assistant with literature search, curated guidelines, methodology checks, and memory. We evaluate METIS against GPT-5 and Claude Sonnet 4.5 across six writing stages using LLM-as-a-judge pairwise preferences, student-persona rubrics, short multi-turn tutoring, and evidence/compliance checks. On 90 single-turn prompts, LLM judges preferred METIS to Claude Sonnet 4.5 in 71% and to GPT-5 in 54%. Student scores (clarity/actionability/constraint-fit; 90 prompts x 3 judges) are higher across stages. In multi-turn sessions (five scenarios/agent), METIS yields slightly higher final quality than GPT-5. Gains concentrate in document-grounded stages (D-F), consistent with stage-aware routing and groundings failure modes include premature tool routing, shallow grounding, and occasional stage misclassification.
Introduction
METIS addresses limited access to expert research mentorship by guiding undergraduates from rough ideas toward publishable conference papers. Its contributions combine a practical workflow, inspectable tools, empirical comparisons, and open reproducibility materials.
- Introduction: METIS guides undergraduates from ideation to a publishable conference paper while grounding advice in instructions, guides, and literature.It also preserves context across sessions for progress over multiple weeks.
- Introduction: The paper contributes a practical mentoring workflow and stage-aware evaluation that turns research mentorship into concrete tasks.
- Introduction: METIS combines literature search, guidelines retrieval, methodology checks, and memory in a simple, inspectable system.
- Introduction: The study compares METIS with GPT-5 and Claude Sonnet 4.5 using single-turn judgments and multi-turn tutoring.
- Introduction: Open prompts, logs, and scripts support reproduction of the reported results.
Related Work
METIS builds on tool-augmented assistants, autonomous scientific agents, personalized tutoring, and assistant-evaluation research. It differs by targeting interactive, stage-aware research-writing mentorship rather than generic tool use, autonomous discovery, or broad tutoring.
- Tool-augmented assistants: Tool-augmented assistants combine reasoning with external tools through modular designs such as ReAct, MRKL, and Toolformer.METIS applies this line to research mentorship and pairs retrieval with lightweight self-critique for evidence and methodology checks.
- Scientific-process agents: Autonomous scientific-process agents target end-to-end research automation, whereas METIS evaluates interactive mentorship and learner progress under constraints.
- Education: Personalized tutoring frameworks structure learning around goals, skills, and progression; METIS focuses this approach on research-writing mentorship and stage-aware evaluation.
- Evaluation: Assistant evaluation commonly combines pairwise preferences with rubric scores, while this work adds student-focused trends and uncertainty estimates.
- Agent benchmarks: Agent benchmarks emphasize multi-turn capability and task success, but METIS uses a systems-level comparison for education-oriented research mentoring.
- Positioning: The evaluation groups prompts by six writing stages and routes tools according to stage-aware research-writing tasks.
System Overview
METIS is a tool-augmented, stage-aware research-writing assistant whose lightweight router selects tools and whose memory preserves context across turns. Its structured prompt also exposes reasoning checks and guides learners through six writing stages.
- System Overview: METIS mediates between the interface and tools for curated guidelines, literature retrieval with citation checks, methodology checks, session memory, and interaction logging.
- System Overview: Replies include Intuition and Why this is principled self-checks, along with guidance across stages A–F from pre-idea through final writing.
- System Overview: The stage detector infers the current stage from conversation context to tailor tool selection and guidance depth.
- System Overview: Prompt instructions implement stage detection and routing, while responses state the inferred stage, track progress, and suggest concrete next actions.
Evaluation
The evaluation compares METIS with GPT-5 and Claude Sonnet 4.5 across six writing stages using pairwise preferences, student-persona rubrics, evidence checks, and short multi-turn tutoring. Results assess both learner-facing quality and evidence/compliance behavior under reproducible protocols.
- Benchmark: 90 single-turn prompts cover six writing stages, with 15 prompts per stage and realistic learner personas, topics, and constraints.
- Pairwise Evaluation: METIS wins 71% against Claude Sonnet 4.5 and 54% against GPT-5 overall in LLM-judge pairwise preferences.
- Benchmark: For document-grounded stages D–F, public arXiv papers are attached in progressively fuller forms, and three diverse judges score each item.
- Pairwise Evaluation: Pairwise preferences are aggregated per stage and overall with Wilson 95% confidence intervals, excluding ties.
- Student-Perspective Evaluation: METIS tracks above both baselines on student-perspective clarity, actionability, and constraint-fit across stages A–F.
- Student-Perspective Evaluation: Student-perspective scores use anchored 0–2 rubrics for clarity, actionability, constraint-fit, and confidence-gain, rated by three LLM judges.
- Multi-Turn Evaluation: Five multi-turn scenarios per agent evaluate final quality from full conversation histories, with success defined as the earliest turn reaching an overall score of at least 1.6.
Results and Analysis
METIS performs best relative to the baselines in later, document-grounded writing stages, while also showing strong evidence and compliance measures. In multi-turn tutoring, it achieves slightly higher final quality than GPT-5 at the cost of modestly more turns, and survey respondents report positive experiences.
- Single-turn and per-stage results: METIS outperforms both baselines overall, with stronger advantages in document-grounded stages D–F.The later stages involve plans, ablations, and submission checks grounded in documents.
- Evidence and compliance: METIS shows near-perfect citation validity and strong stage awareness, leads Claude on evidence integrity, and is close to GPT-5 on RAG fidelity.These measures come from expert judge runs across stages A–F.
- Multi-turn outcomes: Final score is higher than GPT-5 by +0.088 (p=0.043) and comparable to Claude (p=0.289) in multi-turn sessions.The evaluation used five conversations per system.
- Human feedback: 50 testers gave mean ratings of 4.14 for ease of use, 4.22 for helpfulness, 4.08 for goal understanding, and 4.30 for overall experience.Ratings used four 1–5 Likert items and were descriptive.
Discussion and Limitations
The evaluation is limited by small samples, proxy-based student measures, restricted task and baseline coverage, unpriced overhead, and residual retrieval and routing risks.
- Measurement: With 90 prompts and five multi-turn scenarios per system, some effects remain directional rather than definitive.The guidelines-usage split is observational and not causal.
- Scope: The study covers six text-centric writing stages but excludes lab workflows, hardware experiments, and non-text modalities.Student-judge trends are rubric-based proxies rather than longitudinal learning outcomes.
- Practical considerations: METIS trades modest turn-time overhead for higher multi-turn quality, but dollar cost is not analyzed.The practical trade-off is reported without a cost evaluation.
- Practical considerations: Retrieval risks include over-reliance and inadvertent text reuse, while observed failures include premature tool routing, shallow grounding, and rare stage misclassification.Simple routing heuristics mitigate the observed failure modes.
Conclusion and Future Work
METIS is presented as a focused, tool-aware workflow for research mentorship across defined writing stages. The conclusion identifies learned routing and evidence selection, plus component ablations, as next steps for isolating the sources of its gains.
- Conclusion: METIS outperforms Claude Sonnet 4.5 on pairwise preferences, scores higher from the student perspective across stages, and slightly improves multi-turn outcomes versus GPT-5.Its strongest gains occur in document-grounded stages D–F.
- Conclusion: The paper links its gains to stage-aware routing and grounding while acknowledging shallow grounding and occasional stage misclassification.These remain failure modes of the system.
- Future work: A next step is to learn the router and evidence selector from tool-trace logs and run learned-versus-heuristic routing and component-dropout ablations.These experiments are intended to isolate which modules drive the gains.
B Evaluation Scripts (Self-Contained)
The supplementary archive provides scripts and machine-readable prompt and scenario files for reproducing common evaluation tasks.
- Evaluation scripts: Scripts for common evaluation tasks write artifacts into a local outputs/ folder by default.Placeholders in angle brackets are intended to be replaced with local choices.
- Evaluation scripts: The supplementary archive includes machine-readable prompt and scenario files used in the experiments.The named files are evals_single_- turn.jsonl and scenarios.jsonl.
B.1 Single-Turn Pairwise Runner
The evaluation runner compares research assistants across six writing stages using pairwise judgments, student-focused rubrics, and short multi-turn tutoring scenarios. Its prompts and scoring emphasize actionable, constraint-fitting guidance and explicit failure-mode checks.
- Pairwise evaluation: Three LLM judges compare METIS, GPT-5, and Claude Sonnet 4.5 in pairwise evaluations with ties allowed.The runner aggregates overall and per-stage win rates with Wilson 95% confidence intervals.
- Student-centered scoring: Student-focused rubrics score clarity, actionability, and constraint fit on 0–2 scales, alongside binary checks for path readiness and flagged failure modes.Judges assess whether students can execute concrete next steps within their time, compute, and skills constraints.
- Multi-turn evaluation: Five short multi-turn scenarios per system are scored for final quality using student judges.The scenarios cover the same six-stage progression from pre-idea through final paper.
- Evaluation principles: The rubric penalizes generic advice and missing prerequisites while rewarding specific sequenced steps, uncertainty reduction, and explicit respect for student constraints.Critical gaps such as unavailable data, IRB requirements, or missing baselines trigger a failure-mode flag.
- Evaluation inputs: 90 single-turn records cover six stages, A–F, with 15 prompts per stage and metadata for personas and constraints.The released prompt set includes expected checks and stage labels.
C.6 Planning, problem selection, and experiments
The planning framework constrains research guidance through staged feasibility checks, short gated roadmaps, explicit experiment design, and follow-up guardrails. It also supports observational analysis of whether guideline-tool usage coincides with pairwise outcomes.
- Problem selection: Proposed problems are scored on importance, tractability, surprise, generality, and mechanistic payoff, with iteration or scope reduction recommended below 10/15.The tractability criterion is evaluated within the user’s resource and time constraints.
- Experiment suggestions: Each experiment plan specifies its objective, hypothesis, resources, evaluation criteria, outcome interpretations, dependencies, pitfalls, and follow-ups.The framework sequences these elements so students can act on the plan directly.
- Experiment design: An experiment card records the hypothesis, falsifier, minimal test, variables, expected patterns, analysis plan, and stop rule before a study is run or recommended.The card is reused when interpreting results.
- Follow-up guardrails: When earlier questions remain unanswered, the mentor restates the critical question and offers up to three mutually exclusive next steps before broader analysis.Each option is scoped to no more than two hours.
- Guideline-usage analysis: Conditional win rates are computed from existing tool traces, but guideline usage is observational and non-randomized, so differences are not causal estimates.Guidelines are preferentially invoked on more ambiguous or early-stage prompts, concentrating harder cases in the invoked bucket.
E Additional Multi-Turn Results
Scenario-level analysis exposes variability in METIS’s five multi-turn tutoring cases per agent. Higher final scores for Mentor are consistent across scenarios, while efficiency differences remain modest.
- Scenario-level results: Higher final scores for Mentor are consistent across five scenarios per agent, with modest efficiency differences.Figure 5 complements aggregate multi-turn results by showing scenario-level variability.