Evaluation of OpenAI o1: Opportunities and Challenges of AGI

A.6 Low-Resource Language Translation

• No substantive low-resource translation result is provided in the supplied passages.The available excerpts consist only of pagination markers rather than findings, methods, or metrics.Source: §A.6 · p. 6
• The first supplied translation excerpt contains only the page marker 185.No claim about translation performance can be grounded in this excerpt.Source: §A.6 · p. 6
• The available material does not establish a low-resource translation capability, benchmark, or comparison.Neither supplied translation passage contains substantive content beyond pagination markers.Source: §A.6 · p. 6
• The second supplied translation excerpt contains only the page marker 188.No method, comparison, or outcome is stated in this excerpt.Source: §A.6 · p. 6

• The passage provides no substantive content beyond a dotted line ending in page number 191.Source: §A.7 · p. 6

A.8 Logical Reasoning

• A.8 Logical Reasoning has no substantive passage text in the supplied evidence.No result, method, comparison, or metric is provided for this subsection.Source: §A.9 · p. 6

A.9 High School Level Math Competition .

• A.9 High School Level Math Competition is represented only by a page-number marker.The supplied passage does not state any mathematical result, method, comparison, or metric.Source: §A.9 · p. 6
• The first supplied passage contains only a locator ending in 199.It contains no prose describing the paper’s contribution or findings.Source: §A.9 · p. 6
• No quantitative performance result can be recovered from the supplied passages.The visible text contains page-number markers rather than metrics or comparisons.Source: §A.9 · p. 6

A.8 Logical Reasoning

• No distinction between logical reasoning and high-school mathematics findings is supported by the supplied text.The passages do not include claims that would permit a substantive comparison or synthesis.Source: §A.9 · p. 6

A.9 High School Level Math Competition .

• The second supplied passage contains only a locator ending in 205.It likewise provides no substantive evidence for summarizing either subsection.Source: §A.9 · p. 6

A.10 College-level Math Problems .

• The college-level math subsection has no supplied substantive passage text beyond a page reference.The input provides no result, method, metric, or conclusion for this subsection.Source: §A.11 · p. 6

A.11 Electronic Health Record Diagnosis .

• The electronic health record diagnosis subsection is represented only by page references, without substantive claims.No diagnosis performance, comparison, dataset, or method is stated in the supplied text.Source: §A.11 · p. 6

• No quantitative result is recoverable from the supplied passages.The visible numbers are page references rather than reported metrics.Source: §A.11 · p. 6
• No comparison with baselines or competing systems is stated.The passages do not identify candidates, baselines, or outcomes.Source: §A.11 · p. 6
• No experimental conditions, datasets, or evaluation procedures are described.The supplied text contains no methodological details for either subsection.Source: §A.11 · p. 6
• No limitations, mechanisms, or scope qualifiers can be established from the supplied passages.Only page-number references are available for summarization.Source: §A.11 · p. 6

A.12 Stochastic Processes in Statistics .

• The passage is limited to a page-reference line ending in 224, with no reported methods, results, or conclusions.No substantive section content is present in the supplied passage.Source: §A.12 · p. 6

• The passage provides no findings, methods, or conclusions about analogical reasoning, only the page number 234.Source: §A.15 · p. 6

A.17 Public Health Policy Analysis .

• A.17 Public Health Policy Analysis is not substantively represented in the supplied passages.No passage provides findings, methods, metrics, or conclusions for this subsection.Source: §A.19 · p. 6

A.19 Medical Knowledge Question Answer .

• A.19 Medical Knowledge Question Answer is referenced only by page numbers 243, 244, and 259.The supplied passages contain no substantive medical question-answer results or analysis.Source: §A.19 · p. 6

A.18 Medical Genetics and Genomics Reasoning .

• A.18 Medical Genetics and Genomics Reasoning is not substantively represented in the supplied passages.No passage provides findings, methods, metrics, or conclusions for this subsection.Source: §A.19 · p. 6

• The passage is limited to a dotted placeholder and the page number 265.Source: §A.20 · p. 6

1.1.1 OpenAI o1 & Chain-of-Thought Reasoning

• o1 integrates chain-of-thought reasoning into inference and improves with additional train-time and test-time compute, shifting emphasis toward reasoning during inference rather than only preference alignment during training.The paper describes this as enabling longer internal reasoning and potentially multiple reasoning paths before an answer.Source: §1.1.1 · p. 11, §1.1.2 · p. 12

1.2 Motivation

• The evaluation organized 27 tasks into five domains—Creation and Design, Planning, Reasoning, Diagnosis, and Reflection—to assess adaptability across cognitive and real-world challenges.The methodology used relevant tasks within each domain to provide a broad evaluation.Source: §1.2 · p. 12, Figure 1 · p. 15

1.3 Key Findings

• o1-preview demonstrated advanced step-by-step reasoning across high-school mathematics, quantitative investing, and chip design, including complex multi-layered tasks.The evaluation found exceptional logical reasoning abilities in these fields.Source: §1.3 · pp. 12–13
• The model showed broad domain-specific knowledge in medical genetics, radiology, anthropology, and geology, often performing comparably to or better than graduate students or early-career professionals.Its performance indicates substantial breadth across specialized fields.Source: §1.3 · p. 13

– Content Summarization

• o1-preview excelled at nuanced language tasks including sentiment analysis, social media analysis, and content summarization, while still struggling with very subtle emotional nuances.It could capture complex expressions such as irony and sarcasm.Source: §1.3 · p. 13

2.2 Radiology Report Generation

• The model showed strong capabilities in medical diagnosis, radiology report generation, and complex medical question answering, although its reasoning sometimes differed from that of trained medical professionals.These results cover both clinical reporting and medical knowledge tasks.Source: §1.3 · p. 13

1.3 Key Findings

• Despite its strengths, o1-preview remained inconsistent on extremely abstract logical puzzles, real-time dynamic situations, and the most complex advanced mathematics and stochastic-process problems.The paper therefore calls for further refinement and validation before critical real-world deployment.Source: §1.3 · p. 13

AGI-Benchmark 1.0

• AGI-Benchmark 1.0 contributes a public collection of intricate, multi-step reasoning tasks across 27 categories and five cognitive faculties, complementing benchmarks focused mainly on question answering and multiple choice.The benchmark is intended to support transparency, reproducibility, and collaborative progress in evaluating real-world complex reasoning.Source: §1.4 · p. 13, AGI-Benchmark 1.0 · p. 14

2.5 Quantitative Investing

• In quantitative investing, o1-preview dynamically integrated market variables such as price trends, trading volumes, and macroeconomic indicators to refine models and strategies in real time.The stock-trading-QA dataset emphasizes statistical modeling, automation, signal integration, and numerical computation for complex financial queries.Source: §2.5 · pp. 17, 19

2.6 Low-Resource Language Translation

• o1-preview successfully translated common Cherokee phrases and identified grammatical structures, but unknown words sometimes caused incomplete or inaccurate translations requiring expert refinement.The model often produced plausible guesses for unfamiliar vocabulary, indicating some inferential ability despite limited parallel data.Source: §2.6 · p. 19

2.7 Educational Q&A

• o1-preview performed exceptionally well on SciQ, selecting correct scientific answers despite distractors and demonstrating understanding across physics, biology, chemistry, and earth sciences.The evaluation used 13,679 scientifically oriented questions to assess comprehension, inference, and explanation across disciplines.Source: §2.7 · pp. 19–20

2.9 3D Layout Generation

• o1-preview generated coherent, realistic 3D room layouts by respecting spatial constraints, avoiding object overlaps, preserving accessibility, and following design guidelines.The results support potential applications in interior design and virtual environment creation.Source: §2.9 · pp. 20–21

2.10.2 Experimenting with o1-preview in Chip Design

• In chip design, o1-preview was evaluated on engineering assistance, Verilog EDA script generation, and bug-summary analysis as proxies for complex real-world engineering workflows.The evaluation targets technical consultation, code quality and efficiency, and extraction of root causes and solutions from error logs and simulation data.Source: §2.10.2 · p. 22

2.10.3 A Leap Toward AGI and the Future of Chip Design

• o1-preview’s chip-design capabilities are presented as evidence of advanced reasoning across circuit optimization, EDA scripting, and bug analysis, supporting its potential as a general-purpose problem solver toward AGI.The section frames chip design as a high-stakes domain requiring technical precision, adaptive problem-solving, and trade-offs among performance, power, and manufacturability.Source: §2.10.3 · pp. 22–23

2.11 Logical Reasoning

• o1-preview performed extremely well on LogiQA, maintaining high accuracy after repeated recalculation and resisting interference across multiple logical-reasoning problem types.The evaluation covered categorical, sufficient-condition, necessary-condition, disjunctive, and conjunction reasoning using 8,678 QA instances.Source: §2.11 · pp. 23–24

2.14 College-level Math Problems

• o1-preview easily solves basic discrete mathematics but makes various mistakes on more challenging college-level problems involving abstraction, notation, and longer reasoning chains.The manually created problems cover discrete mathematics, calculus, and advanced theorem proofs, although the problem set is relatively small.Source: §2.14 · p. 26

2.15 Electronic Health Record Diagnosis

• o1-preview provides diagnostic reasoning from biomedical abstracts, performs better on shorter texts, and faces challenges evaluating longer, more complex records.The evaluation sampled 10 cardiovascular-disease abstracts from OHSUMED, whose 13,929 documents span 23 categories.Source: §2.15 · p. 27

2.16 Stochastic Processes in Statistics; 2.18 Social Media Analysis; 2.19 Analogical Reasoning; 2.20 Sentiment Analysis

• The evaluation spans stochastic-process statistics, four social-media tasks, semantic analogies, and sentiment polarity, aspect, and informal-text classification, but supplied passages report no outcomes for most tasks.Stochastic-process problems cover Markov chains, Poisson processes, and renewal theory; social-media benchmarks cover sentiment, irony, emotion, and offensive language.Source: §2.16 · p. 27, §2.18 · pp. 28–29, §2.19 · p. 29

2.17 Medical Text Anonymization

• Most privacy information is accurately detected and removed from medical texts, although anonymization performance varies across coarse-to-fine prompt styles.The evaluation uses the 2014 i2b2/UTHealth de-identification dataset and three prompt styles.Source: §2.17 · p. 28

2.21 Anthropology and Geology

• On newly authored expert-reviewed questions, o1-preview produced expert-like paleoanthropology responses and geology explanations that correctly identified phenomena and supplied supporting evidence.The five-question evaluation targeted difficult niche concepts while limiting possible training-set overlap.Source: §2.21 · pp. 30–31

2.22 Educational Measurement and Psychometrics; 2.23 Public Health Policy Analysis; 2.24 Medical Genetics and Genomics Reasoning; 2.25 Medical Knowledge Question Answer

• The evaluations cover ACA health-policy reasoning, 20 genomics QA tasks based on GeneTuring with OMIM-extended information, and 10 MedMCQA questions spanning easy and difficult levels.The ACA dataset is limited in size and scope, and the study reports potential for policy analysis.Source: §2.23 · pp. 31–32, §2.24 · p. 32, §2.25 · p. 33

2.26 Art Education

• o1-preview is assessed as a potential art-education tool for lesson planning, artistic writing, curriculum development, standards alignment, inclusivity, and reflective learning.Prompts cover classroom challenges, creative exercises, curriculum theory, representation, interdisciplinary connections, and culturally responsive practice, including currere.Source: §2.26 · pp. 33–34

2.27 Content Summarization

• Content summarization evaluates whether one-sentence summaries of XSum news articles remain consistent, appropriately detailed, and suitably concise.XSum contains 226,711 BBC articles from 2010–2018 across categories including sports, technology, business, health, and education [125].Source: §2.27 · p. 34

3.1 Foundation Models; 3.3 Chain of Thought

• Transformers underpin modern language and vision models, while Chain-of-Thought prompting exposes intermediate reasoning steps for multi-step mathematical and logical tasks [193].Related methods include manual, automatic, and semi-automatic reasoning annotations, plus Self-Refine, Tree of Thought [202], and Graph of Thought extensions.Source: §3.1 · pp. 34–35, §3.3 · pp. 36–37

3.2 Prompt Engineering

• Prompt engineering guides LLM outputs through manual design or parameter-efficient methods such as Prompt Tuning [72] and Prefix Tuning [81].Its main challenges are interpretability, limited cross-task generalizability, and control of bias and undesirable outputs [143] [179] [87].Source: §3.2 · pp. 35–36

3.4 Multi-modal Large Language Models

• Multimodal models evolved from vision-language architectures such as ViLBERT [111], Oscar [84], CLIP [146], and Flamingo [3] toward stronger reasoning, efficiency, and responsible deployment.Current challenges include computational cost, complex visual-textual reasoning, bias, fairness, and privacy [200].Source: §3.4 · pp. 37–38

3.5 Fine-tuning Large Language Models

• Fine-tuning specializes pretrained models, while parameter-efficient approaches reduce computational cost through adapters, prompt tuning, prefix tuning, and LoRA [53].Full fine-tuning can require substantial resources and cause catastrophic forgetting [90], whereas the efficient methods preserve more general knowledge and improve training speed and generalization.Source: §3.5 · pp. 38–39

3.6 Large Language Model Agent and Retrieval-Augmented Generation

• AI agents autonomously perceive, reason, and act, while retrieval-augmented generation grounds responses in external data to improve factual accuracy and relevance [75].Agents face generalization, accountability, transparency, bias, and resource challenges; RAG adds retrieval latency and depends on source quality, motivating o1-preview’s proposed combination of autonomy, knowledge integration, efficiency, and verification.Source: §3.6 · pp. 39–40

3.7 Large Language Models & Reasoning

• LLMs struggle with multi-step deduction, abstract reasoning, and cross-domain knowledge integration, motivating CoT [193], ToT [202], Self-Consistency [188], and tool-using ReAct [201].These methods expose or search multiple reasoning paths, select consistent answers, and use external tools to compute, retrieve information, and validate steps.Source: §3.7 · p. 40
• Reasoning methods progressed from prompt-based chain-of-thought guidance to Tree-of-Thoughts, self-consistency, and tool-interacting frameworks such as ReAct.These methods explore multiple reasoning paths or combine internal reasoning with external resources to improve handling of ambiguous and complex tasks.Source: §3.7 · p. 41

3.8 Reinforcement Learning with Human Feedback

• RLHF aligns model responses with human expectations through supervised fine-tuning, reward modeling from human preferences, and reinforcement-learning optimization.The approach emerged as a solution for generating responses that satisfy human expectations and underpins widely deployed AI models [18] [220].Source: §3.8 · p. 41, • Policy (π): The strategy the model… · p. 42

• Policy (π): The strategy the model uses to generate responses.

• PPO performs RM and RL stages separately and uses an explicit reward model while providing stable policy updates; DPO combines stages and bypasses the need for an explicit reward model.Source: • Policy (π): The strategy the model… · p. 42

3.9 Evaluation Complex Reasoning Tasks

• o1-preview uses chain-of-thought reasoning to iteratively analyze intermediate steps and synthesize coherent solutions, particularly for sustained multi-step reasoning.Its capabilities are pronounced in scientific analysis, programming, fact-checked legal and medical reasoning, while slower responses can make it less suitable for speed-sensitive tasks than GPT-4 [2].Source: §3.9 · p. 43

4.1 Test Procedure

• The evaluation deployed domain-specific prompts and benchmarks across medicine, education, robotics, mathematics, and other fields to assess coherence, accuracy, relevance, and adaptation.Responses were examined beyond correctness for logical coherence, contextual appropriateness, domain-specific reasoning, and detailed error patterns; limitations remained on intricate specialized reasoning.Source: §4.1 · p. 45

4.2 Code Generation

• o1-preview passed 10 of 12 coding problems, achieving an 83.3% passing rate while failing both hard-level problems.The failures involved time-limit and incorrect-answer issues across three attempts, despite comparable or faster generation times than top human competitors.Source: §4.2 · p. 46, Table 2 · p. 46

4.3 Radiology Report Generation

• o1-preview achieved the highest radiology-report ROUGE scores among six models: R-1 0.3019, R-2 0.0448, and R-L 0.2841.Its average generation time was 15.051 seconds, attributed to extensive inference verification, whereas gpt-3.5-turbo was fastest at 2.024 seconds.Source: §4.3 · p. 49, Table 3 · p. 49

4.4 Robot Command Planning

• o1-preview generated robotic-control code that accurately followed instructions, demonstrating precise and reliable comprehension of complex ROS command structures.The result indicates robustness on intricate robotic-control tasks requiring precise interpretation and synthesis.Source: §4.4 · p. 52
• o1-preview translated abstract instructions into executable four-wheel ROS control code, highlighting potential for robotics automation.The demonstrated code used Twist messages to control forward movement and angular velocity.Source: Figure 7 · p. 52, §4.4 · p. 53

4.5 Nature Language Inference

• Across 10 sampled test cases, o1-preview accurately analyzed logical relationships in natural language inference tasks across general and specialized domains.Table 4 covers NLI and QNLI labels including entailment, contradiction, and neutral; examples include cataloger comparisons, winner claims, medical EKG findings, pleural effusion, and bus-station questions.Source: §4.5 · pp. 54–55, Table 4

4.6 Quantitative Investing

• o1-preview showed solid financial knowledge and statistical modeling skills, explaining ARIMA, mean reversion, stochastic oscillators, autocorrelation, trend, seasonality, and residual analysis for market forecasting.The evaluation describes both conceptual explanations and systematic practical application in financial analysis and forecasting.Source: §4.6 · p. 56, Figure 8 · p. 56, Figure 9 · p. 57

4.7 Low-Resource Language Translation

• In two Cherokee translation cases, o1-preview accurately handled one sentence and basic word recognition but struggled with contextual nuances and precision in the more complex example.It correctly translated “The boys are playing ball,” while the second output identified chickens but diverged from the reference on concepts including roosters, crowing, and dawn.Source: §4.7 · p. 58, Figure 10 · p. 58, Figure 11 · p. 59

4.8 Educational Q&A

• o1-preview correctly answered educational questions identifying the Coriolis effect and exothermic change, demonstrating accurate interpretation of scientifically framed prompts.The examples concern global wind deflection and heat release when a less-ordered state becomes more ordered, such as liquid becoming solid.Source: §4.8 · p. 60, Figure 12 · p. 60, Figure 13 · p. 61

4.9 Student Writing Improvement in Higher Education

• Qualitative writing evaluations found support for linguistic accuracy, coherence, outlines, and initial ideas, but responses could be rigid, lack citations, and require human guidance.The paper cautions that effectiveness varies by learner and context, and that students may passively adopt machine-like structures, limiting creativity [182] [68].Source: §4.9 · p. 62, Figure 14 · p. 63, Figure 15 · p. 64

4.10 3D Layout Generation

• For 3D room-layout generation, the evaluation used an LLplace prompt [199] requiring non-overlapping, functional object placement with spatial coordinates and rotation angles in JSON.The prompt prioritizes placing objects near room edges, aligning objects, and positioning chairs near and facing tables or desks.Source: §4.10 · p. 66

[Task Room Type] /* Input room type */ [/Task Room Type]

• o1-preview accurately interpreted room types, objects, and bounding boxes, producing mostly non-overlapping, aligned layouts that considered accessibility, aesthetics, and spatial relationships.It often placed objects along room edges to increase spaciousness and positioned chairs near and facing tables or desks.Source: [Task Room Type] /* Input room type */… · pp. 66–67
• Although the room layouts were highly functional and aesthetically pleasing, a small number of examples contained three-dimensional object overlaps, leaving room for improved accuracy in complex layouts.The study visualized generated layouts in Figures 17, 18, 19, and 20 and reported object coordinates and rotation angles in JSON.Source: [Task Room Type] /* Input room type */… · p. 67, Figure 17 · p. 68, Figure 18 · p. 69

4.11.1 Engineering Assistant Chatbot

• In engineering-assistant tasks, o1-preview provided more comprehensive and expert responses than ChipNeMo by addressing scalability, clocking conditions, verification edge cases, and retiming trade-offs.Its responses covered parallelizing random tests, adapting assertions across clock domains, and balancing retiming with logic restructuring despite latency and timing constraints.Source: §4.11.1 · pp. 72–73

4.11.2 EDA Script Generation

• For EDA script generation, o1-preview exceeded ChipNeMo by proposing scalable traversal, boundary handling, extensibility, hierarchical-netlist support, and edge-case handling for large designs.The comparisons covered counting flip-flop cells within a rectangle and finding intersecting logic between two pins.Source: §4.11.2 · pp. 73–74

4.11.3 Bug Summary & Analysis

• In bug summary and analysis, o1-preview offered deeper technical and strategic guidance than ChipNeMo, including refined power-measurement isolation, pre-silicon comparison, power-density implications, and task milestones.The task concerned measuring fabric-macro power under varying activity and voltage conditions during design-for-test work.Source: §4.11.3 · p. 74
• Across engineering assistance, EDA scripting, and bug analysis, o1-preview consistently surpassed ChipNeMo in problem-solving depth, scalability, flexibility, and practical relevance.It anticipated edge cases, optimized large-scale workflows, handled complex netlists, and combined technical analysis with strategic task prioritization for chip-design automation.Source: §4.11.3 · pp. 74–75
• o1-preview generated EDA Tcl procedures for counting flip-flops in a defined region and finding logic intersecting the fanout cone of $pinA with the fanin cone of $pinB.The procedures query placement or netlist objects, traverse recursively, and compute intersections; Figure 24 and Figure 25 compare these outputs with ChipNeMo and describe optimization for large designs.Source: §4.11.3 · pp. 80, 82, Figure 24 · p. 81, Explanation： · p. 83

Explanation：

• The intersecting-logic script requires tool-specific Tcl command support and may face long runtimes or memory use for large logic cones.The procedure traverses the netlist at pin level and may need limits, optimization, cell-level traversal, or additional logic-element information.Source: Explanation： · p. 84

2. Test Conditions

• Fabric-power measurements target 0%, 25%, 50%, 75%, and 100% activity at typical low- and high-voltage corners, isolating necessary test-module and fabric logic.Other nonessential logic should be disabled or power-gated, and measurements should be broken down across six fabric modules where feasible.Source: Explanation： · p. 85, §2 · p. 85, §3 · p. 86

2. Measurement Phase

• The proposed power study schedules repeated measurements from November 13–24, 2023, followed by analysis and correlation with pre-silicon estimates from November 27–December 1.The plan seeks reliable data, identifies discrepancies, and reports graphs, charts, and recommendations for adjusting power-estimation models.Source: §2 · p. 86, §3 · p. 86, §4 · p. 86

4. Reporting

• The power request is prioritized because accurate silicon correlation can improve power modeling, guide optimization, and support future chips with higher fabric power density.The response recommends raising severity from 5 - Performance to 3 - Significant and priority from 5 - Normal to 2 - High.Source: §4 · pp. 86–87

4.12 Logical Reasoning

• o1-preview produced accurate, well-organized analyses across regular and complex logical-reasoning problems, including tasks with nested reasoning and multiple conditions.It decomposed problems into components, considered alternative scenarios, and handled mathematical puzzles, sequences, and argument validity; Figure 27 reports an expected answer.Source: §4.12 · p. 88, Figure 27 · p. 89
• o1-preview sometimes diverged from official standard answers, although its exploratory reasoning could expose alternative solution paths and the standard answer was not necessarily accurate.Figure 28 identifies test results differing from the official answer as failed samples, while the analysis emphasizes the model’s convincing reasoning process.Source: §4.12 · p. 88, Figure 28 · p. 90
• Despite generally low error rates and minimal required guidance, o1-preview occasionally mishandled subtle logical relationships and faced bottlenecks when multiple logical systems intersected.The model can process many reasoning problems quickly, but these limitations constrain performance on extremely complex tasks.Source: §4.12 · p. 88

4.13 Table-to-Text Generation

• o1-preview generated coherent, accurate ADNI clinical narratives and interpreted missing measurements contextually rather than reproducing “NaN.”The model preserved clinical information while organizing demographics, cognitive scores, biomarkers, and imaging data into readable reports.Source: §4.13 · p. 91

4.14 High School Level Math Competition

• 100% accuracy across 10 high-school mathematics problems showed that o1-preview handled algebra, counting, probability, and geometry with detailed step-by-step reasoning.It usually answered within seconds, but some level-5 problems required up to 50 seconds.Source: §4.14 · pp. 91, 94

4.15 College-level Math Problems

• In college-level mathematics, o1-preview solved basic combinatorics, one calculus proof, and two real-analysis problems, but failed all three advanced number-theory problems.It also failed a functional-analysis problem because of a misunderstanding of linear independence in infinite-dimensional spaces.Source: §4.15 · pp. 99–101
• The college-level evaluation exposed unreliable proof reasoning, including proof by example, circular reasoning, invalid series comparisons, and loss of rigor across longer reasoning chains.For the calculus series problem, one method was incorrect while the other was correct but relied on graduate-level real-analysis techniques, leading to a questionable evaluation.Source: §4.15 · pp. 99–100
• o1-preview performed well on college-level mathematics, applying advanced theorems and producing distinct Hilbert’s Nullstellensatz proofs, but struggled more with long reasoning chains than abstract problems.Its errors could be human-like, yet it sometimes overgeneralized from observations, used overly advanced techniques, or fell into circular reasoning.Source: §4.15 · p. 101

4.16 Electronic Health Record Diagnosis

• Across 10 cardiovascular EHR cases, o1-preview identified medical patterns and supplied diagnostic reasoning, performing better on concise records than longer, more complex ones.The evaluation used manually selected cases reviewed by the authors, and further EHR-focused reinforcement learning was suggested for improvement.Source: §4.16 · p. 105, o1-preview has demonstrated notable… · p. 106

4.17 Stochastic Processes in Statistics

• 80% accuracy was achieved on basic stochastic-process problems, while more challenging Markov-chain cases exposed errors in determining steady states.The model generally provided detailed step-by-step solutions, but explanations were sometimes overly verbose.Source: §4.17 · p. 111

4.18 Medical Text Anonymization

• Nearly all sensitive information was detected in lengthy medical texts across coarse, medium, and fine prompts, although explicit prompts sometimes removed meaningful words.Prompt appropriateness affected whether the model focused on structure, aligned annotations, or evaluated words individually.Source: §4.18 · p. 114

4.19 Social Media Analysis

• o1-preview generally aligned with ground truth across 11 social-media cases, including sentiment, offensive-language, emotion, and irony tasks, but made errors involving sentiment and emotion ambiguity.It often captured irony, sarcasm, and humor, yet subtle emotional cues, implicit context, cultural background, and nonverbal signals remained difficult.Source: §4.19 · p. 118

4.20 Analogical Reasoning

• Analogical-reasoning experiments tested whether o1-preview could transfer semantic relations and content attributes between domains using relational and multi-layered drawing tasks.The task categories were semantic-structure mapping and semantic-content mapping, with answers constrained to the final drawing.Source: §4.20 · p. 121
• o1-preview correctly solved 10 increasingly complex analogical reasoning questions, mapping semantic structures, eliminating distractors, and applying relational patterns.The experiments covered direct semantic structure mapping and multi-layer analogies.Source: §4.20 · p. 121

4.21 Sentiment Analysis

• o1-preview accurately classified complex sentiments across domains and identified different sentiments toward distinct aspects within the same text.The evaluation included overall movie-review classification and aspect-based analysis across restaurant, electronics, and social-media content.Source: §4.21 · p. 124, Figure 52 · p. 125, Figure 53 · p. 125

4.22 Anthropology and Geology

• o1-preview answered increasingly specific paleoanthropology and geology questions with detailed, field-relevant guidance, but lacked creative thinking and finer specialized detail.The model could provide introductory research information, current status, methods, and peer-like discussion, while remaining limited on narrow details.Source: §4.22 · pp. 127, 133

4.23 Educational Measurement and Psychometrics

• o1-preview excelled at psychometric computation and reasoning at or above first-year graduate level, but misclassified measures of interrater reliability.It correctly handled the Spearman-Brown reliability calculation and standard-error confidence intervals, yet selected nominal agreement and Cohen’s kappa instead of ICC and Coefficient Alpha.Source: §4.23 · p. 134
• The educational measurement evaluation tested o1-preview on specialized questions involving test reliability, standard error of measurement, and interrater reliability.The supplied cases cover reliability concepts and identify consistency between model responses and reference answers using figure annotations.Source: Figure 61 · p. 135, Figure 62 · p. 136, §4.23 · pp. 136–137

4.24 Public Health Policy Analysis

• In public-health policy analysis, o1-preview generally aligned with reference answers on the Affordable Care Act’s coverage, access, disparities, preventive services, and Medicaid impacts, but sometimes omitted disease-specific details.The evaluation targeted ACA effects on insurance coverage, healthcare disparities, preventive services, and Medicaid expansion.Source: §4.24 · p. 138

4.25 Medical Genetics and Genomics Reasoning

• All 20 genomics reasoning instances received correct answers, with explanations tracing conclusions through reasoning chains; false-negative conclusions averaged 20.09s versus 12.67s for true positives.The longer negative-case runtime suggests more extensive reasoning when rejecting a genomics hypothesis.Source: §4.25 · p. 141

4.26 Medical Knowledge Question Answer

• 100% accuracy was achieved on medical knowledge questions, with generally accurate options and explanations despite occasional missed conditions and illogical reasoning.The model handled both simple knowledge questions and complex disease-inference scenarios, but sometimes overlooked key information when assessing alternatives.Source: §4.26 · p. 144

4.27 Art Education

• In art education, o1-preview accurately explained currere and produced structured, creative activity plans, but its cardboard-assemblage lesson was more rigid and less adaptive than Dr. Penfold’s child-centered approach.Its currere explanation closely aligned with William Pinar’s original work, while the activity design lacked flexibility for open-ended exploration.Source: §4.27 · p. 148

4.28 Content Summarization

• The content-summarization evaluation used one-sentence prompts for news articles and compared model summaries with manual summaries on accuracy, abstraction, and length.The design targeted contextual relevance, content understanding, and text generation while constraining output length to one sentence.Source: §4.28 · p. 152
• o1-preview generally met summarization needs and matched reference themes, but produced longer, more detailed outputs and was slightly less abstract than human summaries.Human summaries more often processed details into higher-level expressions, whereas o1-preview commonly retained competition scores and other specifics.Source: §4.28 · p. 152

5 Discussion

• The evaluation found strong scientific reasoning and cross-domain knowledge integration, alongside weaknesses in coding efficiency, advanced mathematics, low-resource translation, medical reliability, concision, and domain grounding.The discussion identifies inconsistent advanced mathematical reasoning, occasional inaccurate diagnoses, overly verbose scientific explanations, and variable practicality in geology and financial analysis.Source: §5 · p. 155

5.1 LLM Agents and o1: Advancing Problem-Solving in Science and Engineering

• Combining o1-preview with agents’ planning, memory, tool use, and self-reflection could support more complex, multi-step scientific research and engineering design, although context, long-term planning, reliability, and consistency remain challenges.Proposed applications include decomposing experiments into literature review, hypotheses, methodology, and analysis, while using specialized software, simulations, and databases.Source: §5.1 · p. 156

6 Conclusion

• The conclusion characterizes o1-preview as a major advance toward general artificial intelligence while emphasizing unresolved limitations in advanced mathematics, specialized knowledge, and consistent performance.Its reported strengths include coding, scientific reasoning, language processing, creative problem-solving, and knowledge integration across domains.Source: §6 · p. 157

A.1 Code Generation

• Coding demonstrations on easy, medium, and hard problems show that the generated code passed Leetcode’s automated judgment system on the first attempt across six documented cases.These cases are Figures 74–79, covering one easy, three medium, and two hard problems.Source: Figure 74 · p. 158, Figure 75 · p. 159, Figure 76 · p. 160

A.2 Radiology Report Generation

• Radiology examples indicate that zero-shot o1-preview reports closely align with human writing patterns through clear organization and concise language.The radiology outputs were translated from Chinese to English for reader accessibility.Source: Figure 84 · p. 168, Figure 85 · p. 169, Figure 86 · p. 170

A.3 Robot Command Planning

• o1-preview generated ROS implementations for SLAM, point-to-point navigation, deep Q-learning robot control, and autonomous-driving control loops.The examples integrate laser scans and odometry, MoveBase goals and quaternion orientation, TensorFlow decision-making, and dynamic speed and steering adjustment.Source: Figure 92 · p. 176, Figure 93 · p. 177, Figure 94 · p. 178

A.4 Nature Language Inference

• Additional NLI examples are presented in Table 8, while the medical-record subsection introduces a cardiovascular patient abstract without supplying a diagnostic result in this part.The available passages establish the presence of examples and a case prompt, but not their substantive outcomes.Source: §A.4 · p. 180, Table 8 · p. 181, §A.11 · p. 214

A.5 Quantitative Investing

• The model explained the stochastic oscillator as a 0–100 momentum indicator, interpreting values above 80 as overbought and below 20 as oversold trading signals.It also explained mean reversion and the combined roles of trend, seasonality, and residual analysis in stock-market forecasting.Source: Figure 96 · p. 182, Figure 97 · p. 183, §A.5 · p. 184

A.6 Low-Resource Language Translation

• In low-resource Cherokee translation, o1-preview often produced detailed grammatical explanations but mismatched the reference meaning in all three presented cases.The examples include a wholly different translation, an incorrect contextual interpretation, and a grammatically analyzed output that failed to capture the intended meaning.Source: Figure 99 · p. 185, Figure 100 · p. 186, Figure 101 · p. 187

A.7 Educational Q&A

• Educational examples show that o1-preview correctly answered questions about radioactive decay and volcanic materials but overthought a cheese-and-yogurt question instead of selecting from the options.The educational dataset is sourced from [195].Source: Figure 102 · p. 188, Figure 103 · p. 189, Figure 104 · p. 190

A.8 Logical Reasoning

• o1-preview provided rigorous, step-by-step analyses across logical reasoning problems, accurately identifying conclusions and explaining relationships among premises, options, and underlying issues.Examples include university-admission deduction, cost-reduction recommendations, policy interpretation, argument evaluation, conditional logic, dialogue similarity, and reasoning-defect matching.Source: Figure 105 · p. 191, Figure 106 · p. 192, Figure 107 · p. 193

A.9 High School Level Math Competition

• High-school mathematics examples covered algebra, counting, and probability problems, with several outputs sharing solution steps and final answers with the references.The cases include vertical asymptotes, parabola distance, algebraic simplification, complementary probability, coin flips, and unequal-probability dice.Source: Figure 113 · p. 199, Figure 114 · p. 200, Figure 115 · p. 201

A.10 College-level Math Problems

• College-level examples were manually judged correct for basic discrete mathematics, calculus, and several advanced theorem proofs, but incorrect for some advanced discrete problems and theorem proofs.The demonstrations therefore show both successful reproduction of mathematical solutions and failures on advanced cases.Source: Figure 119 · p. 205, Figure 120 · p. 206, Figure 121 · p. 207

A.11 Electronic Health Record Diagnosis

• The electronic-health-record section supplies multiple demonstration cases in which outputs are compared with reference diagnoses using consistency-colored figures, but it reports no aggregate diagnostic score.The cases span diverse medical abstracts, including infectious, neurologic, hematologic, dermatologic, and retinal conditions.Source: Figure 128 · p. 214, Figure 129 · p. 215, Figure 130 · p. 216

A.12 Stochastic Processes in Statistics

• The stochastic-process cases report that o1-preview successfully solved several problems, including a betting problem involving a $3 starting stake and an $8 bail threshold.The supplied figures label Cases 3–6 as successful solutions, while the text provides the betting setup but not numerical solution values.Source: Figure 134 · p. 220, Figure 135 · p. 221, §A.12 · p. 221

A.13 Medical Text Anonymization

• The medical-text anonymization cases demonstrate implicit, general, and explicit prompting, with annotations distinguishing human-marked information, model-detected sensitive information, and extra modifications.Six supplied figures cover Cases 4–9, but no numerical accuracy or error rate is reported.Source: Figure 138 · p. 224, Figure 139 · p. 225, Figure 140 · p. 226

A.14 Social Media Analysis

• Social-media demonstrations cover emotion recognition, offensiveness classification, and irony detection, with figures using consistency colors against reference answers.The supplied prompts include four emotion classes and binary offensiveness classification, but no aggregate performance values are given.Source: §A.14 · pp. 230–231, Figure 144 · p. 230, Figure 145 · p. 231

A.15 Analogical Reasoning / A.16 Sentiment Analysis

• The section includes demonstrations of direct semantic mapping, multilayer analogical reasoning, and sentiment analysis, including extraction of positive emotions from food-review text.The analogical figures mark answers as correct, while sentiment figures illustrate model capability without reporting aggregate metrics.Source: Figure 146 · p. 232, Figure 147 · p. 233, §A.16 · p. 234

A.17 Public Health Policy Analysis

• Public-health-policy demonstrations ask about the ACA’s effects on access, insurance coverage, care quality, costs, technology, and chronic-condition outcomes; one response introduces its long-term marketplace impacts.The figures classify responses as partially or fully consistent, but the supplied result passage does not state the substantive impacts.Source: §A.17 · pp. 237–240

A.18 Medical Genetics and Genomics Reasoning

• The medical genetics demonstration shows o1-preview making correct gene–disease inferences from descriptions involving SOD3, SOD1, and SOD2.The example asks for reasoning and argumentation linking a gene to a disease, and the accompanying figure labels the example as positive with correct inferences.Source: §A.18 · p. 243, Figure 156 · p. 243

A.19 Medical Knowledge Question Answer

• The model answered the mandible ossification question correctly as “D: All of the above,” explaining that the mandible uses intramembranous and endochondral ossification.The cited explanation identifies the condylar process, coronoid process, and upper ramus as cartilage-ossifying regions.Source: §A.19 · p. 244
• o1-preview correctly identifies medical-exam answers while providing explanations grounded in immunologic, pharmacologic, and clinical reasoning.Examples include MHC Class II expression, prucalopride’s receptor activity, and androgen insensitivity syndrome.Source: §A.19 · pp. 245, 246, 249, 250, • Conclusion · p. 251
• MHC Class II proteins are expressed by B-cells, dendritic cells, and macrophages, which present extracellular antigens to helper T cells.The response distinguishes these professional antigen-presenting cells from platelets, T-cells, and all nucleated cells, which do not express MHC Class II under the stated conditions.Source: §A.19 · pp. 245–246
• Prucalopride is identified as a selective 5-HT4 agonist that enhances gastrointestinal motility for chronic constipation.The explanation attributes its activity to selective 5-HT4 receptor activation and contrasts it with other serotonin-receptor options.Source: §A.19 · pp. 248–249

• Conclusion:

• The androgen-insensitivity explanation links androgen resistance to estrogen-mediated breast development, phenotypically female external genitalia, sparse pubic hair, and primary amenorrhea.The response describes the condition as involving genetic males with end-organ resistance to androgens and absent or underdeveloped reproductive structures.Source: • Conclusion · p. 251
• Androgen insensitivity syndrome is selected for a patient with primary amenorrhea, well-developed breasts, minimal pubic hair, and ineffective oral contraceptive therapy.The differential explanation rules against premature ovarian failure, Turner syndrome, and Müllerian agenesis because their expected breast, stature, or pubic-hair findings do not match the presentation.Source: §A.19 · p. 250, • Conclusion · p. 251
• The HBV results show isolated anti-HBc IgG positivity, leaving resolved, occult, or false-positive infection possibilities unclear without HBsAg or anti-HBs, while HCV exposure requires HCV RNA testing to establish active infection.The supplied interpretation states that occult HBV is possible but less likely as the primary cause, and that HCV antibodies do not distinguish past from current infection.Source: • Conclusion · pp. 253–254
• The Paul-Bunnell test is based on heterophile antibodies produced during EBV infection, detected through agglutination of animal red blood cells.The explanation identifies heterophile antibody testing as correct and distinguishes it from complement-mediated and latex agglutination methods.Source: • Conclusion · p. 255
• Alport syndrome is identified from anterior lenticonus, early-adult end-stage renal disease, and a maternal-uncle history consistent with X-linked inheritance.The condition is linked to type IV collagen defects affecting lens and glomerular basement membranes, with males more severely affected.Source: • Conclusion · p. 257

A.20 Art Education

• Nick Cave’s Soundsuits help high school students explore identity, creativity, and self-expression by masking social identifiers, encouraging material experimentation, and enabling collaborative performance.Their anonymity supports introspection, empathy, and discussion of diversity, while their materials and performative form promote multimedia and physical expression.Source: §A.20 · p. 260
• Integrating disability studies into art education reframes disability socially and culturally while advancing inclusive curricula, authentic representation, accessible practices, and critical inquiry.The integration promotes belonging and validation through representation and supports participation through adaptive tools, assistive technologies, and Universal Design for Learning.Source: §A.20 · p. 262
• Integrating disability studies in art education challenges ableist norms, critically examines representation, and empowers students with disabilities to express diverse perspectives.The approach also connects art education with sociology, ethics, history, and broader social issues.Source: §A.20 · pp. 262–263
• Art education supports personality development through creativity, emotional expression, critical thinking, confidence, communication, empathy, collaboration, well-being, and lifelong learning.The response also describes gains in motor skills, patience, cultural awareness, academic achievement, and social interaction.Source: §A.20 · p. 264

A.21 Content Summarization

• The content-summarization cases illustrate o1-preview producing one-sentence summaries of news about an alternative-model contest, school funding pressures, fuel prices, a robbery, and a medical emergency at a football match.The associated figures characterize these cases as examples of summarization capability, with yellow indicating partial consistency and green complete consistency with reference answers.Source: Figure 169 · p. 265, §A.21 · pp. 266–270
• The news examples require compressing reports while retaining key events, participants, outcomes, financial figures, price changes, and public or institutional responses.The cases include school savings and funding disputes, supermarket fuel-price reductions, police investigation details, and the abandonment and subsequent fatal outcome of a football match incident.Source: §A.21 · pp. 267–270