PaperlayerView exact revision on arXiv

Source-linked AI summary

K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization

Cheng Deng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, Zhouhan Lin, Junxian He

arXiv:2306.05064v2cs.CLcs.AI

TL;DR

Geoscience lacks sufficiently capable and accessible language models despite abundant domain literature and the success of general LLMs. The paper adapts LLaMA-7B with geoscience pre-training, instruction tuning, GeoSignal, and GeoBench, and reports stronger performance than similar-size baselines on subjective and objective geoscience tasks. K2 also supports tool use and is released with its data and checkpoints.

  • Problem

    Geoscience language models remain sparse and limited in scale, while general LLMs lack reliable geoscience knowledge and accessible open systems for domain research.

    Source: §1 · p. 1

  • Method

    K2 adapts LLaMA-7B through geoscience-domain pre-training and instruction tuning with GeoSignal, evaluates it with GeoBench, and adds tool learning.

    Source: §1 · p. 1, §4.1 · p. 6, §4.3 · p. 7

  • Results

    K2 outperforms similar-size baseline models on both subjective and objective geoscience tasks, including examinations and knowledge reasoning.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    K2 provides an open foundation model and accompanying datasets for geoscience question answering, knowledge understanding, utilization, and research assistance.

    Source: §1 · p. 1, §7 · p. 9

    The data curation and experiments use the March 2023 versions of ChatGPT and GPT-4 unless otherwise specified.

    Source: §3.2.2 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have achieved great success in general domains of natural language processing. In this paper, we bring LLMs to the realm of geoscience with the objective of advancing research and applications in this field. To this end, we present the first-ever LLM in geoscience, K2, alongside a suite of resources developed to further promote LLM research within geoscience. For instance, we have curated the first geoscience instruction tuning dataset, GeoSignal, which aims to align LLM responses to geoscience-related user queries. Additionally, we have established the first geoscience benchmark, GeoBench, to evaluate LLMs in the context of geoscience. In this work, we experiment with a complete recipe to adapt a pre-trained general-domain LLM to the geoscience domain. Specifically, we further train the LLaMA-7B model on 5.5B tokens of geoscience text corpus, including over 1 million pieces of geoscience literature, and utilize GeoSignal's supervised data to fine-tune the model. Moreover, we share a protocol that can efficiently gather domain-specific data and construct domain-supervised data, even in situations where manpower is scarce. Meanwhile, we equip K2 with the abilities of using tools to be a naive geoscience aide. Experiments conducted on the GeoBench demonstrate the effectiveness of our approach and datasets on geoscience knowledge understanding and utilization.We open-source all the training data and K2 model checkpoints at https://github.com/davendw49/k2.

1 INTRODUCTION

K2 addresses the scarcity of geoscience language models by adapting LLaMA-7B with geoscience data, supervised instructions, and evaluation resources. The paper presents K2, GeoSignal, and GeoBench as an open-source foundation for geoscience knowledge understanding and utilization.

Contributions

  • K2 is introduced as a 7-billion-parameter, open-source geoscience language model based on LLaMA.The paper also releases its code, weights, and associated datasets.Source: §1 · pp. 1–2

Method

  • The training pipeline combines further pre-training on geoscience text with instruction tuning using GeoSignal.GeoSignal unifies examples from eight geoscience NLP tasks, including relation extraction, entity recognition, classification, and summarization.Source: §1 · pp. 1–2, Figure 1 · p. 2

Resources

  • GeoBench is built as a geoscience benchmark containing more than 1,500 objective and 939 subjective questions.Its questions are collected from postgraduate and advanced-placement examinations to evaluate geoscience expertise.Source: §1 · p. 1

Applications

  • K2 is designed to answer geoscience questions, follow instructions, use tools, and support knowledge reasoning and research assistance.The paper illustrates these capabilities through academic search tools and relation prediction between geoscience knowledge points.Source: §1 · pp. 1–2

Results

  • Compared with similar-size baseline models, K2 outperforms them on both subjective and objective geoscience tasks.The reported tasks include examinations and knowledge reasoning.Source: §1 · p. 2

2 RELATED WORK

Related work situates K2 among foundation models, domain language models, and parameter-efficient adaptation methods. These lines of work motivate using open-source bases, domain specialization, and lightweight tuning for geoscience applications.

Foundation Language Models

  • Open-source foundation models such as LLaMA and CodeGen support instruction-tuned systems, while tool learning extends models with external modules and information.The paper contrasts these models with API-only systems that are less convenient for further pre-training and development.Source: §2 · p. 2

Domain Language Models

  • Domain language models have been developed for biomedicine, geoscience, geography, geology, materials science, and academic research.Examples include BioGPT, Geographic-BERT, GeoBERT, MatSciBERT, SciBERT, and Galactica.Source: §2 · p. 2

Parameter-Efficient Tuning on LLMs

  • Parameter-efficient methods reduce the need to update all LLM parameters by using soft prompts, prefixes, adapters, or low-rank updates.LoRA represents weight updates through two trainable low-rank matrices while keeping the original network largely fixed.Source: §2 · p. 2

3 DATA COLLECTION AND CURATION

This section builds K2's geoscience training and evaluation resources by collecting and preprocessing domain text, constructing GeoSignal, and establishing GeoBench. The pipeline combines large-scale geoscience sources with human- and expert-aligned instruction data and benchmark questions.

3.1 Pre-training Data

  • 5.5 billion tokens from geoscience Wikipedia, abstracts, and open-access papers form K2's further-pretraining corpus.The corpus includes 1,122,094 open-access paper PDFs and 767,341 Earth-science Wikipedia pages.Source: §3.1 · p. 2, §3.1.1 · p. 3

3.1.2 Text Corpus Preprocessing.

  • Markdown conversion, cleaning, and modality-specific tokens preserve readable structure for figures, tables, citations, and formulas.Figures retain captions, tables become Markdown, citations are replaced with reference-paper titles, and formulas receive special boundary tokens.Source: §3.1.2 · p. 3, Figure 2 · p. 3

3.2 Instruction Tuning Data: GeoSignal

  • GeoSignal combines general instruction datasets, expert-curated geoscience signals, and tool-use examples to align K2 with users and specialists.The data are sampled and cleaned because quality is prioritized over quantity; GeoTool contains 10,000 tool-instruction samples.Source: §3.2 · p. 3, §3.2.1 · p. 4, §3.2.2 · p. 6

3.2.2 Align-to-Expert.

  • GeoSignal draws on GAKG, DDE Scholar, DataExpo, GSO, geoscience QA platforms, and self-instructed questions validated by geoscientists.These resources provide papers, entities, datasets, ontologies, question-answer pairs, and domain-specific generated examples.Source: §3.2.2 · p. 4
  • Expert alignment restructures geoscience sources into input-output tasks spanning explanation, entity recognition, reasoning, verification, summarization, classification, semantics, and question answering.Signals include paper content, categories, references, captions, entities, relations, word descriptions, taxonomies, comprehension data, and factual knowledge.Source: §3.2.2 · p. 4
  • Reasoning training pairs co-occurring geoscience concepts and their paragraphs with verified relation-existence labels, while other tasks derive supervision from summaries, factual statements, and structured resources.Geoscientists verify the reasoning labels, and GeoSignal also includes fact verification and multiple summarization sources.Source: §3.2.2 · p. 5

3.3 Evaluation on Expertise in Geoscience: GeoBench

  • GeoBench evaluates geoscience knowledge understanding using questions from postgraduate and Advanced Placement examinations, with objective and subjective task sets.The benchmark includes 183 NPEE multiple-choice questions, 1,395 AP multiple-choice questions, and 939 NPEE subjective questions evaluated with human assessment.Source: §3.3 · p. 6

4 TRAINING THE K2

K2 is trained through a staged geoscience adaptation recipe that combines domain pre-training, general instruction alignment, and expert-oriented instruction tuning. The process uses LoRA for low-resource tuning and extends K2 with geoscience tool use.

4.1 Geoscience Domain Adaptation Recipe

  • K2’s domain adaptation recipe first further pre-trains LLaMA on geoscience text, then applies general instruction tuning, and finally performs expertise instruction tuning.The authors state that this three-step sequence is evaluated through ablations.Source: §4.1 · p. 6

4.2 Further Pre-training

  • 5.5B geoscience tokens are used to further pre-train LLaMA-7B for one epoch, producing the GeoLLaMA checkpoint.The training uses all 6.7B LLaMA-7B parameters and 30,140 incremental steps.Source: §4.2 · pp. 6–7

4.3 Instruction tuning

  • Sequentially training on Alpaca and then GeoSignal outperforms mixed training for aligning the model with human instructions and geoscience expertise.The authors distinguish general instruction learning from knowledge-intensive instruction learning.Source: §4.3 · p. 7
  • LoRA reduces instruction-tuning updates to 6M parameters and trains attention projections with low-rank matrices.The setup targets k_proj, q_proj, and v_proj, with lora_r set to 8 and lora_alpha to 16.Source: §4.3 · p. 7
  • Tool learning uses LoRA to train K2 to search geoscience knowledge and literature through external APIs.The tool-learning stage trains 4M parameters on eight RTX 3090 GPUs for 21 hours.Source: §4.3 · p. 7

5 EVALUATION AND RESULTS

K2 is evaluated on objective and subjective GeoBench tasks against similarly sized language models, followed by alignment and tool-learning ablations. Results show stronger geoscience performance in several areas and support sequential human-then-expert alignment.

5.1 Objective tasks in GeoBench

  • As training tokens increase during further pre-training, K2’s objective-task benchmark performance improves, indicating acquisition of geoscience knowledge.The authors evaluate saved checkpoints at selected training steps.Source: §5.1 · p. 7
  • K2 outperforms similarly sized models on GeoBench-NPEE, while its AP Test performance is similar to Galactica.The authors attribute Galactica’s comparable AP performance to overlapping high-school geoscience content in its training corpus.Source: §5.1 · p. 7

5.2 Subjective tasks in GeoBench

  • K2 performs better on rationality and correctness while remaining competitive on consistency in subjective GeoBench evaluation.The evaluation uses 50 open questions and ratings from ten geoscience practitioners across three 1-to-3 metrics.Source: §5.2 · p. 8, Table 7 · p. 8

5.3 Ablation on Expert-Alignment

  • Ablations show that knowledge-intensive GeoSignal works best after Alpaca human alignment, whereas mixing the two datasets produces unsatisfactory results.Task-special data and self-instruct data are also reported as less effective than knowledge-intensive data.Source: §5.3 · p. 8, Table 8 · p. 8

5.4 Exploration on Tool Learning

  • The ablations suggest that further pre-training mainly injects knowledge, while supervised fine-tuning teaches knowledge expression and generation patterns.The authors extend this interpretation to tool-use learning experiments.Source: §5.4 · p. 8
  • In tool-learning comparisons, K2 may generate better search-query thoughts than Alpaca-GPT4 because it contains more geoscience knowledge.Both foundation models are trained with the same GeoTool dataset and evaluated on 50 open geoscience questions.Source: §5.4 · p. 8, Table 9 · p. 8

6 APPLICATION

The application studies present K2 as a geoscience research assistant that can generate and revise domain-specific text, use external search tools, and support knowledge reasoning. The examples also explore idea generation from co-occurring scientific concepts.

6.1 Case Study 1: K2 for research assistance

  • K2 can understand geoscience materials, revise statements, generate descriptions, and answer questions as a professional research assistant.The paper frames this capability as acting like a geoscience knowledge base.Source: §6.1 · p. 9
  • With GeoSearch, K2 generates a search thought, calls the GAKG API, updates its action from the observation, and produces an answer prompt.Figure 7 illustrates this tool-augmented interaction.Source: §6.1 · p. 9

6.2 Case Study 2: K2 for knowledge reasoning

  • K2 is used for knowledge reasoning and idea generation by producing abstracts that express potential scientific relations between co-occurring concepts.Table 10 provides an example of this application.Source: §6.2 · p. 9, Table 10 · p. 9

7 CONCLUSION

The paper introduces K2 as a geoscience foundation model together with GeoSignal and GeoBench, and reports effectiveness on geoscience knowledge understanding and utilization. It also presents tool use, knowledge reasoning, research assistance, and open-source release as supported applications and resources.

7 CONCLUSION

  • K2 is introduced as a geoscience foundation model evaluated with the GeoSignal instruction dataset and GeoBench benchmark.The paper reports effectiveness against similar-size language models and describes applications in knowledge reasoning and research assistance.Source: §7 · p. 9

Related papers

Loading 2306.05064v2…