Source-linked AI summary
Efficient Context-Limited Telescope Bibliography Classification for the WASP-2025 Shared Task Using SciBERT
Madhusudhana Naidu
TL;DR
Telescope bibliographies are important for assessing observatory impact and reproducibility, yet their creation remains largely manual. This paper fine-tunes SciBERT for four-category classification within a 512-token context limit. It reports leaderboard F1 of 0.89 despite truncating about half of samples exceeding that limit.
Problem
Creating telescope bibliographies requires identifying relevant papers, disambiguating telescope mentions, and classifying data use, work still largely performed manually.
Source: §1 · p. 1Method
The paper fine-tunes SciBERT for multi-label classification of science, instrumentation, mention, and not telescope using a maximum 512-token input.
Source: §1 · p. 1, §3.2 · p. 2Results
F1 = 0.89 on the leaderboard was achieved despite truncation of roughly 50% of samples exceeding 512 tokens.
Source: §1 · p. 1, §3.2 · p. 2Takeaways & Limitations
The findings support AI-assisted telescope bibliography curation as a potential aid to reproducibility and discoverability of astronomical data.
Source: §1 · p. 1, §6 · p. 3Chunked windows and a 4096-token Longformer backbone are proposed as extensions contingent on more time and compute.
Source: §3.3 · p. 2
Abstract
from arXiv · showhide
The creation of telescope bibliographies is a crucial part of assessing the scientific impact of observatories and ensuring reproducibility in astronomy. This task involves identifying, categorizing, and linking scientific publications that reference or use specific telescopes. However, this process remains largely manual and resource intensive. In this work, we present an efficient SciBERT-based approach for automatic classification of scientific papers into four categories - science, instrumentation, mention, and not telescope. Despite strict context-length constraints (maximum 512 tokens) and limited compute resources, our approach achieved a macro F1 score of 0.89, ranking at the top of the WASP-2025 leaderboard. We analyze the effect of truncation and show that even with half the samples exceeding the token limit, SciBERT's domain alignment enables robust classification. We discuss trade-offs between truncation, chunking, and long-context models, providing insights into the efficiency frontier for scientific text curation.
1 Introduction
Telescope bibliographies support assessment of observatory impact, but identifying and classifying relevant papers remains largely manual. This work develops a lightweight SciBERT approach for the four-way classification task and reports top leaderboard performance under a 512-token limit.
1 Introduction
- Automating bibliography curation could benefit librarians, archivists, and scientists by improving reproducibility and discoverability of astronomical data.The current process of identifying relevant papers, disambiguating telescope mentions, and classifying data use remains largely manual.Source: §1 · p. 1
- The WASP-2025 task asks systems to identify telescope references and classify papers as science, instrumentation, mention, or not telescope.The input may include titles, abstracts, body text, acknowledgments, and grants.Source: §1 · p. 1
- The proposed lightweight SciBERT model operates within a 512-token window despite a combined row sample context of 100k tokens.The paper also analyzes trade-offs between token truncation and classification performance.Source: §1 · p. 1
- Using only Kaggle GPU resources, the system achieves top leaderboard performance with F1 = 0.89; the work also demonstrates that domain-specific SciBERT can outperform large-scale general models in constrained settings.Source: §1 · p. 1
2 Task and Dataset
The task is to determine whether a paper refers to a telescope and classify its relationship to that telescope. Its central data challenge is very long input text relative to the 512-token context window.
2 Task and Dataset
- The classification labels are science, instrumentation, mention, and not telescope, and papers may receive one or more labels.The task is to identify telescope references and classify the paper's relationship to the telescope.Source: §2 · pp. 1–2
- Records include textual fields, metadata, and target labels, while the training data has significant class imbalance.The listed fields include title, abstract, body, acknowledgments, grants, author, year, and a unique bibcode.Source: §2 · pp. 1–2
- Combined text including the body can exceed 50,000 tokens, while many individual sections exceed standard transformers' 512-token window.This length motivates an efficient model using truncated context.Source: §2 · p. 2
3 Methodology
The methodology compares a TF-IDF and logistic-regression baseline with a SciBERT classifier constrained to 512 tokens. It also identifies chunking and a long-context backbone as possible extensions requiring more time and compute.
3.1 Baseline: TF-IDF + Logistic Regression
- The TF-IDF and One-vs-Rest Logistic Regression baseline achieved macro F1 of 0.66 on training and 0.82 on the test leaderboard.Class-balanced weighting addressed label imbalance, and the model used an 80/20 train-validation split.Source: §3.1 · p. 2
3.2 SciBERT with Truncated Context
- SciBERT was fine-tuned for the four labels using concatenated telescope name, year, title, abstract, acknowledgments, and grants, truncated to 512 tokens.Missing text fields were replaced with empty strings; data were split 80/20 for training and validation.Source: §3.2 · p. 2
- Leaderboard F1 reached 0.89 despite truncating roughly 50% of samples that exceeded 512 tokens.The SciBERT setup used three epochs on Kaggle T4 GPUs and selected its best checkpoint by validation macro F1.Source: §3.2 · p. 2
3.3 Potential Extensions
- With more time and compute, the authors propose overlapping chunked windows with stride 128 or a Longformer backbone with 4096-token context.The proposed chunked-window predictions could use majority voting or mean pooling.Source: §3.3 · p. 2
4 Results
SciBERT outperformed models able to process longer inputs while using less than half of the full context. The results suggest that key signals are concentrated in titles, abstracts, and acknowledgments.
4 Results
- The findings suggest that key discriminative signals are concentrated in titles, abstracts, and acknowledgments.The paper links this observation to SciBERT's robustness under truncation.Source: §4 · p. 2
- Table 1 summarizes median, 75th, and 95th percentile token lengths for key fields excluding the body, noting frequent input-limit exceedances in abstracts and acknowledgments.Source: Table 1 · p. 3
- SciBERT processed less than half of the full context yet outperformed models capable of handling longer inputs.Table 2 reports macro F1 for the model and baselines on validation (CV) and the final leaderboard (LB); the supplied passage does not provide cell values.Source: §4 · p. 2, Table 2 · p. 3
5 Discussion
SciBERT achieved strong telescope-bibliography classification performance despite a 512-token context limit and restricted compute. Acknowledgment text was informative for identifying telescope-related data use, while longer-context approaches may capture additional signals.
5 Discussion
- Macro F1 0.89 outperformed GPT-OSS20B (0.31) and the random submission (0.24) on the leaderboard despite the 512-token limit.Experiments used a Kaggle P100 GPU with 15 GB VRAM and 30 GB CPU RAM.Source: §5 · p. 3
- Acknowledgment text strongly correlated with telescope-related data use, particularly for observatories such as Chandra and Hubble.The authors describe acknowledgments as informative evidence of data usage.Source: §5 · p. 3
- Longer-context architectures or chunked SciBERT could potentially capture broader signals from full body text and further improve classification accuracy.The 512-token maximum context meant longer inputs were truncated.Source: §5 · p. 3
6 Conclusion
The conclusion presents a lightweight SciBERT approach that achieved state-of-the-art results for telescope-related literature classification under strict computational limits. It argues concise context paired with a domain-trained encoder can capture scientific intent and data references, and identifies possible extensions.
6 Conclusion
- The system advanced from a TF-IDF baseline to fine-tuned SciBERT and achieved state-of-the-art results under strict computational limits.The conclusion characterizes the approach as lightweight and high-performing for the WASP-2025 shared task.Source: §6 · p. 3
- Concise context combined with a domain-trained encoder can effectively capture scientific intent and data references in astronomy papers.The conclusion frames this as a finding of the study.Source: §6 · p. 3
- Possible extensions include section-wise modeling, hierarchical encoding, and long-context transformer variants to enhance interpretability or recall.The conclusion also connects AI-assisted curation with observatory bibliography and reproducibility efforts.Source: §6 · p. 3