Source-linked AI summary
Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework
Komal Kumar, Aman Chadha, Salman Khan, Fahad Shahbaz Khan, Hisham Cholakkal
TL;DR
Researchers face an expanding literature and fragmented tools for discovering, evaluating, and synthesizing relevant work. Paper Circle addresses this gap with coordinated discovery and analysis pipelines that produce structured, traceable outputs; retrieval performance varies across methods, while review-agent alignment with human judgments remains weak.
Problem
Rapidly expanding scientific literature and limited search-system depth make it difficult to efficiently discover, evaluate, and synthesize relevant work.
Method
Paper Circle combines multi-agent literature discovery with a paper-analysis pipeline that constructs provenance-rich typed knowledge graphs for graph-aware question answering and verification.
Results
Two agent models achieve an 80% hit rate, while BM25 reaches 78% and BM25 plus reranking achieves MRR 0.8692 and R@5 0.9400 at approximately 28× slower speed.
Takeaways & Limitations
Paper Circle provides a reproducible literature-management workflow that unifies retrieval, scoring, structured outputs, knowledge graphs, graph-aware question answering, and coverage checks.
Takeaways & Limitations
Review agents align weakly with human judgments, with correlations below 0.25, so they should not be trusted to compare or rank papers.
Abstract
from arXiv · showhide
The rapid growth of scientific literature has made it increasingly difficult for researchers to efficiently discover, evaluate, and synthesize relevant work. Recent advances in multi-agent large language models (LLMs) have demonstrated strong potential for understanding user intent and are being trained to utilize various tools. In this paper, we introduce Paper Circle, a multi-agent research discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature. The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-criteria scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, experiments, and figures, enabling graph-aware question answering and coverage verification. Both pipelines are implemented within a coder LLM-based multi-agent orchestration framework and produce fully reproducible, synchronized outputs including JSON, CSV, BibTeX, Markdown, and HTML at each agent step. This paper describes the system architecture, agent roles, retrieval and scoring methods, knowledge graph schema, and evaluation interfaces that together form the Paper Circle research workflow. We benchmark Paper Circle on both paper retrieval and paper review generation, reporting hit rate, MRR, and Recall at K. Results show consistent improvements with stronger agent models. We have publicly released the website at https://papercircle.vercel.app/ and the code at https://github.com/MAXNORM8650/papercircle.
1 Introduction
Paper Circle is a collaborative, multi-agent workbench that addresses fragmented literature workflows by supporting discovery, analysis, critique, and synthesis. Its design augments researchers with structured retrieval, knowledge graphs, and specialized review agents.
- Paper Circle addresses the gap between fully autonomous research simulations and the collaborative needs of human research communities.
- The platform supports the literature-engagement lifecycle from discovering and analyzing papers to critiquing and synthesizing them.
- Unlike systems intended to replace researchers, Paper Circle is designed as a collaborative workbench that augments human intelligence.
- Discovery Pipeline: Its Discovery Pipeline uses multi-dimensional scoring beyond keyword matching and produces deterministic structured artifacts at each step.
- Paper Mind Graph: Its Paper Mind Graph represents retrieved literature as a dynamic knowledge graph supporting citation-subgraph-grounded question answering and latent-connection discovery.
- Review Agents: Specialized Review Agents generate detailed critiques and scores that highlight strengths and weaknesses for human reading priorities.
2 Related Work
Paper Circle is positioned within research on autonomous scientific agents, domain-specific scientific systems, and agent-mediated collaboration. It prioritizes transparent, reproducible literature curation while providing a general-purpose foundation for knowledge management.
- Unlike end-to-end AI-Scientist systems, Paper Circle prioritizes curation and reproducibility over full research automation.
- The framework treats literature review as a collaborative, transparent, and verifiable process rather than replacing the researcher.
- Paper Circle complements domain-specific scientific agents with a general-purpose discovery pipeline adaptable across disciplines.
3 Methodology
Paper Circle uses a coder-LLM multi-agent framework to coordinate discovery and paper-analysis pipelines. Discovery iteratively retrieves and ranks literature, while analysis constructs traceable knowledge graphs for graph-aware question answering and verification.
- Background: Multi-agent systems decompose literature search, reading, and reasoning into specialized agent routines that can use external tools.
- Background: The orchestration layer uses a CodeAgent to plan, invoke tools, and delegate work to capability-specific ToolCallingAgents.
- Paper Discovery Agent Design: The Discovery Pipeline combines explicit tool interfaces with intent classification, paper search, analysis, export, and web-search agents.
- Paper Discovery Agent Design: The discovery framework maintains an evolving state and iteratively refines an initially empty draft through orchestrated agentic steps.
- Paper Analysis Agent: The Paper Analysis Agent parses papers, creates structure-aware chunks, constructs typed knowledge graphs, and supports traceable graph-aware question answering.
- Paper Analysis Agent: The knowledge-graph schema represents papers, sections, concepts, methods, experiments, datasets, and visual elements with provenance metadata.
- Paper Analysis Agent: Graph construction stages extract concepts, methods, experiments, visual linkages, and inter-concept relations into a shared MindGraph.
- Paper Analysis Agent: Coverage verification checks figures, tables, sections, and equations for omissions before downstream use.
4 Experiments
Paper Circle is evaluated through retrieval benchmarks, paper-analysis outputs, review generation, user studies, and ablations of retrieval and pipeline configurations. Results show strong retrieval performance, usability, and clear accuracy–efficiency and model-capacity trade-offs.
- Experimental setup: The evaluation combines retrieval metrics, paper-analysis visualizations, review generation, qualitative discovery sessions, usability assessments, and ablation studies.Retrieval experiments measure success rate, hit rate, MRR, and recall across 50 queries; other evaluations examine generated analyses, reviews, user workload, and system components.
- Model comparison: 80% hit rate was achieved by qwen3-coder-30b-Q3KM and qwen3-coder:30b, while qwen3-coder-30b-Q3KM reached MRR = 0.627 at approximately 21–22 seconds per query.BM25 remained competitive at 78% HR, outperforming most agent-based approaches.
- Paper review analysis: Review quality improved with larger models, while qwen3-coder-30B often struggled to sustain a coherent workflow compared with chat-style LLMs such as gpt-oss.The review study used 50 ICLR 2024 papers spanning diverse rating levels.
- Qualitative assessment: 81 discovery sessions across 9 research domains indicated low workload and strong usability, with NASA-TLX workload at 1.2/7 and positive usability items averaging 7.6/10.Negative usability items averaged 2.6/10; learnability was rated 8/10 and learning barrier 1/10.
- Retrieval baseline ablations: BM25-based retrieval outperformed pure semantic retrieval at R@1, while BM25 plus reranking achieved MRR = 0.8692 and R@5 = 0.9400 at approximately 28× the computational cost.The hybrid approach performed on par with BM25, and reranking exposed an accuracy–efficiency trade-off.
5 Conclusion
Paper Circle combines reproducible discovery with knowledge-graph analysis to streamline literature management. Its analysis pipeline supports graph-aware question answering, coverage checks, and human-in-the-loop verification, while future work targets pipeline unification.
- Conclusion: The discovery pipeline unifies heterogeneous search, multi-criteria scoring, deterministic ranking, and synchronized multi-format outputs.It uses a shared agent–tool interface and reproducible workflow.
- Conclusion: The analysis pipeline converts papers into structured knowledge graphs supporting graph-aware question answering and coverage checks.The workflow also includes human-in-the-loop verification.
- Conclusion: Future work will focus on optimizing the unification of the pipeline.
6 Limitations
The review agent does not align reliably with human judgments. Its low or negative correlations mean it should not be trusted to compare or rank papers.
- Limitations: Across models, correlation with human reviewer scores remains low at r < 0.25, with some metrics showing negative correlations.Negative correlations indicate that rankings can oppose human preferences.
- Limitations: The review agent does not reliably distinguish strong from weak submissions.The paper therefore advises against using it as a trusted mechanism for comparing or ranking papers.
- Limitations: The authors report that larger open- or closed-source models may overcome this review-process problem.
A Paper Review Results
The paper evaluates four tool-enabled LLMs on predicting human review scores for 50 ICLR 2024 papers across multiple review dimensions and metrics. The largest model performs best on overall rating and contribution, but correlations with human scores remain weak.
- Paper Review Results: The evaluation samples 50 ICLR 2024 papers and compares four tool-enabled LLMs against human scores.Models predict overall rating, soundness, presentation, and contribution.
- Paper Review Results: The study reports regression error, Pearson and Spearman association, thresholded accuracy, and signed-error statistics.
- Paper Review Results: gpt-oss:120b achieves rating MAE = 1.68 and contribution MAE = 0.62, the best overall accuracy on those dimensions.
- Paper Review Results: gpt-oss:20b is competitive and often stronger on technical subscores such as soundness and presentation.
- Paper Review Results: Despite moderate absolute errors, correlations with human scores generally remain |r| < 0.25 across models.This indicates difficulty preserving the relative ranking of papers.
B System Overview
Paper Circle is a full-stack platform built around complementary discovery and analysis pipelines. The discovery workflow coordinates specialized agents for retrieval and ranking, while the analysis workflow transforms PDFs into traceable knowledge graphs.
- System Overview: Paper Circle provides a React/TypeScript/Vite/TailwindCSS frontend and a Python backend with FastAPI and Supabase storage.The platform supports discovery, reading circles, discussion, and storage for users, communities, papers, and sessions.
- System Overview: The system has complementary Discovery and Analysis pipelines for finding relevant papers and understanding individual papers.
- System Overview: The discovery pipeline uses six agents: intent classification, paper search, sorting, analysis, export, and web search.A CodeAgent coordinates them, selects offline versus online search, and enforces a minimal-step policy.
- System Overview: The analysis pipeline transforms PDFs into typed knowledge graphs through concept, method, experiment, and linkage extraction agents.The resulting graphs support question answering, coverage verification, and traceability to source locations.
- System Overview: PipelineState logs each step and regenerates synchronized JSON, CSV, BibTeX, Markdown, and evaluation artifacts.
- System Overview: Offline retrieval uses BM25 by default, with optional semantic, hybrid, and cross-encoder reranking; online search aggregates multiple scholarly sources.
- System Overview: Retrieved papers are scored using recency, query similarity, novelty, citation count, and BM25, with weighted combined ranking available.
B.4 Analysis and Monitoring
Paper Circle combines offline and online retrieval with intent-aware querying, multi-criteria scoring, reranking, and diversity-aware selection. Its monitoring and ranking controls support reproducible, configurable literature discovery.
- Retrieval: Offline search loads filtered JSON corpora, constructs searchable text, indexes papers with BM25, and returns ranked candidates.Optional semantic scoring, hybrid scoring, and cross-encoder reranking extend the first-stage retrieval process.
- Retrieval: Online search queries arXiv, Semantic Scholar, OpenAlex, and DBLP in parallel, normalizes results, and merges them into a common paper structure.Two-stage deduplication uses DOI matches and normalized titles while preferring richer metadata.
- Query construction: The query-generation agent extracts keywords, constraints, related terms, exclusions, and plausible titles to represent user intent across heterogeneous sources.The specification is intended to provide more consistent query construction than raw keyword matching.
- Scoring: Paper Circle scores papers using similarity, recency, novelty, and BM25 dimensions, then combines them with mode-specific weights or user overrides.Stable mode emphasizes relevance and authority, discovery mode emphasizes novelty, and balanced mode distributes weight across dimensions.
- Ranking: Cross-encoder reranking first retrieves a larger candidate set and then produces the final top-k list to balance efficiency with ranking quality.The reranker computes query-document relevance through a transformer-based cross-attention model.
- Diversity-aware ranking: MMR postprocessing selects papers by combining query relevance with dissimilarity to selected items, improving topical coverage across results.The parameter λ controls the relevance–diversity trade-off, with discovery mode using λ = 0.5.
E.4 Insight Generation
Paper Circle generates aggregate literature insights and synchronized exports from each discovery session. These artifacts support dashboard-based interpretation, auditing, and reuse across curation and reporting tasks.
- Generated insights: The pipeline generates publication trends, source distributions, prolific authors, citation leaders, hot topics, and open-access availability statistics.These insights are stored in summary.json and displayed on the dashboard.
- Generated insights: Open-access availability is reported as the percentage of papers with direct PDF links.The statistic is one of the dashboard’s automatically generated literature insights.
- Structured outputs: Each agent step maintains synchronized artifacts including paper metadata, links, statistics, summaries, and retrieval metrics.Additional CSV, BibTeX, Markdown, and HTML exports support curation, citation management, and reporting.
- Structured outputs: The discovery endpoint returns search specifications, ranked papers, scores, and configurable mode weights through a FastAPI interface.Weights can be queried or overridden at runtime to customize relevance, authority, and novelty trade-offs.
G Evaluation
Paper Circle evaluates retrieval configurations and supports analysis of individual papers through a provenance-rich knowledge graph. Graph retrieval, coverage checking, and human review connect answers to paper evidence.
- Evaluation: Evaluation covers retrieval effectiveness, ranking stability and reproducibility, and the utility of diversity-aware postprocessing.The system computes MRR, Recall@K, Precision@K, and hit rates when ground-truth identifiers are supplied.
- Evaluation: Repeated offline and online runs enable configuration comparisons with MRR and Recall@K, while deterministic scoring supports stable-ranking checks.The illustrative evaluation is designed around discovery goals rather than task-specific question-answering benchmarks.
- Evaluation: Batch benchmarking runs multiple queries concurrently and aggregates mean metrics and timing statistics across retrieval configurations.Comparisons include offline versus online search, BM25 versus semantic retrieval, and reranking enabled versus disabled.
- Knowledge graph: The mind graph represents papers, sections, concepts, methods, experiments, datasets, and visual elements with typed semantic and structural relations.Nodes and edges retain source chunks, page numbers, verification status, confidence scores, and timestamps.
- Knowledge graph: Four specialized extraction agents identify concepts, methods, experiments and datasets, and links between figures or tables and relevant concepts or methods.Extraction proceeds through five phases that update the shared MindGraph.
- Graph-aware question answering: GraphRetriever combines vector retrieval with one-hop graph expansion, returning similar chunks and nodes together with connecting edges for PaperQA answers.Responses include supporting sections, relevant figures and tables, and a confidence estimate.
- Quality assurance: CoverageChecker reports figure, table, section, and equation coverage, along with an overall score, unlinked items, suggestions, and critical issues.VerificationManager supports human edits and records verification status and edit history.
H Implementation and Deployment
Paper Circle uses a Python and FastAPI backend with retrieval libraries and exposes both multi-agent and deterministic discovery pipelines. A React/TypeScript frontend connects to persistent storage and deployment configurations.
- Backend: The backend uses FastAPI, scikit-learn, NumPy, and pandas for service endpoints, retrieval, scoring, and data processing.The system exposes a fast discovery variant designed for low-latency responses.
- Backend: The multi-agent discovery pipeline and refactored deterministic pipeline are implemented in separate backend components and exposed through API servers.Environment variables control API URLs and database credentials for local or hosted deployment.
- Frontend and deployment: The frontend is built with React and TypeScript, while Supabase provides authentication and persistent storage for profiles, communities, sessions, and paper metadata.Docker and deployment configurations support platforms including Railway, Render, and Vercel.