PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperFlow: Profiling, Recommending, and Adapting Across Daily Paper Streams

Fuqiang Wang, Song Tan, Zheng Guo, Jiaohao Fu, Xinglong Xu, Bihui Yu, Jie Dong, Zheng Sun, Siyuan Li, Jingxuan Wei, Cheng Tan

arXiv:2606.07454v1cs.IRcs.AI

TL;DR

Scientific paper recommendation is usually treated as static ranking, despite daily reading, accumulating feedback, and shifting interests. PaperFlow couples structured profiling, multi-signal recommendation, and feedback-driven adaptation within a longitudinal user-day benchmark. It achieves the strongest reported ranking, behavioral-alignment, and blind human-evaluation results, while relying primarily on simulated users and labels.

  • Problem

    Static fixed-candidate ranking does not represent daily scientific reading, accumulating feedback, limited reading budgets, and evolving interests.

  • Method

    PaperFlow couples an editable scholarly profile, multi-signal daily ranking, semantically distinct feedback updates, interest-drift modeling, and a temporally controlled user-day benchmark.

  • Results

    PaperFlow achieves the strongest automatic ranking, highest alignment with simulated reading selections, and highest blind human score among the evaluated baselines.

  • Takeaways & Limitations

    The framework provides a sequential evaluation setting linking daily recommendation quality with simulated behavior and blind expert judgments.

  • Takeaways & Limitations

    The benchmark primarily uses simulated users and relevance labels, is mainly derived from arXiv streams, and may differ in coverage across fields and venues.

Abstract

from arXiv · show

Scientific paper recommendation is typically evaluated as static ranking over a fixed candidate set, yet real scientific reading unfolds as a daily, longitudinal process in which interests shift and feedback accumulates. We introduce PaperFlow, a framework that organizes it into three coupled stages: Profiling, which constructs and maintains a structured, inspectable scholarly profile from heterogeneous cold-start evidence; Recommending, which ranks each date-specific paper stream through multi-signal aggregation under a fixed display budget; and Adapting, which updates user state from semantically distinct feedback signals and models interest drift across days. We further define a longitudinal user-day benchmark that fixes users, dates, candidate pools, visible inputs, and hidden simulated relevance labels under a shared temporal information boundary. The benchmark contains 24 simulated research users, 50 daily paper streams, 1,200 user-day episodes, 20,727 unique papers, and 497,448 episode-paper records. We additionally specify a blind human-evaluation protocol to validate alignment between automatic metrics and expert judgments. Experiments against five scientific recommendation baselines show that PaperFlow achieves the strongest oracle-based ranking, the highest behavioral alignment with simulated reading selections, and the best blind human-evaluation score.

1 Introduction

PaperFlow reframes scientific paper recommendation as a longitudinal daily process requiring coordinated profiling, ranking, and adaptation. It also introduces a temporally controlled benchmark for evaluating this loop without future-information leakage.

  • Traditional recommendation overlooks daily paper streams, limited reading time, skipped papers, and evolving research interests.
  • Prior systems address daily recommendation, profile transparency, or interest drift separately rather than as one controlled temporal user-day task.
  • PaperFlow couples structured profiling, multi-signal daily ranking, signal-aware feedback updates, and behavior-driven drift modeling in one loop.
  • The benchmark fixes users, dates, candidate pools, visible inputs, historical feedback, and hidden relevance labels under a shared temporal boundary.
  • The benchmark contains 24 simulated users, 50 days, 1,200 user-day episodes, 20,727 unique papers, and 497,448 episode-paper records.

2 Related Work

Related work spans scientific recommendation, reading assistance, and dynamic personalization, but these strands are generally studied separately. PaperFlow connects them through an updateable scholarly profile and daily feedback loop.

  • Scientific Paper Recommendation: Scientific recommendation must jointly address relevance estimation, cold start, and profile interpretability.
  • Scientific Paper Recommendation: Prior recommendation methods improve matching, scientific representations, transparency, or conversational interaction through specialized signals and tools.
  • PaperFlow’s Position: PaperFlow maintains a structured, updateable scholarly profile across daily recommendation and feedback loops.
  • Paper Reading Assistants: Reading assistants support search, survey generation, synthesis, screening, interactive reading, and citation-focused comprehension.
  • Dynamic Feedback and Interest Drift: Dynamic recommendation studies distinguish short-term exploration from stable preference migration and examine multi-round feedback and preference updating.

3 PaperFlow Method

PaperFlow implements a dynamic personalized reading loop built around an editable scholarly profile, multi-signal ranking, semantically separated feedback, and behavior-driven interest drift. Its drift mechanism uses sustained, consistent evidence rather than isolated interactions to update topic weights.

  • System Overview: PaperFlow constructs a structured scholarly profile from cold-start evidence and carries it through daily recommendation and feedback updates.The profile includes research directions, topic weights, priors, preferences, report settings, reading behavior, and drift state.
  • Daily Recommendation: Daily ranking aggregates long-term interest matching, author and institution priors, dynamic behavioral signals, and explicit must-read rules.
  • Feedback Updating: Feedback semantics remain separate: selections and edits provide strong interest evidence, skips weak negative evidence, reading behavior updates behavior state, and report feedback updates report preferences.
  • Behavior-Driven Interest Drift: PaperFlow distinguishes transient exploration from sustained migration using drift evidence derived from long-term interests, recent behavior, feedback, and reading state.
  • Behavior-Driven Interest Drift: Four drift states—Stable, Observing, Shifting, and Recovered—transition only after thresholded, sustained, directionally consistent evidence.Per-round topic-weight changes are bounded by an upper update limit.

4 PaperFlow Benchmark

The PaperFlow benchmark converts daily paper streams and simulated researcher profiles into date-frozen user-day episodes with temporally separated inputs and evaluation signals. It supports Top-20 ranking evaluation using reproducible pseudo-oracle labels, behavioral selections, and blind human judgments.

  • Benchmark Construction: The benchmark freezes daily paper pools, initializes dynamic profiles, executes recommendation episodes, records traces, and exports separated inputs and evaluation signals.
  • Benchmark Construction: Date-specific candidate pools contain only papers visible that day, isolating ranking from temporal leakage and later metadata changes.
  • Benchmark Construction: The benchmark initializes 24 simulated researcher profiles with structured interests, preferences, and drift plans for reproducible cold-start and migration tests.
  • Evaluation Protocol: Each episode returns a Top-20 list from a frozen candidate pool while same-day selections, hidden labels, and drift outcomes remain unavailable until evaluation.
  • Benchmark Scale: The benchmark contains 24 researchers, 50 daily pools, 1,200 episodes, 20,727 papers, and 497,448 episode-paper records.
  • Benchmark Analysis: A Top-20 list exposes about 4.8% of an average daily pool, and the simulator selects 3,104 of 24,000 shown papers for deeper reading.The resulting feedback is sparse and temporally ordered.
  • Evaluation Protocol: Automatic metrics combine oracle-based ranking, SelectedNDCG@20 for simulated reading selections, and HumanEval for blind listwise human alignment.

5 Experiments

Experiments show that PaperFlow performs strongly across recommendation quality, behavioral alignment, model comparisons, ablations, and interest-drift adaptation. Results also examine human alignment, token cost, real-user evaluation, and the closed-loop reading workflow.

  • Main Recommendation Results: PaperFlow improves gNDCG@20 from 39.00 to 50.65, RecommendationScore from 46.30 to 55.31, and HumanEval from 55.56 to 65.56 over Scholar Inbox.Scholar Inbox is identified as the strongest external baseline by RecommendationScore.
  • Main Recommendation Results: SelectedNDCG@20 increases from 33.47 to 70.88, indicating stronger alignment with simulated reading selections.The result reflects behavioral alignment in addition to hidden-label relevance.
  • LLM Comparison Results: Grok 4.3 achieves RecommendationScore 56.31 and ModelHumanScore 94.07, while Gemini 3 Flash provides strong ModelAutoScore 64.19 as the default backbone.Model choice is compared under fixed retrieval, embeddings, candidate pools, and evaluation conditions.
  • LLM Comparison Results: Pearson’s r=0.9632 links ModelAutoScore with ModelHumanScore across LLM backbones.The comparison evaluates automatic and blind human-aligned quality under the model-comparison protocol.
  • Ablation Analysis: Full PaperFlow achieves the best SelectedNDCG@20, whereas simplified variants can improve oracle-based relevance concentration.The ablation exposes a static–dynamic trade-off between stable pseudo-oracle relevance and downstream reading alignment.
  • Interest Drift Analysis: PaperFlow leads adaptation-oriented signals, including PostDrift SelectedNDCG@20, NewTopicRecall@20, OldTopicRate@20, adaptation delay, DriftAutoScore 72.76, and AdaptationHumanScore 68.75.The drift analysis compares PaperFlow with versions without drift modeling or with a fixed profile.

6 Conclusion

PaperFlow combines structured profiles, daily multi-signal ranking, feedback-driven updates, and interest-drift modeling in a longitudinal benchmark. Its main evidence is controlled and reproducible, but relies primarily on simulated users and relevance labels.

  • PaperFlow combines structured scholarly profiles, multi-signal daily ranking, feedback-driven state updates, and interest-drift modeling.
  • The benchmark fixes longitudinal user-day episodes, candidate pools, hidden relevance labels, and diagnostic trajectories under a shared temporal boundary.
  • PaperFlow primarily uses simulated research users and simulated relevance labels, while a real-user pilot serves as a user-experience sanity check.
  • The simulated labels support reproducibility and controlled temporal comparison but are not human-annotated truth or real-user deployment logs; coverage is mainly arXiv-based.

A.1 Benchmark Scope

PaperFlow evaluates daily recommendation as a fixed-pool, temporally ordered user-day process rather than as isolated ranking. Each episode records recommendation context, simulated selections, profile dynamics, and evaluation-only fields across a diverse simulated user set.

  • Episode structure: Each episode represents one user on one day, ranks a fixed candidate pool, displays the Top-20 papers, and simulates selection of papers for later reading.
  • Benchmark controls: The fixed-pool design compares all methods on identical candidate sets while preserving temporal order for longitudinal profile and behavior modeling.
  • Episode structure: Evaluation and logging fields such as oracle_label and selected are withheld from ranking inputs to prevent evaluation leakage.
  • User scope: The benchmark spans 24 simulated users across AI, life science, physics, earth science, medicine, social science, and science-of-science topics.
  • User scope: Profiles contain long-term interests, short-term interests, explicit preferences, and behavioral state, rather than a simple keyword set.

C.1 Ranking Score Parameters

PaperFlow’s ranking and adaptation components combine semantic, profile, preference, quality, drift, and reading signals, while its evaluation separates static ranking, behavioral alignment, adaptation, reporting, and human-validation dimensions.

  • Ranking score: The base ranking score combines semantic matching, profile matching, preference signals, paper quality, drift adaptation, reading feedback, and suppression penalties.
  • Ranking signals: Explicit preferences complement embedding similarity by enforcing author, institution, task, or keyword rules that semantic matching may miss.
  • Adaptation: Interest drift improves responsiveness to changing interests but may introduce a small exploration cost on static oracle-based ranking metrics.
  • Adaptation: Reading signals provide lightweight short-term bonuses and require repeated selections within a recent window before becoming short-term preferences.
  • Reading reports: A reading report is post-recommendation assistance rather than part of the ranking metric, helping users decide how and why to read selected papers.
  • Evaluation metrics: DriftAutoScore aggregates post-drift ranking, behavioral alignment, new-topic recall, old-topic suppression, and adaptation speed.

D.13 Relationship Among Metrics

PaperFlow distinguishes static oracle-based ranking from behavior-based alignment and adaptation metrics because these objectives can diverge. Its evaluation also separates human scores by task and uses structured prompts for parsing, explanation, and reading assistance rather than direct label-based recommendation.

  • Metric relationships: Oracle-based metrics assess static relevance ranking, while SelectedNDCG@20 measures agreement with simulated user behavior.
  • Metric relationships: Removing drift can improve conservative oracle-based ranking, whereas full PaperFlow may better match subsequent selections through drift exploration.
  • Metric relationships: This divergence reflects a trade-off between static relevance and dynamic behavioral adaptation rather than a metric error.
  • Prompted components: Prompts support structured parsing, recommendation explanations, and reading-report generation; ranking is determined by profiles, paper content, rules, and structured model fields.
  • Prompted components: Recommendation explanations emphasize faithfulness and restraint, avoiding unsupported claims that topical similarity implies direct methodological relevance.
  • Human validation: Human evaluation uses separate scores for main recommendation quality, LLM comparison, and interest-drift adaptation.

F.4 Inter-Annotator Agreement and Correlation Analysis

The evaluation reports strong agreement among human annotators and substantial alignment between automatic metrics and human judgments across main, drift-specific, and model-comparison analyses.

  • Inter-Annotator Agreement: Mean pairwise Spearman correlations range from 0.844 for DiversityFocusBalance to 0.923 for ProfileMatch across the main listwise evaluation.RankingQuality, DecisionUsefulness, and OverallRank achieve 0.909, 0.872, and 0.901, respectively.
  • Automatic–Human Correlation: RecommendationScore correlates with HumanEval at Pearson r = 0.8626 and Spearman ρ = 0.8631 across 36 annotated Top-20 lists.gNDCG@20 also correlates with RankingQuality at Pearson r = 0.8723 and Spearman ρ = 0.8774.
  • Automatic–Human Correlation: DriftAutoScore correlates with AdaptationHumanScore at Pearson r = 0.9149 and Spearman ρ = 0.8904 across 72 annotated lists.The drift-specific analysis evaluates method–drift-episode recommendation lists.
  • Automatic–Human Correlation: ModelAutoScore correlates with ModelHumanScore at Pearson r = 0.9632 and Spearman ρ = 0.9648 across 14 completed LLM backbones.These correlations assess whether automatic metrics reproduce human-judgment trends without replacing the individual metrics in result tables.
  • Real-User Pilot: The real-user pilot complements the simulated benchmark with five graduate participants using PaperFlow for 5–7 interaction rounds.Participants mark papers worth reading, inspect papers, and optionally request reading reports; the pilot is not statistically powered for deployment evaluation.
  • Real-User Pilot: Prec@5 measures top-list concentration, Prec@20 measures full-list usefulness, and ReadRate captures stronger engagement through concrete reading actions.Satisfaction and ReportUse summarize questionnaire responses, with ReportUse computed only for PaperFlow; participant-round averages are then averaged over five participants.

G.6 Per-User Results

The per-user-results section uses case studies to complement aggregate metrics with qualitative evidence across different recommendation states. The cases examine successful matching, changing interests, behavioral agreement, boundary conditions, and reading support.

  • Scope: The pilot study is limited in scale, so its results are used as complementary evidence without statistical significance claims.Participants are anonymized and only aggregate or per-user averaged statistics are reported.
  • Case-Study Design: Case studies complement aggregate metrics by examining successful recommendation, interest drift, behavior consistency, boundary disagreement, and reading-report support.The selection covers episodes with high ranking quality, drift transitions, behavioral agreement, system–oracle disagreement, or complete PDF-based reports.
  • Case-Study Design: The case-study set avoids selecting only strong outputs and checks aligned matches, changing interests, behavior-aware ranking, and oracle–behavior discrepancies.This design evaluates PaperFlow across different recommendation states rather than only its strongest outputs.

H.2 Successful Recommendation: A Dense Top-20 for an NLP User

The successful-recommendation case shows PaperFlow producing a dense, high-quality Top-20 for an NLP user when profile, topic, and rule signals align. The surrounding cases illustrate adaptation, behavioral agreement, boundary disagreement, reading assistance, and controlled model comparison.

  • H.2 Successful Recommendation: gNDCG@20 = 1.0000, with 20 useful papers, 10 strictly relevant papers, and five selected papers for the NLP user episode.The episode is user_role6::2026-03-16 and Figure 11 visualizes the ranking evidence.
  • H.2 Successful Recommendation: Profile match, topic similarity, and must-read rules concentrate useful papers near the top and remain consistent with later user selections.Representative papers cover long-context question answering, retrieval-augmented generation, and LLM-supported scientific discovery.
  • H.3 Interest Drift: Repeated new-topic evidence triggers an anchor lock that shifts recommendations from GUI agents and web automation toward multimodal reasoning and computer-using agents.PaperFlow waits until a threshold is reached rather than replacing the profile after a single new-topic hit.
  • H.4 Behavior Consistency: SelectedNDCG@20 evaluates agreement with later user choices, whereas oracle labels measure content-level relevance.This distinction helps interpret behavior-consistency and boundary cases where selections and oracle relevance diverge.
  • H.5 Boundary Case: Boundary cases expose possible discrepancies among oracle labels, behavior-selection signals, and system labels, motivating human evaluation.A selected paper may remain useful or inspiring despite being weakly relevant or irrelevant under the oracle.
  • H.6 Reading Assistance: Reading reports combine abstracts, PDF sections, semantic-retrieval evidence, and user profiles to provide profile-aware guidance after screening.They explain relevance, prioritize sections, and describe how a paper may support current research, reducing reading cost after screening.
  • LLM Comparison: TokenCost is reported separately as an efficiency metric and excluded from ModelAutoScore and ModelHumanScore.The comparison fixes candidate pools, profiles, Top-20 budget, embedding model, and evaluation metrics across models.

J.3 Error Handling

The error-handling design distinguishes recoverable failures, which can be bypassed through fallbacks, from unrecoverable failures that require fixes and reruns. Incomplete model experiments must be marked rather than compared directly with complete runs.

  • Error Classes: Recoverable failures include arXiv timeouts followed by successful PDF parsing, while unrecoverable failures include path-length errors, complete LLM unavailability, or unparseable output.Recoverable failures usually do not affect the main flow; unrecoverable failures require code or configuration fixes and rerunning the experiment.
  • Fallbacks: If metadata retrieval fails, PaperFlow attempts PDF download; if PDF retrieval fails, it generates a simplified report from the title and abstract.These fallbacks address selected external API and document-retrieval failures.
  • Reporting Boundary: Incomplete experiments caused by API instability must be marked incomplete instead of being directly compared with complete runs.This prevents partial results from being treated as equivalent to full model runs.
  • Case-Study Coverage: Case studies cover successful recommendation, drift adaptation, behavior consistency, boundary disagreement, and reading-report support.These representative cases inspect PaperFlow beyond aggregate metrics.
Loading 2606.07454v1…