Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

• The motivating hypothesis is that models that compare support counts correctly may use textual collaborative evidence more effectively.Identifying the largest count and predicting a user's later preference require different judgments.Source: §1 · p. 1
• Behavioral measurements can reveal a weakness without showing which alternative will work better for a particular request.Source: §1 · p. 1
• The study compares selectors that choose among recommendation interfaces using existing features, with or without evidence-reading measurements.Both selectors score all interfaces before choosing, so the comparison measures choice quality rather than saved inference work.Source: §1 · p. 1
• The results distinguish useful collaborative evidence from useful measurements of how a model reads it.The practical comparison includes diagnostic-guided choice, existing-feature selection, and an interface fixed using development data.Source: §1 · p. 2

2.1 Data and the controlled ranking task

• Each request ranks one later positive item against seven unseen alternatives matched for past popularity, using eight-candidate NDCG@5 and Hit@1.Candidate scores are averaged across four fixed cyclic orders; these metrics do not represent full-catalog retrieval or online evaluation.Source: §2.1 · p. 2

2.2 Interfaces and reference policies

• The three interfaces are history-only prompting (H), prompting with retrieved-user support counts (T), and score fusion (F).Fusion combines history-only and user-KNN scores; development data sets its neighborhood size, fusion weight, and scaling method.Source: §2.2 · p. 3

2.3 Evidence-reading tests and selectors

• The augmented selector adds six evidence-reading prompts to a baseline using observable and stability features; both select among interfaces after all are scored.The tests ask models to compare support counts, while stability prompts vary wording and candidate order; each feature set selects its own forecaster settings on development data.Source: §2.3 · pp. 3–4

2.4 Endpoints, inference, and analysis history

• The primary endpoint is the augmented selector's change in NDCG@5 over the baseline, with a 0.005 practical improvement target.Inference resamples users and domains; intervals condition on the fitted selectors and exclude fitting uncertainty.Source: §2.4 · p. 4

3.1 Evidence reading and recommendation choices

• Evidence-reading accuracy ranges from 26.2% to 71.5% across checkpoints, while pairwise balanced accuracy ranges from 23.9% to 48.0%.Qwen3 chooses the first alternative in 95.0% of pairwise cases; Gemma answers none of 54 ties correctly, and Phi and Llama often predict ties on non-ties.Source: §3.1 · p. 5, Figure 1 · p. 5
• Most response-derived reading features vary more within than between checkpoints, with checkpoint identity explaining a median 10.0% of weighted variance.Each feature varies within at least 12 of 24 domain/checkpoint cells; the observed variation could overlap with information in V + S6.Source: §3.1 · p. 5
• −0.0019 NDCG@5 for adding reading features, with a 95% interval of [−0.0046, 0.0004], falls short of the 0.005 target.The interval includes zero, and all four domain point estimates are negative.Source: §3.1 · p. 5, Figure 2 · p. 6

3.2 Fixed-policy references and forecast error

• 0.5524 NDCG@5 for Dev-fixed exceeds 0.5447 for B and 0.5428 for B+R on the same candidate sets.Dev-fixed chooses one interface per domain/checkpoint using development data; this reference was added post hoc.Source: Table 1 · p. 5, §3.2 · p. 6
• 0.1274 evaluation MSE for B+R is slightly below B's 0.1277, a relative reduction of 0.27%.The absolute reduction is 0.0003, with interval [−0.0004, 0.0010].Source: §3.2 · p. 6

3.3 Evidence controls and decision changes

• 0.5229 versus 0.4230 NDCG@5: retrieved similar-user evidence beats an activity-matched random-user control by +0.0999.The difference interval is [0.0881, 0.1114], and all four domain point differences are positive.Source: §3.3 · p. 7
• 22.93% of weighted decisions change interface after adding reading features; 4.30% improve NDCG@5 and 4.76% reduce it.Another 13.87% change interface without changing utility; history-only choices rise from 6.91% to 15.30%, while fusion falls from 58.94% to 49.87%.Source: §3.3 · p. 7

3.4 Support, domain, and timestamp sensitivities

• −0.0025 NDCG@5 remains the estimate after removing requests with no retrieved support for any candidate, with interval [−0.0053, 0.0002].Excluding 29 Last.fm positives dated after the documented endpoint shifts the pooled estimate by +0.00015 to −0.0017, with hierarchical interval [−0.0045, 0.0006].Source: §3.4 · p. 7

4 Discussion and Limitations

• Adding reading features does not deliver the targeted 0.005 improvement for the tested selectors, and both trail an interface fixed on development data.The gain forecaster may contribute to the result, and neither a T–H-only specification nor switching from Dev-fixed when predicted gains exceed a development-set margin was evaluated.Source: §4 · pp. 7–8
• The tests compare visible support counts rather than later user preferences, while reading-feature variation may be useful signal, redundant information, or noise.Answer-position tendencies can also weaken interpretation.Source: §4 · pp. 7–8
• The study covers controlled eight-candidate ranking, small pinned checkpoints, and four domains, while pretrained models and item metadata may contain later information.The synthetic test also omits all-zero cases.Source: §4 · p. 8

5 Reproducibility

• The supplement reproduces primary and secondary statistics from saved paired outcomes, averaged across checkpoints within each user and stripped of original identifiers.Scripts cover fixed-interface and random-user comparisons, matched-hyperparameter fits, and subset analyses.Source: §5 · p. 8
• Aggregate feature moments and confusion counts reproduce the diagnostic summaries.The scripts replay saved outcomes rather than regenerating LLM predictions from source data.Source: §5 · p. 8
• Appendix J describes execution environments, model-refitting portability, source-data checks, and incomplete computation records.Source: §5 · p. 8

6 Conclusion

• Across four domains and six checkpoints, evidence from retrieved similar users improves prompting relative to the random-user control.Source: §6 · p. 8
• Adding features from six evidence-reading prompts does not deliver the plan’s 0.005 NDCG@5 improvement in interface selection.Source: §6 · p. 8
• Evaluate evidence-reading tests by whether their features improve recommendation choices beyond existing inputs and fixed interfaces.Report answer preferences and variation across requests within each checkpoint alongside aggregate accuracy.Source: §6 · p. 8

Ethical Considerations

• Collaborative histories can expose sensitive preferences; removing identifiers from the supplement is not a formal privacy guarantee, and ranking metrics leave fairness, satisfaction, and deployment safety untested.Upstream dataset and model redistribution terms still apply.Source: Ethical Considerations · p. 8

A Data construction and analysis history

• The study reports 1,848 development and 3,426 evaluation users, with user-disjoint splits within each domain.MovieLens items are movies?Source: A Data construction and analysis history · p. 10, Table 2 · p. 10

A.1 Preprocessing and candidate construction

• KuaiRec and Last.fm preprocessing uses timestamp cutoffs, earlier positive histories, and prompts with the latest eight distinct items; retrieved peers are ranked by cosine similarity over past-positive histories.Random peers are distinct and activity-matched; outcome labels are stored separately from prompt inputs.Source: §A.1 · p. 10

A.2 Last.fm eligibility

• Last.fm feasibility records show final disjoint cohorts of 350 development and 426 evaluation users, with top-64 peers supporting the later positive for 285 and 326 users, respectively.The records do not retain an exact per-filter attrition chain.Source: §A.2 · p. 10

A.3 Last.fm release dates and exclusion sensitivity

• Of 153,473 events from 204 users on later UTC dates, none enters prompt histories or retrieved/random-neighbor evidence, but 29 of 426 evaluation positives are later-dated.No development positives are later-dated; the latest evaluated positive is June 19, 2009.Source: §A.3 · pp. 10–11
• The distributed Last.fm file has 19,098,862 rows, 52,006 fewer than documented, although all rows parse and the file identifier matches.The supplement provides the source audit and exclusion mask.Source: §A.3 · p. 11
• Removing 29 later-dated positives does not recover the targeted pooled benefit; the flagged group’s larger negative mean contributes to the full-sample estimate.The 29 exclusions leave 397 retained Last.fm users; the primary analysis retains all 426.Source: §A.3 · p. 11
• The primary and its decision rules remain unchanged in the timestamp sensitivity analysis.The retained-user Hit@1 difference is +0.001005 pooled, with H interval [−0.001655, 0.003726] and F interval [−0.001500, 0.003511].Source: §A.3 · p. 11

A.4 Recorded history and unresolved chronology

• Reported intervals account for sampling uncertainty under the recorded analysis but exclude possible outcome-informed design uncertainty.The table summarizes the study history and remaining limits on confirmatory interpretation.Source: §A.4 · p. 12, Table 5 · p. 12

B Checkpoint and diagnostic specification

• Six pinned instruction-tuned checkpoints are evaluated; Qwen2.5 and Qwen3 form one analysis family, while Gemma and Llama use pinned mirrors checked against official revisions.Exact revisions, tokenizers, and configuration files are listed in the supplement.Source: B Checkpoint and diagnostic… · p. 12

B.1 Test cases and observable request mapping

• Development and evaluation each contain 512 test cases, with repeated representations treated as measurements of the same case; accuracy uses the highest-probability answer, while valid-answer mass sums admissible-answer probabilities.A direct answer is valid for any maximum-support candidate, while a pairwise answer must report a tie for equal counts.Source: §B.1 · p. 12, Table 7 · p. 13
• An outcome-independent mapping groups requests by visible support counts, prioritizing tied maxima, sparse counts, a one-count margin, cases containing zero, then remaining all-positive cases.All-zero cases map to tied maxima and have test-coverage flag zero; the test set contains no all-zero case.Source: §B.1 · p. 12

B.2 Recommendation stability prompts

• Recommendation stability prompts vary wording and candidate order while preserving histories, candidates, metadata, and retrieved support counts; wording and order change together, not in a fully crossed design.Main prompts use rotations 0, 2, 4, and 6; additional H and T prompts use rotations 1, 3, and 5.Source: §B.2 · p. 13

B.3 Zero/one check

• The zero/one check presents a maximum- and minimum-support candidate with counts reset to 1 and 0, measuring response to a constant comparison without recommendation history or metadata.The scorer records probability assigned to the valid higher-support answer and whether it is the highest-probability answer.Source: §B.3 · p. 13

C Development-selected settings

• Development-selected settings include domain-specific neighborhood sizes and domain/checkpoint-specific fusion weights and scaling, while gain-forecast regressors are selected separately for each feature set.Five-fold development cross-validation groups all checkpoint rows from each user; selected settings are then applied without additional search.Source: C Development-selected settings · p. 14, Table 8 · p. 14, Table 9 · p. 14

D Numerical inference and decision rules

• Only MovieLens and Amazon meet the 90% interval-containment condition within [−0.005, 0.005], below the required three domains, so neither protocol designation succeeds.The positive designation also fails.Source: D Numerical inference and decision rules · p. 14

D.1 Domain-subset sensitivities

• The hierarchical pooled bootstrap resamples domains and users, while the fixed-domain sensitivity resamples users with four domains held fixed; both estimate the equal-domain mean but reflect different uncertainty sources.Extrapolation from four domains remains limited.Source: D Numerical inference and decision rules · p. 15, Table 12 · p. 15
• Every reported upper bound in the domain restrictions is below 0.005, but these membership sensitivities retain the same fitted selectors and do not establish independent pre-outcome study design.The intervals condition on fitted policies, and some reporting formulas and bootstrap seeds were unspecified in the plan.Source: §D.1 · p. 15, E Secondary endpoints and reference… · p. 15

E.1 Calibration and evidence difficulty

• The held-out evidence-reading calibration combines task formats, and aggregate difficulty accuracy can mask both tie avoidance and excessive tie responses across checkpoints.The actual label-by-response cross-tab is supplied in the appendix.Source: Table 18 · p. 17, §E.1 · p. 17, Table 19 · p. 17

F.1 Development-selected fixed interface

• Development-selected Dev-fixed interface performance is compared with both learned selectors, but the selectors’ Hit@1 point differences are positive with intervals including zero.The fixed reference chooses the best mean development NDCG@5 interface for each domain/checkpoint cell.Source: Table 21 · p. 18, §F.1 · p. 18

F.2 Activity-matched random-evidence prompts

• +0.099896 pooled NDCG@5 separates retrieved-evidence prompts from activity-matched random-neighbor prompts, and retrieved evidence performs better in every domain.The comparison keeps the text-prompt interface and averages four saved main rotations.Source: §F.2 · p. 19, Table 23 · p. 19

F.3 Matched-configuration sensitivity

• The matched augmented policy scores 0.544284 NDCG@5, compared with 0.544714 for B and 0.542833 for B+R.Matching removes the hyperparameter difference while allowing fitted functions to change with inputs; no new configuration search was performed.Source: §F.3 · p. 19, Table 24 · p. 19

G Response-format and feature-variation diagnostics

• Gemma’s direct multiple-maxima accuracy is 96%, but its explicit pairwise-tie accuracy is zero; Phi and Llama frequently respond with ties on non-ties.The checkpoints therefore show distinct answer tendencies.Source: G Response-format and… · p. 19, Table 25 · p. 20

G.1 Class-balanced accuracy and response frequencies

• Qwen3’s 95.0% first-alternative rate inflates its non-tie accuracy relative to balanced comparison performance, while Gemma has the largest balanced-accuracy point estimate despite zero tie recall.Pairwise balanced accuracy averages recall over left, right, and tie labels.Source: §G.1 · p. 20, Table 27 · p. 20

G.2 Within-checkpoint reading-feature variation

• Among 18 response-derived reading features, median checkpoint share is 9.98%; three exceed 50%, but within-checkpoint variation alone does not establish incremental information beyond V + S6.The two excluded covariates add no new per-request measurements given V and the fixed development lookup, though their encoding may affect a finite-capacity forecaster.Source: §G.2 · pp. 20–21

H Checkpoint and family heterogeneity

• The checkpoint-accuracy correlation is −0.942857, but shared users, jointly fitted selectors, and confounding of family with size and test accuracy limit interpretation to this model set.The authors do not assign it a causal or scaling interpretation.Source: H Checkpoint and family heterogeneity · p. 21, Table 30 · p. 22, Table 29 · p. 22

I Analysis coverage and provenance limits

• Table 31 distinguishes planned analyses from post hoc additions, and the analysis plan is included in the supplement.The table caption identifies its scope as coverage of planned and post hoc analyses.Source: I Analysis coverage and provenance… · p. 22, Table 31 · p. 22

J.1 Statistical replay and refit portability

• Saved-outcome statistical replay is exact, but macOS arm64 refitting is not bitwise portable.Linux records also show byte-identical evaluation and additional-candidate rows after feature construction and refitting; replay succeeds under Python 3.13.5 and NumPy 2.3.5.Source: §J.1 · p. 23, §J.2 · p. 23

J.2 Prompt requirements and computation

• The corrected plan defines adding reading features to the baseline selector as the primary comparison; prompt counts are feature requirements, not timed policy executions.The experiment scores the full prompt collection before policy selection.Source: §J.2 · p. 23, Table 32 · p. 23
• 79,089 forward passes, 476,085,792 padded input tokens, and about 50 hours are lower bounds from retained sessions, not complete run totals or per-policy latency.One partial-session record is missing; the largest allocated-memory peak is 20.6 GiB.Source: §J.2 · p. 23

J.3 Statistical artifact boundary

• Replay arrays reproduce feature-set comparisons and control/matched-fit metrics, while histories, user-specific prompts, source identifiers, and full model outputs are excluded.The source-date audit supports the Appendix A.3 restriction, while statistical replay checks array hashes and group accounting without repeating source joins.Source: §J.3 · p. 23