MemTrial: Learning When to Trust Memory in LLM Portfolio Agents

1 Introduction

• Agents write experiences after outcomes are known, then retrieve relevant memories to condition later decisions.Source: §1 · p. 1
• Existing agents increase an experience’s value when a decision using it goes well, or distill lessons from judged successes and failures.Source: §1 · p. 1
• MemTrial puts memory on trial rather than using its credits without a check.Source: §1 · p. 1

P Same date Shared move cancels Trust test Act

• Outcome credit tracks the shared market move: every tested experience-learning agent underperforms equal weighting on PortBench and InvestorBench.Source: P Same date Shared move cancels Trust… · p. 1
• On PortBench, retrieved experiences detectably changed utility in only 2–4 of 56 months, making single-date contribution measurements mostly noise.Source: P Same date Shared move cancels Trust… · p. 2
• Even when experiences changed decisions on about 85% of InvestorBench days, past contributions did not predict the following days.Source: P Same date Shared move cancels Trust… · p. 2
• MemTrial compares drafts with and without each experience on the same date, so their shared market outcome cancels in the difference.Source: P Same date Shared move cancels Trust… · p. 2
• It pools noisy contributions across dates and similar experiences, then acts on learned values only after they predict unseen dates; otherwise it stays anchored near 1/N.Source: Figure 1 · p. 1, P Same date Shared move cancels Trust… · p. 2
• The evaluation covers three real benchmarks and PlantedMem, a semi-synthetic setting with known experience quality; MemTrial is best when experiences matter and stays near 1/N when values fail.Source: P Same date Shared move cancels Trust… · p. 2

2 Related Work

• Financial LLM agents use trading outcomes to reinforce memory importance or reflect on past trades, but the market move dominates those outcomes.Source: §2 · p. 2
• Outside finance, memory methods rank, reuse, or distill experiences from episode outcomes, while retrieval-augmented generation selects by similarity alone.Source: §2 · p. 2
• Data valuation averages marginal contributions, while influence functions and logged-decision methods estimate effects; none measures what memory changes in LLM decisions and tests prediction on future dates.Source: §2 · p. 2
• MemTrial adapts safe deployment and portfolio shrinkage ideas by testing forward scores and anchoring action at a conservative reference until learned values are trusted.Source: §2 · pp. 2–3

3.1 Problem Formulation and Overview

• The agent retrieves four experiences similar to the current market state, and its goal is to learn their decision effects and when those values can be trusted.Source: §3.1 · p. 3

3.2 Counterfactual Attribution: What Does an Experience Change?

• A fractional factorial design uses eight drafts instead of all 16 subsets for four experiences, with each experience appearing in half of the drafts.Source: §3.2 · p. 3
• An experience’s contribution is its average utility difference across drafts with and without it; shared date effects cancel, while the half-fraction aliases it with a three-way interaction.Source: §3.2 · p. 3

3.3 Value Estimation: Pooling Noisy Evidence

• A hierarchical value model pools centered contributions across dates and similar experiences, shrinking sparse estimates toward values predicted from experience content.Source: §3.3 · pp. 3–4

3.4 Trust Test: Has the Learned Value Predicted the Future?

• Before each outcome, MemTrial predicts values; afterward, it scores their ranking of retrieved experiences against measured contributions using out-of-sample forward correlation.Source: §3.4 · p. 4
• MemTrial trusts values only with at least 10 forward scores and significant positive evidence; then it selects the highest-valued positive draft, breaking ties toward fewer experiences.Source: §3.4 · p. 4

3.5 Anchored Action: What to Do Until Memory Can Be Trusted

• Until values are trusted, it blends the conservative reference with the average draft, regularizing the reference weight toward a prior that weakens as evidence accumulates.Source: §3.5 · pp. 4–5

3.6 The Closed Loop and Its Cost

• The same eight drafts support attribution, trusted action, and the average draft; they run in parallel, at eight times the LLM cost of a single-draft agent.Source: Figure 2 · p. 4, §3.6 · p. 5

4.1 Experimental Setup

• The evaluation covers three real benchmarks and PlantedMem, with known experience quality; agents are compared using utility per decision period across three investor profiles.LLM agents use gpt-4.1-mini; MemTrial was frozen before InvestorBench decisions, making InvestorBench, ClassAlloc, and other-LLM runs held-out tests.Source: Table 2 · p. 5, §4.1 · p. 5

4.2 Main Results

• MemTrial is the best of 15 methods in PlantedMem and the best LLM-based method on PortBench and InvestorBench; it stays within 2.2% of 1/N on the latter benchmarks.Across five settings, its utility is 21.2% higher than that of the best experience-learning agent.Source: §4.2 · p. 5, Table 3 · p. 6
• MemTrial tracks 1/N through rising and falling markets and has the best Sharpe ratio and drawdown among LLM-based methods on every benchmark, with highest returns on PortBench and InvestorBench.Source: §4.2 · p. 6

4.3 Ablation Study

• On real markets, replacing the anchored blend with draft averaging costs 16–41% and removing the trust test costs 4–12%; with signal, acting on learned values helps.When experiences are retrieved too rarely, per-experience estimates and an in-sample F-test do not reliably identify informative content.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Robustness to the LLM

• On InvestorBench, MemTrial beats every LLM-based baseline in all eleven configurations, by 4–33% over the best experience-learning agent.On ClassAlloc, it stays within 0.021 pp per month of 1/N across all eight LLMs, but beats the best experience-learning agent with four and trails it with four.Source: §4.4 · p. 7

4.5 Analysis

• At β=0.25, pooled MemTrial values rank experience quality at 0.81 correlation; with informative content, pooling ranks experiences at 0.66 versus 0.55 for raw contributions.When experiences carry no signal, the trust test passes on only 2–3% of decisions; as they matter more, it passes on up to 85% and gains reach 0.30 pp per month over the reference.Source: §4.5 · pp. 7–8
• On ClassAlloc, the memory-free LLM beats 1/N by 0.08 pp per month before its knowledge cutoff and trails by as much afterward; MemTrial gives up most pre-cutoff gains but loses less than other LLM-based methods afterward.Five of six LLMs with published cutoffs show the same before-versus-after pattern.Source: §4.5 · p. 8
• Looser trust-test levels can let chance forward scores pass on up to 17% of PortBench-Full decisions and cost up to 4% utility, while enabling earlier action in informative PlantedMem.A decision costs about USD 0.05 on PortBench; at the same LLM spend, Draft averaging does clearly worse.Source: §4.5 · p. 8

• When experience quality was known, MemTrial achieved the highest utility among 15 methods; on real markets, where learned values did not carry over to unseen dates, it stayed close to 1/N.Source: §5 · p. 8
• MemTrial measures each experience’s changes through same-date counterfactual drafts, pools those measurements across dates and similar experiences, and trusts values only after they predict unseen dates.Until then, it blends toward a conservative reference such as 1/N.Source: §5 · p. 8
• On real benchmarks, the learned values rarely passed the trust test, and even when they did on ClassAlloc, acting on them did not beat the anchor.The paper says gains from memory on real markets remain to be shown; the anchor itself protects baselines similarly.Source: §5 · p. 8
• LLMs can recall market outcomes from before their knowledge cutoffs, creating look-ahead bias that can flatter LLM-based methods on earlier periods.PlantedMem and post-cutoff months are free of this bias; on ClassAlloc before the cutoff, the anchored action also gave up most of the drafts’ advantage.Source: §5 · p. 8

Appendices

• The appendices cover notation, derivations, additional experiments, a case study, algorithms, prompts, and limitations.Source: Appendices · p. 10

A.1 Memory and experience in LLM agents

• Prior LLM memory systems and FinCon use episode success or realized profit and loss to decide what to keep or reuse, without testing whether memory-use rules predict future gains.Examples include MemGPT [40], MemoryBank [61], Voyager [49], and FinCon [55].Source: §A.1 · p. 10

A.2 Valuation and causal effects

• MemTrial estimates experience effects through a designed experiment on each date, holding the market fixed within comparisons, unlike observational treatment-effect methods or online user experiments.Its contribution is a factorial main effect that equals the Banzhaf value when the aliased three-way interaction is negligible [3, 6, 50].Source: §A.2 · p. 10

A.3 Portfolio choice under estimation error

• None of the optimized portfolios evaluated by DeMiguel et al. consistently beats 1/N out of sample, motivating shrinkage and combinations with a conservative portfolio.Source: §A.3 · p. 10
• MemTrial’s anchored action combines shrinkage toward a conservative portfolio with an online blend learned using a regularizer.Source: §A.3 · p. 10

A.4 Forecast evaluation and safe deployment

• MemTrial’s trust test applies sequential out-of-sample evaluation as a one-sided test of positive correlations between experience values and measured contributions.The trust check serves a role like safe policy improvement’s confidence check, while the anchored action serves as its baseline policy.Source: §A.4 · p. 10

C.1 The half-fraction design

• The half-fraction contrast estimates an experience’s main effect, equal to its Banzhaf value over all subsets, while aliasing that effect with a three-way interaction.The shared date-level component cancels, and the four experience contrasts are orthogonal, so two-way interactions do not affect another experience’s contribution.Source: §C.1 · pp. 10–11
• 13–17% extra contribution-noise variance arises from the aliased three-way interaction, while the half-fraction estimate correlates 0.76–0.86 with the full-factorial value.Source: §C.1 · p. 11

C.2 The value model

• Experience-specific values are shrunk toward the centered average experience when measured on few dates, and a new experience’s value is predicted from its content alone.Source: §C.2 · p. 11

C.3 The trust test

• The trust rule is applied on each date once at least 10 forward scores are available, and its false-pass rates were 2.3% and 3.0% in the two no-signal PlantedMem regimes.The scores can be dependent when experiences recur, so the stated 0.05 level is for one test rather than the rate of passing over a run.Source: §C.3 · p. 11

C.4 The anchored action

• The anchored blend begins at its prior weight and, as observations accrue, approaches the blend with the highest mean past utility.Its regularization strength is scaled by estimated utility noise and sample size.Source: §C.4 · p. 11

D More Experiments

• “Improv.” denotes MemTrial’s utility gain over the best experience-learning agent in each setting, expressed as a percentage of that agent’s utility.Source: D More Experiments · p. 11

D.1 Dataset statistics

• PortBench’s test period is only 20 months, and none of its Table 3 differences is significant across test dates.The additional InvestorBench and ClassAlloc runs each involve thousands of executions and reported per-run costs.Source: §D.1 · pp. 11–12

D.2 Baselines

• The baselines include outcome-credit memory learners, uplift credit, counterfactual selection, draft averaging, and Hedge, with methods compared on shared logged drafts where specified.Counterfactual selection uses the two experiences with the highest mean past Banzhaf contribution but omits value estimation, a trust test, and an anchor.Source: §D.2 · p. 12

D.3 Implementation details

• Experiments use gpt-4.1-mini drafts on PortBench and InvestorBench, while ClassAlloc covers five asset classes and monthly decisions scored over 20 sessions.PlantedMem uses 132 monthly dates, with useful, neutral, and misleading experiences in known proportions; experience usefulness depends on the direction it moves the draft.Source: §D.3 · pp. 12–13
• PortBench and InvestorBench retrieve four experiences by embedding similarity; mandate projections constrain risky and defensive shares, and trading costs are 15 bp of portfolio turnover.InvestorBench measures trades from drifted holdings, while PortBench and PlantedMem start each decision from the equal-weight book.Source: §D.3 · p. 13
• The implementation learns from outcomes only after they are realized, and uses the first eight principal components of retrieval embeddings as experience-content features.In PlantedMem, the first content coordinate carries experience quality plus noise; the other coordinates are noise.Source: §D.3 · p. 14
• MemTrial uses eight parallel drafts per decision, costing about USD 0.05 on PortBench and USD 0.005 on InvestorBench; value fitting and trust scoring take milliseconds per decision.The reported real-benchmark seeds vary LLM samples, not market histories; PlantedMem instead uses 100 newly generated test worlds.Source: §D.3 · p. 14

D.4 Why do LLM agents trail 1/𝑁?

• 0.33–0.42 pp per month: on PortBench-Full, memory-free, top-4 retrieval, and eight-draft-average portfolios trail 1/N, with gaps from gross return, trading costs, and risk penalty.LLM drafts move an L1 distance of 1.0–1.2 from the equal-weight book each month, versus 0.17 for projected 1/N; on InvestorBench they pay 1.0–2.5 bp per day in trading costs, versus 0.1 bp for 1/N.Source: §D.4 · p. 14

D.5 Ablation details

• Without the trust test, MemTrial acts on learned values on 25% of PortBench and 34% of InvestorBench test decisions, losing 0.08 pp per month on PortBench-Full, 0.24 pp on PortBench-Raw, and 0.49 bp per day on InvestorBench.Source: §D.5 · pp. 15–16
• An in-sample F-test loses 0.05 pp in informative PlantedMem, while omitting content-aware value estimation loses 0.05 pp there; content-aware learning alone costs 0.07 pp on PortBench-Raw.With 48 experiences in informative PlantedMem, the F-test passes on 15% of decisions versus 45% for the forward trust test.Source: §D.5 · p. 16
• Replacing the anchored action with draft averaging costs 0.36 pp on PortBench-Full, 0.82 pp on PortBench-Raw, 1.70 bp per day on InvestorBench, and 0.03 pp in core PlantedMem.When ClassAlloc drafts beat 1/N, draft averaging gains 0.035 pp per month, whereas always keeping the reference loses 0.008 pp.Source: §D.5 · p. 16

D.6 Wealth over time, market phases and survival

• November 2024 reverses PortBench-Full’s mid-2024 lead: the equal-weighted cryptocurrency class gained 129%, while 1/N gained 15% and agent drafts held 0–7%.Without that month, top-4 retrieval and Hedge beat 1/N and MemTrial on PortBench-Full; dropping any other single month leaves MemTrial the best LLM-based method on both PortBench variants.Source: §D.6 · p. 17
• After gpt-4.1-mini’s knowledge cutoff, every LLM-based method trails 1/N on ClassAlloc, with MemTrial trailing least for every investor.Before the cutoff, the memory-free LLM and most memory-based methods end above 1/N for balanced and aggressive investors.Source: §D.6 · p. 17
• MemTrial’s terminal wealth is highest among LLM-based methods in 38 of 39 PortBench and InvestorBench settings and within 1.4% of 1/N; its Sharpe ratio and maximum drawdown are closest to 1/N.Across falling phases on InvestorBench and ClassAlloc, it loses 0.06% and 0.9%, respectively, against 0.38–0.65% and 1.5–3.5% for the memory-free LLM and experience-learning agents.Source: §D.6 · p. 17

D.7 Hyperparameter analysis

• 2%: MemTrial stays within this distance of 1/N in every setting, while other agents fall as much as 21% behind.The comparison is shown for wealth relative to 1/N.Source: Figure D.1 · p. 18
• At α=0.1 and 0.2, chance scores pass the PortBench trust test on 5% and 12% of decisions; looser tests trust values earlier in PlantedMem and help its informative regime.At α=0.01 versus 0.2, the trust rate is 28% versus 41%, and informative-regime utility is 0.84% versus 0.88%.Source: §D.7 · pp. 18–19
• In PlantedMem, core utility rises from 0.812% per month with k=2 to 0.868 with k=4, 0.896 with k=5, and 0.909 with k=6, with diminishing returns per added draft.At equal draft cost, a half fraction measures one more experience and beats or matches the compared full-factorial designs.Source: §D.7 · p. 19, Table D.10 · p. 21

D.8 Strength of the anchor

• With α0=0.9, MemTrial gives up at most 2.2% where 1/N is strong while retaining the ability to follow better drafts; a higher prior would adapt more slowly when the reference is weak.With α0≥0.9, the tested λ0 settings stay within 0.06 pp per month and 0.06 bp per day of the default.Source: §D.8 · p. 20, §D.7 · p. 20

D.9 Number of retrieved experiences by regime

• When experiences carry no signal, every tested design stays within 0.01 pp of the reference, so larger k adds no risk there; gains come from regimes where experiences matter.Source: §D.9 · p. 21

D.10 PlantedMem by regime

• Trust tracks transfer to unseen dates, not whether experiences change a given day's decision: it reaches 85% at β=0.5, while waiting for forward evidence costs 0.01–0.03 pp versus counterfactual selection when signal matters.When experiences carry no signal, MemTrial stays at the reference while acting without the test falls below it.Source: §D.10 · p. 22

D.11 ClassAlloc before and after the knowledge cutoff

• After the gpt-4.1-mini cutoff, all LLM-based methods trail 1/N on ClassAlloc, with MemTrial least behind at 0.746% versus 0.787%; over all 71 months, no memory method improves on the memory-free draft.The pre- versus post-cutoff difference is not significant (p=0.36, Welch t-test).Source: §D.11 · p. 22

D.12 Other LLMs and temperatures

• In every InvestorBench configuration, the best LLM-based baseline earns 8.97–9.71 bp per day, versus 10.45 for 1/N and 9.84–10.54 for MemTrial.Source: §D.12 · p. 22

D.13 ClassAlloc with other LLMs

• Across eight LLMs, the memory-free draft beats 1/N with four, while MemTrial stays within 0.021 pp per month of 1/N with every LLM.With gpt-5-mini and Qwen3-235B, anchoring gives up most of the memory-free draft's advantage.Source: §D.13 · p. 23

D.14 Look-ahead checks

• Before their published cutoffs, every tested LLM recalls ClassAlloc outcomes (absolute recall 0.10–0.63; relative recall 0.06–0.50), and hardly recalls them afterward.The probe asks LLMs to recall asset-class returns from a date alone.Source: §D.14 · p. 23, Table D.22 · p. 25
• 0.13 and 0.14 pp per month: hiding dates leaves the memory-free draft's pre-cutoff advantage over 1/N intact with gpt-4.1-mini and gpt-5-mini, respectively.The corresponding post-cutoff deficits are 0.10 and 0.07 pp per month.Source: §D.14 · p. 24, Table D.23 · p. 26
• The pre-cutoff advantage may reflect recognizing periods from returns or allocations suited to 2020–2024; it applies to LLM-based methods alike, while PlantedMem is free of it.Source: §D.14 · pp. 24–25

D.15 PortBench after the knowledge cutoff

• Across eleven post-cutoff months, MemTrial is the best LLM-based method with both agents and stays 0.14 and 0.01 pp per month below 1/N; other LLM-based methods trail by 0.36–1.57 pp.Price-file gaps removed six of the twelve newly added months from the shared comparison.Source: §D.15 · p. 26
• In the six new PortBench-Full months, the memory-free draft beats 1/N by 0.80 pp per month and MemTrial gains 0.06 pp; with six months, no difference involving MemTrial is significant.On PortBench-Raw, every LLM-based method is within 0.17 pp of 1/N.Source: §D.15 · p. 27

E Case Study: Two Decisions in the 2022 Bear Market

• In seed 1, acting on learned values cost 1.66 pp versus 1/N in April, while anchoring in September gave up 2.02 pp to the selected draft.The April trust test passed; the September test narrowly failed.Source: Table E.1 · p. 28, E Case Study: Two Decisions in the… · p. 29
• Same-date comparisons cancel outcomes shared by all drafts: despite every draft losing between 6.1% and 3.8%, lesson contributions separated by tenths of a pp.L18/04 helped on both dates (+0.12 and +0.28 pp), while L17/05 hurt on both (−0.42 and −0.21 pp); shrinkage left all learned values below 0.02 pp in absolute value.Source: E Case Study: Two Decisions in the… · pp. 28–29

F Algorithms

• MemTrial drafts eight experience combinations, estimates experience values from past contributions, tests forward predictive scores, then deploys a selected draft or an anchored blend.The algorithm also updates contributions and forward scores after the outcome; Algorithms F.2 and F.3 expand the learning steps.Source: E Case Study: Two Decisions in the… · p. 29, F Algorithms · p. 29

E Case Study: Two Decisions in the 2022 Bear Market

• Across 93 trust-passing ClassAlloc decisions, trusted drafts earned 0.05 pp less (s.e. 0.07) than anchored blends; the never-trust variant was 0.016 pp/month better.Over all 71 months, acting on learned values did not pay on ClassAlloc.Source: E Case Study: Two Decisions in the… · p. 29

G LLM Prompts

• Across the three real benchmarks, LLM-based methods use the same prompts and differ in the memory block; logged drafts also let methods differ by draft selection or combination.On InvestorBench and ClassAlloc, each decision is one chat completion; on PortBench, a draft runs the benchmark's three-stage agent.Source: G LLM Prompts · p. 29

G.3 PortBench

• The frozen historical records summarize earlier allocations and realized returns, but their feedback scope explicitly disclaims causal superiority.Each record covers an earlier decision and includes execution measures, market features, and the realized 20-day net return.Source: §G.3 · pp. 31–32
• The three-stage prompt turns market views into buy, sell, or hold signals, then sparse weights that reduce sell-signal exposure and unnecessary turnover.The example describes a high-volatility regime and bearish views on QQQ, EEM, EFA, and FXI, prompting sell signals for those assets.Source: §G.3 · p. 32
• In the example, the agent assigns equal scores to 93 assets without sell signals and none to the four assets with sell signals.The environment converts the scores into target weights and projects them to the investor’s mandate.Source: §G.3 · p. 32

H Limitations •

• Real-market gains from memory remain unshown: MemTrial passed its trust test on fewer than 1% of PortBench and InvestorBench test decisions and 15% on ClassAlloc, where trusted drafts did not beat the anchored blend.On PortBench and InvestorBench, the same anchor lets baselines perform as well; the authors frame the result as avoiding losses from untested memory, not improving real decisions.Source: H Limitations • · p. 32
• gpt-4.1-mini may have seen outcomes from the InvestorBench test period and most PortBench and ClassAlloc test months during pretraining, though paired comparisons remain fair.The authors note that this may still affect how much any LLM agent gains from memory; PlantedMem is free of this exposure.Source: H Limitations • · p. 33
• The trust test’s false-pass rate has no formal bound because it is applied to dependent scores of a selected learner; the measured rate is 2–3% of decisions without signal.The authors suggest a sequential test with time-uniform error control on block or permutation scores as a possible bound.Source: H Limitations • · p. 33
• MemTrial requires eight drafts per decision, increasing LLM cost even though parallel execution keeps latency from growing.Adaptive designs that spend more drafts only when contributions are uncertain could reduce cost.Source: H Limitations • · p. 33
• The benchmark protocols constrain interpretation: PlantedMem uses a simple tilt model, while PortBench resets monthly decisions to equal weight and fixes its experience pool.InvestorBench adapts a single-stock task to four stocks and cash; ClassAlloc uses equal-weighted, daily-rebalanced class indices and omits cryptocurrency.Source: H Limitations • · p. 33