PaperlayerView exact revision on arXiv

Source-linked AI summary

FACTS-MAS: A Multi-Agent Architecture for Adaptive Forecasting from Heterogeneous Evidence

Madumita Karthikeyan, Umang Rajnikant Bid, Rong Pan

arXiv:2610.38225v1cs.AIstat.AP

TL;DR

Forecasting must combine heterogeneous evidence whose predictive value can vary across horizons and domains. FACTS-MAS tests independent channel forecasts and uses validation-based skill and calibration to combine them. It improves on naive persistence and TimesFM in the reported housing evaluation, but is not more accurate than factor-adjusted damped exponential smoothing.

  • Problem

    Forecasting models that pool heterogeneous inputs bind source encoding, prediction, and attribution together, while evidence can differ in representation and predictive value across horizons, regimes, and domains.

    Source: §1 · p. 1

  • Method

    FACTS-MAS assigns heterogeneous evidence to functional channels, produces independent numerical forecasts, and uses leakage-safe reference-relative skill and validation-selected constants to weight and combine them.

    Source: §1 · p. 2, §6 · p. 25

  • Results

    2.118% versus 4.194% MAPE at 4 weeks on Zillow’s smoothed inventory backtest; FACTS-MAS also outperformed TimesFM but was not more accurate than factor-adjusted damped exponential smoothing.

    Source: §6 · p. 26

  • Takeaways & Limitations

    The housing case study shows that FACTS-MAS can test, distinguish, calibrate, and adaptively weight mixed evidence channels.

    Source: §5 · p. 20

    The Event Agent relies on FEMA declarations, which cover disasters but not the full event ontology contemplated by the method.

    Source: §4.2.3 · p. 19

Abstract

from arXiv · show

Real-world forecasting often requires integrating heterogeneous evidence whose predictive value varies across horizons, operating regimes, and domains. We introduce FACTS-MAS, a domain-agnostic multi-agent forecasting framework that decomposes evidence into specialized channels for temporal dynamics, exogenous context, seasonality, structural information, and discrete events. Each agent generates an independently testable forecast through a common interface, while leakage-safe historical validation measures its predictive skill relative to a naive reference. These skill estimates are used to weight and combine agent forecasts, so that uninformative evidence is either removed or downweighted rather than forced to participate. A calibration procedure selects domain-specific constant means based on historical validation folds. We evaluate FACTS-MAS on weekly for-sale housing inventory from Zillow and weekly unemployment insurance claims from FRED, for 15 U.S. metropolitan areas at 4-, 8-, and 13-week forecasting horizons. Our method outperforms both a naive last-value forecast method and the TimesFM foundation model. However, a factor-adjusted exponential smoothing model performs as well as the fused forecast. The evidence therefore supports FACTS-MAS as a systematic protocol for testing, calibrating, and recomposing evidence channels before assembling them into the strongest single model for a specific application domain. Code, data, and results are available at https://github.com/Madumita24/FACTS-MAS.

1 Introduction

FACTS-MAS addresses the challenge of combining heterogeneous evidence whose predictive value varies by horizon, regime, and domain. It gives each evidence channel an independent forecast, then uses validation-based skill and calibration to determine how those forecasts contribute.

1 Introduction

  • Forecast evidence varies in representation and predictive value, so a signal can help at one horizon or in one regime yet add nothing elsewhere.Sources may include temporal dynamics, exogenous conditions, seasonal cycles, persistent entity attributes, and discrete events.Source: §1 · p. 1
  • On Zillow’s smoothed inventory series, backtest MAPE was 2.118%, 3.622%, and 5.731% at 4, 8, and 13 weeks, versus 4.194%, 7.225%, and 10.814% for last-value persistence.The method was more accurate than TimesFM in the backtest, but not more accurate than factor-adjusted damped exponential smoothing; it was statistically equivalent to that model at 8 weeks.Source: §1 · p. 2
  • FACTS-MAS assigns heterogeneous evidence to functional channels, each of which returns an independent numerical forecast through a common output contract.The five channels are temporal, exogenous, seasonal, structural, and event-based.Source: §1 · p. 2
  • Leakage-safe validation measures each agent’s improvement over a naive reference, aggregates skill scores, and converts them into fusion weights that may vary by horizon or regime when data support it.Regime-specific weights are used only with at least 13 distinct validation weeks; otherwise pooled weights apply.Source: §1 · p. 2
  • Domain-specific constants are selected on historical validation folds and may be set to zero when validation does not support a nonzero effect.The evaluation includes housing inventory across 15 U.S. metropolitan areas at 4-, 8-, and 13-week horizons, with a pre-registered holdout whose configuration and tests were fixed before its 29 weekly origins were accessed.Source: §1 · pp. 2–3

2 Related Work

Related work spans multi-agent forecasting, forecasting models, evidence combination, and interpretation of unstructured information. FACTS-MAS distinguishes itself by independently testing channel forecasts and using validation to weight them.

2.1 Multi-Agent and LLM-Assisted Forecasting

  • Specialized agents alone do not show that each contributes predictive information; FACTS-MAS requires independent numerical forecasts and measures their skill against a reference before assigning fusion weights.Weights may depend on horizon and operating regime when the validation sample supports that conditioning.Source: §2.1 · p. 3

2.2 From Statistical Models to Foundation Models

  • Forecasting has progressed from statistical and global neural models to transformer and pretrained foundation-model forecasters, while FACTS-MAS can use these models as channel implementations or baselines.The housing study uses Holt’s linear-trend exponential smoothing at 4 weeks, a factor-adjusted damped-trend model at 8 and 13 weeks, and TimesFM 2.5 as a baseline.Source: §2.2 · p. 4

2.3 Heterogeneous Evidence and Forecast Combination

  • Forecast combination ranges from equal and performance-based weights to regression stacking, but its central test is whether combining channels beats the strongest single model selectable on the same validation data.FACTS-MAS uses rolling reference-relative skill, median aggregation, and softmax weights; horizon or regime conditioning is treated as a validation choice because it uses fewer observations.Source: §2.3 · p. 4

2.4 Calibration and NEXUS

  • Plausible numerical assumptions can hurt accuracy, so FACTS-MAS selects constants on historical validation folds and permits zero when a nonzero effect is unsupported.These constants include event-severity maps, effect durations, lags, and thresholds.Source: §2.4 · pp. 4–5
  • Unlike NEXUS’s multi-agent language-model reasoning path, FACTS-MAS forecasts independently from specialized channels and re-estimates reference-relative skill weights as the training window expands.The housing comparisons include a chain-of-thought forecast using the NEXUS prompt, a time-series foundation model, and statistical references.Source: §2.4 · p. 5

3 FACTS-MAS: Domain-Agnostic Forecasting Architecture

FACTS-MAS separates domain-specific evidence and channel models from a reusable forecasting procedure. Independent channel forecasts are scored against a reference, weighted, and combined, while calibration selects constants offline.

3 FACTS-MAS: Domain-Agnostic Forecasting Architecture

  • FACTS-MAS separates a reusable forecasting procedure from domain-specific data and models, while combining independently evaluable channel forecasts adaptively.The procedure comprises evidence digestion, channel forecasting, and adaptive combination.Source: §3 · p. 5
  • For each domain, evidence is assigned to functional channels and each agent independently maps its available information to a forecast over the requested horizon.The channel forecasts are scored against a reference, and their skill scores set the weights used to combine them into the FACTS forecast.Source: §3 · p. 5
  • Calibration is a separate offline step that selects channel constants on historical validation folds before the test period begins.Calibration may set a constant to zero.Source: §3 · p. 5, Figure 1 · p. 6
  • Historical validation measures each channel’s skill relative to a naive reference, and that skill sets the fusion weights.The validation window ends at the training cutoff, preserving the stated information boundary.Source: Figure 1 · p. 6

3.1 Functional Evidence-Channel Decomposition

FACTS-MAS organizes evidence by functional role rather than by dataset or application. Its five candidate channels cover target dynamics, external variables, recurring patterns, persistent entity characteristics, and discrete or unstructured information.

3.1 Functional Evidence-Channel Decomposition

  • The temporal channel models endogenous target dynamics from the historical trajectory, including stability, trend, and autocorrelation.Source: §3.1 · p. 6
  • The exogenous channel represents external numerical variables that may influence future target changes, with variable identity and transmission lag depending on the domain.Source: §3.1 · p. 6
  • The seasonal channel represents recurring calendar, periodic, or cyclic behavior.Source: §3.1 · p. 6
  • The intrinsic or structural channel represents persistent entity characteristics that can distinguish entities with similar recent trajectories.Source: §3.1 · p. 6
  • The event channel represents discrete or unstructured information whose forecasting relevance may require semantic interpretation before structuring.Source: §3.1 · p. 6
  • These channels are functional abstractions rather than fixed feature sets, so their variables and event meanings can differ by application.Examples include monetary-policy indicators for housing and weather or demand drivers for energy.Source: §3.1 · pp. 6–7

3.2 Shared Agent Interface

A shared output contract makes agent forecasts comparable and interchangeable in the fusion layer. Leakage-safe historical validation then measures each agent’s skill against a persistence reference, using the median to limit the influence of extreme pointwise ratios.

3.2 Shared Agent Interface

  • The common contract requires agents to forecast the same entity, origin, and horizon, return exactly the requested number of values, and obey domain-valid constraints.For housing, forecasts must be non-negative.Source: §3.2 · p. 7
  • An agent can be replaced without changing the fusion layer if it meets the shared interface, whether it uses statistical, machine-learning, foundation-model, or LLM-assisted methods.Source: §3.2 · p. 7

3.3 Leakage-Safe Agent Evaluation

  • Agent quality is evaluated using historical forecasts constrained to information available at each forecast origin, including publication and availability delays.Source: §3.3 · p. 7
  • Skill is measured relative to a reference, with positive values indicating improvement, near-zero values a match, and negative values underperformance.An agent reproducing the persistence baseline receives skill near zero.Source: §3.3 · p. 7
  • The median aggregates pointwise skill because very small reference errors can make individual skill ratios extreme.The aggregation is over the leakage-safe validation window.Source: §3.3 · p. 7

3.4 Skill-Based Adaptive Fusion

FACTS-MAS converts validated agent skill into non-negative normalized weights and uses them to form a fused forecast. Weights are re-estimated across training folds, while softmax retains distinctions among agents even when their skill is negative.

3.4 Skill-Based Adaptive Fusion

  • Validated skill scores are converted into non-negative normalized agent weights with a temperature-controlled softmax.Source: §3.4 · p. 8
  • The housing configuration sets the softmax temperature to 0.05 by hand from validation-skill magnitudes rather than selecting it by search.A validation-selected temperature was tested but not adopted.Source: §3.4 · p. 8
  • Softmax preserves the ranking of agents, including those with negative relative skill, without assigning every underperforming channel the same minimum weight.This distinguishes agents that are slightly worse than the reference from those that are substantially worse.Source: §3.4 · p. 8
  • Weights are re-estimated as the training window expands; each backtest fold estimates them from a 104-week validation window ending at its training cutoff and holds them fixed during testing.Source: §3.4 · p. 8

3.5 Adaptive Context and Regime Conditioning

FACTS-MAS can condition channel weights on forecast horizon, operating regime, or both, with the regime defined in a domain-specific, leakage-safe way. It selects among eligible groupings using validation error and constrains finer adaptation when historical support is sparse.

3.5 Adaptive Context and Regime Conditioning

  • Weighting may be pooled, horizon-conditioned, regime-conditioned, or conditioned jointly on horizon and regime, with skill recomputed for the relevant grouping.Source: §3.5 · p. 8
  • A regime must represent a meaningful state for the target process and be computable at the forecast origin without future information.Source: §3.5 · p. 8
  • The grouping strategy is selected by historical validation error, but finer conditioning may be unreliable when regimes have few observations.In housing, regime-specific weights require at least 13 distinct validation weeks; otherwise the system falls back to pooled weights.Source: §3.5 · p. 8
  • An earlier implementation assigned equal channel weights when a regime lacked validation data, affecting 780 test cells and inflating the apparent benefit of regime conditioning.Source: §3.5 · p. 8

3.6 Calibration of Domain-Specific Parameters

FACTS-MAS calibrates domain-specific constants on historical validation data rather than relying on intuition alone. Calibration checks whether improvements are consistent across folds and can retain a zero quantitative effect when adjustments are unsupported.

3.6 Calibration of Domain-Specific Parameters

  • Domain-specific constants and mappings can introduce unmeasured forecasting error when assigned by intuition, so FACTS-MAS separates their calibration from held-out evaluation.Source: §3.6 · p. 9
  • Calibration does not promote a candidate solely for the lowest pooled error; it also checks consistency across folds and can select zero adjustment when that performs best consistently.Other channel outputs, such as classification, confidence, or attribution, can remain available when its direct numerical adjustment is removed.Source: §3.6 · p. 9
  • In housing, the zero event-impact scale was the only adopted calibration result and changed test accuracy by at most 0.002 percentage points.Constants were selected using validation windows from all six folds, so selection was not fully nested in time.Source: §3.6 · p. 9

3.7 Factor Attribution

FACTS-MAS attributes the forecasted change to independently forecasted agent channels using weights derived from historical predictive skill. These contributions summarize predictive usefulness under the validation procedure, not causal effects.

3.7 Factor Attribution

  • Each agent’s independent forecast and historically derived fusion weight expose its forecast-level predictive contribution.The contribution is measured for the forecasted change from the last observed value.Source: §3.7 · p. 9
  • An agent that merely repeats the last observed value receives zero contribution to the forecasted change.The change-based form avoids assigning it a share of the forecast level.Source: §3.7 · p. 9
  • The contribution summary indicates which evidence channels were historically most useful under the validation conditions governing the forecast.The displayed weights provide the basis for this interpretation.Source: §3.7 · p. 9
  • These contributions summarize relative predictive usefulness under the selected forecasting and validation procedure, not causal effects.Low weight may reflect redundancy, implementation limitations in extracting signal, or evidence with little predictive value.Source: §3.7 · p. 9

3.8 Domain Adaptation

FACTS-MAS keeps its forecasting procedure domain-agnostic while reconfiguring fitted components for each target domain. Adaptation maps and instantiates channels, calibrates assumptions, and re-estimates attribution from target-domain history.

3.8 Domain Adaptation

  • FACTS-MAS’s domain-agnostic claim applies to the forecasting procedure, not to every fitted component, which requires controlled reconfiguration.Table 1 distinguishes invariant from domain-specific components.Source: §3.8 · p. 9, Table 1 · p. 10
  • Adaptation maps available historical, exogenous, seasonal, structural, and event information to functional channels, then selects or fits a model for each channel.Each implementation preserves the shared output contract.Source: §3.8 · p. 10
  • Domain-specific lags, scales, thresholds, and other configurable quantities are calibrated on historical validation data without accessing the held-out evaluation period.Source: §3.8 · p. 10
  • Agent skill and fusion weights are re-estimated from the target domain’s own historical performance rather than transferred from the source domain.Different weight distributions are expected when evidence-channel importance differs across domains.Source: §3.8 · p. 10
  • Cross-domain transfer succeeds when the common decomposition, validation, calibration, and fusion procedure applies without redesigning the overall architecture.The system learns a domain-appropriate configuration from available evidence.Source: §3.8 · p. 10

4 Case Studies

The case studies apply FACTS-MAS to weekly U.S. housing inventory and unemployment claims, with a frozen configuration also evaluated on a later housing holdout. Energy demand is described only as a planned channel configuration, with no experiments reported.

4.1 Data Preparation and Agent Setups

  • Rolling-origin evaluation restricted fitting, fusion weights, and weighting-rule choice to pre-test information, with a 13-week embargo for weekly housing forecasts.Inputs were admitted only after their availability dates; the reserved period began January 1, 2026, and was excluded from the backtest. Validation selection also calibrated domain-specific constants, with unsupported adjustments eligible to be set to zero.Source: §4.1 · p. 11

4.2 U.S. Housing Inventory Forecasting

  • The Zillow housing case forecasts weekly inventory for 15 U.S. metropolitan areas at 4-, 8-, and 13-week horizons using five evidence channels.The tested channels cover temporal dynamics, national macroeconomic context, annual seasonality, metropolitan structure, and disaster declarations; the intrinsic and event channels calibrated to null quantitative effects and were retired.Source: §4.2 · p. 12, §4.2.1 · p. 13

4.2.1 Domain Instantiation

  • 2.890%, 4.774%, and 6.945% MAPE: the seasonal channel’s standalone error at 4, 8, and 13 weeks.The factor-adjusted damped temporal model had MAPE 1.974%, 3.608%, and 5.664%; the macro magnitude channel was no more accurate than naive, with MAPE 4.258%, 7.297%, and 10.919%.Source: §4.2.1 · pp. 13–14
  • Zero was selected for the intrinsic adjustment in all six validation folds, and the event impact scale was zero-preferred in all six folds.The event-scale sweep’s mean validation MAPE increased as the scale increased; this shows that the particular mapping from FEMA metadata to continuous forecast adjustments lacked demonstrated incremental value, not that real-world events are irrelevant to housing.Source: §4.2.1 · pp. 15–16

4.2.2 Forecasting Results

  • 49.5%, 49.9%, and 47.0% lower MAPE than naive on smoothed housing data; FACTS-MAS does not outperform FA-ETS-d overall.The relative MAPE reductions at 4, 8, and 13 weeks are 23.9%, 34.5%, and 37.1% on the unsmoothed series. FACTS-MAS is less accurate than FA-ETS-d at 4 weeks, equivalent at 8 weeks, and marginally less accurate at 13 weeks.Source: §4.2.2 · pp. 16–17, Table 5 · p. 17
  • 0.243, 1.239, and 2.539 percentage points: FACTS-MAS was more accurate than TimesFM at 4, 8, and 13 weeks.The differences were statistically significant at each horizon; FACTS-MAS also beat naive in every monetary-policy regime at 13 weeks, while FA-ETS-d was similarly accurate across regimes without using regime information.Source: Table 6 · p. 17, §4.2.2 · p. 17

4.2.3 Holdout Evaluation

  • 60%, 65%, and 64% lower MAPE than naive on the smoothed holdout series at 4, 8, and 13 weeks.All four preregistered hypotheses were supported; FACTS-MAS was equivalent to FA-ETS-d at 8 weeks and less accurate by 0.105 percentage points at 4 weeks. The 13-week holdout results are descriptive only, and with a two-week publication lag errors rose but conclusions did not change.Source: Table 7 · p. 18, §4.2.3 · p. 18, Table 8 · p. 19

4 Case Studies

  • Unemployment claims use temporal, exogenous, and seasonal channels; intrinsic and event channels are not implemented, and energy demand is only a planned channel map.The case-study protocol keeps the same information boundary, agent-level scoring, weight estimation, and held-out testing rules while allowing domain data and channel models to change.Source: §4 · p. 10, Table 9 · p. 19

4.2.3 Holdout Evaluation

  • The holdout was evaluated only after the configuration was frozen, with two development choices that used test-window evidence replaced or removed.County-coded hazards may be undercounted, and the structural proxy is limited by only 15 MSAs; nine Zillow observations were forward-filled, creating an artificial flat stretch. FEMA covers only disasters despite a four-category event ontology, and an inauguration declaration fit poorly, though confidence gating zeroed its quantitative effect.Source: §4.2.3 · p. 19

4.3 Unemployment Forecasting

  • The fused forecast beat naive at every horizon (p = 0.002, 0.0004, and 0.0004) and damped FA-ETS at 8 and 13 weeks, but undamped FA-ETS was slightly more accurate at every horizon.The seasonal channel received about three quarters of the weight at every horizon; including the pandemic window raised every method’s MAPE by 0.7 to 2.2 points. A clean holdout is unavailable because an earlier pilot scored claims data through August 2026.Source: Table 10 · p. 20, §4.3 · p. 20

4.4 Energy Forecasting

  • No energy experiments are reported; the paper provides a planned channel map and expects seasonal and exogenous channels to receive substantial weight when validation supports them.Completed energy runs are specified to report the target, frequency, regions, coverage, horizons, leakage controls, channel models, calibrated parameters, baselines, regime definitions, and learned channel weights.Source: §4.4 · p. 20

5 Discussion

FACTS-MAS treats channels as independently testable forecasting hypotheses, allowing validation to downweight weak evidence and calibration to return null effects. Housing results show gains over persistence and TimesFM, but no advantage over the strongest single model; cross-domain transfer preserves the procedure, not fitted weights.

5.1.1 Evidence Channels as Testable Hypotheses

  • Independent channel forecasts let validation distinguish useful evidence from weak or redundant channels without removing or retraining unrelated agents.In housing, temporal skill narrows with horizon unless seasonality is removed; seasonality helps at every horizon, while macro magnitude adds little robust value and intrinsic reversion calibrates to zero.Source: §5.1.1 · p. 21

5.1.2 Macro Channel: A Well-Earned Null

  • The tested macro specification provides little robust incremental value: mortgage rates are null across specifications, and federal funds does not survive robustness and multiple-testing checks.National macro values repeated across MSAs can pseudo-replicate one signal and overstate effective sample size.Source: §5.1.2 · p. 22

5.1.3 Event Channel: Semantic Evidence and a Null Scale

  • The event severity magnitude worsens validation MAPE monotonically, so calibration selects λ = 0 in all six folds and removes its numerical forecast adjustment.The Event Agent retains semantic classification and confidence; useful semantic information does not establish a validated numerical effect.Source: §5.1.3 · p. 22

5.1.4 From Raw Error to Softmax Weights

  • Relative skill avoids rewarding agents that merely imitate persistence, while median aggregation limits instability from near-zero reference errors and softmax preserves negative-skill rankings.This replaces raw-error weighting, which could give a null macro channel weight comparable to stronger agents.Source: §5.1.4 · p. 23

5.1.5 Adaptive Fusion and Its Limits

  • Regime-conditioned weighting shows no predictive information beyond horizon conditioning, and the fused forecast does not outperform the strongest single model.A 13-week apparent regime benefit was concentrated in one fold; regime weights are used only with at least 13 distinct validation weeks, otherwise pooled weights apply.Source: §5.1.5 · pp. 23–24

5.1.6 Predictive versus Causal Attribution

  • Fusion weights represent predictive attribution, not causal contribution; low weight may reflect weak signal, redundancy, implementation quality, rare-regime dependence, or limited validation history.Context-specific weighting reduces the observations behind each decision, and rare regimes often have the least support.Source: §5.1.6 · p. 24, §5.3 · p. 25

5.2 Why Full-Window Evaluation Matters

  • Evaluating all eligible forecast origins avoids the misleading performance picture produced by a single forecast date, including embargo-period scoring.The reported backtest uses 2,610 MSA–origin pairs per horizon, and the holdout uses 29 weekly origins per MSA.Source: §5.2 · p. 24

5.3 What the Housing Results Imply

  • FACTS-MAS improves on persistence at every evaluated horizon and is more accurate than TimesFM, but it is not more accurate than factor-adjusted damped exponential smoothing, matching it at 8 weeks.The fused forecast is slightly less accurate than that single model at 4 and 13 weeks; additional candidates did not change the comparison.Source: §5.1.5 · pp. 23–24, §5.3 · p. 24

6 Conclusion

FACTS-MAS decomposes heterogeneous evidence into independent forecasting channels and uses leakage-safe validation, calibration, and adaptive fusion to recompose those that prove useful. On weekly housing inventory, it beats persistence and TimesFM but does not outperform factor-adjusted damped exponential smoothing, with which it is equivalent at 8 weeks.

6 Conclusion

  • FACTS-MAS treats heterogeneous evidence as testable hypotheses, with each channel forecasting independently and leakage-safe reference-relative skill informing calibration and adaptive fusion.Calibration can return a null effect, and weights may depend on horizon and regime when the sample supports that conditioning.Source: §6 · p. 25
  • 2.118%, 3.622%, and 5.731% backtest MAPE at 4, 8, and 13 weeks on Zillow’s smoothed series, versus 4.194%, 7.225%, and 10.814% for last-value persistence.On the pre-registered holdout, MAPE is 1.243%, 1.929%, and 3.045%, versus 3.122%, 5.445%, and 8.438% for persistence.Source: §6 · p. 26
  • FACTS-MAS is more accurate than TimesFM but not factor-adjusted damped exponential smoothing, matching that model at 8 weeks.Gains are smaller but still significant on the unsmoothed series; holdout error is lower than a chain-of-thought language-model forecast, but not significantly so.Source: §6 · p. 26
  • The supported contribution is a protocol for testing, calibrating, and recomposing evidence channels, with new domain-specific evidence maps, models, and validation-based weights.The shared interface, leakage-safe scoring, and adaptive-fusion procedure remain the same across domains.Source: §6 · p. 26
Loading 2610.38225v1…