PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Detect Symbolic Failure: Machine Learning and the Limits of Black-Scholes

Juli Huang, Jake Cheng, Rupert Lu

arXiv:2609.27764v1cs.LGcs.CE

TL;DR

The paper asks whether machine learning can detect systematic deviations from Black-Scholes and compares learned embeddings, preserved domain structure, and neural-network validation. Tree-based methods outperform kernel dimensionality reduction, while neural-network and Black-Scholes deviation labels agree closely. The authors limit their claims: the findings do not establish exploitable mispricings or market inefficiency.

Abstract

from arXiv · show

We treat options pricing as a representation problem: can machine learning detect systematic deviations from Black-Scholes using 2.6M real option contracts? We compare three regimes: learned abstract embeddings (Kernel PCA), preserved domain structure (tree-based ensembles), and neural network validation. Tree-based methods outperform kernel dimensionality reduction by 21.5 percentage points (93.8% vs 72.3%), and domain-expert features (Greeks, moneyness) outperform engineered features. NN-based and BS-based deviation labels agree 99.9974% of the time, suggesting deviations reflect market structure rather than model artifact. We conclude that in domains with expert-designed symbolic features, preserving structure beats learning abstractions. We make no claim of exploitable mispricings.

Introduction

The paper asks what machine learning reveals when it departs from Black-Scholes’ symbolic structure, and compares representational regimes for detecting deviations. It reports a tree-based advantage over kernel dimensionality reduction, while limiting its claims about market implications.

Symbolic Abstractions and Their Limits

  • Black-Scholes persists as a shared symbolic language for quoting implied volatility, hedging with Greeks, and regulatory accounting despite known shortcomings.Its assumptions include constant volatility, frictionless trading, European-style exercise, and geometric Brownian motion.Source: Symbolic Abstractions and Their Limits · p. 1

The Representation Problem

  • The paper frames its question as what patterns emerge when machine learning on real contracts abandons Black-Scholes’ symbolic constraints.It treats options pricing as a representation problem spanning symbolic, learned, and conceptual dimensions.Source: The Representation Problem · pp. 1–2
  • 21.5 percentage points (93.8% vs 72.3%): tree-based ensembles outperform kernel dimensionality reduction when detecting deviations from Black-Scholes.The authors suggest that learned latent dimensions destroy domain-specific information encoded in structured option-pricing features.Source: The Representation Problem · p. 2
  • 99.9974% agreement between Black-Scholes- and neural-network-based deviation classifications suggests the deviations are not artifacts of the Black-Scholes framework alone.The authors say this does not prove market inefficiency or reveal tradeable alpha.Source: The Representation Problem · p. 2

Three Honest Caveats

  • The authors make no claim of exploitable mispricings or market inefficiency.They identify the 180-day validation window, exploratory ±10% threshold, and neural-network consistency check as limits on what the results establish.Source: Three Honest Caveats · p. 2

I Related Work

The related work connects symbolic models in finance, machine learning on structured data, and the limits of formal abstractions. It positions this paper’s neural network as a validator rather than a primary pricing model.

Machine Learning on Structured Data

  • The paper places its contribution at the intersection of symbolic finance models, machine learning on structured domains, and limits of formal abstractions.It asks whether learned embeddings outperform domain-designed features on tabular data, an open empirical question in the cited work.Source: I Related Work · p. 2
  • Greeks are precomputed summaries of domain knowledge, while tree-based and kernel methods represent distinct trade-offs for structured data.The related work notes that learned representations can excel on images and text yet underperform on expert-informed tabular data.Source: I Related Work · p. 2

Alternative Pricing Approaches

  • Unlike neural-network pricing approaches [7], this paper uses neural networks to validate whether another representation detects the same Black-Scholes deviations.It also compares tree-based models, support vector machines, and kernel methods, implemented with scikit-learn.Source: I Related Work · p. 2

Options Pricing Reality

  • The paper compares market prices with a simplified symbolic baseline rather than proposing to replace practitioners’ alternative pricing tools.Those tools include implied-volatility surfaces, American-exercise adjustments, and jump-diffusion models.Source: I Related Work · pp. 2–3

II Data and Methods

The study uses real option contracts across volatility regimes, with temporally ordered training and testing. Its volatility-smile figure illustrates how implied volatility varies with moneyness relative to Black-Scholes’ constant-volatility assumption.

II.1 Dataset: Option Prices Across Volatility Regimes

  • 2,632,013 real option contracts span 180 trading days, with earlier observations used for training and later observations for testing.The temporal split is intended to prevent look-ahead bias.Source: II.1 Dataset: Option Prices Across… · p. 3
  • Figure 1 depicts implied volatility varying with moneyness, contrary to Black-Scholes’ constant-volatility assumption.The paper identifies this systematic deviation as what its models learn to detect.Source: Figure 1 · p. 3
  • Only 2,044,241 contracts (77.7%) have valid Black-Scholes labels, and results are computed on that labeled subset.The remaining 587,775 contracts (22.3%) have missing labels.Source: II.1 Dataset: Option Prices Across… · p. 3

II.2 Features: Domain-Expert Design

The section compares domain-expert option features with engineered features and defines deviation labels relative to Black-Scholes prices. The ±10% labeling threshold is exploratory rather than economically justified.

II.2 Features: Domain-Expert Design

  • The base features include moneyness, time-to-maturity, implied volatility, and Greeks; 12 engineered features test whether automation can improve on domain expertise.The engineered set includes Greek interactions, moneyness powers, and volatility ratios.Source: II.2 Features: Domain-Expert Design · p. 3
  • 93.9% accuracy for the base 8-feature model exceeds the enhanced 12-feature model’s 93.8% accuracy.The authors interpret the result as suggesting domain experts already identified the essential signals.Source: II.2 Features: Domain-Expert Design · p. 4

II.3 Labels and Classification Task

  • Options are classified by comparing market prices with Black-Scholes prices, using the ±10% threshold to distinguish underpriced and overpriced contracts.The threshold is 2–10 times typical transaction costs, and alternative thresholds would produce different class distributions.Source: II.3 Labels and Classification Task · p. 4
  • The ±10% threshold is exploratory and not economically justified; it was chosen to create sufficient class separation for machine-learning classification.A more rigorous economic approach would use asset-specific, moneyness-adjusted thresholds calibrated to execution costs.Source: II.3 Labels and Classification Task · p. 4

II.4 Three Representational Regimes

Across three representational regimes, full-feature tree ensembles outperform Kernel PCA embeddings, while a neural-network check finds nearly identical deviation labels to Black-Scholes. The approaches test whether preserving financial feature structure or learning abstractions better detects pricing deviations.

II.4 Three Representational Regimes

  • 72.3% accuracy from the best kernel method (Sigmoid KPCA) was only 0.4 percentage points above linear KPCA’s 71.9%.The approach reduced the original features to five principal components and tested four kernels.Source: II.4 Three Representational Regimes · p. 4
  • 93.8% Gradient Boosting accuracy exceeds the best kernel method’s 72.3%, a 21.5-percentage-point gap favoring full-dimensional structure over learned embeddings.Random Forest and Logistic Regression reached 93.0% and 92.0% accuracy, respectively.Source: II.4 Three Representational Regimes · p. 4
  • 99.9974% of BS-based and neural-network-based deviation classifications agreed on 38,833 test contracts, with one disagreement.The neural network learned market prices from Greeks, moneyness, and time-to-expiry using 2.04M contracts.Source: II.4 Three Representational Regimes · p. 4

III.1

Gradient Boosting performs well across volatility regimes and assets, while the results favor preserving domain structure over dimensionality reduction. Domain-expert features dominate engineered additions, and deviation labels show substantial imbalance and strong agreement across pricing models.

Model Performance Across Assets

  • Gradient Boosting achieved high accuracy across three volatility regimes; SPY’s accuracy was 98.2% versus 82.7% for TSLA.The passage suggests SPY’s low volatility and high liquidity and TSLA’s noisier, less-liquid high-volatility setting may relate to the difference.Source: Model Performance Across Assets · p. 5, Table 1 · p. 5

What the Results Reveal

  • 93.8% accuracy from tree-based methods exceeded kernel PCA’s 72.3% by 21.5 percentage points, favoring preserved domain structure over learned abstractions.Figure 2 compares representational regimes; the section also reports 99.9974% agreement between BS- and NN-based classifications.Source: Figure 2 · p. 5, What the Results Reveal · p. 5
  • The enhanced 12-feature model reached 93.8% accuracy versus 93.9% for the base 8-feature model, a 0.1% difference.Figure 3 reports that Greeks, moneyness, and implied volatility dominate feature importance, while engineered interaction terms contribute negligibly.Source: What the Results Reveal · p. 6, Figure 3 · p. 6
  • Among 2,044,241 labeled contracts, 39.1% were underpriced, 34.8% overpriced, and 3.7% fairly priced relative to Black-Scholes under the ±10% rule.Recall was 98–99% on majority classes and 18% on the rare fairly priced class.Source: What the Results Reveal · p. 6
  • Across representational regimes, BS-based and NN-learned classifications agreed 99.9974% of the time on 38,833 test contracts; the deviations did not appear to be artifacts of the BS framework alone.Source: What the Results Reveal · p. 6

IV Robustness to Transaction Costs

Simulated trading signals retain Sharpe ratios of 1.48–2.00 after specified transaction costs, but the evaluation does not establish forward profitability. The temporal split tests generalization within one historical period, not across market regimes.

IV Robustness to Transaction Costs

  • Sharpe ratios of 1.48–2.00 persisted after accounting for 0.1% bid-ask and 0.5% slippage costs, but profitability claims require forward validation.The 180-day temporal split uses data from April 2025–January 2026, so both splits come from the same historical period.Source: IV Robustness to Transaction Costs · p. 6
  • The 180-day evaluation tests generalization within a regime, not across regimes, so it cannot establish persistence through market changes.The authors identify validation on future data spanning multiple regimes as necessary for assessing robustness.Source: IV Robustness to Transaction Costs · pp. 6–7

V.1 When Symbolic Features Beat Learned Embeddings

The paper argues that learned embeddings can discard financial meaning encoded in symbolic features, while tree-based methods maintain those distinctions. It limits its claims: detected Black-Scholes deviations are not evidence of exploitable mispricings, permanent patterns, or a useless pricing model.

V.1 When Symbolic Features Beat Learned Embeddings

  • Greeks encode directional exposure, convexity, time decay, and volatility sensitivity; moneyness and time-to-maturity also carry financial meaning.The paper describes these as semantic features rather than arbitrary measurements.Source: §V.1 · p. 7
  • Kernel PCA’s reduction from 20 features to five latent dimensions captures variance but not financial meaning, whereas tree methods maintain semantic distinctions.The authors argue that abstraction may discard information when experts have already encoded domain knowledge into symbolic features.Source: §V.1 · p. 7

VI Limitations: What We Cannot Claim

  • Detecting deviations from Black-Scholes does not establish profitable trading opportunities; transaction costs, inventory effects, liquidity demand, or hedging constraints could explain them.The neural-network consistency check likewise does not prove inefficiency or exploitable structure.Source: VI Limitations: What We Cannot Claim · pp. 7–8
  • The 180-day validation window is too brief for long-term robustness claims; extended forward testing across crises and regime changes would be required.The paper cites 5–10-year periods as the professional validation horizon.Source: VI Limitations: What We Cannot Claim · p. 7
  • The ±10% Black-Scholes deviation boundary was selected arbitrarily, and its economic relevance depends on varying transaction and financing costs.The authors say the true economic boundary depends on transaction costs, carrying costs, and financing rates.Source: VI Limitations: What We Cannot Claim · p. 7
  • The paper treats Black-Scholes as a shared symbolic language, not a claim of literal accuracy, and does not conclude that the model is useless.It reports that deviations can be learned and detected, rather than indicting the model.Source: VI Limitations: What We Cannot Claim · p. 8

VII Conclusion

The paper presents three findings: Black-Scholes deviations are learnable, preserving domain-expert structure outperforms abstract embeddings, and alternative representations validate the detected deviations.

VII Conclusion

  • 90.8% accuracy: machine learning detects patterns in 2.6M market deviations from Black-Scholes.The authors suggest these deviations are structured rather than random.Source: VII Conclusion · p. 8
  • 21.5 percentage points: tree-based methods outperform Kernel PCA by preserving the semantic meaning of financial features.The features include Greeks, moneyness, and time decay.Source: VII Conclusion · p. 8
  • 99.9974% agreement between Black-Scholes-based and neural-network-learned classifications suggests the detected deviations are consistent across frameworks.The authors propose this as a way to validate claims about symbolic failure.Source: VII Conclusion · p. 8

VIII Future Work

The authors outline future work on broader validation, applications beyond options pricing, interpretability, and testing whether structure preservation generalizes beyond finance.

VIII Future Work

  • Future validation would extend across longer time horizons and volatility regimes.Source: VIII Future Work · p. 8
  • The proposed applications beyond options pricing include currency markets and bond pricing.Source: VIII Future Work · p. 8
  • The authors propose using SHAP and LIME to extract human-readable decision rules from tree models.Source: VIII Future Work · p. 8
  • A formal investigation would test whether structure preservation generalizes beyond financial domains.Source: VIII Future Work · p. 8
Loading 2609.27764v1…