PaperlayerView exact revision on arXiv

Source-linked AI summary

NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees

Weiwei Ye, Hangchen Liu, Renhe Jiang

arXiv:2609.28587v1stat.MLcs.AIcs.LG

TL;DR

Jev-like structured-choice interfaces do not directly provide numerical values at requested precision. NUMERICJEV recursively refines a range through contextual multiway choices without parameter updates, and exceeds direct choice on the reported arithmetic tolerance metric.

  • Problem

    Jev-like models expose structured choices, but these outputs do not directly provide arbitrary numerical values at a requested precision.

    Source: §1 · p. 1

  • Method

    NUMERICJEV repeatedly selects among subintervals in the original question's context and returns the final cell's lower endpoint without training a numerical head.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    83.40% of outputs fall within 5% relative error, compared with 80.47% for direct choice across 256 arithmetic expressions and two option orders.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The tested decoder exposes finite numerical resolution and decision traces, with smaller range-normalized errors than direct selection in the tested arithmetic setting.

    Source: §4 · p. 5, §7 · p. 7

    The controlled evaluation covers one hosted model version, simple generated arithmetic, and a small historical-recall example; known bounds and resolutions are supplied.

    Source: §6 · p. 7

Abstract

from arXiv · show

Large language models can interpret natural lan- guage, yet robust decisions remain challenging. Jev-like models expose structured choices, but these interfaces do not directly provide numeri- cal values at a requested precision. We propose NUMERICJEV, a training-free numerical decod- ing algorithm that enables numerical output from any LLM with a Jev-like structured-choice in- terface. Surprisingly, on our arithmetic bench- mark, it outperforms direct selection from a can- didate list containing the correct answer by 2.93 percentage points (Figure 1). Our motivation comes from the observation that numerical range selection is itself a decision problem that Jev- like LLMs can address. NUMERICJEV recur- sively refines a range through a multiway deci- sion tree while retaining the original question in context, without parameter updates or hidden- state access. On a 100-value grid, a ten-way tree requires only two decision rounds. Range- normalized MAE is 1.84% versus 5.18% for di- rect choice. A separate three-date historical- index study yields 4.58% mean relative recall er- ror and 0% readout error when the value is sup- plied. Code is available at https://github. com/Bring-AI/jev-numeric.

1. Introduction

Jev-like choice interfaces do not directly yield arbitrary numerical values at a requested precision. NUMERICJEV turns numerical range selection into successive contextual choices, producing numerical readouts without training a new head.

1. Introduction

  • Jev-like interfaces expose choices and probabilities, but categorical outputs and rubric scores do not directly supply arbitrary numerical values at requested precision.Prior numerical readouts use trained regression heads or mappings from numeral embeddings; the paper identifies generalization and loss of linguistic context as concerns.Source: §1 · p. 1
  • NUMERICJEV recursively refines a numerical range through multiway choices in the original question's context, without parameter updates or hidden-state access.The decoder handles interval arithmetic and returns the final cell's lower endpoint.Source: §1 · pp. 1–2
  • 83.40% of outputs fall within 5% relative error, versus 80.47% for direct choice; range-normalized MAE is 1.84% versus 5.18% for direct choice.On the 100-value grid, ten-way decoding takes two decision rounds.Source: Figure 1 · p. 2, §1 · p. 2
  • A separate three-date historical-index study reports 4.58% mean relative recall error and 0% supplied-value readout error.These results are tied to that study's protocol.Source: §1 · p. 2

2. Related Work

Prior work connects classification and language-model representations to numerical prediction, while NUMERICJEV differs in relying on a fixed structured-choice interface rather than trained numerical mappings.

2. Related Work

  • Discretization and classification have supported numerical prediction, but NUMERICJEV uses a fixed hosted choice interface with inference-time alternatives and no trained numerical head.The paper describes its analysis as addressing the outer decoder, not providing a learning-theoretic guarantee for the underlying model.Source: §2 · p. 2
  • Numerical language-model work includes digit serialization and predictive distributions; prior results motivate evaluating numerical distance and distinguishing point decisions from distributions.The decoder instead uses successive structured choices, not free-form numerical completion.Source: §2 · p. 2
  • Representation probes require internal representations and training examples, while choice behavior can depend on option identity and order; the paper controls ordering without identifying the cause of Jev's bias.Its focus is numerical error across depth and representation.Source: §2 · pp. 2–3

3. Numerical Decoding through Structured Decisions

NUMERICJEV uses a caller-specified finite grid and repeatedly asks a structured-choice model to select among intervals. Its depth and readout guarantees depend on the selected path, so finer resolution alone does not ensure practical accuracy.

3.1. Interface and finite numerical grid

  • The interface takes an instruction and labeled criteria and returns a selected label and probabilities; those probabilities are not assumed to represent calibrated numerical uncertainty.The caller supplies the numerical range and stopping resolution.Source: §3.1 · p. 3

3.2. Balanced multiway refinement

  • At each step, the decoder partitions the current index interval into nearly balanced subintervals, asks the model to choose one, and returns the selected cell's lower endpoint.It stops at one grid cell and supports grid sizes that are not powers of the branching factor.Source: §3.2 · p. 3, §3.3 · p. 3

3.3. Resolution, depth, and error propagation

  • With N grid cells and branching K, the tree needs at most ceil(log_K N) choices; the error is below ϵ only when every selected interval contains the target.Early divergence can dominate later precision, and greedy refinement can increase lower-endpoint error.Source: §3.3 · p. 3

3.4. Alternative numerical representations

  • Direct choice, index-digit decoding, and index-bit decoding pose different questions; mathematical equivalence does not guarantee identical model behavior.A separate threshold-query interface constructs a projected CDF, but that projection cannot establish calibration; a recorded tree example follows four ten-way choices to [11, 11.01).Source: §3.4 · pp. 3–4, Figure 2 · p. 4

4. Experimental Design

The evaluation fixes the model, prompts, cases, and job order while comparing numerical decoders across a controlled arithmetic grid and separate exploratory tasks. It defines complementary error metrics and uncertainty estimates, with scope limits for zero-valued outcomes and historical recall.

4. Experimental Design

  • The fixed affine-grid study evaluates 64 arithmetic expressions across four transformations, excluding zero-valued results because relative error is undefined for them.The exclusion limits applicability to tasks with zero-valued outcomes.Source: §4 · p. 4
  • Each expression is tested with and without its numerical result supplied, in ascending and reversed option order, across 1,024 condition/order jobs.Cases, prompts, model version, and job order were fixed before execution; this controlled extension is not a blind model-development benchmark or evidence of pretraining OOD.Source: §4 · p. 4
  • The controlled design compares direct choice, interval refinement with K = 2, 4, or 10, decimal index digits, and parallel binary index bits.The stated decision depths are seven, four, and two rounds for the three interval trees; those are dependency depths, not latency measurements.Source: §4 · pp. 4–5
  • MAPE is the primary endpoint, while NMAE complements it by measuring error relative to the specified range and remaining invariant to the four affine transforms for a fixed grid-index error.Near-zero targets can dominate MAPE, and neither metric is an exact-match count.Source: §4 · p. 5
  • Confidence intervals use 10,000 percentile-bootstrap replicates over 64 base families, keeping transformations, orders, and paired input conditions together.The intervals describe this expression generator and model version, not a population of all numerical tasks.Source: §4 · p. 5
  • Archived exploratory evaluations include 12 hand-picked arithmetic questions and a three-date S&P 500 closing-level recall study, with supplied-reference controls.The historical questions test recall rather than forecasting or demonstrated training-data membership.Source: §4 · p. 5

5. Results

Results depend on the error metric, encoding, and evaluation protocol: interval trees improve range-normalized error, but MAPE and domain-level comparisons do not show a uniformly preferred decoder. Supplied-value and historical results are protocol-specific, and the proposed distribution construction remains unvalidated.

5.1. Interval refinement and numerical distance

  • NMAE was 1.76%, 1.60%, and 1.84% for K = 2, 4, and 10 versus 5.18% for direct selection, while MAPE intervals did not establish a population advantage.The paired NMAE-difference intervals were below zero for all three trees; the paired MAPE-difference intervals included zero.Source: §5.1 · p. 5, Table 1 · p. 6

5.2. Supplying the value does not make every encoding equivalent

  • With values supplied, MAPE was 0% for K = 2, 1.27% for K = 4, 2.93% for K = 10, and 3.79% for direct selection; index-digit and bit MAPE remained 59.82% and 77.90%.The arithmetic condition combines computation and readout, so its improvement with the answer supplied cannot be attributed exclusively to either source.Source: §5.2 · p. 5

5.3. Continuous-head transfer and a tolerance-based overview

  • On the full suite, APE ≤5% reached 83.40% for ten-way refinement versus 80.47% for direct choice; continuous-head transfer systems scored 2.73%–6.25%.With the answer supplied, direct choice reached 97.27% and ten-way refinement 99.80%, leaving a 16.41-point gap to the supplied-value control. The continuous-head comparison is between complete systems with different backbones and training.Source: §5.3 · pp. 5–6

5.4. Scale, sign, and option ordering

  • Domain results reverse rankings: direct selection has 0% MAPE on the unshifted integer domain, but ten-way refinement has 30.96% versus direct choice’s 0.52% on the large-scale domain.The affine transformations also change the written expressions, so this is a controlled prompt-and-value transformation, not an isolated intervention on internal numerical representation; reversing insertion order is not a test of all permutations.Source: §5.4 · p. 6, Table 2 · p. 6

5.5. Exploratory arithmetic and historical recall

  • The 12-question exploratory suite favored K = 4 (2.42% MAPE versus direct selection’s 3.37%), while three-date recall had 4.58% mean error and a changed-prompt K = 2 follow-up had 31.17% MAPE.The follow-up also had 0% supplied-value MAPE, but it changed both prompt and branching factor and was not a controlled branching-factor ablation.Source: §5.5 · pp. 6–7

5.6. Numerical distributions need a separate validation

  • In the archived threshold experiment, 25 of 48 query groups had a decrease in their proposed CDF; isotonic projection imposes shape but does not validate calibration or recover a unique ground-truth distribution.Because the arithmetic references are deterministic, they do not supply a conditional outcome distribution for a forecasting study.Source: §5.6 · p. 7

6. Discussion and Limitations

NUMERICJEV’s resolution and accuracy are distinct: more representable values do not ensure lower error, and an early wrong interval can persist. The evaluation is limited in model, task, and target scope.

6. Discussion and Limitations

  • A wrong early interval can remain unrecoverable, and wider trees reduce depth without guaranteeing lower error.The representation requested at the interface also affects results, as shown by the provided-value and affine controls.Source: §6 · p. 7
  • The controlled evaluation covers one hosted model version, simple generated arithmetic, and a small historical-recall example.Targets use evaluator-supplied bounds and resolutions; out-of-range targets and unknown scales remain unsolved.Source: §6 · p. 7
  • The continuous-head baselines use different backbones and prior training tasks, so they do not isolate output-head effects.The study also excludes arithmetic-trained heads, open-ended word problems, and real forecasting datasets.Source: §6 · p. 7

7. Conclusion

The paper presents NUMERICJEV as a way to implement numerical readout through structured choices without a trained numerical head. In tested arithmetic, it reports lower range-normalized errors than direct selection, alongside uncertainty and sensitivity to settings.

7. Conclusion

  • NUMERICJEV implements numerical readout over a structured-choice interface without adding a trained numerical head.The decoder records each decision and exposes its finite resolution.Source: §7 · p. 7
  • Controlled arithmetic experiments show smaller range-normalized errors than direct selection, but MAPE differences are uncertain and results vary with scale, branching, and representation.Supplying the value substantially changes the task without making all encodings reliable.Source: §7 · p. 7
  • A precise-looking decoded number does not itself establish numerical uncertainty or broader regression generalization.The paper says these require additional evidence.Source: §7 · p. 7

Impact Statement

The paper cautions that fine decimal outputs or narrow intervals may be mistaken for accuracy or calibrated confidence. It frames historical index retrieval as a recall diagnostic, not validation of financial forecasting.

Impact Statement

  • A fine decimal output or narrow terminal interval may be mistaken for accuracy or calibrated confidence.The paper distinguishes a precise-looking readout from evidence about uncertainty.Source: Impact Statement · p. 7
  • The three-date historical index experiment is a recall diagnostic and does not validate financial forecasting.Source: Impact Statement · p. 7
  • The paper recommends retaining the decision trace, stating the imposed range and resolution, and evaluating losses and failure costs for the intended application.Source: Impact Statement · p. 7

A. Proofs and Interpretation of the Error Bound

The appendix bounds decoding error through interval refinement and describes the evaluation’s data generation, prompts, and sensitivity measures. Its proof and experiments rely on specified in-range targets and distinguish disagreement from error against truth.

A. Proofs and Interpretation of the Error Bound

  • At depth ceil(log_K N), a correct path reaches the unique grid cell; the error-bound proof applies to in-range targets and the greedy decoder.Source: A. Proofs and Interpretation of the… · p. 9
  • Conditional divergence probabilities follow the chain rule, not returned confidence scores; relative-error bounds require reference magnitudes bounded below by m > 0.Without that lower bound, small absolute deviations can produce large relative errors.Source: A. Proofs and Interpretation of the… · p. 9

B. Dataset Generation and Exact Numerical Checks

  • The arithmetic set uses seeded operand sampling and retains valid results strictly between 0 and 100, excluding 50; exact checks verify labels and decoding mechanics, not model performance.Source: B. Dataset Generation and Exact… · p. 9

C. Complete Prompt Specification

  • Prompts contrast direct candidate selection, interval choices, and digit or bit grid-index readout; reversed order changes dictionary insertion order while labels remain paired with descriptions.Figure 3 reports range-normalized absolute error, with mean markers and 95% family-bootstrap intervals across transforms sharing 100 cells.Source: C. Complete Prompt Specification · pp. 9–10, Figure 3 · p. 10

D. Ordering and Affine Sensitivity

E. Exploratory Records and Distinct Protocols

The exploratory records cover integer and fractional arithmetic questions under multiple numerical-readout protocols. A separate bit-prompt control improves MAPE, but its added computational rule prevents attributing the change solely to clearer wording.

E. Exploratory Records and Distinct Protocols

  • The arithmetic suite uses six integer and six fractional expressions, with candidates 0–15 for integers and sixteenth-step candidates in [0, 1) for fractions.Each question has two option orders and two repeated requests; decimal digits encode the numerical result, while the independent-bit protocol uses four-bit values or binary fractions.Source: E. Exploratory Records and Distinct… · p. 10
  • The protocols include explicit-set questions, four-round adaptive threshold search, and parallel threshold decoding from 15 normalized comparisons.The parallel method pools comparisons with equal-weight isotonic projection and selects the modal grid mass, using lower-value tie breaking; its point estimates differ from the separate midpoint-weighted distribution mean.Source: E. Exploratory Records and Distinct… · p. 10
  • Bit-query MAPE falls from 40.95–46.51% to 20.15–24.31% with explicit weights and arithmetic rules, so the change cannot be attributed solely to wording clarity.The independent bit probe, crossed prompt control, seven-method comparison, historical recall, supplied-value control, and changed-prompt/two-branch follow-up are separate protocols and are not pooled into one method-wide accuracy estimate.Source: E. Exploratory Records and Distinct… · p. 10

F. Continuous-Head Transfer Protocol

The continuous-head transfer protocol evaluates trained distribution heads on matched arithmetic and provided-answer conditions, while the CDF construction does not establish outcome calibration. The fixed evaluation is reproducible from saved records, but re-querying and broader validation remain bounded by the stated artifact limits.

F. Continuous-Head Transfer Protocol

  • Four SCM-trained spline and Beta heads use frozen Qwen3.5 backbones, and arithmetic prompts request a distribution for normalized values without including the answer.The provided condition adds the numerical result field; the prompts are semantically matched to the numerical question, not text-identical to the Jev choice request.Source: F. Continuous-Head Transfer Protocol · pp. 10–11
  • Table 4 reports range-percentage sensitivity to order or affine changes, while Table 5 gives descriptive arithmetic MAPE for six integer and six fraction questions.Predictions use the distribution mean without rounding to the evaluation grid; evaluation uses no gradient steps or calibration.Source: Table 4 · p. 11, Table 5 · p. 11, F. Continuous-Head Transfer Protocol · p. 11
  • Because the transfer baselines differ in backbone, task training, and representation, their poor transfer cannot establish an inherent limitation of continuous functions or a controlled head-architecture advantage.The supplied-answer controls compare direct choice with the same ten-way interval decoder, and the figure's tolerance metric is post-hoc rather than the predeclared primary endpoint.Source: F. Continuous-Head Transfer Protocol · p. 11

G. CDF Construction and Its Limits

  • Normalized yes/no threshold probabilities are projected to a monotone sequence and converted into histogram masses, but neither projection nor midpoint weighting calibrates outcomes.Raw probabilities and monotonicity violations remain in the artifact; Figure 4's displayed repair uses stored probabilities, not outcome calibration data.Source: G. CDF Construction and Its Limits · p. 11, Figure 4 · p. 12
  • Table 6 covers 256 expressions per condition and head, reporting tolerance score, MAPE, and range-normalized MAE with their respective optimization directions.Its caption identifies 64 bootstrap families but supplies no cell values in these passages.Source: Table 6 · p. 12

H. Reproduction and Artifact Boundaries

  • Saved responses, decoding paths, metrics, and bootstrap seed support offline reproducibility, but re-querying may change or fail and the work does not replace evaluation on more models or natural-language numerical tasks.Model weights, training data, and backend execution are not exposed.Source: H. Reproduction and Artifact Boundaries · p. 12
Loading 2609.28587v1…