SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

• SyntaxBench pairs five core character-level tasks across English and character-length-matched random strings, alongside index_to_span, a long-text substring-extraction stress test.The stress-test documents share a 200–500 word band but are not character-length matched.Source: §1 · p. 2
• The benchmark asks whether tokenizer granularity explains input-condition differences, when thinking helps or hurts, whether short-task competence transfers to long texts, and which tests make claims falsifiable.Source: §1 · p. 2
• The framework combines accuracy with confidence intervals, paired tests, effect sizes, tokenization and class-conditional analyses, and multiple-comparison corrections.Source: §1 · pp. 2–3
• Random strings pack 1.892 chars/token versus English’s 3.169, while Qwen3.6-27B palindrome accuracy falls from 0.952 non-thinking to 0.886 thinking at four-shot.Source: §1 · p. 2
• Qwen3.5 four-shot count_chars accuracy rises from 0.698 to 0.880 to 0.958 across 2B, 9B, and 27B, yet the 9B and 27B index_to_span cells remain at 0.5%.Source: §1 · pp. 2–3

• Broad suites such as GLUE [32], SuperGLUE [31], BIG-Bench [29], MMLU, and HELM emphasize semantic capabilities, while SyntaxBench isolates deterministic character-level operations with statistical tests.Source: §2 · p. 3
• Tokenization-free ByT5 and CANINE show improved character-task robustness, while prior work [28] [1] reports character-counting failures that SyntaxBench links to tokenizer granularity.Source: §2 · pp. 3–4
• Few-shot prompting and chain-of-thought motivate shot-mode analysis; prior work reports non-monotonic few-shot results and sensitivity to demonstration selection.Source: §2 · p. 3
• Overthinking research [23] describes chain-of-thought degradation on direct decisions; SyntaxBench reports that thinking can help, hurt, or leave performance unchanged across model-task-shot settings.Source: §2 · p. 3
• Scratchpads [23], MATH, and GSM8K [6] target multi-step numeric reasoning, whereas SyntaxBench evaluates deterministic string operations and a long-horizon index_to_span task.Source: §2 · p. 3
• Benchmarking concerns include point estimates without effect sizes or multiple-comparison correction [20]; SyntaxBench combines paired tests, agreement, correction, and standardized protocols.Source: §2 · p. 3

3.1 Benchmark Overview

• The six tasks comprise five core operations with interpretable outputs and partial-credit metrics, plus index_to_span, which extracts a substring at specified character offsets from a 200–500 word document.Source: §3.1 · p. 4

3.2 Data Sources and Random-String Construction

• For the five core tasks, random strings match the English vocabulary’s character-length distribution; index_to_span instead uses 200–500 word documents with differing mean character lengths across conditions.Observed document means are 321 English words and 336 random words; mean character lengths are 2,128 English and 3,593 random.Source: §3.2 · p. 4

3.3 Models and Thinking Modes

• Eight open-weight models span 11 model-mode configurations, with Qwen3.5 providing a within-family scale comparison and Qwen3.6-27B and Gemma4-31B providing fixed-scale mode comparisons.GPT-OSS-20B and Nemotron-Cascade-2 are evaluated in thinking mode only.Source: §3.3 · pp. 4–5

3.4 Evaluation Protocol

• Each configuration uses zero-, one-, and four-shot prompts with deterministic decoding; each test instance is an independent request, and reasoning traces are captured but excluded from scoring.Source: §3.4 · p. 5

3.5 Scoring and Metrics

• Scoring reports exact-match and relaxed accuracy, parse-error rate, and Cohen’s κ, alongside bootstrap intervals, paired McNemar tests, chi-square tests, Kendall’s τ, and multiple-comparison correction.Tokenizer-stratified analysis bins examples by token count and reports tokenization statistics and per-bucket accuracy.Source: §3.5 · p. 5

4.1 Setup and Main Results

• At four shots, 25 of 32 cells for 27B+ models reach 90% EMA on four core tasks, but edit_distance separates models and index_to_span peaks at 6.75%.GPT-OSS-20B reaches 89.0% on edit_distance, versus 0.0–43.0% for the rest; its 6.75% is the best four-shot index_to_span EMA. Macro EMA hides the floor on index_to_span and near-ceiling behavior on other tasks.Source: §4.1 · p. 5, Figure 1 · p. 6

4.2 Headline 1: Tokenization Mediates the English-vs-Random Gap

• English strings average 3.169 characters per token versus 1.892 for random strings, and count_chars accuracy falls as English examples occupy more tokens.Random inputs remain more accurate in every shared token-count bin; failure analysis localizes nearly every off-by-one English count_chars error to multi-character subwords. Tokenizer-stratified evaluation separates tokenizer granularity from linguistic familiarity.Source: §4.2 · pp. 6–7, Figure 2 · p. 7

4.3 Headline 2: Reasoning Mode Helps Unevenly and Sometimes Hurts

• Thinking can hurt, leave accuracy essentially unchanged, or accompany strong performance, depending on model and task.Qwen3.6-27B palindrome EMA is 0.952 non-thinking versus 0.886 thinking at four shots, while Gemma4-31B is essentially unchanged across four near-saturated tasks; DeepSeek-R1-32B reaches 99.8% on longest_string_finder and 99.4% on letter_container. The authors interpret the pattern on saturated tasks as reasoning-mode interference.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 Diagnostic Depth: What Aggregate Accuracy Misses

• Class-conditional and shot-trajectory diagnostics reveal errors hidden by aggregate accuracy, including positive-class underprediction and non-monotone responses to demonstrations.Qwen3.5-2B palindrome EMA is 0.430 on true palindromes versus 0.965 on negatives; Qwen3.6-27B’s palindrome accuracy dips at one shot in both modes, with thinking remaining below its zero-shot accuracy.Source: §4.4 · p. 8

4.5 Floor: Positional Substring Extraction

• Across configurations, four-shot index_to_span EMA reaches at most 6.75%, and the authors attribute the floor to maintaining character position while extracting spans from long documents.The task uses 200–500-word documents; the Qwen3.5-2B four-shot cell is incomplete, with 0/113 English responses and no random-condition estimate.Source: §4.5 · pp. 8–9

4.6 Pairwise Model Comparisons

• Pairwise McNemar comparisons show GPT-OSS-20B winning 9 and losing 0 of 18 cells against DeepSeek-R1-32B in thinking mode, while comparisons are mode-specific.The tests are not pooled into one ordering because some model families lack a shared thinking mode; several cross-family pairs therefore have no shared-mode test.Source: §4.6 · p. 9

5 Discussion

• Tokenization mediates English-versus-random behavior, reasoning mode helps unevenly and can hurt, and index_to_span remains unsolved across all 11 configurations.The discussion also finds that the interpretation of these results depends on the statistical machinery applied.Source: §5 · p. 9
• 23 of 25 Bonferroni-significant cells favor random inputs; random strings pack 1.892 chars/token versus 3.169 for English, and count_chars accuracy declines as English token count rises.The authors attribute the random-easier majority to tokenizer-mediated character visibility and warn that unstratified evaluations risk mistaking tokenizer artifacts for model capability.Source: §5 · p. 9
• Qwen3.6-27B’s palindrome_test drops by ∆EMA = −0.066 at 4-shot with thinking, Gemma4 is unchanged on four near-saturated tasks, and DeepSeek-R1 benefits on several tasks.The discussion frames reasoning mode as a deployment switch and recommends paired comparisons that also report token cost and latency.Source: §5 · p. 9
• Averaging saturated, discrimination, and floor tasks hides model differences; GPT-OSS-20B’s macro EMA is 0.818 [0.512, 0.989], so per-task and per-mode, per-shot reporting is needed.The interval resamples six task-level EMAs, so its width represents between-task spread rather than a sample bootstrap.Source: §5 · p. 9
• The authors recommend tokenizer stratification, paired tests across modes and shots, class-conditional binary metrics, and both Bonferroni and BH-FDR corrections.They identify edit_distance and index_to_span as the most resistant operations and recommend deterministic library calls until architectures for long-horizon symbolic state become standard.Source: §5 · p. 10

6 Limitations

• The evaluation covers eight open-weight models from 2B–32B, excluding frontier-scale and closed-weight models; tokenizer-stratified results also require public tokenizer access.Closed-tokenizer models are therefore absent from per-bucket results.Source: §6 · p. 10
• Tasks use fixed task-specific prompts, and the benchmark covers only English and synthetic random strings, so findings may differ for other scripts or tokenizer behaviors.Instruction phrasing is not systematically varied; the authors also name agglutinative morphology as an untested scope.Source: §6 · p. 10
• Architecture-specific backends may introduce small per-model variations in tokenization handling and reasoning-trace stripping despite deterministic decoding.Source: §6 · p. 10
• The six tasks exclude document-level constrained generation, encoding/decoding, and multi-step algorithmic reasoning, focusing instead on operations with single-pass ground truth and defined partial-credit metrics.Source: §6 · p. 10
• Thinking-versus-non-thinking comparisons conflate chat-template, decoding-length, and sometimes training-data differences, so they do not isolate the reasoning trace alone.The study treats thinking mode as the deployment-level switch available to users.Source: §6 · p. 10

A.1.2 Length-Matched Random-String Corpus

• The five core tasks pair English words with character-length-matched random strings, while index_to_span documents share a 200–500 word band and are not character-length matched.The random corpus pairs each English word with a uniformly sampled string of the same length; random documents instead sample NLTK-length strings rather than replacing words in Wikipedia excerpts.Source: §7 · p. 10, §A.1.2 · p. 14

7 Conclusion

• 23 of 25 Bonferroni-significant English-vs-random cells favor random inputs; index_to_span peaks at 6.75% four-shot EMA, and Qwen3.6-27B loses 6.6 EMA points with thinking on palindrome_test.The study covers eight open-weight LLMs across 11 model and thinking-mode configurations.Source: §7 · p. 10

A.1.2 Length-Matched Random-String Corpus

• Across 197 complete comparisons, 25 (12.7%) show a Bonferroni-significant English-vs-random gap; random inputs are easier in 23 cells and English in two.One incomplete Qwen3.5-2B index_to_span cell is excluded from this comparison family.Source: §A.1.2 · p. 14

A.2 Task Definitions

• The benchmark defines six deterministic tasks, with task-specific prompts, 0-, 1-, and 4-shot demonstrations, and test samples with ground-truth labels.The core suite covers character counting, letter containment, palindrome detection, edit distance, and longest-string selection; index_to_span asks for a substring at explicit character offsets.Source: §A.2 · p. 14, §A.2.1 · pp. 14–15, §A.2.2 · p. 15

A.2.3 Stress-Test Task

• Index_to_span requires verbatim extraction between two character offsets from documents in the 200–500 word band, whose mean lengths are 2,128 English characters and 3,593 random characters.The random documents are assembled from sampled random-vocabulary strings, not by replacing words in English excerpts.Source: §A.1.2 · p. 14, §A.2.3 · p. 15

A.3 Prompt Engineering

• Prompts use fixed task-specific instructions and held-out demonstrations; each test instance is an independent request, and reasoning traces are excluded from parsed answers.Responses are parsed from a JSON result field, with whitespace and formatting normalization.Source: §A.3 · p. 15, §A.4 · p. 15

A.5.2 Models, Modes, and Inference Backends

• Eight open-weight models are evaluated across 11 model-mode configurations, with some models available in both thinking and non-thinking modes.The configurations vary by model, backend, quantization, and tensor parallelism; not every model supports both modes.Source: §A.5.2 · p. 16

A.5.3 Inference Hyperparameters

• The study uses temperature-zero greedy decoding for reproducibility, with top-p 1.0 (nucleus sampling disabled), repetition penalty 1.0 (disabled), and maximum token limits of 2048 for non-thinking and 8192 for thinking tasks.The English vocabulary contains 236,736 NLTK words, while the palindrome list has 229 entries; index_to_span documents use English Wikipedia excerpts and random-vocabulary documents targeted to 200–500 words.Source: §A.1.1 · p. 14, §A.5.3 · p. 16

A.5.4 Reproducibility

• Seed 42 is fixed for stochastic pipeline operations, and temperature-zero decoding supports reproducible outputs for a given backend version.The fixed seed covers bootstrap and demonstration sampling.Source: §A.5.4 · p. 16
• Each task uses a fixed test set at 0-, 1-, and 4-shot, with standardized prompt templates and no train/test split.The planned run manifest records GPU, backend version, quantization, wall-clock time, and peak VRAM.Source: §A.5.4 · p. 16
• The pipeline and analysis scripts that generate every figure and table are slated for release, and raw model outputs are stored in JSONL format.Source: §A.5.4 · p. 16

A.6 Statistical Methods Reference

• For paired classifier comparisons, McNemar’s statistic uses the counts where each model alone is correct, and their ratio is reported as the odds-ratio effect size.The test also compares thinking and non-thinking modes on the same task-and-shot cell; cells with no discordant samples are flagged as ceiling effects.Source: §A.6 · p. 17
• Bootstrap 95% confidence intervals resample task samples 10,000 times, while the composite macro-EMA interval resamples six task-level EMAs.The two procedures therefore quantify sample-level uncertainty and between-task spread, respectively.Source: §A.6 · p. 17
• English-vs-random comparisons use Pearson’s chi-square test with Yates’ correction when a cell count is below five, reporting Cramér’s V.The conditions are treated as independent samples rather than paired observations.Source: §A.6 · p. 17
• Kendall’s τ relates shot indices 0, 1, and 4 to EMA, with actual values used to classify trajectories as monotone, peak, valley, or flat.With three shot points, τ can take values −1, −0.33, 0.33, or 1.Source: §A.6 · p. 17
• Bonferroni and Benjamini–Hochberg FDR corrections are applied at α = 0.05, with headline tables using the more conservative Bonferroni cutoff.Chi-square tests use the 197 complete English-vs-random comparisons; the incomplete cell is excluded.Source: §A.6 · p. 17

A.7 Tokenizer-Stratified Analysis Methods

• For models with public HuggingFace tokenizers, the analysis records mean tokens, characters per token, and sample counts by condition, plus 0-shot accuracy by token bin.Closed-tokenizer models are omitted from this analysis but retained in English-vs-random chi-square tests.Source: §A.7 · p. 17
• Across five core tasks and public tokenizers, random inputs average 0.542 tokens per character and 1.892 characters per token, versus 0.344 and 3.169 for English inputs.The analysis pools 8,000 samples per condition and reports Cohen’s d = 2.165 for tokens per character; index_to_span is excluded.Source: §A.7 · p. 17
• The tokenization headline excludes the single-character letter query tag and index_to_span, and records its inputs and inclusion rule in a CSV.Source: §A.7 · p. 17

A.8 Datasheet Summary

• The benchmark comprises six tasks and 3,200 samples per complete shot-and-thinking-mode configuration; the Qwen3.5-2B index_to_span four-shot cell is incomplete.Source: §A.8 · p. 17
• English vocabulary comes from NLTK, palindrome words from Wiktionary, Wikipedia excerpts are used under CC BY-SA, and random strings are programmatically generated with a seed.The construction scripts are slated for release.Source: §A.8 · p. 18
• The five core tasks use length-matched English-word and random-string pairs, while index_to_span documents occupy a 200–500-word band without character-length matching.Demonstrations are sampled from a held-out pool with a fixed seed.Source: §A.8 · p. 18
• The dataset is intended for diagnostic evaluation of character-level reasoning, tokenizer-mediated effects, and reasoning-mode behavior—not for broad safety, fairness, or capability claims.Source: §A.8 · p. 18
• The dataset is released under CC BY-SA 4.0, and a Croissant metadata record accompanies it; the Wiktionary-derived palindrome list retains its source’s CC BY-SA 3.0 terms.The code is planned for release, with semantic-version updates and a manifest of evaluated model versions and inference backends.Source: §A.8 · p. 18

B Supplementary Tables and Figures

• Supplementary materials include per-sample model outputs in JSONL files organized by model, task, shot, and mode, with tables reproducible from summary and per-test CSVs.Source: B Supplementary Tables and Figures · p. 18

B.1 Aggregate Performance Tables

• The aggregate tables cover per-task accuracy by reasoning mode, shot ablations, agreement, paired significance tests, shot-count trends, English–random gaps, parse errors, error distributions, thinking costs, effect sizes, and composite leaderboards.Source: §B.1 · p. 18
• Shot ablations report EMA across zero-, one-, and four-shot settings, while Cohen’s κ uses four-shot results and linear weighting for integer tasks.Source: Table 6 · p. 19, Table 7 · p. 19
• Pairwise McNemar tables report the fraction of Bonferroni-significant task-by-shot cells, with tint indicating net wins, losses, or mixed results for each model pair.Source: Table 8 · p. 19, Table 9 · p. 19
• Kendall’s τ summarizes the association between shot count and EMA, but its three-point basis has low power and supports qualitative interpretation only.Source: Table 10 · p. 19
• The four-shot English–random table reports the EMA gap with Bonferroni-corrected significance, while the thinking-cost table ranks task-level EMA per mean thinking token.Source: Table 11 · p. 19, Table 15 · p. 21
• Composite leaderboards average complete task-level EMAs and use bootstrap percentile confidence intervals; incomplete input-condition tasks are omitted.The effect-size table presents the top 12 model-pair effects by absolute h with 95% bootstrap percentile intervals.Source: Table 17 · p. 21, Table 18 · p. 21, Table 16 · p. 21