PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text

Sungjune Lee, Myungjoo Kang

arXiv:2609.06993v2cs.SEcs.CLcs.LG

TL;DR

LLM-generated Markdown is consumed by renderers and automated tools, yet existing evaluations do not isolate boundary correctness from content or surface format. LatentMD benchmarks CommonMark fence-boundary failures and separates those correctness dimensions. Across nine models, boundary failures are widespread, with 38.0% of main-grid generations content-correct but boundary-broken.

  • Problem

    Existing evaluations do not isolate whether LLM-generated Markdown correctly tracks nested fence boundaries, despite downstream tools relying on valid structure.

    Source: §1 · pp. 1–2

  • Method

    LatentMD is a 4,179-prompt benchmark that separates content correctness from CommonMark-level boundary correctness and provides a scoring tool for arbitrary model outputs.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    38.0% of main-grid generations are content-correct but boundary-broken, with failures driven primarily by same-family symmetric-delimiter collisions and only partially mitigated by prompting.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Markdown boundary failures also appear in Python triple-quote docstrings, while JSON remains robust as an asymmetric-delimiter control.

    Source: §7 · p. 9

    LatentMD primarily covers Markdown fenced code blocks, and deployment impact depends on the downstream parser or renderer.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelines. Yet existing evaluations often conflate content quality with format adherence, leaving Markdown boundary failures under-measured. We introduce LatentMD, a benchmark and evaluation protocol for diagnosing CommonMark-level fence-boundary failures in LLM-generated Markdown. LatentMD separates content correctness from boundary correctness, enabling detection of outputs that are content-correct but boundary-broken. The benchmark contains 4,179 prompts and a CLI for scoring arbitrary model outputs. Across 9 LLMs and roughly 37,600 generations, we find that Markdown boundary failures are widespread: 38.0% of valid main-grid outputs are content-correct but boundary-broken, with substantial boundary breakage under unspecified prompts and in a small human-authored validation set. Ablations show that failures are driven primarily by same-family symmetric-delimiter collisions rather than nesting alone, are only partially mitigated by prompt hints, and generalize to Python triple-quote docstrings while JSON remains robust as an asymmetric-delimiter control. LatentMD provides a reproducible diagnostic target for parser-sensitive LLM evaluation.

1 Introduction

Markdown structure is consumed by renderers and automated tools, but existing evaluations do not isolate whether generated fences obey boundary rules. LatentMD addresses this gap by measuring content correctness separately from CommonMark-level boundary correctness.

1 Introduction

  • Markdown boundary failures can break renderers and tools that parse structure, extract code, or choose actions from generated output.Inner fences can prematurely close an outer Markdown wrapper, leaving trailing content outside the intended block.Source: §1 · p. 1
  • Nested-fence breakage has been reported across OpenAI ChatKit, Google Gemini CLI, Microsoft VS Code, and open-source agentic projects.The incidents are described as recurring deployment failures rather than benchmark artefacts.Source: §1 · p. 1
  • LatentMD adds a Boundary axis to a 4,179-prompt benchmark, exposing content-correct but boundary-broken outputs missed by content- or format-only evaluations.Its four-cell decomposition separates content correctness from CommonMark-level boundary correctness.Source: §1 · p. 2

2 Background & Related Work

Prior structured-output evaluations cover adjacent properties, but do not directly test fence balance or safe-length selection within the same delimiter family. LatentMD’s coverage matrix distinguishes direct measurement from partial detection and conceptual precursors.

2 Background & Related Work

  • Existing benchmarks assess structural matching, Markdown content, instruction compliance, format-following, and structured-output reliability, but none directly tests fence balance or safe-length selection within one delimiter family.Formal-language work on balanced-bracket tracking provides a precursor that LatentMD extends to realistic Markdown generation [7].Source: §2 · p. 2
  • The related-work comparison identifies safe-length selection and same-family fence collisions as gaps in existing structured-output evaluations.MDEval [5] and FMBench [22] are omitted from the matrix because their metric definitions do not directly cover the failure types.Source: §2 · p. 2
  • Table 1 maps failure modes to coverage levels: direct measurement, partial detection, conceptual precursor, or no coverage.Partial coverage indicates indirect detection without cause-level diagnosis.Source: Table 1 · p. 2

3 Benchmark Design

LatentMD operationalizes Markdown boundary failures against CommonMark and structures its benchmark around orthogonal prompt axes. Slot-based prompts vary language and task while the design tests wrapper pressure, nesting, and delimiter properties.

3 Benchmark Design

  • Boundary failures are judged against CommonMark 0.31.2, providing an externally specified reference rather than renderer-dependent judgments.The same fenced-code rules apply to GFM.Source: §3 · p. 2

3.1 Experimental Axes

  • Three orthogonal axes isolate wrapper pressure, inner-fence load, and delimiter properties that may drive boundary-state failures.Markdown fences can serve as wrapper, content, or literal text within one generation.Source: §3 · p. 2, §3.1 · p. 3
  • A1 forbids an outer fence, A2 requires a full-response fence, and A3 leaves wrapping unspecified to measure self-wrapping.These conditions vary wrapper pressure, from a no-wrapper baseline to a wrapped stress condition.Source: §3.1 · p. 3
  • B1 uses one code block; B2 and B3 add one or multiple literal Markdown fence examples, while B4 adds non-fence elements.The B-axis varies same-character delimiter load and opportunities for premature closure.Source: §3.1 · p. 3
  • D-axis ablations vary delimiter family and length, including cross-family conditions that remove same-family collision while retaining nesting.D-Inner-Run tests whether models predict a safe outer-fence length in advance.Source: §3.1 · p. 3
  • The 3 A levels and 4 B levels are fully crossed, yielding 12 A×B cells per task-language pair for within-task paired comparisons..Source: §3.1 · p. 3

3.2 Slot-Based Prompt Generation

  • Prompts fill LANG and TASK slots from McEval [3], using 9 programming languages, 3 structured formats, and 15 task instances per language.Tasks are stratified into 5 easy, 5 middle, and 5 hard instances under a fixed random seed.Source: §3.2 · p. 3
  • The released artifact includes prompts, templates, a generation script, and a CLI that scores arbitrary model outputs from JSONL records.The CLI emits per-record labels and aggregate failure rates with Wilson 95% confidence intervals.Source: §3.2 · p. 3

4 Evaluation Metrics

LatentMD evaluates Markdown along separate content and boundary axes, then classifies boundary violations and identifies content-correct outputs that still break spec-level boundaries.

4 Evaluation Metrics

  • LatentMD scores each response on boundary correctness and content correctness as orthogonal axes.Boundary correctness checks fence balance and safe outer-fence length; content correctness uses lightweight structural checks for a meaningful document.Source: §4 · p. 4

4.1 Format Metrics (F1–F3)

  • F1 checks wrapper instructions, F2 checks CommonMark fence balance, and F3 checks whether the outer fence is long enough for same-family inner runs.F3 is measured a-priori for the D-inner-run ablation and post-hoc in the main A×B analysis.Source: §4.1 · p. 4

4.2 Content Metrics (C1–C3)

  • C1 checks for at least two headings, C2 for a code-block language matching the requested language, and C3 for the expected number of literal-syntax fenced blocks.C3 expects zero blocks for B1 and B4, one for B2, and two for B3.Source: §4.2 · p. 4

4.3 Boundary Violation Error Types (E0–E3)

  • E1 denotes premature closure, E2 an unclosed fence, and E3 a latent collision in which nesting passes F2 but safe-length checking fails F3.E1 involves an interior bare same-family fence at least as long as the outer fence; E2 has no such interior bare match. When wrapped, all three failures are fixed by making the outer fence longer than the longest inner same-family run.Source: §4.3 · p. 4, Figure 2 · p. 5

4.4 Four-Cell Decomposition and Latent Failure

  • The four mutually exclusive, exhaustive cells separate content correctness from boundary correctness; the latent-failure cell is content-correct but boundary-wrong.Such outputs may render plausibly in lenient chat interfaces while corrupting downstream consumers that rely on spec-compliant parsing.Source: §4.4 · p. 5, Figure 3 · p. 5

4.5 Validity Filtering

  • The latent failure rate uses valid generations as its denominator, excluding outputs externally truncated by reaching the token limit.The paper reports the rate with a Wilson 95% score interval.Source: §4.5 · p. 5, §4.4 · p. 5

5 Experiments

Across nine LLMs, experiments trace Markdown fence-boundary failures from wrapping and delimiter conditions through mitigations, cross-format probes, and prevalence. The main-grid results show that content-correct but boundary-broken outputs are common.

5 Experiments

  • The study evaluates nine LLMs and organizes experiments from trigger and cause to mitigation, cross-format generalization, and latent-failure prevalence.The model table groups open-weight models by parameter-count tier and closed-source models in a Large/API group because their parameter counts are unavailable.Source: §5 · p. 6, Table 3 · p. 6

5.2 Wrapping × Nesting Triggers Boundary Failure (RQ1)

  • E3 concentrates on the forced-wrap A2 row (49–72%) and peaks at A2×B4, while all no-wrap A1 cells remain below 20%.A3 self-wraps 37% of responses and pools to 40–55% boundary failure; A2 reaches 100% in many model×B cells, a stress condition rather than a prevalence estimate.Source: Figure 4 · p. 6, §5.2 · p. 6

5.3 Same-Family Collision, Not Nesting Alone, Drives Failure (RQ2)

  • Cross-family nesting raises boundary-correct rates to 22.2% and 35.3%, supporting same-family symmetric collision as a dominant driver, not the only source.Switching the outer fence from backtick to tilde raises the rate from 7.2% to 37.0%; residual failures remain high.Source: Table 4 · p. 7, §5.3 · p. 7

5.4 Safe-Length Selection Failure and Prompt Mitigations (RQ3)

  • Eight of nine models rarely choose a safe outer-fence length even when told the inner run length; prompt hints reduce but do not close the gap.H3 is strongest, reducing mean failure by 30.9 percentage points (96%→65%); “at least 4” remains unsafe for inner runs of four or more.Source: §5.4 · p. 7, Figure 5 · p. 7

5.5 Manipulation Check and Natural Prompt Validation

  • The manipulation check finds 8.1% residual A1 failure among compliant responses versus 95.5% A2 failure; 30 human-authored prompts reach up to 56% failure.The small natural-prompt set is an ecological sanity check, not a population-level prevalence estimate.Source: Table 5 · p. 8, §5.5 · p. 8

5.6 Failure Generalizes to Python but Not JSON (RQ4)

  • JSON is 100% parse-valid across models and nesting depths, while Python triple-quote conditions vary widely and reproduce symmetric-delimiter failures.Under Python K1, pass rates are 97.8% for Small, 56.7% for Mid, and 31.7% for Large/API; exact failure modes depend on format and condition.Source: Table 6 · p. 8, §5.6 · p. 8

5.7 38% of Generations Are Latent Failures (RQ5)

  • Latent failures vary substantially by language: code-heavy Shell, C#, and C++ have higher rates than Markdown, whose low content correctness limits latent classification.Rates are Shell 46.2%, C# 45.1%, C++ 42.9%, and Markdown 9.2%; E3 also rises monotonically with nesting complexity (Z = +6.17, p < 10^-3).Source: §5.7 · pp. 8–9
  • 38.0% of main-grid generations are latent failures; among content-correct generations, 57.2% fail boundary metrics.Rates are 38.0% under primary Content and 33.1% under Full-C; model latent-failure rates range from 1.4% to 73.8%.Source: §5.7 · pp. 8–9, Table 7 · p. 9

6 Limitations and Future Work

LatentMD diagnoses Markdown fence-boundary failures against CommonMark 0.31.2, rather than ranking overall LLM reliability. Its validation and cross-format coverage are limited, with broader prompts and formats identified as future work.

6 Limitations and Future Work

  • LatentMD is a diagnostic benchmark, not a universal LLM-reliability ranking; deployment impact depends on whether downstream renderers or parsers are strict.Some failures may be harmless in lenient chat interfaces but corrupt spec-strict agents, code extractors, or parsers.Source: §6 · p. 9
  • The benchmark focuses mainly on Markdown fenced blocks, with smaller Python and JSON probes; its 30 natural prompts are an ecological check, not a prevalence estimate.Future work includes other structured-text formats and larger, more diverse user-derived prompt sets.Source: §6 · p. 9

7 Conclusion

LatentMD separates content correctness from CommonMark boundary correctness, exposing widespread content-correct but boundary-broken outputs. The paper concludes that same-family symmetric-delimiter collisions drive failures, while prompting only partly mitigates them.

7 Conclusion

  • 38.0% of main-grid generations are content-correct but boundary-broken; failures are driven primarily by same-family symmetric-delimiter collisions and only partly mitigated by prompting.The conclusion also reports analogous failures in Python triple-quote docstrings, while JSON remains robust as an asymmetric-delimiter control.Source: §7 · p. 9
  • LatentMD provides a reproducible protocol, prompt suite, diagnostic taxonomy, and scoring tool for assessing Markdown structure used by downstream tools.The conclusion argues that evaluation should measure parser-level boundary correctness, not only visible rendering, instruction compliance, or content quality.Source: §7 · pp. 9–10

A.1 Format Metrics: Full Definitions

  • The four-cell decomposition separates content correctness from boundary correctness, identifying latent failures that content-only or surface-format evaluations can miss.The operational definitions make the four cells mutually exclusive and exhaustive for valid responses.Source: §A.1 · p. 12

A.2 Auto-Detection Algorithms

  • The F2 checker tracks opening fences on a stack, while F3 checks whether the outer fence is longer than the longest same-family inner run.The a-priori F3 variant instead compares outer length with the inner-run constraint specified in the prompt.Source: §A.1 · p. 12, §A.2 · p. 13

A.3 Scope Justification

  • The study focuses on paired delimiters that serve as both opening and closing boundaries, excluding single-marker constructs and YAML front matter.Inline emphasis is also excluded because it does not create the investigated multiline boundary-state challenge.Source: §A.3 · p. 13

A.4 Coverage Matrix: Cell-Level Justification

  • Existing benchmarks provide at most partial or conceptual coverage of some related properties; the cited coverage analysis finds no direct diagnosis of fence-length collisions.StructEval does not compare fence lengths, and FormatBias does not implement a boundary-correctness axis or four-cell decomposition.Source: §A.4 · p. 14

A.5 Latent Failure Downstream Example

  • A visually normal generation can still merge code blocks with prose in a CommonMark-strict editor, despite rendering as expected in a chat interface.The concrete example passes content checks while an inner run matching the outer wrapper causes F3 to fail.Source: §A.5 · pp. 14–15, Table 8 · p. 15

B.1 Visual Overview of Axes

  • The benchmark varies outer constraint, inner nesting, and delimiter properties across its three axes.The visual overview assigns three outer-constraint levels, four nesting levels, and delimiter ablations to axes A, B, and D.Source: Figure 6 · p. 15

B.2 Evaluation Pipeline

LatentMD renders experimental conditions into prompts, collects model responses, and scores format and content before assigning boundary-violation labels. Its design spans a released 4,179-prompt suite and includes main-grid, ablation, and validation tracks.

B.2 Evaluation Pipeline

  • Each evaluation track renders axis values into a prompt, obtains a model response, scores format and content metrics, and assigns one of four boundary-violation labels.The dashed wrapper marks model interaction as the only stochastic stage when temperature is above zero; experiments use T = 0.Source: Figure 7 · p. 16

B.3 LANG Selection: Programming Stratum Consensus

  • The programming-language pool uses a two-of-three ranking consensus, then requires at least one instruction-tuned McEval task for each selected language.The final programming set contains nine languages, alongside Markdown, JSON, and HTML as structured-format values.Source: §B.3 · p. 16

B.4 Prompt Templates: Per-Condition Examples

  • The released prompt set contains 4,179 prompts, with templates varying outer wrapping, nesting, delimiter conditions, hints, and cross-format cases.The examples include a realistic no-wrap-directive prompt and conditions requesting raw Markdown source within a fenced response.Source: §B.4 · pp. 16–18

C.1 Hyperparameter Table; C.2 Closed-source API Usage and Cost

  • Open-weight models use temperature-0 generation, and optional thinking modes are disabled for three models; closed-source APIs cost approximately $122 for the full benchmark.The three APIs are Claude-Sonnet-4, GPT-4o, and Gemini-2.5-Flash; reported costs are reproducibility references and may vary.Source: §C.1 · p. 19, §C.2 · p. 19

D.1 Truncation Statistics

  • Length-truncated responses are excluded from boundary metrics; Gemini-2.5-Flash-Lite is excluded from the main cohort after 19.2% truncation left 3,376 valid responses.Gemini-2.5-Flash has approximately 12–13% truncation, while GPT-4o and Claude-Sonnet-4 show 0%.Source: §D.1 · pp. 19–20

D.2 Per-Language Truncation Distribution

  • Gemini-2.5-Flash truncation is concentrated in Go (26.1%), JavaScript (22.2%), TypeScript (22.2%), and C++ (17.2%).Other models show less than 1% truncation across languages; the authors describe the pattern as consistent with a verbose-output hypothesis.Source: §D.2 · p. 20, Table 12 · p. 20

D.3 Natural Prompt Validation

  • The external-validity check uses 30 human-authored prompts across high, medium, and low elicitation levels, treated as an ecological sanity check rather than a prevalence estimate.The table reports boundary-incorrect responses over observed responses, excluding truncated outputs.Source: §D.3 · p. 20, Table 13 · p. 21

E.1 Full A×B×Model Results

  • Under unspecified A3 prompts, models self-wrap about 37% of the time, with boundary failure between the A1 baseline and A2 stress-test ceiling.The per-model table reports boundary failure across A×B conditions, ordered A1, A3, then A2 by experimental severity.Source: §E.1 · p. 21, Table 14 · p. 21

E.2 Main Results with Wilson 95% Confidence Intervals

The results report high boundary-failure rates in many forced-wrap conditions and use Wilson intervals to represent prompt-sampling uncertainty. Additional analyses describe outer-fence choices, hint-condition comparisons, and a per-model A1 decomposition.

E.2 Main Results with Wilson 95% Confidence Intervals

  • Across most models, A2 cells have boundary-failure rates of at least 90%, with Wilson interval widths of at most 8 percentage points.The intervals reflect prompt-sampling variance, not repeated-sampling variation within prompts under greedy decoding.Source: §E.2 · p. 22

E.3 Outer-Fence Length Distribution (All Nine Models)

  • Most models choose an outer fence of length 3 regardless of the inner-run condition; only GPT-4o non-trivially exceeds the inner length.Several models partially adapt by matching the inner length, but matching does not satisfy the stated strict-greater-than safety condition.Source: §E.3 · p. 22, Table 17 · p. 23

E.4 Hint Ablation: Per-Model Heatmap

  • The hint heatmap encodes per-model boundary-failure rates by hint condition, with models in tier order and row means at right.Deeper red denotes higher failure rate; the caption points readers to §5.4 for the result summary.Source: Figure 8 · p. 23

F.1 A1 Per-Model Decomposition

  • The A1 decomposition table reports four steps: A3 wrap rate, A1 non-compliance, compliance-conditional A1 residual failure, and A2 boundary failure.Its caption identifies the per-model breakdown as dispersion behind the nine-model averages reported in Table 5.Source: §F.1 · p. 23, Table 18 · p. 24

F.2 Full-C Sensitivity Analysis

The sensitivity analysis compares primary and Full-C content definitions, while the stratified error analysis specifies the dimensions along which error types are examined.

F.2 Full-C Sensitivity Analysis

  • 33.1% of generations are latent failures under Full-C, versus 38.0% under the primary content definition; all nine models show higher rates under the primary definition.The ranking-stability statistic is τ = 0.667, below the prespecified 0.8 threshold, so C4 is demoted and primary Content remains C1∧C2∧C3.Source: §F.2 · p. 24

G Stratified Error-Type Analyses

  • The stratified error analysis pools valid, non-truncated responses across nine models by language, model-size tier, nesting condition, and task difficulty.It distinguishes E0 correct, E1 premature, E2 unclosed, and E3 latent-collision outcomes.Source: G Stratified Error-Type Analyses · p. 24

G.1 Per-Language Distribution

Boundary failures vary by language, model tier, nesting condition, and wrapper instruction, while difficulty-level rates remain nearly flat. The analyses distinguish boundary mechanisms and show that unclosed fences usually reflect imbalanced nesting rather than missing closing attempts.

G.1 Per-Language Distribution; G.7 Per-Language Latent Failure Distribution

  • Markdown has E0=64.8% but latent failure is 9.2% (content correctness is 21.6%); Shell’s latent failure is 46.2%, while JSON, Shell, and HTML have E3 rates ≥39%.The passages attribute cross-language E3 variation to syntactic similarity between fence delimiters and inner content tokens, not language semantics.Source: §G.1 · pp. 24–25, §G.7 · p. 26, Table 25 · p. 27

G.2 Per-Model-Size-Tier Distribution

  • Mid-tier models have the lowest E0 rate (22.3%) and highest E3 rate (58.8%); Small and Large tiers are closer on E0 (53.4% vs 48.3%).The non-monotonic tier ordering is driven by Gemma-3-27B’s outlier behavior.Source: §G.2 · p. 25, Table 20 · p. 25

G.3 Per-B-Condition Distribution

  • E1 peaks at B2 (19.4%), while E3 is high at B1 (39.6%) and B4 (43.3%), where inner blocks compete most directly with the wrapper run length.B2 requires a code example plus a literal Markdown explanation.Source: §G.3 · p. 25, Table 21 · p. 25

G.4 Per-Difficulty Distribution

  • E0 rates across easy, middle, and hard McEval difficulty levels are 43.6%, 45.2%, and 43.7%, respectively, indicating a nearly flat distribution.This pattern suggests boundary-state tracking failures are not primarily explained by McEval semantic task difficulty.Source: §G.4 · p. 25, Table 22 · p. 25

G.5 A×B Joint Distribution (Pooled)

  • Across pooled A×B cells, A2 is E3-dominant (49–72%); A1×B2 and A1×B3 instead have dominant E2 rates of 12.8% and 11.6%.A3 cells fall between A1 and A2, with E3 dominant at 15–39%.Source: §G.5 · pp. 25–26, Table 23 · p. 26

G.6 A×B Joint Distribution (Per-Model)

  • Under A2, E3 accounts for ≥92% of cell mass in Claude-Sonnet-4 and GPT-4o, while Gemma-2-9B and Llama-3.1-8B retain substantial E2 mass.The per-model patterns show that smaller models also leave fences unclosed, rather than failing only through length-safety collisions.Source: §G.6 · p. 26, Table 24 · p. 26

H.1 F1×F2 Joint Failure Analysis; H.2 Block Compliance by Boundary Condition; H.3 E2 Unclosed-Fence Sub-Patterns

  • F1/F2 and C3 distinguish instruction and block compliance from boundary balance; most E2 records reflect an imbalanced nesting stack, not absent closing attempts.C3 is most informative for B2 and B3; small models drop to single digits on B3 while large models retain ≥50%.Source: §H.1 · p. 27, §H.2 · p. 27, §H.3 · p. 28

H.4 Error Cascade Analysis

The analyses find that wrapper-required judgments differ sharply from direct-output judgments, while boundary-failure susceptibility varies by model and shows an aggregate trend with nesting complexity. The study also leaves temperature sensitivity untested and finds a contrast between symmetric-delimiter Python docstrings and asymmetric-brace JSON.

H.4 Error Cascade Analysis

  • Content correctness is 70.0% conditional on E1 versus 66.1% without E1, a small aggregate gap suggesting premature closure does not systematically truncate or omit content.Per-model estimates vary substantially, and some rows have small non-E1 sample sizes.Source: §H.4 · p. 28, Table 29 · p. 29

I.1 Inter-Model Agreement (Fleiss’ Kappa)

  • Aggregate Fleiss’ κ is 0.349 across nine models; the Mid tier’s κ is +0.008, reflecting little agreement between Gemma-3-27B and Qwen3-32B on which prompts fail.The passages characterize prompt susceptibility as at least partly model-specific.Source: §I.1 · p. 29, Table 31 · p. 29

I.2 Cochran–Armitage Trend Test

  • The pooled E3 trend with B-condition complexity is positive (Z=+6.17, p<10^-3), although only 5/9 models show statistically significant positive trends.The four exceptions have low E3 throughout or U-shaped profiles.Source: §I.2 · p. 29, Table 32 · p. 30

I.3 Chi-Squared Pairwise Comparisons

  • Pairwise Cramér’s V ranges from 0.01 for Claude versus Gemma-2-9B to 0.61 for Claude versus Gemma-3-27B; the four highest-V pairs involve Gemma-3-27B or Claude-Sonnet-4.The reported effect sizes correlate with the models’ per-model latent-failure rates.Source: §I.3 · p. 30, Table 33 · p. 30

I.4 McNemar Paired Test (A1 vs A2)

  • In 4,509 discordant prompts, models passed under A1 but failed under A2, versus only 4 prompts showing the reverse; the paired odds ratio b/c is approximately 1127.This is the within-prompt counterpart to the approximately 95% A2 failure reported in Table 14.Source: §I.4 · p. 30, Table 34 · p. 30

J.1 Cross-Format Analysis

  • JSON parse-validity is 100% across models and nesting depths, while Python triple-quote conditions show large model-specific failure rates; K2 often improves over K1.The cross-format results support extending the symmetric-delimiter boundary-tracking concern beyond Markdown, with JSON as an asymmetric-delimiter control.Source: §J.1 · p. 31, Table 35 · p. 31

J.3 Temperature Sensitivity (Future Work)

  • Main experiments use temperature=0; whether higher temperatures exacerbate or mitigate boundary failures remains untested.Temperature sensitivity across 0.0–1.0 is left to future work.Source: §J.3 · p. 33

J.2 Inline Code Span Results

The additional analyses test inline-span collisions, downstream harm, newer models, and naturally occurring prompts. They show that failures persist in realistic settings and can disrupt renderers and extractors, while observed rates depend on elicitation and wrapping behavior.

J.2 Inline Code Span Results

  • At inline runs 1 and 2, 8 of 9 models failed 100% of cases; the 96%+ pooled verbatim-match failure rate includes interpretation errors, not only collisions.No inline record was classified as a latent failure; structural failures concentrated in asymmetric-close and length-matched-collision modes.Source: Table 36 · p. 31, §J.2 · pp. 32–33

K Practical Evidence: Full List

  • Table 37 compiles independently reported boundary-state tracking failures across LLM platforms and developer tools.Its caption notes that URLs were verified at the time of writing.Source: Table 37 · p. 33

L Additional Experiments

  • The additional analyses cover headline-rate decomposition, real-consumer harm, newer models, expanded organic prompts, and diagnostic diversity with a coverage-preserving core.They use the main study’s decoding protocol and released CLI unless otherwise stated.Source: L Additional Experiments · p. 34

L.1 Realistic-Condition Decomposition of the Headline Rate

  • Among content-correct A3 responses, 51.2% were boundary-broken; within A3, latent failures occurred in 85.8% of self-wrapped versus 8.3% of non-wrapped responses.A3 and the natural sets estimate realistic elicitation prevalence, whereas A2 measures a forced-wrap stress ceiling; the pooled grid rate is not deployment-weighted.Source: §L.1 · p. 34, Table 38 · p. 34

L.2 Downstream Harm Under Real Consumers

  • 99.7% of E3 records yielded a truncated block even with an AST-based extractor, and 94% corrupted rendered output.Cross-renderer disagreement was at most 0.2 percentage points; the E0 nested-literal-fence control had 41.0% corruption versus 0.0% for plain documents.Source: §L.2 · p. 35, Table 41 · p. 35, Table 40 · p. 35

L.3 Newer-Generation Models

  • Six of seven newer models still failed at 96–99.7% under forced wrapping, while lower pooled rates tracked reduced self-wrapping rather than improved capability.Python triple-quote collisions persisted, while JSON remained at 100% for all 16 models under every nesting condition.Source: §L.3 · p. 36, Table 42 · p. 36

L.4 Expanded Natural-Prompt Set (n = 248)

  • 17.5% of self-wrapped organic responses failed, versus 0.77% of non-wrapped responses; the 16-model natural set’s overall failure rate was 1.97%.The prompts were curated for boundary relevance, so these rates describe Markdown-document requests, not population-wide user traffic; the set reports boundary metrics only.Source: §L.4 · p. 37, Table 43 · p. 37

L.5 Effective Diagnostic Diversity and a Coverage-Preserving Core

The analysis tests whether LatentMD’s diagnostic axes measure distinct dimensions and whether a smaller prompt core can preserve the benchmark’s findings. It finds six separable dimensions, while showing that screening and diagnosis require different amounts of coverage.

Effective Diagnostic Diversity

  • The ten diagnostic axes yield six conservatively separable dimensions: median inter-axis |ρ| is 0.36, and only 4 of 45 pairs reach |ρ| = 0.70.The forced-wrap axis, the headline capability measure, does not induce a common ordering on the other axes.Source: §L.5 · p. 39, Table 52 · p. 41

Coverage-Preserving Core

  • A 144-prompt coverage-preserving core retains the aggregate rate and broad model ordering, but loses resolution for small deltas and paired attribution.Same-task A1–A2 pairs fall from 720 to 48 in the paired core and to 3 under independent sampling.Source: §L.5 · p. 40, Table 49 · p. 40, Table 50 · p. 40
  • Eight prompts per cell detect the unusually large Claude Sonnet 5 run-3 versus run-4 contrast, but not small generational deltas or diagnosis across dimensions.The authors conclude that a core suits low-cost screening, while diagnosis across six dimensions requires the full suite.Source: §L.5 · pp. 40–41, Table 51 · p. 41
Loading 2609.06993v2…