PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparing Chunking and Embedding Strategies for Turkish RAG Systems

Mustafa Sertaç Türkel, Fatma Nur Korkmaz, Ahmet Tuğrul Bayrak

arXiv:2608.26192v1cs.CLcs.AI

TL;DR

Chunking and embedding choices in Turkish RAG have not been systematically studied, despite Turkish morphology and document layouts posing distinct retrieval considerations. This paper compares chunking, embedding, and generator combinations and finds that chunking’s value depends on content, with Docling delivering a 17.0-point table-question accuracy gain over fixed-length chunking.

  • Problem

    Evidence on Turkish RAG chunking and embeddings is limited, despite suffix-driven morphological variation and institutional documents mixing prose with tables.

    Source: §I · p. 1

  • Method

    The study fully crosses three chunking methods, five embedding models, and two generators across 300 questions, producing 9,000 graded evaluations with paired significance testing.

    Source: §I · p. 1, §III · p. 2

  • Results

    17.0 points: Docling improved table-question accuracy over fixed-length chunking, and its advantage was conditional on tabular rather than prose content.

    Source: §VI · p. 5

  • Takeaways & Limitations

    The best complete configuration cannot be chosen by combining marginal component rankings: Docling with Mursit-large and GPT-4o mini scored 87.00%.

    Source: §VII · p. 6

    Accuracy reflects an LLM judge’s verdicts, which were not validated against human annotation.

    Source: E. Evaluation Protocol and Operational… · p. 3

Abstract

from arXiv · show

How documents are segmented into retrievable chunks and how those chunks are embedded strongly affect Retrieval-Augmented Generation (RAG) quality, yet neither has been systematically studied for morphologically rich languages such as Turkish. We compare Turkish document question answering across three chunking strategies (fixed-length, semantic, and layout-aware Docling), five embedding models, and two generator LLMs, over three documents with contrasting layouts. The fully crossed design yields 9,000 graded question-answer evaluations, each scored by an independent judge model, and component comparisons are tested by paired McNemar tests under Holm correction. Four findings follow. The chunking strategy determines how much the embedding choice matters: layout-aware chunking compresses the spread between the modern embedding models to about a point. The three leading embedding models are statistically indistinguishable, so language specialization yields no measurable retrieval advantage. The faster generator is not the more accurate one. And the preferred configuration depends on content type, since layout-aware chunking helps documents containing tables far more than prose. The best individual components therefore do not compose into the best complete configuration, which reaches 87.0%.

I. INTRODUCTION · II. RELATED WORK

The paper motivates a controlled study of Turkish RAG, where chunking, embedding, language morphology, and document layout can interact, and connects that study to open questions in retrieval and segmentation research.

I. INTRODUCTION

  • RAG combines retrieval with generation conditioned on retrieved passages,, so chunking and embedding choices made before generation affect answer quality, [4].Source: §I · p. 1
  • Turkish suffixation creates many inflected forms that mismatched subword vocabularies fragment inconsistently, while splitting tables across chunks makes their facts harder to retrieve.These morphological and layout challenges motivate evaluating Turkish documents containing both prose and tables.Source: §I · p. 1
  • Chunking governs how much embedding choice matters, the leading embedding models are statistically indistinguishable, and preferred chunking depends on content type.The paper identifies these as central findings of its Turkish document question-answering comparison.Source: §I · p. 1
  • The fully crossed comparison grades 9,000 question-answer pairs and reports fourteen planned component comparisons using paired significance tests under Holm correction.It evaluates chunking strategy, embedding model, and generator LLM on the same question set, with operational metrics alongside accuracy.Source: §I · p. 1

II. RELATED WORK

  • Retrieval moved from sparse lexical matching to dense query–passage embeddings compared by inner product or cosine similarity, with approximate nearest-neighbour indexing such as FAISS,.Sentence-BERT and LaBSE use sentence-level objectives suited to semantic similarity, while multilingual-e5 [13] and commercial services support broad multilingual representation.Source: §II · pp. 1–2
  • Turkish-specific models outperform multilingual models on some downstream tasks,, but whether that advantage transfers to retrieval ranking remains open.The paper directly examines whether Turkish language specialization yields a retrieval advantage.Source: §I · p. 1, §II · p. 2
  • Chunking research spans lexical-cohesion segmentation, supervised neural segmenters, fixed-length baselines, semantic segmentation, and layout-aware parsing such as Docling [18].The paper contributes a controlled Turkish comparison across documents combining prose and tabular content, with operational metrics and significance tests.Source: §II · p. 2

III. DATA

The study uses three Turkish documents spanning table-heavy, balanced, and text-heavy layouts, with the same chunking and embedding methods applied across them. A fully crossed design evaluates 30 pipeline configurations on 300 questions, yielding 9,000 graded evaluations.

III. DATA

  • Three documents represent table-heavy, balanced, and text-heavy Turkish content, isolating layout effects by applying identical chunking and embedding methods to each.The documents differ in the proportion of content presented as tables rather than flowing text.Source: §III · p. 2
  • 300 distinct questions comprise 105 table-grounded and 195 text-grounded questions, with each question labelled by whether its answer requires table information.Table-grounded questions number 80, 22, and 3 in the table-heavy, balanced, and text-heavy documents, respectively; the text-heavy figure is reported for completeness, not interpretation.Source: §III · p. 2
  • 9,000 graded evaluations result from crossing 3 chunking strategies, 5 embedding models, and 2 generators across 300 questions.The fully crossed design applies the same questions to every configuration, avoiding confounding from uneven question or document allocation and enabling paired tests.Source: §III · p. 2

IV. METHOD · A. RAG Pipeline · B. Chunking Strategies

Configurations share a controlled RAG pipeline and differ only in chunking, embedding, and generator; the chunking strategies range from fixed-length segments to topical and layout-aware segmentation.

A. RAG Pipeline

  • Each document is segmented, its chunks are embedded and stored in a FAISS index [8], and queries use the same embedding model.The selected chunking and embedding models determine segmentation and vector encoding.Source: A. RAG Pipeline · p. 2
  • At query time, the system retrieves the most similar chunks by cosine similarity and inserts them with the question into a fixed prompt template.The selected generator produces the answer, which is graded against the reference as described in Section IV-E.Source: A. RAG Pipeline · p. 2
  • The number of retrieved chunks, prompt template, and similarity metric remain constant, making the 3×5×2 design a controlled comparison.This holds retrieval and prompting settings constant so observed differences are attributable to the three factors under study.Source: A. RAG Pipeline · p. 2

B. Chunking Strategies

  • Fixed-length chunking splits text at a target character length regardless of content, serving as the baseline.It represents the strategy with no awareness of document structure.Source: B. Chunking Strategies · p. 2
  • Semantic chunking places boundaries where topical similarity between adjacent sentences drops and typically produces shorter chunks,.It is described as the practical descendant of lexical-cohesion segmentation.Source: B. Chunking Strategies · p. 2
  • Docling chunking parses document layout and keeps structurally coherent elements, particularly tables, together within larger chunks [18].It represents explicit layout awareness, compared with topical structure in semantic chunking and none in fixed-length chunking.Source: B. Chunking Strategies · p. 2

C. Embedding Models

The study evaluates five embedding models, from a lightweight subword baseline to Turkish-oriented and general multilingual or commercial encoders, in a shared RAG pipeline. This selection tests whether sentence encoders suit Turkish retrieval and whether Turkish specialization adds benefit.

C. Embedding Models

  • FastText is the lightweight subword baseline; TurkEmbed was trained on Turkish inference and sentence-similarity data, while Mursit-large [21] was retrieval-tuned on a Turkish-dominant corpus.The passage describes TurkEmbed's training data and Mursit-large's pretraining corpus orientation.Source: C. Embedding Models · p. 2
  • The RAG procedure embeds chunked documents for FAISS retrieval, then generates and judges answers; 30 configurations across 300 questions yield 9,000 verdicts.Chunking strategy, embedding model, and generator vary while retrieval depth, prompt template, similarity metric, and judge remain constant.Source: Figure 1 · p. 3, C. Embedding Models · p. 3
  • The other candidates are multilingual-e5-large [13], a large multilingual encoder, and text-embedding-3-small, a general-purpose commercial model accessed through a hosted API.Together, the five models span Turkish-oriented, multilingual, and general-purpose options.Source: C. Embedding Models · p. 3
  • All five models were evaluated in the identical pipeline, varying only the embedding step, to test whether sentence encoders suit Turkish retrieval and whether specialization adds benefit.Dense sentence encoders have largely replaced word-level representations for retrieval, motivating the comparison.Source: C. Embedding Models · p. 3

D. Generator LLMs · E. Evaluation Protocol and Operational Metrics

The study compares two generators under identical prompts and retrieved context, while evaluating answer accuracy with a third-party LLM judge and recording operational metrics relevant to production trade-offs.

D. Generator LLMs

  • GPT-4o mini and qwen-plus received the same template and retrieved context, isolating generation quality and latency from retrieval.Source: D. Generator LLMs · p. 3

E. Evaluation Protocol and Operational Metrics

F. Statistical Analysis · V. RESULTS

Results show that accuracy varies substantially by configuration: Docling performs best overall among the chunking strategies, while embedding and generator choices also affect accuracy and operational tradeoffs. Comparisons use paired tests with Holm–Bonferroni correction.

F. Statistical Analysis

  • Component comparisons use paired McNemar tests with continuity correction, and p-values for 14 planned comparisons receive Holm–Bonferroni adjustment.A likelihood-ratio test compares logistic regression models with and without a chunking-by-embedding interaction.Source: F. Statistical Analysis · p. 3

V. RESULTS

  • Across all 9,000 evaluations, overall accuracy is 68.10%, serving only as a reference because accuracy varies widely by configuration.Source: §V · p. 3
  • multilingual-e5-large reaches 80.00% accuracy, while the Turkish-oriented models and text-embedding-3-small range from 75.78% to 78.67%; FastText scores 27.56%.The four competitive models have chunk sizes of 1,836–1,909 characters and response times near 4.0 s, so accuracy differences are not explained by context length.Source: §V · p. 3, Table I · p. 3
  • GPT-4o mini is more accurate than qwen-plus (69.58% versus 66.62%), while qwen-plus is faster (2.60 s versus 5.62 s).The accuracy gap is statistically reliable; the speed-accuracy tradeoff may suit latency-sensitive deployments but not those where accuracy is the binding constraint.Source: §V · p. 3
  • Docling accuracy is 74.37% overall and 80.57% on table-grounded questions; semantic and fixed-length chunking score 65.80% and 64.13% overall.Docling's advantage is concentrated on table-grounded questions, consistent with keeping tables intact.Source: §V · pp. 3–4, Figure 2 · p. 4
  • Chunking reverses its relative ranking by question type: semantic beats fixed-length on table questions (71.90% versus 63.62%), while fixed-length slightly beats semantic on text questions (64.41% versus 62.51%).Docling chunks average 3,195 characters, compared with 1,451 for fixed-length and 1,019 for semantic chunking, increasing per-query context cost; mean response times are 3.72–4.34 s.Source: §V · p. 4

A. Significance of Component Differences

Ten of 14 planned paired comparisons remained significant after Holm correction. The top three embedding models and semantic versus fixed-length chunking were not reliably distinguishable, while chunking and generator effects were statistically reliable.

A. Significance of Component Differences

  • Multilingual-e5-large differed nonsignificantly from TurkEmbed (+1.33 points, pHolm = 0.323) and text-embedding-3-small (+1.50, 0.315); TurkEmbed and text-embedding-3-small were indistinguishable (+0.17, 0.900).Semantic and fixed-length chunking also could not be separated (+1.67, 0.211).Source: A. Significance of Component Differences · p. 4
  • All four dense-embedding-versus-FastText comparisons were significant (48.22 to 52.44 points, all pHolm < 0.001).The chunking and generator effects were statistically reliable.Source: A. Significance of Component Differences · p. 4
  • 10 of 14 planned paired comparisons remained significant after Holm correction, while the top-three embedding ranking should not be over-interpreted.Both Docling comparisons, the generator comparison, and all three comparisons against Mursit-large were significant.Source: Table II · p. 4, A. Significance of Component Differences · p. 4

B. Chunking–Embedding Interaction · C. Performance by Document Type

Chunking changes how much embedding choice matters, while Docling’s advantage depends on document structure. The document-type results also show that table-question performance varies by chunking strategy rather than reflecting an inherent difficulty of table-grounded questions.

B. Chunking–Embedding Interaction

  • The best embedding changes by strategy: Mursit-large leads with Docling, multilingual-e5-large with semantic, and TurkEmbed with fixed-length chunking.Among the four modern embeddings, chunking sensitivity ranges from 7.00 points for TurkEmbed to 15.00 for Mursit-large; FastText varies by 22.17 points.Source: B. Chunking–Embedding Interaction · p. 4
  • Under Docling, modern embeddings span 1.33 points (82.50–83.83), versus 7.67 under semantic and 7.17 under fixed-length chunking.The table reports accuracy for each of the fifteen chunking–embedding pairs, each over 600 evaluations.Source: Table III · p. 4, B. Chunking–Embedding Interaction · p. 4
  • Averaging over chunking obscures the within-strategy ranking: under structure-agnostic chunking, embedding choice costs up to 7.17 points, and Mursit-large falls fifteen points below its best case.Where segmentation preserves structure, a strong embedding can instead be chosen on cost, licensing, or latency grounds with little penalty.Source: B. Chunking–Embedding Interaction · p. 4

C. Performance by Document Type

  • Table questions score below text questions on the table-heavy document (73.58% versus 78.50%), but above them corpus-wide (72.03% versus 65.98%).The reversal is confined to fixed-length chunking: table questions score 61.50% versus 79.50% for text questions; under Docling and semantic, table-question scores are 82.62 versus 81.00 and 76.62 versus 75.00, respectively.Source: C. Performance by Document Type · p. 4
  • The balanced document is hardest at 57.73%, compared with 74.57% and 72.00% for the other two document types.Table IV reports accuracy by document type.Source: C. Performance by Document Type · p. 4, Table IV · p. 5
  • Docling’s advantage depends on document type: strategies differ by at most 2.1 points on the text-heavy document, while Docling beats the best alternative by 6.0 points on the table-heavy and 12.3 on the balanced document.The table-heavy comparisons are 82.30 versus 76.30, and the balanced-document comparisons are 67.70 versus 55.40.Source: C. Performance by Document Type · p. 4, Table IV · p. 5

D. Best Complete Configurations

The best complete configuration reaches 87.00%, but its status as the single best is indicative rather than established. The preferred chunking and latency–accuracy trade-off vary by question content.

  • Docling appears in eight of the ten best overall configurations, while embedding model and generator vary.The sixth-to-tenth-ranked qwen-plus configurations score 80.67–81.33% at 2.04–3.63 s, within 6.5 points of the best at roughly half its latency.Source: D. Best Complete Configurations · p. 5
  • 93.33%: Table-question leaders are dominated by Docling and semantic chunking, and several qwen-plus configurations rank among the strongest, narrowing the latency–accuracy gap.Source: D. Best Complete Configurations · p. 5
  • 84.10%: The best text-question configuration reaches this accuracy, and fixed-length chunking enters the top ten for text but not table questions.This is consistent with fixed-length chunking being less damaging for flowing text than tables, where a boundary can separate a value from its row or column header.Source: D. Best Complete Configurations · p. 5
  • 87.00%: Docling chunking with Mursit-large and GPT-4o mini is best overall, compared with the 68.10% average.Each configuration is evaluated on 300 questions, giving accuracy a 95% confidence interval of roughly ±4 points; selection across thirty configurations biases the maximum upward, so the winner is indicative rather than established.Source: Table V · p. 5, D. Best Complete Configurations · p. 5

VI. DISCUSSION · A. Does Language Specialization Help Retrieval?

The discussion finds that chunking matters most for table-heavy content, while strong embedding models show no measurable retrieval advantage from Turkish specialization. Embedding differences also depend on chunking, and weak non-contextual embeddings perform poorly.

VI. DISCUSSION

  • Docling improves table-question accuracy by 17.0 points over fixed-length and 8.7 points over semantic chunking, while text-heavy content differs by at most 2.1 points.Both table comparisons survive multiple-testing correction; Docling retrieves whole tables rather than fragments, but its larger chunks consume more context tokens.Source: §VI · p. 5
  • Under Docling, four embedding models span 1.33 points, versus 7.17 points under fixed-length chunking, where Mursit-large falls fifteen points below its own best case.The four modern models differ by about four points overall, and the top three cannot be statistically separated; avoiding an inadequate model matters more than choosing among strong ones.Source: §VI · p. 5

A. Does Language Specialization Help Retrieval

  • Language specialization shows no measurable retrieval advantage: multilingual-e5-large is not significantly ahead of TurkEmbed (+1.33, pHolm = 0.323) or text-embedding-3-small (+1.50, 0.315).Those latter models are also indistinguishable (+0.17, 0.900); only Mursit-large is reliably behind, so cost, latency, or licensing can guide the choice among strong models.Source: A. Does Language Specialization Help… · p. 5
  • FastText scores 27.56% and trails every dense embedding model by roughly fifty points at pHolm < 0.001, consistent with the limits of word-level representations lacking contextual composition.Because there is no non-agglutinative control language, the gap cannot be attributed specifically to Turkish morphology; large multilingual models may instead succeed through subword coverage and contextual composition, consistent with tokenization work.Source: A. Does Language Specialization Help… · p. 5
  • Turkish-specific pretraining benefits classification,, but that advantage does not transfer automatically to retrieval.Source: A. Does Language Specialization Help… · p. 6

B. Generator Choice and Trade-offs · VII. CONCLUSION

Across Turkish document question answering, the highest-scoring pipeline combined Docling, Mursit-large, and GPT-4o mini, despite those not being the best marginal components. Chunking changes how much embedding choice matters, while generator choice presents an accuracy–latency trade-off.

B. Generator Choice and Trade-offs

  • qwen-plus answered in under half GPT-4o mini’s time but gave up 2.96 points, a statistically reliable difference (pHolm < 0.001).Several qwen-plus configurations nevertheless ranked among the strongest on table questions; generator choice depends on whether latency or accuracy takes priority.Source: B. Generator Choice and Trade-offs · p. 6

VII. CONCLUSION

  • 87.00% was the highest-scoring pipeline, pairing Docling with Mursit-large and GPT-4o mini, about nineteen points above average despite Mursit-large ranking last among four modern embedding models marginally.The best individual components were Docling chunking, multilingual-e5-large, and GPT-4o mini, but they did not compose into the best complete configuration.Source: §VII · p. 6
  • Across three chunking methods, five embedding models, and two generators, the fully crossed design produced 9,000 graded question-answer pairs from documents with three contrasting layout types.Source: §VII · p. 6
  • Layout-aware chunking compressed the spread between strong embedding models to little more than one point, making embedding rankings unstable and the leading models indistinguishable under that strategy.Source: §VII · p. 6
  • Paired tests found chunking and generator effects reliable but the ranking of the three leading embedding models unreliable, indicating no measurable retrieval advantage from language specialization on this corpus.Source: §VII · p. 6
  • The preferred configuration depends on content type, and accuracy should be weighed against cost and latency; for predominantly prose documents, the penalty for fixed-length chunking is modest.For mixed content on this corpus, the conclusion favors layout-aware chunking with any of the three leading embedding models.Source: §VII · p. 6, B. Generator Choice and Trade-offs · p. 6
  • Future work includes broader document collections, validating automatic grades against human annotations on a stratified sample, and hybrid chunking that routes tabular and prose content differently.Source: §VII · p. 6

Related papers

Loading 2608.26192v1…