Source-linked AI summary
Comparing Chunking and Embedding Strategies for Turkish RAG Systems
Mustafa Sertaç Türkel, Fatma Nur Korkmaz, Ahmet Tuğrul Bayrak
TL;DR
Turkish RAG research lacks systematic evidence on how chunking and embedding choices affect document question answering, despite challenges from morphology and mixed layouts. The paper fully crosses three chunkers, five embedding models, and two generators across three documents, producing 9,000 judged evaluations. Layout-aware chunking is especially valuable for tables, strong embedding models are statistically similar, generator speed trades off against accuracy, and the best components do not form the best pipeline, which reaches 87.00%.
Problem
Evidence is limited on how chunking and embedding choices affect RAG retrieval for morphologically rich Turkish documents with mixed prose and tabular layouts.
Method
The study fully crosses three chunking strategies, five embedding models, and two generators across 300 questions and three documents, yielding 9,000 graded evaluations.
Results
Layout-aware chunking improves table-question accuracy by 17.0 points over fixed-length and 8.7 points over semantic chunking, while the top three embedding models cannot be statistically separated.
Takeaways & Limitations
The preferred RAG configuration depends on content type and component interactions: the highest-scoring pipeline pairs Docling, Mursit-large, and GPT-4o mini at 87.00%.
Abstract
from arXiv · showhide
How documents are segmented into retrievable chunks and how those chunks are embedded strongly affect Retrieval-Augmented Generation (RAG) quality, yet neither has been systematically studied for morphologically rich languages such as Turkish. We compare Turkish document question answering across three chunking strategies (fixed-length, semantic, and layout-aware Docling), five embedding models, and two generator LLMs, over three documents with contrasting layouts. The fully crossed design yields 9,000 graded question-answer evaluations, each scored by an independent judge model, and component comparisons are tested by paired McNemar tests under Holm correction. Four findings follow. The chunking strategy determines how much the embedding choice matters: layout-aware chunking compresses the spread between the modern embedding models to about a point. The three leading embedding models are statistically indistinguishable, so language specialization yields no measurable retrieval advantage. The faster generator is not the more accurate one. And the preferred configuration depends on content type, since layout-aware chunking helps documents containing tables far more than prose. The best individual components therefore do not compose into the best complete configuration, which reaches 87.0%.
I. INTRODUCTION
RAG quality depends on how documents are chunked and embedded before generation, and these choices interact with Turkish morphology and document layout. The paper therefore conducts a controlled comparison of chunking, embedding, and generator choices for Turkish document question answering.
- Motivation: RAG quality depends on chunking and embedding decisions made before the generator answers from retrieved context.Chunking determines retrievable units, while embedding maps chunks and queries into a shared vector space for similarity search.
- Motivation: Turkish morphology creates retrieval challenges because suffixation produces many inflected forms and mismatched subword vocabularies can fragment them inconsistently.An embedding model inheriting this mismatch may place semantically similar passages far apart.
- Motivation: Document layout adds a separate retrieval challenge when chunking splits tables across boundaries.Institutional documents mix prose with tables, and fragmented tables can make their facts harder to retrieve regardless of the embedding.
- Contributions: The study compares chunking strategy, embedding model, and generator LLM in a fully crossed design of 9,000 graded question-answer pairs.The harness reports operational metrics alongside accuracy and tests component comparisons with paired significance tests under Holm correction.
- Related work: Prior work leaves open whether Turkish language specialization transfers from classification to retrieval, while chunking has received less systematic attention.The paper addresses both questions through a controlled comparison of Turkish documents mixing prose and tabular content.
III. DATA
The dataset uses three Turkish institutional documents with contrasting proportions of tables and flowing text, and evaluates the same question set across every pipeline configuration. This fully crossed structure isolates layout effects while enabling separate table- and text-grounded analyses.
- Corpus: Three institutional Turkish documents represent table-heavy, balanced, and text-heavy layouts.The same chunking and embedding methods are applied to all documents to isolate the effect of layout.
- Questions: 300 distinct questions are distributed as 100 questions per document and labelled by whether their answers require table information.The labels support separate analyses of table-grounded and text-grounded questions.
- Questions: 105 questions are table-grounded and 195 are text-grounded, with table-grounded counts of 80, 22, and 3 across the three document types.The uneven distribution intentionally samples three densities of tabular content.
- Limitations: Only 3 table-grounded questions occur in the text-heavy document, so that figure is reported for completeness rather than interpretation.This is the dataset’s clearest boundary for interpreting table-question results by document type.
- Experimental design: 9,000 graded evaluations result from crossing 3 chunking strategies, 5 embedding models, and 2 generators over 300 questions.Each configuration uses the same questions, enabling paired component comparisons without uneven allocation confounding.
IV. METHOD
Each RAG configuration segments documents, embeds chunks into a FAISS index, retrieves similar chunks for each question, and generates an answer from a fixed prompt. The comparison varies chunking strategy while holding the rest of the retrieval-and-generation pipeline constant.
- Pipeline: Each document is segmented, its chunks are embedded and indexed in FAISS, and cosine similarity retrieves the most similar chunks for each question.The question is embedded with the same model used for chunks before retrieval.
- Pipeline: The retrieved chunks are inserted into a fixed prompt template containing the question before answer generation.The selected generator then produces the response from that retrieved context.
- Chunking strategies: Fixed-length chunking uses target character lengths regardless of content, whereas semantic chunking places boundaries when adjacent-sentence similarity drops.Semantic chunking is the practical descendant of lexical-cohesion segmentation and typically produces shorter chunks.
- Chunking strategies: Docling is layout-aware: it parses document structure and keeps structurally coherent elements, especially tables, together within chunks.This design directly preserves table structure during retrieval.
C. Embedding Models
The study evaluates five embedding models and two generators in the same RAG pipeline, varying only the selected component under comparison. It includes Turkish-oriented, multilingual, commercial, and classical subword-based representations while holding retrieval and prompting settings constant.
- Embedding models: FastText is a classical subword-based baseline, while TurkEmbed and Mursit-large are publicly available models oriented toward Turkish.TurkEmbed was trained on Turkish inference and sentence-similarity data, and Mursit-large was retrieval-tuned on Turkish-dominant pretraining.
- Experimental controls: The architecture varies only chunking strategy, embedding model, and generator while holding retrieval depth, prompt template, similarity metric, and judge constant.Repeating the pipeline across 30 configurations and 300 questions produces 9,000 verdicts.
- Embedding models: multilingual-e5-large is a large multilingual encoder, and text-embedding-3-small is a general-purpose commercial embedding service.All five embedding models are evaluated in the identical pipeline, so only the embedding step varies.
- Embedding models: The model selection tests whether dense sentence encoders outperform word-level representations for Turkish retrieval and whether language-specialized models add benefit.The comparison targets retrieval ranking rather than classification labels.
- Generator models: GPT-4o mini and qwen-plus are compared as compact commercial generators receiving the same prompt template and retrieved context.Their comparison reflects generation quality and latency rather than retrieval differences.
E. Evaluation Protocol and Operational Metrics
The study uses binary LLM judging against reference answers and paired statistical tests, while tracking chunk, answer, and latency metrics for operational tradeoffs.
- Evaluation Protocol: Answers were graded as correct or incorrect by an independent Llama 3.3 70B Instruct judge using the question, reference, and candidate answer.The judge came from a third provider, but its verdicts were not validated against human annotation.
- Operational Metrics: Three operational metrics measure retrieved-chunk size, generated-answer length, and end-to-end response time.Chunk size serves as a proxy for context-token consumption and per-query cost.
- Operational Metrics: A slightly more accurate configuration may not be preferred if it is slower or more expensive.The operational measures support quality–cost–latency decisions for deployment.
- Statistical Analysis: Paired McNemar tests with continuity correction compare factor levels because every configuration answers the same 300 questions.Holm–Bonferroni adjustment accounts for the fourteen planned comparisons.
V. RESULTS
Results show substantial variation by embedding, generator, and chunking strategy. Layout-aware Docling performs best overall and especially on table-grounded questions, while embedding and generator choices involve distinct accuracy tradeoffs.
- Embedding Models: 80.00% overall accuracy makes multilingual-e5-large the strongest embedding model, while FastText reaches 27.56%.The other competitive models range from 75.78% to 78.67%.
- Generators: 69.58% versus 66.62% gives GPT-4o mini higher accuracy than qwen-plus.The accuracy difference is statistically reliable despite being small in absolute terms.
- Generators: 2.60 s versus 5.62 s makes qwen-plus faster than GPT-4o mini, trading about three accuracy points for latency.This tradeoff may suit latency-sensitive deployments but not settings where accuracy is binding.
- Chunking Strategies: 74.37% overall accuracy makes Docling the strongest chunking strategy.Semantic and fixed-length chunking reach 65.80% and 64.13%, respectively.
- Chunking Strategies: 80.57% accuracy on table-grounded questions is Docling’s largest advantage, consistent with preserving tables intact.Its chunks average 3,195 characters, increasing per-query context cost relative to the alternatives.
A. Significance of Component Differences
Paired comparisons show that most planned component differences are statistically significant, but the three leading embedding models cannot be reliably distinguished from one another after Holm correction.
- Multiple-Comparison Results: 10 of 14 planned comparisons remain significant after Holm correction.All four dense-embedding comparisons against FastText are significant, with differences from 48.22 to 52.44 points and pHolm < 0.001.
- Embedding Models: +1.33 points separates multilingual-e5-large from TurkEmbed without significance after correction (pHolm = 0.323).The comparison does not support treating the two models as reliably different.
- Embedding Models: +1.50 points separates multilingual-e5-large from text-embedding-3-small without significance after correction (pHolm = 0.315).The observed ranking should not be over-interpreted.
- Embedding Models: +0.17 points separates TurkEmbed and text-embedding-3-small without significance after correction (pHolm = 0.900).The three leading embedding models are statistically indistinguishable in these comparisons.
B. Chunking–Embedding Interaction
Chunking changes the apparent importance of embedding choice: layout-aware Docling makes modern embeddings perform nearly alike, while structure-agnostic methods widen their spread. Its advantage is concentrated in documents containing tables, with little effect on text-heavy content.
- Embedding sensitivity: Under Docling, the four modern embedding models span only 1.33 points, compared with 7.67 and 7.17 points under semantic and fixed-length chunking.The embedding spread therefore depends strongly on how document structure is segmented.
- Document dependence: On the text-heavy document, chunking strategies differ by at most 2.1 points, but Docling leads the best alternative by 6.0 and 12.3 points on table-heavy and balanced documents.Layout-aware segmentation matters most when tables are present.
- Document dependence: In the table-heavy document, table questions score 73.58% versus 78.50% for text questions overall, because fixed-length chunking reverses this pattern.Docling and semantic chunking instead give table questions slightly higher scores than text questions there.
D. Best Complete Configurations
The best complete pipeline combines Docling chunking with Mursit-large and GPT-4o mini, although rankings vary by question type and individual configuration estimates remain uncertain. Faster qwen-plus alternatives stay relatively competitive, especially on table-grounded questions.
- Overall ranking: 87.00% is achieved by Docling with Mursit-large and GPT-4o mini, about nineteen points above the 68.10% average.This single-configuration result is indicative rather than established because each configuration uses 300 questions and the maximum over thirty configurations is selection-biased.
- Overall ranking: The five configurations ranked sixth to tenth use qwen-plus, span 80.67–81.33% accuracy, and run in 2.04–3.63 seconds.They remain within 6.5 points of the best configuration at roughly half its latency.
- Question-type rankings: On table questions, leading configurations reach 93.33% and are dominated by Docling and semantic chunking, while several qwen-plus configurations rank among the strongest.
- Question-type rankings: On text questions, the best configuration reaches 84.10%, and a fixed-length configuration enters the ten best there but not among the table-question leaders.This pattern is consistent with fixed-length chunking being less damaging for flowing text than for tables.
VI. DISCUSSION
The discussion finds that chunking is most consequential for tabular content, while strong modern embeddings are broadly interchangeable and weak embeddings are costly. Turkish-specific embedding specialization does not improve retrieval measurably, although morphology-related weaknesses appear in FastText.
- Chunking and layout: Docling improves table-question accuracy by 17.0 points over fixed-length and by 8.7 points over semantic chunking, with both comparisons surviving multiple-testing correction.The gain comes from retrieving whole tables rather than fragments, but larger chunks increase context-token consumption; prose-heavy documents gain little.
- Embedding quality: The four modern embedding models differ by about four points overall, while FastText reaches only 27.56%; the top three modern models cannot be statistically separated.Under Docling their spread is 1.33 points, whereas fixed-length chunking expands it to 7.17 points.
- Language specialization: Language specialization confers no measurable retrieval advantage: multilingual-e5-large is not significantly ahead of TurkEmbed or text-embedding-3-small.Only Mursit-large is reliably behind among the compared modern models.
- Language specialization: FastText falls to 27.56% and is separated from every dense embedding model by roughly fifty points at pHolm < 0.001.This is consistent with its word-level subword-n-gram representations lacking contextual morphological composition.
- Language specialization: Turkish-specific pretraining helps classification, but that benefit does not transfer automatically to retrieval.
B. Generator Choice and Trade-offs
The generator choice presents a clear accuracy–latency trade-off within the fully crossed comparison. qwen-plus is substantially faster, whereas GPT-4o mini is more accurate, and the leading retrieval settings remain stable across generators.
- Accuracy–latency trade-off: qwen-plus answers in under half the time of GPT-4o mini while giving up 2.96 accuracy points, a statistically reliable difference (pHolm < 0.001).Several qwen-plus configurations nevertheless rank among the strongest on table questions.
- Deployment choice: Generator selection can prioritize accuracy or latency largely independently of chunking and embedding, whose leading settings are stable across both generators.
- Accuracy–latency trade-off: The highest-scoring pipeline pairs Docling with Mursit-large and GPT-4o mini at 87.00%, despite Mursit-large ranking last among the four modern embeddings marginally.This result reflects interaction between retrieval-side factors rather than independently selecting the best marginal components.