PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading a Legal Question Word by Word: Embedding Trajectories of 2,144 Vietnamese Legal Headlines

Tran Minh Quan

arXiv:2609.08372v1cs.CL

TL;DR

The paper asks how a dense retriever’s representation develops when a legal question is read one word at a time, rather than encoded only as a complete string. It evaluates prefix trajectories across Vietnamese legal headlines, sub-questions, and answers, finding that retrieval usually locks onto the gold article after the topic-bearing early words and is best characterized as a context-modulated additive walk.

  • Problem

    Prefix-level evidence is limited on when a decoder retriever reaches the correct passage and which words determine retrieval, despite queries arriving sequentially.

  • Method

    The study encodes 2,144 held-out Vietnamese legal headlines word by word with four decoder embedders, ranking 65,444 prefixes against 20,034 articles and analyzing sub-question, answer, rank, cosine, and step trajectories.

  • Results

    Gold articles reach rank 1 after a median of 6–7 content words across encoders, while the first sub-question largely determines retrieval and the walk is a noisy, context-modulated additive process.

  • Takeaways & Limitations

    Retrieval can often begin after the topic noun phrase, with the first sub-question carrying most of the headline’s retrieval signal and later interrogative material usually confirming rather than changing it.

  • Takeaways & Limitations

    The gold label measures self-retrieval of one publisher’s editorial headline–article pairs rather than general question answering, and Vietnamese word boundaries are syllable spaces.

Abstract

from arXiv · show

A dense retriever encodes a question as one vector, but the question arrives one word at a time. We read 2,144 held-out headlines from Thu Vien Phap Luat (Vietnamese legal library) word by word with Nemotron-3-Embed 8B/1B and Qwen3-Embedding 8B/0.6B, encoding 65,444 prefixes against 20,034 articles, plus every prefix of 3,438 sub-questions from 1,112 multi-question headlines and of 168 answers. (i) The gold article becomes rank 1 after a median of 6-7 content words in every encoder, before the interrogative frame is read, and stays there to the end in 78-85% of cases. (ii) In a multi-question headline the lock is inside the first sub-question 94-98% of the time; the second leaves rank unchanged in 89-95%; encoded alone, the second reaches rank 1 in 42-58% vs 91-96% for the first, at the same lock word (95-97% identical). (iii) Numbers, dates and instrument identifiers move the embedding twice as far as content words and four times as far as interrogative words; 72-78% of steps move toward the gold article, and the closing interrogative frame moves against that direction in 95-99% of headlines. (iv) Rank/cosine clustering yields six archetypes (instant, typical, unstable, late, never-locking) that differ by legal area and form (chi-squared p < 1e-8): real-estate and litigation headlines never lock on a number; environmental and accounting headlines do so a third of the time. (v) An answer read word by word retrieves its article after 8-16 words and addresses the sub-questions in order asked in 83-89% of cases. (vi) A word's step keeps a consistent direction across headlines (cosine 0.25-0.33; 0.44-0.60 for numbers); a preceding question rotates that step by about 60 degrees and a greeting by about 30 degrees; steps shrink as i^{-0.8}; and a two-question headline is within 12-17 degrees of a linear mix of its two questions. We call this a context-modulated additive walk.

1 INTRODUCTION

This paper measures how decoder-based retrievers build an answer-relevant representation as a legal question unfolds word by word. It addresses a corpus-scale gap in trajectory analysis by separating topic, interrogative, compound-question, and noninformative words in Vietnamese legal headlines.

  • Motivation: Decoder retrievers usually encode a query as one vector, leaving the prefix-level path to the gold passage empirically unmeasured.The paper asks when the vector reaches the gold article, whether it stays there, and which words drive the transition.
  • Motivation: Vietnamese legal headlines provide long, structurally varied queries in which topic phrases, interrogative frames, second questions, greetings, and attributions can be separated.The corpus has a mean headline length of 32 words, and 52% contain two or more questions.
  • Prior work and novelty: The study extends embedding visualization from static lexical or latent-space views to word-level retrieval trajectories.It uses rank in the article index as the primary signal rather than only projected proximity.

3 DATA, MODELS AND THE PREFIX EXPERI MENT

The experiment encodes prefixes of held-out Vietnamese legal headlines and independently split sub-questions, ranks them against a 20,034-article index, and records retrieval trajectories and answer-prefix behavior. It also visualizes trajectories using principal-component projections of the gold article, competitors, and path.

  • Data: The corpus contains 20,034 articles and 2,144 held-out headlines, with each headline paired to its own article as the gold passage.A rule-based splitter removes greetings and trailing attributions and separates sub-questions at question marks.
  • Models: Four open-weight decoder embedders are tested: Nemotron-3-Embed-8B/1B with mean pooling and Qwen3-Embedding-8B/0.6B with last-token pooling.The models use query prompts for headline prefixes and passage prompts for answer prefixes.
  • Prefix experiment: The study forms 65,444 headline prefixes, ranks each against 20,034 passage vectors, and records rank, cosine, competitors, and prefix-step measures.It also encodes 61,568 standalone sub-question prefixes and 15,839 answer prefixes sampled every eight words.
  • Prefix experiment: The insight point is the first prefix where the gold article reaches rank 1, while a hold requires rank 1 to persist through the final word.Trajectories are projected using principal components of the gold vector, nearest final-prefix passages, and the path itself.

4 WHERE A QUESTION LOCKS ON ITS ANSWER

Gold articles usually become rank 1 after only a short content prefix, before the interrogative frame, and generally remain there. Compound-question results show that retrieval is principally determined by the first sub-question, while the second usually leaves the rank unchanged.

  • Insight point: Half of headlines reach gold rank 1 after six content words with Nemotron-3-Embed-8B and seven with the other three encoders.The 90th percentile is 13–16 words, and only 2.0–4.3% never reach rank 1.
  • Insight point: 84.5% of Nemotron-3-Embed-8B paths hold rank 1 through the final word, compared with 78.4–81.7% for the other encoders.Final rank 1 is reached by 92.5–96.3% of headlines.
  • Compound questions: The gold rank remains unchanged from the first sub-question’s end through the full headline for 95.1% of Nemotron-3-Embed-8B headlines and 89.1–92.5% for the other encoders.The second question helps in 3.3–8.3% and hurts in 1.6–3.9%.
  • Compound questions: Standalone first sub-questions reach rank 1 in 90.5–95.9% of cases, whereas second sub-questions reach it in 41.7–57.9%.Among Nemotron-3-Embed-8B two-question headlines, both halves retrieve the article alone in 54.9%, while only the first does so in 41.5%.

6 WHAT MOVES THE VECTOR

Word classes move the embedding by very different amounts: digits produce the largest shifts, while interrogative and attribution words produce the smallest. Despite tortuous paths, most steps point toward the gold article, whereas the interrogative frame reverses direction after retrieval has already locked.

  • 0.465 is the mean step for digit-containing words in Nemotron-3-Embed-8B, versus 0.245 for ordinary content words and 0.115 for interrogative or function words.For Qwen3-Embedding-8B, the corresponding values are 0.394, 0.219 and 0.109.
  • 72–78% of steps have a positive component toward the gold article across the four encoders.The encoder-specific values range from 72.1% to 77.9%.
  • The path length is 5.3–6.0 times the net displacement, making retrieval trajectories substantially more tortuous than their endpoint movement.Nemotron-3-Embed-8B has 1.3 direction reversals per headline, compared with 3.0–4.5 for the other reported models.
  • The interrogative frame makes a U-turn after the insight point, moving against the accumulated direction toward the gold article.

7 ADDITIVE OR AUTOREGRESSIVE? WHAT KIND OF WALK THIS IS

The prefix trajectory is neither an independent sum of fixed word steps nor an unconstrained rewrite: word directions persist, context rotates them, and step sizes decay with position. Across pooling architectures, these effects make the walk approximately additive but context-modulated.

  • The study tests whether prefix trajectories are independent walks with word-specific steps or autoregressive processes that rewrite vectors using prior context.
  • Context dependence grows with preceding length: the mean cosine falls from 0.59 for first questions of at most six words to 0.47 when they exceed fifteen.For Nemotron-3-Embed-8B, yes/no second questions show cosine 0.54, while definitions show 0.43.
  • Two-question headline vectors are evaluated against the first-question vector and against the best linear mixture of the two standalone question vectors.Figure 6d uses 922 two-question headlines to compare these alternatives.
  • Content-word steps shrink with position, from 0.73–0.89 at the second word to 0.09–0.13 at the thirtieth, with fitted slopes of −0.72 to −0.85.The decline occurs in both mean-pooled Nemotron and last-token Qwen models, so it is not attributed to pooling alone.
  • A word’s step retains a word-linked direction, while consecutive steps are uncorrelated in Nemotron-3-Embed-8B and mildly anti-correlated in Qwen.Mean cosine is 0.00 for Nemotron-3-Embed-8B and −0.05 to −0.09 for Qwen, with 63–68% of turns exceeding 90°.

8 SIX ARCHETYPES, AND WHERE THEY LIVE

Clustering produces six trajectory archetypes whose prevalence and decisive words vary across legal areas. Most lock near the first content words, but unstable, late, and never-locking paths form distinct exceptions.

  • Archetypes: Six archetypes emerge from clustered rank, cosine, and step trajectories.The clustering uses resampled curves and k-means with k = 6.
  • Archetypes: 30% of headlines are instant locks, reaching rank 1 at content word 4; typical clusters lock at words 7–8 and hold in 87–88% of cases.The instant cluster begins near rank 300, while typical clusters begin near rank 3,000–5,000.
  • Archetypes: 13% are unstable locks, arriving at word 6 but holding only 67% of the time; 6% are late locks, and 1% never lock.Unstable paths average 0.35 exits per headline, while 82% of never-locking headlines never reach rank 1.
  • Where they live: Number-based insight words range from 0% in real estate and litigation procedure to 32% in natural resources and environment.Accounting and audit reaches 29%, while criminal liability reaches 21%.

9 THE ANSWER READ WORD BY WORD

Answer prefixes retrieve their source article early, while later answer content determines where similarity to the headline peaks. Compound answers usually address sub-questions in the order asked.

  • Retrieval: The article reaches rank 1 after a median of 8 answer words, or 16 words with Nemotron‑3‑Embed‑1B.The answer’s restated heading supplies the retrieval signal.
  • Similarity position: Headline similarity peaks at 14–27% of document answers, 63–75% of yes/no answers, and 51–71% of sanction answers.Document answers name the template early; yes/no and sanction answers state the verdict or fine near the conclusion.
  • Question order: 89.3% of multi-question answers address their first sub-question before the second for both Nemotron models.The corresponding rates are 85.7% for Qwen3‑Embedding‑8B and 83.3% for Qwen3‑Embedding‑0.6B.
  • Question order: The first and second answer-sub-question peaks are separated by a median gap of 35–54% of the answer.This gap accompanies the ordering pattern across the four encoders.

10 DO THE ENCODERS AGREE?

The encoders agree on aggregate insight-point distributions more than on the exact word that triggers retrieval. Exact agreement is uncommon, while near agreement is substantially higher.

  • Cross-encoder agreement: All four encoders identify the same lock word in 16% of headlines and a lock within one word in 37%.These figures summarize agreement across all four models.
  • Cross-encoder agreement: Pairwise same-word agreement ranges from 34% to 42.9%, while within-one-word agreement ranges from 58% to 66.5%.The highest pairwise values are 42.9% exact and 66.5% within one word.

11 IMPLICATIONS

The results support early, prefix-based retrieval and prioritizing the first sub-question, while also warning that numeral removal and word-only caching can discard context-sensitive signals.

  • Retrieval timing: Half of headlines resolve after six content words, and 90% resolve after 13–16 words.The authors propose probing once the topic noun phrase is complete and treating the interrogative frame as confirmation.
  • Compound questions: The first sub-question alone reproduces headline retrieval in 91–96% of cases, whereas the second alone reaches rank 1 in fewer than 60%.The proposed workflow retrieves for the first question, then re-queries for the second with the first topic prepended.
  • Numbers: A date or instrument number moves the vector twice as far as other words and decides retrieval in a third of environmental and accounting headlines.The authors warn that numeral normalization or removal can remove this signal.
  • Context dependence: Context-modulated steps make word-keyed caching unsuitable because the step for a word changes after another question.The paper specifically contrasts the step of mẫu (template) after one question with its step after another.

12 LIMITATIONS

The study’s scope and interpretation are bounded by self-retrieval of one publisher’s editorial headlines, rule-based preprocessing, sampled answer walks, and prompt- and projection-dependent analyses.

  • Scope: The gold label pairs each headline with its own article from one publisher, so rank measures editorial-headline self-retrieval rather than general question answering.The corpus and gold-label design constrain what retrieval performance represents.
  • Preprocessing: Word boundaries are syllable spaces, making Vietnamese compounds count as two or three reading steps.The splitter and sub-question forms are rule-based, while interrogative forms were validated on 60 headlines and treated as covariates.
  • Answer analysis: Answer walks use a stride of eight words and cover 168 answers, rather than every answer prefix in the corpus.This sampling choice limits the granularity and coverage of answer-trajectory results.
  • Visualization: The 2-D planes are per-panel PCA projections, so distances between panels are not comparable and apparent star-to-path separation can be a projection artefact.A distant-looking final point does not contradict rank 1 when the final rank is 1.
  • Interpretation: The additivity tests measure departure from a linear mix of separately encoded prefixes and sub-questions, not the network’s internal mechanism.The tests characterize observed vector geometry without identifying how the model computes it.

A GALLERIES

The galleries visualize word-by-word retrieval paths across headline forms and legal areas, highlighting which words reach rank 1 and how trajectories vary by case.

  • Gallery design: The gallery set contains trajectories for every question form and legal area, using six headlines per form where available.Each panel draws from the full vectors of all 2,144 headlines, with three single- and three multi-question examples where available.
  • Headline forms: Document-request headlines lock on the template name, while mẫu and tải add nothing.The figure isolates the template term as the retrieval-defining part of this form.
  • Headline forms: Amount/time headlines place the requested quantity in the frame, but the retrieval lock occurs on the subject.This contrasts the semantic request with the word that determines article proximity.
  • Headline forms: The sanction form is described as the most stable, although one example begins with Cho tôi hỏi.The opening greeting is part of the displayed trajectory rather than the defining retrieval cue.
  • Headline forms: A procedure headline can lock late: one example needs twelve content words before the procedure phrase becomes specific enough.The gallery uses this case to show delayed specificity.
  • Headline forms: Non-legal utility headlines feature dates and lookup tables, with the number serving as the lock.The figure presents numeric identifiers as the decisive trajectory point in these examples.
Loading 2609.08372v1…