PaperlayerView exact revision on arXiv

Source-linked AI summary

Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS

Junchuan Zhao, Chenglin Xu, Wei Zeng, Haoyang Li, Yiwen Guo, Ye Wang

arXiv:2610.11437v1cs.SDeess.AS

TL;DR

Instruction-based TTS lacks a direct way to edit a reference voice while also specifying delivery changes across text segments. EDICT uses an edited acoustic reference as a shared voice anchor and reconstructs the KV cache at segment boundaries; evaluations report improved timbre editing and a favorable balance among local adherence, speaker consistency, and transitions.

  • Problem

    TTS interfaces based on voice cloning or text descriptions do not directly let users modify a reference voice, while utterance-level instructions leave delivery changes across segments underspecified.

    Source: §1 · pp. 1–2

  • Method

    EDICT edits a source voice into a shared codec reference and rebuilds the KV cache at instruction boundaries using the updated conditions and selected acoustic context.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Objective evaluations, listening tests, and ablations show improved timbre-edit accuracy and target-voice matching, with a favorable balance between local instruction adherence, voice consistency, and natural transitions.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The evaluations treat global timbre editing and local instruction control as complementary forms of control, including their composition.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Instruction-based text-to-speech (TTS) offers control over voice characteristics and speech expression through interfaces including voice cloning and text-based voice design. Voice cloning reproduces a reference voice, whereas text-based voice design creates a voice from a natural-language description. However, neither interface directly enables users to modify the timbre of a given reference and synthesize speech with the modified voice. Meanwhile, utterance-level expressive instructions leave changes across individual text segments underspecified. We introduce \textbf{EDICT}, a framework that unifies global timbre editing and local expressive control by using an edited acoustic reference to anchor voice identity across segments. To enable synthesis with an instruction-edited voice, EDICT combines reference audio with structured timbre edits to generate an edited reference in codec-token space. This representation serves as a shared voice anchor for a frozen TTS backbone, allowing segment-specific natural-language instructions to guide expression. To accommodate instruction changes while supporting acoustic continuity, EDICT rebuilds the KV cache at each segment boundary, refreshing instruction conditioning while retaining bounded acoustic context from previously generated speech. Evaluations on our proposed TimbreEdit-Bench and IntraTTS-Bench demonstrate improved timbre editing and a favorable balance between local instruction adherence, speaker consistency, and transition quality. Audio demos are available.

1 INTRODUCTION

EDICT addresses TTS requests that need a reference voice’s timbre to remain consistent while delivery changes across text segments. It combines reference-conditioned timbre editing with segment-level instruction control and evaluates both through two proposed benchmarks.

Motivation

  • A single utterance may require a shared voice anchor alongside delivery changes across successive text segments.The motivating example changes delivery from calm to excited to soft while keeping a brighter edited reference voice.Source: §1 · p. 1
  • Voice cloning uses an acoustic reference and voice design uses a textual description, while reference-conditioned editing specifies changes relative to an existing voice.EDICT uses the edited voice as a shared conditioning reference for new text under changing delivery instructions.Source: §1 · pp. 1–2
  • EDICT rebuilds the cache at each instruction boundary from the current instruction, edited reference, and bounded acoustic context to address control leakage while supporting repeated switching and continuity.The introduction contrasts this with retaining full history, which may weaken new instructions, and discarding it, which removes continuation context.Source: §1 · p. 2

Contributions

  • EDICT’s trainable editor maps a global request to attribute operations and creates a shared edited codec reference for a frozen TTS backbone.At instruction boundaries, the system reconstructs the KV cache under the next instruction while retaining selected initial and recent acoustic states.Source: §1 · p. 2
  • Experiments on Qwen3-TTS-VD and CosyVoice 2 show improved instruction following over the respective baselines for timbre editing and intra-utterance control.The evaluations also assess cache reconstruction across backbones and its combination with timbre editing.Source: §1 · p. 2
  • TimbreEdit-Bench evaluates requested voice edits and preservation of unspecified attributes, while IntraTTS-Bench evaluates local instruction following, speaker identity, and natural transitions.The benchmarks support objective metrics, listening tests, and ablations for global, local, and combined control.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses language-based voice and style control, reference-anchored voice editing, and local control of speech delivery. EDICT combines an edited codec reference with cache reconstruction across successive instruction switches.

Language-based voice and style control

  • Language-based systems control voice and delivery through descriptions, while ControlSpeech and FlexiVoice combine voice references with language-based style control.ReStyle-TTS adjusts style relative to a reference, and Vevo separates timbre and style prompts.Source: §2 · p. 2

Reference-anchored voice editing

  • Voice cloning specifies a voice acoustically, voice design specifies it textually, and voice editing applies instructed changes to a source voice.Prior systems include relative attribute edits, text-guided conversion, selective changes, and approaches integrating editing with TTS.Source: §2 · p. 2

EDICT

  • EDICT produces a reusable edited codec reference for changing local instructions and recomputes retained acoustic states at successive switches.Its cache is rebuilt under updated conditions using initial context for voice cues and recent context for continuity; requested changes and preservation are evaluated separately.Source: §2 · p. 3, Figure 1 · p. 3

Local control and acoustic history

  • Local-control work spans time-varying style, word-level styles, emotion trajectories, and timing; TED-TTS uses segment-aware causal masking and monotonic alignment.Kang et al. address acoustic style self-referencing with cache swapping and sliding-window attention for a source-to-target transition (Kang et al., 2026).Source: §2 · p. 3

3 METHODOLOGY

EDICT uses a shared edited codec reference to anchor voice identity while segment-specific instructions control delivery. It reconstructs the KV cache at instruction boundaries to refresh conditioning while retaining selected acoustic context.

3.1 TASK FORMULATION AND OVERVIEW

  • EDICT cascades a trainable timbre editor and frozen TTS backbone to generate successive segments in one shared edited voice.The edited codec reference conditions every segment, while local instructions change delivery.Source: §3.1 · p. 3

3.2 REFERENCE-CONDITIONED TIMBRE EDITING

  • Nine timbre attributes are encoded as operations: ordinal attributes receive direction and magnitude, categorical attributes receive target categories, and unmentioned attributes are kept.Operations are serialized in a fixed attribute order and interpreted with the acoustic reference.Source: §3.2 · pp. 3–4
  • Paired source–target profiles supply editing operations and codec targets for supervised fine-tuning, followed by weighted DPO with auxiliary supervised loss.Training tuples cover single-attribute and compositional edits, same or different texts, and within- and cross-language cases.Source: §3.2 · p. 4

3.3 SEGMENT-LEVEL INSTRUCTION CONTROL

  • At each non-final segment boundary, EDICT replaces the old KV cache with a prefill conditioned on the next instruction, text, and shared edited reference.The prefill includes acoustic inputs from the first S and most recent K generated positions.Source: Figure 2 · p. 4, §3.3 · p. 5
  • Reconstruction recomputes retained acoustic states under updated conditions: initial context preserves evidence from the utterance’s start, while recent context supports continuation.This bounds retained history; the cache then grows normally within each segment.Source: §3.3 · p. 5
  • During continuous decoding, each query attends to the rebuilt prefill and current-segment acoustic inputs through its position, preserving causal attention.Omitted history remains in the output codec stream, and all segments are decoded into one waveform.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments describe bilingual training and evaluation data, implementation choices, and metrics for intelligibility, quality, similarity, instruction compliance, and human ratings.

4.1 DATA AND BENCHMARKS

  • TimbreEdit-Bench contains 1,955 source–target pairs across nine edit categories, while IntraTTS-Bench contains 997 requests with two or three instructed text segments.Both benchmarks cover Chinese and English; annotators screen TimbreEdit-Bench pairs for audible edits, attribute preservation, and audio quality.Source: §4.1 · p. 6

4.2 IMPLEMENTATION DETAILS

  • The editor is fine-tuned from Qwen3-TTS-12Hz-1.7B-Base, while local generation uses frozen Qwen3-TTS-VD and also tests CosyVoice 2.The local-control comparisons include independent segment synthesis, joint prompting, and a TED-TTS adaptation without KV cache reconstruction.Source: §4.2 · p. 6

4.3 EVALUATION METRICS

  • Evaluation uses language-specific intelligibility metrics, perceptual-quality and speaker-similarity measures, instruction-compliance scores, and five-point human ratings.Human ratings cover naturalness, edit accuracy, target similarity, instruction following, and transitions, with 95% confidence intervals from audio-item bootstrapping.Source: §4.3 · p. 6

5 RESULTS

EDICT improves instruction-conditioned timbre editing and, with KV-cache reconstruction, local instruction adherence and transition quality while balancing speaker consistency against content fidelity. The ablations indicate that retaining both initial and recent acoustic context supports this balance.

5.1 TIMBRE EDITING

  • EDICT Stage I leads instruction-conditioned systems in target similarity and attribute edit direction and category accuracy, while Qwen3-TTS-VD better preserves unchanged attributes.The target-similarity comparison uses both ERes2Net and WavLM; target-reference voice conversion is a separate upper-bound setting with target audio.Source: §5.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5.2 INTRA-UTTERANCE INSTRUCTION CONTROL

  • EDICT Stage II consistently improves segment and emotion accuracy and transition quality over TED-TTS across both backbones, with modestly higher recognition error than TED-TTS.Speaker consistency remains comparable to TED-TTS; Concat. favors instruction following but has weaker transitions, while Joint favors speaker consistency.Source: §5.2 · pp. 6–7, Table 3 · p. 8

5.3 JOINT TIMBRE AND DELIVERY CONTROL

  • EDICT achieves the highest target similarity and predicted perceptual quality in joint timbre-and-delivery control, with speaker consistency near Joint but lower segment accuracy than independent synthesis.Replacing concatenation with KV-cache reconstruction improves temporal continuity and speaker consistency, at the cost of higher recognition error.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 SUBJECTIVE EVALUATION

  • In listener ratings, EDICT Stage II outperforms TED-TTS in instruction following and transition quality, with naturalness and transition ratings close to Joint.For timbre editing, EDICT Stage I has the highest edit-accuracy and target-similarity ratings among instruction-conditioned systems, with naturalness comparable to Qwen3-TTS-VD.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 ABLATION STUDIES

  • More SFT data improves target matching for structured instructions but gives mixed results for descriptive instructions; a larger DPO candidate pool improves SIM-T under both forms.wDPO achieves the highest SIM-T but has higher Chinese CER than SFT (1M), with mixed changes in predicted speech quality.Source: §5.5 · p. 8, Table 5 · p. 9
  • KV-cache reconstruction improves all three control metrics over persistent or truncated caches; removing initial context improves local accuracy but weakens speaker consistency, while removing recent context sharply degrades transitions.The full configuration retains the first S = 4 and most recent K = 5 acoustic positions; reconstruction also raises recognition error relative to the persistent cache.Source: §5.5 · pp. 8–9, Table 6 · p. 9

6 CONCLUSION

EDICT combines reference-conditioned timbre editing with segment-level delivery control, and evaluations report improved editing and target-voice matching alongside a favorable balance among adherence, consistency, and transitions.

6 CONCLUSION

  • EDICT combines an attribute-based editor that anchors a shared voice with segment-wise KV-cache reconstruction that updates delivery conditions while retaining selected acoustic context.The framework evaluates these controls with TimbreEdit-Bench and IntraTTS-Bench.Source: §6 · p. 10
  • Objective evaluations, listening tests, and ablations demonstrate improved timbre-edit accuracy and target-voice matching, with a favorable balance among instruction adherence, voice consistency, and natural transitions.Joint-control experiments show edited references supporting successive delivery changes within one utterance.Source: §6 · p. 10
  • Future work will expand timbre-editing data to more diverse voice characteristics and editing requests, and explore conversational speech generation.Source: §6 · p. 10
  • Generative AI assisted synthetic-data and benchmark construction, automatic speech evaluation, and language refinement; authors reviewed and validated the materials and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

APPENDICES

The appendices collect task definitions, data and benchmark construction, implementation details, evaluation protocols, and supplementary analyses.

APPENDICES

  • The appendix covers task taxonomy and method comparison, timbre-editing training data, benchmark construction, implementation, automatic evaluation, listening tests, attention analysis, and supplementary results.Source: APPENDICES · p. 16
  • Table 7 organizes speech-control tasks by typical inputs, targets, and temporal scope, while noting that inputs are not exhaustive and tasks may overlap.Source: Table 7 · p. 17

A TASK TAXONOMY AND METHOD COMPARISON

The taxonomy distinguishes speech-control tasks by what they modify, their inputs, and their temporal scope; the capability comparison is limited to named models and cited settings.

A TASK TAXONOMY AND METHOD COMPARISON

  • Speech content editing changes recorded words, whereas timbre editing changes specified voice attributes while aiming to retain others.Source: A TASK TAXONOMY AND METHOD COMPARISON · p. 17
  • Voice cloning and voice design specify a voice through an acoustic exemplar and a description, while global and local expressive control target utterances and individual spans.Instructions serve as an interface across these tasks, not a separate task.Source: A TASK TAXONOMY AND METHOD COMPARISON · p. 17
  • Table 8 records capabilities in cited settings; a positive timbre-modification entry does not establish selective preservation, and EDICT's inherited cloning and design capabilities are not evaluated.Source: A TASK TAXONOMY AND METHOD COMPARISON · p. 17
  • The comparison defines intra-utterance control as delivery assigned to words or segments, excluding content edits and paralinguistic insertions.Source: A TASK TAXONOMY AND METHOD COMPARISON · p. 17

B TIMBRE-EDITING TRAINING DATA

The appendix details synthetic timbre-editing supervision, benchmark construction, editor training, and segment-level decoding; its benchmarks cover global edits and intra-utterance delivery control.

B TIMBRE-EDITING TRAINING DATA

  • The nine-attribute schema organizes synthetic profiles; 6,000 profiles yield 239,794 recordings totaling 518.23 hours, with 200 profiles held out for TimbreEdit-Bench.Filtering used generation status and duration, without ASR transcript checks or perceptual attribute screening.Source: B TIMBRE-EDITING TRAINING DATA · pp. 17, 19, Table 9 · p. 19
  • One million directed training pairs average 4.92 changed attributes, include single and compositional edits, and contain no all-keep examples.The pairs derive categorical targets and ordinal level differences from paired profiles; their edit statistics do not measure audio realization accuracy.Source: B TIMBRE-EDITING TRAINING DATA · pp. 19–20

C BENCHMARK CONSTRUCTION

C.1 TIMBREEDIT-BENCH

  • TimbreEdit-Bench retains 1,955 screened pairs: 975 Chinese and 980 English, including 358 single-attribute and 1,597 compositional edits.Three annotators assessed audible requested changes, preservation of unmodified attributes, and audio quality for reliable judgment.Source: §C.1 · p. 21

C.2 INTRATTS-BENCH

  • IntraTTS-Bench contains 997 bilingual requests with two or three segments, totaling 2,393 segments and 1,396 adjacent boundaries.The requests are distributed across dialogue, observation, and vivid-description topics.Source: §C.2 · p. 21

D.1 TIMBRE-EDITOR TRAINING

  • The editor is fine-tuned on one million pairs and post-trained with preference pairs selected by target-speaker similarity, which does not directly account for intelligibility or attribute preservation.The preference pool excludes all 200 held-out profiles from both source and target sides.Source: §D.1 · pp. 22–23

D.2 SEGMENT-LEVEL DECODING

  • At segment boundaries, decoding rebuilds selected initial and recent acoustic states under the next instruction, then resumes causal generation from that cache.Excluded intermediate acoustic history is unavailable in the cache, although its codec frames remain in the output sequence.Source: §D.2 · p. 23

D.3 BASELINE AND ABLATION CONFIGURATIONS

The baselines vary how timbre edits and segment instructions are supplied, while ablations compare editor training choices and retained acoustic context. EDICT’s cache reconstruction is contrasted with reuse of existing KV states.

Baseline implementations

  • Qwen3-TTS-VD and Qwen3-TTS-Base use source audio in context with edit instructions, while CosyVoice 2 uses it as a speech prompt; FreeVC and Seed-VC use source and target audio without edit instructions.FreeVC and Seed-VC are evaluated separately on a same-text subset; instruction-conditioned baselines use both structured and descriptive instructions.Source: §D.3 · p. 23
  • Concat. synthesizes segments independently and concatenates them, Joint generates the utterance in one pass, and TED-TTS switches conditions through attention masking while retaining causal acoustic history.TED-TTS is adapted to each backbone and does not reconstruct the KV cache.Source: §D.3 · p. 23
  • Joint-control Qwen3-TTS-VD receives the original reference, global edit, and local instructions, while EDICT uses a shared precomputed edited reference for continuous generation with cache reconstruction.Editor + EDICT Stage I + Concat. uses the same edited reference but synthesizes segments independently.Source: §D.3 · p. 23
  • Timbre-editor ablations compare SFT data scale, DPO variants, preference scoring, and an additional wDPO round, with training-budget differences limiting attribution of DPO-RP effects to its scoring rule.DPO-RP uses M = 16 candidates; both DPO baselines start from the 1M-pair SFT checkpoint and use auxiliary SFT.Source: §D.3 · pp. 23–24
  • The cache ablation contrasts persistent KV states, truncated initial and recent context with reused states, and EDICT reconstruction under updated conditions.The full configuration uses (S, K) = (4, 5); variants remove initial or recent context while retaining reconstruction.Source: §D.3 · p. 24

E AUTOMATIC EVALUATION PROTOCOLS

The automatic protocols separately assess recognition, speech quality, speaker similarity, instruction adherence, and boundary transitions. The listening study uses blinded ratings from 19 participants, with distinct criteria for timbre editing and intra-utterance control.

E AUTOMATIC EVALUATION PROTOCOLS

  • Objective evaluation uses language-specific CER and WER, UTMOS and DNSMOS (OVRL) for speech quality, and WavLM4 and ERes2Net5 for speaker similarity.SIM-I averages similarities over unordered segment pairs within each utterance, using generation-provided segments when available and forced alignment otherwise.Source: E AUTOMATIC EVALUATION PROTOCOLS · p. 24
  • Gemini judges segment-level instruction and emotion realization separately, boundary-level speaker continuity and natural connection, and timbre attribute operations against source audio.The evaluation distinguishes edit direction, categorical targets, and preservation, and assesses transitions separately from instruction following.Source: E AUTOMATIC EVALUATION PROTOCOLS · pp. 24–25
  • Human–Gemini agreement on intra-utterance criteria ranges from 83.2% to 96.0%, with the highest agreement on transition quality; downward edits and preservation have lower agreement in timbre judgments.Pairwise agreement uses jointly valid labels; each timbre-editing annotator provides 100 valid judgments per criterion.Source: E AUTOMATIC EVALUATION PROTOCOLS · p. 25, Table 10 · p. 25, Table 11 · p. 25

F SUBJECTIVE LISTENING TEST

  • The listening study involves 19 participants, hidden system identities, and 1–5 ratings for 50 timbre-editing samples and 48 intra-utterance samples.Means are item-level listener averages with 95% percentile intervals from 10,000 item-bootstrap resamples.Source: F SUBJECTIVE LISTENING TEST · p. 26, Figure 8 · p. 26
  • Timbre-editing ratings separate naturalness, edit accuracy relative to the source, and target-timbre similarity, revealing the target reference only after edit accuracy is rated.The three criteria use five-point scales.Source: F SUBJECTIVE LISTENING TEST · pp. 26–27, Figure 9 · p. 27
  • Intra-utterance ratings assess naturalness, whether delivery attributes occur at the corresponding text spans, and whether expression changes naturally with voice continuity.Transition ratings accept natural pauses and instruction-appropriate changes, but penalize audible splices, unnatural gaps, truncation, and apparent speaker changes.Source: F SUBJECTIVE LISTENING TEST · pp. 27–28, Figure 9 · p. 27

G ATTENTION ANALYSIS OF KV CACHE RECONSTRUCTION

EDICT reconstructs selected acoustic context under each new segment’s instruction and text, unlike instruction masking that leaves prior acoustic states cached. On a 200-pair real-speech evaluation set, EDICT-Stage I outperforms instruction-conditioned baselines in target-speaker similarity across languages and instruction forms.

G ATTENTION ANALYSIS OF KV CACHE RECONSTRUCTION

  • At segment transitions, TED-TTS masks the previous instruction but retains preceding acoustic states, whereas EDICT selects initial and recent acoustic inputs and recomputes their representations under updated conditions.Figure 10 compares attention on the same two-segment request; the difference concerns both context selection and representation computation.Source: G ATTENTION ANALYSIS OF KV CACHE… · p. 28
  • Figure 11’s replay comparison changes both cache and conditioning, so it diagnoses reconstruction rather than isolating its effect on generation quality or control.Retained acoustic inputs are recomputed during full prefill under new conditions; quantitative cache ablations assess effects separately.Source: G ATTENTION ANALYSIS OF KV CACHE… · p. 28, Figure 11 · p. 29

H.1 TIMBRE EDITING WITH REAL SPEECH REFERENCE

  • The real-speech evaluation tests transfer from synthetic training and reference voices using 200 compositional editing pairs, evenly divided between Mandarin and English.The set uses neutral recordings from 20 speakers; requests specify attributes to modify and preserve based on annotated differences between same-sentence, same-language speaker pairs.Source: §H.1 · p. 29, Figure 12 · p. 30
  • EDICT-Stage I outperforms instruction-conditioned baselines in target-speaker similarity across languages and instruction forms on real speech, and is the only system with positive target–source similarity margins in every condition.It also has the highest DNSMOS in three of four language–instruction conditions; its advantage persists with descriptive instructions, though target similarity is lower than with structured instructions.Source: §H.1 · p. 29

H.2 TIMBRE EDITING WITH TIMBREEDIT-BENCH

EDICT-Stage I leads on requested attribute edits but still has a preservation gap. On MED-TTS, EDICT-Stage II improves emotion accuracy over TED-TTS in both languages with comparable speaker similarity, while producing lower boundary-discontinuity scores.

H.2 TIMBRE EDITING WITH TIMBREEDIT-BENCH

  • EDICT-Stage I achieves the highest Up, Down, and Category accuracy under both instruction forms, while Qwen3-TTS-VD has the highest preservation scores but lower edit accuracy.EDICT-Stage I’s remaining preservation gap means accurate requested changes do not guarantee that other attributes remain unaffected.Source: §H.2 · p. 30
  • Among single-round editor configurations, wDPO achieves the highest SIM-T across both instruction forms and speaker encoders; a second round reduces English WER but also lowers ERes2Net similarity and predicted speech quality.The paper retains single-round wDPO to balance target matching and speech quality; training differences prevent attributing the DPO-RP gap solely to its source-similarity penalty.Source: §H.2 · p. 30, Table 13 · p. 32
  • The auxiliary SFT coefficient λSFT = 0.01 gives the highest WavLM target-voice similarity for both instruction forms and the highest descriptive-instruction ERes2Net similarity.Larger coefficients can reduce English WER and improve UTMOS, but do not consistently improve target matching.Source: §H.2 · p. 30, Figure 13 · p. 32

H.3 INTRA-UTTERANCE INSTRUCTION CONTROL

  • On a 500-request IntraTTS-Bench subset, increasing initial context generally improves SIM-I but reduces EmoSim, while recent context beyond K = 4 largely plateaus in SIM-I and tends to lower EmoSim.The authors use (S, K) = (4, 5) as a compromise between speaker consistency and similarity to independently synthesized emotional references.Source: §H.3 · p. 31, Table 14 · p. 33, Table 15 · p. 33
  • On MED-TTS, EDICT-Stage II improves emotion accuracy over TED-TTS in both languages with comparable speaker similarity, decreases Chinese CER but increases English WER, and obtains the lowest TCD in both languages.TCD is a continuity proxy, not a direct measure of transition naturalness; phonetic content and intended expression changes also affect it.Source: §H.3 · p. 33, Table 16 · p. 33
  • EDICT-Stage II’s English emotion gains occur in Emotional Dialogue and Novelistic Description, while TED-TTS is slightly stronger on Observational Narrative; EDICT has the lowest TCD in all six language–category combinations.Table 17 reports results by language and text category.Source: §H.3 · p. 33, Table 17 · p. 34
Loading 2610.11437v1…