PaperlayerView exact revision on arXiv

Source-linked AI summary

StateWise: Teaching Language Models to Track and Obey Evolving Constraints in Multi-Turn Dialogue

Samarth Varshney, Ritu Yadav

arXiv:2610.33766v1cs.CLcs.AIcs.LG

TL;DR

Instruction-tuned models can forget active constraints or keep enacting withdrawn ones, while the prior repair described here uses an external ledger. StateWise trains one LLM to track revisable constraint state and respond using it, then adds preference learning from relapse contrasts. Across two backbones, relapse falls by 8–22 points relative to supervised self-training, while the recipe has a documented breadth cost.

  • Problem

    Evolving-constraint dialogue requires models to retain active requirements and stop enacting withdrawn ones, while the prior repair identified here uses an external ledger.

    Source: §1 · p. 1

  • Method

    StateWise fine-tunes one LLM on state summarization and state-conditioned responses, then self-trains with exact round-trip filtering and relapse-contrastive preference learning.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Relapse falls by 8–22 points relative to supervised self-training on two backbones, two training seeds, and unseen phrasings.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Tracking constraint state and obeying it are separable skills, and the round-trip check becomes a useful negative signal when used for preference learning.

    Source: §8 · p. 9

    The experiments use programmatically checkable formatting constraints and scripted trajectories, while semantic constraints and real-world user phrasings are broader.

    Source: §9 · p. 9

Abstract

from arXiv · show

Users of conversational assistants rarely state all requirements up front: they add constraints, amend them, withdraw them, digress, and return. Instruction-tuned LLMs fail here in two ways: active constraints stop being honored as turns accumulate (persistence failure), and withdrawn constraints keep being enacted (relapse). The only existing repair we know of is an external ledger that re-injects a compiled specification every turn. We propose StateWise, a self-improving approach that teaches a single LLM to maintain its own revisable constraint state, a typed state with tombstones for revoked constraints, so persistence and relapse are measured by executable checkers and ground-truth states are compiled for free. StateWise fine-tunes the LLM on two complementary skills, summarizing a dialogue into its constraint state and responding conditioned on that state, then self-trains on its own trajectories, keeping only turns that pass an exact round-trip check against the compiled state. This teaches 7-8B models to track the state almost perfectly but, surprisingly, not to obey it: on Llama-3.1-8B every supervised variant roughly doubles relapse. We therefore add a preference stage on self-generated relapse contrasts, where the rejected response enacts a revoked rule and the chosen one passes the round-trip check. On two backbones with held-out constraint families and unseen phrasings, StateWise cuts relapse by 8-22 points relative to supervised self-training, lifts conversation success by 8-19 points over the backbone, and beats an oracle compiled-ledger baseline on relapse with no inference-time scaffolding. We also characterize the recipe's cost on general instruction-following benchmarks and study weight interpolation as a mitigation.

1 INTRODUCTION

Evolving constraints expose persistence and relapse failures, while existing training work does not teach models to maintain revisable constraint state. StateWise combines state tracking, round-trip-checked self-training, and relapse-specific preference learning; its supervised stage alone can worsen relapse.

Related work

  • Existing training approaches address single-turn constraint following or information revealed across turns, but none trains a revisable constraint state.The prior repair identified in the paper is an external tombstone ledger re-injected into each request, at roughly 1.5× delivery cost and only for code.Source: §1 · pp. 1–2

StateWise

  • StateWise teaches a single LLM to maintain revisable constraint state using complementary state-summarization and state-conditioned response skills.It composes verifiable constraints into scripted ADD, AMEND, and REVOKE trajectories and checks emitted states against compiled ground truth.Source: §1 · p. 2

The supervised-stage result

  • On Llama-3.1-8B, every supervised variant roughly doubles relapse, even when the model emits the correct state with tombstones.The model produces the correct in-distribution state on 92% of turns, showing that tracking and obeying the state are separable.Source: §1 · p. 2

The preference-stage result

  • Relapse-specific preference learning cuts relapse by 8–22 points on two backbones, two seeds, and unseen phrasings, below both the backbone and an oracle compiled-ledger system.The chosen and rejected responses are labeled using the round-trip check; active-constraint-only preference pairs raise relapse, while shuffled pairs do nothing.Source: §1 · p. 2

Contributions

  • The study formalizes persistence and relapse with executable metrics and reports conversation-success gains of 8–19 points over the backbone.Experiments also characterize costs on cumulative-constraint and single-turn benchmarks; backbone weight interpolation removes that cost at the price of the relapse gain.Source: §1 · p. 2

2 RELATED WORK

Prior work studies multi-turn degradation, verifiable instruction following, information fragmentation, and revocation inertia. StateWise differs by training one model on multi-turn constraint edits with explicit state targets and checker-labeled preference pairs.

Multi-turn instruction following

  • Multi-turn benchmarks document degradation with dialogue depth, topic switches, and revised goals; StateWise adds amended and withdrawn constraints and measures relapse directly.The paper uses Multi-IF as a held-out test.Source: §2 · p. 2

Verifiable instruction following

  • Single-turn verifiable instruction-following methods train on individual prompts, whereas StateWise trains on multi-turn edit trajectories with explicit state targets.Single-turn constraint training is a StateWise baseline and raises both persistence and relapse.Source: §2 · p. 3

Lost in conversation and dialogue state

  • Methods for information fragmentation and fixed-ontology dialogue tracking do not address constraint revision; StateWise uses an open set of executable predicates and tombstones in the responding model.Its state is supervised by self-generated data.Source: §2 · p. 3

Revocation inertia

  • ReBIND repairs revocation inertia with an external compiled ledger for code-generation dialogues, while StateWise trains the model and extends to general formatting constraints.The paper also compares against compiled-ledger baselines.Source: §2 · p. 3

Self-improvement with round-trip filtering and preference signals

  • StateWise uses exact structured-state matching and executable checks for filtering, and the same checkers label preference pairs without a judge model.Its methodological template is SimRAG’s combination of complementary skills, self-generated data, and a consistency filter.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper represents evolving constraints as edit trajectories over a typed active-and-revoked state, with executable checkers and compiled ground truth. Persistence, relapse, and conversation success are defined over responses to those states.

3 PROBLEM FORMULATION

  • A constraint pairs a family with parameters, and dialogue edits ADD, AMEND, or REVOKE constraints, with DISTRACT and RETURN representing digressions and task resumption.The state records active constraints and tombstones for revoked or superseded ones; re-adding a tombstoned constraint revives it.Source: §3 · p. 3
  • Compiled states provide exact ground truth because the edit history yields an order-independent canonical serialization.This makes the state available for training and evaluation without annotation.Source: §3 · p. 3
  • Persistence is the fraction of active constraints satisfied, while relapse is the fraction of eligible tombstoned constraints satisfied by a response.Relapse excludes families with chance-satisfaction rate above 0.10 and amended constraints whose new version logically implies the old one.Source: §3 · p. 3
  • Conversation success requires every active constraint to be satisfied and zero relapse at every turn.Metrics are averaged over conversations with conversation-level bootstrap intervals, and arm comparisons are paired on shared trajectory identifiers.Source: §3 · p. 3
  • Evaluation includes 240 in-distribution conversations and two 300-conversation sets, one with six untrained families and one with alternative phrasings and new task prompts.Trajectories contain four to eight turns, and test sets are fixed before training.Source: §3 · p. 4

4 METHODOLOGY

StateWise induces state-tracking and state-conditioned response skills, then self-trains on turns that pass an exact round-trip filter. A final preference stage contrasts relapsed responses with round-trip-clean responses; inference can also test behavior without explicit state prompting.

4.1 STAGE I: SKILL INDUCTION

  • Stage I fine-tunes on single-turn instruction following, dialogue-to-state summarization, and state-conditioned response generation using scripted trajectories and a backbone-generated teacher signal.Responses for the state-conditioned mixture are obtained by rejection sampling from the backbone.Source: §4.1 · p. 4

4.2 STAGE II: SELF-TRAINING WITH ROUND-TRIP FILTERING

  • Stage II keeps generated turns only when the emitted state exactly matches compiled ground truth, active checkers pass, and no low-chance tombstoned constraint is satisfied.The model is fine-tuned on kept turns; relapse is the dominant rejection reason, but filtering supplies no signal against relapse.Source: §4.2 · p. 4

4.3 STAGE III: RELAPSE-CONTRASTIVE PREFERENCE LEARNING

  • Stage III pairs a response that enacts a tombstoned rule with a passing response from the same prefix, labeling pairs with compiled states and executable checkers.DPO uses a frozen reference and a fixed budget of 59 optimizer steps over 931 pairs.Source: §4.3 · p. 5

4.4 INFERENCE

  • Inference evaluates state-emitting and state-free modes, and backbone weight interpolation offers a no-training way to trade relapse gains against breadth.The interpolation uses α values of 0.5 and 0.75.Source: §4.4 · p. 5

5 EXPERIMENTAL SETUP

The experiments compare StateWise training stages and ledger baselines on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, using preregistered metrics and gates. They also report public-benchmark checks and training costs.

Backbones and arms

  • Llama-3.1-8B-Instruct is the primary backbone (two training seeds), and Qwen2.5-7B-Instruct the secondary (one seed); arms span supervised stages, StateWise, and two matched-prompt-token ReBIND-style external-ledger baselines.A6B lists only rules in force and uses compiled ground-truth state at inference, making it an oracle ledger.Source: §5 · p. 5

Evaluation sets and metrics

  • Evaluation covers held-out, in-distribution, and paraphrase conversations, plus Multi-IF, IFEval, and MT-Bench, using persistence, relapse, conversation success, and state exact-match.The held-out set has 18 families, including six unseen families; the paraphrase set uses author-written phrasings and new tasks.Source: §5 · p. 5

Preregistration

  • Hypotheses, arms, metrics, and pass/fail gates were fixed before each run; failed gates are reported as failures.Gates include comparison with the backbone and oracle ledger, state-free inference, Stage III outcomes, replication, and public-benchmark checks.Source: §5 · p. 5

Main results

  • Table 2 reports the main Llama-3.1-8B-Instruct results and state exact-match for A2, A4, and A7; A6 and A6B are oracle baselines.The reported exact-match values are 0.527 / 0.981 for A2, 0.510 / 0.916 for A4, and 0.500 / 0.893 for A7, held-out / in-distribution.Source: §5 · p. 5, Table 2 · p. 6

6 EXPERIMENTAL RESULTS

StateWise’s preference stage reduces relapse across replications and improves conversation success, while supervised self-training can track constraint state without reliably obeying it. An initial diagnostic also finds that prompting an untrained model to maintain its own state worsens performance.

6.1 THE FAILURE EXISTS AND PROMPTING DOES NOT FIX IT

  • 0.887 first-turn persistence falls to 0.653 by turn eight, while prompting self-state worsens every metric and matches compiled states only 0.3% of the time.The diagnostic used 240 trajectories and 1,438 assistant turns; revoked low-chance constraints were enacted on 11% of post-edit turns.Source: §6.1 · p. 6

6.2 MAIN RESULTS

  • A4 relapse is 0.246 held-out versus the backbone’s 0.130, despite exact compiled-state output on 92% of in-distribution turns.Every supervised arm roughly doubles relapse; rejection-sampling SFT filters relapsed examples but supplies no signal against relapse.Source: §6.2 · p. 6
  • 0.246 to 0.094 held-out relapse follows 59 DPO steps on 931 self-generated contrasts; conversation success rises 0.080 over A4 held-out.State exact-match is unchanged in-distribution, indicating the gain is in obeying the state rather than tracking it.Source: §6.2 · pp. 6–7

6.3 REPLICATION: SECOND SEED, SECOND BACKBONE, UNSEEN PHRASINGS

  • Stage III cuts relapse 13–17 points on Llama and 7.6 points on Qwen, while raising conversation success 10–19 points over the backbone across replications.The replication covers a second Llama seed, a second backbone, and unseen phrasings; the strict two-part replication gate passes on Qwen but narrowly fails on Llama seed 2.Source: Table 3 · p. 7, §6.3 · p. 7

6.2 MAIN RESULTS

  • StateWise has lower relapse than the oracle ledger everywhere, matches its held-out success, and beats its in-distribution success.The oracle ledger that explicitly lists withdrawn rules under a “do not follow” header nearly triples relapse to 0.380.Source: §6.2 · p. 7

7 ANALYSIS

The ablations show that relapse-specific preference contrasts are needed to reduce relapse, while active-violation contrasts can worsen it. Specialization costs breadth, and weight interpolation trades some relapse gain for broader benchmark performance; relapse clusters in silently enacted formatting rules.

7.1 ABLATION STUDIES: WHAT THE PREFERENCE STAGE NEEDS

  • 30% active-violation pairs add 4.7 points of relapse reduction, while the relapse-specific contrast is necessary and active-only contrasts raise relapse.Label-shuffled pairs do nothing; both preregistered ablation gates pass.Source: §7.1 · pp. 7–8

7.2 THE COST OF SPECIALIZATION AND A CONTROLLABLE TRADE-OFF

  • Specializing an 8B model to 12 formatting families lowers Multi-IF turn-3 performance by 6.5 points and IFEval prompt-strict accuracy by 6.5 points versus the backbone.A weak MT-Bench judge also signals a small first-turn regression.Source: §7.2 · p. 8
  • At α = 0.5, conversation success is 0.413 and persistence is +0.087 over A0, while relapse returns to the backbone level.This setting is within 1.9 points of the backbone on IFEval and above it at Multi-IF turns 2–3; the preregistered interpolation gate is not met by either α.Source: §7.2 · p. 8, Table 5 · p. 8

7.3 WHERE THE RELAPSE GOES

  • Relapse concentrates on silently enacted rules such as no commas, no digits, sentence count, and exact paragraphs; Stage III removes most without changing state exact-match.The authors describe this as the behavioral signature of knowing a rule was withdrawn but not that this mattered.Source: §7.3 · pp. 8–9

8 CONCLUSION

StateWise addresses forgetting active constraints and re-enacting withdrawn ones by teaching a model to track constraint state and then using relapse contrasts as preference signals. Across two backbones, relapse falls by 8–22 points relative to supervised self-training, with breadth costs and a trade-off through interpolation.

8 CONCLUSION

  • After 59 steps of DPO on self-labeled contrasts, the method cuts relapse by 8–22 points, below the backbone and an oracle compiled ledger, across two backbones, two seeds, and unseen phrasings.Source: §8 · p. 9
  • The exact round-trip filter teaches a 7–8B model to track constraint state almost perfectly but does not teach it to obey that state.Every supervised Llama variant roughly doubles relapse, and none of the Qwen variants reduces it.Source: §8 · p. 9
  • A relapse-contrastive preference stage supplies a negative signal against relapse, while ablations identify the relapse-specific contrast as necessary.The recipe costs breadth on cumulative-constraint and single-turn benchmarks; backbone interpolation trades that cost against relapse reduction.Source: §8 · p. 9

9 LIMITATIONS

The evaluation is limited to scripted trajectories and formatting-level constraints, with phrasing shift tested using a paraphrase set. Results concern 7–8B LoRA backbones, and the relapse metric excludes families with high chance-satisfaction rates.

9 LIMITATIONS

  • Constraint families are formatting-level and programmatically checkable; trajectories are scripted, and only a paraphrase set tests phrasing shift.Semantic constraints and user phrasings in the wild are broader than this evaluation.Source: §9 · p. 9
  • The relapse metric excludes high-chance families by construction, understating relapse on constraints that are easy to satisfy accidentally.Forbidden word is excluded because its measured chance-satisfaction rate is 0.94–0.95; the other reported family rates and exclusions are specified in the trajectory setup.Source: §9 · p. 9, A CONSTRAINT FAMILIES AND TRAJECTORIES · p. 11
  • Results cover 7–8B backbones with LoRA; stronger models may be near the relapse floor, and breadth costs may differ.Held-out success intervals have roughly five-point half-widths, and relapse relative to the backbone crosses zero on one seed and touches it on the paraphrase set.Source: §9 · p. 9

A CONSTRAINT FAMILIES AND TRAJECTORIES

  • The authors plan to release the constraint library, trajectory generator, checkers, compiler, filters, pair construction, configurations, scripts, and result artifacts.Generated trajectories are stored because Python random sequences can differ across interpreter versions.Source: REPRODUCIBILITY STATEMENT · p. 9

B TRAINING DETAILS AND FILTER STATISTICS

The appendix reports SFT and DPO training settings and filter yields, then summarizes preregistered gates and their outcomes. Relapse was the leading filter rejection reason across stages and backbones.

B TRAINING DETAILS AND FILTER STATISTICS

  • SFT used one epoch with LoRA rank 64 and learning rate 10^-4; Llama Stage I trained on 9,238 examples and Stage II kept 9,920 turns.Stage I took 55–61 minutes on two A100s, and Stage II took 68 minutes.Source: B TRAINING DETAILS AND FILTER STATISTICS · p. 12
  • DPO used 931 preference pairs for 59 steps, with at most 30% active-violation pairs and a chosen-response NLL weight of 0.5.The pairs comprised 663 relapse and 284 active-violation examples; 16 were dropped for length.Source: B TRAINING DETAILS AND FILTER STATISTICS · p. 12
  • Relapse was the leading rejection reason at every stage and on both backbones, including 887 relapse versus 411 active-violation reasons in Stage II.Stage-I Llama yielded 9,238 of 10,723 attempted turns (0.86); Stage II yielded 9,920 of 11,243 (0.88).Source: B TRAINING DETAILS AND FILTER STATISTICS · p. 12

C PREREGISTRATION SUMMARY

  • The protocol fixed arms, metrics, seeds, and gates before training, with no gate or hyperparameter changed after results were seen.The summary records mixed outcomes, including a narrow G2 pass and a G1 fail.Source: C PREREGISTRATION SUMMARY · p. 12

D ADDITIONAL RESULTS

Additional results compare Stage-III preference ablations, per-turn behavior, and interpolation with oracle-ledger baselines. Interpolation changes the success, relapse, and persistence trade-off, while A7 reduces relapse across turn depths.

Stage-III ablations

  • A7 reduced relapse versus A7-act by 0.266, versus A7-shuf by 0.199, and versus A7-rel by 0.061; versus A7-rel, success increased by 0.092.These are in-distribution Table 4 comparisons.Source: D ADDITIONAL RESULTS · p. 12

Per-turn behavior

  • A7 lowered relapse below the backbone at every diagnostic-set depth, while A4 raised it; trained arms shifted persistence upward by 4–9 points.Backbone persistence declined from 0.887 to 0.653 across the listed turn indices.Source: D ADDITIONAL RESULTS · p. 12

Interpolation vs. oracle ledger

  • At α = 0.5, interpolation versus A6B yielded success +0.023, relapse +0.004, and persistence +0.008, with all intervals including zero.At α = 0.75 versus A0, success was +0.107 and persistence +0.058; relapse was −0.005.Source: D ADDITIONAL RESULTS · pp. 12–13

Prompts

  • A6 prepends compiled rules in force plus a withdrawn-rules “do not follow” list, while A6B prepends only rules in force on every user turn.Both ledger baselines were matched in token count to the state block.Source: D ADDITIONAL RESULTS · p. 13

Related papers

Loading 2610.33766v1…