Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

1 INTRODUCTION

• The authors frame factual queries as constraint-satisfaction problems whose constraints completions must satisfy to be correct.The introduction illustrates constraints such as a movie’s director or award name.Source: §1 · pp. 1–2
• SAT Probe uses attention to constraint tokens to predict constraint satisfaction and factual correctness.The paper reports that it performs comparably to model confidence and can identify errors partway through the forward pass.Source: §1 · p. 2
• Attention to constraint tokens correlates with factual correctness, with less attention associated with inaccurate responses.Source: §1 · p. 2
• The study uses 10 datasets with more than 40,000 prompts to evaluate factual-error prediction.The datasets include single- and multi-constraint queries.Source: §1 · p. 2

2 BACKGROUND: LANGUAGE MODELS AND FACTUAL RECALL

• Transformer layers transform input embeddings into hidden states, and the final-layer representation at the last token is used to predict the next token.Source: §2 · p. 2
• Attention updates a token’s state using earlier token states, making it the operation of interest for studying token interactions.The section says standard Transformer MLP layers do not capture token interactions.Source: §2 · p. 2
• In autoregressive LLMs, attention weights are lower triangular because each token can attend only to previous tokens.Source: §2 · p. 3
• Prior work suggests that MLP layers store factual associations and attention transfers factual information from subject tokens to where it is used.For example, the section describes attention transferring information about LeBron James from the subject tokens to the final token used to generate “basketball.”Source: §2 · p. 3
• Unlike most cited factual-recall studies, this work investigates mechanisms involved when LLMs produce factually incorrect text.Source: §2 · p. 3

3 FACTUAL QUERIES AS CONSTRAINT SATISFACTION PROBLEMS

• Binary correctness labels cannot distinguish easy queries from barely-correct ones or reveal which query parts drive failure.Source: §3 · p. 3
• A factual query is represented by constraints and verifiers, and a completion is an error if any constraint’s verifier returns zero.Source: §3 · p. 3
• The framework focuses on conjunctions of constraints and assumes access to the tokens that identify the constraints in its evaluations.The text notes that the framework could be generalized to other operations.Source: §3 · p. 3
• LLM performance is better for more popular entities, and larger models perform better for less popular entities.Popularity is approximated using the number of site links on a WikiData page.Source: §3 · p. 4
• More constrained problems, with fewer potential solutions, have lower LLM performance, while larger models perform better across constrainedness levels.Source: §3 · p. 4, Figure 2 · p. 4
• The CSP lens is proposed as a useful way to capture factual-query difficulty and discuss how LLMs process queries and generate errors.Source: §3 · p. 4

• The analysis tracks attention contributions from constraint tokens to the final token used for generation, including constraints composed of multiple tokens.Source: §4 · pp. 4–5
• Across Llama-2 7B, 13B, and 70B, attention predicts constraining-entity popularity with held-out Spearman’s Correlation ρ ≥0.65 and p-value p ≈0.The authors note that popularity may not be available as a clean measure, motivating a more direct indicator of factual correctness.Source: §4 · p. 5
• Correct generations show stronger constraint-token attention than failures, and total attention to constraints correlates with accuracy.The analysis compares attention across confident and less-confident predictions and relates total constraint attention to accuracy across query groups.Source: §4 · p. 5
• Across model scales, greater relative attention in both models generally accompanies shared success, while larger-model-only success occurs when the larger model attends more.The authors report that performance improvements in larger models correlate with increased attention to constraint tokens.Source: §4 · p. 6

5 PREDICTING FACTUAL ERRORS USING ATTENTION TO CONSTRAINTS

• The benchmark contains 10 datasets with over 40,000 queries, evaluated using Llama-2 7B, 13B, and 70B variants.Single-constraint queries use exact match when there is one solution and WikiData verification when multiple solutions are possible.Source: §5 · p. 7

5.1 PREDICTING FACTUAL CORRECTNESS

• SAT PROBE estimates constraint satisfaction from self-attention, using a linear probe over attention weights across layers and heads.For multi-constraint queries, it combines predictions for individual constraints.Source: §5.1 · p. 7
• Exact-match verification is a strict criterion that can introduce noise into evaluations where it is used.The authors identify this as a limitation of their verification approach.Source: §5.1 · p. 7
• SAT PROBE is often comparable to CONFIDENCE and better than POPULARITY, while COMBINED performs best in most scenarios.Across single- and multi-constraint settings, SAT PROBE also provides fine-grained feedback; the figure reports standard errors over 10 random train/test splits.Source: Figure 6 · p. 8, §5.1 · p. 8
• Attention alone significantly outperforms the CONSTANT baseline and sometimes exceeds CONFIDENCE, but it does not explain every failure.The authors report cases where attention to constraints is present even though the model still fails.Source: §5.1 · p. 8

5.2 EXTENSIONS

• For Llama-2 7B and 13B, early stopping preserves average performance while saving 50% of failure-case wall-clock time on most datasets; 70B shows a slight performance drop.The predictor uses attention only up to an intermediate layer.Source: §5.2 · p. 9
• SAT PROBE predicts satisfaction for individual constraints with results comparable to single-constraint prediction, unlike raw CONFIDENCE, which gives one value for all constraints.The authors suggest partial constraint-satisfaction reports may help debug issues such as failure to follow specific instructions.Source: §5.2 · p. 9
• A single predictor trained across mixed single-constraint datasets performs competitively with predictors trained separately and better than POPULARITY.The authors present this as potential for developing general factual-error detectors.Source: §5.2 · p. 9

6 RELATED WORK

• Meng et al. (2022) and Geva et al. (2023) study factual recall mechanisms but do not investigate the generation of factually incorrect text.The paper also notes that many attention-head studies analyze functions such as syntax or induction.Source: §6 · p. 9
• Li et al. (2023) study truthfulness directions across total attention contributions, whereas this work uses constraint-only attention weights to predict factual errors and fine-grained constraint satisfaction.The paper also identifies early stopping as an additional contribution.Source: §6 · p. 9
• Black-box inconsistency checks can require several forward passes and may convey conflicting signals, such as an LLM refuting its initial claim (Mündler et al., 2023; Manakul et al., 2023; Zhang et al., 2023).The passage notes that such conflicting signals may diminish user trust (Liao & Vaughan, 2023; Huang et al., 2020).Source: §6 · p. 9

7 CONCLUSION AND FUTURE WORK

• The analysis covers conjunctive factual queries, while possible constraints also include instructions and disjunctive queries, so the framework does not cover all factual queries.The approach also assumes access to constraint tokens; extracting them is left for future work.Source: §7 · p. 9
• The information carried by attention patterns remains opaque and warrants further investigation.Source: §7 · p. 9
• Future work could investigate actions to fix errors, including steering model behavior by manipulating attention to constraints.The authors hope these methods and analyses contribute to the pursuit of reliable and safe LLM generations.Source: §7 · p. 9

A MODEL DETAILS

• Llama-2 70B uses 8-bit quantization to fit on one A100 GPU with 80GB of memory, and all models use greedy decoding at temperature 0.The models are accessed through HuggingFace Transformers.Source: A MODEL DETAILS · p. 14
• Grouped-Query Attention reduces the KV cache loaded at each inference step, but does not change analyses of attention weights or attention contributions.The passage describes Llama-2 70B as using an 8 key-value projection variant of GQA.Source: A MODEL DETAILS · p. 14
• The study examines attention contributions rather than MLP contributions, which may also contain relevant signals.The authors leave exploration of MLP contributions to future work.Source: A MODEL DETAILS · p. 14

B.1 SINGLE-CONSTRAINT EXPERIMENTS

• Each dataset is split into train and test sets, and features are normalized to zero mean and unit variance using the training split.Source: §B.1 · p. 14
• A logistic regressor with C = 0.05 L1 regularization is trained on one subset and evaluated on the other across 10 random seeds.The reported performance is the mean with standard error for each dataset and method.Source: §B.1 · p. 14
• SAT PROBE(A), which uses attention weights, performs better than SAT PROBE(a), which uses attention contribution.Source: §B.1 · p. 14

B.2 MULTI-CONSTRAINT EXPERIMENTS

• Multi-constraint splits keep identical constraint pairs out of both training and test sets, and correctness requires satisfying both query constraints.Constraint pairs are treated as sets, so prompts with reversed constraint order stay in the same split.Source: §B.2 · p. 14

B.3 PREDICTING PARTIAL CONSTRAINT SATISFACTION

• Partial-satisfaction experiments predict each constraint separately; confidence is not a baseline because it cannot make per-constraint predictions.The authors point to future work on individual instructions and queries with logical operators such as OR.Source: §B.3 · p. 15

B.4 EARLY STOPPING EXPERIMENTS

• Up to 50% less computation time causes little or no failure-prediction performance drop for most datasets, while Llama-2 70B benefits more from later layers.The authors suggest attention patterns could support early abstention or escalation.Source: §B.4 · p. 15

B.5 GENERALIZING THE FAILURE PREDICTOR

• A single predictor trained on mixed single-constraint datasets remains better than POPULARITY and competitive with individual predictors, with only minor performance fluctuations.The mixture uses 50% of each dataset for training and 50% for testing.Source: §B.5 · p. 15, Figure 8 · p. 16

C.1 PREDICTING POPULARITY

• Basketball-player popularity is predicted from attention using Lasso regression on entities with at least 15 site links, split equally into training and test sets.The regression uses a 0.005 regularization strength.Source: §C.1 · p. 16

C.2 COMPUTING ATTENTION TO A CONSTRAINT

• Constraint attention is measured as the maximum across all constraint tokens, since tokenization and ambiguity can make earlier tokens more informative than the last token.For Kelly Oubre Jr., the passage reports little attention to “Jr.” and the most attention on “Oubre.”Source: §C.2 · p. 17

D.1 DATA CURATION FOR SINGLE-CONSTRAINT QUERIES

• Four non-CounterFact query groups use WikiData entities with at least five site links and fields selected to support exact-match evaluation.The site-link threshold is intended to avoid extremely tail knowledge and low-quality entity samples.Source: §D.1 · p. 17
• CounterFact queries use mother tongue, citizenship, and headquarters-location relations selected for less variation in valid exact-match answers.The authors report nontrivial model performance on these relations despite using exact match.Source: §D.1 · p. 18

D.2 DATA CURATION FOR MULTI-CONSTRAINT QUERIES

• The multi-constraint datasets contain 1,352 word prompts, 1,290 Nobel-winner prompts from 645 cities, and 1,492 book prompts from 746 books.Each of the three datasets has at least one correct completion per query.Source: §D.2 · p. 19

D.3 VERIFICATION

• For single-constraint queries, exact match checks whether the first |Y| generated tokens equal the reference, and prompts are shaped to elicit the desired format.For example, a question can include “the year founded in” and the response can begin “The team was founded in.”Source: §D.3 · p. 19
• Multi-constraint book verification looks up the generated entity in WikiData, retrieves relevant fields, then uses GPT-3.5 to check whether each constraint appears in those fields.The GPT-3.5 step is limited to multi-constraint queries because it is not cheap.Source: §D.3 · p. 19
• Word-query constraints are checked by testing the first or last character of the completion’s first word.Source: §D.3 · p. 19

E COMPARING CONFIDENCE AND ATTENTION

• The comparison uses test-set predictions for Football Teams and Basketball Players, with a 0.5 decision threshold for judging whether each model predicts correctly.Source: E COMPARING CONFIDENCE AND ATTENTION · p. 19
• Figure 15 places confidence-only failure predictions on the x-axis and attention-only predictions on the y-axis, with bins colored by prediction correctness.Source: E COMPARING CONFIDENCE AND ATTENTION · p. 19, Figure 15 · p. 21
• Attention and confidence predictors sometimes disagree on failure calls, with disagreement becoming more pronounced for larger LLMs.Source: E COMPARING CONFIDENCE AND ATTENTION · p. 20, Figure 15 · p. 21