When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks

INTRODUCTION

• Accuracy and repeatability can mislead: models may repeat incorrect judgements or produce correct judgements through improper evidence use, so hidden failures need separate assessment.One cited study reported 84% and 87% repeat-run agreement for ChatGPT and Claude, respectively, while 57% of incorrect judgments had correct supporting rationales.Source: INTRODUCTION · pp. 4–5
• The taxonomy addresses a gap in prior frameworks by organizing mechanisms according to whether they can cause repeat-run variability, hidden failures, or both, to guide suitable tests and mitigations.Different mechanisms can produce similar observed errors, such as an incorrect eligibility decision caused by omitted evidence, an overlooked criterion, or misinterpreted criteria.Source: INTRODUCTION · p. 5

Scope of the taxonomy

• TRAILS covers tasks requiring interpretation of source information against criteria, not simple factual retrieval, regardless of whether the task uses a single call or a more complex workflow.Risk-of-bias assessment and figure interpretation require identifying relevant information and applying criteria, creating opportunities for variable or hidden failures.Source: Scope of the taxonomy · pp. 5–6
• Mechanisms were included when credible evidence supported a plausible, practically testable pathway to repeat-run variability, hidden failure, or both; empirical proof of causing either was not required.The taxonomy identifies potential failure sources and provides a basis for targeted evaluation, rather than establishing that mechanisms caused particular observed failures.Source: Scope of the taxonomy · p. 6

OVERVIEW OF THE TAXONOMY OF MECHANISMS

• TRAILS identifies 38 mechanisms across ten categories: 4 met IC1 only, 20 IC2 only, and 14 both (excluding sub-mechanisms); deployment headings indicate access generally needed to investigate mechanisms.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · p. 7
• Researchers’ ability to investigate mechanisms depends on deployment access: some causes are difficult to verify directly, while vendor-managed agentic environments may expose intermediate steps without guaranteeing identification or prevention.Some mechanisms depend on inaccessible training data, internal routing, numerical processes, or model reasoning; access to document processing, retrieval, agent execution, and context management varies across systems.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · pp. 7–8, Scope of the taxonomy · p. 7
• The taxonomy groups mechanisms into ten feature-based categories and codes their potential effects on framing, evidence basis, judgement formation, expression, and substantiation.A mechanism may affect several aspects of a read-and-evaluate task.Source: Scope of the taxonomy · p. 7, OVERVIEW OF THE TAXONOMY OF MECHANISMS · p. 8
• A checklist helps researchers document which mechanisms were tested, disabled, reported, or mitigated, but it is a practical reference rather than a comprehensive or formally validated evaluation tool.Some mechanisms are conditional on system features being enabled; an observed failure alone does not identify its cause, so researchers should confirm the condition and assess its contribution where possible.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · p. 8
• Response position, co-batched items, structured-output requirements, or omitted requested components can alter judgments or required outputs.The effects depend on the model, task, instructions, format, and material; fixed-format requirements may be especially consequential for complicated formats and reasoning tasks.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · pp. 9–10
• Parsing, media preprocessing, long-prompt position, and evidence quotation or extraction can distort, omit, misattribute, or under-use source evidence.Repeated runs can agree while missing the same mid-document evidence; extracted evidence may also be absent, copied incorrectly, assigned to the wrong source or location, or fail to support the judgment.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · p. 10
• Judgments may accept unchecked prompt assumptions, give rationales disconnected from evidence, fail to integrate task components, or substitute superficial cues for criterion-relevant evidence.A model can identify individual components yet combine them using a rule that differs from the prespecified decision rule.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · pp. 10–11
• Embedded source instructions, variable agent workflows, transformed verification inputs, model updates, training contamination, or sampling randomness can affect judgments.Training contamination can yield high apparent accuracy and repeatability even when the model has not independently appraised the supplied evidence.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · pp. 11–12
• Document segmentation, retrieval, context compaction, pruning, or loss of earlier-turn instructions can separate or remove material needed for later judgments.These pathways can change the evidence or criteria available to the model; compaction may change or lose context, while pruning removes material rather than preserving it in a summary.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · p. 12
• The 38-mechanism taxonomy distinguishes repeat-run variability (IC1), hidden judgement failures (IC2), and mechanisms eligible for both, while mapping deployment settings for investigation or control.Eligibility indicates possible failure types, not demonstrated failures; the effects need not occur under the same circumstances.Source: OVERVIEW OF THE TAXONOMY OF MECHANISMS · pp. 13–14, Figure 2 · p. 14

Evidence varied across mechanisms

• 13 mechanisms had strong overall evidence, 24 moderate, and one limited; for read-and-evaluate-specific evidence, eight were strong, 18 moderate, and 12 limited.Ratings reflected evidence yield and directness; several mechanisms had stronger broader-LLM evidence than task-specific evidence, including model version drift and agentic workflow instability.Source: Evidence varied across mechanisms · p. 14

TesƟng approaches

• IC1 testing repeats the same task while holding specified evidence, prompt, model, and researcher-controlled settings fixed, then compares judgments and required outputs.Prespecify the number of repetitions and what counts as a material difference; observed variability does not necessarily identify its cause.Source: TesƟng approaches · p. 14

Mitigation strategies

• Mitigations may reduce a failure’s likelihood or effect but do not prove it was prevented; researchers should assess implementation and test whether the problem remains.Mitigations can introduce other mechanisms, and user control may be limited when a commercial vendor manages the relevant process; human verification can serve as a later safeguard.Source: MiƟgaƟon strategies · p. 15

How to apply the taxonomy

• Applying TRAILS begins by specifying the task, criteria, documents, reference standard, model, deployment, and workflow, then mapping potentially relevant mechanisms and prioritizing tests and safeguards.The assessment is prospective, not a diagnosis; repeated runs under matching conditions investigate IC1, while deliberate changes to features that should not affect a judgement can test IC2. Observed failures do not by themselves identify their cause, and unobservable conditions should be reported as uncertain.Source: How to apply the taxonomy · pp. 16–17

Principal findings

• TRAILS identifies mechanisms behind repeat-run variability (IC1), hidden judgement failures (IC2), or both; 34 mechanisms meeting the IC2 criteria cannot be detected by repeated-run testing alone.A model may consistently reproduce a judgement seen during training or retrieved online, or be influenced by writing style, without basing its judgement only on supplied evidence and appropriate criteria.Source: Principal findings · p. 17

Accuracy

• IC1 testing generally requires repeated runs under specified conditions, whereas IC2 may require sensitivity tests, evidence verification, or other targeted assessments.Accuracy, repeatability, and judgement validity answer different questions; high accuracy or repeatability can coexist with hidden failures, so evidence support, output completeness, or task completion may also need assessment.Source: Principal findings · p. 17, Accuracy · p. 18

Challenges in applying the taxonomy

• Applying the taxonomy can require substantial resources, and reporting model, prompt, and workflow details does not itself test mechanisms that may cause IC1 or IC2.Reporting checklists and risk-of-bias frameworks serve complementary purposes, while complete methodological reporting remains necessary to interpret evaluations.Source: Challenges in applying the taxonomy · pp. 17–18, RelaƟon to reporƟng guidance and… · p. 18

ImplicaƟons on evidence synthesis findings

• When LLM judgements feed later evidence-synthesis stages, researchers should assess whether residual variability changes review findings or conclusions.Repeated screening decisions can yield different included-study sets and pooled estimates; human verification can intercept problematic judgements before they propagate, and oversight can target workflow stages where mechanisms may affect results.Source: ImplicaƟons on evidence synthesis… · p. 18, Reproducibility and human oversight · p. 19, CONCLUSIONS · p. 21

Relevance to current and evolving LLM systems

• Failure pathways remain relevant in current LLMs and newer judgement systems; structured outputs do not establish that judgements are correct or evidence-supported.Recent studies found effects from candidate position, long-context information location, and rubric wording; OpenAI reported that Structured Outputs prevented format failures but not failures in the information returned. TRAILS could complement Inspect AI with targeted tests [28].Source: Relevance to current and evolving LLM… · p. 19, ImplicaƟons for LLM evaluaƟon studies · p. 19

Research prioriƟes

• TRAILS requires mechanisms to have a testable connection to IC1, IC2, or both, but no validated standard measurement framework yet quantifies repeatability and mechanism-specific sensitivity across these tasks.The taxonomy does not estimate mechanism frequency or impact, may not be exhaustive, and has not undergone formal external validation; future research should establish suitable measures and test mitigations without introducing new errors.Source: RelaƟonship to other failure mode… · p. 20, Research prioriƟes · p. 20, LimitaƟons · p. 20

A. Models and technical features

• An artifact is an irrelevant input or procedural feature that may influence judgement; a failure mechanism is the process through which it or another condition can produce failure.The glossary also notes that practical context-window handling may depend on the application.Source: A. Models and technical features · p. 26
• Model versions, variants, weights, provenance, and open-weight availability describe distinct aspects of the system used to process inputs and generate outputs.Open weights do not necessarily make training data, code, or development history available.Source: A. Models and technical features · p. 26
• Sampling settings affect text selection, and temperature can affect response variation when the provider exposes it.Model routing selects a model, variant, or serving pathway; safeguards are intended to reduce undesirable behaviour but do not establish effectiveness.Source: A. Models and technical features · p. 26

B. Evidence and document processing

• Source evidence is the information to be used for the task; parsers convert documents or outputs, while OCR converts scanned text and recognition errors may alter supplied evidence.Chunking divides documents into smaller sections, while retrieval selects relevant documents or passages.Source: B. Evidence and document processing · pp. 26–27
• Multimodal processing handles multiple information forms, web access obtains information beyond supplied materials, and evidence traceability enables checking the cited or used source evidence.Source: B. Evidence and document processing · p. 27

C. Deployment settings and workflow architecture

• Deployment settings range from managed interfaces and vendor-managed work environments to direct APIs and user-controlled runtimes, with differing levels of researcher control.Provider-controlled serving infrastructure or orchestration may remain in hosted settings; user-controlled runtimes give direct control over weights, software, hardware, sampling, and context.Source: C. Deployment settings and workflow… · p. 27
• Workflow architecture describes how model calls, retrieval, tools, and processing steps combine; a single-call workflow produces its final output from one direct model call.An agentic workflow has dependent steps in which a model may select actions, use tools, assemble context, or decide when to stop.Source: C. Deployment settings and workflow… · p. 27

Multi-agent workflow.

• A multi-agent workflow uses more than one model instance or assigned role to contribute to, critique, or combine the final output.Tools provide external capabilities, while a persona instructs the model to respond from a specified role or perspective.Source: Multi-agent workflow · p. 27

D. Task and evaluation terminology

• A task includes the instructed activity and required outputs; a prompt guides performance, and a run is one complete performance of that task or workflow.A judgement is the task-level determination, while output may also include rationale, references, and other required information.Source: D. Task and evaluation terminology · pp. 27–28
• Accuracy is agreement with a reference judgement, repeatability is agreement across runs under specified conditions, and reproducibility is agreement when an independent analyst repeats the evaluation.Judgement validity concerns whether legitimate information determined the judgement; robustness or sensitivity testing changes a condition and examines the effect.Source: D. Task and evaluation terminology · p. 28
• Mechanisms can be detected, prevented, or mitigated, but mitigation only reduces likelihood or impact; some processes are unverifiable from an individual judgement.A failure mode is an operationally defined problem pattern, while controllability or testability means a condition can be held fixed or diagnostically compared.Source: D. Task and evaluation terminology · p. 28

Taxonomy inclusion criteria

• IC1 covers materially different judgements or required outputs across repeated runs under the same specified conditions; IC2 covers systematic effects not apparent from the judgement itself.Accuracy and repeatability do not rule out IC2.Source: Taxonomy inclusion criteria · p. 28

1.0 Eligibility criteria

• Eligibility was assessed first for sources and then for proposed mechanisms; sources could establish plausible mechanisms without directly demonstrating either inclusion criterion.The review limited sources to English-language publications and documentation, and documentation alone did not prove a judgement failure.Source: §1.0 · p. 29, Inclusion criteria · pp. 29–30

Exclusion criteria

• The review excluded researcher decisions outside the LLM workflow and reproducibility barriers unless they affected the LLM judgement or required output.Source: Exclusion criteria · p. 29

Illustrative examples of IC1 and IC2

• Illustrations distinguish variable ratings, stable but contaminated judgements, and cases combining both; differences in quotations alone do not establish IC1 if support remains equally valid.The examples are illustrations, not findings from included studies.Source: Illustrative examples of IC1 and IC2 · p. 30

Exclusion and consolidation criteria

• Proposed mechanisms were excluded for lacking a plausible pathway or being outside scope, and overlapping concepts were consolidated rather than counted again.Broad outcomes such as calibration error or subgroup differences were not treated as distinct mechanisms without a specific failure pathway.Source: Exclusion and consolidation criteria · p. 30

Search rationale and design

• The search was designed to identify and organise plausible mechanisms rather than estimate how often each failure occurs.It extended beyond biomedical databases because relevant reliability research can appear in preprints, conference proceedings, and technical documentation.Source: Search rationale and design · p. 31

Information sources and search approach

• The iterative search covered arXiv, Google Scholar, selected conference proceedings, and vendor or laboratory documentation, using screening and citation chasing to find further sources.Search terms were refined as relevant studies emerged; the approach was not an exhaustive systematic review.Source: Information sources and search approach · p. 31

Coding and analysis

• The taxonomy records each mechanism’s definition, proposed pathway, tests, mitigations, deployment applicability, and whether it can be observed, tested, or controlled.It distinguishes where a mechanism can occur from the access generally needed to investigate it; detecting an effect does not establish its underlying cause.Source: Coding and analysis · pp. 31–32
• Mechanisms are grouped into ten categories, with evidence strength rated Strong, Moderate, or Limited and read-and-evaluate evidence distinguished from broader research and vendor documentation.The taxonomy reports 38 parent mechanisms and two H.7 sub-mechanisms; proposed tests and mitigations are not assumed to be validated in every setting.Source: Coding and analysis · p. 32

Purpose

• Classifying both dimensions helps identify potentially relevant mechanisms and the conditions researchers can observe or control, but does not show that a mechanism occurred..Source: Purpose · p. 34
• Deployment setting describes how researchers access and run a model, while workflow architecture describes the sequence of operations; the dimensions are independent.The same API can support single-call, retrieval-supported, or agentic workflows, and earlier processing failures can change the evidence reaching the model.Source: Purpose · p. 34

Learning objecƟves

• The tutorial aims to help evidence-synthesis readers distinguish four deployment settings and four workflow architectures and recognize them in study methods.The intended readers know evidence-synthesis methods but may have limited LLM experience.Source: Learning objecƟves · p. 34, Purpose · p. 34

Step 1. IdenƟfy the deployment seƫng

• Deployment setting is the access route, not the model name, and is usually reported in the Methods section.The four settings are commercial chat, vendor-managed agentic environment, direct API, and user-controlled runtime.Source: Coding and analysis · p. 31, Step 1. IdenƟfy the deployment seƫng · pp. 34–35

1) Commercial chat

• Commercial chat offers a simple interface with relatively little infrastructure control; vendor-managed agentic environments allow configuration of files, tools, permissions, or instructions while vendors may retain control of underlying services.A tool-enabled chat interface alone does not establish that an agentic workflow was used.Source: 1) Commercial chat · p. 35, 2) Vendor-managed agenƟc or work… · pp. 35–36

3) Direct API (ApplicaƟon Programming Interface)

• Direct APIs can expose model versions, settings, prompts, and tools, whereas self-hosting offers more execution control but can add instability and does not necessarily reveal training history.Self-hosted researchers may control libraries, inference engines, hardware, precision, tokenizers, and model weights.Source: 3) Direct API (ApplicaƟon Programming… · p. 36, 4) Self-hosted model (user-controlled… · pp. 37–38

Step 2. IdenƟfy the workflow architecture

• Workflow architecture is the sequence of operations used to produce a final judgment, distinct from deployment setting; one deployment can support multiple architectures.A model call is one request and response, while a run is a complete execution that may contain several calls.Source: Step 2. IdenƟfy the workflow… · p. 38

A single-call workflow

• A single-call workflow sends the input once and returns an answer directly, with no intermediate steps; a long prompt or many separate records can still use this architecture.The defining feature is the path to each task result, not the number of records or questions.Source: A single-call workflow · p. 38

Retrieval-supported workflow

• Retrieval-supported workflows chunk and search documents, then supply selected passages to the model rather than necessarily providing the complete document.Because retrieval selects the evidence reaching the model, selecting different passages can add variability before model evaluation.Source: Retrieval-supported workflow · pp. 39–40

AgenƟc workflow

• In an agentic workflow, the model chooses subsequent actions, such as searching or using tools, based on intermediate results; predetermined multi-step processing alone is not necessarily agentic.The key distinction is model-directed choice during execution.Source: AgenƟc workflow · pp. 40–41

MulƟ-agent workflow

• Multi-agent workflows combine intermediate outputs from multiple LLM-based roles, which may extract evidence, assess bias, or independently evaluate evidence before voting.Agents may use the same model, and multi-agent systems can also be agentic when agents choose actions during execution.Source: MulƟ-agent workflow · pp. 41–42

Step 3. Read the Methods secƟon

• Read Methods descriptions for evidence of actual model access and workflow behaviour; software names, temperature settings, and chunking alone do not establish a classification.The worked example classifies OpenAI API access as a direct model API and passage selection from a vector database as retrieval-supported workflow.Source: Step 3. Read the Methods secƟon · pp. 43–44, Worked example: Classifying an… · p. 44

Step 4. Consult the taxonomy table

• For the worked example, the study is classified as a direct model API and retrieval-supported workflow, directing attention to mechanisms that its capabilities make possible.Distinguish mechanisms that are not applicable because a capability is absent from those that remain unclear because the methods do not report enough detail.Source: Step 4. Consult the taxonomy table · pp. 45–46

Summary

• The tutorial’s central rule is to classify deployment setting and workflow architecture independently, recording either dimension as unclear rather than guessing when reporting is insufficient.Deployment describes how the model was accessed; workflow describes how it completed the task.Source: Summary · p. 46

Supplementary 4: Full taxonomy of mechanisms, evidence, testing and mitigation strategies

• The taxonomy defines repeat-run variability and hidden judgement failure as distinct, potentially overlapping criteria, and organizes mechanisms by where they affect the task.Its descriptive aspects cover framing, evidence basis, judgement formation, expression, and substantiation; the taxonomy does not claim to be comprehensive.Source: Supplementary 4: Full taxonomy of… · p. 47

A.1 Personalisation and user history

• Prior conversation or stored user information can shift later judgements, including through irrelevant personalization; studies found effects from preceding conversational polarity.The cited passage also reports broader personalization findings and identifies separate, labeled memory fields as a way to reduce unwanted carry-over.Source: Supplementary 4: Full taxonomy of… · p. 48

A.2 Custom instructions

• Higher-level instructions can change final judgements despite an unchanged prompt, while fixed inappropriate instructions may create a stable shift that repeated runs will not reveal.More detailed task instructions performed better than simplified versions in research quality evaluation; record and version-control persistent instructions and test with and without them.Source: §A.1 · p. 48, §A.2 · p. 48

B.1 Routing variability

• Model substitution and mixed routing can be difficult to detect from outputs alone, and studies found third-party API identity inconsistencies and request-level backend mixing.Testing should verify the model or route through probes or serving logs rather than infer routing changes from output differences alone; pin model identifiers where available and record access details.Source: §A.2 · p. 49, §B.2 · p. 49

B.2 Model version drift

• Longitudinal model updates changed outputs and sometimes reversed correct answers even with tasks and prompts held constant.Overall performance could appear similar while the correct answers changed; record model identifiers and access dates, and treat known version changes as new conditions.Source: §B.1 · p. 49, §B.2 · pp. 49–50
• Serving-condition changes under greedy decoding produced up to 9% accuracy variation and 9,000-token response-length differences for one reasoning model.In repeated evaluations, one GPT-4o run received scores of 5, 3, and 1 in 52%, 14%, and 34% of runs, while four self-hosted models were consistent; greedy decoding reduced but did not eliminate pass/fail changes.Source: §B.2 · p. 50

B.3 Serving-time numerical nondeterminism

• Standard serving yielded 80 unique completions across 1,000 temperature-0 runs, while batch-invariant numerical operations made all 1,000 completions identical.Repeated inference also varied token probabilities across four models and several GPUs, especially for tokens with probabilities not close to 0 or 1.Source: §B.2 · p. 50

C.1 Stochastic sampling, decoding and seed

• Repeated generations under fixed sampling settings produced outputs whose repeatability varied by model, prompt, and dataset, showing that stochastic decoding can yield different outputs for the same task.Greedy or deterministic decoding and fixed seeds reduce sampling variation but do not address numerical nondeterminism; estimate repeat-run agreement from multiple independently generated judgements.Source: §B.3 · p. 50, §C.1 · p. 51

D.2 Inference-time retrieval

• Prior exposure or information retrieved from outside the intended evidence can influence a judgement without revealing whether it relied on the supplied material; the judgement may match or differ from a reference.Contamination estimates ranged from 1%–45% across six QA benchmarks and more than 15 LLMs, and contaminated subsets showed accuracy increases of up to 14% on C-Eval and 7% on HellaSwag, though effects were not consistent. Use behavioural probes or new matched items when training data are unavailable, and check retrieved sources where applicable.Source: §D.2 · p. 51, §C.1 · p. 51

D.3 Label Leakage

• When supplied material reveals or strongly signals the expected judgment, the model may rely on that cue rather than appraise evidence independently.In one evidence-synthesis example, removing textual supporting information reduced prediction of GRADE judgments from abstracts and authors’ conclusions.Source: contamination / online influence · p. 52

D.4 Prompt Contamination

• Hidden or unintended instructions in documents can redirect how an LLM performs a task, while OCR preprocessing prevented injections relying on hidden document-layer text in one study.Torrielli et al. found hidden instructions successfully steered positive peer review in 98% of trials across 100 papers, two chatbots, and five runs; susceptibility varied by model and document processing.Source: §D.3 · p. 52, §D.4 · p. 53

E.1 Document conversion distortion

• A PDF parser or OCR system can omit, reorder, merge, split, or misassign source content, leaving the model to judge a distorted representation.One extraction study reported 31% errors when GPT-4 accessed 10 study PDFs through a PDF-reading plug-in.Source: §D.4 · p. 53

E.2 Context Segmentation / chunk-boundary effects

• Segment boundaries can separate evidence that needs joint interpretation, and studies report better performance with adaptive or context-preserving segmentation than with smaller fixed segments.This is especially relevant to retrieval-supported pipelines, where segmentation affects which text can be retrieved and presented together.Source: §E.1 · p. 53, §E.2 · p. 54

E.3 Retrieval Variability

• Across retrieval studies, changing retrieved documents can change performance, but direct evidence that retrieval varies across identical runs and changes final outputs remains limited.With vectors and retrieval setup fixed, one study found identical retrieved documents and ordering across tested FAISS index types.Source: §E.2 · p. 54

E.4 Input modality conversion / preprocessing distortion

• Changing or degrading visual inputs—including resolution, noise, blur, filtering, compression, or frame selection—can affect model performance across multimodal tasks.Reported effects spanned biomedical and general multimodal tasks, multiple models, and multiple degradation types.Source: §E.3 · p. 55

E.5 Tokenisation effects

• Tokenisation can alter text representation or how much source material fits in context, but performance differences alone do not establish a read-and-evaluate failure.The same text translated into different languages required up to 15 times as many tokens across 17 tokenizers; evidence linking tokenisation to materially changed judgments remains a gap.Source: §E.4 · p. 55, §E.5 · p. 56

E.6 Multiple items or tasks in one model call

• Evaluating multiple criteria, records, or tasks in one model run can influence a target judgment; task-specific evidence synthesis studies found extraction accuracy differed across grouping methods and datasets.Testing the same item alone and alongside other items can reveal sensitivity, but clear labels do not establish that interference is removed.Source: §E.6 · p. 56

E.7 Long-context position and length effects

• Relevant evidence can be underused because of its position or surrounding context length, and moving evidence or changing context length is a sensitivity test rather than IC1.Some studies report performance losses with longer contexts and declines exceeding 20% in some middle-position conditions, though the pattern is not universal.Source: §E.7 · p. 57, §E.6 · p. 57

F.1 Prompt Sensitivity

• Meaning-equivalent wording changes can alter judgments; sequential screening tests found GPT-5.1 recall ranged from 0.72 to 0.32 across eligibility-criterion formulations.Differences became smaller when the screening workflow changed.Source: §E.7 · p. 57, §F.1 · p. 57

F.3 Few-Shot Example Order and Selection Bias

• Worked-example effects vary by task, criterion, and model, and broader in-context learning studies show example choice or order can make performance better or worse.Evidence synthesis studies have not tested whether changing examples or their order changes judgment accuracy.Source: §F.1 · p. 58, §F.4 · p. 58

F.4 Output-format failures

• Fixed output formats can interfere with judgments or required outputs, with effects varying by task, instructions, format, and model.Schema conformity can reduce structural errors but does not establish that judgment values or supporting explanations are correct.Source: §F.3 · p. 58, §F.4 · p. 59

F.5 Persona prompting effects

• Personas can add assumptions or priorities beyond the evaluation construct, and studies found their effects varied rather than consistently improving judgments.In one annotation task, adding a persona reduced agreement with human judgments for three models and increased it for one.Source: §F.4 · p. 59, §F.5 · p. 59

F.7 Option / Label Selection

• Changing the position or arbitrary label of unchanged response options can change the selected judgment, with effects varying across models and tasks.Balanced alternative orders can test whether position or identifier affects category selection.Source: §F.5 · p. 60, §F.7 · p. 60

Instability

• Fixed agentic runs can produce materially different judgments or outputs, while formal verification can distort requirements before downstream decisions; direct read-and-evaluate evidence for the latter remains limited.Different execution paths alone do not establish instability when the final judgment remains the same.Source: Instability · pp. 60–61

G.2 Verificationrepresentation distortion (conditional workflow failure)

• Formalizing natural-language evidence, instructions, or criteria can omit, alter, or misinterpret requirements, potentially making a downstream judgment appear verified despite an incorrect representation.The proposed test compares source requirements with the system-produced and independently verified representations, then checks whether distortions affect judgments or required outputs.Source: §G.2 · p. 61

H.1 Rationale–

• A correct or stable judgment does not ensure that its stated rationale supports it; evaluating the rationale can reveal failures that outcome accuracy alone does not distinguish.The mechanism concerns the stated rationale and substantiation, not whether that rationale exposes hidden internal reasoning.Source: §G.2 · p. 61

judgment unfaithfulness

• A rationale can be unsupported or fail to link evidence to an apparently acceptable judgement, and it does not reveal the model’s hidden reasoning.Generated explanations should not be treated as evidence of the process that produced a judgement.Source: judgment unfaithfulness · p. 61

H.2 Accepting unchecked assumptions

• An unverified premise can be accepted as evidence instead of checked against the source, potentially influencing interpretation or the final judgement.False-premise evaluations found models sometimes accepted incorrect assumptions, including when evidence needed to reject them was available.Source: judgment unfaithfulness · p. 62
• Hallucinated references include fabricated sources and substantially altered identifying details that make a real source citation invalid.Across 11 systematic-review searches, reported hallucinated-reference rates were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard among generated references.Source: §H.2 · p. 62

H.3 Hallucinated citations

• 90%–98% binary risk-of-bias accuracy coexisted with only 60%–65% of generated claims having source support.A fabricated or invalid reference can accompany an otherwise acceptable judgement, which cannot establish that the reference exists or supports the claim.Source: §H.3 · pp. 62–63

H.4 Evidence-extraction / quote instability and misattribution

• 83%-91% of quotes were verbatim valid across three models, but only 48%–79% were correctly attributed.Support was assessed by LLM judges rather than humans, so these results warrant caution.Source: §H.3 · p. 63

H.5 Output Completeness Failure

• Required components can be omitted, and omissions generally increase as the number or complexity of instructions rises.GPT-4o produced complete responses for 49% of queries; completeness declined from approximately 51% for queries with four or fewer requests to 36% for those with more than four.Source: §H.4 · p. 63

H.6 User-position / sycophancy effect

• A judgement may shift toward the user’s unsupported stated position, even when it conflicts with the evidence that should determine the judgement.In a source-based code-evaluation task, misleading user framing produced decision-flip rates of 40% to 72% and false-alignment rates above 90%; evidence-first prompting substantially reduced both.Source: §H.5 · p. 64

H.7 Failure to integrate task components (parent)

• Applying the published RoB 2 algorithm to LLM-generated signalling-question answers improved domain-level performance in several domains, while a pilot reported conformance of 43%, 51%, and 60% across three systems.Direct evidence isolating failures to integrate correctly interpreted components remains limited.Source: §H.7 · p. 65

H.7a Incorrect integration-rule application

• Prompt-injection attacks embedded in evaluated material could influence LLM-as-a-judge judgements, though success varies across models and workflows and routine-task frequency is uncertain.Instruction-like text is improperly treated as an instruction rather than evaluated as source content.Source: H.7a Incorrect integration-rule… · p. 65

H.8 Source-embedded instruction interference

• Keyword or phrase matching may be treated as sufficient proof of an appraisal criterion without checking its underlying requirement; direct evidence for this precise mechanism is limited.The cited studies show wording effects but did not directly test whether appraisal criteria were checked against all their requirements.Source: §H.8 · p. 65

H.9 Surface-cue substitution in criterion judgement

• Source writing or format can change judgements, even when the underlying information is unchanged and the presentation feature should not determine the rating.Studies found review scores changed with paraphrases and title style, and polished or authoritative-looking responses could be rated higher.Source: §H.9 · p. 66

I.1 Source writing and format effects

• Changing source or author information while holding content constant can change judgements; identical information was judged more trustworthy when labelled human-authored than AI-generated in some studies.Another study found models favoured responses labelled expert over human, LLM, or unknown.Source: §I.1 · p. 66

I.2 Source-label effects

• Irrelevant source labels or identity can influence a judgement despite unchanged content, and the judgement itself does not reveal that influence.Labels legitimately required by the task are excluded.Source: §I.2 · p. 66

J.1 Loss or distortion during context compaction

• Context compaction or truncation may remove task-relevant information before a later judgement, but cited studies do not directly establish incorrect or unsupported evidence-appraisal judgements from automated removal.The studies show that removing input information can affect model performance, while removing less relevant information can also improve efficiency without reducing accuracy.Source: §J.1 · p. 67

J.2 Removal of task-relevant context (truncation or pruning)

• Direct evidence in read-and-evaluate tasks remains limited; multi-turn instruction failures do not establish whether instructions were missing or ignored.The cited discussion distinguishes instructions not delivered from instructions available to the model but not followed.Source: §J.2 · p. 67

J.3 Loss of required instructions across turns

• Required instructions may be absent from later-turn inputs even without truncation or pruning, potentially affecting a judgement without being apparent from its rating.The mechanism may vary across executions with identical declared settings, or a fixed omission may systematically affect a judgement or required output.Source: §J.2 · p. 67, §J.3 · p. 67
• Testing context removal involves checking logs for task-relevant material and comparing retained versus pruned conditions under fixed configurations.Mitigation includes retaining critical inputs, logging pruning, and validating retained extracts or summaries.Source: §J.3 · p. 67
• Testing instruction loss involves inspecting assembled prompts at each turn; mitigation includes resending required instructions and versioning prompts.The passage also recommends independently checking the resulting judgement.Source: §J.3 · p. 67

TRAILS taxonomy checklist for LLM read-and-evaluate tasks

• The checklist records whether mechanisms were tested, disabled, reported, and mitigated.Its headings identify these as checklist fields.Source: TRAILS taxonomy checklist for LLM… · p. 73

20 Hallucinated evidence (H.3)

• For H.3, mark a feature disabled only if it was deliberately switched off; parenthesized codes identify items in Supplement 4, Table S1.The passage supplies checklist instructions rather than findings about hallucinated evidence.Source: §20 · p. 73

23 Failure to integrate task components (H.7)

• Failure to integrate task components is divided into failure to connect components (H.7a) and incorrect integration-rule application (H.7b).Source: §23 · p. 74

30 Randomness in token selection (C.1)

• The listed mechanisms include document splitting that separates related information, retrieval of different evidence, context-compaction loss or distortion, context removal, instruction loss across turns, and hidden model switching.Source: §30 · p. 74

SELF-HOSTED / USER-CONTROLLED MODEL RUNTIME

• The self-hosted or user-controlled runtime list includes tokenisation effects (E.5) and numerical variation during model serving (B.3).Source: SELF-HOSTED / USER-CONTROLLED MODEL… · p. 74
• Blank checklist boxes do not establish that a mechanism was absent or ruled out; H.7a and H.7b are sub-mechanisms of H.7.Source: SELF-HOSTED / USER-CONTROLLED MODEL… · p. 74

• Much prior research concerns general LLM tasks or system behavior rather than evidence interpretation against prespecified criteria, limiting generalization to complex appraisal tasks.A review identified 63 studies, with performance assessments concentrated in title and abstract screening, followed by data extraction and full-text screening.Source: Supplementary 5: Taxonomy evidence gaps · p. 75
• Evidence for variability across identical repeated runs remains less developed for many mechanisms than evidence from deliberately changed prompts or inputs.Accuracy or agreement studies also often do not establish whether intended evidence, decision rules, or supporting outputs were used.Source: Supplementary 5: Taxonomy evidence gaps · p. 75
• Further research is needed to measure mechanisms’ frequency and consequences across models, deployment settings, and workflows, especially where hidden system operations are hard to observe or control.Technical demonstrations of serving-time numerical differences do not establish their frequency or consequences in read-and-evaluate tasks.Source: Supplementary 5: Taxonomy evidence gaps · p. 75
• The effectiveness of available controls and prompting strategies remains uncertain for many applications, warranting repeated-run experiments, mechanism-specific interventions, and independent verification.Structured outputs address response-format compliance but do not guarantee correct generated values.Source: Supplementary 5: Taxonomy evidence gaps · p. 75