Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

• iCRs address underspecification or ambiguity that prevents an instruction follower from acting with enough certainty, despite generally understanding the instruction.They are a type of clarification request associated with the level of uptake.Source: §1 · p. 1
• A useful iCR dataset should provide spontaneous, specific requests that are frequent, diverse, relevant to task success, and regular rather than idiosyncratic.The desiderata emphasize player-initiated requests during real interaction and a distinct iCR category.Source: §1 · pp. 1–2
• The paper asks whether data-trained instruction followers can learn when to request more information and instruction givers can learn to recognize and respond to iCRs.Source: §1 · p. 2
• The authors annotate naturally occurring CoDraw iCRs, analyze their relation to game dynamics, and discuss two subtasks and models for handling them.They present CoDraw-iCR (v1) as a resource for data-driven clarification research.Source: §1 · p. 2

• Compared with clarification at perception and understanding levels, level-4 intention, uptake, or task-level clarification has received less research attention.The paper positions its work as contributing to this gap.Source: §2 · p. 2
• Earlier instruction-following studies include small corpora and larger annotated games, but some datasets do not clearly distinguish level-4 requests.Rodríguez and Schlangen (2004) report 22.17% level-4 CRs; the cited Minecraft Dialogue Corpus annotations report clarification-question and instruction-level-question categories [Narayan-Chen et al., 2019; Lambert et al., 2019; Shi et al., 2022].Source: §2 · pp. 2–3
• Several large clarification datasets use staged or synthetic collection, which abstracts away the joint decision-making and repair present in CoDraw.CoDraw-iCR retains sequential interaction and the instruction follower's spontaneous initiative to ask.Source: §2 · p. 3
• CoDraw-iCR combines many iCRs and dialogues with short games, a relatively constrained action space, and an annotation category specific to iCRs.Source: §2 · p. 3

3.1 Formalisation: Instruction-Following Dialogue Games

• A game consists of an instruction giver, an instruction follower, a hidden target scene, alternating utterance-action rounds, and a similarity objective for the final reconstruction.The instruction giver may or may not see the follower's incremental scene state.Source: §3.1 · p. 4
• Players are assumed equally competent and jointly motivated to maximize scene similarity while minimizing effort, so clarification is warranted when its cost is lower than the potential information gain.Source: §3.1 · p. 4

3.2 Tasks

• The two subtasks are binary decisions before the next utterance: the follower decides whether to ask after an instruction, and the giver decides whether the follower's utterance is an iCR.The ask decision uses dialogue context, the current instruction, and scene state; recognition uses dialogue context and the original scene.Source: §3.2 · p. 4

• CoDraw contains 9,993 English dialogues in which a teller instructs a drawer to reconstruct a scene by placing cliparts.The teller may peek at the drawer's canvas once; messages are limited to 140 characters, and scene similarity is scored from 0 to 5.Source: §4 · pp. 4–5
• Scenes use 6–17 clipart types drawn from 58 types, with multiple trees, hats, clouds, glasses, and balls creating opportunities for ambiguity resolution.Source: §4 · p. 5
• An utterance is annotated as an iCR when it requests information about earlier instructions to support action, likely because they were underspecified, ambiguous, or unclear.The annotation is a binary decision over drawer utterances.Source: §4 · p. 5
• To reduce annotation effort, rare utterance types received a one-utterance context window, while repeated forms were collapsed into one datum without context.The original CoDraw baselines assumed a silent drawer, leaving identification of necessary clarification and its generation for future work.Source: §4 · p. 5

5.1 Descriptive Statistics

• 8,807 iCRs make up 11.36% of drawer utterances, and two annotators labeled 13,727 utterance types with Cohen’s κ of 0.92.Among dialogues containing at least one iCR, 24.36% of utterances are iCRs; 59.45% of dialogues contain none.Source: §5.1 · pp. 5–6, Table 2 · p. 6
• With 7,260 types occurring only once, iCRs show diverse surface forms; frequent forms concern size, position, orientation, and disambiguation.Common initial bigrams include polar and wh-questions related to placement, resizing, flipping, and disambiguation.Source: §5.1 · p. 6, Figure 2 · p. 6, Figure 3 · p. 6
• Of 2,200 drawer vocabulary types, 1,468 occur in iCRs, whose frequent words include question terms and object-placement language.Non-iCR utterances more often include task, greeting, thanks, and acknowledgement words.Source: §5.1 · p. 6, Figure 4 · p. 7

5.2 Relations to Game Dynamics

• iCR rounds have more actions and greater average score improvement than non-CR rounds; the action difference is larger in post-iCR rounds.The authors report statistically significant differences in actions and score at turns realizing and following iCRs.Source: §5.2 · pp. 6–7, Table 3 · p. 7
• Nearly 60% of games contain no iCRs, which the authors take as evidence that clarification reflects the follower’s private decision-making.They attribute the naturalness of iCRs to synchronous human interaction without direct incentives to ask for clarification.Source: §5.2 · p. 7

6 Models and Experiments

• Both tasks predict the positive iCR label on imbalanced datasets, whose label distributions are summarized in Table 4.The experiments use the original CoDraw train, validation, and test splits.Source: §6 · p. 7, Table 4 · p. 7

6.1 Models

• The neural model combines scene, dialogue-context, and last-utterance representations, then uses a two-layer classifier to predict the iCR label.Its output is the sigmoid probability of the positive label; Figure 5 illustrates the architecture.Source: §6.1 · p. 8, Figure 5 · p. 8

6.2 Evaluation

• Average Precision and macro-average F1 Score evaluate the binary tasks, with logistic regression on utterance features and input representations as trivial baselines.The feature sets use utterance length and task-specific lexical indicators; Average Precision is used because ROC AUC can mislead on imbalanced data.Source: §6 · p. 7, §6.2 · p. 8

6.3 Embeddings

• AP= 0.91 and macro F1 Score= 0.86 on the validation set show that iCR occurrence is partly encoded in pretrained dialogue-context representations.The logistic-regression probe uses context before the peek action in dialogues containing a peek turn.Source: §6.3 · p. 8

7 Results

• Task 2 is comparatively tractable with logistic regression, while Task 1 remains challenging even for the neural model.Table 5 reports Average Precision and macro-average F1 Score on validation and test sets; feature-based baselines improve over random performance for Task 1 and considerably for Task 2.Source: Table 5 · p. 8, §7 · p. 8
• The image representation appears underused, although it should constrain possible actions in Task 1; pretrained clipart encoding may be inadequate.The authors call for investigation with other models and fine-tuning.Source: §7 · p. 9
• For Task 1, context is more relevant than the last utterance, and removing drawer utterances nearly harms performance as much as removing the whole context.Removing teller utterances causes little change; the authors infer that the model may exploit drawer-behavior patterns.Source: §7 · p. 9, Table 6 · p. 9

8 Discussion

• Predicting when to ask remains difficult across tested approaches, whereas identifying iCR utterances is uncomplicated even with logistic regression.The authors report similar outcomes after imbalance-handling techniques and model variations, aligning with Aliannejadi et al. (2021) and Shi et al. (2022) on the difficulty of predicting when to make a CR.Source: §8 · p. 9
• Task 1 does not yet satisfy regularity, and the current tasks model an overhearer rather than an agent learning clarification decisions alongside game actions.The authors expect clarification decisions to emerge more readily in representations that also make actions.Source: §8 · p. 9
• Drawer utterances appear informative, challenging the original CoDraw models’ “silent drawer assumption.”Removing drawer utterances may discard dialogue phenomena relevant to the game.Source: §8 · p. 9

9 Conclusion

• CoDraw-iCR (v1) supports investigation of instruction-level clarification at scale, and iCR and post-iCR turns imply different game dynamics relevant to successful play.The authors conclude that agents need to handle iCRs because they affect both dialogue acts and game moves.Source: §9 · p. 9
• The models perform well at detecting iCRs and provide a starting point for research on predicting when an iCR should be made.The proposed roadmap integrates iCR decisions into the full instruction follower so they are learned alongside game actions.Source: §9 · p. 9
• A second annotation phase will add fine-grained iCR form and content categories grounded in game objects, opening research on generation..Source: §9 · p. 9

10 Limitations

• CoDraw is a simplified, controlled instance, so broader conclusions depend on additional resources and collaborative efforts.The authors describe it as representative of instruction-giving/following dialogue games while noting that controlled data collections have inherent limitations.Source: §10 · p. 10
• Because CoDraw actions were reversible and tellers could peek, the authors cannot assess performance if players were unable to make CRs.Peek may incentivize tellers to give less exhaustive instructions and drawers to make approximate drawings for later refinement.Source: §10 · p. 10
• Without crowdworker IDs, the authors cannot study individual clarification strategies or determine whether dialogues without iCRs reflect no need for repair or lower willingness to ask.They note that experience may affect how players identify underspecified instructions and adapt them, but cannot investigate these effects.Source: §10 · p. 10
• The first annotation step labels utterance types rather than full dialogues; fine-grained annotation of each identified iCR in context is planned.The authors cite limited resources and the frequency of non-iCR utterances as reasons for this choice.Source: §10 · p. 10
• The models omit the drawer’s clipart gallery, an informative input; preliminary experiments did not improve results, and a suitable representation remains future work.Source: §10 · p. 10

A Data Statement

• CoDraw comprises 9,993 crowdworker dialogues, written in English and collected through synchronous, alternating-turn interactions with messages up to 140 characters.The dataset was released under a CC BY-NC 4.0 license; crowdworker demographic information was not released.Source: A Data Statement · p. 14

B Additional Corpus Analysis

• The analysis excludes task-level questions that do not directly refer back to a given instruction from its iCR examples.The authors give “what do you see in the sky?” and “anything else to change?” as negative examples.Source: B Additional Corpus Analysis · p. 14, Table 7 · p. 14
• Most iCRs occur early in dialogues averaging 7.7 rounds, and dialogues with more than five iCRs are rare.Figure 7 shows the round of occurrence; Figure 8 shows the number of iCRs per dialogue.Source: B Additional Corpus Analysis · p. 14, Figure 7 · p. 15, Figure 8 · p. 15
• Dialogues with more than 10 iCRs are outliers, while most dialogues have no more than 15 rounds.Figure 9 breaks down iCR counts by dialogue-round count.Source: B Additional Corpus Analysis · p. 15, Figure 9 · p. 15
• 14.58% of validation and 11.96% of test iCR utterance types also occur in training, while the corresponding utterance overlaps are 17.50% and 14.81%.The authors describe this overlap as low and desirable for reducing memorization shortcuts.Source: B Additional Corpus Analysis · p. 15
• Drawing actions are grouped as addition, edit, or no action; edits include deletion, movement, flipping, and resizing, each counted separately.Moves are computed from consecutive-round scene strings, assumed to represent the canvas when the drawer sends the message.Source: B Additional Corpus Analysis · p. 15

C Reproducibility

• Training uses the original CoDraw train/validation/test splits and selects the checkpoint with the highest validation Average Precision for test evaluation.Models train for up to 20 epochs; the final configuration follows a hyperparameter search, using the second-best setting for more stable learning curves.Source: C Reproducibility · p. 16

D Detailed Results

• Figure 10 plots validation-set average precision per round.Source: Figure 10 · p. 16, D Detailed Results · p. 17
• Table 8 presents hyperparameter search bounds and optimal values.Source: Table 8 · p. 17
• Figures 11 and 12 show validation-set ROC and precision-recall curves, respectively, for the best checkpoint.Source: Figure 11 · p. 17, Figure 12 · p. 17, D Detailed Results · p. 17