PaperlayerView exact revision on arXiv

Source-linked AI summary

Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing

Quanze Chen, Jonathan Bragg, Lydia B. Chilton, Daniel S. Weld

arXiv:1810.10733v1cs.HC

TL;DR

Difficult crowdsourcing problems remain challenging when aggregation fails and existing argumentation offers only limited, generic interaction. CICERO uses synchronous, multi-turn discussions targeted to workers’ reasoning. It improves individual accuracy more than one-shot argumentation and extends effectively to tasks with many answer choices, though at higher cost.

  • Problem

    Existing aggregation can fail on difficult problems, while one-shot argumentation lacks personalized back-and-forth and is impractical for many-answer tasks.

    Source: INTRODUCTION · pp. 1–2

  • Method

    CICERO pairs workers with opposing answers for contextual, multi-turn discussions, supported by synchronous implementation and argumentation training.

    Source: INTRODUCTION · p. 2, CONCLUSION & FUTURE WORK · p. 10

  • Results

    16.7 percentage points versus 6.8 percentage points improvement on Relation Extraction, and 98.8% accuracy versus 66.7% initial accuracy on Codenames.

    Source: CONCLUSION & FUTURE WORK · p. 10

  • Takeaways & Limitations

    Contextual, multi-turn discussion substantially improves worker accuracy and supports non-binary tasks that one-shot argumentation cannot practically handle.

    Source: INTRODUCTION · p. 2, CONCLUSION & FUTURE WORK · p. 10

    Synchronous discussions require workers to be online together, increasing waiting and requester cost.

    Source: DISCUSSION · p. 10

Abstract

from arXiv · show

Traditional approaches for ensuring high quality crowdwork have failed to achieve high-accuracy on difficult problems. Aggregating redundant answers often fails on the hardest problems when the majority is confused. Argumentation has been shown to be effective in mitigating these drawbacks. However, existing argumentation systems only support limited interactions and show workers general justifications, not context-specific arguments targeted to their reasoning. This paper presents Cicero, a new workflow that improves crowd accuracy on difficult tasks by engaging workers in multi-turn, contextual discussions through real-time, synchronous argumentation. Our experiments show that compared to previous argumentation systems which only improve the average individual worker accuracy by 6.8 percentage points on the Relation Extraction domain, our workflow achieves 16.7 percentage point improvement. Furthermore, previous argumentation approaches don't apply to tasks with many possible answers; in contrast, Cicero works well in these cases, raising accuracy from 66.7% to 98.8% on the Codenames domain.

INTRODUCTION

CICERO addresses weaknesses in one-shot crowdsourced argumentation by pairing workers with opposing answers for contextual, multi-turn discussions. Its workflow combines dynamic task allocation, synchronous discussion, and training designed to improve argument quality.

PREVIOUS WORK

  • Existing aggregation methods can fail on difficult problems when unreliable workers or a mistaken majority drive the result.The paper discusses majority voting, gold questions, and expectation maximization as approaches that may remain inaccurate on hard tasks.Source: INTRODUCTION · p. 1, PREVIOUS WORK · p. 2

Rationales & Feedback Can Improve Accuracy

  • MICROTALK improves accuracy through assess, justify, counterargument, and reconsider stages, but its one-round, pre-collected arguments are not personalized or interactive.Generic counterarguments are especially impractical when a task has many possible answers.Source: INTRODUCTION · pp. 1–2

Cicero DESIGN

  • CICERO pairs workers with opposing beliefs in a chat-style interface where multiple turns support context-dependent rebuttals, rule citations, and misconception resolution.The workflow focuses discussions on specific claims rather than generic justifications.Source: INTRODUCTION · p. 2, Contextual and Multi-Turn Discussion · p. 3

Instructions for the Task & the Meta-Task

  • Training covers both the domain task and argumentation practices, including rating justifications and emphasizing logical connections and reasonable conclusions.The paper also introduces justification training to help workers recognize and synthesize good arguments.Source: INTRODUCTION · p. 2, Instructions for the Task & the… · p. 3

INTRODUCTION

  • 16.7 percentage points versus 6.8 percentage points improvement in individual accuracy shows CICERO outperforming one-shot argumentation on Relation Extraction.The comparison is reported for CICERO’s contextual, multi-turn workflow against MICROTALK’s one-shot argumentation.Source: INTRODUCTION · p. 2
  • 98.8% accuracy, up from 66.7%, shows CICERO working on a many-answer Codenames task where one-shot counterarguments are impractical.The improvement is a 32.1 percentage point gain over initial accuracy.Source: INTRODUCTION · p. 2

Real-Time Crowdsourcing

  • CICERO-SYNC implements the workflow with real-time paired discussions, preserving conversational context while requiring both workers to participate simultaneously.The implementation was chosen to evaluate multi-turn argumentation in a controlled setting.Source: Real-Time Crowdsourcing · p. 3

Workflow Overview

  • The event-based workflow dynamically assigns assess or discussion tasks, matching workers with incompatible beliefs and preventing repeated pairings until disagreement opportunities are exhausted.Requesters can terminate earlier when agreement thresholds are reached.Source: Workflow Overview · pp. 3–4

CICERO-SYNC: A Real-Time Implementation

CICERO-SYNC realizes CICERO through simultaneous, real-time chat between matched workers. This preserves continuous multi-turn context but introduces waiting, coordination, and higher requester cost.

CICERO-SYNC: A Real-Time Implementation

  • CICERO-SYNC keeps both matched workers online in a chat-like discussion and assigns them no other tasks during the session.The synchronous design supports continuous dialogue spanning many turns while preserving context.Source: CICERO-SYNC: A Real-Time Implementation · p. 4
  • Synchronous discussions require workers to overlap in time, causing partner wait time and higher requester costs.The implementation also involves practical real-time communication, recruitment, and payment challenges.Source: CICERO-SYNC: A Real-Time Implementation · p. 4

Discussion Interface

CICERO centers its workflow on a familiar, synchronous chat interface for multi-turn worker discussions. The interface supports domain-specific features, including seeded justifications, alternate answers, and an exit mechanism.

Discussion Interface

  • The discussion task is CICERO’s defining workflow component, implemented as a free-form instant-messaging interface after pilot designs proved cumbersome.Workers can freely send and receive messages in the chat-style setting.Source: Discussion Interface · p. 4
  • Workers can end discussions to signal either consensus or that no agreement is possible.Source: Discussion Interface · p. 4
  • Domain adaptations seed Relation Extraction discussions with earlier justifications and let Codenames workers switch among alternate answers.The Codenames control accommodates questions with many possible answers.Source: Discussion Interface · p. 4
  • Workers required minimal training to use the discussion interface and quickly participated effectively in both domains.Source: Discussion Interface · p. 4
  • Relation Extraction instructions include five citable rules with bold shorthands for efficient discussion and justification references.Source: Figure 3 · p. 5

Optimizing Task Instructions

CICERO optimized instructions for fast, unambiguous reference during synchronous argumentation. Relation Extraction used concrete shorthand rules, while Codenames combined general guidance with reviewable training examples.

Optimizing Task Instructions

  • Explicit references to task guidelines were more persuasive, but inconsistent citation practices caused confusion during time-sensitive discussions.Source: Optimizing Task Instructions · p. 5
  • Relation Extraction guidelines were reorganized into five concrete, easy-to-cite rules with shorthands for unambiguous discussion references.Citing became more consistent, with workers frequently using the shorthands.Source: Optimizing Task Instructions · p. 5
  • Codenames instructions combine general guidelines with a way for workers to review relevant training examples when referencing them.Source: Optimizing Task Instructions · p. 5

Selecting and Training Effective Workers

CICERO uses training and gating to improve workers’ understanding of complex task instructions rather than relying mainly on post-hoc selection. The process trains workers while screening those who fail the required threshold.

Selecting and Training Effective Workers

Selecting and Training Effective Argue-ers

CICERO adds justification training so workers learn to recognize and produce useful arguments, with domain-specific exercises for binary and many-answer tasks. The training targets more critical discussion.

Selecting and Training Effective Argue-ers

Worker Retention and Real-Time Quality Control

CICERO-SYNC uses lobby feedback and task incentives to keep workers active during synchronous matching and discussion. Anti-cheating controls and peer reporting further support participation throughout experiments.

Worker Retention

  • Lobby status information and work statistics encourage workers to remain available while waiting for synchronous task matches.Workers may voluntarily exit after waiting without matches or completing enough tasks.Source: Worker Retention and Real-Time Quality… · p. 6

Quality Control

EXPERIMENTS

The experiments test whether synchronous multi-turn discussion improves individual accuracy, works when one-shot justifications are impractical, and exhibits contextual interaction. They evaluate these questions in binary Relation Extraction and multi-choice Codenames tasks.

Research Questions

  • CICERO-SYNC experiments ask whether multi-turn discussion improves individual accuracy over one-shot reconsider workflows.Source: EXPERIMENTS · p. 6
  • The experiments also test whether discussion works when one-shot justifications are impractical and whether exchanges are genuinely multi-turn and contextual.Source: EXPERIMENTS · p. 6

Domains

  • Relation Extraction supports comparison with one-shot argumentation, while Codenames evaluates CICERO on a non-binary, multi-choice task.Source: EXPERIMENTS · p. 6

Organization

  • The paper proceeds from experimental setup to domain-specific results, then analyzes discussion characteristics and effects on future accuracy.Source: EXPERIMENTS · p. 6

Experiment Setup

The experiments isolate contextual, multi-turn argumentation by sequencing training, assessment, and discussion stages under a blocking task assigner. A reproduced MicroTalk workflow provides the one-shot reconsider comparison.

Experiment Setup

  • The blocking assigner isolates assessment from argumentation by assigning workers one task type until its questions are completed.This controlled setting prevents interleaved concurrent tasks during evaluation.Source: Experiment Setup · p. 6
  • Workers complete gold-standard assessments before other questions, allowing quality evaluation and filtering after training and gating.Source: Experiment Setup · p. 6
  • Discussions are withheld until workers finish all assessments, so initial answers are collected before argumentation begins.Source: Experiment Setup · p. 6
  • The experimental discussion condition sequences Training, Assess, and Discussion stages, with tasks allocated on demand within each stage.Source: Experiment Setup · p. 6

Comparison Condition

  • A reproduced MicroTalk workflow adaptively assigns justification or reconsider tasks, showing a previously justified opposing answer when disagreement occurs.Source: Experiment Setup · p. 6

Recruiting and Incentives

The experiments recruited experienced Mechanical Turk workers and calibrated payments by task time and domain. Incentives were designed to discourage cheating and align with the MicroTalk comparison, while Figure 4 summarizes Relation Extraction accuracy improvement across question batches.

Recruiting

Evaluation Visual

  • Figure 4 compares average worker-accuracy improvement for individual question batches and the entire Relation Extraction question set.Source: Figure 4 · p. 7

Incentives

  • Payments were calibrated to approximately $7 per hour using pilot estimates, with a $1.00 bonus for completing training and the gating quiz.Source: Recruiting and Incentives · p. 7
  • Relation Extraction incentives paid for assessments, justifications, discussions, and reconsider tasks, with discussion bonuses differing by condition.Source: Recruiting and Incentives · p. 7
  • Codenames incentives combined base pay, correct-answer bonuses, and discussion participation and correctness bonuses.Source: Recruiting and Incentives · p. 7
  • The incentive structure aimed to discourage cheating and remain consistent with MicroTalk [10] and recent crowdsourcing research.Source: Recruiting and Incentives · p. 7

Relation Extraction Domain: Binary Answer

The study evaluates crowdsourced identification of the TAC KBP LivedIn relation, a binary judgment governed by deceptively complex annotation rules. It uses challenging questions, gating, and worker feedback to support reliable evaluation.

  • The task asks workers to judge whether a sentence supports that a person lived in a location, despite complex official annotation guidelines.The domain concerns structured information extraction from natural language.Source: Relation Extraction Domain: Binary… · p. 7
  • 23 challenging TAC KBP questions were divided into three experimental batches, with three simple gold-standard questions used for gating.Workers also completed a six-question quiz and had two attempts to pass the gating threshold.Source: Relation Extraction Domain: Binary… · p. 7

Multi-turn vs. One-shot Workflows

CICERO-SYNC improves worker accuracy through synchronous, contextual, multi-turn discussions, outperforming one-shot reconsideration on binary relation extraction and extending effectively to many-answer tasks. The discussions frequently use contextual argumentation patterns, although their synchronous implementation requires substantially more time and cost.

Multi-turn vs. One-shot Workflows

  • 16.7 percentage points versus 6.8: CICERO improves average worker accuracy more than one-shot reconsideration in Relation Extraction.The difference is statistically significant at p = 0.0143, with comparable initial worker quality across conditions.Source: Multi-turn vs. One-shot Workflows · p. 7

Codenames Domain: Multiple Choice with Many Answers

  • Multi-turn dialogue addresses many-answer scaling because workers can argue against a partner’s specific choice instead of pre-collecting arguments for every alternative.The Codenames task used around 7–10 candidate words alongside positive and negative clues.Source: Codenames Domain: Multiple Choice with… · p. 8

Evaluating on Multiple Choice Tasks with Many Answers

Discussion Characteristics

  • Codenames discussions averaged 8.3 turns, and more turns correlated with correct convergence; Relation Extraction averaged 7.5 turns without a significant turns–convergence relation.Relation Extraction discussions included seeded justifications, whereas Codenames used contextual turns to communicate and debate alternatives.Source: Discussion Characteristics · pp. 8–9
  • 77.6% of Relation Extraction discussions and 86.8% of Codenames discussions used at least one contextual argumentation pattern.The coded patterns included refutation, queries, counters, and references to previous reasoning; their apparent utility varied by domain.Source: Discussion Characteristics · p. 9
  • 79% of Relation Extraction answer changes followed persuasion by an argument, compared with 55% in Codenames, while other changes reflected task learning or concept-meaning agreement.The category distributions were 18%, 3%, 79% for Relation Extraction and 17%, 28%, 55% for Codenames.Source: Discussion Characteristics · p. 9

Do Workers Learn Through Discussion?

  • Post-test accuracy trended upward after discussion but showed no statistically significant interaction between experiment condition and individual evaluation accuracy.Relation Extraction accuracies were 66.7%, 69.3%, and 73.9% for baseline, reconsider, and discussion; Codenames reached 52.0% versus 46.7% for baseline.Source: Do Workers Learn Through Discussion? · p. 9

DISCUSSION

  • CICERO-SYNC improves accuracy at higher cost because synchronous workers wait for partners, with asynchronous execution proposed as a way to reduce overhead.A CICERO-SYNC discussion averaged 225.3 seconds versus 13.6 seconds for one-shot reconsideration.Source: Multi-turn vs. One-shot Workflows · p. 7, DISCUSSION · p. 10
Loading 1810.10733v1…