PaperlayerView exact revision on arXiv

Source-linked AI summary

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

Milagros Miceli, Martin Schuessler, Tianling Yang

arXiv:2007.14886v2cs.HCcs.CVcs.LG

TL;DR

The paper investigates how annotators make sense of image data and how structures, standards, and decision-makers shape that process. Fieldwork at two annotation companies finds that hierarchical power relations shape classifications and are often naturalized, motivating power-aware documentation of annotation.

  • Problem

    The paper asks how annotators make sense of data, what structures shape that work, and who decides which classifications define data points.

    Source: §1 · p. 1

  • Method

    The authors conducted several weeks of constructivist grounded theory fieldwork and 24 interviews at two annotation companies.

    Source: §1 · p. 2

  • Results

    Power asymmetries in labor and client-service relationships shape labels, datasets, and systems, while workers naturalize classifications imposed by higher-ranked actors.

    Source: §6 · pp. 20–21

  • Takeaways & Limitations

    The authors propose power-aware documentation and reflexive practices to restore annotation context and support deliberative accountability.

    Source: §1 · p. 2

    The findings are bounded to image-data annotation in impact-sourcing companies; the framework could inform comparative research in other settings.

    Source: §5.3 · p. 20

Abstract

from arXiv · show

The interpretation of data is fundamental to machine learning. This paper investigates practices of image data annotation as performed in industrial contexts. We define data annotation as a sense-making practice, where annotators assign meaning to data through the use of labels. Previous human-centered investigations have largely focused on annotators subjectivity as a major cause for biased labels. We propose a wider view on this issue: guided by constructivist grounded theory, we conducted several weeks of fieldwork at two annotation companies. We analyzed which structures, power relations, and naturalized impositions shape the interpretation of data. Our results show that the work of annotators is profoundly informed by the interests, values, and priorities of other actors above their station. Arbitrary classifications are vertically imposed on annotators, and through them, on data. This imposition is largely naturalized. Assigning meaning to data is often presented as a technical matter. This paper shows it is, in fact, an exercise of power with multiple implications for individuals and society.

1 INTRODUCTION

The paper investigates how image annotators make sense of data and which conditions and actors shape classification. It argues that hierarchical power relations impose meanings on annotation and proposes power-aware documentation to restore context.

1 INTRODUCTION

  • The study asks how annotators make sense of data, what conditions and standards shape that work, and who decides classifications at each stage.The passage introduces these as the paper’s three research questions.Source: §1 · p. 1
  • The investigation combines several weeks of fieldwork at two annotation companies with 24 interviews and defines annotation as assigning meaning to data through labels.The interviews included annotators, management, and computer vision practitioners; annotation involves multiple actors and iterations, beginning when clients turn needs into instructions.Source: §1 · p. 2
  • Hierarchical structures and top-down impositions shaped by client and market demands profoundly inform data interpretation, extending analysis beyond annotators’ individual subjectivities.The paper uses symbolic power to analyze how these imposed meanings become naturalized.Source: §1 · p. 2
  • The authors propose power-aware documentation to restore context and argue that reflexive practices can improve accountability and preserve knowledge about data work.They also encourage situated, power-aware reflection in research on data creation and in researchers’ own processes.Source: §1 · p. 2

2 RELATED WORK

Related work treats data work as human-influenced, situated, and collaborative, and connects classification to power. This paper brings those strands together to examine how power structures shape annotation and the meanings encoded in datasets.

2.1 Data Work as Human Activity

  • Data work involves human discretion and interlocking design choices, while the paper frames annotation as situated collaboration between clients and annotation teams within organizational routines.The paper extends attention beyond individual subjectivity to power structures that enable imposed meanings and classifications.Source: §2.1 · p. 3
  • Prior annotation research has examined individual preconceptions and factors such as guidelines, annotation context, styles, and interactions among items in a batch.Geiger et al. also identify annotator background, training, qualifications, pre-screening, and inter-rater reliability as factors likely to influence annotations.Source: §2.1 · p. 3

2.2 Data, Classification, Power

  • Classification choices shape what datasets include or exclude, and the paper treats datasets as consequential representations of a worldview rather than neutral records.Decisions about what to collect and how to measure or interpret it can make some aspects visible while excluding others.Source: §2.2 · pp. 3–4
  • Symbolic power is authority to classify social reality and legitimize some definitions over others; its uneven distribution is tied to economic, cultural, and social capital.Arbitrary classifications can become naturalized when authority is recognized and their arbitrariness is misrecognized.Source: §2.2 · p. 4
  • The paper uses a relational view of data and power to examine how interactions among annotators, data, and corporate structures shape assumptions encoded in datasets.This approach treats data as produced through human intervention rather than as raw or independent of people.Source: §2.2 · p. 5

3 METHOD

The study uses constructivist grounded theory to investigate how annotators interpret data, what structures shape that work, and who decides classifications. It combines iterative observation and interviewing with systematic reflection on the researchers’ role.

3 METHOD

  • The research questions ask how annotators make sense of data, which conditions and standards shape that work, and who decides classifications for each data point.The questions also ask at what stages those classification decisions are made.Source: §3 · p. 5
  • The study follows a constructivist grounded theory approach in which data and theory are formed through researchers’ interactions with the field and participants.The method supports systematic reflection on researchers’ position, subjectivity, and interpretation during fieldwork and coding.Source: §3 · p. 5
  • Data collection and analysis proceeded iteratively through participatory observation and qualitative interviews, with each informing the other and constant comparison identifying differences and similarities across sites.Sensitizing concepts organized field inquiry without serving as hypotheses or preconceptions.Source: §3 · p. 5

3.1 Data Collection

Data collection combined open-ended, guided observation with interviews. Researchers used sensitizing concepts to keep fieldwork manageable and included expert interviews to learn about company structures and the wider market.

3.1.1 Participatory Observations.

  • Open-ended observations supported inductive theme development, while sensitizing concepts such as training and quality assurance helped orient observation.Observation ranged from shadowing to active participation at two annotation companies, where confidentiality agreements restricted disclosure of client information.Source: §3.1.1 · pp. 5–6

3.1.2 Qalitative Interviews.

  • Interviews included annotators and management, with expert interviews used to obtain information and assessments about company processes and the overall market.Interview partners could choose code names or were anonymized to protect their identities.Source: §3.1.2 · p. 6

3.2 Sample

The study sampled annotation companies and practitioners to trace how client needs become annotation tasks and standards. Its sites and informants span different company settings and roles.

3.2 Sample

  • The researchers chose hierarchical impact-sourcing companies to trace labeling decisions, then added larger firms and a dataset-using computer-vision company to examine task and quality-standard translation.The sample also reflected field access and a preconception that power tensions might be more prominent among vulnerable workers.Source: §3.2 · p. 6

3.2.1 S1: The Annotation Company in Buenos Aires.

  • S1 in Buenos Aires employed about 200 workers, including creators and reviewers, in a structured annotation workforce serving local and regional clients.The company also conducted content moderation and software testing; its workforce strategy was tied to impact sourcing.Source: §3.2.1 · pp. 6–7

3.2.2 S2: The Annotation Company in Sofia.

  • S2 in Sofia handled ten projects with three salaried employees and about 60 freelance contractors, serving mostly European and North American clients.The company focused on visual annotation and employed refugees and migrants; contractors generally worked remotely with flexible hours and project-based pay.Source: §3.2.2 · p. 8

3.2.3 S3: The Experts.

  • Three expert interviews with managers and a machine-learning director added perspectives on company structures, training, and annotation markets.The interviewees worked in Iraq, Kenya, and India.Source: §3.2.3 · p. 8

3.2.4 S4: The Practitioners.

  • Practitioners described managers as mediating client requirements into worker tasks; the researchers interviewed four roles at a Berlin computer-vision company using labeled images.That company was not a direct client of the sampled annotation firms.Source: §3.2.4 · pp. 8–9

3.3 Data Analysis

The researchers analyzed interviews, field notes, and documents through iterative grounded-theory coding, linking categories to the core phenomenon of imposition of meaning.

3.3 Data Analysis

  • The analysis combined 24 interview transcripts with field notes and client instructions, quality-assurance metrics, and impact assessments.The researchers systematically applied open, axial, and selective coding.Source: §3.3 · p. 10
  • Open coding produced 28 codes, which iterative axial coding refined into seven categories before selective coding combined several into “imposition of meaning.”The researchers repeatedly revisited material to test and revise their developing interpretation.Source: §3.3 · p. 10
  • The researchers connected salient categories to the core phenomenon as causal conditions, context, intervening conditions, strategies, or consequences.These relations were organized in a paradigm model.Source: §3.3 · p. 10

4 FINDINGS

The findings section frames visual annotation as interpretive work and previews three project cases alongside recurring practices and four salient observations.

4 FINDINGS

  • Visual annotation interprets image content and assigns labels; the observed work mainly involved segmenting objects and then labeling them.Labels can name previously distinguished objects and may add keywords or attributes that describe what constitutes a class.Source: §4 · p. 10
  • Semantic segmentation separates image objects into classes, while labeling names those classified objects.Keywords and attributes can further specify the meaning of each class.Source: §4 · p. 10
  • The authors present three especially relevant annotation projects, note practices and tensions consistent across projects and companies, and report four salient observations.Source: §4 · p. 10

4.1 Project 1: Drawing Polygons

In the vehicle-damage project, client instructions and review practices shaped how remote annotators marked images, while workers often lacked the product context behind the task.

4.1 Project 1: Drawing Polygons

  • The project used damaged-vehicle images for a client whose dataset purpose was unclear to the Bulgarian team; the founder believed it would support an early-stage insurance damage-reporting app.The reported aim was to train models to identify damage automatically.Source: §4.1 · p. 10
  • Ali selected and briefed annotators, reviewed their work, and answered questions, yet he lacked information about the product; limited pipeline awareness was common among annotators.Several interviewed annotators could not relate machine learning or artificial intelligence to their work, and one said they had not asked why the client needed the images.Source: §4.1 · p. 11
  • Annotators followed client instructions on a dedicated platform to select vehicle parts, draw damage polygons, and classify damage type and severity.The client restricted disclosure of the specific commands and labels.Source: §4.1 · p. 11
  • Eva and Ali used briefings, Slack support, and alignment meetings to address questions and inconsistencies and standardize labels across workers.Ali also summoned annotators to work through unsatisfactory polygons or visible inconsistencies.Source: §4.1 · p. 11
  • The client-side account described labeling problems as arising from misunderstood requirements and differing interpretations, especially when damage boundaries and severity were not straightforward.It emphasized shared understanding and standardization of workers’ interpretations.Source: §4.1 · pp. 11–12

4.2 Project 2: Building Categories

In a competitive test project, annotators built categories rather than following a client taxonomy, yet the client’s anticipated expectations still oriented their decisions.

4.2 Project 2: Building Categories

  • The gas-station task unusually asked annotators to create mutually exclusive vehicle categories, adding classes as new vehicle types appeared.The team began with five categories and added pick-ups, SUVs, and semi-trucks; questions about category boundaries remained.Source: §4.2 · p. 12
  • Unresolved label questions moved up the hierarchy, with the client holding the final say.Annotators asked a leader, then QA, and ultimately the client when earlier levels could not resolve questions.Source: §4.2 · p. 12
  • Despite freedom to shape categories, the team tried to anticipate what the client would value most.Nati described imagining the client’s interests while developing categories for the project.Source: §4.2 · pp. 12–13
  • Assigning one annotator per category reduced completion time but raised concerns about accuracy checks and reviewer capacity.Nati asked how accuracy could be checked when only one annotator handled each class and reviewers were insufficient.Source: §4.2 · p. 13
  • The project’s ambiguity challenged QA to infer the client’s preferred taxonomy while optimizing annotator performance for a competitive offer.The client would not accept the costs of cross-checking; QA saw an opportunity to offer valuable data and potentially sell more hours.Source: §4.2 · p. 13

4.3 Project 3: Classifying Faces

The face-labeling project used concise client instructions for sensitive categories, which workers described as straightforward despite concerns about ethics and communication.

4.3 Project 3: Classifying Faces

  • Client instructions required face boxes and labels for sex, age, and ethnicity, plus five freely chosen keywords per image.The project manager handled briefings, tracking, and bounding-box revisions, while the founder oversaw quality control without further provenance information.Source: §4.3 · p. 13
  • Annotators described assigning age, sex, and race labels as easy and obvious, even when the categories could be difficult to judge.Ali pointed to facial features such as wrinkles and said uncertainty would be referred to the client.Source: §4.3 · p. 14
  • Eva said the company usually followed client interpretations, while describing the project as controversial and raising moral questions about bias.She believed clients would probably not invest time or money in addressing these issues.Source: §4.3 · p. 14
  • Anna linked difficulty discussing sensitive categories with workers’ limited vocabulary for nuances and the challenge of explaining them.She also suggested that a lack of explicit curiosity among workers could contribute.Source: §4.3 · p. 14

4.4 Salient Observations

Annotation practices standardize client expectations through hierarchical controls, while those arrangements are often treated as natural and shaped by commercial priorities.

4.4.1 Standardization.

  • Quality was generally defined as meeting client expectations, with instructions and QA standardizing interpretation toward the intended product and revenue plan.The account describes standardization across both annotation instructions and quality assurance.Source: §4.4.1 · pp. 14–15

4.4.2 Layering.

  • Annotation involves hierarchical layers of workers, reviewers, QA, and sometimes additional managers or external evaluators, with different status and authority.The figure characterizes the layers as differentiated by payment, occupational status, and epistemic authority.Source: §4.4.2 · p. 15, Figure 1 · p. 15

4.4.3 Naturalization.

  • Top-down meanings were rarely perceived as imposed: labels seemed self-evident, and doubts were usually resolved vertically rather than through peer discussion.Belief that clients knew best about their products and data helped naturalize the classification hierarchy.Source: §4.4.3 · pp. 15–16

4.4.4 Profit-Orientation.

  • Companies sought faster, more homogeneous annotation, while market priorities could inscribe profit-oriented values even in sensitive classifications.Clear guidelines and smaller label sets also provided workers orientation; ethical documentation and quality control involve costs clients may not bear.Source: §4.4.4 · p. 16

5 DISCUSSION

The discussion argues that hierarchical power relations naturalize imposed classifications, shaping labels and datasets while diffusing accountability. It recommends power-aware documentation and reflection for practitioners and researchers, while limiting its findings to image annotation in impact-sourcing companies.

5 DISCUSSION

  • Workers often accept both client-imposed labels and higher-status actors’ authority as self-evident, allowing discretionary classifications to pass as natural.The authors describe this twofold naturalization as central to the efficacy of meaning imposition.Source: §5 · pp. 16–17
  • Multiple layers of classification and control standardize labels while making decision responsibility hard to identify; the process often serves economic interests rather than social responsibility.The authors caution that standardization is not inherently harmful, but can stabilize power structures when guided solely by profit maximization.Source: §5 · p. 17
  • Data classifications can shape visibility and resource decisions, while reproducing discriminatory categories, surveillance, or misclassification harms.The discussion links data assumptions to potential effects on individuals and communities, including people left unseen by residual-category erasure.Source: §5 · pp. 17–18

5.1 Implications for Practitioners

  • Practitioners should document dataset purpose, composition, actors, decisions, and the rationale behind taxonomies through a collaborative, power-aware process.The proposed records include client instructions, platform choices, and procedures for resolving ambiguity; disclosure checklists should be developed democratically rather than imposed from academia.Source: §5.1 · pp. 18–19
  • Documentation may support accountability, regulatory compliance, traceability, and organizational learning, though companies may resist its costs or treat some details as trade secrets.The paper identifies deliberative accountability, compliance, and persistent practical knowledge as potential benefits.Source: §5.1 · p. 19

5.2 Implications for (CSCW) Researchers

  • Researchers should examine organizational hierarchies alongside individual subjectivities, make data work visible, and reflect on their own interpretive position.The authors present constructivist grounded theory as a tool for reflecting on researchers’ perspectives and interpretations.Source: §5.2 · p. 20

5.3 Limitations and Future Work

  • The findings concern image-data annotation for machine learning in impact-sourcing companies; comparative research could examine other settings, including crowdsourcing.Source: §5.3 · p. 20

6 CONCLUSION

The conclusion reports that hierarchical classifications constrain annotators’ subjectivity and shape datasets through market-oriented standards and multiple layers of decision-making. It advocates power-aware documentation and analysis to make those dynamics visible.

6 CONCLUSION

  • The investigation uses constructivist grounded theory, fieldwork at two companies, and interviews to examine structures and standards shaping data classification.Source: §6 · p. 20
  • Annotators’ subjectivity is structurally constrained by classifications imposed from above, while briefings, guidelines, and quality control serve client and market demands.Dataset creation follows cost-effectiveness, output optimization, and label-standardization logics, often at the expense of ethical considerations.Source: §6 · pp. 20–21
  • Power asymmetries in labor and client-service relationships fundamentally shape labels, datasets, and systems, while workers naturalize imposed classifications and higher-ranking actors’ authority.Multiple actors and iterations contribute to meaning imposition and dilute responsibility for potential harms from arbitrary labels.Source: §6 · p. 21
  • The authors advocate power-aware documentation of the actors and rationale behind assigned meanings, and a similar perspective in CSCW research.They also call for researchers to reflect on their own position and power.Source: §6 · p. 21
Loading 2007.14886v2…