Source-linked AI summary
AI-Assisted Submissions in Online Research Are Rare and Highly Concentrated but Routinely Approved
Neil K. R. Sehgal, Manuel Tonneau, Dunigan Folk, Lyle Ungar, Sharath Chandra Guntuku
TL;DR
How often workers use AI assistance in online research remains unclear because prior estimates rely on self-report or inconsistent detection rather than direct observation. The authors link donated ChatGPT histories to Prolific submission records and find assistance in only 1% of submissions, with no detectable increase over more than three years.
Problem
Estimates of AI assistance range from 0% to 35% and are difficult to compare because they often cover selected task types and rely on self-report or detection tools that disagree.
Source: Introduction · p. 1Method
The authors survey 2,500 Prolific workers and link donated ChatGPT histories to records covering 712,930 submissions by 408 weekly ChatGPT users.
Source: Introduction · p. 1Results
68% of observed workers had used ChatGPT on a study, but assistance appeared in only 1% of submissions, with no detectable increase over more than three years.
Source: Discussion · p. 3, Results · p. 3Takeaways & Limitations
The findings do not support the view that AI assistance currently poses an existential threat to online research, but indicate platforms are poorly prepared if it becomes wider.
Source: Discussion · pp. 3–4Other AI services were not tracked, while selecting weekly ChatGPT users may raise estimates relative to Prolific overall, leaving the net direction of bias uncertain.
Source: Discussion · p. 4
Abstract
from arXiv · showhide
Online research platforms underpin much of what science claims about people, on the assumption that a human produced each response. Generative AI threatens that assumption by letting participants delegate responses to a chatbot, yet how often they do so remains unclear because prior estimates rely on self-report or automated detection rather than direct observation. We surveyed 2,500 workers on a high quality online research platform and directly observed AI assistance by linking donated ChatGPT histories to platform submission records of weekly ChatGPT users, covering 712,930 submissions across more than 127,000 studies. Although one in eight surveyed workers reported ever using AI on a study and 68% of observed workers had done so, assistance appeared in only 1% of submissions, with no detectable increase over more than 3 years. Assistance was often temporally localized within tasks and highly concentrated among workers, with 5% accounting for 64% of assisted submissions. Where assistance occurred, workers were virtually always paid, even when study instructions prohibited AI use, with overall approval rates similar to those for unassisted submissions. Half of assisted submissions involved bounded responses, outside the scope of the platform's LLM detector for open-ended responses. Taken together, our results do not support the view that AI assistance currently poses an existential threat to online research, but reveal limited payment consequences for prohibited use and gaps in platform safeguards, leaving platforms poorly prepared should that threat materialize.
Introduction
Prior estimates of AI assistance in online research vary widely and lack direct observation, leaving its prevalence, concentration, and payment outcomes unclear. The authors survey Prolific workers and link donated ChatGPT histories to submission records to examine these questions.
Introduction
- Published estimates range from 0% to 35%, but often come from single, selected task types and rely on self-report or detection tools that disagree.The authors identify three open questions: what share of submissions is affected, whether assistance concentrates among workers, and what its forms and payment outcomes reveal about safeguards.Source: Introduction · p. 1
- The study surveys 2,500 workers and links donated ChatGPT histories to Prolific records for 408 weekly ChatGPT users across 712,930 submissions and more than 127,000 studies.The records cover studies conducted from December 2022 to February 2026.Source: Introduction · p. 1
- A submission counts as affected when visible ChatGPT output answers or materially contributes to a response, regardless of permission.Source: Introduction · p. 2
Results
AI assistance appeared in a small share of observed submissions, often in localized exchanges, and was concentrated among a few workers. It spanned response types and formats, while assisted submissions were almost always approved for payment.
Results
- 68.4% of observed workers had ever used ChatGPT for assistance, while only 13.5% had assistance on at least 2% of their submissions.In the survey, 11.31% reported ever using AI on a Prolific submission when asked directly; the list experiment estimate was 14.24%, with no statistically significant difference.Source: Results · pp. 2–3
- 1.00% of 712,930 submissions involved AI assistance, and 40% of affected submissions contained one ChatGPT exchange; in the rest, messages spanned a median 29% of study duration.The rate was below most published estimates.Source: Figure 1 · p. 2, Results · p. 3
- 64.1% of affected submissions came from the top 5% of workers, compared with 78.2% from the top 10%.Excluding the top 5% reduced the affected-submission rate from 1.00% to 0.40%, a 59.4% relative reduction.Source: Figure 1 · p. 2, Results · p. 3
- Among affected submissions, 55.1% included subjective or person-specific content, 52.2% cognitive or knowledge content, and 52.6% bounded responses.Free-form responses appeared in 57.5%; formats and domains were nonexclusive, so shares could exceed 100%.Source: Figure 1 · p. 2, Results · p. 3
- 99.6% of AI-assisted submissions were approved for payment, including 99.7% of bounded and 99.5% of free-form submissions.All 60 submissions with a payment decision despite explicit instructions prohibiting AI use were approved and paid.Source: Results · p. 3
- AI assistance showed no detectable time trend over more than three years and was not associated with rejection after adjustment; each standard-deviation increase in daily ChatGPT use was associated with higher odds of assistance (OR 2.09, 95% CI 1.63–2.69).Higher rewards and longer task duration were also associated with higher odds of assistance.Source: Results · p. 3
Discussion
Across observed Prolific activity, AI assistance remained rare despite many workers having used it; the findings do not support an existential-threat view of online research.
Discussion
- 68% of observed workers had used ChatGPT on a study, but assistance appeared in only 1% of submissions, with no detectable increase over more than three years.The authors describe use as temporally localized within tasks.Source: Discussion · p. 3
- The findings suggest limited displacement of human input at current prevalence, but platforms are poorly prepared if AI assistance becomes a wider threat.The authors call for detecting repeated use across response formats and focusing on the small number of workers responsible for most assistance.Source: Discussion · pp. 3–5
- Approval exceeded 99% across assisted and unassisted groups, including submissions with confirmed prohibitions; affected free-form submissions were approved 99.50% and bounded submissions 99.73%.Unaffected submissions were approved 99.70% of the time.Source: Figure 2 · p. 4, Discussion · p. 4
- Prior simulations show task-dependent bias: attenuation from 0.40 to 0.36 SD at 10% assistance versus inflation of 2.3 percentage points at 4.4% assistance.The authors caution that the direction and magnitude of bias are not uniform.Source: Discussion · p. 4
- 5% of workers accounted for nearly two thirds of assisted submissions, while half of assisted submissions used bounded responses outside the platform detector’s open-ended-question scope.The authors recommend response-level checks across formats and worker-level vetting that aggregates quality signals across studies.Source: Discussion · p. 4
- The estimate may omit assistance through other AI services, while selecting weekly ChatGPT users may raise it relative to Prolific overall, leaving net bias uncertain.Study-specific AI policies were usually unknown, and the findings concern a single platform.Source: Discussion · p. 4
Materials and Methods
The methods combine a randomized survey with linked ChatGPT-history and Prolific-submission records to measure and characterize assistance.
Materials and Methods
- The study paired a randomized survey of 2,500 workers with linked ChatGPT histories and Prolific records, classifying overlapping exchanges and modeling prevalence and correlates.The classifier’s submission-level assistance validation against human annotations yielded Cohen’s κ = 0.76.Source: Materials and Methods · p. 5
Supplementary Information
The supplied supplementary-information passages identify the paper and its authors.
Supplementary Information
- The supplementary-information material is associated with “AI-Assisted Submissions in Online Research Are Rare and Highly Concentrated but Routinely Approved,” by Neil K. R. Sehgal, Manuel Tonneau, Dunigan Folk, Lyle Ungar, and Sharath Chandra Guntuku.Source: Supplementary Information · p. 7
S1 Survey of AI assistance
The survey used randomized direct and indirect questions about lifetime AI assistance, with a double-list procedure and checks for response-design effects and sample comparability.
S1.1 Participants and design
- 2,500 adults in the United States, United Kingdom, or Canada were randomly assigned to direct-report or one of four double-list cells about lifetime AI assistance.The sensitive item asked about using ChatGPT or another AI tool to help complete a Prolific study, irrespective of whether instructions permitted it.Source: S1.1 Participants and design · p. 7
S1.2 Comparison with the active Prolific pool
- The survey sample resembled the active Prolific pool on most measured demographics within each country, but workers with fewer than 100 prior approved submissions were underrepresented in all three countries.Source: S1.2 Comparison with the active… · p. 7
S1.3 Control-item selection
- Control items were chosen using pilot response distributions, endpoint frequencies, ambiguity, cross-pilot stability, and within-list associations, with high-, medium-, and low-endorsement items in each list.Source: S1.3 Control-item selection · p. 7
S1.4 Double-list procedure
- Double-list respondents counted true statements without identifying them, completed both lists, and encountered the sensitive item exactly once; list order and item order were randomized.Respondents first completed a worked count example, repeating it after reviewing instructions if initially incorrect.Source: S1.4 Double-list procedure · p. 8
S1.5 Statistical analysis
- Prevalence was estimated from the difference in mean counts between lists with and without the sensitive item, averaging the two list estimates equally.Percentile 95% confidence intervals used 10,000 respondent-level bootstrap samples stratified by the five randomized cells.Source: S1.5 Statistical analysis · p. 8
- Endpoint counts were low, tests found no evidence that the sensitive item changed control responses, and demographic balance tests detected no differences after Holm adjustment.The average list-order interaction was imprecise at 6.7 percentage points (95% CI, −3.6 to 17.2; P=0.213).Source: S1.5 Statistical analysis · p. 8
S2 Linking ChatGPT histories to Prolific records
This section builds a linked-history cohort of established weekly ChatGPT users, defines eligible Prolific task windows, and describes exchange annotation and validation. Its sample comparisons and selection stages bound the scope of the observational analysis.
S2.1 Participants and design
- 425 of 767 prior ChatGPT-history donors uploaded Prolific records, and 408 contributed at least one eligible task window after multiple selection stages.The recontact frame had selected established ChatGPT users and experienced Prolific workers.Source: S2.1 Participants and design · pp. 8–9
S2.2 Sample comparisons and selection into donation
- Within established users, donors and non-donors showed little use-intensity or demographic difference, but eligible donors differed from survey participants most in country composition.Among eligible donors, 76.5% reported using another AI service more than once a month, but those services’ histories were not collected.Source: S2.2 Sample comparisons and selection… · p. 9
- Follow-up donation was not materially associated with greater observed ChatGPT-use intensity or large demographic differences within the already selected cohort of established users.The follow-up invitation used coded linkage to prior research records, and the protocol was IRB-approved with informed consent.Source: S2.1 Participants and design · pp. 8–9, S2.2 Sample comparisons and selection… · p. 9
S2.3 Prolific records and primary task windows
- Eligible task windows lasted no more than two hours and fell entirely within each worker’s observed ChatGPT-history period, yielding 712,930 windows.Distinct studies were approximated by publisher-name and study-title combinations, which could merge separate studies with identical names.Source: S2.3 Prolific records and primary task… · pp. 9–10
S2.4 Temporal overlap and exchange construction
- A task window with at least one timestamped ChatGPT user message during the task counted as temporal overlap; 12,564 windows across 325 workers met this criterion.The 12,564 overlapping tasks contained 46,691 target exchanges, constructed from in-window user messages and subsequent in-window assistant messages.Source: S2.4 Temporal overlap and exchange… · p. 10
S2.5 Annotation
- The annotation classified task relationship, actionable input, assistant contribution, response target and format, visible AI policy, and evidence, with task endpoints using an any-positive rule.Detailed labels were reduced into broader response-type and response-format categories.Source: S2.5 Annotation · p. 10
S2.6 Annotation Validation
- Task-level κ was 0.715 for current-task interaction, 0.626 for usable task help, and 0.759 for ChatGPT-contributed responses; explicit-prohibition flag precision was 83.6%, with recall unestimated.The prohibition review covered 73 flagged submissions, and the precision estimate came from 61 confirmed prohibitions.Source: S2.6 Annotation Validation · pp. 10–11
S2.7 Statistical analysis
- Primary prevalence estimates were task-weighted with confidence intervals from 4,000 worker-level bootstrap samples; concentration, timing, and logistic regression were also analyzed.The regression covered submissions approved or rejected for payment and included task and worker-level predictors.Source: S2.7 Statistical analysis · p. 11
S3 Annotation Prompt
The annotation prompt separates task relevance, response opportunity, assistant contribution, supported response type and format, and visible AI policy. It relies on visible evidence and specifies rules for applying and recording those judgments.
S3.1 System prompt
- Each target exchange is labeled separately; prior-context messages may clarify it but do not receive their own label.The prompt instructs the classifier to use only visible evidence and not infer that output was copied, submitted, authorized, or prohibited.Source: S3.1 System prompt · p. 11
- The prompt makes five exchange-level judgments and one task-level AI-policy judgment.The judgments cover relationship, response opportunity, contribution, response target, response format, and visible policy.Source: S3.1 System prompt · pp. 11–12
- Task relationship is distinct from response opportunity: context-only input is not actionable, and an unestablished relationship makes downstream judgments not assessed.The prompt distinguishes an actionable request or item from instructions or stimulus context without an answerable item.Source: S3.1 System prompt · pp. 11–13
- Contribution labels distinguish model-added measured content from procedural help or editing that does not materially add what the study sought to measure.Response targets include personal, attitudinal, evaluative, knowledge, comprehension, reasoning, creative, and other categories.Source: S3.1 System prompt · p. 12
- Response format depends on the expected response space, not observed answer length; translation is free-form unless task instructions require fixed choices.A word or sentence limit alone does not make a response bounded.Source: S3.1 System prompt · pp. 12–13
- Visible AI policy requires task-specific instructions: explicit encouragement or prohibition determines the label, while no visible policy does not establish permission.An AI-themed study title alone does not establish a policy.Source: S3.1 System prompt · p. 13
S3.2 Output schema
- The output schema encodes categorical fields for relationship, opportunity, contribution, response target and format, and visible AI policy.The schema supplies named values and descriptions for these label sets.Source: S3.2 Output schema · p. 14