Gricea: An Open Science Platform for Conversational AI Research

1 Introduction

• Gricea represents conversational AI studies as deployable artifacts that researchers can inspect, reproduce, and extend, with the study representation also governing runtime execution.Immutable versions and reusable templates preserve designs as resources for cumulative research.Source: §1 · pp. 2–3

2.1 Infrastructure for Crowdsourced and Online Studies

• Reusable experiment frameworks such as oTree, Empirica, and jsPsych let study components support subsequent studies, while Gricea additionally couples procedure with conversational task behavior.Deliberate Lab [76] brings agent behavior into human–AI research infrastructure, but Gricea aims to represent a broader set of conversational AI studies.Source: §2.1 · p. 3

2.2 Infrastructure for Open Science

• Open-science infrastructure can preserve and share materials, but conversational AI reproduction also requires connecting procedure, interface, agent behavior, and materials.Gricea’s executable representation links study specification to implementation and makes the configured method available for inspection and reuse.Source: §2.2 · p. 3

2.3 Infrastructure for visual programming of CAI studies

• Visual and declarative tools expose computational choices, while Gricea brings this control into human-subject studies by connecting assignment, participant interaction, and measurement.Prior research includes visual composition and debugging [95], comparison of prompt and model variations [8], editable workflows [14], and structured specification and testing [34].Source: §2.3 · p. 4

3 Formative Analysis: The Science of Conversational AI Studies

• The analysis coded 57 papers for study focus, participants, variables, procedure, system components, and analysis methods to derive recurring requirements for infrastructure.The team collected 100 candidate papers and filtered for participant-facing conversational or agentic systems with sufficient study detail.Source: §3 · p. 4
• Staged diagrams, branches, and flowcharts communicate condition assignment, task sequences, and follow-up measures, motivating executable visual representations of study logic.The intended representation lets researchers design, inspect, communicate, and run a study without manually reconstructing its logic in code.Source: §3 · p. 4

3.1 What studies on conversational AI investigate

• Conversational AI studies span tasks such as co-writing, search, learning, recommendations, and planning, and examine outcomes including trust, reliance, persuasion, learning, and collaboration.The analysis identifies a need to configure interactions and collect self-reports, behavioral traces, and task outputs.Source: §3.1 · p. 4

3.2 The manipulation space of conversational AI studies

• The experimental object is a configured interaction condition combining agent behavior, interface, context, and procedure, including choices manipulated and settings held constant.The analysis synthesizes six interacting dimensions: Interface Condition, Agent Condition, Context & Grounding, Task & Modality, Domain & Audience, and Study Procedure.Source: §3.2 · p. 4, Table 1 · p. 5

3.3 The Procedural Anatomy of Conversational AI Studies

• Conversational AI studies couple structured procedures—such as consent, assignment, and measures—with an interactive runtime in which participants complete tasks.Prior examples include co-writing, conversational search, and repeated planning conversations with surveys and interviews.Source: §3.3 · p. 5
• Condition assignment and pre-task information can configure agent behavior and context, while in-task actions can shape interaction paths and post-task measures.This motivates connected Study Flow and Task Flow representations.Source: §3.3 · p. 5

3.4 Synthesis of Infrastructural Challenges

• Controlled conversational AI studies require coordinating interfaces, orchestration, models, assignment, deployment, and logging, creating engineering barriers for researchers without sufficient expertise or resources.Commercial systems may also expose or preserve behavior incompletely, so configuration and dependencies need to be inspectable alongside what participants encountered.Source: §3.4 · p. 6

• The paper derives five design desiderata from its formative analysis.They concern infrastructure for study design, execution, and reuse.Source: §4 · p. 6
• The desiderata address infrastructure for designing conversational AI studies.Source: §4 · p. 6
• The desiderata also address study execution and reuse.Source: §4 · p. 6

D1: Explicit Representation of Study Conditions.

• Represent the configured interaction participants experience, making explicit how procedure, interface, agent behavior, and context relate and what is manipulated or held constant.The representation should support checking whether assignment, task behavior, and measures correspond to the intended design.Source: D1: Explicit Representation of Study… · p. 6

D2: Coupled Support for Study Flow and Task Flow.

• Keep Study Flow and Task Flow distinct but connected: one specifies study stages and measures, while the other specifies within-task actions, responses, and branching.Researchers should be able to author and modify each flow separately while specifying how task information, endings, and outputs connect to study stages.Source: D2: Coupled Support for Study Flow and… · p. 6

D3: Lower Technical Barriers to Controlled Study Authoring.

• A no-code interface lets researchers without systems backgrounds specify and connect study components using staged diagrams, branches, and condition flows.These forms reflect recurring representations identified in the formative analysis.Source: D3: Lower Technical Barriers to… · p. 6
• Reusable participant interfaces, model integration, deployment, and data collection are intended to support executable studies without rebuilding surrounding infrastructure.The stated goal is to lower barriers so a broader range of researchers can contribute.Source: D3: Lower Technical Barriers to… · p. 6
• The platform is designed to reduce the bespoke engineering required to build and deploy controlled conversational AI studies.Source: D3: Lower Technical Barriers to… · p. 6

D4: Reproducibility Through Inspectable and Reusable Artifacts.

• Each study version retains its procedure, task logic, prompts, model and context settings, and participant-facing materials in executable form.Source: D4: Reproducibility Through… · p. 6
• Recorded interactions remain linked to the corresponding version so researchers can inspect both the authored condition and participants’ experience.Source: D4: Reproducibility Through… · p. 6
• Researchers can inspect, redeploy, adapt, and build on an artifact, with changes made explicit across versions.The passage presents this continuity as a shared reference for reproducing studies and building cumulative knowledge.Source: D4: Reproducibility Through… · p. 6

D5: Extensibility Across Tasks, Modalities, and Study Settings.

• The platform is intended to support text chat, coding, multimodal interaction, voice, longer-running procedures, and community-facing workflows.Source: D5: Extensibility Across Tasks,… · p. 6
• New task types, modalities, and settings should reuse the platform’s mechanisms for study execution and data collection rather than reimplementing surrounding systems.Source: D5: Extensibility Across Tasks,… · pp. 6–7
• Extensions remain configurable within the study representation, so resulting studies can be inspected, versioned, and reused in the same authoring environment.Source: D5: Extensibility Across Tasks,… · p. 7

5.1 Core System Abstractions

• Study Flow graphs govern participant progression, while nested Task Flow graphs specify interactive-condition behavior; publication serializes both for direct runtime execution.Study versions also include their configurations and assets.Source: §5 · p. 7

5.2 Researcher Authoring Environment

• No-code node-based editors let researchers author procedures and interactive conditions together while retaining separate control over each.Study-level variables connect procedural decisions to task settings, and publication locks the authored definition into a version while drafts continue to change.Source: §5.2 · p. 7

5.3 Participant Runtime and Instrumentation

• The runtime executes the published study’s procedural and task graphs, renders participant interfaces, and records configured interaction traces linked to the immutable study version.Supported tasks include chat, search, voice, coding, annotation, image generation, and split-view configurations.Source: §5.3 · pp. 7–8

5.4 Publication, Reuse, and Community Workflows

• Published study artifacts preserve prompts, task parameters, graph structures, assignment logic, and participant-facing materials as immutable, inspectable versions.Shareable links make the executable study available alongside its written description.Source: §5.4 · p. 8
• Researchers can inspect a study’s graph and configuration, experience it as a participant, then fork it to replicate or modify conditions while retaining the published source.The source and revised artifacts provide a shared reference for what changed and what remained.Source: §5.4 · p. 8

5.5 Architecture and Extensibility

• The architecture separates authoring, execution, and publication services so new task capabilities can reuse the shared study representation and its mechanisms.Extensions can add task node types and participant-facing components within the existing flow model.Source: §5.5 · pp. 8–9

6.1 Evaluation Rationale and Questions

• The evaluation asked whether Gricea could reproduce studies beyond its formative corpus, support researchers’ intended designs, represent diverse studies, and reveal remaining authoring frictions.It paired reconstruction of CUI 2026 papers with a usability study in which researchers and practitioners authored studies around their own questions.Source: §6.1 · p. 9

6.2 RQ1: Replication of Published CUI 2026 Studies

• 27 of 29 eligible CUI 2026 papers were replicated as full study configurations or conversational portions: 10 completely and 17 partially.Missing source materials and external dependencies limited complete replication; unavailable interview and survey materials prevented replication of the other two studies.Source: §6.2 · p. 10
• Incomplete reporting or unavailable materials occurred in 28 of 29 eligible papers, most often questionnaire or interview wording, agent configuration information, and stimulus materials.The categories appeared in 21, 15, and 11 papers respectively, with overlap; omissions limited replication when the information was needed to implement a study.Source: §6.2 · p. 10

6.3 Participants

• Ten participants from varied backgrounds completed a 90-minute session combining an assisted walkthrough and open-ended study authoring, followed by surveys and interviews.The team analyzed completion, timing, hints, and breakdowns descriptively, and used thematic analysis for think-aloud and interview data.Source: §6.3 · p. 10, §6.4 · p. 10, §6.5 · p. 10

6.6.1 Overcoming the Prototyping and Reproducibility Bottleneck.

• At least 3 to 6 months of effort was estimated as savings from using Gricea, and participants reported high likelihood to recommend it (M=6.8, SE=0.13).Participants also expressed confidence that Gricea-authored studies could be reliably reproduced by other researchers (M=5.9, SE=0.37).Source: §6.6.1 · p. 11

6.6.2 Participants from diverse backgrounds were able to author studies.

• Participants reported high ease of use (M=6.1, SE=0.31) and said Gricea’s connected visual building blocks matched how they reasoned about study design.They valued the visual canvas, unified procedure and runtime, and preview and validation features for checking what the platform would execute.Source: §6.6.2 · p. 11

6.6.3 Gricea supported RQs across the Conversational AI Design

• Participants operationalized diverse research questions (M=5.5, SE=0.40), spanning creativity, AI disclosure, search systems, coding agents, debate, tool use, fact-checking, and model tiers.The implemented studies included domain-specific protocols, and participants reported that Gricea accommodated their experimental manipulations.Source: §6.6.3 · p. 11, Table 2 · p. 12

6.6.4 High Degrees of Freedom and the Need for additional scaffold-

• High degrees of freedom created friction with variables, branching, and the transition between Study Flow and the more granular Task Flow.Participants requested stronger scaffolding, particularly for researchers without technical backgrounds; they also described Gricea as better suited to concrete designs than early-stage ideation.Source: §6.6.4 · p. 11

7.1 Conversational AI Studies as Research Artifacts

• Gricea treats the executable study as part of the methodological record, connecting materials and configurations to versions that teams can inspect, reproduce, and reuse.Participants valued versioning, templates, inspectability, and shareable links; the replication study also found that papers often lacked materials needed to reconstruct implementation.Source: §7.1 · p. 12

7.2 Related Research Questions Require Inspectable Study Configurations

• Related research questions can use substantially different tasks, procedures, participants, and measures, so inspectable configurations help researchers identify what to retain or vary.The discussion illustrates this with studies of uncertainty and reliance, and studies of conversational contact with an outgroup.Source: §7.2 · p. 12

7.3 Study Flow and Task Flow as a Methodological Abstraction

• Study Flow separates assignment, ordering, and measurement from Task Flow’s interactive behavior, allowing either to change while the other is retained.Task components can reuse surrounding study infrastructure, and the distinction may also apply to online studies of decision aids, visualizations, educational interfaces, and collaborative tools.Source: §7.3 · p. 13

7.4 Lowering Technical Barriers Requires Methodological Scaffolding

• Researchers from varied backgrounds implemented runnable studies, while evaluation surfaced a need for guidance translating research intent into variables, branching, randomization, and validation.The discussion presents scaffolding as a way to connect research questions to executable logic while retaining researcher visibility and control.Source: §7.4 · p. 13

7.5 Making Design-to-Implementation Fidelity Inspectable

• Researcher review, a published study definition, and version-linked execution records provide a common basis for examining whether implementation matches intended design.Structural validity alone does not ensure fidelity: a valid within-subject flow could still misrepresent an intended between-subject comparison.Source: §7.5 · p. 13

7.6 Open-Science Workflows for Cumulative Conversational-AI Research

• Connected study artifacts could help teams trace evolving hypotheses and distinguish shared configurations from deliberate extensions, but cumulative value depends on sustained community use.Synthesizing findings across studies also requires examining differences in populations, measures, and contexts.Source: §7.6 · pp. 13–14

7.7 AI-Assisted Research Through a Shared Study Representation

• A shared representation could let authoring agents propose or modify study flows while researchers inspect changes to assignment, behavior, measurement, and participant experience.The discussion distinguishes agents that help conduct research from agents whose behavior is part of an experiment, and proposes linking agent changes to source studies for review.Source: §7.7 · p. 14

• The authoring study included ten participants and does not capture the full range of prospective researchers or research practices.Source: §8 · p. 14
• The evaluation covers only a subspace of Gricea’s intended methods, excluding facilitated co-design, situated observation with personal devices or assistive technologies, and embodied agents’ physical behavior.Source: §8 · p. 14
• The published-study evaluation is limited to CUI 2026 and does not cover the full range of social-science experiments or complex combinations Gricea permits.Source: §8 · p. 14
• Planned work includes an inspectable AI copilot for proposing configurations, scaffolding for early ideation, and AI participant simulations for pre-deployment checks and pilots.Source: §8 · p. 14

A.1 Participant Details

• Table 3 summarizes 10 usable completed participant records spanning multiple roles and diverse conversational-AI study ideas.The participant summary is provided in Table 3.Source: Table 3 · p. 18, §A.1 · p. 18

A.2 CUI 2026 Replication Corpus

• Table 4 lists the 29 eligible CUI 2026 papers attempted for recreation.The table presents the complete replication corpus.Source: §A.2 · p. 18, Table 4 · p. 19