PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing the Future of User Feedback for Generative AI

Alisa Frik, Julia Bernd, Amitis Karami, Mohammad Tahaei

arXiv:2610.02631v1cs.AI

TL;DR

Usable, actionable post-deployment feedback mechanisms for GenAI remain under-specified despite the role of user feedback in model refinement. In collaboration with eBay, the researchers benchmarked existing approaches and developed and tested a prototype. They identified recurring design issues and translated them into recommendations, while noting that the eBay-centered participants limit generalizability.

  • Problem

    Practical guidance is limited for designing user feedback mechanisms that are usable for consumers and actionable for product teams.

    Source: §1 · p. 2

  • Method

    In collaboration with eBay, the researchers benchmarked industry approaches, conducted usability and expert evaluations, and developed and user-tested a feedback-tool prototype.

    Source: §1 · p. 3

  • Results

    The study identified recurring issues in discoverability, clarity, consistency, transparency, and perceived user value, and translated them into design recommendations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors judge the major prototype design patterns broadly relevant beyond eBay and e-commerce, despite the eBay-specific design and scenarios.

    Source: §3.5 · p. 7

    Heuristic evaluators and user-testing participants were eBay stakeholders and users, which limits the findings’ generalizability.

    Source: §3.5 · p. 7

Abstract

from arXiv · show

Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and features. When implemented effectively, giving such feedback can increase users' engagement with and trust in GenAI systems. Government regulations and industry guidelines call for post-deployment user engagement, but there is little guidance on designing mechanisms that are usable for consumers and provide actionable input for product teams. We conducted a multi-phase study as a collaboration between academic researchers and eBay. Our benchmark evaluation of current industry approaches identified common issues including lack of discoverability, unclear terminology, and inattention to user value. Based on these findings, we developed best-practice recommendations and designed and tested a prototype feedback-collection tool. The tool aimed to provide users with an efficient, flexible, and positive feedback-giving experience, and provide product teams with rich data on performance and potential problems in a usable format.

1 Introduction

Post-deployment user feedback can support GenAI monitoring and refinement, but practical guidance for designing usable, actionable feedback mechanisms is limited. This study benchmarks existing approaches and develops recommendations and a user-tested prototype with eBay.

1 Introduction

  • User feedback can provide cost-effective, scaled, contextual, and potentially more representative signals for refining model outputs, while supporting user engagement and trust.The paper also connects feedback to company reputation and efforts to address safety and ethical challenges.Source: §1 · p. 1
  • Existing governance guidance leaves limited practical direction for feedback mechanisms that are discoverable, legible, privacy-aware, and actionable for users and organizations.The paper describes this as an important but understudied layer of GenAI governance.Source: §1 · p. 2
  • The study asks how products collect feedback about GenAI features and what researchers, product stakeholders, and users consider best practices and usability issues.Source: §1 · p. 2
  • A multi-phase collaboration with eBay benchmarked industry approaches, assessed usability, and refined a prototype with stakeholder and user input.Across phases, the researchers identified recurring issues in discoverability, clarity, consistency, transparency, and perceived user value.Source: §1 · p. 3
  • The contributions include a taxonomy of feedback approaches, heuristic evaluation, a user-tested prototype, and design guidelines for GenAI feedback tools.Source: §1 · p. 3

2 Related Work

Prior work examines human feedback for training and governance, but offers limited operational guidance and user-centered evidence about GenAI feedback interfaces. This paper positions feedback design as part of the system that shapes user input and proposes concrete guidance to address that gap.

2.1 User Feedback Mechanisms for Generative AI

  • Machine-learning research studies human feedback for model training, but effective design patterns for capturing that feedback remain a critical gap.Binary preference votes can also leave unclear which output aspects shaped a rater’s choice.Source: §2.1 · p. 3
  • Feedback interfaces shape user choices and perceptions; prior work also motivates human control and agency, which the authors hypothesize well-designed tools could support.Source: §2.1 · p. 3

2.2 Existing AI Governance Guidelines about Feedback for GenAI Systems

  • AI governance frameworks recommend audits and risk assessments but do not explain how to use direct user feedback for continuous performance monitoring.Existing interface guidelines call for feedback and privacy transparency but give limited concrete direction on signals, entry points, wording, and options.Source: §2.2 · p. 4

2.3 User Research about Feedback Mechanisms for GenAI Systems

  • User-centered research on GenAI feedback mechanisms is scarce, and prior studies of feedback types and learning outcomes did not assess mechanism usability.A ChatGPT interface analysis found the design encouraged simple, frequent, performance-focused feedback while discouraging collective input and discussion Cooper and Zafiroglu [2025].Source: §2.3 · p. 4

2.4 Existing gap and our contribution

  • The paper identifies a lack of standardized frameworks and detailed guidance for capturing high-quality, output-specific feedback, then contributes evidence-driven analysis and concrete design recommendations.Its proposed contribution frames post-deployment feedback tools as human-centered governance mechanisms.Source: §2.4 · pp. 4–5

3 Methods

The study benchmarks GenAI feedback mechanisms, obtains cross-functional expert assessments, and develops and tests a prototype with eBay users. Its methods combine structured inspection and iterative user testing, with participant and product selection boundaries that shape generalizability.

3 Methods

  • The researchers combined a benchmark, usability inspection, stakeholder heuristic evaluations, and eBay user testing to refine the prototype and guidelines.Source: §3 · p. 5

3.1 Benchmark and Usability Inspection

  • The benchmark selected popular GenAI products across sectors using traffic rankings and inclusion criteria, then documented feedback mechanisms across 16 design dimensions.The resulting set comprised 32 products that asked users for feedback.Source: §3.1 · p. 5

3.2 Heuristic Evaluation

  • Twenty-four eBay experts assessed 12 products against 11 heuristics using a four-point Likert scale and provided issue, strength, and improvement comments.The heuristic set drew on Nielsen’s method and usability issues found during inspection.Source: §3.2 · pp. 5–6

3.3 User Testing

  • The prototype covered four eBay GenAI features, and 13 US-based buyers and sellers tested it through moderated sessions using issue scenarios and think-aloud walkthroughs.Researchers iteratively tested screen wording and variants, then used thematic analysis to refine the prototype and guidelines.Source: §3.3 · p. 6

3.4 Ethics and Confidentiality

  • Researchers used pseudonymous participant numbers, stored contact details separately, removed identifiers from transcripts, and restricted access to recordings and transcripts.The user study was determined exempt from IRB oversight, and participants consented electronically.Source: §3.4 · pp. 6–7

3.5 Limitations

  • Evaluated products were free, popular English-language services from a subset of industries, while user testing and expert evaluation were tied to eBay.The authors note e-commerce-specific scenarios and eBay design guidance, while judging the major design patterns broadly relevant.Source: §3.5 · p. 7

4 Results by Research Phase

The phase results show that feedback tools were common in the inspected products but often remained generic, while stakeholder evaluation and user testing surfaced usability concerns and perceived value in a GenAI-tailored prototype.

4.1 Benchmark and Usability Inspection

  • 80% of popular products in the 40-product sample had feedback mechanisms, but implementations often reused generic satisfaction or bug-reporting patterns that missed GenAI-specific issues.The overlooked issues included biased, offensive, or inaccurate responses and unrealistic generated images.Source: §4.1 · p. 7
  • Negative feedback was almost always requested, while positive input was rarer; feedback on images or whole chats was also less common than feedback on specific text outputs.Inspection also surfaced issues with entry points, feedback modalities and flow, and the submission step.Source: §4.1 · p. 7

4.2 Heuristic Evaluation

  • eBay evaluators detailed accessibility blocks, unintuitive menus, privacy-disclaimer gaps, and hard-to-notice cues; neither dedicated GenAI tools nor e-commerce features led in usability or satisfaction ratings.Evaluators also distinguished practices adaptable across products from those tied to particular use cases.Source: §4.2 · pp. 7–8

4.3 User Testing

  • Among 13 interviewees, some doubted feedback would be acted on, while participants said the GenAI-tailored prototype increased their confidence that the company would attend to and try to resolve issues.Participants included eight mainly buyers and five mainly sellers.Source: §4.3 · p. 8

5 Findings about Usability of Feedback Mechanisms

This section examines usability findings about feedback mechanisms, including how users understood their value, discoverability, purpose, scope, status, and use of identity or privacy controls. It also summarizes participants’ preferences and concerns around giving feedback and sharing identifiers.

5.1 User Value and Motivation; 5.2 Consistency

  • Forms often failed to show how feedback benefits users or appreciate their effort, while web and mobile versions inconsistently presented options, wording, privacy information, and confirmation.Participants appreciated a thank-you message and felt it could signal that feedback would be taken seriously.Source: §5.1 · p. 8, §5.2 · p. 8

5.3.1 Entry Point Signal; 5.3.2 Entry Point Placement; 5.3.3 Text Accompanying Entry Point

  • Thumbs were familiar for routine helpfulness votes but less clearly suited to serious or technical issues; feedback was harder to find when hidden, distant from output, or buried in menus.Participants noticed thumbs below responses more readily than above, while placement did not always clarify whether feedback concerned one output or the whole feature. Labels such as “Give feedback” could clarify that users could provide details, especially for complex outputs.Source: §5.3.1 · p. 9, §5.3.2 · p. 9, §5.3.3 · p. 9

5.4 Navigation; 5.4.1 Feedback Formats; 5.7 Clarity of Wording/Information

  • Mobile flows were often longer, and some forms were cluttered or lengthy; participants valued quick preset choices and free-text comments, while some were surprised when presets submitted feedback immediately.About half of participants thought presets could also simplify positive feedback.Source: §5.4 · p. 10, §5.4.1 · p. 10, §5.7 · p. 13

5.4.2 Structure and Presentation of Multiple-Choice Preset Categories; 5.4.3 Responsible AI

  • Participants often could not tell whether preset categories allowed multiple selections, and too many or unclear options could overwhelm or discourage users; responsible-AI concerns were sometimes hard to discover or report.Participants generally considered eight presets manageable, but “Other” without a free-text field could feel uninviting; serious issues also sometimes had separate reporting flows.Source: §5.3.1 · p. 9, §5.4.2 · p. 10, §5.4.3 · p. 10

5.4.4 Presentation of Free-Text Box; 5.4.5 Option to Include a Screenshot or Recording Demonstrating the Issue

  • Free-text prompts could imply comments were only for issues outside the presets, and small text boxes seemed less inviting; most participants preferred screenshots to recordings but wanted screenshots captured automatically.Many said they would rarely use a screenshot option, though some wanted to annotate screenshots if teams needed more context.Source: §5.4.4 · p. 11, §5.4.5 · p. 11

5.5 User Control and Freedom; 5.6 Visibility of System Status

  • Some forms made required fields unclear, and users often lacked ways to edit or withdraw feedback; participants understood a filled thumb as submission but were confused by clicking it again or seeing it revert.Most understood “Unsend” when using it, but many initially missed the link; some wanted more than six seconds or a confirmation before unsending.Source: §5.5 · p. 11, §5.6 · pp. 11–12

5.7 Clarity of Wording/Information

  • Participants preferred concrete feature-specific wording, while evaluators found jargon, internal names, and unclear presets impeded comprehension; concise versions were generally preferred over conversational ones.Participants found “unsafe” unclear and some legal or privacy categories hard to judge; the authors recommend clearer combined categories, including “Offensive or biased” and “Potential legal, copyright, or privacy issue.”Source: §5.7 · pp. 12–13

5.8 Clarity about the Process, Including Additional Information, Privacy Disclaimers, and Transparency

  • Products rarely explained what feedback would be used for, who would see it, what data would accompany it, or possible consequences; disclosures were often late, incomplete, hard to find, or difficult to navigate.Participants generally found the tested disclaimer clear, but some wanted specifics about third-party access and most expected interaction context to accompany feedback.Source: §5.8 · p. 13
  • Most participants were not very concerned about non-anonymous feedback; some valued user IDs for context or follow-up, while others doubted or misinterpreted the toggle.A few expected their data would be sent regardless of the toggle, and some assumed it requested follow-up.Source: §5.8 · p. 14

6 Design Recommendations

The recommendations pair recognizable, clear feedback paths with flexible ways for users to share input, while emphasizing user value, control, and transparency about data use.

6.1 User Value and Motivation

  • Use inviting language that explains user benefits, thank users after submission, and explain how their feedback will be used.If there is room to highlight user value only once, prioritize doing so before submission; the authors also recommend reassuring users that feedback will be taken seriously.Source: §6.1 · p. 14

6.2 Consistency

  • Keep feedback forms, options, disclaimers, wording, and visual treatments consistent across platforms and screen sizes where feasible.The recommendations also call for alignment with system design, content, and brand guidelines.Source: §6.2 · p. 14

6.3.1 Entry Point Signal

  • Use visible thumbs-up/down entry points, add a separate path for serious problems, and place feedback near the relevant output when possible.For complex features, offer separate entry points for response components or let users select which part they mean; placing feedback below all results may reduce responses when users do not scroll.Source: §6.3.1 · p. 14, §6.3.2 · p. 15

6.3.3 Text Accompanying Entry Point

  • Label feedback requests clearly, specify their scope when needed, and use accessible signals to help users discover and understand them.The authors recommend friendly bold labels, optional hover text on web, and accessibility-guideline adherence; labels such as “Like” may narrow attention toward aesthetics or user value rather than other concerns.Source: §6.3.3 · p. 15, §6.4 · p. 15

6.4.1 Feedback Formats

  • Offer binary feedback alongside optional presets and free text, with multiple selections and categories that include Responsible AI concerns.The recommendations include six to eight preset options, “Other” last, and nested options when space is limited; the authors also advise against removing rarely used Responsible AI categories.Source: §6.4.1 · p. 15, §6.4.2 · pp. 15–16, §6.4.3 · p. 16

6.4.4 Presentation of Free-Text Box

  • Make detailed feedback optional, provide a broadly labeled free-text field, and include screenshots only when other records cannot clarify the problem.Users should be able to exit easily and, where feasible, undo a submission; the recommendations also call for clear submission status and visible selections.Source: §6.4.4 · p. 16, §6.4.5 · p. 16, §6.5 · p. 16

6.6 Visibility of System Status

  • User testing found the least confusing pattern was one feedback-form submission per AI output, with thumbs disabled after submission.The rule applies whether or not a confirmation message is still visible.Source: §6.6 · p. 16

6.8 Clarity about the Process, Including Additional Information, Privacy Disclaimers, and Transparency

  • Use clear, concise wording and relevant privacy disclosures that explain data access and use, with further details available through a tailored “Learn more” resource.Although testing did not show a desire for anonymous feedback, the authors recommend testing that option because preferences may vary by culture, product, and context; a thank-you message may also link to support or troubleshooting.Source: §6.7 · p. 17, §6.8 · p. 17

7 Discussion and Future Work

The discussion frames post-deployment feedback as a socio-technical participation mechanism, while identifying limits in current designs and proposing broader, more inclusive engagement. It argues that feedback tools should complement—not replace—deeper participation in AI development.

7.1 Implications: The Value of Usable, Actionable Feedback Mechanisms for GenAI

  • Binary feedback can yield many but shallow signals, while richer feedback is often difficult to elicit and current mechanisms may not be usable or structured for product improvement.The authors identify sparse practical guidance on how post-deployment feedback should be designed.Source: §7.1 · pp. 17–18
  • The eBay collaboration produced practical recommendations and a flexible prototype, and the paper treats feedback mechanisms as socio-technical systems rather than interaction features alone.The authors say the collaboration grounded recommendations in industry needs and helped eBay optimize its feedback processes.Source: §7.1 · p. 18

7.2 Future Work: Expanding User Engagement beyond Feedback Mechanisms

  • Interface feedback is a provider-structured form of participation, and the authors caution that it should not be users’ only route to influence AI design.They characterize feedback collection as “work” or “consultation” in participation frameworks and call for exploring higher participation levels.Source: §7.2 · p. 18
  • Future work should explore ongoing participation and feedback processes embedded across deployment, governance, model improvement, and product decisions.Some study stakeholders preferred iterative engagement over one-off signals.Source: §7.2 · p. 18
  • Standardized feedback can streamline pipelines but may homogenize input; the authors recommend open-ended comments and other participation alongside preset categories.They note risks of amplified bias, suppressed perspectives, and values that do not fit particular contexts.Source: §7.2 · pp. 18–19
  • Future work should broaden participation and address accessibility and representativeness, including multimodal feedback and attention to potential sample bias.The authors note that participatory AI projects have focused on a few homogeneous user groups and suggest collecting metadata and sociodemographic information to help control for potential sample bias.Source: §7.2 · p. 19

8 Conclusion

The paper reports a collaborative study that examined current GenAI feedback practices, studied user views, and developed and tested a flexible prototype. It identifies common tool issues and offers recommendations for feedback collection.

8 Conclusion

  • The study combined industry-mechanism review, heuristic evaluation, research with eBay users, and development and user testing of a flexible prototype.The collaboration involved academic researchers and eBay partners.Source: §8 · p. 19
  • The recommendations emphasize recognizable entry points, language that conveys appreciation and user benefit, and choices among binary, preset, and free-text feedback.The suggested options let users choose their level of effort and engagement.Source: §8 · p. 19
  • The recommendations also call for clear signals that feedback was submitted and explanations of what will happen to it.Source: §8 · p. 19

Disclosure of Interests

The authors disclose eBay funding and employment ties, the company’s involvement in review, and steps taken to keep user testing independent. eBay plans to use some recommendations in a feedback survey template.

Disclosure of Interests

  • Two authors received an eBay grant, and two were eBay employees conducting the research as part of their regular employment.The project also used eBay features as examples and consulted eBay employees; heuristic evaluators were from eBay.Source: Disclosure of Interests · p. 19
  • eBay reviewed the manuscript and made edits to avoid disclosing confidential information, and plans to use some recommendations in a GenAI feedback survey template.Source: Disclosure of Interests · pp. 19–20
  • Non-eBay researchers planned, conducted, and analyzed prototype user testing, while eBay employees could not access records linking participant identities to responses.The researchers explained their role to participants to address concerns about giving offense or possible platform consequences.Source: Disclosure of Interests · p. 20

Use of Generative AI

The paper used generative AI to assist with prototype and manuscript text, with human review intended to preserve the original content and meaning.

Use of Generative AI

  • Figma Make converted human-composed prototype screens into clickable flows, which a human checked for functionality and unchanged text and visuals.The tool was used for the user-testing version.Source: Use of Generative AI · p. 20
  • ChatGPT-assisted edits smoothed some visual-guideline text, with human review and revision to preserve its intended meaning.Source: Use of Generative AI · p. 20
  • Gemini 3.1 Pro and GPT 5.5 Pro edited some paper passages for clarity, consistency, and grammar, subject to author review for fidelity.The original drafter reviewed the edits to preserve tone, content, and facts.Source: Use of Generative AI · p. 20

Data Availability

User walkthrough and interview transcripts are not publicly released because participants did not consent to sharing, even in de-identified form; the protocols are included in supplementary materials.

Data Availability

A.1 Dimensions for Taxonomy and Usability Inspection

The taxonomy records how feedback is presented and what it asks users to provide, alongside product and company context.

A.1 Dimensions for Taxonomy and Usability Inspection

  • The taxonomy records feedback entry points, invitation text, feedback targets, response types, and preset negative-feedback options.It also records whether Safety/Harm, Bias, and Accuracy appear among the options and whether detailed feedback is optional.Source: §A.1 · p. 25
  • It captures post-submission confirmation messages, persistent submission signals, editing or unsending options, process information, and login requirements.Source: §A.1 · p. 25
  • For each product, the researchers also documented its function, login requirement, user choice about interacting with the feature, company, and sector.Examples of product functions include generating text, images, or video and recommending products.Source: §A.1 · p. 25

A.2 GenAI Products Reviewed

The appendix lists reviewed GenAI products, describes the heuristic-evaluation template, and presents the final feedback guidelines. Together, these materials cover product examples, evaluation questions, and guidance for feedback forms and flows.

A.2 GenAI Products Reviewed

  • The benchmark includes search, productivity, messaging, and eBay features, including Bing and Google search summaries, macOS Writing Tools, and WhatsApp Sticker Create.Source: §A.2 · pp. 25–26
  • Asterisks identify products selected for heuristic evaluation; eBay Background Generator was excluded because it lacked a dedicated feedback mechanism as of March 2025.Source: §A.2 · pp. 25–26
  • Reviewed e-commerce examples include Google Shopping AI Overview, Google Shopping Try-On Models, and Shop App AI; Bing Copilot Answers is listed from another sector.Source: §A.2 · p. 26

B Template for Heuristic Evaluation

  • Table 1 illustrates the evaluation template's heuristic, guiding-question, rating, and written-evaluation fields; Table 2 contains the full heuristic list.Source: Table 1 · p. 27, B Template for Heuristic Evaluation · p. 27
  • The evaluation template asks reviewers to assess effort, timing of information, required versus optional fields, and ease of navigation, then record ratings, issues, positives, and recommendations.The task instructions ask reviewers to assume a user perspective and inspect web and mobile interaction recordings.Source: B Template for Heuristic Evaluation · pp. 27–28

C Model User Feedback Template and Guidelines

Loading 2610.02631v1…