Source-linked AI summary
GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant
Zhuokang Shen, Yifan Wang, Hanyu Chen, Yunhang Shen, Wenxuan Huang, Gaoqi He, Jiao Xie, Rongrong Ji, Shaohui Lin
TL;DR
Multi-user group-chat assistants remain underexplored, especially for proactive intervention in complex, multimodal conversations. GroupGPT combines specialized agents with MUIR and achieves 4.72/5.0 response quality while supporting efficient, privacy-aware deployment.
Problem
Multi-user group-chat assistants remain underexplored, particularly for accurate, timely, and multimodal intervention in complex evolving conversations.
Method
GroupGPT uses an edge-cloud multi-agent framework with intervention judging, privacy sanitization, multimodal processing, and the MUIR benchmark for intervention reasoning.
Results
4.72/5.0 response quality was achieved in LLM-based evaluation, with participants generally receiving the chatbot positively.
Takeaways & Limitations
GroupGPT supports effective and efficient deployment of multi-user group-chat assistants, while MUIR enables quantitative study of intervention reasoning.
Takeaways & Limitations
Constructing MUIR training samples introduces label-leakage and inference-latency issues.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have enabled increasingly capable chatbots. However, most existing systems focus on single-user settings and do not generalize well to multi-user group chat interactions, where agents require more proactive and accurate intervention under complex, evolving contexts. Existing approaches typically rely on LLMs for both intervention reasoning and response generation, leading to high token consumption, limited scalability, and potential privacy risks. To address these challenges, we propose GroupGPT, a token-efficient and privacy-preserving agentic framework for multi-user chat assistant. GroupGPT adopts an edge-cloud model collaboration architecture to decouple intervention timing from response generation, enabling efficient and accurate decision-making while preserving user privacy through on-device processing of sensitive information. The framework also supports multimodal inputs, including memes, images, videos, and voice messages.To support evaluation of timing accuracy and response quality, we further introduce MUIR, a benchmark dataset for multi-user chat assistant intervention reasoning. MUIR contains 2,500 annotated group chat segments with intervention labels and rationales. We evaluate a range of models on MUIR, spanning from open-source to proprietary variants, including both LLMs and their smaller counterparts. Extensive experiments demonstrate that GroupGPT generates accurate and well-timed responses, achieving an average score of 4.72/5.0 in LLM-based evaluation, and is well-received by users across diverse group chat scenarios. Moreover, GroupGPT reduces the token usage by up to 3 times compared to baselines, while providing privacy sanitization of user messages before cloud transmission. Code is available at: https://github.com/Eliot-Shen/GroupGPT .
1 Introduction
Existing chatbot research remains focused on single-user interactions, while group-chat systems face token, multimodal, and evaluation limitations. GroupGPT addresses these challenges with a specialized multi-agent framework, privacy-aware edge-cloud processing, and the MUIR benchmark for intervention reasoning.
- Motivation: Existing group-chat frameworks commonly rely on passive, rule-driven responses and exhibit high token consumption, limited multimodal support, and insufficiently diverse evaluation.The identified multimodal gap concerns effective integration of images, audio, and videos in real-world group interactions.
- Proposed framework: GroupGPT combines specialized sub-agents with edge-cloud collaboration to support proactive, multimodal group-chat assistance while protecting sensitive user information.Its components include an Intervention Judge, Privacy Transcriber, Multimodal Processor, and Final Respondent.
- Evaluation: 4.72/5.0: the GroupGPT-instantiated chatbot achieved this overall response-quality score under LLM-as-a-judge evaluation and was well received by most participants.The evaluation targets intervention behavior and response quality in group-chat scenarios.
- Benchmark: MUIR contains 2,500 group-chat segments with human-annotated intervention rationales, enabling quantitative evaluation of multi-user chat-assistant intervention behavior.The paper presents MUIR as a benchmark dataset constructed through an innovative data-curation pipeline.
2 Related Work
Prior dialogue research has focused mainly on single-user LLM chatbots, while multi-user work has developed conversational strategies, personalization, and application-specific group agents. Parallel industry deployments and privacy research highlight the growing practical relevance and security challenges of group-oriented conversational systems.
- Dialogue-system research has primarily centered on single-user LLM chatbots for task-oriented domains such as ticket booking and restaurant reservation.
- Group-chatbot research has introduced conversational dimensions for what, when, and who to answer, alongside strategies for more natural response timing.
- Other multi-user systems address personalization through reflection–analysis–feedback workflows and socially informed group recommendations.
- Industry deployments increasingly apply proactive group agents across ChatGPT, TikTok, Yuanbao, WhatsApp, Telegram, and Feishu.
- LLM conversational agents face conventional data-security threats and memorization risks, prompting sanitization, differential privacy, unlearning, and PII detection or rewriting.
3 Framework Architecture
GroupGPT addresses timely intervention, efficiency, privacy, and multimodal understanding in dynamic group chats through a modular architecture. Lightweight edge components judge intervention and sanitize inputs, while processed context and chat-flow signals support final response generation.
- Design Challenges: The architecture must balance accurate intervention timing, inference latency, token efficiency, privacy, and contextual reasoning in evolving multi-user conversations.Premature or delayed interventions can respectively disrupt dialogue or reduce utility, while cloud processing can expose sensitive group-chat content.
- Core Components: GroupGPT combines five cooperative components to minimize token consumption, protect privacy, and maintain engagement in dynamic group chats.The architecture includes an intervention judge, privacy transcriber, multimodal message processor, chat frequency logger, and final respondent.
- Intervention Judge: The intervention judge uses a lightweight language model on a short-term context window to select whether and how the assistant should intervene, avoiding frequent LLM invocation.It infers an intervention action from the defined action set, including staying silent, emotional support, suggestions, and fact correction.
- Privacy Transcriber: The privacy transcriber independently detects and generalizes personally identifiable information before messages are used in downstream processing.It performs message-level sanitization by identifying sensitive spans and producing equivalent generalized transformations.
- Multimodal Processing: The multimodal message processor converts images, memes, videos, and voice messages into tagged textual representations, enabling a text-based LLM to generate responses from consistent context.Images and memes are classified and captioned, while videos and speech are directly captioned by multimodal models.
4 MUIR: Intervention Reasoning Dataset for the Multi-User group chat
MUIR is a privacy-conscious benchmark for intervention reasoning in multi-user group chats, built from anonymized real-world logs and collaboratively annotated with LLM labeling and human verification. Its construction converts multimodal conversations into text, segments them with overlapping windows, and produces interpretable intervention supervision while avoiding response-generation latency.
- Motivation: The dataset addresses limitations of existing resources, including small scale, outdated content, text-only formats, narrow topic coverage, and privacy constraints.Synthetic conversational data is also described as having limited modality diversity.
- Dataset overview: MUIR contains 2,500 group chat segments constructed from anonymized real-world logs and annotated through LLM labeling plus human verification.The dataset is specifically designed for intervention reasoning in multi-user group chats.
- Dataset construction: Construction begins by converting non-textual messages into text captions and segmenting long-term chat streams with overlapping sliding windows.Overlap mitigates boundary fragmentation and helps capture cross-turn dependencies.
- Annotation: GPT-4o identifies intervention types, rationales, responses, and message-level intervention positions within each formatted chat window.Annotations associate each intervention with a position, label, reason, and response.
- Training segment construction: Training instances retain intervention labels and rationales but exclude responses, because responses are generated at inference time and waiting for them would add latency.A decision range X controls the supervision labels and the proportion of stay-silent instances.
5 Experiment
Experiments show that GroupGPT’s small intervention models achieve strong MUIR performance, while user studies indicate high response quality, privacy preservation, and satisfaction. The framework also reduces intervention-related costs and benefits substantially from its core modules.
- MUIR Benchmark: GPT-4o trails human performance by 1.4% in accuracy and 1.9% in Macro-F1 on Chime-in Reason, while Chime-in Timing accuracy remains at or below 67%.Lightweight embedding models achieve moderate timing performance but generally only around 30% accuracy on Chime-in Reason.
- MUIR Benchmark: 86.3% accuracy and 81.0% Macro-F1 make Qwen-2.5-Instruct-3B the strongest overall model evaluated on MUIR’s intervention tasks.The finetuned small language models outperform GPT-4o on Chime-in Timing and are well-suited as Intervention Judges.
- Response Quality: 300 GroupGPT responses received high marks across relevance, coherence, fluency, and helpfulness under GPT-4 evaluation on a 1–5 Likert scale.The evaluation used a stratified random sample and reported both average ratings and score distributions for each dimension.
- Token Consumption: About 2B yearly input tokens would be consumed by a dedicated LLM-only agent for one active group chat, motivating GroupGPT’s token-efficiency analysis.The estimate uses approximately 1,500 daily messages and a sample of 500 logged group-chat messages.
- Ablation Study: Removing the Intervention Judge lowers chime-in timing from 4.38 to 3.11, whereas removing the Privacy Transcriber slightly improves overall scores.The ablation compares the full system with variants lacking the Intervention Judge, Privacy Transcriber, Chat Frequency Logger, or long-term window.
- User Study: 84% of users felt rewritten messages removed most private information, 88% said original meaning was preserved, and over 70% found contributions helpful and contextually accurate.Additionally, 64% felt the agent joined conversations at the right moments.
6 Conclusion
The paper presents GroupGPT, an agentic framework for multi-user chatbots, and introduces MUIR as a publicly available benchmark for intervention reasoning in group chats. Experiments demonstrate GroupGPT’s effectiveness and efficiency in multi-user chat scenarios.
- 6 Conclusion: GroupGPT addresses design challenges in multi-user chatbots through an agentic framework, while MUIR provides the first publicly available, high-quality benchmark for intervention reasoning in multi-user group chats.The benchmark is specifically designed to study intervention reasoning in multi-user group chats.
- 6 Conclusion: Extensive experiments demonstrate the effectiveness and efficiency of deploying GroupGPT in multi-user group chat scenarios.
A Future Work
The paper identifies several future directions for multi-user chat assistants, spanning group-level personalization, multimodal understanding, data generation, training, evaluation, multi-agent interaction, and digital personas. These directions aim to improve capability, scalability, alignment, and deployment relevance in real-world group chats.
- Future Directions: Future systems could model group-level personalization through collective memory and shared concepts, including persistent recognition of group entities across multimodal interactions.This extends personalization beyond individuals to shared group knowledge.
- Future Directions: Synthetic multimodal group-chat data could approximate real-world distributions, reducing the cost and time required to collect conversation datasets.LLMs and image or video generation models could support standalone or hybrid training paradigms.
- Future Directions: Future training could use human preference signals to build feedback datasets for RLHF or other post-training alignment techniques beyond supervised fine-tuning.The current intervention model uses supervised fine-tuning.
- Future Directions: Unified multimodal models could improve interpretation of text, images, videos, and memes while assigning different reasoning levels to meaningful and casual content.The proposal specifically distinguishes semantically meaningful images from casual content such as memes.
- Future Directions: More comprehensive benchmarks with diverse tasks and metrics could enable fair, reproducible evaluation and reduce reliance on costly large-scale user studies.The proposed benchmarks would provide standardized assessment protocols.
- Future Directions: Future work could explore multi-agent group chats with specialized roles and integrate digital personas that participate autonomously in socially coherent interactions.Examples of specialized roles include philosophers, artists, and teachers, while digital personas represent users’ virtual counterparts.
B Details of GroupGPT
GroupGPT is deployed with vLLM on two 3080Ti GPUs and profiled for end-to-end latency and GPU memory usage. Its average response latency is approximately 4.3 seconds, comparable to human reply behavior in group conversations.
- Inference latency and GPU usage: 4.3 seconds is GroupGPT’s average end-to-end inference latency from receiving a message to generating a response.The reported response time is perceptually on par with human reply behavior in group conversations.
- Inference latency and GPU usage: GroupGPT is deployed using two 3080Ti GPUs with 12GB memory each and the vLLM inference framework.
- Inference latency and GPU usage: The deployment systematically profiles end-to-end latency and GPU memory usage for each component.
C Details of Dataset · C.1 Topic Diversity
This section describes MUIR’s dataset coverage and construction-related statistics, with Topic Diversity examining the semantic range of collected group-chat segments through a word-cloud analysis. The conversations span everyday, technical, educational, entertainment, and coordination-oriented themes.
- C Details of Dataset: MUIR’s additional dataset details cover topic coverage, label distribution, intervention-distance statistics, and data-construction design considerations.
- C.1 Topic Diversity: The dataset analysis examines topic diversity across collected group-chat segments to characterize the conversations represented in MUIR.
- C.1 Topic Diversity: Figure 5 visualizes the conversation-message topics in MUIR using a word cloud.
- C.1 Topic Diversity: The word cloud highlights daily life as a recurring discussion theme in the dataset.
- C.1 Topic Diversity: Technical questions and learning resources are also frequent themes in the collected conversations.
- C.1 Topic Diversity: Entertainment and coordination among group members further contribute to MUIR’s conversational topic coverage.
C.2 Label Distribution · C.3 Intervention Distance Distribution
MUIR captures intervention decisions through labeled conversation segments, with a deliberately balanced label distribution and intervention distances reflecting natural, proactive group-chat dynamics.
- C.2 Label Distribution: MUIR annotates each conversation segment with a label indicating whether the assistant should intervene at that moment.The dataset focuses on the intervention decision problem in group conversations.
- C.2 Label Distribution: Setting X to 5 makes stay silent the second most frequent final label, encouraging a more balanced intervention policy.This proportion is explicitly controlled through Algorithm 4.
- C.2 Label Distribution: Overall, MUIR represents diverse and realistic intervention behaviors, emphasizing selective and context-aware participation by conversational agents.The label distribution highlights that assistants should not participate uniformly.
- C.3 Intervention Distance Distribution: Intervention distance measures the number of messages between consecutive intervention labels, capturing how interventions are separated in group conversations.The measure characterizes the nonuniform timing of assistant interventions.
- C.2 Label Distribution: Figure 6 summarizes MUIR’s intervention-label distribution, including final labels computed solely from training targets.The final-label distribution is presented alongside a complementary perspective in Figure 6.
- C.3 Intervention Distance Distribution: Figure 7 shows that most interventions occur after several messages rather than immediately after another intervention, reflecting natural conversational dynamics.The annotation protocol’s relatively proactive strategy makes intervention instances comparatively frequent.
C.4 Dataset Construction Pipeline · D Prompts
MUIR is constructed to train a lightweight judge that predicts whether and why to intervene on the latest group-chat message. The pipeline prevents label leakage, limits latency, and handles multi-turn requests through explicit routing and prompt design.
- C.4 Dataset Construction Pipeline: MUIR trains a lightweight intervention judge to determine when and why to intervene from recent group-conversation messages.The model predicts an intervention decision for the latest message based on a sequence of recent messages.
- C.4 Dataset Construction Pipeline: A naive fixed-window construction risks label leakage because neighboring samples can expose intervention labels and enable shortcut prediction.The dataset therefore treats leakage prevention as a central construction problem.
- C.4 Dataset Construction Pipeline: Including neither historical interventions nor responses causes repeated interventions and prevents the judge from recognizing already-addressed causes or multi-turn interactions.In such cases, the correct decision may be to stay silent, but the model can trigger intervention redundantly.
- C.4 Dataset Construction Pipeline: Including responses introduces inference latency and implicit label leakage, while including both responses and historical interventions also requires scarce realistic multi-turn interaction data.Responses correlate strongly with intervention reasons, and waiting for them delays inference.
- C.4 Dataset Construction Pipeline: The final pipeline uses a sliding window that preserves some interventions as historical context while reserving others as labels, preventing leakage without discarding intervention patterns.Historical responses are excluded from the judge context to reduce inference latency.
- C.4 Dataset Construction Pipeline: When users directly address the agent, the intervention judge is bypassed and the request is forwarded to the final respondent through an explicit prompt-based protocol.This handles user-initiated multi-turn interactions separately from autonomous intervention decisions.
- D Prompts: The detailed prompts used in the experiments are provided in the code repository.This repository contains the implementation-level prompt specifications for the reported experiments.
E Qualitative Analysis
The qualitative analysis presents visual examples of GroupGPT participating in group-chat discussions, using randomly sampled segments organized by topic type.
- The examples show GroupGPT participating in discussions within group chats.
- The group-chat segments were randomly sampled based on topic type.
- Figure 8 presents qualitative results of chat responses obtained by GroupGPT.