Leveraging Large Language Models for Collective Decision-Making

Motivation

• Meeting scheduling is a demanding collective-decision setting because members have diverse, nuanced preferences and existing tools cannot capture general constraints such as avoiding back-to-back meetings.The challenge is heightened by coordination across organizations and hybrid work contexts.Source: §1 · pp. 1–2

Contribution

• The system extracts individual preferences from conversations, consolidates them, proposes options, and explains its suggestions to support group decisions.Its four components are a database, intent extraction module, coordination module, and evaluation module.Source: §1 · p. 2

Evaluation

• The large-scale study uses LLMs to simulate employees with synthetic profiles and meeting scenarios, enabling controlled, scalable evaluation of communication efficiency, preference satisfaction, and balanced option quality.The authors note that simulated behavior may not perfectly replicate humans but supports experimentation at scale.Source: §1 · p. 2

Findings

• The system reduces interactions over coordination rounds while increasingly satisfying and balancing members’ preferences, and human participants report strong preference aggregation, reasoning, and confidence in its use.A second study asked participants to assess proposed options, generated justifications, and their confidence in using LLMs for collective decision-making.Source: §1 · p. 3

Broader scope

• The framework is intended to extend beyond meeting scheduling to group discussions and brainstorming where members have conflicting opinions or need creative input.The paper emphasizes collective rather than single-user interaction with LLMs in future work environments.Source: §1 · p. 3

2.1 Participants

• The requirements study recruited technology-company employees, receiving 45 effective responses from participants with varied roles and meeting workloads.Most participants had at least three years of experience, and 54% reported three to five meetings per day.Source: §2.1 · p. 4, §2 · p. 4

2.2 Usage Contexts and User Needs

• Meeting scheduling was considered an important copilot use case by 95% of participants, followed by group discussion at 94%, brainstorming at 78%, and event planning at 27%.The result motivated meeting scheduling as the paper’s case study.Source: §2.2 · p. 4
• Administrators coordinate frequent meetings and events involving internal and external participants, with time-zone differences and extensive communication among their common challenges.They also balance factors such as executive attendance, travel, budget, and space availability.Source: §2.2 · pp. 4–5

2.3 Existing Methods or Systems

• Traditional polling tools emphasize availability and majority choices, whereas the proposed system incorporates nuanced preferences and rationales into interpretable recommendations.The paper therefore treats direct comparison with polling methods as limited because the systems process different information.Source: §2.2 · p. 4, §2.3 · p. 5
• Traditional tools are less effective when external participants’ calendars are inaccessible, creating back-and-forth communication that can delay scheduling.The proposed system aims to streamline preference collection and aggregation in this setting.Source: §2.3 · p. 5
• Compared with Calendar.help and optimization-based decision support, the proposed system processes free-text preferences, generates explanations, and improves plans across conversational rounds.The authors describe these approaches as complementary because optimization packages handle complex hard constraints precisely, while the proposed system directly aggregates preferences and supplies rationale.Source: §2.3 · p. 6

3 SYSTEM DESIGN

• The framework models multi-member decisions as repeated rounds in which members privately communicate preferences and opinions while the LLM system proposes group options.The process is designed for decisions such as scheduling a meeting with external customers.Source: §3 · p. 6

3.1 Modules

• The four modules are intent extraction, coordination, evaluation, and database access, combining conversational preferences with background information about members.The database can include expertise, responsibilities, and meeting-coordination involvement.Source: §3.1 · pp. 6, 8
• The intent extraction module communicates with each member and converts the conversation into compact restatements of individual preferences, which form the preference profile for coordination.The coordinator uses the latest extracted preferences, group intent, and database information to generate options.Source: §3.1 · pp. 6–7
• Each option records the proposed decision, the members whose preferences it fulfills, and reasons for the suggestion; the evaluation module scores satisfaction and equity across members.Satisfaction is averaged across members, while equity uses the Gini coefficient of member-level values.Source: §3.1 · pp. 7–8

3.2 Coordination Process

• The decision candidate first maximizes the number of members whose preferences are met, then uses satisfaction score to break ties and replaces the candidate only when the new option improves a relevant measure.This rule minimizes the number of members whose preferences are entirely unfulfilled before optimizing aggregate satisfaction.Source: §3.2 · pp. 8–9
• At each round, the coordinator generates K options, retaining the prior decision after the first round and adding K−1 options that satisfy at least as many members.The study assumes no new member information requires database updates.Source: §3.2 · p. 8

• Meeting scheduling is used to study how the system coordinates diverse interests, balances tradeoffs, and assists consensus.Source: §4 · p. 9
• The system elicits each member’s preferences, sends them to a coordination module, and proposes meeting options for the group.An example run starts with an organizer’s meeting-purpose description, followed by preference conversations with each member.Source: §4 · p. 9
• In later rounds, members assess proposed options and the system refines them, such as narrowing a morning meeting to a specific time.Source: §4 · p. 9
• The system evaluates coordination through interaction efficiency, satisfied-member coverage, preference fulfillment, balanced quality, and the acceptability of its reasons and options.Source: §4 · p. 11
• Evaluation combines scalable LLM-simulated meeting scenarios with a subsequent human-participant assessment of reasoning and option suggestions.Synthetic simulations address implementation and scalability challenges associated with large user studies.Source: §4 · p. 11

5.1 Synthetic Data

• The study models a 34-employee synthetic medical start-up with roles, reporting lines, seniority, responsibilities, schedule preferences, meetings, and a social graph.The database contains public employee information and social relationships, while schedule preferences remain private and are elicited by the system.Source: §5.1 · pp. 11–12

5.2 Experiment Setup

• The evaluation varies meeting size, option count, database incorporation, baseline design, and paraphrased preferences across the coordination process.The experiments include schedule-preference-only settings and settings that also impose database and attendance constraints.Source: §5.2 · pp. 12–13
• The simulations use T=4 rounds, meeting sizes n∈{3, 4, 5}, and 120 meeting scenarios, with n>K to support progressive consensus refinement.Source: §5.2 · p. 13

5.3 Evaluation Metrics

• Performance is measured by average member-system interactions, satisfaction ratio, satisfaction score, and equity score.The metrics respectively capture communication efficiency, satisfied-member coverage, preference fulfillment, and balance across members.Source: §5.3 · pp. 13–14
• Figure 4 summarizes average percentage changes from round t=1 across 20 scenarios for T=4, K=2, and n∈{3,4,5}.Source: Figure 4 · p. 14

6.1 Effect of Meeting Size n

• Across meeting sizes, interactions decreased while satisfaction ratio and satisfaction score increased; for n=5, these changes reached ≈50%, ≈110%, and ≈120%, respectively.The equity score decreased for n=5 but fluctuated for n=3 and n=4.Source: §6.1 · pp. 14–15

6.2 Effect of Number of Options K

• Increasing options from K=2 to K=4 reduced the final satisfaction-ratio increase from ≈100% to ≈50% and generally weakened performance.The satisfaction-score increase was ≈100% for K=2, ≈50% for K=3, and ≈100% for K=4, while equity improved less as K increased.Source: §6.2 · p. 15

6.3 Effect of Incorporating the database

• Adding the database and attendance constraints worsened every evaluation dimension relative to schedule preferences alone.The interaction decrease fell from ≈50% to ≈25%, satisfaction-ratio growth from ≈110% to below ≈50%, and satisfaction-score growth from ≈120% to ≈75%.Source: §6.3 · p. 16

6.4 Comparison with Baselines and Robustness Checks

• The proposed conversational multi-round system outperformed two baselines in almost every metric, although intent extraction initially reduced first-round performance before later rounds produced considerable improvements.Source: §6.4 · p. 16

6 SIMULATION STUDY RESULTS

• The satisfaction-ratio improvement correlated with improvements in satisfaction score, equity score, and interaction efficiency.Source: §6.4 · p. 17

6.4 Comparison with Baselines and Robustness Checks

• Results remained almost unchanged after repeatedly paraphrasing schedule preferences, indicating robustness to paraphrasing.Source: §6.4 · p. 17, Table 2 · p. 17

7.1 Procedure and Tasks

• The survey used a synthetic meeting with n=3 members and K=2 options generated after the first coordination round, presenting preferences, options, satisfaction information, and reasons.Groups A, B, and C evaluated the scenario from the perspective of different meeting members.Source: §7.1 · p. 17
• Participants completed three evaluation parts covering member-specific preferences, all members’ preferences, and impressions of system use for meetings with n=3 and n=5.The final part asked participants about the suitable number of proposed options and their confidence in using the system for meeting scheduling.Source: §7.1 · p. 17

7.2 Measures and Analysis

• Numerical responses used a 5-point Likert scale, and averages were reported for all participants and groups with two-sided 95% confidence intervals.Confidence intervals were computed with Student-t distributions using degrees of freedom based on the aggregate and group sample sizes.Source: §7.2 · p. 18

7.3 Evaluation Results

• Participants generally rated the system’s reasoning above average, but noted that some options suited larger groups better and that preferences could carry different weights.Participants also disagreed about whether particular options satisfied members’ needs because they interpreted the same preferences and reasons differently.Source: §7.3 · p. 18
• Most participants preferred K=3 suggested options for both n=3 and n=5 meetings, despite simulations indicating that K=2 performed best because more options could complicate consensus.The reported preferences were 57% for n=3 and 64% for n=5; confidence in using the system ranged from 3.22 to 4.09 across groups.Source: §7.3 · p. 18

8.1 Interpreting the Results

• Approximately 50% fewer interactions occurred in most settings, because members accepted proposed options without further refinement in later rounds.The reduction concerned interactions between members and the intent extraction module.Source: §8.1 · pp. 19–20
• Satisfaction ratios generally increased across rounds, but larger option sets hindered later refinement and added attendance or database complexity reduced the improvement magnitude.Smaller K achieved higher satisfaction ratios, whereas integrated attendance constraints and databases still accommodated preferences less effectively.Source: §8.1 · p. 20
• Satisfaction scores improved across settings, especially with more members, whereas increasing options from K=2 to K=3 or 4 reduced scores and left equity effects inconclusive.The reported equity comparison for successively proposed options was inconclusive.Source: §8.1 · p. 20

8.2 Ablation Study

• The intent extraction module became beneficial after the first round by clarifying preferences and supporting option adjustments, while remaining robust to paraphrased preferences.Its initial-round performance was worse than the comparison baselines, but its later-round effect was positive.Source: §8.2 · p. 21
• Several coordination rounds outperformed a single-round process across all metrics because the module could solicit additional opinions and refine the decision candidate.Human assessments also aligned with evaluation-module scores, supporting its measurement of satisfaction.Source: §8.2 · p. 21

8.3 Challenges and Limitations in System Implementation

• Practical deployment faces token limits, many LLM requests, latency for large meetings, changing preferences, manipulation attempts, and weaker performance with social graphs or mathematically encoded calendars.The system does not implement preference tracking, and database or hard-calendar constraints degraded performance.Source: §8.3 · pp. 21–22

8.4 Privacy Concerns

• Private member communication can expose sensitive preferences through generated explanations, making redaction, filtering, or preference perturbation relevant privacy mitigations.The concern includes undisclosed preferences, sensitive attributes, and confidential information available through the database.Source: §8.4 · p. 23

8.5 LLM-simulated Users versus Human Users

• LLM-simulated users cannot fully capture human inconsistency, emotion, trust concerns, privacy behavior, or strategic manipulation, limiting how directly simulation results transfer to human settings.The authors identify these behavioral differences as difficult for LLMs to simulate accurately.Source: §8.5 · p. 23

9.1 Collective Decision-Making in Work Environments

• Meeting scheduling research addresses complex, distributed coordination, but existing assistants often cannot handle diverse free-text preferences and organizational asymmetries.The paper contrasts its approach with Calendar.help, which primarily considers workers’ calendars.Source: §9.1 · pp. 23–24
• Scheduling tools that protect focus time complement this system’s aim to reduce non-essential coordination work and accelerate collective tasks such as brainstorming.The paper presents this relationship as complementary rather than as a direct comparison.Source: §9.1 · p. 24

9.2 Understanding Human Preferences Using Large Language Models

• LLMs have been used to infer individual preferences and aggregate group opinions, while this system extracts preferences from member dialogues and generates group suggestions without specifically training preference models.The paper contrasts its approach with AGREE and fine-tuned consensus-generation methods.Source: §9.2 · p. 24

9.3 Simulating Human Behavior Using Large Language Models

• LLM-simulated agents support scalable studies across social, market, interview, and crowdsourcing settings, reducing the logistical costs of recruiting and managing human participants.The paper uses simulated members to evaluate its system at scale, addressing the time and cost of human user studies.Source: §9.3 · pp. 24–25

9.4 Evaluating the Performance of Large Language Models

• Existing evaluation frameworks assess LLM capabilities through multiple metrics or interactive human feedback, whereas this work focuses on coordination metrics using large-scale simulated discussions plus a smaller human study.The paper distinguishes coordination evaluation from frameworks designed primarily to assess general LLM capabilities.Source: §9.4 · p. 25

9.5 AI and Collective Decision-Making

• Related AI research spans graph mining, consensus generation, communicative agents, preference negotiation, and multi-agent debate, while this paper evaluates collective decision-making through LLM-simulated agents.The paper positions its contribution within broader machine-learning approaches to decision-making and agent communication.Source: §9.5 · pp. 25–26

10 CONCLUSION

• The system increasingly fulfills more members’ preferences in a balanced way during meeting scheduling, according to a large-scale study with LLM-simulated members.The framework targets mutually agreeable meeting times while coordinating multiple members with different preferences.Source: §10 · p. 26
• Human participants report robust preference aggregation and reasoning, while the framework is proposed for discussions, brainstorming, and work-plan creation.The paper also presents LLM-simulated participants as a way to conduct otherwise costly and time-consuming large-scale studies.Source: §10 · p. 26

11 ETHICS STATEMENT

• The study underwent formal privacy review, obtained participant consent, anonymized results, and withheld names, email addresses, and individually analyzed data.The reported safeguards cover both organizational review and participant-level data handling.Source: §11 · p. 26
• The researchers omitted sensitive metadata such as seniority and job function and restricted dataset access to the project research team.Participants were also compensated for the user study.Source: §11 · p. 26

Appendix

• The appendix identifies the synthetic-data and prompt materials as part of the paper’s supporting documentation.This passage provides only the appendix section label, not an additional ethical safeguard.Source: Appendix · p. 30

A.1 Member Data

• Synthetic member, meeting, and social-graph materials provide the data inputs for the meeting-scheduling experiments.The appendix labels example synthetic member and meeting tables and an example social graph associated with Figure 7.Source: Table 4 · p. 30, Table 5 · p. 31, §A.3 · p. 31

B.1 Intent Extraction Prompts

• The intent-extraction prompts summarize each member’s dialogue into preferences and record an agreed option in JSON format.The prompts prohibit fabricated preferences and use prior coordination results in later rounds.Source: §B.1 · pp. 32–33

B PROMPT DESIGN FOR GROUP MEETING SCHEDULING

• The appendix specifies JSON-only outputs and instructs the prompts to base responses and reasons only on supplied user data.These constraints appear across intent extraction and coordination prompts.Source: §B.1 · p. 33, §B.2 · p. 35

B.2 Coordination Module Prompt

• The coordination prompt combines preferences, meeting information, relationships, roles, responsibilities, and a candidate option to generate at most K alternatives.It requires the candidate option, favors options satisfying at least as many users, and considers needed attendees, collaborators, teammates, and managers.Source: §B.2 · pp. 33–35

B.3 Evaluation Prompt

• The evaluation prompt assigns each user-option pair a score from 1 to 3 based on how many of the user’s preferences the option satisfies.A score of 1 satisfies few preferences, 2 satisfies at least half but not all, and 3 satisfies all preferences.Source: §B.3 · pp. 35–36

B.4 Synthetic Member Prompts

• Synthetic-member prompts supply names, roles, managers, teammates, collaborators, and schedule preferences while directing members to express preferences and answer clarifying questions.The simulated member’s stated goal is to find a feasible meeting time without mentioning responsibilities in responses.Source: §B.4 · pp. 36–37

C SIMULATION PLOTS

• Detailed simulation plots are presented, with their aggregated results reported in Figures 4 to 6.Source: C SIMULATION PLOTS · p. 37

D.1 Consent and Intake Forms

• The human-participant study evaluates coordination scenarios through intake questions, qualitative feedback, and questions about the need for a coordination agent.Source: C SIMULATION PLOTS · p. 39, §D.1 · p. 39

D.2 Eligibility Criteria

• Participants must be at least 18, based in the United States, experienced with meeting-scheduling tools, and outside designated sensitive groups.Source: §D.2 · p. 40

D.3 Requirements Analysis Form

• The requirements analysis asks about participants’ roles, experience, meeting sizes and frequency, and interest in LLM copilots for collective decision-making scenarios.Source: §D.3 · p. 40

D.4 Synthetic Meeting Scenario

• The scenario involves a meeting organized by John Doe for Norma Fisher, Elizabeth Woods, and Theodore Mcgrath.Source: §D.4 · pp. 40–41
• The paper withholds results for the role question to mitigate privacy risks.Source: §D.4 · p. 40
• Members have distinct scheduling preferences: morning meetings and afternoon deep work, midday meetings, or availability throughout the day based on customers’ needs.Source: §D.4 · p. 41
• The system proposes options with explicit satisfaction labels and reasons tied to members’ reported availability and scheduling preferences.Source: §D.4 · p. 41
• The survey asks participants to judge preference coverage, acceptability of explanations, reflection of group preferences, desired option counts, and confidence in coordination.Source: §D.4 · pp. 41–42

E ADMINISTRATOR INTERVIEW QUESTIONS

• Administrator interviews examine coordination frequency, time demands, participant counts, tools, power dynamics, option generation, unresolved conflicts, and equitable preference consideration.Source: E ADMINISTRATOR INTERVIEW QUESTIONS · p. 42