Source-linked AI summary
SETU: An Agentic Ecosystem for Multilingual, Persona-Aware Communication Coaching
Jonnalagadda Maruthi Tejas, Uponika Barman Roy, Tilottama Goswami, Samir Goswami, Mousita Dhar
TL;DR
Multilingual workplace communication coaching often lacks integrated, auditable analysis across modalities and audience context. SETU addresses this gap with specialized agents, trust-aware orchestration and human oversight for formative sales-pitch coaching. Across 18 sales sessions, it achieved coach agreement ρ = 0.78, explainability 4.3/5 and language macro-F1 0.86, while reducing coach review time from 22 to 13 minutes per session.
Problem
Existing coaching systems often score modalities in isolation and rarely combine multimodal, persona-aware reasoning for multilingual and code-mixed sales speech.
Method
SETU integrates specialized multimodal agents, buyer-persona context, trust-aware routing and aggregation, and human oversight into a structured sales-coaching pipeline.
Results
Across all 18 sales sessions, SETU obtained coach agreement ρ = 0.78, explainability 4.3/5 and language macro-F1 0.86.
Takeaways & Limitations
Case-study evaluation indicates improved coach agreement and reduced pitch-coaching effort compared with text-only and non-agentic baselines.
Takeaways & Limitations
The current pilot is small and internal, with 18 sales videos; larger expert-labeled benchmarks are needed before deployment at scale.
Abstract
from arXiv · showhide
Corporate training teams need scalable and explainable tools to improve workforce communication in multilingual settings. Existing systems often score text, audio, or video in isolation, or produce black-box outputs that are difficult to audit for coaching use. This paper presents SETU, an agentic ecosystem for corporate communication coaching aimed at recruiters, frontline sales professionals and training units who prepare for audience specific conversations. SETU is designed for two scoped scenarios: (i) recruiter-candidate eligibility-and-interest calls with persona context and (ii) sales pitches with target-audience adaptation; owing to limited evaluation resources, this paper reports results on scenario (ii) only. The ecosystem decomposes analysis into specialized video, audio-speech, text-relevance, scoring, notification and reporting agents coordinated through trust-aware orchestration. It generates modality-attributed coaching reports for formative training, with human reviewers retaining final judgment. The name SETU (bridge in several Indic languages) reflects the goal of bridging communication gaps across regional languages and audience expectations.
I. INTRODUCTION
SETU addresses multilingual, multimodal workplace communication coaching through specialized, explainable agents that incorporate audience persona and task context. Its design emphasizes trust-aware aggregation and human oversight, while evaluation in this study is limited to sales-pitch coaching.
- Workplace communication requires evaluating content, delivery clarity and adaptation to the listener’s role, intent and context.
- Existing tools often provide generic textual feedback or single-modality scores rather than combining visual, acoustic, linguistic and persona-aware reasoning.
- The study evaluates the sales-pitch scenario; recruiter–candidate interactions are supported by design but not evaluated because of limited resources.
- Trust-weighted orchestration can down-weight unreliable signals, while agent boundaries support score attribution, replacement and debugging.
- SETU decomposes coaching into specialized video, audio–speech, text–relevance, scoring, notification and reporting agents.
- The ecosystem adapts feedback to performer role, target audience, task prompt and multilingual language conditions, with human reviewers retaining final judgment.
III. METHODOLOGY
The methodology structures spoken product-pitch practice around a buyer persona, objective, time limit and multilingual recording. A running payroll-SaaS scenario illustrates the pipeline from pitch design through analysis.
- The running case uses a first-meeting pitch for a cloud HR and payroll SaaS platform aimed at an SME CEO.
- The pitch prompt requires addressing the SME pain point, product value, ROI impact and a clear next step in 90 seconds.
- The illustrative input is a 78-second webcam recording of English–Bengali code-mixed speech.
- The weak case opens with company background and product modules, whereas the strong case leads with compliance pain, gives an ROI example and closes with a demo call-to-action.
B. Proposed High-Level Agentic Framework
SETU organizes sales-pitch coaching as a routed multimodal pipeline that aligns perceptual and contextual evidence, progressively enriches state, and delivers human-reviewed reports. Trust-aware orchestration assigns subtasks to specialized agents and downweights unreliable modality signals during aggregation.
- Pipeline architecture: SETU routes synchronized pitch recordings through speech, video, audio, and relevance analysis before semantic fusion, scoring, alerts, and human-reviewed scorecards.The dual-stream design separates perceptual analysis from buyer-persona and pitch-prompt constraint mapping.
- System formulation: The ecosystem represents coaching with agents, tasks, state, environment, orchestration, communication, and memory in a unified tuple.These elements define the system-level formulation of SETU.
- Agent decomposition: Specialized agents cover synchronization, visual cues, speech and acoustics, text quality, persona-aware sales relevance, and coaching-report generation.The full task set assigns t1 through t6 to these functions.
- Orchestration: Each subtask is mapped to the most capable available agent rather than sending all inputs to one monolithic model.This routing assignment is the orchestration core.
- State enrichment: Context is progressively enriched from recorded posture and vocal-delivery features to transcript text, pitch relevance, and the final coaching report.The sequence produces s2, s3, and s4 as increasingly informed states.
- Trust-aware aggregation: Agent trust combines modality accuracy, score consistency, and runtime reliability, allowing poor-SNR audio evidence to be downweighted during aggregation.The trust formulation applies to agent outputs rather than the sales trainee.
D. Sales Pitch Mapping Design and Data Acquisition
SETU frames spoken product-pitch practice as a structured matrix linking products, buyer personas, pitch stages, objectives, and timed prompts. Data collection uses controlled webcam video, office-noise audio, short clips, and coach-rated pitch rubrics.
- Training matrix: The sales pitch mapping design specifies the product, buyer persona, and pitch objective for each timed practice entry.It is intended for structured spoken product-pitch training.
- Matrix layers: The design layers performer role, target buyer persona, pitch stage, and timed pitch prompt.Examples include an SME CEO target and a 90-second payroll SaaS pitch requiring ROI and a next-step close.
- Data acquisition: Pitch clips use 720p single-speaker webcam capture, typical office-noise audio, and durations of 45–120 seconds.These conditions define the pitch data-collection setting.
- Human evaluation: Sales coaches label pitches Good, Fair, or Poor using rubrics for opening hook, buyer relevance, evidence or ROI, delivery quality, and closing call-to-action.The labels are based on pitch-specific coaching criteria.
E. Video Agent
The video agent extracts nonverbal delivery cues from webcam recordings and combines smoothed behavioral subscores into a visual performance score. In Case A, visual weaknesses during pricing and ROI discussion corresponded to a low score and a confidence-related coaching observation.
- Cue extraction: The video agent measures posture stability, eye-contact approximation, facial expressiveness, and head movement using MediaPipe Holistic landmarks.The resulting subscores are temporally smoothed before combination.
- Case A result: 3.8/10 visual score in Case A followed camera aversion during pricing and lost upright posture during the ROI segment.The report linked weak nonverbal delivery to reduced perceived pitch confidence, independent of language choice.
F. Speech and Audio Agents
SETU’s speech and audio agents transcribe multilingual, code-mixed pitches, detect language through multiple signals, and compute acoustic delivery features. Case A combined English–Bengali speech, frequent fillers, moderate SNR, and a 4.1/10 audio score with targeted pacing guidance.
- Speech and acoustic analysis: The audio/speech agent transcribes multilingual and code-mixed speech and computes RMS energy, pitch variation, silence ratio, SNR, WPM, and filler density.Transcription uses the Sarvam saaras:v3 codemix model, while acoustic features use Librosa.
- Language detection: Language detection combines API confidence, script-based fallback, and parallel language probes for code-mixed Indian sales speech.These signals address cases where script and audio cues disagree.
- Case A result: 4.1/10 audio score in Case A followed high filler density, moderate SNR, and Bengali code-switching during the ROI explanation.The coaching alert advised slowing down and emphasizing benefit statements clearly.
- Text and relevance analysis: The text agent evaluates grammar, fluency, professionalism, semantic completeness, and alignment with the sales-task prompt and buyer persona.It scores both delivery quality and whether the correct sales communication objective is addressed.
- Language policy: Code-switching is not penalized by default, while switching among more than three languages in one session is flagged as excessive.The coaching target is English plus up to two regional languages.
- Case A text result: 4.6/10 text score in Case A reflected fragmented pitch flow and missing value-proposition language when features were not converted into buyer outcomes.The example pitch listed attendance, payroll, and leave modules without articulating buyer outcomes.
H. Relevance Agent
The relevance agent evaluates whether a pitch fits the persona-task prompt and produces traceable, modality-linked coaching guidance. In the case study, it distinguished a partial CEO-targeted pitch from one containing pain, ROI and a clear next step.
- Relevance scoring enriches the pitch prompt with performer and buyer personas before applying an LLM rubric scored from 0–50 and normalized to 0–10.The agent evaluates fulfillment of the persona-task prompt.
- Case A scored 5.4/10 because it omitted payroll-compliance pain, ROI evidence and a next-step ask.The deductions reflected missing pain point, missing ROI evidence and missing close.
- Case B addressed the CEO prompt with payroll-error pain, up to 30% admin-time reduction and a 15-minute demo request.
- The report links visual, audio, text and relevance findings to recommendations and deduction traces for coach inspection.
- Case A’s rewrite recommendations covered SME pain, buyer-relevant benefits, ROI, eye contact, a demo call-to-action and selective English–Bengali switching.
- Human sales coaches validate evidence, adjust rubrics and approve feedback, keeping SETU a formative coaching assistant rather than an autonomous decision engine.The validation step supports auditability and fairness governance.
IV. EXPERIMENTS AND PERFORMANCE ANALYSIS
The experiments evaluate SETU through a pilot sales-pitch case study focused on buyer-persona fit and persuasive delivery. The protocol covers timed pitches, varied products and personas, multilingual recordings, and coach rubric labels.
- SETU’s evaluation is a pilot study with a corporate sales training partner, restricted to sales pitch mapping.Each pitch presents a solution to a defined buyer persona within 45–120 seconds.
- The evaluation asks whether each spoken pitch fits the buyer persona and achieves the prompt objective, rather than treating recordings as free-form conversations.
- The study measures delivery-scoring validity, multilingual handling, buyer-persona relevance, feedback explainability and coaching-time reduction.
- The pilot dataset contains 18 recorded sales pitch videos, with products spanning HR–payroll SaaS, EdTech learning platforms and insurance cross-sell plans.
- Target personas include CEOs of SMEs, school teachers, mid-level managers, DINKs in IT firms and retired government officers.
- Pitch stages cover Lead Generation, Connecting, Profiling, Needs Gathering and Product Mapping.
- Recordings range from 45–120 seconds, use 720p webcam and mono office audio, and include English, Hindi, Telugu and English–Hindi or English–Telugu code-mixing.Coaches label pitches Good, Fair or Poor using hook, relevance, evidence, delivery and close rubrics.
C. Evaluation Protocol
The evaluation protocol processes each pitch through multimodal and language-aware analysis, compares SETU with text-only and non-agentic baselines, and reports agreement, explainability, latency and language performance. Results also quantify coaching-time and practice-cycle changes.
- Evaluation protocol: Each video passes through audio extraction, speech-to-text, language profiling, video and acoustic analytics, LLM rubric scoring and weighted aggregation.Livemode tests on six sessions measure alert latency.
- Baselines: The baselines are transcript-only LLM feedback and a single-pass non-agentic multimodal summary without agent-level traces.
- Metrics: The evaluation reports modality sub-scores, overall score, coach agreement, explainability, language macro-F1, end-to-end latency and coaching-time reduction.
- Representative results: SETU’s representative overall scores were 4.6 for Case A, 7.7 for Case B and 7.4 for Case C.Scores used batch weights of 0.20 visual, 0.25 audio, 0.20 text and 0.35 relevance.
- Aggregate results: Across 18 sessions, SETU achieved ρ = 0.78 coach agreement, 4.3/5 explainability and 0.86 language macro-F1, with 38.6-second batch latency and 2–4-second live refresh.
- Comparative analysis: SETU exceeded both baselines in agreement and explainability, while buyer-persona conditioning improved agreement by 0.11 in ρ over non-persona prompting.Coach review time fell from 22 to 13 minutes per session, a 40.9% reduction.
G. Case Study Insights
SETU’s sales-pitch case study indicates that multimodal, persona-aware decomposition can produce explainable coaching that coaches can act on immediately. The pilot also identifies deployment boundaries and priorities for strengthening the system at scale.
- Case-study findings: Code-mixed pitch delivery is feasible to coach when ASR, script cues and confidence filtering are combined, with human review retained.Human coaches validate pitch judgments and help avoid over-penalizing natural language switching.
- Boundaries: The current pilot is constrained by video/audio quality, ASR accuracy, prompt design, rubric fairness and a small internal sample of 18 sales videos.Visual cues may vary with culture, disability, camera angle and lighting.
- Future directions: Future work will expand buyer-persona and task coverage, add expert-labeled benchmarks, improve low-resource language support, and conduct longitudinal and fairness analyses.Planned audits cover gender, accent, skin tone, disability and regional language groups.
- Case-study findings: SETU combines multimodal agent decomposition and buyer-persona context to produce specific, explainable coaching for multilingual sales pitches.The case study frames persona context as important for judging whether product language fits the target buyer.