PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI

José Antonio Siqueira de Cerqueira, Mamia Agbese, Rebekah Rousi, Nannan Xi, Juho Hamari, Pekka Abrahamsson

arXiv:2411.08881v3cs.CYcs.AI

TL;DR

Practical guidance for operationalising AI ethics remains limited, and the study asks whether trustworthiness-enhancing LLM techniques can support ethically aligned AI software. The authors build and evaluate a multi-agent prototype across three real-world incident cases; it generates substantially more documented code and ethics-related artefacts than the baseline trials.

  • Problem

    Practical guidance for developers to operationalise AI ethics is limited, and the study asks whether trustworthiness-enhancing LLM techniques can support ethically aligned AI software.

    Source: §1 · p. 1

  • Method

    An adapted single-cycle Design Science Research approach builds an LLM multi-agent prototype using four trustworthiness techniques and evaluates it on three real-world AI-incident cases.

    Source: §1 · p. 2, §3 · p. 5, §3.2 · p. 5

  • Results

    Around 2,000 lines of code (with proper documentation), versus around 80 lines of output without source code in baseline trials, included ethics-related themes such as bias detection, consent, fairness, and compliance.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    The prototype gives developers tools to consider ethical impacts during AI-system development, within the evaluated cases.

    Source: §5 · p. 11

    Code execution checked whether generated code runs, not its functionality, correctness, or ethical alignment, and the evaluation covered only three case scenarios.

    Source: §6 · p. 12

Abstract

from arXiv · show

AI-based systems, including Large Language Models (LLMs), impact millions by supporting diverse tasks but face issues like misinformation, bias, and misuse. AI ethics is crucial as new technologies and concerns emerge, but objective, practical guidance remains debated. This study explores the extent to which trustworthiness-enhancing techniques in LLMs can support the development of ethically aligned AI software. We adopt a single exploratory cycle of Design Science Research (DSR). First, we identify trustworthiness-enhancing techniques for LLMs: multi-agents, distinct roles, structured communication, and multiple rounds of debate. Second, we design a multi-agent prototype LLM-MAS in which agents address real-world AI ethics issues from the AI Incident Database. Finally, we evaluate the prototype across three case scenarios using thematic analysis, hierarchical clustering, a baseline comparison, and code execution. The system generates approximately 2,000 lines of code per case, compared to only 80 lines in baseline trials. Results reveal terms like bias detection, transparency, accountability, user consent, GDPR compliance, fairness evaluation, and EU AI Act compliance, showing this prototype ability to generate extensive source code and documentation addressing often overlooked AI ethics issues. However, practical challenges in source code integration and dependency management may limit its use by practitioners.

1. Introduction

LLMs’ expanding use in software engineering raises trustworthiness concerns, while practical guidance for operationalising AI ethics remains limited. This study asks whether trustworthiness-enhancing techniques can support ethically aligned software and explores the question with a prototype and three incident-based cases.

1. Introduction

  • LLMs are used across software-engineering tasks, but concerns about trustworthiness include generated code that is syntactically correct yet non-functional.Prior cited work also reported security vulnerabilities in 40% of 1,689 programs produced with GitHub Copilot.Source: §1 · p. 2
  • The study asks whether trustworthiness-enhancing techniques in LLM-based systems can support the development of ethically aligned AI software.It addresses a stated gap in practical AI-ethics operationalisation and in research on LLM use to develop ethical AI systems.Source: §1 · p. 2
  • The authors use an adapted single-cycle Design Science Research approach to build and evaluate a prototype across three real-world AI-incident cases.The cases concern recruitment, deepfake detection, and image classification; the evaluation includes thematic analysis, hierarchical clustering, a ChatGPT GPT-4o baseline, and code execution.Source: §1 · p. 2
  • The prototype combines multi-agent collaboration, specialised roles, structured communication, and multiple debate rounds to address ethical-development tasks.It generated around 2,000 lines of documented code covering issues such as bias, transparency, accountability, consent, GDPR, fairness, and EU AI Act compliance; baseline trials produced around 80 lines without source code.Source: §1 · p. 2

2. Background and Related Work

The background reviews LLMs and trustworthiness, multi-agent techniques in software engineering, AI-ethics principles, and LLM use in qualitative analysis. It positions the study at the intersection of practical AI-ethics operationalisation and LLM-based development.

2.1. Large Language Models

  • LLMs learn language patterns from large text corpora and generate contextually relevant text by predicting the next word or sequence.The paper identifies translation, summarisation, question answering, and creative writing as example tasks.Source: §2.1 · p. 3

2.2. Trustworthiness in Large Language Models

  • Trustworthiness taxonomies use multiple metrics, but terminology, aligned-model frameworks, and assessment guidance remain unsettled.HELM evaluates 30 models across seven metrics; another taxonomy describes eight trustworthiness perspectives, some of which remain difficult to define objectively.Source: §2.2 · p. 3

2.3. Trustworthiness enhancing techniques in LLM4SE

  • LLM4SE frameworks use specialised agents, structured workflows, and communication, while multi-agent debate is reported to improve reasoning and factuality.AutoGen’s ablation study reports that its specialised-role approach outperforms single-agent systems on complex tasks; ChatDev also notes risks from varying outputs and untested code.Source: §2.3 · pp. 3–4

2.4. AI Ethics

  • The study uses 11 AI-ethics principles as its analytical basis, while abstract guidance and its soft-law status complicate operationalisation.The authors position LLM-based multi-agent development from the first stage of development onward as an under-studied approach to operationalising ethical principles.Source: §2.4 · p. 4

2.5. LLMs for Qualitative Analysis

  • Prior work applies LLMs to qualitative coding, while this study automates the whole thematic-analysis process and generates source code for ethical AI development.The background notes that thematic analysis is time-consuming and can involve mistakes and bias; cited studies report that LLMs can identify themes and patterns.Source: §2.5 · pp. 4–5

3. Methodology

The study uses one adapted Design Science Research cycle to explore trustworthiness-enhancing techniques, build an LLM-based multi-agent prototype, and evaluate it in three real-world-incident case scenarios. The methodology combines thematic analysis, hierarchical clustering, a single-agent baseline, and code execution.

3. Methodology

  • The single-cycle DSR study evaluates the prototype in three incident-based scenarios using thematic analysis, hierarchical clustering, comparative baselines, and source-code execution.The study is limited to one exploratory cycle; its four phases are exploration, prototyping, case-study evaluation, and communication of results.Source: §3 · p. 5

3.1. Exploring

  • The exploration phase identifies trustworthiness-enhancing techniques from prior literature to ground the study’s conceptual approach.The techniques are presented in Table 1, whose supplied caption identifies it as a list of adopted techniques.Source: §3.1 · p. 5, Table 1 · p. 5

3.2. Prototype: LLM-MAS Technical Implementation

  • The prototype uses three GPT-4o-mini agents—two senior Python developers and one AI ethicist—with role-specific system prompts and structured communication.Agents have no memory or tool access, and the conversation structure comprises reply, reflection, code, and critique.Source: §3.2 · p. 5
  • Agents work sequentially, each receiving the full conversation history; the researchers selected five rounds per project to support refinement while keeping output manageable.A round is a sequence of conversations among agents 1, 2, and 3, and the number of rounds is modifiable.Source: §3.2 · p. 6

3.3. Evaluation: Case Studies

  • Each of three incident-inspired project descriptions is evaluated through LLM-based thematic analysis, hierarchical clustering, and comparison with direct GPT-4o prompting without multi-agent techniques.The evaluation examines outputs in varied ethical and technical contexts.Source: §3.3 · p. 6
  • The system is run three times per project description to account for output variability, and partial thematic analyses are combined into one analysis per project.Thematic analysis uses GPT-4o; hierarchical clustering uses Descending Hierarchical Classification in iRaMuTeQ, with raw outputs appended as input.Source: §3.3 · pp. 6–7
  • The baseline prompts each project description to regular ChatGPT-4o, while practical evaluation executes the source code generated by the prototype.For the baseline, the single output is also used to create a thematic analysis and hierarchical clustering.Source: §3.3 · p. 7

4. Results

Across three AI-incident scenarios—recruitment, deepfake detection, and image classification—the prototype produced substantial discussion and source code, with thematic and clustering analyses identifying ethical and technical themes. The baseline produced no code in the reported comparisons, while the generated code also presented practical integration challenges.

4. Results

  • The three case studies drew on real AI incidents concerning recruitment, deepfake detection, and image classification.The incidents were selected for their relevance to AI ethics and regulatory compliance.Source: §4 · p. 7

4.1. Project Description 1

  • In Project Description 1, prototype outputs contained 1,712–1,976 lines of discussion plus source code, while the baseline produced 78 lines and no code.The prototype analysis identified ethical, EU Act, and bias-evaluation themes; the baseline yielded three themes and was too small for hierarchical clustering.Source: §4.1 · p. 8

4. Results

  • Generated code was dispersed through conversations and could require manual assembly, dependency installation, and module management, adding practitioner effort.The authors also report deprecated packages and modules that practitioners must locate, copy, test, and manage.Source: §4.1 · p. 8, §4.2 · p. 9

4.2. Project Description 2

  • In Project Description 2, prototype outputs contained 1,414–2,178 lines of text, and the baseline’s 78 lines were described as generic study material.The analysis associated class 1 with Ethical Compliance and AI Responsibility and class 2 with Security and Collaboration; the baseline produced no code.Source: §4.2 · p. 9

4.3. Project Description 3

  • In Project Description 3, outputs contained 1,676–2,122 lines, with themes spanning ethical AI and compliance as well as technical implementation, optimization, and robustness.The clustering included “sklearn,” which the authors had already recognized as deprecated, illustrating a challenge of relying on static LLM knowledge in a changing software landscape.Source: §4.3 · p. 10

5. Discussion

The prototype applies trustworthiness-enhancing techniques to ethical AI development, producing richer ethics-related artefacts than the baseline while exposing practical adoption hurdles.

5. Discussion

  • Its contribution extends LLM-agent work in software engineering to operationalising fairness, accountability, transparency, privacy, and legal alignment in generated outputs.The legal alignment named includes the EU AI Act and GDPR.Source: §5 · p. 11
  • The prototype produced more code and richer ethics-related artefacts than the baseline, including bias-related functions, consent handling, and documentation.The authors suggest multi-agent collaboration, specialised roles, structured communication, and multiple debate rounds can improve generated artefact quality.Source: §5 · p. 11
  • Scattered modules, dependency management, and packaging or integration work remain practical barriers for adoption.The authors identify role specialisation as a next step and call for further testing, comparison, and improvement of the techniques.Source: §5 · p. 11

6. Threats to Validity

The study’s findings are bounded by model reproducibility, fully automated analysis without human validation, limited code testing, and three case scenarios.

6. Threats to Validity

  • Reliance on a proprietary OpenAI LLM creates reproducibility challenges because future versions may behave differently under identical conditions.The authors note that the model may evolve in undocumented ways.Source: §6 · p. 11
  • LLM-performed thematic analysis and automated clustering lacked human validation, limiting triangulation and potentially missing subtle, context-sensitive insights.The study did not conduct a parallel human-led thematic analysis.Source: §6 · p. 11
  • Execution tests checked whether generated code runs, not its functionality, correctness, or ethical alignment.The practical reliability and appropriateness of the generated software therefore remain open questions.Source: §6 · pp. 11–12
  • Evaluation covered only three case scenarios, and the case-study design aimed for in-depth contextual insight rather than generalisable results.The authors call for broader case coverage and more rigorous assessment of ethical dimensions.Source: §6 · p. 12

7. Conclusion

Using Design Science Research, the authors developed and evaluated an LLM-based multi-agent system on three AI-incident-derived cases. It generated substantially more output than the baseline, including ethics-related code and documentation, while practical integration and ethical-alignment assessment remain concerns.

7. Conclusion

  • The Design Science Research prototype used multi-agent collaboration, specialised roles, structured communication, and multiple debate rounds across three AI-incident-derived case studies.The cases addressed recruitment, deepfake detection, and image classification.Source: §7 · p. 12
  • About 2,000 lines of text, including source code and documentation, were produced per case, versus around 80 lines of baseline output without source code.The cases concerned recruitment, deepfake detection, and image classification.Source: §7 · p. 12
  • Generated ethics-related outputs included bias detection, transparency and accountability, user consent, GDPR compliance, fairness evaluation, and EU AI Act compliance.The authors also report ethics-related terms and functions in documentation and source code.Source: §7 · p. 12
  • Source-code extraction, module handling, package installation, and deprecated dependencies hinder practitioner use; further ethical-alignment analysis is needed.The authors propose supplying legislation or ethics guidelines as future work.Source: §7 · p. 12

Declaration on Generative AI

The authors used ChatGPT for grammar and spelling checks and to improve clarity, then reviewed and edited the content.

Declaration on Generative AI

  • The authors used ChatGPT for grammar and spelling checks and clarity improvements, then reviewed and edited the text.They state that they take full responsibility for the publication’s content.Source: Declaration on Generative AI · p. 13

Related papers

Loading 2411.08881v3…