PaperlayerView exact revision on arXiv

Source-linked AI summary

The Ethics of Artificial Intelligence in Military Operations

Nicolas Drapier, Florian Mauberger, Aladine Chetouani, Aurelien Chateigner

arXiv:2609.26507v1cs.CYcs.AIcs.HC

TL;DR

Opaque military AI and gaps in existing governance raise the question of how responsible human judgment can be preserved. The paper proposes procedural oversight and technical safeguards, then examines them through counterfactual case analyses. The cases indicate that documented harms involve deficiencies in multiple governance dimensions and that no single safeguard is sufficient in isolation.

  • Problem

    The paper asks whether responsible human judgment can be preserved when military decisions are delegated to opaque systems, against a background of gaps in existing governance coverage.

    Source: §1 · p. 2, §2.5 · p. 6

  • Method

    The paper proposes procedural governance combining named accountability structures, traceable records, enforceable human controls, a NATO evaluation standard, and tiered deployment profiles.

    Source: §3.1 · p. 8, §3.3 · p. 9, §3.6 · p. 19

  • Results

    Every case involving documented harm shows deficiencies in at least two governance dimensions, and no single safeguard would have sufficed in isolation.

    Source: §3.7.6 · p. 22

  • Takeaways & Limitations

    The paper concludes that ethical constraints should be encoded in institutions and procedures when they cannot be reliably encoded in a loss function.

    Source: §4 · p. 23

    Military AI model exchange is constrained by national doctrine, which embeds nation-specific tactical assumptions, priorities, and decision heuristics.

    Source: §3.3 · p. 10

Abstract

from arXiv · show

Deep learning systems now mediate military decisions to use force, yet their internal logic resists inspection, their evaluation practices are gameable, and their deployment fractures accountability across dispersed stakeholders. The ethical challenge posed by these systems is fundamentally epistemic: not just whether autonomous weapons should be permitted to kill, but whether the conditions for responsible human judgment can survive when critical functions are delegated to opaque algorithms. We show that this epistemic condition produces a concrete accountability gap: responsibility diffuses across designers, operators, and policymakers while International Humanitarian Law presupposes capacities for judgment that current AI systems lack. To address this gap, we propose a governance framework that proceduralizes ethical constraints through named accountability roles, adversarial auditing with undisclosed benchmarks, tiered deployment thresholds, and a proposed NATO evaluation standard. Counterfactual analysis of eight documented cases (1988-2025) shows that each governance mechanism addresses a documented class of failure, but no single safeguard suffices in isolation: effective governance of military AI requires not only technical constraints but the institutional infrastructure to keep human judgment meaningful.

1 Introduction

The paper reframes military AI’s ethical challenge as preserving the epistemic conditions for responsible human judgment, not only deciding whether autonomous systems may select and engage targets. It diagnoses how opacity compounds accountability and IHL problems, then proposes a procedural governance framework evaluated against operational cases.

1 Introduction

  • The central ethical question is whether responsible human judgment can survive when critical military functions are delegated to systems whose internal logic resists inspection.The paper argues that this epistemic challenge is more fundamental than the binary debate over whether autonomous systems should be permitted to select and engage targets.Source: §1 · p. 2
  • Deep learning already supports targeting, surveillance, logistics, and command decisions at speeds and scales beyond human cognitive capacity.The paper calls this integration of autonomous inference into the chain of command “Algorithmic Warfare.”Source: §1 · p. 2
  • Opacity compounds across the chain of command, distorting operator trust, fragmenting accountability, and straining IHL principles that presuppose human judgment.Responsibility is displaced and redistributed rather than eliminated when decisions are delegated to algorithms.Source: §1 · p. 2
  • The paper diagnoses three compounding epistemic failures and proposes procedural governance evaluated through counterfactual analysis of eight operational cases from 1988–2025.The framework includes named accountability roles, adversarial auditing with undisclosed benchmarks, a proposed NATO evaluation standard, tiered deployment thresholds, and interface design principles.Source: §1 · p. 2

2 Background

The background traces how opacity, adversarial fragility, institutional incentives, and fragmented responsibility undermine judgment and accountability. It also argues that existing governance instruments leave important military AI applications and model behavior insufficiently covered.

2.1 Opacity, Explainability, and Adversarial Fragility

  • Post hoc explanations can satisfy the need for justification without revealing a model’s actual reasoning, while adversarial fragility can induce competence-boundary failures that operators cannot detect.The paper argues that partial transparency may suppress skepticism more than acknowledged opacity, and that opacity and adversarial fragility are usually studied separately despite their interaction in military decisions.Source: §2.1 · p. 3

2.2 Trust, Institutional Incentives, and Systemic Vulnerabilities

  • Categorical outputs and known evaluation criteria can foster overconfidence, while benchmark gaming, weight poisoning, and contracting arrangements separate technical knowledge from deployment authority.The paper notes that poisoned models can retain benchmark performance while behaving anomalously in high-stakes scenarios, and that Project Maven exposed a gap between state ownership and access to code or training data.Source: §2.2 · pp. 3–4

2.3 The Accountability Gap

  • Responsibility is fragmented across designers, data curators, operators, and policymakers, while effective human control requires both authority to intervene and interpretive access to system outputs.When operators align with AI outputs without challenge, their role can become performative rather than an exercise of critical judgment.Source: §2.3 · pp. 4–5

2.4 IHL Under Strain

  • IHL principles such as distinction and proportionality require contextual judgment, and lawful deployment is framed through foresight, administration, and traceability.The cited analysis also identifies restraint toward surrendering, evacuated, or otherwise hors de combat persons as a setting current systems handle poorly.Source: §2.4 · p. 5

2.5 Existing Governance Instruments and Their Limits

  • Existing rules have uneven reach: the EU AI Act excludes purely military systems, while DoDD 3000.09 covers weapon systems and other military regimes offer advice or principles rather than requirements.The binding military rule does not cover logistics, medical triage, intelligence analysis, or cyber operations, even when their outputs can lead to lethal consequences.Source: §2.5 · p. 6
  • Governing a platform rather than its learned model can leave the model’s decision unconstrained when the same model is moved to another application.The paper argues that a model can behave differently as inputs drift while the approved platform baseline remains unchanged; configuration checks may pass without testing what they were meant to establish about model behavior.Source: §2.5 · pp. 6–7

3 Design Principles for Accountable Military AI

The paper argues that ethical constraints cannot yet be reliably encoded in learning systems across operational contexts. It therefore proposes procedural governance, complemented by limited algorithmic safeguards, as the practical design approach.

3 Design Principles for Accountable Military AI

  • The paper says current methods cannot reliably encode ethics in learning systems robustly across diverse contexts, adversarial manipulation, and distribution shifts.It argues that formal constraints can satisfy the letter of a rule while violating its spirit or fail outside their training distribution.Source: §3 · p. 7
  • Rather than treating morality as an optimizable loss function, the paper proposes encoding ethical constraints in institutions, certification regimes, standard operating procedures, and technical safeguards.This proceduralization is intended to address the gap between convergence on high-level ethical principles and divergence in implementation.Source: §3 · p. 7
  • The proposed pragmatic approach combines limited algorithmic constraints with external procedural controls, including calibrated uncertainty, forced abstention, audits, authorization gates, and traceability.The paper presents this as compatible with a future hybrid approach as alignment research matures.Source: §3 · p. 7
  • The framework develops accountability structures, adversarial evaluation, a NATO standardization proposal, interface principles, tiered deployment thresholds, and case-based testing.The paper describes six components, including testing the framework against eight operational cases.Source: §3 · p. 8

3.1 Actionable Accountability

The section proposes making distributed responsibility actionable through defined human roles, traceable records, and enforceable controls. These measures are intended to clarify obligations and reduce conditions for scapegoating without eliminating the many-hands problem.

3.1 Actionable Accountability

  • The framework aims to assign a human authorizer to each high-stakes decision, a human owner to each deployed system, and evidence sufficient to reconstruct and attribute critical failures.It proposes three complementary requirements to make dispersed responsibility actionable.Source: §3.1 · p. 8
  • Each operational AI capability should have separate System Owner, Operational Authorizer, Operational User, and Independent Evaluator roles, with responsibilities written into doctrine.The roles cover lifecycle risk management, mission-level activation, real-time judgment and anomaly reporting, and certification and reassessment.Source: §3.1 · p. 8
  • Tamper-evident logs should record model and data versions, sensor provenance and inputs, uncertainty estimates, operator actions, and timing for post hoc review.The design goal is a factual record sufficient for legal scrutiny.Source: §3.1 · p. 8
  • For potentially harmful operations, systems should enforce stage-gated approvals, abort capability, and fallback modes that operators can exercise under time pressure.The paper argues that substantive human control must be implemented as an operable control surface.Source: §3.1 · p. 8
  • Explicit, auditable, and enforceable responsibilities are intended to reduce scapegoating, though they do not eliminate the many-hands problem.The proposal reframes distributed decision-making as distributed obligation.Source: §3.1 · p. 8

3.2 Adversarial Evaluation

Known benchmarks can reward benchmark-specific optimization rather than operational competence. The section proposes adversarial auditing, robustness evaluation, and negative testing to make military AI evaluation better reflect deployment conditions.

3.2 Adversarial Evaluation

  • Known evaluation criteria can drive developers to optimize for those measures at the expense of broader competence.The section frames benchmark disclosure as a military concern because controlled demonstrations may not predict operational performance.Source: §3.2 · pp. 8–9
  • 11-14% accuracy drops across classifiers followed replication of ImageNet test-set creation, illustrating that fixed-benchmark performance may not predict generalization.The passage reports no change in the underlying data distribution.Source: §3.2 · p. 9
  • Undisclosed, regularly rotated benchmarks can reduce optimization for known test content while keeping evaluation procedures transparent and standardized.The existence of some evaluation sets should itself be classified.Source: §3.2 · p. 9
  • Evaluation should test adversarial robustness because patches and targeted perturbations can cause high-confidence misclassification, while common corruptions degrade classifiers despite high clean-data accuracy [9].Military evaluation should reflect contested and unpredictable operating conditions.Source: §3.2 · p. 9
  • Negative testing asks whether systems reject inputs they should reject, including civilian objects resembling targets, atypical military targets, surrender, medical evacuation, and distribution-boundary cases.The section treats this as a direct operationalization of distinction.Source: §3.2 · p. 9

3.3 Toward a NATO Evaluation Standard

NATO has standards for sensor data and responsible-AI principles, but lacks a framework for certifying the algorithms that use the data. The section proposes a standardized evaluation protocol while identifying data-standardization and model-sharing constraints.

3.3 Toward a NATO Evaluation Standard

  • Sensor-layer standards are mature, while algorithmic and autonomy layers remain under development and study-stage STANAGs have not been ratified [49].NATO’s AI Strategy commits to governability, traceability, and reliability but does not specify certification mechanisms.Source: §3.3 · p. 9, Table 1 · p. 10
  • Exchanging trained models can expose national doctrine encoded in their weights to adversarial reverse-engineering, making unrestricted allied model exchange impractical for the foreseeable future.STANAG 5669 targets exchange of trained neural networks independent of training-time software and hardware.Source: §3.3 · p. 10
  • The authors propose a NATO STANAG evaluation protocol with state-facing rules for benchmark secrecy, methodology, deployment prerequisites, and independent auditing.The proposed protocol has a second part intended for developers.Source: §3.3 · pp. 10–11
  • Publicize metric categories such as robustness, calibration, and latency, while classifying specific implementations, thresholds, and weights to balance preparation with benchmark gaming risk.The authors say this middle path channels optimization toward relevant properties but does not eliminate Goodhart risk.Source: §3.3 · p. 11
  • A workable data-audit compromise pairs statistical dataset summaries with periodic sample-level inspections under confidentiality agreements, since summaries alone may miss some biases.The standard should also mandate canonical annotation formats and cross-cutting provenance, uncertainty, context, and versioning provisions.Source: §3.3 · p. 11

3.4 The Assistant Principle: Preserving Human Judgment

The operator interface should make system capabilities, limitations, and confidence perceptible so that human understanding tracks the system’s state. The proposed hybrid trajectory pairs procedural safeguards with technical guardrails and expands autonomy only as operational reliability is demonstrated.

3.4 The Assistant Principle: Preserving Human Judgment

  • Interfaces should make capabilities, limitations, and confidence immediately visible so operators’ mental models track the system’s actual state.Uncertainty should be perceptually salient to help recalibrate trust.Source: §3.4 · pp. 11, 13
  • Where operational constraints permit, transparent models are preferred for high-stakes decisions; post hoc explanations remain audit artifacts, not proof of faithful reasoning or decision legitimacy.The authors identify generalized additive models and decision trees as transparent model classes and cite saliency maps and SHAP values as diagnostic signals.Source: §3.4 · p. 13
  • The hybrid trajectory combines calibrated uncertainty and abstention, hard interlocks outside the authorized envelope, and progressive autonomy bounded by those interlocks.Below a task-specific confidence threshold, the system defers to the operator.Source: §3.4 · p. 13
  • Autonomy expands only after traceability records demonstrate sustained reliability under operational conditions, with procedural safeguards defining permissible boundaries at each stage.The authors describe this as a temporal process rather than a fixed architecture, motivating tiered deployment.Source: §3.4 · p. 13

3.5 Tiered Deployment Thresholds

The framework assigns minimum safeguards by operational risk and reversibility, classifying systems across five dimensions rather than by application domain alone. It also treats some configurations as impermissible and requires reassessment when capabilities or context change materially.

3.5 Tiered Deployment Thresholds

  • Tiering uses lethality, reversibility, autonomy, speed, and scope to classify operational consequences; this differs from application-domain classification.The table describes profiles across five measurable dimensions, with escalation on any single dimension potentially triggering reclassification.Source: §3.5 · pp. 13–14, Table 3 · p. 14
  • Lower tiers require standard engineering assurance and domain validation, while intermediate tiers call for independent testing, command-level authorization, visible confidence estimates, and guaranteed manual fallback.Intermediate tiers cover reconnaissance and command support, where errors may affect situational awareness and become irreversible once acted upon.Source: §3.5 · p. 15
  • Tier 5 autonomous cyber operations require national-authority approval, code-enforced scope limits, and continuous monitoring with automatic suspension triggers.The passage cites machine-speed effects, jurisdictional reach, and cascading consequences as reasons for these safeguards.Source: §3.5 · p. 15
  • Tier 6 lethal autonomous weapons marks a threshold for political and legal deliberation, not authorization to deploy under sufficiently robust safeguards.The passage cites the ICRC position on human control and prohibition of systems incapable of IHL compliance.Source: §3.5 · p. 15
  • Procedural compliance is insufficient for systems unable to reliably distinguish combatants from civilians or intended for settings where that distinction is structurally unachievable.The authors state such systems should not be deployed regardless of procedural safeguards.Source: §3.5 · p. 15
  • A system’s tier can shift as functions or contexts change, so material changes require periodic reassessment and mandatory re-certification.The passage gives a reconnaissance system gaining target-nomination features as an example of risk migration.Source: §3.5 · p. 15

3.6 From Dimension Profiles to Tiers: A Criticality Score

Section 3.6 replaces a weighted sum of five dimension ranks with a multiplicative criticality score converted to tiers, because weighted sums cannot reproduce Table 3’s tier structure.

3.6 From Dimension Profiles to Tiers: A Criticality Score

  • With three levels across five dimensions, 243 profiles are possible, but Table 3 names only six, leaving a rule needed for the other 237.The profiles use ranks 0, 1, and 2 on each dimension.Source: §3.6 · pp. 15–16
  • Any weighted sum puts the Tier 3 profile halfway between Tiers 1 and 6, so it cannot reproduce Table 3’s Tier 3 assignment.The figure visualizes this midpoint mismatch; the proof also establishes that no weights make all six tiers equally spaced.Source: Figure 1 · p. 16, §3.6 · p. 16
  • The score multiplies factors for veto opportunity, action gravity, and reversibility; logarithmic tier units make the factors’ contributions additive.Moving up one tier represents the same multiplicative increase in expected damage.Source: §3.6 · p. 17
  • The veto-window factor depends on autonomy and effect speed, while gravity and reversibility capture the action’s severity and how long its harm lasts.Advisory systems leave the decision to a human; autonomous systems acting in milliseconds leave no practical veto window.Source: §3.6 · p. 17
  • Each tier step has a constant damage multiplier, and the six reference profiles score 0.00, 0.65, 2.32, 2.68, 3.99, and 5.34 for Tiers 1–6.Raising any single dimension never lowers the score, so escalation can maintain or raise the tier.Source: §3.6 · pp. 17–18
  • The Table 4 values were fitted to no data; six expert-reviewable constraints set them, and panels must decide whether to accept the implied weighting of autonomy against scope.The framework excludes profiles combining Direct lethality with Full reversibility; the closest admissible profile sits one-tenth of a tier from a boundary.Source: §3.6 · pp. 18–19

3.7 Operational Precedents: Counterfactual Analysis

Across eight counterfactual cases, every case involving documented harm exhibited deficiencies in at least two governance dimensions, and no single safeguard sufficed. Each mechanism addresses a documented failure class, but the analysis cannot establish that safeguards would have prevented harm.

3.7 Operational Precedents: Counterfactual Analysis

  • The framework was assessed counterfactually against eight cases spanning Tiers 3–6 and 1988–2025, but the analysis cannot establish that safeguards would have prevented harm.Lavender and Gospel rely primarily on investigative journalism and have not been subject to official inquiry.Source: §3.7 · p. 19

3.7.1 Tiered deployment thresholds.

  • Maven’s shift from Tier 3 ISR analysis to Tier 4 targeting recommendations lacked public evidence of formal reassessment; Kargu-2 lacked Tier 6 governance, while Iron Dome operated within constrained parameters.The tiering framework requires risk-based classification and re-certification when capabilities migrate across tiers.Source: §3.7.1 · pp. 19–20

3.7.2 Actionable accountability.

  • Maven exposed an accountability gap: contractor-held knowledge and state-held authority were not bridged by named roles, while engineers lacked formal influence over deployment.The framework separates responsibilities among a System Owner, Operational Authorizer, and Independent Evaluator.Source: §3.7.2 · pp. 20–21

3.7.3 Traceability by default.

  • Stuxnet, Kargu-2, and Lavender lacked traceability needed to reconstruct or challenge decisions, making post hoc accountability procedurally impossible.Tamper-evident logs with sensor provenance would have made Stuxnet’s scope creep detectable.Source: §3.7.3 · p. 21

3.7.4 Adversarial evaluation.

  • Patriot’s friendly-fire incidents and the Vincennes tragedy illustrate confusion failures that negative testing and adversarial evaluation would require testing before deployment.Proposed tests include ambiguous IFF environments, overlapping military and civilian signatures, and high-stress time-critical conditions.Source: §3.7.4 · p. 21

3.7.5 The assistant principle.

  • Lavender’s reported 10% error rate corresponded to approximately 3,700 misidentifications among 37,000 flagged individuals despite roughly twenty-second human reviews.Gospel’s target volume could overwhelm human review capacity, while Aegis displayed raw data without highlighting an anomaly.Source: §3.7.5 · pp. 21–22

3.7.6 Technical enforceability.

  • Every case involving documented harm exhibited deficiencies in at least two governance dimensions, and no single safeguard would have sufficed in isolation.The mechanisms address documented failure classes, but prospective certification is needed to test predictive power.Source: Table 6 · p. 22, §3.7.6 · pp. 22–23
  • Stuxnet spread to approximately 115,000 systems across multiple countries without hard scope limits; Iron Dome’s engagement criteria bounded autonomous operation.The framework proposes code-level target lists, network boundaries, and time windows.Source: §3.7.6 · p. 22

4 Conclusion

The conclusion argues that opaque, gameable systems and dispersed knowledge can erode the conditions for meaningful human judgment. It proposes procedural governance while acknowledging limits in scope and the need for further institutional and empirical testing.

4 Conclusion

  • Opaque reasoning, gameable evaluation, and dispersed knowledge can erode the conditions under which human judgment remains meaningful.An operator alone does not ensure oversight; the operator needs time to intervene and access to interpret when intervention is warranted.Source: §4 · p. 23
  • The framework shifts ethical constraints into institutions through named roles, evaluation regimes, deployment thresholds, and interfaces that keep uncertainty visible.It focuses on documented process and standards rather than resolving who bears moral responsibility for an algorithmic output.Source: §4 · p. 23
  • The tiered classification is schematic, and the framework addresses state military organizations within treaty regimes rather than non-state actors.Operationalizing the framework would require negotiation among states with divergent legal traditions and procurement cultures.Source: §4 · p. 23
  • Expert consultation could test the framework against institutional practice, while forced abstention and salient uncertainty still require empirical study under time pressure.The paper identifies these interface principles’ effect on decision quality as a behavioral question the framework currently answers only by assertion.Source: §4 · p. 23
  • Assessment of whether a particular system in a particular context meets International Humanitarian Law demands remains a human judgment requiring adequate knowledge and real authority.Source: §4 · p. 24

Declarations

The declarations report no specific funding, disclose one author’s defense-technology employer, and state that no datasets were generated or analyzed.

Funding

Competing interests

  • Authors A, B, and D are employed by SAS Impact, which uses machine learning systems for defense applications; Author C declares no competing interests.Source: Competing interests · p. 24

Data availability

  • No datasets were generated or analyzed; the cases are documented in publicly available cited sources.Source: Data availability · p. 24
Loading 2609.26507v1…