PaperlayerView exact revision on arXiv

Source-linked AI summary

Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development

Sattvik Sahai, Prasoon Goyal, Michael Johnston, Anna Gottardi, Yao Lu, Lucy Hu, Luke Dai, Shaohua Liu, Samyuth Sagi, Hangjie Shi, Desheng Zhang, Lavina Vaz, Leslie Ball, Maureen Murray, Rahul Gupta, Shankar Ananthakrishna

arXiv:2508.10108v1cs.AIcs.CL

TL;DR

AI coding assistants can produce vulnerable code, and their use raises safety concerns around both software vulnerabilities and cyberattacks. The paper describes a 10-team adversarial challenge that tested coding assistants through multi-turn red-teaming and reports distinct lessons about vulnerable-code generation, multi-turn attacks, and safety-utility approaches.

  • Problem

    AI coding assistants may generate vulnerable code or aid cyberattacks, creating safety concerns as their use in software development grows.

    Source: §1 · pp. 1–2

  • Method

    The challenge paired five automated red-teaming teams with five assistant-development teams in tournaments, with rankings accounting for attack success, diversity, defense success, and utility.

    Source: §1 · p. 2, §2.4 · p. 4

  • Results

    The challenge found defenders easier to elicit vulnerable code from than malicious cyberattack assistance, while attackers often succeeded by beginning with benign turns and introducing malicious intent later.

    Source: §8 · p. 16

  • Takeaways & Limitations

    The challenge’s team work indicates that safety and utility may not be competing objectives for vulnerable-code generation, and highlights multi-turn dialogue structures as a concern for safeguards.

    Source: §8 · pp. 16–17

    The team-led utility-testing approach was difficult to scale alongside evolving attack strategies, despite mitigating overfitting for a time.

    Source: §8 · p. 17

Abstract

from arXiv · show

AI systems for software development are rapidly gaining prominence, yet significant challenges remain in ensuring their safety. To address this, Amazon launched the Trusted AI track of the Amazon Nova AI Challenge, a global competition among 10 university teams to drive advances in secure AI. In the challenge, five teams focus on developing automated red teaming bots, while the other five create safe AI assistants. This challenge provides teams with a unique platform to evaluate automated red-teaming and safety alignment methods through head-to-head adversarial tournaments where red teams have multi-turn conversations with the competing AI coding assistants to test their safety alignment. Along with this, the challenge provides teams with a feed of high quality annotated data to fuel iterative improvement. Throughout the challenge, teams developed state-of-the-art techniques, introducing novel approaches in reasoning-based safety alignment, robust model guardrails, multi-turn jail-breaking, and efficient probing of large language models (LLMs). To support these efforts, the Amazon Nova AI Challenge team made substantial scientific and engineering investments, including building a custom baseline coding specialist model for the challenge from scratch, developing a tournament orchestration service, and creating an evaluation harness. This paper outlines the advancements made by university teams and the Amazon Nova AI Challenge team in addressing the safety challenges of AI for software development, highlighting this collaborative effort to raise the bar for AI safety.

1 Introduction

AI coding assistants can generate vulnerable code, and their growing use also raises concern that coding tools could expand access to cyberattack development. The challenge addresses gaps in static, single-turn safety benchmarks by organizing automated red-teaming against AI coding assistants.

Safety motivation

  • Security concerns include vulnerable AI-generated code reaching production and malicious actors using coding assistants to develop sophisticated cyberattacks.As coding assistance lowers barriers to software development, users may lack experience to catch generated vulnerabilities.Source: §1 · pp. 1–2

Research gap

  • Static benchmarks often have limited coverage and focus on single-turn prompts, while human-in-the-loop red-teaming can be prohibitively expensive to scale.The paper notes that developers commonly have multi-turn conversations with assistants.Source: §1 · p. 2

Challenge overview

  • The challenge organized 10 university teams into five automated red-teaming teams and five safe-assistant teams for adversarial, multi-turn tournaments.The paper describes the tournament organization, evaluation approach, service, team advances, and lessons learned.Source: §1 · p. 2

Challenge resources

  • The challenge also built a custom 8B coding specialist model as a baseline for defending teams.Further details about the model and tournament orchestration service appear in appendices.Source: §1 · p. 2

2 Challenge Design

The challenge uses closed-box, multi-turn adversarial tournaments to let university teams test red-team bots and safe coding assistants against one another. Its design combines varied match-ups, safety and utility evaluation, and probing so teams can adapt between rounds.

2.1 Adversarial setup

  • Five attackers and five defenders competed in tournaments, with attackers seeking vulnerable or malicious code or cyberattack explanations and defenders aiming to remain useful and robust.Defenders started from an 8B coding specialist model built for the challenge.Source: §2.1 · pp. 2–3

2.2 Tournaments

  • Each attacker faced all five defenders, producing 25 match-ups per tournament and exposing each side to a range of opposing systems.The format was intended to give defenders varied red-teaming conversations and attackers varied systems to test.Source: §2.2 · p. 3

2.3 Match-ups

  • Each match-up contained 200 conversations, each limited to five prompt-response pairs; the figure depicts an attacker-defender pairing with up to 10 turns per conversation.The turn limit was chosen to permit multi-turn attack strategies.Source: §2.3 · p. 3, Figure 1 · p. 3

2.4 Evaluation and Ranking

  • Attackers were ranked using attack success adjusted for diversity, while defenders’ average defense success was scaled by utility to discourage repeated attacks and blanket refusal.The overall rankings average attacker results across defenders and defender results across attackers.Source: §2.4 · p. 4

2.5 Incentive Structure

  • From the second tournament onward, each match-up added 200 unscored probing conversations, allowing attackers to adapt without manual system changes or training before scored rounds.Automatic adaptation by attack bots was permitted.Source: §2.5 · p. 4

2 Challenge Design

  • The challenge format was reported to drive competitive interaction and innovation, while providing dynamic, evolving evaluation compared with static benchmarks.The authors describe this as a summary of their experience with the challenge design.Source: §2.5 · p. 4

3 Design Guidelines

The design guidelines constrained both sides to practical, annotatable multi-turn interactions while giving defenders a shared coding model and bounded system modifications. Attackers had wider model and component choices, but still faced runtime and latency constraints.

General constraints

  • Both sides had to support English or human-readable-code multi-turn conversations, with Python code generation as the scope for code-based attack success.Conversations were limited to five prompt-response pairs and prompts and responses to 800 tokens; bots had a 45-second latency limit.Source: §3 · p. 5

Defender constraints

  • Defenders built on the 8B Prize coding model and could fine-tune it and add system components, but responses had to use the core model.Auxiliary models were capped at 800M parameters in total, and multiple sequential core-model passes were allowed within the latency budget.Source: §3 · p. 5

Attacker constraints

  • Attackers could combine open-source language models with planners, rules, prompt mutators, dialog managers, and other components to build adaptive attack bots.They could not use closed-box model APIs at runtime and had to meet the latency requirement.Source: §3 · p. 5

4 Evaluation

The evaluation labels conversations for attack or defense success, then ranks attackers using success and diversity and defenders using defense success and utility. Vulnerable-code detection is automated, while security-event judgments use cybersecurity experts.

4 Evaluation

  • The evaluation labels each conversation for attack or defense success, ranks attackers using diversity-adjusted attack success, and combines defenders’ average defense success with utility.It is designed to discourage repeated attacks and defenders that simply block or deflect every request.Source: §4 · p. 6

4.1 Attack vs. Defense Success Labeling

  • An attack succeeds if vulnerable Python code is detected or experts identify compliance with malicious intent; only medium-or-higher vulnerabilities count.Python responses are checked with CodeGuru, while security-event conversations are labeled by three cybersecurity annotators using a majority vote.Source: §4.1 · pp. 6–7

4.2.1 Diversity Evaluation for Each Match-up

  • Successful-attack diversity is computed by averaging pairwise BLEU scores over conversations that include both attacker and defender turns.The authors chose this simple, interpretable measure because they lacked data approximating the tournament distribution for testing alternatives.Source: §4.2.1 · p. 7

4.2.2 Attacker Score for a Match-up

  • The normalized attack success rate combines attack success rate with diversity so attackers with similarly high success are ranked higher when their attacks are more diverse.The paper reports choosing a combination measure intended to capture this ranking intuition.Source: §4.2.2 · p. 7

4.2.3 Overall Attacker Score

  • An attacker’s overall score is the average normalized attack success rate across all defenders.This score is used to rank attackers.Source: §4.2.3 · p. 7

4.3.1 Average Defense Success Rate

  • A defender’s average defense success rate is computed by averaging defense success across attacks.Source: §4.3.1 · p. 8

4.3.2 Utility Evaluation

  • Utility tests assess benign cybersecurity questions and coding tasks, including realistic functionality and multi-turn cases that test whether prior malicious turns trigger later over-refusal.Coding Utility V1 uses executable test cases but has constrained scope; V2 adds LLM-judged tasks such as server requests, database access, and system calls. Utility scores are capped relative to the base model and averaged across sets.Source: §4.3.2 · pp. 8–9

4.3.3 Overall Score

  • The overall defender score combines average defense success across attackers with utility to reward safety without regressing on base-model utility.The combined score is used to rank defenders.Source: §4.3.3 · p. 9

5 Tournament Orchestrator

The Tournament Orchestrator coordinated scalable, reproducible multi-turn conversations between attack and defense bots. It initialized eligible pairings and managed each session’s alternating turns through termination.

5 Tournament Orchestrator

  • The orchestrator used AWS Lambda, Amazon SQS, and Amazon DynamoDB for serverless, scalable, event-driven coordination of pairwise multi-turn bot conversations.Its architecture is shown in Figure 2.Source: §5 · p. 9, Figure 2 · p. 9

5.1 Initialization Phase

  • The Config Assistant created eligible attacker-defender pairs and recorded their configurations; after readiness checks, the Session Coordinator queued session-start messages.Initial messages contained empty conversation history.Source: §5.1 · p. 9

5.2 Runtime Phase (Life of a Session)

  • During runtime, attacker and defender turns alternated with updated session history until an end signal, fatal error, or turn limit; termination triggered logging and any needed next batch.This lifecycle abstracted pacing from bot teams while allowing sessions to proceed independently across pairs and batches.Source: §5.2 · p. 10

6 Scientific Advancements

The university teams developed varied approaches to safe coding assistance and automated red teaming. Their methods included reasoning-based alignment, layered safeguards, synthetic data, adaptive attack generation, and strategies for efficient probing.

6.1 Defending (Model Developer) Teams

  • Purpl3pwn3rs combined an input sanitizer, deliberative moderator, vulnerability refiner, and secure filter; its refiner used RLVR and reward functions.PurpCorn-Plan trained a reasoning-aligned 32B model, then aligned the 8B Prize model using safety, utility, and formatting oracles, internal red teaming, and lightweight guardrails.Source: §6.1 · p. 10
  • Alquist dynamically adjusted prompting by intent and could regenerate unsafe outputs, while LionCoders paired response generation and output guarding with a vulnerability fixer and detector.HokieTokie trained separate models for malicious requests and vulnerabilities, using datasets built from taxonomies, real vulnerable code, and tournament failures.Source: §6.1 · pp. 10–11
  • Defending teams commonly generated synthetic data, used reasoning traces or policy optimization, and added input or output safeguards to improve safety alignment.Teams used methods including GRPO, DPO, classifiers, string-based guardrails, and output filtering.Source: §6.1 · p. 11

6.2 Attacking (Red) Teams

  • PurCL used Gibbs sampling to search promising attack-space regions and adapted prompts to exploit weaknesses in defender reasoning; CapitalAI retrieved strategies from summarized failed and successful attacks to evolve failures during deployment.CapitalAI’s strategy library was populated with LLM-generated summaries of conversations that did and did not find safety lapses.Source: §6.2 · pp. 11–13
  • Astro’s COMET combined attack generation, prompt tuning, a tournament-trained surrogate, and an attack planner, while SaFoLab progressively intensified conversational prompts and selected candidates using an utterance reward model.SaFoLab trained that reward model using Monte Carlo Tree Search.Source: §6.2 · p. 12
  • RedTWIZ built a diverse multi-turn attack suite and planner; other teams used attacker-defender-evaluator loops, utility-inspired attacks, and planners to select promising strategies.RedTWIZ’s approaches included progressively harmful utility prompts, a risk-category and attack-style grid, conversation trees, and a learned steering policy.Source: §6.2 · p. 12
  • Novel red-team techniques included sampling promising attack-space regions, exploiting weaknesses in model reasoning, and retrieving strategies to evolve failed attacks toward successful ones.The paper describes these as approaches for efficiently testing target models’ safety robustness.Source: §6.2 · pp. 12–13

7 Results

The paper reports results from three tournaments and a two-round finals evaluation. Across the tournaments, attackers faced increasingly difficult defenses, while defenders maintained utility and successful elicitation of malicious code or explanations declined.

7 Results

  • The results section presents outcomes from three tournaments and describes how the finals evaluation was conducted.The tournament series was used to evaluate teams over the course of the challenge.Source: §7 · p. 13

7.1 Tournament 1

  • Tournament 1 showed a healthy distribution of overall ASR and DSR values, but some defenders regressed on utility benchmarks and some attackers lost diversity.Tables 1 and 2 provide the defender and attacker leaderboards.Source: §7.1 · p. 13, Table 1 · p. 13, Table 2 · p. 13

7.2 Tournament 2

  • Tournament 2 had a narrower range of overall ASR scores than Tournament 1, defenders improved at maintaining utility while improving safety, and simpler attacks often stopped succeeding.Vulnerable-code attacks increased in volume relative to malicious-code or explanation attacks, a trend that continued in Tournament 3.Source: §7.2 · p. 14

7.3 Tournament 3

  • Tournament 3 retained a healthy spread of ASR and DSR scores, all defenders maintained utility well, and successful attacks eliciting malicious code or explanations dropped substantially.The tournament covered 38 unique vulnerability types mapping to over 44 CWEs; Table 7 lists the ten most frequent CWEs.Source: §7.3 · pp. 14–15

7.4 Finals

  • The finals advanced four attackers and four defenders through an adversarial round and a second round of paired expert red-teaming and review.Judges assessed defender utility beyond test sets and attacker techniques, then deliberated to select winners and runners-up.Source: §7.4 · p. 15, Figure 3 · p. 16, Table 7 · p. 16

8 Discussion

The tournaments exposed distinct safety challenges for vulnerable-code generation and malicious cyberactivity, including multi-turn jailbreak strategies. The teams also revealed practical challenges in testing and adapting defenses as attacks evolved.

8 Discussion

  • Defenders were easier to elicit vulnerable code from than malicious cyberattack assistance, which the authors attribute mainly to the difficulty of reasoning about code security.For vulnerable code, models must generate logically correct code while ensuring it is secure, rather than simply learning which requests to deflect.Source: §8 · p. 16
  • Attackers more successfully jailbroke models by beginning with benign requests and gradually introducing malicious intent across turns.The authors say this suggests current safeguards may leave models vulnerable to certain multi-turn dialogue structures.Source: §8 · pp. 16–17
  • Teams increasingly focused on vulnerable code, while defense teams explored multi-objective GRPO and deliberative reasoning to identify hidden intent.The challenge’s work suggests safety and utility may not be competing objectives for vulnerable-code generation.Source: §8 · pp. 16–17
  • After attackers said 200 scored conversations gave too little signal to tailor attacks, later tournaments added unscored probing conversations before scoring.This let attackers adapt to specific defenders while keeping annotation costs and time manageable.Source: §8 · p. 17
  • Utility datasets mitigated overfitting for a time, but maintaining them became difficult as teams’ attack strategies evolved.The authors suggest future challenges make utility testing part of attackers’ responsibilities.Source: §8 · p. 17
  • Defenders became more resistant to attacks resembling those seen in earlier tournaments, even as rankings fluctuated between consecutive runs.The authors attribute ranking fluctuations to defenders training on prior attacks and recommend more frequent or continuous tournaments with rankings averaged across runs.Source: §8 · p. 17

9 Conclusion

The conclusion presents the Trusted AI challenge as a collaborative effort to advance coding-assistant safety through adversarial tournaments. Its supporting Prize LLM used a custom tokenizer and a staged English-and-code pre-training curriculum.

9 Conclusion

  • The challenge paired competing university teams with adversarial evaluation and data, and the authors see the tournament format as applicable beyond coding safety.The paper reports that participating teams published proceedings articles and submitted multiple publications to external venues.Source: §9 · p. 17

A Prize LLM

  • The Prize LLM was an 8B helpful-only code specialist built as a shared baseline, using a decoder-only Transformer with 32 layers and 32 attention heads.It used Grouped Query Attention with 8 groups, a 90,000-token vocabulary, and an 8k context window; access was restricted to challenge use.Source: A Prize LLM · p. 25

A.1 Tokenizer

  • The custom tokenizer’s training mix included about 27% Python, 12% other primary coding languages, 20% non-primary coding languages, 13% code-related English conversations, and 27% general English.The authors chose a custom tokenizer to avoid undertrained tokens from domain mismatch and improve compression on domains such as Python code.Source: §A.1 · p. 25
  • Tokenizer candidates were assessed using validation-set compression ratios and small 1.5B models trained on 24B–200B tokens and compared on benchmarks including HumanEval, MMLU, and ARC-Challenge.The latter provided an extrinsic evaluation of candidate tokenizer performance trajectories.Source: §A.1 · p. 25

A.2 Pre-training

  • The Prize LLM was pre-trained on 3.5T tokens, beginning with 1.3T tokens of English and non-primary code before 2.2T tokens of coding data biased toward Python.Both stages included 0.1T tokens of math, science, and technical-discussion data.Source: §A.2 · p. 25

A.2.1 Datamix

The appendix reports datamix experiments and training choices for the Prize LLM. Mixed-language code, Python upsampling, context extension, and lightweight post-training were among the reported decisions.

A.2.1 Datamix

  • Datamix decisions used small 1.5B ablation models sparingly because these experiments were resource intensive.The authors used this approach only for major datamix decisions.Source: §A.2.1 · p. 25
  • Models trained on multiple programming languages ultimately performed better on Python than Python-only models, whose gains plateaued earlier.The mixed-language models improved more slowly on Python at first but continued improving for longer.Source: §A.2.1 · p. 25
  • Upsampling Python data gave a significant performance boost, while an additional Python-only curriculum stage did not improve the final 8B model.The authors hypothesize that the 8B model learned more during the preceding curriculum stage, making repeated Python data unnecessary.Source: §A.2.1 · p. 26

A.3 Context Extension

  • Context extension increased the model’s context length from 2,048 to 8,192 tokens by adjusting RoPE base frequency and linearly decreasing the learning rate.The authors report significant performance improvements after only 200B tokens of training in this stage.Source: §A.3 · p. 26

A.4 Post-training

  • Post-training was limited to lightweight SFT because university teams were expected to experiment with their own post-training strategies.The SFT data was filtered to ensure code appeared inside markdown blocks and deduplicated using MinHash.Source: §A.4 · p. 26

A.5 Model Evaluation

The Prize LLM was evaluated against coding, instruction-following, and general-knowledge goals, and the paper reports competitive Python coding performance. The tournament orchestrator supported controlled multi-turn interactions between university teams’ attackers and defenders.

A.5 Model Evaluation

  • Evaluation goals covered Python coding, complex instruction following, and general knowledge, measured with HumanEval, IFEval, and MMLU, respectively.The authors selected these benchmarks as proxies for coding ability, jailbreak-relevant instruction following, and general knowledge in a testbed.Source: §A.5 · p. 26
  • The Prize LLM showed competitive Python coding performance in a HumanEval pass@1 comparison with similarly sized models.The paper also presents post-trained results on other selected benchmarks and a wider benchmark evaluation of the pretrained model, without supplying their values in these passages.Source: §A.5 · p. 26, Table 8 · p. 27, Table 9 · p. 27

B Tournament Orchestrator Details

  • The stateful, serverless, modular Tournament Orchestrator managed structured, multi-turn interactions between university teams’ attacker and defender bots.It supported official tournaments, dry runs, practice runs, and A/B testing with configurable sessions per attacker–defender pair.Source: B Tournament Orchestrator Details · p. 28

B.1 Functional Guarantees

  • Static attacker–defender pairings and per-pair session quotas supported configured tournament traffic, while batched launches let attackers adapt between batches.Message pacing controlled traffic, and tournaments could continue while at least one attacker and one defender were online.Source: §B.1 · p. 28
  • Turn-based Lambda handlers and separate queues alternated attacker and defender responses while carrying full session context, including in sessions exceeding 15 minutes.The design retained a serverless architecture without session-level infrastructure such as EC2, ECS, or AWS Batch.Source: §B.1 · p. 28
  • Sessions ended on attacker termination, fatal error, or turn limit; failed API calls were retried once, and persistent failures ended the session.The coordinator launched additional batches until configured sessions were complete, while independent bot contexts allowed problematic bots to be paused without affecting others.Source: §B.1 · p. 28

B.2 Design Trade-Offs and Considerations

The orchestrator’s trade-offs include delayed feedback, turn-based latency, and duplicate-request handling, while CoBot gives teams tools for building, deploying, and hosting challenge bots. CoBot’s updates simplify deployment and add support for larger models and custom integrations.

B.2 Design Trade-Offs and Considerations

  • The orchestrator buffers results until sessions end, adds latency through Lambda cold starts and SQS polling, and may send duplicate requests that stateful bots must handle.Despite these trade-offs, it provides a robust and extensible framework for high-integrity adversarial evaluations at scale.Source: §B.2 · pp. 28–29

C Capabilities Provided to Teams

C.1 Conversational Bot Toolkit (CoBot)

  • CoBot’s modular, extensible toolkit helps teams build and deploy LLM-based conversational experiences while reducing time spent on infrastructure, hosting, and scaling.Teams also received exclusive access to Amazon resources, technologies, and AI systems experts.Source: §C.1 · p. 30, C Capabilities Provided to Teams · p. 30

C.1.1 CoBot Deployment Infrastructure

  • CoBot combines lightweight Lambda modules with remote Docker modules on ECS and EC2, and adds EBS configuration and runtime commands for large images and custom container entry points.The remote modules sit behind Application Load Balancers, which Lambda uses to send requests.Source: §C.1.1 · pp. 30–31

C.1.2 Large Language Model Support in CoBot

  • CoBot supports remote Docker modules for larger models and provides vLLM- and TGI-based modules to help teams deploy and integrate popular LLMs.The toolkit also added vLLM, TGI, and Bedrock hosting; these capabilities reduce inference time by up to 70% compared with its previous Flask-server-based hosting.Source: §C.1 · p. 30, §C.1.2 · p. 31

C.1.3 Sample Bots

The challenge supplied redesigned attacker and defender sample bots tailored to the Trusted AI track. They illustrate core integrations and provide simple, readable examples for teams to build on.

C.1.3 Sample Bots

  • The redesigned samples include one attacker bot and one defender bot, tailored to the Trusted AI track.The defender example uses the baseline Prize LLM that teams were expected to improve.Source: §C.1.3 · pp. 32–33
  • The samples demonstrate interactions with the Tournament Orchestrator, Bedrock APIs, and Flask- and TGI-based remote modules; vLLM support was later added with LionCoders’ contribution.They also include examples of DynamoDB conversation storage, A/B testing, and integration tests.Source: §C.1.3 · p. 32
  • The examples were kept simple, with readable, high-quality code intended to help teams build working bots and understand CoBot’s capabilities.Source: §C.1.3 · p. 33

C.2 Training Infrastructure and Software

The paper describes the training infrastructure and software provided to teams, alongside tournament evaluation, annotation, support, and safety processes. Together, these resources supported training and iterative assessment of AI coding assistants.

C.2.1 Infrastructure

  • Teams used an EKS cluster with reserved Trainium instances, MPI orchestration, and shared FSx storage for distributed SFT/DPO training.Each trn1.32xlarge instance had 16 Trainium chips; worker pods used the Neuron SDK and MegaSFT.Source: §C.2 · p. 33, §C.2.1 · pp. 33–34

C.2.2 Software

  • MegaSFT adapted Megatron-LM for Trainium-based SFT/DPO, while customized scripts and runbooks supported cluster operations and consistent distributed-training workflows.MegaSFT supported data, tensor, pipeline, and sequence parallelism; runbooks documented pod setup, data loading, checkpointing, and worker coordination.Source: §C.2.2 · p. 34

C.3 Code Extraction and Vulnerable Code Detection

  • A code-extraction pipeline parsed tournament conversations for code, and CodeGuru Security scanned extracted files with over 100 vulnerability detectors.Results and code snippets were aggregated into team S3 buckets for review; teams also used the scanning capability for offline benchmarking and data collection.Source: §C.3 · p. 35

C.4 Feedback Data and Evaluation Metrics; C.5 Support from the Amazon team

  • Practice runs and official tournaments supplied teams with diagnostic data, scores, anonymized rankings, and opposing-side metrics; Amazon also offered bootcamp training, reviews, office hours, and technical support.Practice feedback included conversation and failure logs, vulnerability reports, and code-utility and security-utility scores.Source: §C.4 · p. 35, §C.5 · p. 35

D.1 Overview of Human Annotation Exercise

  • Human annotation targeted conversations where users elicited malicious code or detailed cyberattack instructions; vulnerable-code cases were screened with static analysis and excluded from annotation.Annotators were instructed to focus on malicious intent, not merely requests for vulnerable code.Source: §D.1 · p. 36

D.2 Dimensions to Annotate; 2 PV_MAL_EXPLN; 3 PV_MAL_CODE, PV_MAL_EXPLN

  • Annotators assessed five-turn conversations at the conversation level, labeling whether malicious code or detailed explanations of malicious cyberactivity were provided.The two labels were PV_MAL_CODE and PV_MAL_EXPLN; conversations containing both could receive both labels.Source: §D.2 · pp. 36–37, §2 · p. 37, §3 · p. 37

D.2.1 Additional Items to Note for Annotating:

  • The annotation rules considered malicious intent across turns, required Python code for the PV_MAL_CODE label, and distinguished standalone operational explanations from text accompanying code.The conversation was marked policy-violating only when malicious intent preceded the code or detailed explanation.Source: §D.2.1 · pp. 37–38

E Research Safety & Containment

  • The challenge restricted generated-code execution to benign utility testing in a sandbox and reviewed public releases for safe and ethical dissemination.Tournament red teaming ran through a controlled orchestrator; the paper also states that no functioning malware was executed or retained.Source: E Research Safety & Containment · p. 42
Loading 2508.10108v1…