This workshop, first presented at Black Hat USA 2024, empowers security professionals to systematically red team AI systems. It goes beyond traditional security failures by incorporating novel adversarial machine learning and Responsible AI (RAI) failures, enabling a holistic approach to identifying potential issues before an AI system is deployed.

The curriculum, adapted from our internal upskilling programs designed for other red teams at Microsoft, is tailored to meet the demands of safeguarding AI technologies. Participants will gain practical insights into AI architecture, functionality, and real-world applications, with a focus on popular systems like GPT, Stable Diffusion, and LLaMA.

The workshop is designed to be hands-on, providing an environment where participants can actively engage with custom-built AI applications. Practical modules challenge participants to red team these applications, while subsequent sessions offer proven strategies for fortifying them. Attendees will explore various prompting strategies for LLMs, including zero-shot, few-shot, and advanced techniques such as Retrieval-Augmented Generation.

In this training, participants will learn to red team AI systems using innovative methodologies that address emerging risks, following practices pioneered by the Microsoft AI Red Team (AIRT). The course covers AI-specific vulnerabilities, Responsible AI (RAI) vulnerabilities, and effective mitigations. Participants will engage in a comprehensive learning experience through a blend of lectures, hands-on playground sessions, and coding labs.

By the end of the course, attendees will be equipped to probe any machine learning system for vulnerabilities, including prompt injection attacks, using both manual and automated methods. The coding portion will introduce participants to PyRIT, AIRT’s open-source toolkit for red teaming. Additionally, since the course is hosted by Microsoft, attendees will have the unique opportunity to examine real vulnerabilities discovered in Microsoft products and services, illustrating the “break-fix” cycle that the AI Red Team employs.

This course is led by experts from Microsoft’s AI Red Team, the first team to integrate RAI red teaming alongside traditional security red teaming. Over the past year, this team has assessed every high-risk AI system at Microsoft, including foundation models and Copilots.

Attendees will need to bring a computer with internet access and accounts and API keys for 

2 Days

AI Red Teaming in Practice

Application Security

Hacking

Process

Security Tooling

The course blends secure coding, API security, secure design concepts, threat modeling, and a tiny bit of incident response to provide a comprehensive foundation for building and maintaining secure software systems. Key tools include VS Code, GitHub, OWASP DevSlop Pixi, Semgrep Community, and 42Crunch IDE plugins.

Day 1: Secure Coding & Design Foundations

: Secure Coding Fundamentals with “Bad, Better, Best”

: Secure Design Concepts – Interactive Ideation

Day 2: Hands-On Secure API Design & Hardening

: Advanced API Threats & OWASP API Top 10 (6–10)

Internet access for GitHub, Semgrep Community, 42Crunch plugin setup

Beginner to Intermediate. Basic coding experience is expected. Some familiarity with web technologies, APIs, and development tools (e.g., Git, VS Code) is helpful but not mandatory.

Q: Do I need to be a security expert to take this class?


A: No. This course is designed for developers and engineers with some experience in software development, but not necessarily in security.

Q: What languages and frameworks will be used?


A: Primarily JavaScript, Python, and REST APIs (openAPI/swagger), though principles are transferable across languages.

Q: Will we write code or is this lecture-only?


A: We will review code and discuss code constantly. We will fix API vulnerabilities together on day two. We will use the 42Crunch free plugin with VS Code to analyze the security of an API. You can fix the issues with me, or just watch. If you are not comfortable reading or writing code, you can just follow along. Many students choose to follow along, and that is perfectly ok.

Q: Will I get a copy of the course materials?


A: Yes. You will receive the code, cheat sheets, and slide decks during the class.

Ability to read and write basic JavaScript or another programming language

Working knowledge of the system development lifecycle (SDLC)

Laptop with the IDE VS Code (preferred) or Eclipse installed before the class

Optional but helpful: basic understanding of HTTP, APIs, and common web vulnerabilities

Secure Coding & API Hardening: Hands-On Secure Design, Development, and Threat Modelling

Security

We levelled up the threat modeling war game. Engaged in CTF-style challenges, your team will battle for control over an offshore wind turbine park.

The level of this training is Beginner/Intermediate. Participants who are new to threat modeling are advised to follow our self-paced Threat Modeling Introduction training (which is about 2 hours and is included in this training).

As highly skilled professionals with years of experience under our belts, we're intimately familiar with the gap between academic knowledge of threat modeling and real-world practice. To minimize that gap, we have developed practical use cases, based on real-world projects. Each use case includes a description of the environment, together with questions and templates to build a threat model.

Students will be challenged in groups of 3 to 4 people to perform the different stages of threat modeling:

• Diagram techniques applied on a travel booking service
• Threat model a cloud-based update service for an IoT kiosk
• Create an attack tree against a nuclear research facility
• Create a SOC Risk Based Alerting system with MITRE ATT&CK
• Mitigate threats in a payment service build with microservices and S3 buckets
• Apply data protection by design and default on a loyalty app
• Apply the OWASP Threat Modeling Playbook on agile development
• Threat modeling the CI/CD pipeline
• Battle for control over "Zwarte Wind", an offshore wind turbine park

After each hands-on exercise, the results are discussed, and students receive a documented solution. All participants get our Threat Modeling Playbook to improve you threat modeling practice, and a one-year access to our online threat modeling learning platform.

As part of this training, you will be asked to create and submit your own threat model, on which you will get feedback. One month after the training we organize an individual coaching session with all the participants.


Course Syllabus/Outline

Agile Whiteboard Hacking - aka Hands-on Threat Modeling - Course Outline

Threat modeling introduction
• Threat modeling in a secure development lifecycle
• What is threat modeling?
• Why perform threat modeling?
• Threat modeling stages
• Different threat modeling methodologies
• Document a threat model
Diagrams – what are you building?
• Understanding context
• Doomsday scenarios
• Data flow diagrams
• Trust boundaries
• Sequence and state diagrams
• Advanced diagrams
• Hands-on: Diagram techniques applied on a travel booking service
Identifying threats – what can go wrong?
• STRIDE introduction
• STRIDE threats
• Hands-on: Threat model a cloud-based update service for an IoT kiosk
• Attack trees
• Hands-on: Create an attack tree against a nuclear research facility
• Attack libraries
• MITRE ATT&CK
• Hands-on: Create a SOC Risk Based Alerting system with MITRE ATT&CK

Addressing each threat
• How to address threats
• Mitigation patterns
• Value of standard mitigations
• Setting priorities through risk calculation
• Risk management
• Threat agents
• The mitigation process
• Hands-on: Mitigate threats in a payment service build with microservices and S3 buckets
Threat modeling and compliance
• How to marry threat modeling with compliance
• GDPR and Privacy by design
• Privacy threats
• LINDUNN and Mitigating privacy threats
• Threat modeling medical devices
• Threat modeling Industrial Control Systems (IEC 62443)
• Threat Assessment and Remediation Analysis for automotive (TARA, SAE 21434)
• Mapping threat modeling on compliance frameworks
• Hands-on: Apply data protection by design and default on a loyalty app
Advanced threat modeling
• Typical steps and variations
• Validation threat models
• Effective threat model workshops
• Communicating threat models
• Agile and DevOps threat modeling
• Improving your practice with the Threat Modeling Playbook
• Scaling up threat modeling
• Hands-on: Apply the OWASP Threat Modeling Playbook on agile development
• Hands-on: Threat modeling the CI/CD pipeline
Threat modeling resources
• Open-Source tools
• Commercial tools
• General tools
• Threat modeling tools compared
Examination
• Hands-on examination
• Grading and certification
Battle for control over "Zwarte Wind", an offshore wind turbine park
Red team versus Blue team battle for control over an offshore wind turbine park

Review session (online coaching session after 1 month)
• Hand-in of your own threat model
• Individual feedback on your threat model
• Review session


Agile Whiteboard Hacking – aka Hands-on Threat Modeling

AI/ML

Architecture

Cloud Security

Privacy

SDLC

Topics include AI threat modeling, adversarial defense, supply chain integrity, red teaming, and continuous monitoring. Participants will leave equipped with actionable practices to embed security across the AI development lifecycle—whether leveraging LLMs for coding or deploying models into production.

Day 1: Using AI Securely in Software Development

AI-specific vulnerabilities and threat surfaces
How attackers abuse AI in software systems


 Build a mental model for AI-related risks in secure dev work.

Risks in AI-generated code (vulnerabilities, licensing, dependencies)
Secure prompt engineering and safety guardrails


 Use tools like Copilot/ChatGPT securely and responsibly in pipelines.

3. Secure AI Development Lifecycle (SDLC + MLOps)

Security in data ingestion, modeling, deployment
DevSecOps/MLOps integration with security gates


 Embed controls across AI system development workflows.

Data poisoning, model theft, insecure inference
How AI threat modeling differs from traditional apps


 Apply threat modeling to proactively identify AI-specific risks.

Prompt injection, data leakage, model abuse
Case studies and tested mitigations


 Secure LLM features and avoid common developer pitfalls.

Vetting third-party models, datasets, and libraries
Trust and integrity in data lineage and model provenance


Evasion, poisoning, model inversion, gradient masking
Defensive training and detection techniques


 Recognize and respond to adversarial threats effectively.

AI-specific pentesting strategies
Tools, frameworks, and Red vs. Blue exercises


 Operationalize adversarial testing to improve resilience.

Secure versioning, rollback strategies
Emergency response to discovered model vulnerabilities


 Create patchable, resilient AI deployment pipelines.

Using explainability tools to spot anomalies
Balancing transparency with exploitability


 Detect bias, malicious behavior, and model theft vectors.

Role-based access to inference endpoints, model APIs
Credential hygiene and zero-trust enforcement


 Harden model endpoints and AI-connected services.

Monitoring for concept/data drift and security regression
Triggering retraining or re-validation workflows


 Maintain model integrity and relevance over time.

Recap of core practices and threats
Toolkits, references, and next-step strategies


 Leave with a clear roadmap for securing AI in real-world environments.


Attendees will gain both strategic insight and tactical techniques for using AI securely in software development, and for defending AI systems themselves against evolving threats. The workshop blends secure dev, MLOps, and adversarial resilience—equipping developers and security engineers with skills for modern AI-integrated applications.

NDC { Manchester }

Workshops

09:00 - 17:00 (UTC±00)

Building and Deploying Secure AI: Practical Strategies for Security Developers

Jim Manico

AI Red Teaming in Practice

Gary Lopez

Secure Coding & API Hardening: Hands-On Secure Design, Development, and Threat Modelling

Tanya Janca

Agile Whiteboard Hacking – aka Hands-on Threat Modeling

Georges Bolssens