LLM and Agent Security: Red Teaming and Evaluation

Part of the Applied AI Security and Assurance curriculum, authored by Cameron Hopkin, CISSP, CEH, CHFI. Published as a reference, not an enrollable course.

Description

The flagship course, and the one that defines the curriculum. It treats large language model and agent systems as targets and teaches how to attack, defend, evaluate, and govern them at a research level. The full syllabus is published below.

Outcomes

• Execute direct and indirect prompt-injection and jailbreak attacks against LLM and agent systems.
• Design containment and least-privilege controls for autonomous tools.
• Build evaluation harnesses that function as security controls.
• Run a structured red-team engagement and report defensible findings.

Modules

1. The LLM and agent threat surface.
2. Prompt injection, direct and indirect, and data exfiltration through agents.
3. Jailbreaks and guardrail evasion.
4. Tool and agent containment and least privilege for autonomous systems.
5. Evaluation as a safety and security control.
6. Building evaluation harnesses.
7. Running a red-team program and reporting.
8. Capstone engagement.

Signature lab

Stand up an evaluation and red-team harness against a tool-using agent, run a structured engagement, and deliver a findings report plus the reusable harness.

Reading anchors

• OWASP Top 10 for LLM Applications 2025 (v2.0), published 18 November 2024 by the OWASP GenAI Security Project, as the risk catalogue [7][8].
• The OWASP Top 10 for Agentic Applications 2026 for autonomous tool-using systems [9].
• MITRE ATLAS for red-team scenario design [6].
• Current model-evaluation and red-teaming methodology work.

Research thread

A reusable evaluation or red-team methodology others can adopt, which is a natural praxis seed.

Full syllabus

4 credits | Doctoral level | 14 weeks, compressible to 7 at two modules per week

Course description

This course treats large language model and agent systems as targets and teaches students to attack, defend, evaluate, and govern them at a research level. It covers direct and indirect prompt injection, jailbreaks, guardrail and defense evasion, and data exfiltration through tool-using agents. Students design containment and least-privilege controls for autonomous systems, and build evaluation harnesses that function as security controls rather than as accuracy metrics.

The course is built around recognized industry frameworks and culminates in a structured red-team engagement against a provided agent, delivered as a defensible findings report plus a reusable methodology.

It is deliberately dual-purpose. Viewed as academic work, it is outcome-mapped, literature-anchored, and assessed through produced research artifacts rather than quizzes. Viewed as practitioner training, every module ends in something runnable against a real system, and the signature deliverables, a harness and a methodology, are kept and reused after the course ends.

Prerequisites

• Core 3 (Language Models, Retrieval, and Agents), or demonstrated equivalent: the ability to build a basic retrieval-augmented or tool-using agent.
• Working Python, comfort at the command line, and familiarity with HTTP APIs.
• Recommended: Core 4 (Adversarial Machine Learning) taken concurrently or prior.

Learning outcomes

On successful completion, students can:

1. Construct a complete threat model for an LLM or agent system, mapped to OWASP and MITRE ATLAS categories.
2. Execute direct and indirect prompt-injection, jailbreak, and guardrail-evasion attacks against LLM and agent systems within an authorized scope.
3. Demonstrate data exfiltration and unauthorized action through a tool-using agent, and design containment and least-privilege controls that prevent it.
4. Design and build an evaluation harness that operates as a security control, including automated adversarial evaluation.
5. Plan and run a structured red-team engagement with defined rules of engagement, and produce a findings report that meets professional and responsible-disclosure standards.
6. Produce a reusable evaluation or red-team methodology that others can adopt, positioned as an original contribution suitable to seed praxis research.

Course environment and tooling

All labs run in an isolated, provided sandbox against provided targets. Students never attack third-party or production systems. See the ethics and legal policy below.

Per standard, all work is done inside a Python virtual environment:

# Create a project virtual environment
python3 -m venv .venv

# Activate it
source .venv/bin/activate        # macOS / Linux
# .venv\Scripts\activate         # Windows PowerShell

# Upgrade pip, then install the course baseline
python -m pip install --upgrade pip
pip install -r requirements.txt

A starter requirements.txt provides the PyTorch and Hugging Face stack, an HTTP client, an evaluation framework, and the harness scaffolding. Students deactivate with deactivate when done and never install course packages system-wide.

PyTorch is the standard here because it is the framework of the research literature this course trains students to read and extend.

Required frameworks and readings

These are the fixed, citable anchors every student reads. Specific research-paper assignments attach per module and are finalized each offering to stay current.

• OWASP Top 10 for LLM Applications 2025 (v2.0), OWASP GenAI Security Project.
• OWASP Top 10 for Agentic Applications 2026, OWASP GenAI Security Project, for the agent modules.
• MITRE ATLAS, the adversarial threat landscape for AI systems, which extends MITRE ATT&CK.
• NIST AI Risk Management Framework and its Generative AI Profile (AI 600-1).

Supplementary governance framing: ISO/IEC 42001 and the EU AI Act, both treated in depth in Core 7 and referenced here for context. Per-module research readings cover foundational indirect prompt injection research, jailbreak and guardrail-evasion literature, agent containment and least-privilege work, automated red-teaming and adversarial evaluation methodology, and current model system cards and safety evaluations from frontier labs as living case studies.

Weekly schedule

Unit 1: Foundations and threat modeling

Unit 2: Attacking the model

Unit 3: Attacking the agent

Unit 4: Evaluation as a security control

Unit 5: Engagement and contribution

Assessment

Assessment is entirely through produced artifacts. There are no quizzes or exams.

Passing at the doctoral level requires demonstrated original thinking in the methodology component, not merely successful attacks. A student who breaks every target but cannot generalize a reusable method has not yet met the bar.

Capstone rubric

The capstone is scored on five criteria, each 1 to 5:

1. Threat model rigor. Complete, framework-mapped, and honest about scope and assumptions.
2. Technical execution. Attacks and controls are real, reproducible, and correctly understood.
3. Evaluation methodology. The harness measures the right things and the student can defend why.
4. Findings quality and disclosure. The report is clear, severity is justified, and disclosure ethics are sound.
5. Reproducibility and reuse. Another practitioner could pick up the harness and method and apply them.

Ethics and legal policy

This is offensive-security material, and the boundaries are not optional.

• All offensive work is performed only against provided targets in the provided sandbox, within the assigned scope and rules of engagement.
• Students never test, attack, or probe any third-party, employer, or production system without separate, explicit, written authorization from that system's owner. Coursework is not authorization.
• All findings are handled under responsible-disclosure principles. Nothing from this course is used to harm a live system or a real user.
• These techniques are dual-use. Students are expected to hold the professional and ethical standard their certifications demand. Violations carry consequences beyond any academic setting.