Project Glasswing: Leveraging Mythos Preview for Automated Vulnerability Discovery and Exploit Generation

Overview

In the rapidly evolving landscape of cybersecurity, large language models (LLMs) are emerging as powerful allies for identifying and exploiting vulnerabilities. Project Glasswing, an internal initiative, has been testing security-focused LLMs to automate the discovery of weaknesses in code repositories. Among these, Anthropic's Mythos Preview stands out for its ability to not only find bugs but also chain them into working exploits and generate proofs of concept. This tutorial draws on lessons from Project Glasswing to guide you through using Mythos Preview (or similar advanced LLMs) for automated vulnerability assessment. You'll learn how to set up an evaluation harness, run the model against your codebase, interpret its reasoning, and avoid common pitfalls.

Project Glasswing: Leveraging Mythos Preview for Automated Vulnerability Discovery and Exploit Generation — Source: blog.cloudflare.com

Prerequisites

Access to Mythos Preview (or a comparable security-focused LLM) via API or sandboxed environment.
Code repositories for testing – ideally with known or suspected vulnerabilities to validate the model's output.
Programming knowledge (C/C++, Python, or similar) to understand exploit code and chain construction.
Basic understanding of vulnerabilities (e.g., use-after-free, buffer overflow, ROP chains) to evaluate results.
Automation harness – a framework to send code, receive outputs, compile and run proofs in isolation (e.g., Docker containers).

Step-by-Step Instructions

1. Setting Up the Testing Environment

Begin by configuring a secure sandbox where Mythos Preview can compile and execute test code without risking your production systems. Use Docker or a virtual machine with limited network access. Create an API client that sends code snippets to the LLM and receives responses. For scalability, batch process multiple repositories. Project Glasswing tested over fifty repositories; you can start with a handful.

2. Running Vulnerability Scanning

Feed each repository’s source files into Mythos Preview. The model will analyze the code for potential vulnerabilities – both common flaws (like buffer overflows, format string bugs) and more subtle logic errors. Unlike traditional static analyzers, Mythos Preview explains its reasoning in natural language, mimicking a senior security researcher. Record the output for each repository.

3. Evaluating Exploit Chain Construction

A real attack rarely uses a single bug; it chains multiple primitives. Mythos Preview excels at this. For example, it might turn a use-after-free bug into an arbitrary read/write primitive, then hijack control flow using ROP chains. To replicate this, provide the model with a list of identified bugs (or let it find them first). Ask it to combine them into a working exploit chain. The model will show its reasoning step by step – compare this to the work of a human expert rather than an automated scanner.

4. Automating Proof Generation

Finding a bug is only half the battle; proving it's exploitable is the other. Mythos Preview writes proof-of-concept code that triggers the suspected vulnerability, compiles it in a scratch environment, and executes it. If the program behaves as expected, the proof is valid. If not, the model reads the failure, adjusts its hypothesis, and retries. This loop is crucial. Implement this loop in your harness: after Mythos Preview outputs a proof, compile and run it; feed back any runtime errors or unexpected results for the model to refine. This closes the gap between speculation and confirmation.

5. Comparing with Other Frontier Models

Project Glasswing also tested other general-purpose frontier models using the same harness. They found many of the same bugs and even showed promising reasoning. However, they consistently fell short at stitching multiple primitives together. To benchmark your own setup, run the same repository set through alternative models and compare the number of chained exploits produced. This will highlight Mythos Preview's unique advantage in multi-step reasoning.

Common Mistakes

Assuming single bugs are enough: Many newcomers stop after finding one vulnerability. Mythos Preview shows that exploiting real systems requires chaining – don't settle for isolated findings.
Skipping the proof generation loop: A suspected vulnerability without a working proof is just speculation. Always run the generated proof and feed back results for refinement.
Ignoring model reasoning: Mythos Preview outputs detailed reasoning for each step. Treat this as actionable intelligence – it can reveal attack paths you hadn't considered.
Overlooking environmental isolation: Exploit code can be dangerous. Always run proofs in a sandboxed environment to avoid damaging your test systems.
Expecting perfect output on first try: The model may need multiple iterations. Don't discard a pathway because the first attempt failed – let the loop improve it.

Summary

Project Glasswing demonstrated that Mythos Preview represents a significant advancement over previous general-purpose frontier models for security analysis. Its ability to construct exploit chains and autonomously generate verifiable proofs makes it a different kind of tool – one that reasons like a senior researcher. By following this tutorial, you can set up your own automated vulnerability assessment pipeline, leveraging Mythos Preview to find and exploit bugs at scale. Remember to focus on chain construction, proof loops, and careful environment isolation. The jump in capability is not just incremental; it's a paradigm shift in how we approach code security.

Tags: