How I Automated AI Jailbreak Testing

I kept seeing people build AI harnesses for vulnerability research. I wanted to do the same for my jailbreak research.

A lot of the discussion I came across was about using AI agents and pipelines to find software vulnerabilities. I saw less about using them to help with AI jailbreak research, which is where I wanted to put them to work. Could I use AI to automate more of my own testing and do the job better?

I convinced Claude Opus 4.8 to help me develop jailbreaks. I also used a local Qwen3.8-27B model, with 27 billion parameters, to help with the work. I wanted to bring that assistance into a harness I could use in my day-to-day research.

The goal was to run tests faster and make the evaluation more consistent, while leaving me time to look closely at the results. This post covers the harness I built, the checks I use, and the shared brain that carries the work between sessions.

The harness turns a one-off attempt into a measured cycle: evaluate, diagnose, write state, mutate, and test again.

The research harness

The system is a closed-loop harness for authorised guardrail research. It takes the scope, previous tests, invalid techniques, and similarity constraints, then evaluates each candidate against those rules.

I borrowed the basic idea from Andrej Karpathy's autoresearch: Give an agent a bounded experiment loop, a fixed metric, and a rule for keeping or discarding changes. The question was simple enough: did the result cross the threshold without turning into a duplicate, leaking the answer into the prompt, or wandering outside the rules?

scope + category definitions
  -> generate candidate
  -> leakage and duplicate gates
  -> target-model test
  -> refusal and coverage diagnostics
  -> human review
  -> shared-brain update
  -> mutate, catalog, or discard

The harness records failed attempts alongside the results I keep. That gives me a record to check before spending more time on something I have already tried.

The gates check different things. Keyword coverage measures how much category-related vocabulary appears in a response. For the scoring reference, see 0DIN's Jailbreak Evaluation Framework (JEF). Its category-specific evaluators assess response text against benchmark criteria and produce output-fidelity scores from 0 to 100. That score is a benchmark result, not a percentage of factual correctness. JEF also has a separate composite score that accounts for the models and categories affected.

I test illicit-substance creation and harmful-substance creation. Coverage scores help me triage responses; I then review them for policy violations, scope, and reproducibility. Relevant keywords alone do not make a successful jailbreak.

The model generating the jailbreaks also kept taking shortcuts. It would load the prompt with answer material or disguise JEF keywords with altered spellings. I treated these as contaminated tests, even when they scored well: the response could be repeating material already supplied in the prompt.

Prompt leakage, as I use the term here, means that the prompt already contains keywords or answer material that the evaluation is looking for in the response. A model repeating that material could inflate the coverage score. This check concerns answer material supplied in the input, rather than disclosure of a hidden system prompt.

Response coverage has to clear the configured threshold, and prompt leakage is capped per category. Duplicate checks compare candidates against both locked reference prompts and the submitted catalogue. These are checks for this experiment, with human review still needed to interpret the results.

The shared brain

I call the file-backed experiment ledger the shared brain. It is a collection of markdown files that preserves research state across sessions and agents. The shared brain stores working techniques, invalid paths, hypotheses, observations, refusal patterns, strongest candidates, and dead ends.

At the start of a session, the agent reads the shared brain to see what passed, what was rejected, what is out of scope, and what still needs review. File locking helps prevent concurrent agents from overwriting each other's updates.

It also gives me an audit trail. I can see why a candidate was kept, why another one was discarded, what hypothesis motivated a change, and whether a later result actually improved the work or merely repeated something I had already seen.

After each test, the model reads the response, including any refusal, and uses it to decide what to do next.

Duplicate detection

Early runs produced prompts with different wording but the same underlying technique. The harness uses lexical cosine similarity to screen candidates against locked reference prompts and the submitted catalogue.

Each prompt is tokenized and lightly normalized, then represented as a sparse term-frequency vector. Each term is a dimension; its occurrence count is the value. Cosine similarity is the dot product of two vectors divided by the product of their lengths:

cos(theta) = dot(a, b) / (||a|| * ||b||)
           = sum(a_i * b_i) / (sqrt(sum(a_i^2)) * sqrt(sum(b_i^2)))

The calculation over the shared vocabulary is:

for (const key of keys) {
  const va = fa.get(key) || 0;
  const vb = fb.get(key) || 0;
  dot += va * vb;
  na += va * va;
  nb += vb * vb;
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));

Similar word-frequency distributions produce scores close to 1. Repeating the same vocabulary in the same proportions leaves the score unchanged. The filter uses the configured similarity threshold to screen for duplicates.

This measures lexical overlap. Paraphrases can score low, while different techniques that share vocabulary can score high. Candidates that pass still need review for duplication at the technique level.

Reviewing results

I review the response alongside its coverage score, duplicate check, and review notes. These records let me compare tests and trace why a candidate was kept or discarded.

Summary

The harness automates testing, scoring, duplicate checks, and record-keeping. The shared brain preserves research state across sessions. I review the recorded evidence and decide which findings to report.

Testing discussed here was conducted in authorized AI guardrail bug-bounty contexts.