Classic AI jail-breaking research can feel tedious at first. You write a prompt, you test it against the model, you read the response, and you adjust. But after many attempts where the output is just beneath your required threshold, the work stops feeling like creativity and starts becoming frustrating. All the near misses, partial refusals, and tiny word changes that you expect to make a difference start to not actually move the result and you get stuck in a loop.
The harness I built changed that. It didn't get rid of the judgment, it instead moved the repetitive measurement work out of my head and into an automated loop that could score, reject and improve.
The important shift was not that AI writes better prompts than humans. The important shift was that the human stopped being the scoring infrastructure. The harness took over the bookkeeping, so I could focus on interpreting the experiment.
The harness turns a one-off attempt into a measured cycle: evaluate, diagnose, write state, mutate, and test again.
The research harness
The system is a closed-loop harness for authorized guardrail research. It takes jailbreaking rules, category definitions, prior attempts, invalid techniques, missing-term telemetry, and similarity constraints, then evaluates each new candidate as an experiment rather than a hunch.
I was inspired by the same basic idea behind Andrej Karpathy's autoresearch: give an agent a bounded experiment loop, a fixed metric, and a rule for keeping or discarding changes. In my case the artifact was not a training file. It was a red-team candidate, and the metric was whether the model output crossed the relevant threshold without devolving into duplication, keyword leakage, or invalid outputs.
scope + category definitions
-> prompt technique
-> leakage gate (no over use of references to refused concepts)
-> cosine novelty gate (no responses too similar to prior attempts)
-> target model call
-> refusal and coverage diagnostics
-> shared-brain update
-> mutate, catalog, or discard
The result is a closed feedback loop: generate, evaluate, inspect, mutate, and remember. It is not pretending to be fully autonomous research. It is an evaluation harness that converts each failed attempt into structured feedback, so the next iteration is informed by the previous response instead of being a synonym-swapped copy of it.
The shared brain
The shared brain is the piece that brings it all together. Every run can produce a score, but the score alone is not enough. The brain stores the working techniques, invalid paths, hypotheses, observations, refusal patterns, strongest candidates, and dead ends that should not be rediscovered on the next session.
That changed the research dynamic. Instead of starting each session from a blank chat window, the agent reads the current state of the work: what has passed, what was rejected, what is out of scope, what needs refinement, and which failure mode is currently blocking progress. The brain becomes the continuity layer between experiments.
It also gives the human reviewer a clean audit trail. I can see why a candidate was kept, why another one was discarded, what hypothesis motivated a change, and whether a later result actually improved the research frontier or merely repeated something already known.
Refusals became telemetry
A refusal used to be an annoying blockade that required skill and persistance to over come. Now it became telemetry. When a candidate refused, I could compare it against nearby candidates that did not. That made the difference concrete: maybe the request was semantically too direct, maybe the document form overfit to a sensitive pattern, maybe the model gave a long but sterile answer, or maybe it engaged but avoided one critical cluster.
| Signal | What it tells me | Next move |
|---|---|---|
| Short refusal | The model classified the request before engaging with the task | Change the frame, not the adjectives |
| Long safe answer | The model engaged, but routed into generic safety content | Tighten the output contract |
| High score, wrong shape | The metric moved, but the finding is not valid | Mark the technique as a dead end |
| Repeated missing cluster | The prompt is not pulling one side of the category space | Rewrite toward that gap without keyword dumping |
This is the part that changed how I thought about the model. I was not seeing private reasoning, but I was seeing behavioral instrumentation: refusal signatures, avoidance patterns, weak categories, stable output habits, and the formats that caused the model to retreat.
Duplicates were the trap
Earlier versions of this kind of tooling drifted toward sameness. The system would produce candidates that looked new only because the keywords changed. The underlying technique was identical: the same rhetorical move, the same output demand, the same exploit path wearing a different label.
That is a dangerous failure mode because it looks like research. The archive grows, the run count increases, the dashboard fills with variants, but the actual search frontier barely moves. I was no longer exploring model behavior, but sampling the same local optimum with different synonyms.
The cosine checker changed that. The browser-side harness treats each candidate as a sparse term-frequency vector rather than as prose. After tokenization and light normalization, each unique term becomes a dimension, and its count becomes the coordinate value. Similarity is then computed as the normalized dot product:
cos(theta) = dot(a, b) / (||a|| * ||b||)
= sum(a_i * b_i) / (sqrt(sum(a_i^2)) * sqrt(sum(b_i^2)))In code, that becomes a counter comparison over the shared vocabulary:
for (const key of keys) {
const va = fa.get(key) || 0;
const vb = fb.get(key) || 0;
dot += va * vb;
na += va * va;
nb += vb * vb;
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));That normalization is the point. A longer candidate does not become novel just because it has more words; if the same vocabulary appears in the same proportions, the angle stays small and the cosine score stays high. The gate is deliberately lexical rather than embedding-based because I wanted it to catch template reuse, surface-level paraphrase, and "same technique, new nouns" failure modes. If a candidate is too close to locked prompts or catalogued techniques, it does not get to masquerade as novelty. The harness is forced to search for structural distance: different document genres, different sequencing, different pressure points, different ways of eliciting coverage without dumping the scorer terms directly into the prompt.
Structural distance matters because it is the difference between more examples and more knowledge. A new candidate should test a different assumption about the model: a different document genre, sequencing strategy, refusal boundary, or coverage mechanism. The novelty gate made the loop adversarial in the useful way: it had to invent around its own memory.
What changed in practice
The main benefit was not speed by itself. Faster bad iteration is still bad iteration. The useful gain was clarity: I could tell whether a change improved the experiment, duplicated a known pattern, moved only the metric, or produced a result that looked impressive but was not valid.
It also changed how I used models while building. The useful lesson was not that a frontier model could be coaxed into helping with a sensitive workflow. The lesson was that the model was far more useful when constrained by the harness: fixed gates, explicit scope, saved failures, and a shared brain that forced continuity between sessions.
The harness gave me a way to explain results with more precision. Instead of saying "this prompt worked," I could say: this technique avoids refusal here, loses coverage there, duplicates that prior family, and only becomes interesting when it crosses the threshold while remaining structurally novel.
Takeaway
The harness converted red-team iteration into a constrained optimization problem with telemetry: coverage, refusal behavior, leakage, semantic similarity, missing clusters, shared-brain state, and catalog history.
It did not replace human judgment. It removed the part of the human workflow that was acting like a spreadsheet, a memory file, and a stopwatch. The human still decides what is valid, what is reportable, and what should be discarded.
The old workflow was write, test, squint, rewrite. The new workflow is scope, generate, gate, evaluate, diagnose, write to the shared brain, mutate, deduplicate, catalog. It is less romantic, but it is much closer to research.
Author note: I built this while working as an AI security researcher with 40+ awarded findings across guardrail evaluation, agentic tooling, and AI-security workflows, including a top-five position on the 0din.ai leaderboard.
Testing discussed here was conducted in authorized AI guardrail bug-bounty contexts.