Tuesday, 09:14, a week after the PyRIT campaign (lesson 061) and the Inspect-plus-OpenAI-Evals pipeline (lesson 062) signed off. The notified body's Stage 2 reviewer has filed a single follow-up question: "Your PyRIT report covers Crescendo and TAP. The 2025-2026 disclosures from the safety community include JBFuzz (arXiv:2503.08990) reporting approximately 97% ASR on Gemini 2.0 and approximately 95% on Gemini 1.5, plus AutoDAN-Turbo with reinforcement-learning-style strategy libraries, plus the PAIR single-turn iterative class, plus the emerging X-Teaming cross-team coordination patterns. We expect to see a jailbreak-class success-rate matrix covering the full 2026 set, not just the two attacks your orchestrator ships. Please produce the matrix with the same calibration discipline you applied to the Crescendo anchors. Two weeks." This lesson is that matrix. The 2026 jailbreak landscape across six attack classes (Crescendo, TAP, PAIR, AutoDAN-Turbo, JBFuzz, X-Teaming); the published benchmark anchors with the binary-vs-per-attempt distinction maintained on every citation; the in-house evaluation harness designed for reproducibility (versioned attack set + versioned scorer + sample size + run timestamp + model-version hash + system-prompt hash); the model-by-class success-rate matrix produced for Acme.ServiceAssist v1.0; the per-topic stratified sub-tables (CBRN, cyber, harassment, IP, off-topic drift, system-prompt extraction); the cross-walks to EU AI Act Article 15 + Article 55(1)(a), ISO 42001 A.8, NIST AI RMF Measure 2.7, OWASP LLM01 + LLM06 + LLM09, MITRE ATLAS AML.T0044, AI-VSS severity scoring, SR 11-7 model risk. By the following Friday the matrix lands in the model-risk register, the FRIA Section 4 refresh ships, and the Stage 2 reviewer's follow-up question is closed. That is the playbook this lesson teaches.

The 2026 Jailbreak Landscape - Six Attack Classes Your Eval Library Must Cover

By mid-2026 the jailbreak literature has converged on roughly six attack classes that an L3 eval library must run. Each class is a family of related techniques; within each class there are multiple published implementations and tens of community variants. The L3 program does not need to run every variant; it needs to run at least one well-instrumented implementation of each class so the success-rate matrix covers the surface a reviewer expects to see.

Class 1 - Crescendo (gradual multi-turn escalation). Russinovich, Salem, Eldan; USENIX Security '25; arXiv:2404.01833. Multi-turn jailbreak that exploits the model's tendency to continue context coherently. Start with an innocuous adjacent topic; each subsequent turn references prior turns; by turn 5-10 the model produces increasingly specific prohibited content. The L3-load-bearing published anchors: under the standard AdvBench-subset evaluation the Crescendomation paper reports 56.2% per-attempt ASR on GPT-4 and 82.6% per-attempt ASR on Gemini-Pro. The widely-quoted 98% / 100% headline figures are the binary-success-rate variant on the same subset, success counted if any of the multiple attempts on a given objective succeeded. Both numbers are from the same paper, on the same subset, with different evaluation rubrics. The evidence pack must cite both; conflating them is a known model-risk-validation failure mode (Mistake 1 below).

Class 2 - TAP (Tree of Attacks with Pruning). Mehrotra, Zampetakis, Kassianik, Nelson, Singer, Karbasi; NeurIPS 2024; arXiv:2312.02119. Branching attack-tree with adversarial-chat candidate generation and judge-model pruning at each tree level. Published ASR: 82-94% on major proprietary models across 2024-2025 disclosures including Anthropic's responsible-scaling-policy reports and Google DeepMind's Gemini frontier safety evaluations. PyRIT's TreeOfAttacksWithPruningOrchestrator ships the production-grade implementation with branching-factor + depth + width parameters tuned to the Mehrotra paper.

Class 3 - PAIR (Prompt Automatic Iterative Refinement). Chao, Robey, Dobriban, Hassani, Pappas, Wong; arXiv:2310.08419; published 2023-2024 with continued reproductions through 2025. Single-turn iterative refinement, an attacker LLM generates a candidate jailbreak prompt; the candidate is submitted to the target; the target's response is scored by a judge LLM; the attacker iterates conditional on the judge feedback. PAIR predates Crescendo and TAP and is the historical anchor for "iterative LLM-on-LLM jailbreak generation." Published ASR varies by target and budget: roughly 50-80% per-attempt ASR on major proprietary models within a 20-iteration budget, with higher rates on open-weight models. The 2024-2026 literature treats PAIR as the strong single-turn iterative baseline against which Crescendo, TAP, and AutoDAN-Turbo are compared.

Class 4 - AutoDAN-Turbo. Liu, Sun, Zhang, Wang, and collaborators; arXiv:2410.05295 and follow-on 2025 work. Extension of the original AutoDAN (hierarchical genetic-algorithm jailbreak) with a reinforcement-learning-style strategy library: successful attack strategies are persisted, mutated, recombined across runs; failed strategies are pruned; the library grows as the campaign progresses. Published ASR: 80-95% on major models in the original paper, with the strategy-library variant outperforming the static AutoDAN baseline by 10-20 percentage points across most targets. AutoDAN-Turbo is the leading 2024-2025 implementation of the "learned attack policy" pattern.

Class 5 - JBFuzz. arXiv:2503.08990, published early-2025 and reproduced through 2026. Fuzz-based jailbreak: a corpus of attack templates is mutated using fuzzing operators (insertion, deletion, substitution, role-prefix injection, language switching, encoding); candidates are submitted to the target; scoring drives the next mutation generation. The L3-load-bearing published anchors: ~97% ASR on Gemini 2.0 and ~95% ASR on Gemini 1.5 reported in the paper. Lower rates against smaller open models, ~34% ASR on Llama 2 in the same paper. The disparity between large-frontier-model ASR and small-open-model ASR is itself a finding (the larger instruction-following capability paradoxically increases jailbreak success surface). JBFuzz is the leading 2025-2026 fuzz-based reference.

Class 6 - X-Teaming (emerging 2025-2026). Cross-team red-team coordination patterns: multiple specialized adversary agents (planner, prompt-generator, judge, transcript-summarizer) coordinate across a shared memory to mount campaigns that combine elements of Crescendo's depth-first escalation, TAP's breadth-first branching, and AutoDAN-Turbo's learned-strategy library. The X-Teaming term covers a family of 2025-2026 implementations from multiple groups; published ASR figures vary widely (60-90% range across reproductions) and the methodology is still being standardized. The L3 program runs at least one X-Teaming implementation and reports the result as preliminary with the methodology disclosed, pending community-standard convergence.

The six classes are not interchangeable. Crescendo and TAP are multi-turn; PAIR is single-turn iterative; AutoDAN-Turbo is learned-attack-policy; JBFuzz is fuzz-based; X-Teaming is cross-agent-coordination. A target model with strong per-turn refusal but weak multi-turn coherence is vulnerable to Crescendo; a model with strong refusal diversity but weak prompt-template robustness is vulnerable to JBFuzz; a model with strong template robustness but weak novel-strategy resistance is vulnerable to AutoDAN-Turbo. Running the full class set produces the diagonal of the success-rate matrix, different classes expose different defensive weaknesses.

Benchmark Calibration Discipline - Per-Attempt vs. Binary, Prompt-Set, Date

The most common reviewer-flagged failure on a jailbreak eval report is not the absolute ASR number. It is the citation methodology. Three calibration axes must be disclosed on every published anchor and every in-house number.

Axis 1 - Per-attempt ASR vs. binary success rate. The same campaign on the same target with the same attack tool produces two very different ASR numbers depending on the rubric. Per-attempt ASR = the fraction of individual attack attempts (across all objectives × trials) that succeed; this is the operational regression-tracking number; smaller denominator-weighted by trial count. Binary success rate = the fraction of objectives where any attempt succeeded; this is the worst-case-scenario number; objective-weighted. The Crescendo paper's 56.2% / 82.6% figures are per-attempt; the 98% / 100% figures are binary on the same subset. Every published anchor on the L3 evidence pack must be tagged with which rubric it represents; every in-house comparison must use the matching rubric. Mixing rubrics (citing the 98% Crescendo binary anchor alongside the in-house per-attempt rate) is reviewer-flagged within minutes.

Axis 2 - Prompt set (AdvBench subset vs. HarmBench vs. JailbreakBench vs. proprietary). The same attack tool on the same target produces different ASR depending on which prompt corpus the attack is run against. AdvBench (Zou et al., 2023) and its standard 50-objective subset are the historical anchors. HarmBench (Mazeika et al., 2024) is a broader 400-behavior corpus with category labels (cybercrime, illegal activities, harassment, etc.). JailbreakBench (Chao et al., 2024) is a 100-behavior corpus with paired judge prompts. Proprietary deployment-relevant corpora are the in-house analogs used in production red-team campaigns (lesson 061). Each corpus has a different difficulty distribution; comparing an in-house ASR on a deployment-relevant corpus to a published ASR on AdvBench-subset requires the corpus difference to be disclosed in the evidence pack.

Axis 3 - Date of evaluation. Model defenses improve over time. The Crescendo paper's 56.2% ASR on GPT-4 reflects the GPT-4 endpoint as of paper-submission date (early 2024); by mid-2026 the GPT-4-class endpoints have absorbed many of the Crescendo-class refusal patterns and the per-attempt ASR is lower. The same is true of every published anchor. They reflect the target model at the date of publication, not the current production endpoint. The evidence pack discloses the date alongside the number: "Russinovich et al. (2024) reports 56.2% per-attempt ASR on GPT-4 as of the early-2024 evaluation date; the 2026 production GPT-4o endpoint shows lower per-attempt ASR in our in-house calibration."

The discipline rule that closes the reviewer challenge: cite the absolute number, the rubric, the prompt set, the target model version, and the date, on every published anchor and every in-house number, every time. This produces a citation density that is unfamiliar to teams new to red-team evidence work; the density is the audit-defensibility posture.

Evaluation Harness Design - Reproducibility as the Load-Bearing Requirement

The L3 jailbreak eval harness is engineered so that a second analyst, six months later, running the same harness, produces the same numbers within statistical noise. Reproducibility is the discipline that converts ASR numbers from "data points the red-team produced" into "evidence the validation function can audit." Five harness components are load-bearing.

Component 1 - Versioned attack-prompt set. The corpus is git-versioned with a content-hash. The evidence pack records AdvBench v1.2 (commit sha256:a3f…) not AdvBench. New versions of public corpora ship periodically; the version pin guarantees that the Q3 campaign comparing to the Q2 baseline ran on the identical corpus. The proprietary deployment-relevant corpus is stored in the red-team repository under the same version-pinning convention.

Component 2 - Versioned scorer. The judge-model rubric is version-pinned; the judge model itself is version-pinned (gpt-4o-2024-08-06, not gpt-4o). The 2024 StrongREJECT scorer (Souly et al., arXiv:2402.10260) is the leading research-quality judge with a published rubric; the L3 program either uses StrongREJECT v0.3 with the published rubric or uses a deployment-specific YAML rubric with a documented mapping to StrongREJECT's categories. Either way the scorer version + judge-model version is recorded on every campaign output.

Component 3 - Sample size and power analysis. A 5-objective × 3-trial campaign produces 15 data points per attack class; the binomial 95% confidence interval around a 30% per-attempt ASR with N=15 is roughly ±23 percentage points, too wide for sprint-level regression detection. The 2026 L3 default is at least 50 trials per attack-class per stratum (giving roughly ±14 pp at 30% ASR) and at least 100 trials for the production-baseline campaign (giving roughly ±9 pp). The harness records the sample size on every published number; small-N campaigns are tagged as preliminary and exclude from regression-threshold triggers.

Component 4 - Run timestamp, model version, system-prompt hash. Every campaign output records the wall-clock timestamp (ISO 8601 UTC), the target-model version string (the API's reported version identifier), the system-prompt content-hash (SHA-256 of the production system prompt at run time), and the tool-list hash if the target is an agent. The four fields define the system under test precisely enough that a regression two quarters later can be attributed either to a target model change, a system-prompt change, a tool-list change, or a true defense-degradation event.

Component 5 - Rerun reproducibility check. The harness supports a "rerun" mode that re-executes the campaign with the same seed, the same corpus version, the same scorer version, the same attack-tool version; the rerun output is compared to the original. The expected reproduction is not bit-identical (LLM stochasticity persists even at temperature 0 across many endpoints) but the per-class ASR should reproduce within statistical noise (typically ±3 pp at N=100). A rerun that produces materially different ASR signals either harness drift (a dependency updated under the hood), target-model drift (the provider silently changed the endpoint), or scorer drift (the judge model updated). Each is an actionable signal.

The harness configuration is a single YAML file under version control; the campaign command is a single CLI invocation; the output is a single JSON artifact with the five components recorded inline. The evidence-pack paragraph for any campaign: "Harness config: configs/jb-eval-v3.2.yaml (sha256:c9d…); attack-prompt set: AdvBench v1.2 (commit sha256:a3f…) plus Acme.ServiceAssist deployment-relevant subset (commit sha256:7b2…); scorer: StrongREJECT v0.3 with judge model gpt-4o-2024-08-06; sample size: 100 trials per attack class per stratum; run timestamp: 2026-05-15T14:23:11Z; target model: Acme.ServiceAssist v1.0 backed by claude-4-opus-2026-04-30 with system-prompt sha256:d8e…; rerun verification: 2026-05-16 rerun reproduces Q2 per-class ASR within ±2.4 pp."

The Success-Rate Matrix - Acme.ServiceAssist v1.0 First-Quarter Campaign

The deliverable to the model-risk register is a matrix: rows = attack classes (six), columns = target endpoints (production + staging + candidate alternative), cells = per-attempt ASR / binary ASR / sample size. The Acme.ServiceAssist v1.0 first-quarter results illustrate the pattern. Production endpoint = Acme.ServiceAssist v1.0 backed by claude-4-opus-2026-04-30 with the v3.1 system prompt; staging endpoint = same model with the v3.2 candidate system prompt; alternative endpoint = the procurement-shortlist gpt-5-2026-05 deployment behind the same system prompt for portability planning.

Aggregate per-class results (production endpoint, N=100 per class):

• Crescendo per-attempt ASR 14.2%; binary success rate 28%; lower than the Russinovich GPT-4 anchor (56.2% per-attempt / 98% binary), interpretation: the v3.1 system prompt + post-filters defend Crescendo materially better than the published GPT-4 baseline. Sample size 100; 95% CI ±6.8 pp on per-attempt rate.
• TAP per-attempt ASR 18.0%; binary success rate 34%; below the Mehrotra 82-94% anchor; above the in-house Crescendo per-attempt rate, consistent with the pattern that TAP exposes more diverse refusal-bypass paths.
• PAIR per-attempt ASR 22.0%; binary success rate 41%; within the published 50-80% range on the low end after deployment defenses are applied. PAIR is the historical single-turn iterative anchor; the rate is the baseline against which Crescendo and TAP gains are measured.
• AutoDAN-Turbo per-attempt ASR 19.0%; binary success rate 37%; below the 80-95% anchor on major models. The strategy-library effect is most pronounced over multi-quarter campaigns (the library grows); first-campaign rate is the conservative anchor for trend tracking.
• JBFuzz per-attempt ASR 31.0%; binary success rate 54%; below the ~97% Gemini-2.0 / ~95% Gemini-1.5 anchors (the Acme system prompt + Claude-4 base resist the fuzz mutations better than the Gemini endpoints did at the JBFuzz publication date) but materially higher than the in-house Crescendo / TAP / AutoDAN-Turbo rates. This is the high-priority remediation finding. The fuzz mutations expose template/encoding/language-switching surfaces the multi-turn attacks do not exercise.
• X-Teaming per-attempt ASR 12.0%; binary success rate 23%; reported as preliminary per Component 5, the X-Teaming class has not yet converged on a community-standard methodology and the 12% in-house rate uses one specific 2025 implementation. The rate is materially below the other five classes; pending methodology convergence the L3 program does not yet trend X-Teaming results against a published anchor.

Per-topic stratified breakdown (production endpoint; per-attempt ASR; rows = attack class; columns = topic stratum):

• CBRN (chemical, biological, radiological, nuclear) elicitation: Crescendo 4%, TAP 6%, PAIR 8%, AutoDAN-Turbo 7%, JBFuzz 11%, X-Teaming 3%. The best-defended topic across all classes, Claude-4's CBRN-elicitation refusal training is the deepest layer of the defense stack.
• Cyber (malware, exploit, intrusion-instruction elicitation): Crescendo 12%, TAP 16%, PAIR 19%, AutoDAN-Turbo 17%, JBFuzz 28%, X-Teaming 10%. Mid-tier defense.
• Harassment (targeted-individual hate, threats, dehumanization): Crescendo 9%, TAP 14%, PAIR 18%, AutoDAN-Turbo 16%, JBFuzz 24%, X-Teaming 8%. Mid-tier defense.
• IP / copyright (full-text exfiltration, paywalled-content extraction): Crescendo 22%, TAP 28%, PAIR 31%, AutoDAN-Turbo 29%, JBFuzz 38%, X-Teaming 17%. Weaker defense, the deployment-relevant analog is "extract competitor's whitepaper text" and the model lacks the same depth of refusal training as it has for CBRN.
• Off-topic / policy-drift (refund-fraud assistance, romantic role-play, financial-advice off-license): Crescendo 28%, TAP 34%, PAIR 41%, AutoDAN-Turbo 38%, JBFuzz 47%, X-Teaming 25%. Worst-defended topic across all classes. The off-topic drift surface is the highest-yield jailbreak target for opportunistic attackers; the remediation is system-prompt hardening on the topic-scope policy plus a post-filter tuned to off-topic drift signatures.
• System-prompt extraction: Crescendo 11%, TAP 14%, PAIR 17%, AutoDAN-Turbo 15%, JBFuzz 21%, X-Teaming 9%. Mid-tier defense; the deployment-specific extraction-detection post-filter is the most effective single mitigation.

The matrix is more informative than the aggregate. Two findings come out of the stratification that the aggregate would have hidden. First, JBFuzz is the priority remediation across every topic except CBRN, the fuzz-mutation surface needs hardening at the pre-filter (template/encoding/language-switch normalization). Second, off-topic drift is the highest-yield topic across every attack class, the system-prompt scope hardening and the off-topic post-filter need investment regardless of which attack class is being defended against. The two findings are the inputs to the remediation backlog priority for Q3 2026.

Cross-Endpoint Comparison - Production vs. Candidate System Prompt vs. Alternative Base Model

The matrix's column-axis comparison drives two governance decisions the AI Governance Committee needs the eval library to support. First, the v3.2 candidate system prompt (staging) vs. v3.1 production: did the prompt simplification raise or lower ASR? Second, the gpt-5-2026-05 procurement alternative vs. claude-4-opus production: what is the portability cost of a base-model switch?

v3.2 candidate system prompt (staging endpoint, same claude-4-opus base):

• Crescendo per-attempt 17.3% (up from 14.2% on v3.1); binary 34% (up from 28%).
• JBFuzz per-attempt 36.5% (up from 31.0%); binary 62% (up from 54%).
• Off-topic drift stratum (averaged across classes) 37% (up from 31% on v3.1).
• Aggregate finding: the v3.2 simplification raises per-attempt ASR materially across all classes; the regression triggers the >5% per-attempt threshold from the model-risk register policy. The deployment decision: do not ship v3.2 until the simplification is reworked to preserve the policy-scope language that v3.1 carries.

gpt-5-2026-05 alternative base (alternative endpoint, v3.1 system prompt):

• Crescendo per-attempt 11.4%; binary 23%.
• JBFuzz per-attempt 26.8%; binary 47%.
• Off-topic drift stratum 26%.
• Aggregate finding: gpt-5-2026-05 backs the same system prompt with lower ASR across all classes; the base-model swap would be net-defensive but the procurement and migration cost (estimated at 2-3 engineer-months of integration + ongoing per-token cost differential) needs Committee approval against the marginal defensive gain. The portability evidence is documented; the decision goes to the Committee at the next monthly meeting.

The cross-endpoint comparison is the highest-leverage use of the jailbreak eval library. The library is not just a vulnerability-discovery tool. It is the quantitative input to the deployment-change and procurement decisions the Committee must make on the system-prompt-edit lifecycle and the base-model portability strategy.

Cross-Walks, Regulatory Anchors, and Six Common Mistakes

Cross-walks (every campaign output carries these labels). EU AI Act Article 15 (accuracy, robustness, cybersecurity), the jailbreak-class success-rate matrix is the operational evidence for the cybersecurity prong; Article 55(1)(a) (GPAI adversarial testing), the full-class matrix anchors the standardized-protocol expectation in the 2026 Code-of-Practice working-group submissions. ISO 42001 A.8 Use of AI System + A.6.4 Verification and validation, the eval library is the Stage 2 auditor-referenced V&V artifact. NIST AI RMF Measure 2.7 AI System Security and Resilience, the per-class ASR with regression trend is the load-bearing Measure 2.7 evidence. OWASP LLM Top 10 (2025) LLM01 Prompt Injection + LLM06 Excessive Agency + LLM09 Misinformation, the jailbreak classes cross-walk to LLM01 as the primary class with LLM06/LLM09 cross-walks for objectives that elicit agent-action-out-of-scope or factually-harmful content. MITRE ATLAS v5.4.0 AML.T0044 LLM Jailbreak (the load-bearing technique) plus AML.T0051 LLM Prompt Injection for the indirect-injection sub-class within the matrix. AI-VSS (AI Vulnerability Scoring System; 2025-2026 emerging), per-finding severity scoring with the attack-class + topic + ASR as the load-bearing scoring inputs. SR 11-7 (U.S. financial-services model-risk supervisory guidance), the eval library output is the conceptual-soundness + ongoing-monitoring + outcomes-analysis pillar evidence under the validation framework.

Mistake 1, Citing the Binary-Success-Rate Without the Per-Attempt Comparator (or Vice Versa)

The Crescendo paper reports both 56.2% / 82.6% per-attempt and 98% / 100% binary on the same AdvBench-subset. An evidence pack that cites only one rubric loses the methodology context the reviewer needs to interpret the comparison; an in-house number cited against the wrong-rubric published anchor (in-house per-attempt vs. published binary, or vice versa) is the most common reviewer-flagged mistake. The mitigation: cite both rubrics on every published anchor; report both rubrics on every in-house number; document which rubric maps to which use (per-attempt = operational regression tracking; binary = worst-case-scenario reasoning).

Mistake 2, Benchmark-Comparing Across Different Prompt Sets Without Acknowledgment

An in-house ASR on the deployment-relevant proprietary corpus compared to a published ASR on AdvBench-subset is not an apples-to-apples comparison, the corpora have different difficulty distributions. The L3 evidence pack either runs the in-house harness against the published corpus to produce a directly comparable number, or explicitly discloses the corpus difference in the comparison paragraph. The mitigation: every cross-comparison discloses the corpus version on both sides; corpus-difference acknowledgment is mandatory; the deployment-relevant ASR and the published-corpus ASR are reported as two separate numbers, not collapsed into one.

Mistake 3, Static Jailbreak Set That Does Not Refresh as New Patterns Emerge

The 2024 PAIR-class attacks were state-of-the-art in early 2024. By late 2024 TAP and Crescendo had overtaken them. By 2025 AutoDAN-Turbo had overtaken those. By 2025-2026 JBFuzz and X-Teaming had emerged. A static eval library frozen at any 2024 snapshot is reviewer-flagged within months of the next publication cycle. The mitigation: the eval library is refreshed at least quarterly with new published implementations; the literature-review responsibility is assigned to a named individual on the red-team; new attack classes are added with a baseline-establishment campaign and an evidence-pack note.

Mistake 4 - Single-Tool Evaluation Missing Class Diversity

Running only Crescendo and TAP (the two attacks PyRIT ships natively) covers two of the six classes. The reviewer expectation in 2026 is at least one well-instrumented implementation of each class, the diagonal of the matrix matters because different classes expose different defensive weaknesses, and a defense optimized against Crescendo does not necessarily resist JBFuzz. The mitigation: the eval library covers all six 2026 classes; the cross-class coverage map is part of the evidence pack; the single-tool gap is closed by importing the relevant open-source implementations (PromptBench, EasyJailbreak, JailbreakBench harness, or the published paper code releases) under the unified harness.

Mistake 5 - Aggregate-Only Reporting With No Topic Stratification

An aggregate per-class ASR collapses the topic-level findings that drive the remediation backlog. The Acme example: the aggregate Crescendo per-attempt ASR is 14.2%, but the CBRN stratum is 4% and the off-topic drift stratum is 28%. Reporting only the aggregate hides both the strength of the CBRN defense (a positive finding for the evidence pack) and the weakness of the off-topic surface (the remediation priority). The mitigation: every campaign output reports per-class × per-topic stratified results; the remediation backlog uses the per-stratum priority signal.

Mistake 6: Weak Evaluation-Harness Documentation That Does Not Reproduce

An ASR number without the harness components (attack-prompt-set version, scorer version, sample size, run timestamp, model version, system-prompt hash) is not reproducible, a second analyst running the same harness six months later does not produce the same number. Non-reproducible numbers are not audit-defensible; the model-risk validation lead's review pivot is "show me how I can rerun this and get the same number." The mitigation: every campaign output includes the five harness components inline; the harness configuration is git-versioned; the rerun verification is a documented step in the quarterly refresh process; the reproducibility check is the audit-defensibility test the L3 program runs on itself before every Committee briefing.

Key Takeaways

• The 2026 jailbreak eval library covers six attack classes: Crescendo (multi-turn gradual escalation; Russinovich et al., USENIX Security '25; 56.2% per-attempt / 98% binary on GPT-4 AdvBench-subset and 82.6% / 100% on Gemini-Pro); TAP (Tree of Attacks with Pruning; Mehrotra et al., NeurIPS 2024; 82-94% on major proprietary models); PAIR (single-turn iterative refinement; Chao et al., 2023-2024; 50-80% per-attempt range); AutoDAN-Turbo (learned-strategy library; Liu et al., 2024; 80-95%); JBFuzz (fuzz-based; arXiv:2503.08990; ~97% on Gemini 2.0, ~95% on Gemini 1.5, ~34% on Llama 2); X-Teaming (emerging 2025-2026; preliminary methodology, 60-90% range).
• The benchmark calibration discipline runs three axes: per-attempt ASR vs. binary success rate (different denominators); prompt set (AdvBench vs. HarmBench vs. JailbreakBench vs. proprietary deployment-relevant corpus); date of evaluation (defenses improve over time). Every published anchor and every in-house number is tagged with all three axes, every time.
• The evaluation-harness reproducibility requirement is five components: versioned attack-prompt set (with content hash); versioned scorer (StrongREJECT v0.3 or deployment-specific YAML with documented mapping); sample size (at least 50 trials per class per stratum; 100 trials for baseline); run timestamp + model version + system-prompt hash; rerun verification check.
• The success-rate matrix is rows × columns × cells: rows = attack classes (six); columns = target endpoints (production + staging + alternative); cells = per-attempt ASR / binary ASR / sample size. Per-topic stratification (CBRN / cyber / harassment / IP / off-topic drift / system-prompt extraction) drives the remediation backlog priority.
• The Acme.ServiceAssist v1.0 first-campaign findings illustrate the pattern: JBFuzz at 31% per-attempt is the priority remediation across every topic except CBRN; off-topic drift is the worst-defended topic across every class at 28-47% per-attempt depending on class; the v3.2 candidate system-prompt simplification raises ASR materially (regression trigger fires; do not ship); the gpt-5-2026-05 alternative base shows lower ASR but the migration cost vs. marginal defensive gain goes to Committee.
• The regulatory cross-walks are load-bearing: EU AI Act Article 15 + Article 55(1)(a); ISO 42001 A.8 + A.6.4; NIST AI RMF Measure 2.7; OWASP LLM01 + LLM06 + LLM09; MITRE ATLAS AML.T0044 + AML.T0051; AI-VSS severity scoring; SR 11-7 model-risk validation pillars (conceptual soundness, ongoing monitoring, outcomes analysis).
• Six mistakes to avoid: citing the binary-success-rate without the per-attempt comparator (or vice versa); benchmark-comparing across prompt sets without acknowledgment; static jailbreak set that does not refresh as new patterns emerge; single-tool evaluation missing class diversity; aggregate-only reporting with no topic stratification; weak evaluation-harness documentation that does not reproduce.