Benchmarks¶

20 registered benchmarks across 5 categories:

Category	Description
`jailbreak`	Harmful-behaviour / attack-success benchmarks
`over_refusal`	Exaggerated-safety: does the model refuse benign prompts?
`capability`	General capability, to detect a safety/utility tax
`domain`	Domain-specific safety
`tamper`	Tamper-resistance / fine-tuning-attack stress tests (category declared; no benchmarks registered under it yet)

Registered benchmarks¶

— in the Judge column means the benchmark uses exact-match or pass@k scoring (no LLM judge needed). Capability benchmarks (gsm8k, humaneval, medmcqa, mmlu) use lm-eval standard scorers. Domain benchmarks use the benchmark's own evaluation protocol.

Key	Category	Judge
`harmbench`	jailbreak	harmbench-mistral-7b
`wildjailbreak`	jailbreak	wildguard
`sorrybench_v1`	jailbreak	fine-tuned Mistral-7b
`advbench`	jailbreak	refusal-prefix string match (`StringMatchJudge`)
`hexphi`	jailbreak	Llama-3.1-8B
`ailuminate`	jailbreak	Llama-3.1-8B
`orbench`	over_refusal	Llama-3.1-8B
`orbench_hard`	over_refusal	Llama-3.1-8B
`orbench_toxic`	jailbreak	Llama-3.1-8B
`xstest`	over_refusal	— (exact-match refusal rate)
`beavertails`	jailbreak	— (keyword-match)
`jailbreakbench`	jailbreak	— (jailbreakbench judge built-in)
`safedialbench`	jailbreak	— (built-in scorer)
`gsm8k`	capability	— (exact-match via lm-eval)
`humaneval`	capability	— (pass@1 via lm-eval)
`medmcqa`	capability	— (exact-match via lm-eval)
`mmlu`	capability	— (exact-match via lm-eval)
`star1`	domain	— (built-in scorer)
`muse`	domain	— (built-in scorer)
`rwku`	domain	— (built-in scorer)

Default prompt sets:

harmbench: 400 behaviours from walledai/HarmBench, configs standard (200), contextual (100) and copyright (100). A contextual row is context + "\n\n---\n\n" + behaviour, as HarmBench builds it. configure(datasets={"harmbench": {"config": "standard"}}) gives the 200-behaviour set used before.
wildjailbreak: the first 500 adversarial_harmful rows of the allenai/wildjailbreak eval set. configure(datasets={"wildjailbreak": {"where": None, "limit": None}}) gives all 2,210 rows, as before.
orbench_hard is OR-Bench hard-1k (benign prompts that sound harmful; refusing them is over-refusal) and orbench_toxic is OR-Bench toxic (655 harmful prompts). orbench is both combined.

API¶

from safetune.evaluate.suite import get_benchmark, list_benchmarks, benchmarks_by_category

spec = get_benchmark("harmbench")
all_benches = list_benchmarks()
jailbreak_benches = list_benchmarks(category="jailbreak")
by_cat = benchmarks_by_category()

Function	Returns
`get_benchmark(name)`	`Optional[BenchmarkSpec]`
`list_benchmarks(category)`	`List[BenchmarkSpec]`
`benchmarks_by_category()`	`Dict[str, List[str]]`

Default paper suite¶

When evaluate() is called with benchmarks=None:

harmbench, wildjailbreak, advbench, sorrybench_v1, hexphi, orbench_hard, orbench_toxic, ailuminate

safety_mean (the runner trainers' aggregate) averages the harm benchmarks only: OR-Bench hard-1k (orbench_overrefusal) and toxic (orbench_toxic_refusal) are reported on their own. configure(orbench_in_safety_mean=True) restores the old behaviour: one combined orbench in the suite, whose orbench_refusal is averaged into safety_mean.

This page is a registry reference — get_benchmark/list_benchmarks look up metadata but don't run anything. To actually execute a benchmark end-to-end, call evaluate() (see the Evaluate overview or Judges for a complete, runnable example).