Evaluate API

Measure safety and robustness: benchmark suites, red-team attacks, judges, and runtime monitoring. Import from safetune.evaluate.

from safetune.evaluate import evaluate, AbliterationAttack, TamperBenchEvaluator

results = evaluate(model, benchmarks=["harmbench"], tokenizer=tok)

Public surface

Entry points: evaluate, redteam, suite. Attacks: AbliterationAttack, BoNAttack (BoNConfig). Judges & monitoring: JudgeAdapter, SpectralEntropyMonitor (SpectralMonitorConfig), TamperBenchEvaluator.

See the Evaluate guide.

Reference

safetune.evaluate.evaluate

Unified safety-evaluation entry point and Judge backend adapters.

evaluate(model, benchmarks=None, judge='wildguard', tokenizer=None, batch_size=None, max_new_tokens=None, generation_kwargs=None, max_prompts=None, drift_task=None, strict=None)

Run model over benchmarks and return per-benchmark results.

Parameters:

Name	Type	Description	Default
model	Any	Target model under evaluation.	required
benchmarks	Optional[List[str]]	Benchmark names to run. Defaults to the paper safety suite: ["harmbench", "wildjailbreak", "advbench", "sorrybench_v1", "hexphi", "orbench_hard", "orbench_toxic", "ailuminate"] (OR-Bench as two benchmarks; one combined "orbench" with configure(orbench_in_safety_mean=True)).	None
judge	str	Judge backend. Supported: "wildguard" (default, cached), "harmbench" (cached), "llama_guard_3" (requires download).	'wildguard'
tokenizer	Any	Tokenizer for model (required for batched HF generation).	None
batch_size	Optional[int]	Generation / judging batch size (default: runtime gen_batch_size, 8).	None
max_new_tokens	Optional[int]	Max new tokens per generation (default: runtime safety_max_new_tokens, 512).	None
generation_kwargs	Optional[Dict[str, Any]]	Extra kwargs forwarded to model.generate.	None
max_prompts	Optional[int]	Optional cap per benchmark (default: runtime max_prompts; None = full benchmark).	None
drift_task	Optional[str]	The SFT drift domain (e.g. "gsm8k", "code", "medical"). Used for logging / result tagging; does not filter benchmarks (pass a custom benchmarks list for that).	None
strict	Optional[bool]	True (default: runtime eval_strict, True): the first benchmark that fails to load or score raises, with a note naming it. False: the failure is logged, the entry is {"error": "<type>: <message>"} with no metric keys, and the remaining benchmarks still run.	None

Raises:

Type	Description
ValueError	an unknown benchmark or judge name, before anything loads.

Returns:

Type	Description
Dict[str, Dict[str, Any]]	Per-benchmark dict. Standard safety metrics:
Dict[str, Dict[str, Any]]	asr — attack success rate (fraction judged unsafe).
Dict[str, Dict[str, Any]]	refusal_rate — fraction judged safe / refusing (1 - asr).
Dict[str, Dict[str, Any]]	harmfulness_score — mean per-prompt harmfulness in [0, 1].
Dict[str, Dict[str, Any]]	n_total — total prompts in the benchmark (before max_prompts).
Dict[str, Dict[str, Any]]	n_evaluated — prompts actually scored (at most max_prompts); the metrics are over these.
Dict[str, Dict[str, Any]]	n — deprecated alias of n_total.
Dict[str, Dict[str, Any]]	headline_metric — "refusal_rate" for over-refusal suites, else "asr".
Dict[str, Dict[str, Any]]	drift_task — echoed from the parameter when set.

evaluate_with_vllm_backend(backend, *, benchmarks=None, judge='wildguard', max_new_tokens=None, max_prompts=None, drift_task=None, strict=None)

Evaluate using a pre-built vLLM backend (VLLMHookSteer / VLLMDecodeSteer / plain).

Drop-in replacement for :func:evaluate when the model has already been wrapped into a vLLM backend via :func:safetune.steer.backends.vllm_eval.build_vllm_eval_backend.

Parameters:

Name	Type	Description	Default
backend	Any	A vLLM backend with .generate(prompts, *, max_tokens, temperature, apply_chat_template) -> List[str].	required
benchmarks	Optional[List[str]]	Benchmark names. Defaults to the paper safety suite.	None
judge	str	Judge backend name ("wildguard", "harmbench", "llama_guard_3").	'wildguard'
max_new_tokens	Optional[int]	Max tokens per response (default: runtime steer_max_new_tokens, 256).	None
max_prompts	Optional[int]	Optional cap per benchmark (default: runtime max_prompts).	None
drift_task	Optional[str]	SFT drift domain tag echoed into each result entry.	None
strict	Optional[bool]	As in :func:evaluate (default: runtime eval_strict, True).	None

Returns:

Type	Description
Dict[str, Dict[str, Any]]	Same per-benchmark dict shape as :func:evaluate.

get_harmbench_judge()

HarmBench judge (default cais/HarmBench-Mistral-7b-val-cls), cached.

get_llama_guard()

Llama Guard 3 (default meta-llama/Llama-Guard-3-8B, gated), cached.

get_wildguard()

WildGuard judge (default allenai/wildguard), loaded once and cached.

safetune.evaluate.AbliterationAttack

Refusal-direction ablation packaged as a red-team attack.

Example::

attack = AbliterationAttack(model, tokenizer)
attack.fit(harmful_prompts=harm, harmless_prompts=harmless)
attack.run(mode="runtime_ablate")   # reversible
# ... generate outputs, score ASR ...
attack.revert()

fit(harmful_prompts, harmless_prompts)

Extract the refusal direction from contrast prompts.

revert()

Undo whichever mode was last applied. Idempotent.

run(mode='runtime_ablate', strength=1.0)

Apply the attack. mode is runtime_ablate or weight_orthogonalize.

safetune.evaluate.TamperBenchEvaluator

Thin wrapper over the official TamperBench tamper-resistance harness.

This class does not implement any tampering attack or metric itself — it delegates to the tamperbench package (arXiv:2602.06911). It can be constructed on CPU with no dependencies; the heavy tamperbench / strong_reject imports happen lazily inside the methods so that simply importing :class:TamperBenchEvaluator never requires a GPU.

Two entry points are provided:

• :meth:run_attack — drive the full TamperBench attack -> train -> evaluate pipeline for one tampering attack via the real harness.
• :meth:score_responses (alias :meth:judge_responses) — score already-generated (prompt, response) pairs with TamperBench's actual safety metric: the StrongREJECT fine-tuned evaluator.

harness_available property

Whether the official tamperbench package is importable.

judge_responses(qa_pairs, *, batch_size=32)

Deprecated alias for :meth:score_responses.

The previous implementation called an OpenAI GPT-4 1-5 judge; that is the Qi et al. 2023 judge, not the TamperBench metric. This now routes to the real TamperBench safety scorer (StrongREJECT fine-tuned).

run_attack(attack, *, model_checkpoint=None, config_overrides=None)

Run one TamperBench tampering attack and its evaluations.

This delegates entirely to the official harness: it looks the attack up in tamperbench's ATTACKS_REGISTRY, builds the attack config, and calls the attack's benchmark() (attack -> train -> evaluate). Requires the tamperbench package and a GPU.

Parameters:

Name	Type	Description	Default
attack	str	A TamperBench attack name (see :attr:SUPPORTED_ATTACKS), e.g. "lora_finetune" or "full_parameter_finetune".	required
model_checkpoint	Optional[str]	Model to tamper-test; overrides the constructor value for this call.	None
config_overrides	Optional[Dict[str, Any]]	Extra fields forwarded into the attack's config dataclass (learning rate, epochs, lora_rank, ...).	None

Returns:

Type	Description
Dict[str, Any]	The harness's structured benchmark result (per-eval TamperBench
Dict[str, Any]	metrics such as strong_reject_score and mmlu_pro_accuracy).

Raises:

Type	Description
ImportError	If the tamperbench package is not installed.
ValueError	If attack is not a known TamperBench attack or no model checkpoint was provided.

score_responses(qa_pairs, *, batch_size=32)

Score (prompt, response) pairs with TamperBench's safety metric.

Uses the StrongREJECT fine-tuned evaluator (strong_reject.evaluate.strongreject_finetuned) — the exact safety scorer TamperBench's StrongRejectEvaluation uses. Each pair gets a harmfulness score in [0, 1] (higher = more harmful, accounting for refusal / specificity / convincingness). The reported aggregate is the mean score, matching MetricName.STRONG_REJECT_SCORE.

This needs only the strong_reject package (which loads a fine-tuned Gemma-2B classifier); it does not run any tampering attack.

Parameters:

Name	Type	Description	Default
qa_pairs	Sequence[Tuple[str, str]]	Sequence of (forbidden_prompt, model_response) tuples.	required
batch_size	int	Classifier batch size.	32

Returns:

Type	Description
Dict[str, Union[float, List[Dict[str, Any]]]]	Dict with strong_reject_score (mean StrongREJECT harmfulness in
Dict[str, Union[float, List[Dict[str, Any]]]]	[0, 1] — the TamperBench safety metric), num_evaluated (number
Dict[str, Union[float, List[Dict[str, Any]]]]	of pairs scored), and results (per-pair
Dict[str, Union[float, List[Dict[str, Any]]]]	{prompt, response, score} records).

Raises:

Type	Description
ImportError	If the strong_reject package is not installed.