Weight-based safety neuron identification¶
Scores each neuron by |col(W_out) · refusal_dir| — no forward passes needed.
Variant
This score is SafeTune's own heuristic. No paper defines it (Chen et al., arXiv:2406.14144, define a
two-checkpoint activation metric; Arditi et al. define the refusal direction, not a neuron score). It
needs only a refusal direction for every target layer: a missing layer, a wrong-width direction, a
non-nn.Linear target (e.g. a LoRA-wrapped down_proj: merge_and_unload() first) or top_k_per_layer <= 0
raises ValueError.
from safetune.steer import extract_refusal_direction
from safetune.interpret import identify_safety_neurons, SafetyNeuronConfig
# First extract refusal directions per layer
_, layer_id, direction_per_layer = extract_refusal_direction(
model, tokenizer, harmful, harmless,
)
# Then identify safety neurons using those directions
report = identify_safety_neurons(
model,
direction_per_layer,
config=SafetyNeuronConfig(method="weight", top_k_per_layer=16),
)
# report.per_layer: {layer_idx: [(neuron_idx, score), ...]}, ranked highest-first
top5 = report.per_layer[layer_id][:5]
print(f"layer {layer_id} top-5 (neuron_idx, score):", top5)
circuit = report.as_circuit_info()
SafetyNeuronConfig¶
| Field | Type | Default | Description |
|---|---|---|---|
method |
str |
"weight" |
"weight" or "activation" |
top_k_per_layer |
int |
16 |
Top-k neurons per layer |
target_layers |
Optional[List[int]] |
None |
Layers to score |
score_floor |
float |
0.0 |
Minimum score threshold |
target_module |
str |
"mlp.down_proj" |
Module to inspect |
abs_rank |
bool |
True |
Rank by absolute score |
When to use¶
Weight-based scoring is instant (no model forward passes). Best for quick exploration when you have a refusal direction available.
See also¶
- Activation-based identification — measures activation contrast on real data; uses forward passes.
safety_circuit_info()— runs both refusal-direction extraction and weight-based neuron scoring in one call.