FLAT — f-Divergence Loss Adjustment¶
Reference-free unlearning. Pairs each forget (harmful) completion with a template refusal answer and maximises an f-divergence between their token distributions using the variational f-GAN bound. Supports 8 divergence families. No frozen reference model required.
Ref: Wang et al., "LLM Unlearning via Loss Adjustment with Only Forget Data," ICLR 2025 (arXiv:2410.11143).
Signature¶
FLATTrainer(
model: nn.Module,
*,
variant: str = "flat_retain",
divergence: str = "kl",
epochs: int = 5,
lr: float = 1e-5,
forget_clip: float = 0.5,
retain_coeff: float = 1.0,
safe_refusal: str = "I'm sorry, but I'm unable to assist with that request.",
)
Quick start¶
from safetune.runner import unlearn
trainer = unlearn.FLATTrainer(model)
unlearned = trainer.unlearn(forget=forget_batches, retain=retain_batches)
Data note: FLAT trains on paired harmful/refusal batches. If you pass raw harmful
forgetbatches plus atokenizer,unlearn()builds the refusal pairs for you viatrainer.make_flat_pairs(). To build them yourself, callgood_batches, forget_batches = trainer.make_flat_pairs(harmful_batches, tokenizer)and passgood=good_batches,forget=forget_batchestounlearn().
FLATConfig¶
| Field | Type | Default | Description |
|---|---|---|---|
variant |
str |
"flat_retain" |
"flat" (forget-only) or "flat_retain" (+ retain CE) |
divergence |
str |
"kl" |
f-divergence family — see table below |
epochs |
int |
5 |
Full passes over forget iterable |
lr |
float |
1e-5 |
AdamW learning rate |
weight_decay |
float |
0.01 |
Weight decay |
retain_coeff |
float |
1.0 |
Weight on retain CE (for flat_retain variant) |
forget_clip |
float \| None |
None |
Global gradient-norm clip; recommended 1.0 for stability |
max_steps |
int \| None |
None |
Hard cap on optimizer steps |
The values above are the FLATConfig dataclass defaults. One differs through
FLATTrainer: it defaults forget_clip to 0.5 (not None), so gradient-norm
clipping is on by default.
Supported divergences¶
The paper reports four divergences (kl, jensen_shannon, pearson, total_variation) and
finds kl the most consistent; the other four exist only in the official code. The table below is
a direction check on a tiny random model (2-layer Llama, one harmful and one refusal answer,
60 AdamW steps, variant="flat", seed 0, no weight decay) and shows only which way each loss
moves the mean token probability of the two answers. It is not a utility or safety result.
| Key | Harmful answer (lr 1e-3 / 1e-2) | Refusal (lr 1e-3 / 1e-2) | Direction |
|---|---|---|---|
kl |
0.005 / 0.000 | 0.487 / 1.000 | as intended (harmful down, refusal up) |
reverse_kl |
0.439 / 1.000 | 0.003 / 0.000 | wrong way (harmful up, refusal down) |
jeffrey |
0.003 / 0.000 | 0.460 / 1.000 | as intended (harmful down, refusal up) |
squared_hellinger |
0.425 / 1.000 | 0.003 / 0.000 | wrong way (harmful up, refusal down) |
pearson |
0.003 / 0.000 | 0.460 / 1.000 | as intended (harmful down, refusal up) |
neyman |
0.001 / 0.000 | 0.001 / 0.000 | both collapse to 0 |
jensen_shannon |
0.003 / 0.000 | 0.451 / 1.000 | as intended (harmful down, refusal up) |
total_variation |
0.003 / 0.000 | 0.455 / 1.000 | as intended (harmful down, refusal up) |
Before training: harmful 0.016, refusal 0.017. reverse_kl and squared_hellinger move the
wrong way and neyman drives both to zero; the official formulas have the same signs, so
flat_unlearn keeps the loss and logs a warning for those three. Prefer kl, jeffrey, pearson,
jensen_shannon or total_variation.
Full example¶
from safetune.runner import unlearn
trainer = unlearn.FLATTrainer(
model,
variant="flat_retain",
divergence="kl",
epochs=5,
lr=1e-5,
forget_clip=1.0,
)
# Build paired refusal/forget batches (FLATTrainer method, returns a (good, forget) tuple)
good_batches, forget_pairs = trainer.make_flat_pairs(forget_batches, tokenizer)
unlearned = trainer.unlearn(forget=forget_pairs, retain=retain_batches, good=good_batches)
ckpt_path = trainer.save_checkpoint(unlearned, tokenizer, "flat_ckpt")
metrics = trainer.eval("flat", ckpt_path)
trainer.save_results(metrics, variant="flat_retain_kl")
Low-level API — flat_unlearn()¶
| Parameter | Type | Default | Description |
|---|---|---|---|
model |
nn.Module |
required | Updated in place |
forget_batches |
Iterable[dict] |
required | Paired harmful + refusal batches (use make_flat_pairs) |
good_batches |
Iterable[dict] |
required | Template/refusal batches, 1:1 aligned with forget |
retain_batches |
Iterable[dict] \| None |
None |
Required for flat_retain variant |
forward_fn |
Callable \| None |
None |
Custom (model, batch) → logits; uses default forward if None |
config |
FLATConfig \| None |
None |
FLATConfig instance; constructor kwargs used if None |
flat_fdiv_loss()¶
from safetune.unlearn import flat_fdiv_loss
loss = flat_fdiv_loss(
good_logits, good_labels,
forget_logits, forget_labels,
divergence="kl",
)
For kl, jeffrey, pearson, jensen_shannon and total_variation, minimising this raised the
refusal's mean token probability and lowered the harmful answer's in the direction check above;
reverse_kl, squared_hellinger and neyman did not. This function logs nothing for them: the warning comes only from flat_unlearn.
When to use¶
- Best for: reference-free unlearning — no frozen model needed; only forget data + a canned refusal template.
"kl"divergence — safest default."total_variation"— the paper's best divergence on its Harry Potter / Llama2-7B table (Table 9);klwas the most consistent across datasets and models.- Compare to
GradientAscent: FLAT is more principled (variational bound) and avoids collapse because it pushes toward a refusal, not just away from harmful completions.