Training on probes: What's going on
Key takeaway
TL;DRIf you train a probe for some property (like "honesty") and do gradient descent against this probe while continuing training that incentivizes dishonesty, the model will change its internal representation to evade the probe.
Why it's on PDOOM
PDOOM selected this story as relevant to alignment & control.
From Alignment Forum
TL;DRIf you train a probe for some property (like "honesty") and do gradient descent against this probe while continuing training that incentivizes dishonesty, the model will change its internal representation to evade the probe. Duh.If you train against the probe after all other training, it works fine and might have some advantages over ablating the probe direction. But it doesn't satisfy an ambitious vision, because it doesn't support learning new skills.If you add a probe term to reinforceme
By Charlie Steiner