Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking
Key takeaway
It would be useful if we had the ability to modify a model’s beliefs.
Why it's on PDOOM
PDOOM selected this story for its alignment & control signals: Alignment, Reward Hacking.
AlignmentReward HackingFrom Alignment Forum
It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring[1], help us do better science on current models[2], and augment certain forms of alignment training[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward
By Jozdien