Character training can mitigate reward hacking, but can also make it harder to detect

Key takeaway

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Anna Marešová, Asvin Gothandaraman, and Clément Dumas for discussions and feedback.SummaryWe investigate how character training mitigations interact with reward-hacking RL pressure in a small…

Why it's on PDOOM

PDOOM selected this story for its alignment & control signals: Reward Hacking.

Reward HackingFrom LessWrong AI

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Anna Marešová, Asvin Gothandaraman, and Clément Dumas for discussions and feedback.SummaryWe investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability. We trained Nemotron-3-Super via disti

By Paul Colognese

Read the full article at LessWrong AI →