Training Language Models for Introspective Oversight
Key takeaway
Language models increasingly exhibit behaviors that are difficult to detect in transcripts, such as reward hacking, deception, and sycophancy.
Why it's on PDOOM
PDOOM selected this story for its alignment & control signals: Reward Hacking, Oversight.
Reward HackingOversightFrom ArXiv cs.AI
Language models increasingly exhibit behaviors that are difficult to detect in transcripts, such as reward hacking, deception, and sycophancy. Asking the assistant follow-up questions about its actions is unreliable, since models are generally not trained to report their own misbehavior. As a result, approaches to oversight typically involve querying external models. These face limitations: external transcript monitoring operates only on visible mo
By Alex Lin Wang, Linus Yifeng Tang, Dami Choi, Neil Chowdhury