Foundation Models for Oversight
To oversee an AI model, we'd ideally like to ask questions such as:What are important situations where the model sandbags?Does the model have an objective it wouldn't admit to if asked directly?Does the model treat a user differently once it infers something about their identity, and along what axis?Is the model's chain of thought load-bearing, or is it a post-hoc rationalization of an answer that was already settled on?Is it reward hacking on this input, or
By jsteinhardt