Desiderata for the Theory of Agents
Key takeaway
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata.
Why it's on PDOOM
PDOOM selected this story for its alignment & control signals: Alignment.
AlignmentFrom Alignment Forum
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata. A Theory of What? The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") tha
By Vanessa Kosoy