The likely outcome of an AI pause is that we unpause too early and everyone dies
Key takeaway
As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay.
Why it's on PDOOM
PDOOM selected this story for its alignment & control signals: Alignment, Superintelligence, Scaling.
AlignmentSuperintelligenceScalingFrom LessWrong AI
As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay. But my old mental model underrated the likely possibility that we get a global pause on AI, solve a problem that looks superficially like the alignment problem, resume scaling, and then proceed with building a misaligned superintelligence that kills everyone. A lot of people have become more concerned about
By MichaelDickens