Neural nets are exceptionally vulnerable to goal self-hack
Key takeaway
Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value.
Why it's on PDOOM
PDOOM selected this story as relevant to alignment & control.
From Alignment Forum
Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.This rich definition is contained in a sophisticated LLM, abl
By Stuart_Armstrong