Training on probes: Research ideas
Key takeaway
RecapSequel to Previous Post.
Why it's on PDOOM
PDOOM selected this story for its capabilities signals: AGI.
AGIFrom Alignment Forum
RecapSequel to Previous Post. This post might not make sense without it.Training on probes might let our judgments on easy domains generalize to harder domains by leveraging an AI's model of the world. Taking the gradient of the probe teaches the model to fool the probe, but doing RL against the probe doesn't teach the model to fool it!Following the brain-like storyMy hardwired instincts for eating fresh fruit have recruited my learned knowledge about supermarkets, and now I want to go to the su
By Charlie Steiner