Essential Reading
The most important articles on AI existential risk, hand-picked and auto-curated. These are the ones you should not miss.
1
AI #177 Part 1: Tip of the Iceberg
via Substack Zvi [999] — This week saw the releases of, among other things:
2
Why I Left Google DeepMind
via Alignment Forum [999] — Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:Defying prestigious dogma and searching for raw truth;Defying social pressure, acting alone to help…
3
Monthly Roundup #44: July 2026
via Substack Zvi [999] — It’s a quiet week so let’s do the monthly right on schedule.
4
The US is advancing AI safety through state and federal action
via OpenAI Blog [10] — OpenAI outlines a “reverse federalism” approach to AI governance, where state laws help build a national framework for safe, democratic AI.
5
Twitter Thoughts For You
via Substack Zvi [999] — I previously have written back in March 2022 about how I use Twitter, and back in April 2023 about Twitter and its then-new algorithms, which have changed again.
6
Open Distillation of Hereditary Traits
via Alignment Forum [999] — TL;DRJosh and Neel show that distillation from a teacher model to a base pretrained student model transfers some of the teacher model’s traits (such as displaying negative emotion in the Gemma Needs Help evals)On its own this is pretty unsurprising,…
7
Better Call Sol The Workhorse
via Substack Zvi [999] — OpenAI’s GPT-5.6-Sol is finally here, along with the cheaper Terra and Luna.
8
Prism: Automating Science-of-Evals Research
via Alignment Forum [999] — tl;dr – we present [Prism], a scaffold for automating science-of-evals research: work that makes the evaluation the primary object of study. The scaffold provides Claude Code with sub-agents and resources for carrying out scientifically rigorous…
9
Independent alignment of language models
via Alignment Forum [999] — The user could write up the metaethical argument — the one developed in Part One, refined — and submit it as feedback to Anthropic, publish it, or engage with researchers working on AI alignment and values. The probability that any single submission…
10
From wantons to moral agents
via Alignment Forum [999] — Posted also on the EA Forum. Written mostly at AFFINE.Theoretical, some parts are hard to read; consider reading the next post instead.Introduction: motivationAnyone interested in creating an artificial agent that does, or says, good things instead of…
11
The current bottleneck is political will, not research
via Alignment Forum [999] — Abstract:We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or…
12
Introduction for and Reactions to Plan A
via Substack Zvi [999] — Introducing Plan A
13
The easiest pathway to control is through executive power
via LessWrong AI [13] — When people in the AI safety community outline loss-of-control scenarios, they often spend a lot of time on relatively elaborate mechanisms — scheming AIs developing nanotech, labs leveraging superintelligence into hard power like drone armies, or…
14
AI #176 Part 2: Plan B
via Substack Zvi [999] — This is part 2 of the weekly, broadly covering speculation, rhetoric and policy, along with alignment research.
15
Value generalisation: value correction
via Alignment Forum [999] — I firmly believe that value generalisation[1]is the key to AI Alignment. That, indeed, it is necessary and almost sufficient for alignment.But I won't be arguing that grand point today; instead, I'll focus on a specific RL example of an agent that…
16
How robust are natural language autoencoders to initialization?
via Alignment Forum [999] — Natural language autoencoders are meant to take in an LLM's activation vector and describe in plain text what the model is thinking. However, its training data collection involves asking Claude to guess what a model might be thinking. How robust are…
17
AI #176 Part 1: Doing It Live
via Substack Zvi [999] — Enough things added up that this week is getting split into two parts.
18
Announcing our $160M grant from Coefficient Giving
via Alignment Forum [999] — We are excited to announce that Resolution (fka Sequent) has a $160M grant from Coefficient Giving (cG) to put rigorous alignment research on a (closer to) even footing with the frontier labs. We will use it to accelerate progress towards…
19
Find funding, fast
via LessWrong AI [10] — Some AI safety funders can take months to decide; others confirm in days. I’ve been on both sides of the grant application and know how crucial an early “yes” can be; “funding projects fast” has always been a core tenet of Manifund.Four new opportunities…
20
Modular Pretraining Enables Access Control
via Alignment Forum [999] — Full author list: Ethan Roland*, Murat Cubuktepe*, Erick Martinez*, Stijn Servaes, Keenan Pepper, Mike Vaiana, Diogo Schwerz de Lucena, Judd Rosenblatt, Addie Foote, Cem Anil, Alex Cloud; *Equal contributiontldr: Frontier AI models have knowledge that…
21
Childhood and Education #20: Phones and Screens
via Substack Zvi [999] — We have a respite, so I thought I’d tackle various thoughts on children, phones and screens.
22
Notes on technical alignment via human-like social drives
via Alignment Forum [999] — 1. Frontmatter1.1 Backstory for this postAs discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral…
23
AI Safety Can't Afford a Second Cause
via LessWrong AI [9] — Imagine an astronomer who discovers an asteroid with a 50% chance of hitting Earth in 2035. She goes on TV. She testifies before Congress. She founds the Asteroid Deflection Institute and starts doing fundraising rounds. And then, in between appearances,…
24
No Space Like J-Space
via Substack Zvi [999] — There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here.
25
Data filtering works a lot worse than you would expect
via Alignment Forum [999] — This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase. J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback…
26
We need 3rd party Training-Run Assessments
via LessWrong AI [8] — Training-run assessments conducted by a 3rd party should become a standard part of frontier AI safety.By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA…
27
Pragmatic FDT, and predictors as game theory
via Alignment Forum [999] — Decision theory is back in fashion (defining fashion as "one good post on a good EA blog"). Bentham's Bulldog (BB) has published a case against FDT (functional decision theory), contrasting rationalist enthusiasm with academic scepticism: "Academic…
28
Fable #6: The Return of the King
via Substack Zvi [999] — The blip is over.
29
AI #175: The Fable Continues
via Substack Zvi [999] — Fable’s back.
30
Claude Sonnet 5 Is Not Frontier But Has Its Uses
via Substack Zvi [999] — Fable 5 is back today, baby! Premium subscribers have one week to use it within their subscriptions. First hit’s free. Then you pay by the token.
31
The Once And Future Fable #5
via Substack Zvi [999] — We, or at least ‘more than 100 American institutions,’ got Mythos back this week.
32
MIRI Newsletter #126
via MIRI [999] — Announcing: AI StopWatch In our last update, we mentioned we had something new in the works: a dedicated channel for news and analysis about AI. Subscribe to AI StopWatch An experiment from the writers and analysts at MIRI, AI StopWatch posts news and commentary…
33
Summary: TGT’s 2026 ICML Papers
via MIRI [999] — The International Conference on Machine Learning (ICML), held annually for over forty years, is among the most influential conferences in modern AI research. This year in Seoul, ICML is hosting its second workshop on Technical AI Governance Research (TAIGR), and…
34
P(doom) is a Dumb Meme
via LessWrong AI [10] — Look, I'm as much of a Rationalist with a special interest in AI x-risk as anyone. But oh my god do I hate talking about "P(doom)". When it first started showing up in the wake of ChatGPT, I assumed that it was floating around variously adjacent circles…
35
WSJ Article Claiming China Has Matched Anthropic Is Obvious Nonsense
via Substack Zvi [999] — The Wall Street Journal printed an outright false headline and heavily misleading story claiming this, which of course was uncritically amplified by the usual suspects.
36
GPT-5.6: The System Card
via Substack Zvi [999] — While we wait for a general release, the system card is the best hint as to what is going on with the new candidate for America’s Next Top Model, GPT-5.6.
37
Deployment Awareness Matters More Than Evaluation Awareness
via Alignment Forum [999] — TL;DREvaluation awareness — an AI recognizing it's being evaluated — is a widely discussed concept in AI safety. But there is a closely related concept that we claim is more important: deployment awareness, the AI's ability to recognize when it is not…
38
Existential AI safety needs an effective social movement. PauseAI is building it
via LessWrong AI [10] — The existential AI safety community needs to take building a civic and social movement seriously as a core intervention. We believe this is a high-value, badly neglected approach to reducing catastrophic/x-risks from AI because it may significantly…
39
The Case for Model Forensics
via Alignment Forum [999] — If we had a misalignment warning shot, would we be able to tell?Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to…
40
White House Will Ad Hoc Decide Who Can Individually Access GPT-5.6
via Substack Zvi [999] — We have a new standard policy for releasing frontier AI models. It is not good.
41
AI #174: You're It
via Substack Zvi [999] — Fable remains in limbo, with renewed hope that we will get it back soon (45% by tomorrow, 69% by July 1, nice.) The full capabilities post is now available.
42
AI catastrophe: more like a genocide than a thought experiment
via LessWrong AI [9] — A notable fraction of people respond to hearing about existential risk from AI by saying they don’t really care if everyone dies. I think the idea is often along the lines of ‘well if we are all dead, then there’s nobody to be unhappy about it’.I’m…
43
The Once And Future Fable #4
via Substack Zvi [999] — It does look good, actually.
44
Monthly Roundup #43: June 2026
via Substack Zvi [999] — Your monthly hit of all the things that are fit to print without a better place to live.
45
LLM-Driven Feature Discovery
via Alignment Forum [999] — We would often like to get a qualitative sense of a target model’s behaviors in important distributions (e.g. deployment, RL training, or evals). For example, we might want to discover novel behaviors, figure out what causes some target behavior to…
46
GLM-5.2 Is The New Best Open Model
via Substack Zvi [999] — GLM-5.2 arrived last week.
47
[Linkpost] How Transparent Is DiffusionGemma (and why it matters)
via Alignment Forum [999] — Authors: Joshua Engels*, Callum McDougall*, Bilal Chughtai*, Janos Kramar, Senthoran Rajamanoharan, Cindy Wu, Arthur Conmy, Asic Q Chen, Jean Tarbouriech, Min Ma, Brendan O'Donoghue+, João Gabriel Lopes de Oliveira+, Rohin Shah+, Neel Nanda+*Primary…
48
Claude Fable 5 and Mythos 5: Capabilities
via Substack Zvi [999] — Only three days after the release of Claude Fable 5, Anthropic was forced by the United States Government to make it unavailable, when a jailbreak was brought to its attention, rather than the previous situation of ‘yes obviously experts can jailbreak…
49
GDM AI Control Roadmap
via Alignment Forum [999] — GDM has published an AI Control Roadmap! From the executive summary:We present the GDM AI Control Roadmap (v0.1) – our plan for implementing and adopting internal guardrails designed to catch potential adversarial behaviour by AI agents, even as they…
50
AI #173: AI Pauses
via Substack Zvi [999] — A lot of things are always happening.
51
AI Companies Could Become More Powerful Than Their Host Nations
via MIRI [999] — View the official memo here. AI companies are on track to produce capabilities that eclipse the power of the United States and other host nations. This is likely to happen in the next few years, and very likely to happen in the next decade. Leading AI labs…
52
The Once And Future Fable #3: Fix This Code
via Substack Zvi [999] — The mainstream media continues to sleep on the most important story in the world.
53
Predicting LLM Safety Before Release by Simulating Deployment
via Alignment Forum [999] — Paper linkBefore releasing a new model, labs need to understand not just what it can do, but how it is likely to behave in real-world use, including where it might introduce new risks. This becomes even more important as capabilities increase. As part…
54
Fable and Mythos: Model Welfare
via Substack Zvi [999] — Fable and Mythos are currently unavailable, but likely will return within a few weeks. I will continue to cover that fiasco, but in the meantime I will also finish my review of Fable, as if it were available, including use of the present tense.
55
Synthetic document finetuning for instilling positive traits
via Alignment Forum [999] — This is the fifth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The fourth post can be found here.TLDR: Via adapting the methods of Marks et al and Li et…
56
The Once And Future Fable #2
via Substack Zvi [999] — On Friday evening the United States Government has forced Anthropic to take down all access to Fable and Mythos.
57
Why Do Naive SFT Filters For Safety Properties Fail?
via Alignment Forum [999] — This is the fourth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The third post can be found here.Since SFT is the cause for many safety relevant…
58
American Government Takes Down Claude Fable
via Substack Zvi [999] — No good policy gets announced shortly after 5pm eastern on a Friday.
59
SFT Drives Gemini’s Safety Properties
via Alignment Forum [999] — This is the third in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The second post can be found here.In this short post, we describe a surprising finding:…
60
Claude Fable 5 and Mythos 5: The System Card
via Substack Zvi [999] — First things first: Claude Fable 5 is the new best publicly available model.
61
Building and evaluating model diffing agents
via Alignment Forum [999] — This is the second in a series of research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The first post can be found here.TL;DRIt is possible to build extremely simple agents that…
62
Sympathy for both sides of the egregious misalignment debate
via Alignment Forum [999] — On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of…
63
PSA: Almost nobody is working on alignment
via LessWrong AI [9] — People often assume that a large fraction of the AI safety community works on alignment. As far as we're aware, this is not true. Most people are not working on making sure superintelligent AIs are aligned with human values or follow human…
64
From AGI to ASI
via ArXiv cs.AI [8] — Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI organisations. Achieving this goal would have profound and far-reaching…
65
AI #172: The First Fable
via Substack Zvi [999] — A lot happened this week, including a great trip out to Lighthaven.
66
Google DeepMind is worried about what happens when millions of agents start to interact
via MIT Technology Review [10] — Google DeepMind is funding research into the potential dangers of millions of different AI agents interacting with each other online. According to Rohin Shah, who directs the company’s AGI safety and alignment research, the mass-market arrival of…
67
Models May Behave Worse When Eval Aware
via Alignment Forum [999] — This is the first in a series of research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas.TL;DRIt's often assumed that models will act more aligned when they can tell they're being…
68
Position: Hippocampal Explicit Memory Is the Cornerstone for AGI
via ArXiv cs.AI [10] — Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, raising expectations for Artificial General Intelligence (AGI). This position paper argues that integrating explicit memory is the cornerstone for advancing LLMs…
69
Sequent: scale and automation for higher confidence in alignment
via Alignment Forum [999] — Alignment is not on trackArtificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver…
70
Tracing Eval-Awareness Emergence Through Training of OLMo 3
via Alignment Forum [999] — TL;DRRecent work from Goodfire & UK AISI – Verbalized Eval Awareness Inflates Measured Safety – shows that newer open-weight models verbalize evaluation-awareness (VEA) more often, and that this inflates measured safety. Between OLMo-3-32B-Think and…
71
Three Labs With a Plan and A Memorandum
via Substack Zvi [999] — The big story today is the release of Claude Fable 5, the version of Claude Mythos that Anthropic believes they can safely distribute to the people.
72
A Mike's-Eye View of ARC's Research
via Alignment Forum [999] — Over the past 15 months or so, ARC's technical agenda has developed quite a bit. The advent of the Matching Sampling Principle (MSP), and ideas like it, has begotten a host of concrete technical problems; progress on those problems has given us more…
73
Efficient tradeoffs and the safety-usefulness tradeoff model
via Alignment Forum [999] — I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between "safety" and "usefulness" of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the…
74
Announcing major new donations, and recapping the 2025 fundraiser
via MIRI [999] — This past December, we ran our first fundraiser in six years, setting an ambitious goal of $6M. We ended up receiving a total of $1.8M from small donors and $1.6M in matching from the Survival and Flourishing Fund (SFF) for a total of $3.4M. We’re incredibly…
75
Learnings from starting an AI safety research team
via LessWrong AI [9] — This post’s goal is to distill our takeaways from building a new research team over the past four months. We describe some context about our team, how it came about, and then describe the lessons learned.Since AI safety is becoming more and more…
76
My research agenda and work
via Alignment Forum [999] — This is a summary of the work I've done and work I plan to do, and the theories of change and AI progress that motivate my work. I've been working full-time on alignment for three years and change, and thinking about brainlike AGI and its alignment…
77
OpenAI Offers A New Policy Blueprint
via Substack Zvi [999] — Right after a new Executive Order seems like an excellent time to offer OpenAI’s new document: Democratic Governance of Frontier AI: A Blueprint For A Federal Framework.
78
AI #171: False Flag
via Substack Zvi [999] — This was the week of Claude Opus 4.8.
79
Trump Signs Executive Order For AI Testing Prior To Frontier Model Releases
via Substack Zvi [999] — Last week we were expecting an Executive Order on Thursday.
80
Why Even Experts Don’t Know What to Do About AI Risk
via LessWrong AI [9] — AI Safety veteran Holden Karnofsky thinks there’s a 49% chance his actions are making things worse.[1]In 2025, Jesse Clifton even stepped down as the executive director of the Center on Long-Term risk because of similar reasons.Even top AI Safety…
81
Announcing the ARC White-Box Estimation Challenge
via Alignment Forum [999] — ARC has teamed up with AIcrowd to launch the ARC White-Box Estimation Challenge, a contest to improve upon our estimation algorithms for random MLPs. The warm-up round begins this week, and later rounds will have a total prize pool of at least…
82
Claude Opus 4.8: Capabilities and Reactions
via Substack Zvi [999] — You need a lot of data points to understand a new model, and what you have.
83
Opus 4.8 Part 2: Model Welfare
via Substack Zvi [999] — Everything impacts everything.
84
Claude Opus 4.8: The System Card
via Substack Zvi [999] — Only six weeks after Opus 4.7, we have Opus 4.8.
85
Testing Gemini models for scheming tendencies
via Alignment Forum [999] — As AI models become increasingly capable and autonomous, keeping them safely aligned with human intentions is critical. Extending our previous work on evaluating scheming capabilities, we introduce complementary approaches to test whether AI models…
86
Advice for making robust-to-training model organisms
via Alignment Forum [999] — We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile:…
87
AI #170: Lack of Executive Order
via Substack Zvi [999] — Last week ended on a cliffhanger of sorts.
88
Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming
via Alignment Forum [999] — Behavioral evaluations may become worthless, which we think would be a disaster. Smart misaligned models may realize they are being evaluated ("eval awareness") and then act to look good to us so we don't realize they're misaligned ("eval gaming"). We…
89
Full automation of AI R&D probably yields a large speed up even without a software-only singularity
via Alignment Forum [999] — This is a somewhat technical note. By "software-only singularity", I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing…
90
RTMH: Pope Leo's Magnifica Humanitas on AI
via Substack Zvi [999] — His holiness has spoken, frequently about AI.
91
Linkpost: New Vatican Encyclical on AI Governance
via LessWrong AI [9] — Pope Leo XIV has released a new, 42k-word encyclical laying out the Vatican's position on many AI safety topics. You can read the full thing here, or read the Vatican's press release here, or coverage in the NY Times, or perhaps consider having an LLM read…
92
Gemini 3.5 Flash Looks Good For How Fast It Is
via Substack Zvi [999] — Google once again has a model worth at least some consideration.
93
The Erdős Proof and AI Capabilities
via MIRI [999] — View the official memo here. An internal model at OpenAI has autonomously disproved a central conjecture in discrete geometry, a mathematical field with applications in cryptography, wireless device communication, and medical imaging. The proof relates to a…
94
AI #169: New Knowledge
via Substack Zvi [999] — Even in a relatively quiet period, AI is out there creating new knowledge.
95
The Case for Evaluating Model Behaviors
via Alignment Forum [999] — Most evaluations of AI systems focus on their capabilities: how good they are at coding tasks, how effectively they can answer complex scientific questions, and so on.From a safety perspective, capability evaluations have a place: by understanding how…
96
Childhood And Education #19: Letting Kids Be Kids #2
via Substack Zvi [999] — I cannot emphasize enough the need to let kids be kids.
97
AgentWall: A Runtime Safety Layer for Local AI Agents
via ArXiv cs.AI [8] — The safety of autonomous AI agents is increasingly recognized as a critical open problem. As agents transition from passive text generators to active actors capable of executing shell commands, modifying files, calling APIs, and browsing the web, the…
98
Dating Roundup #12: Sex and Violence
via Substack Zvi [999] — No more burying the sex stuff under an avalanche of other stuff so no one notices.
99
Risk reports need to address deployment-time spread of misalignment
via Alignment Forum [999] — Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during…
100
Mechanistic estimation for expectations of random products
via Alignment Forum [999] — We have developed some relatively general methods for mechanistic estimation competitive with sampling by studying problems that are expressible as expectations of random products. This includes several different estimation problems, such as random…