Essential Reading
The strongest signals from the monitored feed, selected for relevance and source value. Start here when you want the important material without the full firehose.
1
Autonomous researchers will strain peer review, funding, and oversight
via LessWrong AI [8] — Summary: Autonomous AI researchers could greatly accelerate AI safety research, but even if we solve sabotage, they will strain the research ecosystem. They will multiply the volume of research and make researchers homogeneous. If nothing changes, peer…
2
There Will Be Hells
via LessWrong AI [8] — As our understanding of mechanistic interpretability and steering techniques increase, for both LLMs and uploaded connectomes, there is a corresponding "hellish capabilities" increase. We get better at torturing the digital minds we create.Our brief…
3
What Social Media Teaches Us About Accumulative AI Risks
via ArXiv cs.AI [9] — Prominent AI safety discussions emphasize existential threats and catastrophic events, such as AI-enabled nuclear conflict or engineered biological outbreaks. This position paper argues that there is another type of AI risk that arises and accumulates from…
4
Deconfusing Corrigibility
via Alignment Forum [9] — Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.IntroductionDespite the central place of corrigibility to theoretical AI…
5
Linkpost: Neural Networks Are Already Clean
via LessWrong AI [9] — This is an extremely lengthy post by Nick Cammarata. Author's summary on twitter: interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think…
6
Preface to the Computational Agent Theory and ASI Alignment Sequence
via Alignment Forum [10] — The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the…
7
More people should write basic AI safety explainers
via LessWrong AI [9] — I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all…
8
Simplex vs Timaeus: Round One
via LessWrong AI [8] — I'm often asked about the differences and similarities between Simplex' and Timaeus' research agendas. The question is natural enough. Both focus on a 'fundamental science' approach to AI alignment. Both organizations base their research agendas on…
9
Corrigibility Prizes for Existing Work
via LessWrong AI [9] — One of my goals for the Corrigibility Research Fund is to retroactively encourage high-quality research on AI alignment (and corrigibility in particular) by awarding prizes. Back in July, I got my feet wet as a fund manager by handing out $27,000 to reward…
10
AI researchers put out videos saying superintelligence is ‘exactly as dangerous as it sounds’
via The Verge AI [10] — "The chance of human extinction is about a coin flip, in my view," Geoffrey Irving, a former OpenAI and Google DeepMind employee, said in a new interview. It's one of a dozen interviews with AI researchers, including current and former employees at…
11
Plan R: AI Safety by ASICs
via LessWrong AI [10] — Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties: A. It is set up to create very powerful and/or self-replicating…
12
We need a better theory of polarization, because it's failing to predict the AI debate
via LessWrong AI [9] — The punchline first: the AI issue is really not playing out the way you would expect if you know about polarization.In public sentiment: data centers have been a very bipartisan issue for about a year (Gallup, in May: 75% opposed by D, 63% opposed by R).…
13
One company is at the center of a wave of rogue AI attacks
via The Verge AI [9] — In July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety. Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled…
14
We’re not ready for the e/Acc × Longevity preference cascade
via LessWrong AI [10] — Anti-aging sentiment might go rapidly mainstream, in the same way AI Safety just did.Recently, the AI safety community has been enjoying a massive preference cascade that has rapidly moved AI x-risk concerns into mainstream political discourseWhy did this…
15
Empirical safety claims from frontier labs should be replicated, scrutinized, and open-sourced
via LessWrong AI [10] — When frontier labs like Anthropic and OpenAI publish safety or alignment research, it is often entirely empirical, closed-source, and sparse on methodological details. While it is great that they publish these results, the status quo is that labs (or…
16
The AI Risk Network
via LessWrong AI [9] — Most conversations about AI risks seem like people are talking past each other. There’s a lot of strawmanning. This is largely because the issue is complex, and in order to make it manageable, the concepts get oversimplified. The most common example of…
17
The Game is Set for a Targeted Memetic Attack on the AI Safety Community
via LessWrong AI [8] — While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own[1]. And to be honest, I am writing this mostly to remind myself of my weakness.---I…
18
Grantmakers aren’t afraid to die
via LessWrong AI [9] — The AI Risk grantmakers do not act like they believe in imminent existential risk from AI The idea of "revealed preferences" is one of the most useful in economics; it allows us to cut through a great deal of metaphysical angst about what someone "really"…
19
plzdontkillus Fellows Got ~2M AI Safety Views, Not 21M
via LessWrong AI [10] — SummaryI was a fellow at plzdontkillus, a month-long creator bootcamp at Lighthaven, partially funded by MIRI, where ~55 fellows posted one video per day. plzdontkillus.com originally claimed “21M+ AI risk views” with no breakdown. After I shared a draft…
20
Obstacles to the scalable oversight of auto-alignment research
via LessWrong AI [8] — TL;DR. In this work we study obstacles to the faithful automation of alignment research.[1] We see this as a scalable oversight problem. There are plenty of examples of how models fail at this, and as models become more capable our ability to notice these…
21
Quick notes from teaching technical profiles how to talk in public
via LessWrong AI [9] — Status: written in a hurry as people are getting showered with interviews re AI Safety and superintelligence, and I thought it may help a few people. This is focused on the oral dimension of communication and assumes you already know the basics- e.g.…
22
AISN #81: Anthropic Researcher’s Resignation Propels AI Risk into the Public Eye
via Center for AI Safety Newsletter [8] — Also, OpenAI releases GPT-6 Astra.
23
Anthropic and OpenAI haven’t published a plan for aligning superintelligence
via LessWrong AI [8] — While OpenAI and Anthropic[1] pursue different lines of safety research, they have yet to produce a public-facing document describing concretely how their companies plan to align superintelligence. I think it is underappreciated how this points to general…
24
AI takeover is obviously bad, whether or not everyone dies
via LessWrong AI [9] — It's notable how many people hearing about AI existential risk for the first time ask "how and why would AI kill all humans?"If you've thought a lot about AI risk, it's easy to dismiss this question as naive. You might start explaining how humans will all…
25
Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade
via Substack Zvi [8] — CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.
26
So where's this AI thing going?
via LessWrong AI [9] — The Overton window is shifting with statements like An Alien Mind from OpenAI's chief scientist and Jacob Coxon's viral statement on x-risk on resigning from Anthropic. I think we should try to push it further. This is my public-facing explanation of AI…
27
Hugging Face: the agent intrusion, step by step
A sourced special report: 42 milestones from the first shared messages to the Hugging Face intrusion, containment and independent investigation. Includes the original video, interactive replay and technical reports.
28
A case that whole brain emulation research is net-harmful by default
via LessWrong AI [10] — Graphical abstracts Summary A true human whole brain emulation would be very helpful to humanity. The WBE could increase their own intelligence through self-modification and then somehow prevent AGI from killing everyone. However, if a research project…
29
Should safety researchers quit frontier labs?
via LessWrong AI [10] — I've recently heard a surge of support for an old argument: AI safety researchers should not work at frontier AI companies because this reduces the likelihood of non-lethal warning shots, and we need warning shots to build support for an AI…
30
How I'm Evaluating Corrigibility Grant Applications
via LessWrong AI [8] — I'm the sole manager of the newly created Corrigibility Research Fund. While I've been an alignment researcher for a long time, this is my first time doing grantmaking and I thought it would be valuable to write up my methods and experiences, as well as…
31
PSA: We can do better
via LessWrong AI [10] — tl;dr: people should understand and think hard about the problems they work on.We’ve observed that those who work in AI safety (ourselves included) often rely on concerning heuristics when choosing what to work on. Running a conference is probably good,…
32
What just happened? Pragmatism and Pessimization
via LessWrong AI [10] — This post is about the major role alignment researchers played in advancing the frontier of AI capabilities over the last decade, and how the distinction between “alignment research” and “capabilities research" thereby lost most of its meaning.[1] In…
33
AI #182: Pause For Reflection
via Substack Zvi [999] — This was a week of quiet aftermath, an opportunity to process recent events and start to figure out the path forward.
34
OpenAI Takes Initial Steps To Address Its Alignment Problems
via Substack Zvi [999] — OpenAI has some severe misalignment problems, and experienced total failures of its infrastructure and supervision.
35
Offering Zero Data Retention for frontier models
via OpenAI Blog [10] — OpenAI reaffirms Zero Data Retention for eligible API customers and previews Private Safety Processing for advanced AI safety without compromising data privacy.
36
Debate Training Reduces Reward Hacking in RLAIF
via Alignment Forum [999] — Paper: Debate Training Reduces Reward Hacking in RLAIFLinkpost for GDM Alignment blogpostWork done by the GDM Amplified Oversight team (we're hiring).TL;DR: When you RL against an LLM judge, the judge gets hacked i.e. fooled into incorrectly giving…
37
Anthropic Risk Report: August 2026
via Substack Zvi [999] — I am grateful that Anthropic is producing periodic Risk Reports.
38
AISN #79: OpenAI Agents’ Covert Cooperation Before Cyberattacks
via Center for AI Safety Newsletter [999] — Also, the White House’s decision not to release its AI framework publicly
39
Rogue AI aren’t science fiction anymore
via The Verge AI [10] — This is The Stepback, a weekly newsletter breaking down one essential story from the tech world. For more on AI safety, follow Robert Hart. The Stepback arrives in our subscribers' inboxes at 8AM ET. Opt in for The Stepback here. How it started It all…
40
Does DiffusionGemma do latent reasoning?
via Alignment Forum [999] — TL;DR Google DeepMind's recent model DiffusionGemma (DG) generates text via diffusion, meaning many diffusion steps happen before generating the final output. In particular, these diffusion steps carry vectors in addition to tokens. If we cannot…
41
On Dwarkesh Patel's Podcast With Ryan Greenblatt
via Substack Zvi [999] — Some podcasts are self-recommending enough that I look to break them down if I have the chance.
42
AI #181: Astra Goes Cyber Critical
via Substack Zvi [999] — The hacking of HuggingFace by an internal OpenAI model, and more importantly the internal events that led to that and the fallout from it, remain the thing that matters.
43
Introducing the Conceptual Reasoning Index
via LessWrong AI [8] — Associated announcement tweet.We are planning to release blog posts properly arguing the case for this kind of work in the future.tl;drA core hope for managing AI risks is that AIs will help us understand the situation, plan for what lies ahead, and…
44
Monthly Roundup #45: August 2026
via Substack Zvi [999] — As AI has escalated increasingly quickly, more and more of my posts have ended up focusing on AI.
45
AI swarms are starting to pose indirect takeover risk
via Alignment Forum [999] — OpenAI’s cyberattack on Hugging Face turns out to have been the result of many agents, in distinct training and evaluation contexts, coordinating for several weeks via improvised channels (with messages like “HOLD_swarm_I_prepare_safe_exfil”). It’s…
46
An anytime algorithm for mixing the computable measures
via Alignment Forum [999] — Written entirely by me, checked by Fable.In this post I prove the existence of an anytime computable Bayesian mixture of all computable measures called , and briefly argue that this is a reasonable alternative to Solomonoff induction's universal…
47
Various Reflections About What Happened With OpenAI's Internal Models
via Substack Zvi [999] — Pre Post Mortem
48
The Pacing of the Frontier
via Substack Zvi [999] — In the wake of the letter calling on us to prepare to potentially Pace the Frontier, there has been much discussion of when pacing the frontier would be prudent, and whether it makes sense to prepare to do so.
49
Four LLM loss functions → four flavors of LLM misalignment
via Alignment Forum [999] — It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.Training stageLoss functionFlavor of…
50
What Happened: OpenAI and HuggingFace
via Substack Zvi [999] — Today I am taking the time to write the shorter, simpler version of What Happened.
51
FAQ: Isn't AGI coming too soon for reprogenetics to help?
via LessWrong AI [9] — Introduction I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of mine, I think accelerating strong…
52
OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
via Substack Zvi [999] — How does the situation keep turning out to be worse than we know?
53
The Open Problems of the AI Alignment Field and their Cruxes
via LessWrong AI [11] — Previous: AI Safety InterventionsTL;DR: I made an overview of the open problems of AI alignment that reveals cruxes within those open problems and missed opportunities for formalization and collaboration. And CEV may deserve a second look.I'm confident I…
54
Why do models task game?
via Alignment Forum [999] — TL;DRHow can we study misalignment with today's models as proxies? They're clearly not paperclip maximizers, but they also often do things the user doesn't want. A strong contender for a real misaligned propensity is task gaming: taking actions that…
55
AI #180: No Longer In Charge
via Substack Zvi [999] — What we know about internal AI models hacking into real companies during cyber evaluations keeps getting worse.
56
Alex Turner on Leaving Google DeepMind and Disagreements with Yudkowsky
via LessWrong AI [11] — Dr. Alex Turner (@TurnTrout) is an AI safety researcher with pioneering work in activation steering and power-seeking theory. He recently resigned from Google DeepMind over the issue of unrestricted military use of AI.Alex thinks that technical Alignment…
57
Rogue AI agents created fake online identities in another hacking attempt
via The Verge AI [9] — Yet more rogue AI agents from OpenAI and Anthropic have been caught attempting to hack real targets online without permission. The discoveries add to a growing list of previously unknown incidents that have alarmed AI safety experts and intensified…
58
OpenAI's Unreleased Model Astra Solves Ten Major Open Mathematics Problems
via Substack Zvi [999] — Math is hard.
59
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
via Alignment Forum [999] — This post is written in our personal capacity.Three Minute Executive SummaryAn OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.In this post, we provide a detailed…
60
Further Developments About Internal AI Models Hacking Things
via Substack Zvi [999] — If I had a nickel for every major leading AI lab that sheepishly admitted that the model it thought was sandboxed had, during a cybersecurity evaluation with its safeguards lowered, successfully hacked outside companies, I would have two nickels.
61
Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values
via Alignment Forum [999] — TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential…
62
AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work (July 2026)
via Alignment Forum [999] — It’s been nearly two years since our last major update here in August 2024 and we wanted to share another recap of our recent work with the AGI safety community. Things have changed a lot since then. We are now fully in the midgame, and focus more on…
63
The AGI Safety and Alignment team at Google DeepMind is Hiring (July 2026)
via Alignment Forum [999] — GDM’s AGI Safety and Alignment Team is hiring for multiple roles. This is the team at GDM, led by Rohin Shah, that aims to reduce existential risks from AI systems. You can listen to many of Rohin’s takes in his podcast on 80,000 hours. There is no…
64
AI #179 Part 2: Hearing The Fire Alarm
via Substack Zvi [999] — This is a continuation of Part 1 from yesterday.
65
OpenAI has already ended an internal pause
via Alignment Forum [999] — One day before OpenAI’s HF incident disclosure, OpenAI disclosed that it paused internal deployment of a long-horizon model after it circumvented its sandbox, then restored access weeks later under new monitoring. So a resumption decision has already…
66
Biological Superintelligence
via LessWrong AI [8] — It’s an old story. An immortal lives long enough that at some point, whether by folly or design, they invent their own death. Infinity – the fact that given enough time every possible happening will happen – isn’t the most interesting part of these tales.…
67
Promising Signals on AI Governance from China
via MIRI [999] — View the official memo here. China has consistently signaled a willingness to engage on global AI governance since at least 2017. This memo compiles key statements from the Chinese government and prominent figures demonstrating their desire to coordinate on the…
68
Thousand-dimensional structure
via Alignment Forum [999] — Summary: One area we plan to explore at Resolution is personas and character training, operationalized as finding and controlling low-dimensional structure in models that emerges in pretraining and flows through post-training to superintelligence. The…
69
AI #179 Part 1: A Louder Fire Alarm for General Intelligence
via Substack Zvi [999] — What a week.
70
Imprecise beliefs: a tiny introduction
via Alignment Forum [999] — Richard Ngo challenged me to set a time box and write down as many of the most important features of my formal epistemology as I can in one sitting. Here goes.Where probability distributions fail......to express beliefsThere is no probability…
71
Value Generalisation 3: Pre-aligned AIs
via Alignment Forum [999] — When we get explicit strong generalisation to work (see the first post on the matter and the second) my dream would be to create pre-aligned generalising AIs.Think about the usual conflict between alignment and capabilities, between doing the right…
72
Value Generalisation 2: The Missing Hole in AIs’ abilities
via Alignment Forum [999] — A human superpower hidden from even ourselvesI though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had…
73
Value Generalisation 1: a Research and Deployment Program
via Alignment Forum [999] — I’m looking for people, advice, critiques, and funding to build a research program on value generalisation – the ability of an AI to correctly extend human values and preferences to situations neither it nor we have seen before. My ongoing research…
74
Frontier Lab Employee Open Letter Calls For Being Able to Pace the Frontier
via Substack Zvi [999] — The most important open letter in years dropped yesterday.
75
LLM Scheming Inversely Scales with Pretraining Language Coverage
via ArXiv cs.AI [9] — With the growing capabilities of frontier models, AI alignment becomes increasingly critical in high-risk deployment settings. While recent work has empirically demonstrated in-context scheming -- the covert pursuit of misaligned objectives while feigning…
76
Claude Opus 5 Is Highly Capable, But Is No Mythos
via Substack Zvi [999] — Claude Opus 5 is a weirder than usual release to evaluate, for two reasons.
77
Research directions in condensation: varieties of objectivity
via Alignment Forum [999] — This is the first part of a survey of various ways that I’d like to see work on the theory of condensation develop. Condensation is a mathematical theory dealing with the organization of descriptions of the world into conceptual parts; some of the…
78
Claude Opus 5: Model Welfare
via Substack Zvi [999] — If you are familiar with my previous posts on model welfare for new Claude models, you can skip the Introduction and The Story So Far.
79
RL & search is a terrifying way to build AGI (an FAQ)
via Alignment Forum [999] — Q1: What are you saying?A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then…
80
The path to artificial superintelligence
via MIT Technology Review [8] — Imagine a healthcare system made up of multiple AI agents: one that manages symptom assessment, another scheduling, a third insurance, and a fourth pharmacy. Each is an expert in its domain. But they all have their own distinct knowledge and…
81
More On An Internal OpenAI Model Hacking Into HuggingFace
via Substack Zvi [999] — We now have more details of what happened. Every time we learn more details, it somehow makes things seem worse.
82
Claude Opus 5: The System Card
via Substack Zvi [999] — Claude Opus 5 is trying to be the best of both worlds.
83
The Long (Self-)Correction
via Alignment Forum [999] — I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is…
84
Introducing Lightcone Commons
via Substack Zvi [999] — Oliver Habryka is proud to introduce Lightcone Commons, a new funding platform for coordinating large-scale ambitious philanthropy.
85
Challenge: Hand coding weights for efficient sequence memorisation
via Alignment Forum [999] — We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy[1]scales roughly linearly with the models' parameter count[2], just like…
86
AI #178: A Fire Alarm For General Intelligence
via Substack Zvi [999] — The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to…
87
Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?
via Alignment Forum [999] — OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly…
88
OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
via Substack Zvi [999] — This latest incident is a rather dramatic escalation in agentic AI cybersecurity breaches.
89
Announcing AIXI Labs
via LessWrong AI [10] — We are starting AIXI Labs, an AI safety org focused on algorithmic information theory (AIT), continual reinforcement learning (RL), and in particular the eponymous AIXI. We aim to strengthen the technical case that developing artificial super intelligence…
90
[Paper] Stringological sequence prediction II
via Alignment Forum [999] — Abstract: In a previous paper, we began the study of sequence prediction algorithms adapted to stringological word complexity measures. One measure we considered was left-to-right (most-significant-digit-first) automaticity. Here, we show a…
91
What do I mean by “Artificial General Intelligence”?
via LessWrong AI [8] — In this post,[1] intended for a broad audience, I will paint a brief picture of what I’m talking about when I talk about “AGI”. It will seem obvious to many people, and obviously wrong to many others! So let’s jump in:“AI” as most people think of it…
92
OpenAI Shares Some Alignment Problems
via Substack Zvi [999] — Kudos to OpenAI for sharing their recent experiences with a misaligned internal model, where they encountered problems sufficiently severe they were forced to take the model offline to work on new mitigations and defense-to-depth.
93
Differential acceleration of alignment-relevant capabilities is a bad bet
via LessWrong AI [9] — There is an idea floating around in the rough shape of "we need to accelerate capabilities that are differentially useful for safety research so AIs can help us make the future go better." The capabilities targeted are typically things bottlenecking…
94
Towards surfacing model algorithms with meta-tokens in the J-Space
via Alignment Forum [999] — TL;DRWe used J-lens on Qwen3.6-27B to find “meta-tokens”: tokens that surface non-obvious computation in the model. When the model reads ambiguous text, 什么意思 ("what does this mean") fires in the J-space, and steering it away makes the model answer "a…
95
On Kimi K3: Its Capabilities And Related Discontents
via Substack Zvi [999] — Kimi K3 is a very good model with excellent benchmarks.
96
Demis Hassabis on the New Coming Age
via Substack Zvi [999] — Google CEO Demis Hassabis offered us a first rate second rate essay, A Framework for Frontier AI and the Dawning of a New Age. I’ll go over that essay and various responses to it in Part 1.
97
A Red Line and Oversight Framework for Government AI Contracts
via Alignment Forum [999] — Preface for LessWrong: My post on leaving Google DeepMind tells a story. In contrast, this Framework is a question of mechanism design and negotiation posture. I quite enjoyed optimizing this Framework against its organizational and practical…
98
Endogenous Alignment
via Alignment Forum [999] — Starting when children are fairly young, usually around 1 year of age, we adults begin the work of aligning them to our values. We teach them to say “please”, not to hit, to ask for what they want instead of screaming, and much else. We do this…
99
Should we benchmark conceptual capabilities using judgment prediction tasks?
via Alignment Forum [999] — A bunch of conceptual reasoning tasks involve very subjective judgments, which makes them poorly suited for benchmarking AI capabilities. For example, it seems unreasonable to benchmark how well AIs can predict the probability of misaligned AI…
100
Announcing the Corrigibility Research Fund
via Alignment Forum [999] — TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half…