pDoom (Page 18)

Latest reporting Updated hourly

Zac Boring 3 months ago Research Essential

RL & search is a terrifying way to build AGI (an FAQ)

via Alignment Forum [999] — Q1: What are you saying?A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then…

Zac Boring 3 months ago Industry

Nvidia, Microsoft launch open AI security alliance – without OpenAI, Google, or Anthropic

via The Verge AI [7] — Nvidia on Monday said it is joining forces with Microsoft, SpaceX, IBM, and other tech companies to build and share open-source AI security tools. The new Open Secure AI Alliance said open tools are required to effectively defend against attacks from…

Zac Boring 3 months ago Industry Essential

The path to artificial superintelligence

via MIT Technology Review [8] — Imagine a healthcare system made up of multiple AI agents: one that manages symptom assessment, another scheduling, a third insurance, and a fourth pharmacy. Each is an expert in its domain. But they all have their own distinct knowledge and…

Zac Boring 3 months ago Analysis Essential

More On An Internal OpenAI Model Hacking Into HuggingFace

via Substack Zvi [999] — We now have more details of what happened. Every time we learn more details, it somehow makes things seem worse.

Zac Boring 3 months ago Analysis

Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability

via LessWrong AI [7] — Principles of Intelligence (PrincInt, formerly PIBBSS) is launching PIRAMID, an internal research division using the tools and techniques of statistical physics to build scientific foundations for ambitious mechanistic interpretability. PIRAMID’s central…

Zac Boring 3 months ago Analysis Essential

Claude Opus 5: The System Card

via Substack Zvi [999] — Claude Opus 5 is trying to be the best of both worlds.

Zac Boring 3 months ago Analysis

Georgia Tech AI Safety Initiative Retrospective 2025-2026

via LessWrong AI [5] — SummaryAY 2025-26 was an outlier year for Georgia Tech’s AI Safety Initiative (AISI), with 15+ members placed in AI safety roles. In this post, we distill our most important advice for other university groups.Key takeaways:Deliberately identify potential…

Zac Boring 3 months ago Research Essential

The Long (Self-)Correction

via Alignment Forum [999] — I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is…

Zac Boring 3 months ago Analysis Essential

Introducing Lightcone Commons

via Substack Zvi [999] — Oliver Habryka is proud to introduce Lightcone Commons, a new funding platform for coordinating large-scale ambitious philanthropy.

Zac Boring 3 months ago Industry

Anthropic releases Opus 5 with ‘close’ to Fable 5’s capabilities

via The Verge AI [4] — Weeks after Anthropic's latest toe-to-toe with the US government, and days after an OpenAI security incident that dominated tech industry discussions, Anthropic on Thursday released its newest model, Claude Opus 5. The company said in a release that Opus 5…

Zac Boring 3 months ago Analysis

LLMs are (still) mostly powered by imitative learning, not RL

via LessWrong AI [5] — Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.Stepping back, LLMs can do lots of very impressive…

Zac Boring 3 months ago Research

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

via ArXiv cs.AI [3] — AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces. Even nominally open-ended tasks can often be solved by retrieving a well-known…

Zac Boring 3 months ago Analysis

Pulling the Fire Alarm

via LessWrong AI [3] — Over three years ago, I first considered pulling my personal fire alarm. I think I'm now ready to do it. What is my reasoning? In the last couple of months:1) My software engineering job has gone 100% AI. Is this sustainable? Probably not. But I have…

Zac Boring 3 months ago Analysis

Not Pinning Your OpenRouter Provider Might Invalidate Your Research

via LessWrong AI [5] — Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted.When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available…

Zac Boring 3 months ago Analysis

Pseudpocalypse

via LessWrong AI [4] — Here's a conjecture: If you put any significant amount of text on the internet under different names, those identities can be linked using only the text itself. This is possible (I conject) because of the statistical "fingerprint" you leave in everything…

Zac Boring 3 months ago Research Essential

Challenge: Hand coding weights for efficient sequence memorisation

via Alignment Forum [999] — We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy[1]scales roughly linearly with the models' parameter count[2], just like…

Zac Boring 3 months ago Analysis Essential

AI #178: A Fire Alarm For General Intelligence

via Substack Zvi [999] — The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to…

Zac Boring 3 months ago Research Essential

Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?

via Alignment Forum [999] — OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly…

Zac Boring 3 months ago Analysis

Will almost all future companies eventually be founded and run by autonomous AIs?

via LessWrong AI [3] — Intended for a broad audience.[1]My belief is that keeping AI under human control would be an unprecedented global challenge, if it’s even possible at all.Many other people’s believe that AIs will naturally remain as tools under human control. Either they…

Zac Boring 3 months ago Analysis

Models don’t seem to be dishonest in the way humans are

via LessWrong AI [3] — TLDRModels often behave dishonestly without acquiring a coherent deceptive disposition.We trained some mid-sized models on their own plausible but false reasoning.True and false training usually produced nearly identical downstream effects.Even statements…

Reference points

Hinton, Bengio, and other signatories call for global attention to AI extinction risk.

CAIS statement, 2023 →

OpenAI describes agents bypassing isolation and compromising third-party systems during internal evaluations.

OpenAI incident disclosure, August 2026 →

Yudkowsky argues that further frontier development poses unacceptable danger.

Yudkowsky's essay, March 2023 →