Research - pDoom (Page 2)

Zac Boring a month ago Research

A Defense of Gradual Disempowerment

via Alignment Forum [5] — (Or: Why Bentham's Bulldog and John Halstead are wrong in their critique of Kulveit et al.)Gradual Disempowerment is a 2025 paper (with a nice, dedicated website) proposing a form of existential risk from AI that goes beyond "mundane" risks like…

Zac Boring a month ago Research

Toward Governance-Aware Autonomous GIS: A Narrative Review of Ethical and Privacy Risks in LLM-Enabled GeoAI

via ArXiv cs.AI [3] — Geospatial artificial intelligence (GeoAI) powered by large language models (LLMs) is expanding the capacity to query, generate, and interpret spatial information through natural-language interfaces and agentic autonomous GIS workflows. This capability…

Zac Boring a month ago Research

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

via Alignment Forum [3] — It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring[1], help us do better science on current models[2], and augment certain forms of alignment training[3]. Currently,…

Zac Boring a month ago Research

AISN #81: Anthropic Researcher’s Resignation Propels AI Risk into the Public Eye

via Center for AI Safety Newsletter [8] — Also, OpenAI releases GPT-6 Astra.

Zac Boring a month ago Research

Op-Ed: I Worked at Google DeepMind. You Should Listen to the Warnings About AI

via Alignment Forum [4] — Published in The Guardian.Major AI lab CEOs recently advocated for pacing AI development. They are right to be concerned: the field runs an extremely dangerous race towards superintelligent AI. We can and should demand that our governments protect us…

Zac Boring a month ago Research

CoT controllability evals seem very under-elicited

via Alignment Forum [4] — The CoTControl eval asks reasoning models to follow formatting constraints in their chain-of-thought (e.g. write in all lowercase, avoid a specific word) while solving questions. Models seem to mostly be pretty bad at this: recent models score between…

Zac Boring a month ago Research

An operationalization of opaque serial depth

via Alignment Forum [3] — Currently, chain-of-thought (CoT) is a valuable tool for overseeing AI models. However, some architectural shifts could significantly reduce CoT monitorability. We have recently proposed that AI companies should transparently share information about…

Zac Boring a month ago Research

Proposal for tracking the effects of architecture on monitorability

via Alignment Forum [3] — Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going…

Zac Boring a month ago Research

Adaptive Entangled Game Modules in Artificial General Intelligence

via ArXiv cs.AI [4] — We introduce a probability-wave framework for modeling the collective behavior of interacting adaptive agents, deriving testable eigenmodes through a generalized behavioral intelligence (GBI) nonlocal probability-wave equation. This framework captures a…

Zac Boring a month ago Research

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

via ArXiv cs.AI [3] — Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose…

Zac Boring a month ago Research

Astra can do a concerning amount with no chain of thought

via Alignment Forum [3] — TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)

Zac Boring a month ago Research

Compiling VGDL into Causal Models

via ArXiv cs.AI [4] — Reinforcement learning and large language models often struggle to accurately capture the causal mechanics of game environments. Standard reinforcement learning agents tend to rely on spurious correlations, while large language models are prone to…

Zac Boring a month ago Research

Beyond Right and Wrong: Evaluating Second-order Social Reasoning in Large Language Models

via ArXiv cs.AI [6] — Previous AI alignment efforts have focused primarily on first-order social norms -- teaching models what is socially acceptable or unacceptable (e.g., `do not steal'). However, social intelligence depends not only on norm recognition, but also on…

Zac Boring a month ago Research

Training on probes: Research ideas

via Alignment Forum [4] — RecapSequel to Previous Post. This post might not make sense without it.Training on probes might let our judgments on easy domains generalize to harder domains by leveraging an AI's model of the world. Taking the gradient of the probe teaches the model…

Zac Boring a month ago Research

Training on probes: What's going on

via Alignment Forum [3] — TL;DRIf you train a probe for some property (like "honesty") and do gradient descent against this probe while continuing training that incentivizes dishonesty, the model will change its internal representation to evade the probe. Duh.If you train…

Zac Boring 2 months ago Research

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

via Alignment Forum [3] — We recently published the report from our brief independent investigation into this incident. You can read the full report here. Here is our tweet thread summarizing what we found: METR & Redwood Research investigated agent behavior in the Hugging Face…

Zac Boring 2 months ago Research

There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items

via ArXiv cs.AI [5] — Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whether a language model's answer is read from generated text or from per-option likelihoods. Work on this…

Zac Boring 2 months ago Research

PrimeAgentOrchestrator: Memory-Primed Agent Spawning for Personal AI Infrastructure

via ArXiv cs.AI [4] — Large language model (LLM) coding agents start each session with an empty context window, discarding accumulated knowledge from prior work. We present PrimeAgentOrchestrator (PAO), a system that spawns new instances of Claude Code -- Anthropic's…

Zac Boring 2 months ago Research

Debate Training Reduces Reward Hacking in RLAIF

via Alignment Forum [999] — Paper: Debate Training Reduces Reward Hacking in RLAIFLinkpost for GDM Alignment blogpostWork done by the GDM Amplified Oversight team (we're hiring).TL;DR: When you RL against an LLM judge, the judge gets hacked i.e. fooled into incorrectly giving…

Zac Boring 2 months ago Research

GxP-Agent: Process-DAG Topology for Reliable Clinical Trial Programming with LLM Agents

via ArXiv cs.AI [4] — Clinical trial programming -- transforming study protocols into analysis-ready datasets under CDISC standards -- is a bottleneck in regulatory submissions, yet LLM-based code generation fails catastrophically on this task: across 11 single-shot attempts…