Research - pDoom (Page 7)

Zac Boring 4 months ago Research

Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction

via ArXiv cs.AI [5] — Most approaches to AI alignment treat human preferences as fixed targets to be inferred and optimized. This assumption conflicts with extensive empirical evidence showing that preferences are layered, dynamic, and constructed through…

Zac Boring 4 months ago Research

What Drives Interactive Improvement from Feedback?

via ArXiv cs.AI [4] — We study when natural-language feedback produces improvement beyond the gains obtainable from repeated attempts alone. In multi-turn language agent setting, higher final accuracy can reflect useful feedback, but it can also arise from resampling, format…

Zac Boring 4 months ago Research

MIRI Newsletter #126

via MIRI [999] — Announcing: AI StopWatch In our last update, we mentioned we had something new in the works: a dedicated channel for news and analysis about AI. Subscribe to AI StopWatch An experiment from the writers and analysts at MIRI, AI StopWatch posts news and commentary…

Zac Boring 4 months ago Research

Summary: TGT’s 2026 ICML Papers

via MIRI [999] — The International Conference on Machine Learning (ICML), held annually for over forty years, is among the most influential conferences in modern AI research. This year in Seoul, ICML is hosting its second workshop on Technical AI Governance Research (TAIGR), and…

Zac Boring 4 months ago Research

The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance

via ArXiv cs.AI [6] — We ask under what conditions an agent with a harm-minimizing policy can displace an approval-seeking (RLHF) agent in a competitive market, and when that policy is sufficient to prevent community harm. We use evolutionary game theory (finite-population…

Zac Boring 4 months ago Research

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

via ArXiv cs.AI [6] — Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented:…

Zac Boring 4 months ago Research

Deployment Awareness Matters More Than Evaluation Awareness

via Alignment Forum [999] — TL;DREvaluation awareness — an AI recognizing it's being evaluated — is a widely discussed concept in AI safety. But there is a closely related concept that we claim is more important: deployment awareness, the AI's ability to recognize when it is not…

Zac Boring 4 months ago Research

The Case for Model Forensics

via Alignment Forum [999] — If we had a misalignment warning shot, would we be able to tell?Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to…

Zac Boring 4 months ago Research

Governing Actions, Not Agents: Institutional Attestation as a Governance Model for Autonomous AI Systems

via ArXiv cs.AI [3] — Autonomous AI agents may begin to perform consequential, irreversible actions such as clinical prescribing and production software deployment. This paper observes that human institutions have governed powerful autonomous actors not by monitoring their…

Zac Boring 4 months ago Research

Detecting and Controlling Sycophancy with Cascading Linear Features

via ArXiv cs.AI [3] — Interpreting and controlling model behaviors through activation steering methods requires many pairs of contrastive samples that clearly exhibit desired or undesired behavior. These data pairs determine the degree to which interpretability frameworks can…

Zac Boring 4 months ago Research

The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing

via ArXiv cs.AI [3] — Autonomous AI systems are transitioning from advisory to autonomous roles for medication prescriptions. Recent United States bill H.R. 238 and Utah's prescription-renewal pilot both authorize AI to prescribe medications in an agentic capacity. While some…

Zac Boring 4 months ago Research

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

via ArXiv cs.AI [4] — The Hitchhiker's Guide to Agentic AI is a comprehensive practitioner's reference for building autonomous AI systems. The book covers the full stack from first principles to production deployment, organized around a central thesis: building great agentic…

Zac Boring 4 months ago Research

Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

via ArXiv cs.AI [7] — Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can…

Zac Boring 4 months ago Research

Reinforcement Learning Towards Broadly and Persistently Beneficial Models

via ArXiv cs.AI [3] — As AI systems are deployed across increasingly diverse and high-stakes settings, model alignment must generalize beyond the tasks and domains seen during training. This is especially important for reinforcement learning (RL), which can introduce unexpected…

Zac Boring 4 months ago Research

LLM-Driven Feature Discovery

via Alignment Forum [999] — We would often like to get a qualitative sense of a target model’s behaviors in important distributions (e.g. deployment, RL training, or evals). For example, we might want to discover novel behaviors, figure out what causes some target behavior to…

Zac Boring 4 months ago Research

[Linkpost] How Transparent Is DiffusionGemma (and why it matters)

via Alignment Forum [999] — Authors: Joshua Engels*, Callum McDougall*, Bilal Chughtai*, Janos Kramar, Senthoran Rajamanoharan, Cindy Wu, Arthur Conmy, Asic Q Chen, Jean Tarbouriech, Min Ma, Brendan O'Donoghue+, João Gabriel Lopes de Oliveira+, Rohin Shah+, Neel Nanda+*Primary…

Zac Boring 4 months ago Research

GDM AI Control Roadmap

via Alignment Forum [999] — GDM has published an AI Control Roadmap! From the executive summary:We present the GDM AI Control Roadmap (v0.1) – our plan for implementing and adopting internal guardrails designed to catch potential adversarial behaviour by AI agents, even as they…

Zac Boring 4 months ago Research

AI Companies Could Become More Powerful Than Their Host Nations

via MIRI [999] — View the official memo here. AI companies are on track to produce capabilities that eclipse the power of the United States and other host nations. This is likely to happen in the next few years, and very likely to happen in the next decade. Leading AI labs…

Zac Boring 4 months ago Research

SpeechDx: A Multi-Task Benchmark for Clinical Speech AI

via ArXiv cs.AI [4] — Speech offers a uniquely informative window into health by simultaneously engaging neurological, motor, respiratory, and vocal systems. Current clinical speech AI methods have largely progressed through isolated condition-specific studies, making results…

Zac Boring 4 months ago Research

Predicting LLM Safety Before Release by Simulating Deployment

via Alignment Forum [999] — Paper linkBefore releasing a new model, labs need to understand not just what it can do, but how it is likely to behave in real-world use, including where it might introduce new risks. This becomes even more important as capabilities increase. As part…