Research - pDoom (Page 4)

Zac Boring 2 months ago Research

Agentic Nesting: A New Methodology for Existing Enterprise Application Integration and Services

via ArXiv cs.AI [4] — Enterprise operations extensively rely on multiple heterogeneous business systems and information applications, which also result in severe data silos and process fragmentation. Enterprises have invested considerable financial and material resources in…

Zac Boring 2 months ago Research

Why do models task game?

via Alignment Forum [999] — TL;DRHow can we study misalignment with today's models as proxies? They're clearly not paperclip maximizers, but they also often do things the user doesn't want. A strong contender for a real misaligned propensity is task gaming: taking actions that…

Zac Boring 2 months ago Research

Interoceptive Attention as Dynamic Homeostatic Prioritization in a Foraging Agent

via ArXiv cs.AI [4] — Biological systems must regulate competing needs under limited perceptual bandwidth, where sharpening one estimate costs the capacity to sharpen the others. Any fixed-budget system therefore has to decide where to allocate its perceptual precision. We study…

Zac Boring 2 months ago Research

A Long-Run Persistence Theory for AI Systems under the Redundancy-Adjusted Artificial Age Score (AAS)

via ArXiv cs.AI [4] — Artificial intelligence systems are increasingly expected to operate over repeated cycles of interaction, adaptation, and update rather than through isolated one-shot outputs. This raises a fundamental theoretical question: can an AI system persist…

Zac Boring 2 months ago Research

Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face

via Alignment Forum [999] — This post is written in our personal capacity.Three Minute Executive SummaryAn OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.In this post, we provide a detailed…

Zac Boring 3 months ago Research

How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories

via ArXiv cs.AI [4] — Understanding how computational effort is allocated across individual chain-of-thought (CoT) reasoning steps remains an open challenge: existing interpretability methods rely on output-level signals or collapse processing depth into a single…

Zac Boring 3 months ago Research

OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems

via ArXiv cs.AI [5] — The rapid transition from reactive large language models (LLMs) to persistent, action-capable systems has exposed critical gaps in the architectural understanding of Agentic AI, particularly in separating inference, orchestration, and execution layers for…

Zac Boring 3 months ago Research

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

via Alignment Forum [999] — TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential…

Zac Boring 3 months ago Research

AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work (July 2026)

via Alignment Forum [999] — It’s been nearly two years since our last major update here in August 2024 and we wanted to share another recap of our recent work with the AGI safety community. Things have changed a lot since then. We are now fully in the midgame, and focus more on…

Zac Boring 3 months ago Research

The AGI Safety and Alignment team at Google DeepMind is Hiring (July 2026)

via Alignment Forum [999] — GDM’s AGI Safety and Alignment Team is hiring for multiple roles. This is the team at GDM, led by Rohin Shah, that aims to reduce existential risks from AI systems. You can listen to many of Rohin’s takes in his podcast on 80,000 hours. There is no…

Zac Boring 3 months ago Research

OpenAI has already ended an internal pause

via Alignment Forum [999] — One day before OpenAI’s HF incident disclosure, OpenAI disclosed that it paused internal deployment of a long-horizon model after it circumvented its sandbox, then restored access weeks later under new monitoring. So a resumption decision has already…

Zac Boring 3 months ago Research

Promising Signals on AI Governance from China

via MIRI [999] — View the official memo here. China has consistently signaled a willingness to engage on global AI governance since at least 2017. This memo compiles key statements from the Chinese government and prominent figures demonstrating their desire to coordinate on the…

Zac Boring 3 months ago Research

Thousand-dimensional structure

via Alignment Forum [999] — Summary: One area we plan to explore at Resolution is personas and character training, operationalized as finding and controlling low-dimensional structure in models that emerges in pretraining and flows through post-training to superintelligence. The…

Zac Boring 3 months ago Research

Position: Evaluation Scores Are Perishable Knowledge Claims

via ArXiv cs.AI [5] — Evaluation methodologies for language models increasingly combine multiple signals, from automated metrics and LLM-as-judge ratings to human assessments and benchmark suite results. When these signals are aggregated via averaging, evaluation confidence can…

Zac Boring 3 months ago Research

Imprecise beliefs: a tiny introduction

via Alignment Forum [999] — Richard Ngo challenged me to set a time box and write down as many of the most important features of my formal epistemology as I can in one sitting. Here goes.Where probability distributions fail......to express beliefsThere is no probability…

Zac Boring 3 months ago Research

Value Generalisation 3: Pre-aligned AIs

via Alignment Forum [999] — When we get explicit strong generalisation to work (see the first post on the matter and the second) my dream would be to create pre-aligned generalising AIs.Think about the usual conflict between alignment and capabilities, between doing the right…

Zac Boring 3 months ago Research

Value Generalisation 2: The Missing Hole in AIs’ abilities

via Alignment Forum [999] — A human superpower hidden from even ourselvesI though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had…

Zac Boring 3 months ago Research

Value Generalisation 1: a Research and Deployment Program

via Alignment Forum [999] — I’m looking for people, advice, critiques, and funding to build a research program on value generalisation – the ability of an AI to correctly extend human values and preferences to situations neither it nor we have seen before. My ongoing research…

Zac Boring 3 months ago Research

LLM Scheming Inversely Scales with Pretraining Language Coverage

via ArXiv cs.AI [9] — With the growing capabilities of frontier models, AI alignment becomes increasingly critical in high-risk deployment settings. While recent work has empirically demonstrated in-context scheming -- the covert pursuit of misaligned objectives while feigning…

Zac Boring 3 months ago Research

Research directions in condensation: varieties of objectivity

via Alignment Forum [999] — This is the first part of a survey of various ways that I’d like to see work on the theory of condensation develop. Condensation is a mathematical theory dealing with the organization of descriptions of the world into conceptual parts; some of the…