Research - pDoom (Page 9)

Zac Boring 5 months ago Research

Testing Gemini models for scheming tendencies

via Alignment Forum [999] — As AI models become increasingly capable and autonomous, keeping them safely aligned with human intentions is critical. Extending our previous work on evaluating scheming capabilities, we introduce complementary approaches to test whether AI models…

Zac Boring 5 months ago Research

Advice for making robust-to-training model organisms

via Alignment Forum [999] — We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile:…

Zac Boring 5 months ago Research

Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming

via Alignment Forum [999] — Behavioral evaluations may become worthless, which we think would be a disaster. Smart misaligned models may realize they are being evaluated ("eval awareness") and then act to look good to us so we don't realize they're misaligned ("eval gaming"). We…

Zac Boring 5 months ago Research

Full automation of AI R&D probably yields a large speed up even without a software-only singularity

via Alignment Forum [999] — This is a somewhat technical note. By "software-only singularity", I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing…

Zac Boring 5 months ago Research

Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems

via ArXiv cs.AI [4] — Long-lived AI agents are increasingly deployed as persistent operational systems, yet they are still evaluated like freshly initialized models. Day-one benchmarks miss a basic systems question: how long does an agent remain reliable after deployment? Even…

Zac Boring 5 months ago Research

The Erdős Proof and AI Capabilities

via MIRI [999] — View the official memo here. An internal model at OpenAI has autonomously disproved a central conjecture in discrete geometry, a mathematical field with applications in cryptography, wireless device communication, and medical imaging. The proof relates to a…

Zac Boring 5 months ago Research

The Case for Evaluating Model Behaviors

via Alignment Forum [999] — Most evaluations of AI systems focus on their capabilities: how good they are at coding tasks, how effectively they can answer complex scientific questions, and so on.From a safety perspective, capability evaluations have a place: by understanding how…

Zac Boring 5 months ago Research

AgentWall: A Runtime Safety Layer for Local AI Agents

via ArXiv cs.AI [8] — The safety of autonomous AI agents is increasingly recognized as a critical open problem. As agents transition from passive text generators to active actors capable of executing shell commands, modifying files, calling APIs, and browsing the web, the…

Zac Boring 5 months ago Research

Fast-tracking genetic leads to reverse cellular aging

via DeepMind Blog [4] — Biologists use Co-Scientist to find novel factors that successfully rejuvenate human cells.

Zac Boring 5 months ago Research

Verifiable Agentic Infrastructure: Proof-Derived Authorization for Sovereign AI Systems

via ArXiv cs.AI [3] — Modern cloud and enterprise systems rely on identity-centric authorization, assuming that callers possessing valid credentials are safe to execute commands. The emergence of autonomous AI agents invalidates this assumption: agents can generate syntactically…

Zac Boring 5 months ago Research

SDOF: Taming the Alignment Tax in Multi-Agent Orchestration with State-Constrained Dispatch

via ArXiv cs.AI [3] — Multi-agent orchestration frameworks such as LangChain, LangGraph, and CrewAI route tasks through graph-based pipelines but do not enforce the stage constraints that govern real business processes. We present SDOF, a framework that treats multi-agent…

Zac Boring 5 months ago Research

A Two-Dimensional Framework for AI Agent Design Patterns: Cognitive Function and Execution Topology

via ArXiv cs.AI [3] — Existing frameworks for LLM-based agent architectures describe systems from a single perspective: industry guides (Anthropic, Google, LangChain) focus on execution topology -- how data flows -- while cognitive science surveys focus on cognitive function --…

Zac Boring 5 months ago Research

Risk reports need to address deployment-time spread of misalignment

via Alignment Forum [999] — Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during…

Zac Boring 5 months ago Research

Mechanistic estimation for expectations of random products

via Alignment Forum [999] — We have developed some relatively general methods for mechanistic estimation competitive with sampling by studying problems that are expressible as expectations of random products. This includes several different estimation problems, such as random…

Zac Boring 5 months ago Research

The safe-to-dangerous shift is a fundamental problem for eval realism; but also for measuring awareness

via Alignment Forum [999] — 1) The safe-to-dangerous shift is a fundamental problem for eval realismSuppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A…

Zac Boring 5 months ago Research

Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

via ArXiv cs.AI [6] — Agent benchmarks have become the de facto measure of frontier AI competence, guiding model selection, investment, and deployment. However, reward hacking, where agents maximize a score without performing the intended task, emerges spontaneously in frontier…

Zac Boring 5 months ago Research

RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking

via ArXiv cs.AI [4] — Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction. A key challenge, however, is learning an accurate critic in large state--action spaces with limited dataset coverage.…

Zac Boring 5 months ago Research

Summary: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence

via MIRI [999] — If anyone, anywhere builds a superhuman artificial intelligence using present methods, the most likely outcome is catastrophe. There have accordingly been widespread calls for an international agreement prohibiting the development of superintelligence. In…

Zac Boring 5 months ago Research

Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria

via ArXiv cs.AI [6] — Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing…

Zac Boring 5 months ago Research

Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)

via Alignment Forum [999] — 1.1 Tl;drAlignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last…