Research - pDoom (Page 10)

Zac Boring 5 months ago Research

Hidden Coalitions in Multi-Agent AI: A Spectral Diagnostic from Internal Representations

via ArXiv cs.AI [5] — Collections of interacting AI agents can form coalitions, creating emergent group-level organization that is critical for AI safety and alignment. However, observing agent behavior alone is often insufficient to distinguish genuine informational coupling…

Zac Boring 5 months ago Research

Clarifying the role of the behavioral selection model

via Alignment Forum [999] — This is a brief elaboration on The behavioral selection model for predicting AI motivations, based on some feedback and thoughts I’ve had since publishing. Written quickly in a personal capacity.The main focus of this post is clarifying the basic…

Zac Boring 5 months ago Research

Intelligent CCTV for Urban Design: AI-Based Analysis of Soft Infrastructure at Intersections

via ArXiv cs.AI [4] — Artificial intelligence (AI) and computer vision are transforming transportation data collection. This study introduces an AI-enabled analytics framework leveraging existing CCTV infrastructure to evaluate the impact of soft interventions, such as temporary…

Zac Boring 5 months ago Research

Understanding Annotator Safety Policy with Interpretability

via ArXiv cs.AI [3] — Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or…

Zac Boring 5 months ago Research

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

via Alignment Forum [999] — AbstractWe introduce Natural Language Autoencoders (NLAs), an unsupervised method for generating natural language explanations of LLM activations. An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text…

Zac Boring 5 months ago Research

Mechanistic estimation for wide random MLPs

via Alignment Forum [999] — This post covers joint work with Wilson Wu, George Robinson, Mike Winer, Victor Lecomte and Paul Christiano. Thanks to Geoffrey Irving and Jess Riedel for comments on the post. In ARC's latest paper, we study the following problem: given a randomly…

Zac Boring 5 months ago Research

Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?

via ArXiv cs.AI [4] — Modern coding agents increasingly delegate specialized subtasks to subagents, which are smaller, focused agentic loops that handle narrow responsibilities like search, debugging or terminal execution. This architectural pattern keeps the main agent's…

Zac Boring 5 months ago Research

[Linkpost] Interpreting Language Model Parameters

via Alignment Forum [999] — This is the latest work in our Parameter Decomposition agenda. We introduce a new parameter decomposition method, adVersarial Parameter Decomposition (VPD)[1] and decompose the parameters of a small[2] language model with it. VPD greatly improves on…

Zac Boring 5 months ago Research

Motivated reasoning, confirmation bias, and AI risk theory

via Alignment Forum [999] — Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias.- From Scott Alexander's review of Julia Galef's The Scout Mindset.…

Zac Boring 5 months ago Research

Understanding Emergent Misalignment via Feature Superposition Geometry

via ArXiv cs.AI [6] — Emergent misalignment, where fine-tuning on narrow, non-harmful tasks induces harmful behaviors, poses a key challenge for AI safety in LLMs. Despite growing empirical evidence, its underlying mechanism remains unclear. To uncover the reason behind this…

Zac Boring 5 months ago Research

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

via ArXiv cs.AI [9] — Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and…

Zac Boring 5 months ago Research

Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models

via ArXiv cs.AI [5] — Safety trained large language models (LLMs) can often be induced to answer harmful requests through jailbreak prompts. Because we lack a robust understanding of why LLMs are susceptible to jailbreaks, future frontier models operating more autonomously in…

Zac Boring 6 months ago Research

Exploration Hacking: Can LLMs Learn to Resist RL Training?

via Alignment Forum [999] — We empirically investigate exploration hacking (EH) — where models strategically alter their exploration to resist RL training — by creating model organisms that resist capability elicitation, evaluating countermeasures, and auditing frontier models…

Zac Boring 6 months ago Research

Risk from fitness-seeking AIs: mechanisms and mitigations

via Alignment Forum [999] — Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call…

Zac Boring 6 months ago Research

Binary Spiking Neural Networks as Causal Models

via ArXiv cs.AI [4] — We provide a causal analysis of Binary Spiking Neural Networks (BSNNs) to explain their behavior. We formally define a BSNN and represent its spiking activity as a binary causal model. Thanks to this causal representation, we are able to explain the output…

Zac Boring 6 months ago Research

Distill-Belief: Closed-Loop Inverse Source Localization and Characterization in Physical Fields

via ArXiv cs.AI [3] — {Closed-loop inverse source localization and characterization (ISLC) requires a mobile agent to select measurements that localize sources and infer latent field parameters under strict time constraints.} {The core challenge lies in the belief-space…

Zac Boring 6 months ago Research

Research Sabotage in ML Codebases

via Alignment Forum [999] — One of the main hopes for AI safety is using AIs to automate AI safety research. However, if models are misaligned, then they may sabotage the safety research. For example, misaligned AIs may try to:Perform sloppy research in order to slow down the…

Zac Boring 6 months ago Research

Sparse Personalized Text Generation with Multi-Trajectory Reasoning

via ArXiv cs.AI [6] — As Large Language Models (LLMs) advance, personalization has become a key mechanism for tailoring outputs to individual user needs. However, most existing methods rely heavily on dense interaction histories, making them ineffective in cold-start scenarios…

Zac Boring 6 months ago Research

Recursive forecasting: Eliciting long-term forecasts from myopic fitness-seekers

via Alignment Forum [999] — We’d like to use powerful AIs to answer questions that may take a long time to resolve. But if a model only cares about performing well in ways that are verifiable shortly after answering (e.g., a myopic fitness seeker), it may be difficult to get…

Zac Boring 6 months ago Research

Towards Causally Interpretable Wi-Fi CSI-Based Human Activity Recognition with Discrete Latent Compression and LTL Rule Extraction

via ArXiv cs.AI [3] — We address Human Activity Recognition (HAR) utilizing Wi-Fi Channel State Information (CSI) under the joint requirements of causal interpretability, symbolic controllability, and direct operation on high-dimensional raw signals. Deep neural models achieve…