Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
Key takeaway
TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits.
Why it's on PDOOM
PDOOM selected this story for its alignment & control signals: Interpretability.
InterpretabilityFrom LessWrong AI
TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the outcomes of these experiments no better than agents that just read the transcript. Using it as training data, we find that models trained to predict how prompt edits change t
By Adam Karvonen