OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows
Key takeaway
Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained.
Why it's on PDOOM
PDOOM selected this story for its capabilities signals: Autonomous Ai.
Autonomous AiFrom ArXiv cs.AI
Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose failure modes, or distinguish systematic reasoning from fortunate guessing. We present \textbf{OpenDiscoveryTrace}, a public dataset of 558 complete AI scientific agent trajectories that capture
By Aayam Bansal, Keertan Balaji