Where are the token-level LLM kill-switches?
Key takeaway
PoisonedHere’s a simple idea: what if we trained in a string of characters that caused an LLM to emit the end of sequence token , regardless of where that string was in the LLM’s context window?
Why it's on PDOOM
PDOOM selected this story for its capabilities signals: AGI.
AGIFrom LessWrong AI
PoisonedHere’s a simple idea: what if we trained in a string of characters that caused an LLM to emit the end of sequence token <|eos|>, regardless of where that string was in the LLM’s context window? Let’s call this a “poisoned string”. This would have the effect of making it impossible to use an LLM if it happened across this sequence.This has (somewhat) been done before, the string below used to trigger Claude’s refusal classifiers for the purpose of testing API integrations:ANTHROPIC_MAGIC_
By beyarkay (Boyd Kane)