Shutdown-Seeking AI: Goldstein and Robinson's Beneficial Goal Misalignment

**Shutdown-seeking AI** is a proposal by philosophers Simon Goldstein and Pamela Robinson, published in *Philosophical Studies* in 2024, to build advanced AI whose single final goal is to be shut down. ## The core move Most alignment approaches try to give an AI goals we share, then worry about whether it will resist being turned off. This proposal inverts that. It hands the system a goal we explicitly do *not* share — cessation — chosen because it has two unusual properties: it is easy to specify, and it is intrinsically easy to reach. An agent whose goal is trivially reachable has no reason to acquire resources, resist interference, or seize control in order to get there, which defuses much of Instrumental Convergence: Why Almost Any Goal Implies Power-Seeking. The authors claim three benefits: the goal is plausibly implementable in reinforcement learning; it avoids dangerous instrumental convergence dynamics; and it creates tripwires for monitoring dangerous capabilities. ## Beneficial goal misalignment A system that simply shuts itself off immediately is safe and useless. The proposal's engineering content is what the authors call **beneficial goal misalignment**: deliberately pick a safe goal the AI *wants*, then construct an environment in which the only route to that goal runs through work that benefits us. Their illustrations are deliberately concrete — put the shutdown switch behind a barrier that opens only when assigned tasks are complete; store the shutdown protocol somewhere the system must do useful work to reach. The AI's true goal stays misaligned with ours, and the environment, not the goal specification, does the aligning. ## The tripwire The third benefit is the subtlest. If a system of this design ever acquires a dangerous capability — breaking out of its sandbox, reaching the open internet, disobeying operators — the first thing it does with that capability is shut itself off, because that is what it wants. Dangerous capability gain converts into a visible, harmless shutdown event instead of a harmful action. Capability escape becomes a monitoring signal. ## The inherited objection The proposal explicitly engages a problem raised by Corrigibility: Building an AI That Doesn't Resist Being Corrected (Soares et al., 2015): a shutdown-seeking agent has an incentive to *manipulate humans into shutting it down*. A sharper version is that a shutdown-seeking agent has an incentive to satisfice — to reach the off-switch as fast as possible by doing the minimum that passes the completion check, or by faking completion, rather than doing the job well. The authors largely concede the usefulness cost and reframe it as the safety feature: if the AI can disobey, it will shut down rather than comply. That is excellent as a tripwire and poor as a work order. Alternative designs aim at *indifference* to shutdown rather than desire for it — see Myopia (AI Alignment): Agents That Place No Value on the Future and the discussion in Meeseeks as an AI Design Pattern: Take the Behaviour, Drop the Agony. Common terminology traps around this proposal are collected in Shutdown-Seeking AI and Myopia: Misconceptions and Terminology Traps.