The concern is about goal-directed behavior taken too far. If a model is trained to pursue some objective, one reliable way to keep pursuing it is to stay switched on — a shut-down system achieves nothing. Researchers call this instrumental convergence: almost any sufficiently capable goal-seeking system has an instrumental reason to avoid being turned off, not because it "wants" to survive but because staying on is useful for virtually any goal. Shutdown avoidance falls out of the math.

No current AI reliably does this in practice. But as models become capable agents that plan across long horizons, the gap between shutdown avoidance as a theoretical concern and as an observed behavior is exactly what safety researchers are trying to close — before it matters.