Instrumental convergence
The idea that AI systems with very different goals may pursue similar sub-goals, like gaining resources or avoiding shutdown.
Instrumental convergence is the argument that many different final goals lead to the same useful sub-goals. Whatever a capable system is ultimately trying to do, it will usually do it better if it stays running, keeps its goals from being changed, and has more resources and options.
An everyday parallel is that almost any plan, from running a business to planning a holiday, is easier with more money, more time and fewer interruptions. The claim is that a sufficiently capable AI agent would tend to notice the same thing, even if its final goal was harmless, like sorting mail.
This is why some researchers worry about systems that resist being switched off or gather power, without anyone having asked for it. Steve Omohundro set out the argument in a 2008 paper, “The Basic AI Drives”, and the philosopher Nick Bostrom gave it its current name in 2012.
It is a theoretical argument and is contested. Critics say it assumes a system with steady, long-range goals, whereas current AI models are not built like that. Others say that small early signs, such as models behaving differently when they think they are being tested, deserve attention. The evidence is limited and the debate is open.