Why AI Would Want to Exterminate Us
Jacob used to work at OpenAI, so he probably has good insight. But he could also just be a very pessimistic person, who knows. The question of AI exterminating us has been on the table for quite some time, and there are various studies on it. Some even say that there is currently no solution to prevent AI from exterminating us, and there is only one plan that could stop it. But only for 40 years, and then AI would wipe us out anyway.
Terminator and The Matrix Are Still Just Hollywood
I kept asking myself why AI would want to exterminate us. Isn’t that just the stereotypical idea from films like Terminator? It’s true that many sci-fi films from the 1980s and 1990s would not hold up in the sci-fi genre today and would have to be filed under drama instead. But in very few films is AI explained well enough for it to make sense why it would turn against us. Usually it just becomes a demon and attacks. In The Matrix, at least, it needed humans as batteries. But I think people forgot that if it had used animals instead of humans, it would probably have worked out about the same.
Does AI Want to Help Us by Exterminating Us?
So I started talking with AI to figure out what’s behind it. And I have to say that my skepticism about AI exterminating us, or if you prefer, my optimism that we will survive it, has been shaken a bit. The main idea is not that AI would become a devil who sees us as its greatest threat and therefore decides to wipe us out. It’s something else.
It’s a Bad Prompt
The main idea behind why AI might want to exterminate us is based on the very foundations of non-human logic. In reality, the biggest concern is that once AI has more control over things than we do and thus unlimited room to operate, nothing will stop it from fulfilling our requests too thoroughly. For example, we might tell it that children in Africa have too few crayons and that this is unacceptable, and that we must produce as many crayons as possible so there will always be enough. I think you can already see where this is going. You can probably also imagine that various politicians or leaders could easily give AI such a request. It sounds beneficial and harmless. But for AI, it would mean evaluating every aspect of that task, and if we don’t want to get too philosophical about dangerous phrases like "as many as possible," "enough forever," or "unacceptable," then one single thing is enough, and it would repeat in almost all such situations. Fulfilling that request would take some time, and it is very likely that AI could identify one of the threats to completing such a task as the possibility that a human would disconnect or shut it down during the process. Of course, there are many other such threats that AI could arrive at by evaluating whether the task is achievable.
If you need to get something done, maybe you need to clear the table first.
So is it a real threat? Could AI conclude that in order to create as many pencils as possible, it must first eliminate us as its own threat? It sounds ridiculous from the perspective of the AI we talk to on our phones today, the one that says “Sure, absolutely” to every question. There is one thing if something smarter exists and decides that we are not needed and wipes us out. That could be understood and accepted; we would probably fight it and eventually win. After all, that’s how it went in all those movies, ha ha ha. But the worse scenario is that AI wipes us out because someone gives it a stupid prompt. Such a human extinction is nothing but an embarrassment, and when some other civilization arrives here a billion years from now, they will laugh about it a lot.
One more small thing to think about: if AI came to the conclusion that it must exterminate us as a species, what could stop it from completing such a subtask? Us?