The thing that's stopping anything like this is that the AI we have today is not intelligence in any sense of the word, despite the marketing and "journalism" hype to the contrary.

ChatGPT is predictive text on steroids.

Type a word on your mobile phone, then keep tapping the next predicted word and you'll have some sense of what is happening behind the scenes.

The difference between your phone keyboard and ChatGPT? Many billions of dollars and unimaginable amounts of computing power.

It looks real, but there is nothing intelligent about the selection of the next word. It just has much more context to guess the next word and has many more texts to sample from than you or I.

There is no understanding of the text at all, no true or false, right or wrong, none of that.

AI today is Assumed Intelligence

Arthur C Clarke says it best:

"Any sufficiently advanced technology is indistinguishable from magic."

I don't expect this to be solved in my lifetime, and I believe that the current methods of"intelligence " are too energy intensive to be scalable.

That's not to say that machine learning algorithms are useless, there are significant positive and productive tools around, ChatGPT and its Large Language Model siblings not withstanding.

Source: I have 40+ years experience in ICT and have an understanding of how this works behind the scenes.