Technology

Alignment with our goals. I'm not talking about biases, those are inherent to the training material.
Alignment refers to how well the network's learned goal aligns with the intended goal.
The underlying issue is that there are several layers of indirection in the process. How well does the training data align with the intended goal? How well does the learned, internal goal align with the goal described by the training data? How well does the goal internal goal with what the network understands or goal to be?

We're trying to teach an intelligence that understands us, maybe better than we assume, but that we don't understand in turn. LLMs are not programmed. They're selected for, so the process is more like growth or evolution, where we can exert some, but not full control.

I'm glossing over a ton of intricacies here. If you're interested I highly recommend Robert Miles' YouTube, where he published several videos on the different types of alignment and/or If Anyone Builds it, Everyone Dies if you prefer books. This isn't just fear mongering or luddites opposing progress. These are people with a deep understanding of all the technicalities who came to the conclusion that this technology (maybe not LLMs specifically, but artificial intelligence in general) has the potential to create an intelligence vastly superior to our own, that employs ways of thinking that might as well be extraterrestrial and whose motives we cannot even fathom. And unless we're absolutely, beyond the shadow of a doubt certain, that its goals perfectly align with our own, we absolutely shouldn't try to create it. Because we're very unlikely to get a second chance if we miss our first shot.