Hmm, these numbers seem very low. I wonder how these scores were determined.

They weren't, because LLMs don't have reasoning ability, at least not in the way you as a human do. They are generative models, so the short answer is the model most likely made the numbers up, though there's a chance they pulled them directly from some training data that's likely completely unrelated to the user's prompt.

What they generate is supposed to have similar multidimensional correlation as the input data, so there are complex relationships between what the question asked and the output it gave, but these processes don't look anything like the steps you would go through to answer the same question.

"garbage in, garbage out" my beloathed

Not the first time this has happened Google's own AI overviews have misinterpreted u/fucksmith, eaten rocky onions and hallucinated cats on the moon before) but this is probably the worst such incident

Anyways, sidenote time:

Right now, there's no legal precedent determining whether or not "AI overviews" like Google's are protected under Section 230, but between shit like this and the recent lawsuit against character.ai, I suspect there's gonna be plenty of effort to deny them Section 230 protection.

If that happens, I expect it will put an immediate end to public-facing autoplag like this, as such products immediately become legal timebombs waiting to go off. I suspect it will also kill any future attempts at AI for the foreseeable future, for similar reasons.

As for AI as a concept, which I've discussed previously, I expect this incident will help further a public notion of "artificial intelligence" being an oxymoronic concept, and of intelligence being something that either cannot be replicated by artificial means, or something which should not be replicated by artificial means.

I don't understand the title. LLM hallucinations have nothing to do with JAQing off.