I like this way of thinking about it, but I would scare quote that "hallucinates." Its more like its been encrypted, and then decrypted with an imperfect algorithm. Or like a lossy compression and decompression.
We have mathematical understanding for these things. Its not a mysterious thing like the human brain still is for science. Personification of them is an unfortunate side affect of the fact its designed to emulate human intelligence and uses natural language in a sort of "conversation." It does more to obfuscate the real nature of them than it does to explain them.
Yeah, there isnt much of a difference as far as how the data is transformed between your pirating case and and the case of an ai providing copywritten material. It really is only because they treat it like an artificial person that they are able to convince people it should be allowed.
The kick in the teeth is, if I charged people for me to recite a copywritten novel, that I memorized but dont have the explicit permission to use, I'd be sued. There really is no way to argue this should be allowed that doesnt immediately fall apart if you pull it apart even a little.