The base assumption of those with that argument is that an AI is incapable of being original, so it is "stealing" anything it is trained on. The problem with that logic is that's exactly how humans work - everything they say or do is derivative from their experiences. We combine pieces of information from different sources, and connect them in a way that is original - at least from our perspective. And not surprisingly, that's what we've programmed AI to do.

Yes, AI can produce copyright violations. They should be programmed not to. They should cite their sources when appropriate. AI needs to "learn" the same lessons we learned about not copy-pasting Wikipedia into a term paper.