Ultimately ChatGPT is a text generator. It doesn't understand what its writing, it's just observed enough humans' writing that it can generate similar text that closely matches it. Which is why if you ask ChatGPT for information that doesn't exist, it will generate convincing lies. It doesn't know it's lying - it's doing its job of generating the text you wanted. Was it close enough, boss?

As long as humans talk about a topic, generative AI can mimic their commentary. That includes love, empathy, poetry, etc. Writing text can never be an answer for captcha; it would need to be something that can't be put in a dataset - even a timestamped photo can be spoofed with the likes of thispersondoesnotexist.com.

The only things AI/bots currently won't do are whatever's deliberately disabled on the source AI for legal reasons (since almost nobody is writing their own AI models), but I doubt you want a captcha where the user lists every slur they can think of, or bomb recipes.