Begrudgingly Yeast (@begrudginglyyeast.bsky.social) on bsky informed me that I should read this short story called 'Death and the Gorgon' by Greg Egan as he has a good handle on the subjects/subjects we talk about. We have talked about Greg before on Reddit.
I was glad I did, so going to suggest that more people he do it. The only complaint you can have is that it gives no real 'steelman' airtime to the subjects/subjects it is being negative about. But well, he doesn't have to, he isn't the guardian. Anyway, not going to spoil it, best to just give it a read.
And if you are wondering, did the lesswrongers also read it? Of course: https://www.lesswrong.com/posts/hx5EkHFH5hGzngZDs/comment-on-death-and-the-gorgon (Warning, spoilers for the story)
(Note im not sure this pdf was intended to be public, I did find it on google, but might not be meant to be accessible this way).
They just don't call it model collapse, and that fixes the problem. (Somebody on bsky said 'hey that isn't actually model collapse, read this article on it written by the AI company'. And then the article described model collapse, but didn't call it model collapse, it was amusing. (And I went into it, thinking I could be wrong, going 'ah so it is only related to a specific set of instructions', but then the article also showed that models without those instructions had the the problem (in a much smaller way, but the bad instructions were spreading due to synthetic data))).