People who train neural networks do not write a bunch of tokens and weights.

Reading this made me think of an analogy of generated code. This is basically exactly the same thing as distributing the code of your program but not in the source language, rather the assembly listing of the final binary, and calling it open source. You can turn any defense of the AI model of "open-source" into a defense of that model of distributing code. You can run my AI/code (if you have a powerful/similar enough machine), you can inspect it (it's just not going to tell you anything), you can modify it (lol), so it's open source!

Edit: The more I think about it the more I come to the realisation that the assembly listing is actually still vastly more useful than the AI models. Like at least a very dedicated and insane enough programmer could technically track down a bug in the assembly and correct it if given enough coffee.