Like, Stable Diffusion, a tool used to generate images, would fall under this. It's very questionable that it, however, would be terribly useful in doing anything dangerous.

My concern is how short a hop it is from this to "won't someone please think of the children?" And then someone uses Stable Diffusion to create a baby in a sexy pose and it's all down in flames. IMO that sort of thing happens enough that pushing back against "gateway" legislation is reasonable.

California putting a restriction like that in place, absent some kind of global restriction, won't stop development of models.

I'd be concerned about its impact on the deployment of models too. Companies are not going to want to write software that they can't sell in California, or that might get them sued if someone takes it into California despite it not being sold there. Silicon Valley is in California, this isn't like it's Montana banning it.