we accomplished all of that with the theoreticals

...No. There were plenty of tests, for example the atomic pile built at the University of Chicago. They certainly didn't just do a bunch of math and then build an industrial nuclear power plant on the first go.

there is criticism from actual physicists over these colliders

I don't think you're making the same criticisms they are (I last read some of those criticisms a while ago, though).