There was an off-hand joke in the most recent episode about this, and if I remember correctly, there have been similar jokes before.

Prof. David Kaiser, an MIT professor who is both a physicist and a historian (aka the coolest guy possible) has done extensive research on this, and his work is particularly interesting because he has the expertise in all the relevant fields do dig through the archives.

It's been a long time since I've read him, but he concludes that the physics was widely known outside of secret government operations, and the fundamental challenges to building an atomic bomb are engineering challenges -- things like refining uranium or whatever. In other words, knowing that atoms have energy inside them which will be released if it is split was widely known, and it's a very, very, very long path engineering project from there to a bomb.

This cultural understanding that physicists working for the Manhattan project built the bomb is actually precisely because the engineering effort was so big and so difficult, but the physics was already so widely known internationally, that the government didn't redact the physics part of the story. In other words, because people only read about physicists' contributions to the bomb, and the government kept secret everything about the much larger engineering and manufacturing effort, we are left with this impression that a handful of basic scientists were the main, driving force.