Windows 95 originally didn't support FAT32, limiting partition sizes to 2GB. OSR2 added FAT32 support but I dunno if the installer still properly supports it or not. I would bet the issue is related to that. You could try limiting the partition size to under 2GB and see if that helps.
Any reason not to use Windows 98SE for this? It would very likely not have this issue but games would work just as well.
And yes, when installing any Win 9.x, you want to to always copy the installation files to the hard disk first and run the setup from there. The main reason is that after the first reboot the system might not yet have the correct driver for the CD drive but it still needs access to the installation files to continue.