That's unusual behaviour for a torrent client. Even on a very popular torrent, I see between 50 and 80 connections.
Make sure you are enforcing encryption and not accepting unencrypted peer connections.
That aside, the hexs (refresh) will be plenty, assuming you have symmetric upload/download. I have an old Hex (no SFP) I use to cordon off my storage network from anything but my servers, and it passes the full 940mbps (no jumbo frames).
You may want to check what is happening with your torrent client with top and iotop. Did you give torrents queue preference or something like that? They should not be interfering with other traffic that aggressively.