Realistically no one is cracking my super long randomized password either. Seems fine to leave it on as backup login.

Then yeah idk that makes no sense to me. The only time I had problems with Jellyfin chromecast was when I wasn't on https.

You need to be hosting jellyfin behind a reverse proxy so that it works through https. Chromecast doesn't support unsecured sources.