I don't have specific troubleshooting advice to give, but I have been using an Nvidia Shield Pro as my main streaming device for about 4 years now. It is overpriced for what it is, but it has been rock solid for streaming via both Jellyfin and Plex.

The official Jellyfin app is definitely not as polished as Plex, but it has consistently worked for me on the Shield.

I know people knock it, but Plex has worked without issue for me for years. They have apps dedicated to every type of media for easy browsing. It’s so simple my mom can use it without help.

If it ain’t broke, don’t fix it.

You may want check out Infuse for the AppleTV. I have found it fixed every audio drift and video jitter concern that I’ve ever had with Plex or Jellyfin.

You can point it either directly at an SMB share, or a library hosted on Jellyfin or Plex. The advantage of this is it caches the artwork in the library, not on the AppleTV, because the AppleTV will periodically flush its local cache, leading to long re-fetching times and waiting to watch things.

I have no recommendations for the Chromecast.

My solution to that issue is to share my media over NFS with a raspberry pi connected to the TV, the pi runs OSMC (https://osmc.tv/) and has the media direct. you can control OSMC on the pi with your TV remote

Have you tried non official jellyfin clients? I had issues with the official app on AppleTV as well, so I switched to Infuse. The free app supports most things and I think the paid gives access to additional codecs if I recall correctly. The pro upgrade isn’t too expensive and the app is sleek across Apple devices. Only issue I’ve really had is sometimes if a video is not played to the end (as in, stopped during credits) it still shows up in the continue watching, but that might be a problem on my end.

I think MXPlayer is similar on Android which might be worth checking out.

Unless you have a commitment to only using open source software, I'd recommend Plex over Jellyfin. Mostly because I've found the client software for Jellyfin to be lacking, especially on AppleTV.

For the issues with the GoogleTV, you mention that it's on WiFi, would it be possible to use a wired connection or get another set top box for it? Some TVs have the WiFi antenna behind the screen causing interference, so even though other devices get a strong signal the TV doesn't.

Also, how's the hardware on your server? Is the CPU powerful enough or do you have a GPU for transcoding? Also, is the server on WiFi or wired?

It's worth noting that a lot of settop boxes have limited codec support, which might be forcing transcoding even if everything should otherwise support direct play.

I am using the "Chromecast with Google TV" connected to wifi via 5ghz and use mkv files from collection via Jellyfin. I used to transcode from the i7-8700u but larger files took too long to start playing (with quicksync and proper drivers and passthrough) and tried turning off transcode and surprising works fine without issues. Ive tried 15gb files without issues.

So i would suggest checking your set up to make sure not something configured wrong and if you use mkv files try turning off all transcode to see if it works better.

Since you are against using plex even with using it only locally I would suggest you look into the Jellyfin clients support. Specifically look at each client device documentation and take a look at what audio and video codex they natively support. If there is one they both support then I suggest you do some testing and stream a movie or show that is encoded with that specific format. This will let you know if it will work across both clients whether it’s a stream issue hardware issue or Wi-Fi issue.

This will prevent transcoding that can cause audio sync issues.

Edit - if that works then I suggest you convert you library to that format.

Does it need to use the remote? I have a Chromecast (the casting device, not the app running device) and I use the Jellyfin app on my phone and I cast to the Chromecast. Works fine for me.

I stream from Jellyfin to an Apple TV without your issues. It might be a network or configuration problem.

Have you tried Infuse, SenPlayer or other clients?

I use Emby. It’s similar to Jellyfin, but the Apps get a little more attention to detail. Worth a try, and if you don’t need gpu transcoding you don’t need to pay.

But, if I was still using an Apple TV, I would use Infuse.

The jellyfin problem is probably it downmixing surround 7.1 really shittily. You can run tdarr to downmix surround into stereo before you watch. Kind of a pain but it's the only workaroumd I've got for the shit audio