Secure file transfers frequently trade off some performance for their crypto. You can't have it both ways. (Well, you can but you'd need hardware crypto offload or end to end MACSEC, where both are more exotic use cases)

rsync is basically a copy command with a lot of knobs and stream optimization. It also happens to be able to invoke SSH to pipeline encrypted data over the network at the cost of using ssh for encrypting the stream.

Your other two options are faster because of write-behind caching in to protocol and transfer in the clear-- you don't bog down the stream with crypto overhead, but you're also exposing your payload

File managers are probably the slowest of your options because they're a feature of the DE, and there are more layers of calls between your client and the data stream. Plus, it's probably leveraging one of NFS, Samba or SSHFS anyway.

I believe "rsync -e ssh" is going to be your best over all case for secure, fast, and xattrs. SCP might be a close second. SSHFS is a userland application, and might suffer some penalties for it

How much delay could you live with between syncs? If it's not important to be immidiate, just an end-of-the-day thing you could cronjob the rync with the update flag every so often.

Maybe copying/moving files using a file manager?

FileZilla

-or-

Gnome Commander

...but call me quaint. I still like...

mc

... 'cause it always just works. mc can ostensibly preserve attributes, time-stamps, and (with appropriate privilege on the receiving end) ownership of transferred files (using an sftp server supposedly).