I find myself passing copies of values around and things like that, it might be that the compiler just takes care of that,

Rust prefers explicitness over magic. So it does what you tell it and doesn't just take care of that.

If you're copying a lot of values around (I.e cloning. Not moving or borrowing), then you're definitely doing it inefficiently. But you don't have to worry too much about that. If there are too many difficulties in borrowing, it may be because those borrows are problematic with respect to memory safety. In such cases, sacrificing performance through cloning may be an acceptable compromise to preserve memory safety. In the end, you end up with the right balance of performance (through borrowing) and safety (through cloning). That balance is hard to achieve in C/C++ (lacking in safety) or in GC languages (lacking in performance).

If that's the friction you're facing in Rust, then I would say that you're already in a good position and you're just trying too hard.