I sometimes have to. Not because someone tells me to, but because my physical presence is actually needed.
Sometimes I have to fly in from my middle-of-nowhere home office because whatever system I was troubleshooting wasn't fixable via VPN. And said system can be anywhere in the world, so it doesn't really matter where I live.
So returning to an office based work location? Yeah, that ain't happening..