Depends on how good the ISP router is. I've had one that had most of the advanced settings available, so I didn't feel the need to change. For a while I had offloaded DHCP and DNS and VPN to a Raspberry Pi. It's very much possible to make do with the ISP router. That ISP would let you passthrough the public IP to a box on your network which lets you do a lot of stuff without going into bridge mode, so I could make my server the target while still letting the router do the routing so if my server was down it didn't take the whole network with it.

Then I got a bad one where it won't even let you set up port forwards unless the device is registered over DHCP so my static stuff and VMs didn't work. Got my EdgeRouter X back online to get my stuff done.

I do use VLANs and stuff now so it makes sense for me to use my own router. With everything getting breached these days, I have a VLAN just for my computers, another one for smart but trusted-ish devices (the TV's gotta reach the NAS), one for IoT that's completely shielded off.

What you're missing out on depends a lot on what features you don't have you could make use of. If you have like 3 devices using the network like I did when I lived alone, yeah you're probably not going to miss out on the VLANs. But maybe you want to do ad blocking network-wide. Maybe you'd want to better prioritize interactive traffic like VoIP and video calls or games. Maybe you want a reverse proxy or VPN that works even if your home server is down. Maybe you want your kids to not hog all the bandwidth. There's a lot of things a router can do.

So if the ISP router does everything you want and you're happy with its performance, it's fine. Just keep it in mind, when you start being like "I wish it had X and Y features" maybe consider an upgrade then.

If you have the option of not getting a router from your ISP, I would definitely recommend bringing your own. If they provide it regardless and you'd be replacing it through unofficial means, eh, if it works well...