Pact of the chain warlock player here. It's hard to make it fun for everyone.

As a player, my fantasy is having my invisible or spider shaped familiar go in and get us intelligence so we can get the jump on stuff. I want to know there's four bandits eating lunch and one on watch so we can make an informed plan.

It is extremely not fun when the DM has characters stomp on my rat specifically. Or when they just happen to hear an invisible creature and just happen to decide to shoot it. Nevermind that there are rats everywhere, and an invisible flying creature is just as likely to be something Really Dangerous as a familiar. It's annoying metagaming.

The dnd rules are also kind of bad about invisibility. Shooting a flying invisible creature that's out of range 300' away is the same disadvantage as shooting a visible flying creature that's 300' away, or an invisible one that's 30' away. It is extremely annoying when the bandit decides he hears something funny up above so he decides to shoot it. The tiny invisible thing. And then rolls two tens and kills it.

When the party has someone with something like pact of the chain, the dm needs to move away from the surprise as tension. You can generate tension in many other ways. Time sensitivity. Or just forcing the players to make a hard decision. Cross the troll bridge or go under through the crocodile River. Knowing the troll is there ahead of time isn't that big a deal.

The player needs to not try to take all the spotlight. Don't try to pull a "I poison all of their food with my familiar". Dm needs to speed up the scouting part.

Also hidden traps kind of suck. Familiar won't find most of them, but probably don't use them. Stuff the players make decisions about are more interesting