Your knock on the door analogy is exactly right--when I started my instance, I had to search every community that I wanted to see directly by URL. Then my server would send a message to that community's server saying that I subscribed to that community. Now, every time a post is made at that community, it's server sends my server an update. If I post a comment to a community on lemmy.ca (like I am now), from my kbin instance (remy.city), and you are reading it from kbin.social, that means my server first saved my comment locally, then sent it to lemmy.ca, and lemmy.ca sent it to your kbin.social because you subscribed to the community. So in that case, lemmy.ca is the 'authority', and is responsible for sending updates out to subscribed parties.

There is no such thing for instances--each new instance has to manually make a connection to another (i.e. a user on the new instance must subscribe to something from another instance). I think the tools like fediverse.observer are reading comments or other activity from popular instances, and are then compiling a list of the instances they find by doing that. But there is no central server/authority that makes communication between instances possible. Each instance has to talk to each other instance for it to happen. It's a bit inefficient but is necessary for decentralized communication.