You're right, Signal is not P2P. The way Signals messaging pipeline works is like this - note I'm oversimplifying it for accessibility.

Sending a message to Bob

1. You press Send.
2. The message is encrypted on your device with a key that can only be unlocked by Bob.
3. The message is then "sealed" so that there's only a "deliver to" field visible (not a "from").
4. The "deliver to" field is addressed with a hashed/salted label for Bob - this means Signal's server can see its a unique user, but not what their name is.
5. The message is finally sent to Signal's servers.
6. Your message sits on Signals servers until it can be delivered to the intended recipient.

you can’t really do user lookups without some sort of middleware in the cloud.

See their blog post about Private Contact Discovery, they've spent a long time figuring out how to engineer a method to know as little as possible about you.