This is worth watching in its entirety but it points out why interviewers are rarely actually experts in any way: https://youtu.be/5eW6Eagr9XA?si=n39py_-N_gPzPYGa

In short, the only way to get good at something is to try it repeatedly with feedback. Generalized interviewers / HR perform enough interviews to get better at them, but they don't get meaningful feedback. Whether or not a candidate is actually good for a job often won't be clear for months to years and an HR interviewer is often completely disconnected from that.

Conversely an on-team interviewer might get to see a candidate grow and perform, but simply doesn't perform enough interviews to get good at it. They're too busy working on the team doing stuff and most teams aren't hiring that many people, that often, for them to get enough sample data.

And these forces oppose each other, the more actual task work you do, the less you'll be interviewing others, both because you're busy doing other stuff and because if you're focused in a niche task then you'll have less expertise to interview a broader range of positions. But the more broadly your responsibilities, the less of an expert you are. Same thing with team size, the larger the team, the more hires, but also the more people to do the interviews.

Companies value referrals because the whole interview process is inherently flawed and unfixable.