There's an odd trend of labeling everyone with even the slightest advantage a, "nepo baby".

Nepotism is when you give friends or relatives special consideration for jobs or positions. As far as I know the only job Buffet ever had from a relative was working in his grandfather's grocery store. The closets I could find for Elon Musk was that he started one of his companies with his brother.

Elon's father was an engineer. That certainly put him in a comfortable position, particularly as a white engineer in South Africa but it definitely doesn't get you recognition from old money families. Buffet went to public school.

They both had advantages growing up but if we expand nepotism to include people like that, it becomes a pretty meaningless term.