Your average American hasn't attended for a couple of reasons:

a) It's SUPER expensive. Not just to go, but if you notice, wherever the Superbowl is held, it's in a neutral stadium, which means fans of BOTH teams have to travel to get there. Which means planes, hotels, rental cars, etc. etc. etc. beyond the extraordinary cost of tickets themselves.

b) It's super hard to plan for. This year, the Superbowl will be held in New Orleans in 59 days, which means right now is the prime time to book plane tickets, hotels, etc. But who's going to be playing? Is it going to be "your team"? Nobody knows, because the playoffs don't even start until January 11th, and the two teams going to the bowl won't be known until January 26th, just 13 days before the Superbowl itself.

Which means fans have just 13 days to figure out how to get there to support their team, hope they can get tickets, hotels, etc. Short term travel booking like that is super expensive (see point a) and beyond the reach of average Americans.

c) It's actually a better experience at home. Unless you LIKE being in a stadium with 76,467 other fans, paying premium prices for snacks, and dealing with the traffic nightmare getting to and from the stadium, it's better to sit at home, in your jammies, in front of a 65" television, watching all the plays in slow-mo 4K detail, with color commentary telling you what to think about it.