“We’ve been made aware that two of our songs were played at the anti-immigration marches over the weekend by members [of] a neo-Nazi group. To be absolutely clear, the goals and philosophies of these marches are abhorrent to us, and we don’t support them in any way,” the band said in a statement.

“The Living End’s worldview is about equality, and we aim for our songs to be a positive influence. We believe in a fair go for everyone. Our songs are written to bring people together, not to divide them, and especially not to be used in this way.

“In summary: Fuck Nazis,” they concluded.

You do raise a good point, unfortunately life is too fast to always avoid generalisations and unfortunate namecalling. The bottom line is:

There most likely were many people who were not there to participate in racism. Simply mislabeling them 'racist' may trivialise their real difficulties and alienate them from real solutions.
Many are actually racists, feigning ignorance.
The rallies, taken as a collective, were largely organised and promoted by a neo-Nazi organisation and their crypto-fascist allies (such as Auspill and Bec Freedom). Open Nazis were front and centre in many rallies including Sydney, Melbourne and Townsville, and attempts to prevent them doing the same at Adelaide ended the event. Most or all other rallies failed to reject them in a meaningful way, regardless of whether any attendees personally took issue. Their involvement was known and argued for by the official FB pages, was reported on by mainstream media including Sky News, and there's really little doubt anyone didn't know Nazis would be there. It was a Nazi-friendly event. The main ones were Nazi-hosted events.
One can argue that knowingly being at a Nazi-run event and marching alongside Nazis doesn't necessarily imply that makes them racist. But it does make them a racist enabler, a racism collaborator, a supporter of racists, an ally of racists. And if that doesn't make them think twice and regret their actions, then for all intents and purposes, they're equivalent to a racist.

All that to say:

Maybe we shouldn't just call them a racist if we're not sure. But we should absolutely tell them they're supporting racists.