You're not alone. After my dad left my mom, he and his new wife became fairly wealthy, they're probably millionaires now. While this happened his kids and ex squatted in their old home for a few years until they were kicked out and then moved into a leaky house that barely had heat in New England. I never got shit from him in adulthood besides being able to stay on my step-mother's health insurance which he would get pissed at me every time I used. Though I've been lucky to never be in a bad financial spot, I still worry constantly, sometimes to the point of panic, about money because of how I grew up and he's done irreparable harm to me and the people I love by hoarding wealth.

Some parents hate their kids. Some just have no empathy for them. Either way I think it's quite reasonable to be upset about it and define your relationship around it. It gets easier to deal with over time.