Don't apologise, it's a phrase people use and you weren't being racist. Euphemism treadmill is slippery by definition, if we're all mates there's no need to get too into language policing.

No need to apologize, I happened to have said it a year or so ago and then had a...wait a minute, down the Internet rabbit hole and here we are. The purpose of mentioning it is that it wasn't originally or historically ever offensive and probably outside of the US wouldn't raise an eyebrow today, but since I do live in the US when I read more about it I committed that one to memory.

I recently had a phrase come to mind a family member used to say as an intensifying adjective, such as "God damned" in "call the God damned police!", but the intensifier my family member said growing up was "cotton picking" as "get the cotton picking car out of the way!" I'm from pacNW US and only decades later had my brain tripwire catch on that it was clearly derogatory towards either slaves or at best those who did low-paid agricultural work when my family member was growing up. How that phrase made it into his family's vocabulary is a wonder in the PacNW with no agricultural ties nor family history anywhere near the US south where you might expect it to be more common. Point is, we all have things that we absorb from the language around us, I just try to generally be cognizant of roots and context but it's difficult even when you try sometimes!