I was always taught that the towel's final role is to abrade and collect the dirt/grime/skin that has been loosened by showering, but that wasn't washed away (which iirc is mostly just the skin and oily grime, not dirt or other large particulates).

If it works for you then you do you! It's just odd to discover that people think towels are somehow clean after being rubbed all over your body. It's probably fine, the literature I've just dug up seems to indicate that it's not ideal but safe to use the same towel for 1-3 days so long as you're not sharing it, (depending on environment, it seems that they get a "disconcertingly large bacteriological load" (heh) if left in a humid bathroom) but still. ew.