I would think it's a liability thing for sure. If you have an spo2 meter you could also check how you do with it? No permanent harm feels likely as we should adapt to respiratory resistance just fine.

Assuming you are wearing a n95 type mask of dome kind, maybe this study could give you some peace of mind? It states that there are changes in prolonged use, but those are likely to resolve with adaptation and none are outside the normal range.

Anecdotally I've personally done weightlifting in a FFP2 mask, it was uncomfortable at first, but I got used to it.

Edit. Here is a systemic review looking spesifically at exercise. I would think that this can be compared to harder labor.

fun fact, here at least firefighters need to pass a fitness test that includes a treadmill run with full gear and a mask. to train for it they run while wearing two N-95s

so it does add some restriction but probably not enough to worry about

Dampness may also be a factor, if you’re sweating into your mask.

My personal experience with exercising in a respirator is that it is perfectly manageable and not a safety issue, but I also have not had Covid that I know of, have an athletic background, and was pretty well-trained up til about 6 months ago which means I've had a lot more leeway. Your experience may vary.

There are some N95 respirators (Gerson pouch respirators) that are specifically designed to have very low breathing resistance, comparable to a surgical mask, while providing real filtration. I think it's due in part to the large volume. You could get a sample pack and try doing some heavy exercises (bike strenuously, do some sprints, lift some weights) with and without it on and with and without your preferred mask to see.

I think I can answer more questions if you provide more information about the mask type.

Is it disposable or elastomeric? What filtration type? N95, N99, KN95, KN100? How well does it fit? Do you find it getting humid or difficult to breathe in after a while? Does it have an exhalation valve?

In terms of the physics of what's happening in a mask go, masks do not really store CO2 or limit the flow of air, because the pore sizes are humongous compared to the size of these molecules. However, they do trap water vapor and the humidity can impact your lung's gas exchange efficacy. When I use my elastomeric mask for long periods, I have to periodically go outside, take it off, and shake condensed water droplets out. This buildup of humidity can be mitigated by getting a mask with an exhalation valve -- while that doesn't protect others and so was frowned upon earlier in the pandemic, at this point since most people aren't wearing a mask it doesn't matter as much.