Most likely you are more focused on the parts you dislike when you take the break than the positives, since you have already experienced them, even if it is subconsciously. Kind of like in your mind you have already done the thing, and the break feels like an end point, so there is no need to start over doing the thing.
You don't get the feeling as long as you keep going because you haven't hit an end point.
I have ADHD and this is not an uncommon feeling for me. When stuff sucks to do I can often power through it fine, but stopping and restarting is extremely difficult to do. When stuff is fun, but exhausting, then it is a very similar feeling to dreading something I don't want to do.
In my case canoeing and other fun outdoor physical activities don't trigger this because the fun outweighs the negatives so I can resume whatever I'm doing. Yard work that is only somewhat fun, hikes without an overall end goal, and other tiring activities can trigger it pretty easily.
Not saying you are neurodivergent, just that the pattern of feeling like you are already done with a thing if you take a break is relatable.