I think that's going to be a bit of a tall order to get a really good quality small form-factor speaker with a good quality mp3 player as an all-in-one unit.

Am I recalling correctly that you do fieldwork? If so then you might be interested in some of the ruggedized Ulefone offerings – aside from being designed for construction sites etc. and often being waterproof with a larger battery, they tend to boast a much more substantial speaker than any other phone I've come across. But they come at a premium as they are a mobile phone so maybe it's something to put on the horizon as a purchase down the track.

Otherwise my ideal setup would be a Hiby R2 or R3 or a Shanling M0 + a small bluetooth speaker, maybe a small Soundcore or Tribit unit.

If one part of that setup dies then you can replace it, plus there's more options available because independent bluetooth speakers will work with anything.

Also it depends on what size pockets we're talking here. All of this is going to be way above the budget in your post tho.