The object doesn't absorb their mass, but rather their energy (which admittedly can be equated to a mass via a factor of c^2, but that's not actually what's happening). The change in momentum that results from a photon hitting you isn't caused by a change in m, it comes from a change in v. If mass were the quantity being transferred, solar sails wouldn't work to move anything; they would just sit there and get more massive as photons hit them.