You don't grow during your downtime, you grow while you're scrambling around in some hole half-dead. Your changes are significant, and they happen in-game, not as an after effect of finishing the night's fun. As a general rule, if I describe some of the ways your character has grown since their creation (more HP, scars, oddities) you should be able to tell me the specific event that caused that change, not just see the change as a general reward for actions.