Its been a while since I've seen the movies, and I've never read the books, but here's a wiki article that explains it a bit.
It's basically summed up as an unpredictable instinct in young children. I'd guess that maybe that's lost as they get older and can think more concretely. Probably adding wand foci and being drilled in using then so much in Hogwarts also diminishes that innate ability. It seems that as you become more dependent on foci, the more difficult and unpredictable wandless magic becomes.