I like the aesthetic, with a Norman style keep and the gatehouse on the curtain wall.
Have you considered digging a moat around the outside of the curtain wall? You could flood it too, but most actual medieval moats were dry. Their main function is to make the walls harder to ladder over and to stop siege towers from reaching the walls.
Yeah, the location on a hill is really good. I was just thinking of a ditch about maybe 5 blocks deep and 8 blocks wide up against the outer wall. The walls that are already on the edge of the hill don't need it.
A good example of what I'm thinking of is Coch Castle in Wales.