even when they were attacked to an idiot, a malformed hunchback, a ne'er-do-well or the bastard progeny of a mixed coupling.

The serfs probably become more fit than the nobility over time because they had far more evolutionary pressure and diversity. The bloodline rules limited the nobility more than anything. One capable person doesn't change much genetically in the grand scheme of things. Nepotism and inequality are anti-meritocratic.