That's because the human body does not have a system to detect how much oxygen is in the blood, it only detects how much carbon dioxide is in the blood. When you hold your breath and feel the need to breath it is not caused by a lack of oxygen, but from a buildup of CO2.

That is why odorless gases are so dangerous. If there's no oxygen in the gas you are breathing but also no CO2 the body thinks "this is fine" until you pass out.