Materials like stone in some of the US (I'm specifically thinking places with very little airflow like in parts of the deep swampy south) would be deadly without constant, properly-running mechanical ventilation, for one. In a power outage in the south, people would die without it as stone buildings spend all day getting heat and radiating it back out. That type of house suits some cold climates fine, but is very bad in 35 degrees with 80% humidity. Likewise, in high-humidity environments, mold becomes a real issue without that mechanical ventilation (opening windows does nothing when outside is that humid and particularly when there is no wind).

In earthquake zones, you WANT flexibility; stone and brick are deadly as mortar joints fail and the structure collapses.

I think you underestimate how strong properly-built wooden-construction homes can be. There are also materials and cost issues to building with other things. Finally, as I started with, home construction should be appropriate to the climate in which the house is built with consideration for the local materials and safety.

Here in Japan, we have wood and steel-reinforced concrete. Rarely, you'll find reinforced block, but I think that was a fad that passed. Anything brick-and-motor now has all kinds of steel bracing added for earthquake protection. The house I'm in is not far from Fukushima and survived the 2011-03-11 earthquake and tsunami with only a minor thing to be fixed (and some cosmetic damage to wallpaper). It's made of wood.

Edit: fixed an "a lot of the US" that was left from a previous wording and further clarified what I had in mind. A lot of the US was indeed wrong and not what I meant to post.