Simple explanation: 21st century tech.

A palm sized quadcopter, has more sensors and processing power, than many 20th century rockets.

SpaceX can afford to build dozens of (relatively) cheap prototypes, fill them with all kinds of sensors, hook them up to StarLink, and gather massive amounts of real-world data instead of some make-believe simulations, even when the rocket turns into thin dust. No video or flight recorders required.

For this latest flight 4, keep in mind that the damage to the flap would have thrown any simulation-based and verified flight computer program into the ground... but whatever they used, managed to adapt, compensate, and essentially land a rocket that was falling apart... all the while streaming live video and telemetry.

In software, a problem once solved is gone forever

That is not correct, and why having tests to detect regressions is important.

Not sure how much "technical debt" SpaceX might be incurring, but my guess is that each of these flights is providing massive amounts of data to plug into simulations of future designs, which might be more valuable than having a single "meticulous design" that would fail spectacularly if something like a rubber seal were to get too cold the night before.