I don’t think that the software world has done a great job of letting people control that data lifetime. And I think that it’s something that a user should reasonably be able to expect out of their computer.

That's true.

I once thought about this, that maybe it's a good idea to use a tagged and maybe log-style filesystem, where 1) every directory name in file path becomes a tag for it, other than the user-added tags (which can be searched separately), 2) there are temporary and permanent files, where temporary ones are deleted once their lifetime passes or that plus once the space is required, while permanent ones are stored indefinitely, 3) with hardlink functionality transparently available to the user, from the GUI, 4) the GUI itself should drop the bullshit and return to DOS times in the sense of control - with this thing I describe it may well be that the casual user won't feel as lost as they do now.

Maybe (again, transparent and user-accessible) filesystem overlays for every application are a good idea too here, like with Docker, chroots, MacOS DMG images, etc.

In addition to that indexing file contents may make sense too, like you said.

Frankly there are so many good things one can do which haven't been done, before just OCR`ing everything on the screen and storing it.

About "why MS chose this" - because they consistently choose the dumbest and ugliest way to deal with any problem. The heaviest artillery available, to look relevant.

Offtopic - the searchable start menu problem is what scares me off Gnome every time I try it. You just get that tablet-like one-level place with a search field and icons. A frigging lot of fscking icons for every dot-desktop file Gnome found. Then I panic and get back to FVWM.