For the athletes, it's a not a waste of time and effort.

For the hosting countries/cities?

They cost a TON of time and money, and it's a dubious proposition that the host cities ever recover that investment. Olympic villages are also rarely the best possible use of prime land, and often involve buying up and gentrifying entire neighborhoods for the purpose of building the facilities, and for the duration of the event, and the planning leading up to it, your city basically gives up autonomy to an organization that spends your resources as if they were their own in order to create a platform to sell advertising rights in the guise of sportsmanship. For the Olympics to be more sustainable and less corrupted by commercial interests, they should focus less on spectacle and expenditure and rely more on existing infrastructure.

Furthermore, while sport is an important and noble endeavor, it is by no means any more important than any other human endeavor, and every four years we have to be exposed to the intersection of sports and politics in discussions about nations that, for example, allow medalists to avoid conscription or escape poverty in various countries in exchange for bringing national prestige, and rarely do we take the opportunity to discuss what this says about our priorities, simply accepting the elevated prestige we place on this particular sporting event without question.

In short, I don't agree their cancellation would cause NOTHING of value to be lost, but the Olympics as an event in its current incarnation has PLENTY worth reevaluating, and we could all benefit greatly from reexamining its scale and role.