Years ago, no one wanted to stage the games, and they were riddled by boycotts, claims of cheating, and the fact that they were too expensive to stage.
Then big business got involved and suddenly countries were falling over themselves to stage them. Occasionally they are shamelessly used by an authoritarian power to bolster its image, Berlin 1936, Sochi 2014 for example. Otherwise I say lets play. If the money really would get spent elsewhere then by all means do so but it wouldn't .