Well, you see, Ju$tice is expensive and unreliable.

So the decision of using or not the Justice System for enforcing one's rights isn't purelly "am I likely to win", it's also about "is it worth it bringing it to Court", which is especially important when the damages one is entitled to get are low (like the price of a game).

At which point "who controls the thing under dispute", which is theoretically unimportant in a perfect Justice System, becomes the main deciding factor.

Maybe an example will make it clear:

• You have a game which the publisher wants to take away from you. They're actually in their right to do so: you actually read through and accepted BEFORE PURCHASING (this is important in jurisdiction which aren't legal jokes, unlike the US) a set of terms and conditions that gave them that right and it was clearly that this wasn't a sale but a time limited licensing. However you have the actual installer for a single player game (in your computer, physical disk, whatever), totally free to install, no phone-home DRM authorization check - you control that copy. They have to actually take you to court to force you to delete that game from your system and destroy all copies. This is a $50 game. Are they really going to do it for a $50 game?!
• Similarly but reversed: you bought a physical disk with a game, it cost $50, it has phone-home DRM to install and to run. You didn't agreed to anything before the purchase and you're in a legal jurisdiction (such as Germany) which is not a joke so the implicit rights from a sale cannot be altered unilaterally post sale by a forced change of the contract terms of the sale (i.e. EULAs aren't valid there). You're in the right yet they block you from installing or running that game. To get back what you're entitled to or compensation (all of $50) you have to take them to court. Are you really going to do it for a $50 game?!

Anyways, the point I'm making here is that well before a Court of Law actually goes through the whole thing and determines who is in the Right and orders a certain action in favor of and/or compensation for the injured side, de facto the outcome is often decided by the actual stakeholders deciding "is it worth it taking this to court?" and in a system where Justice has costs (the bigger the costs the more that's the case) "who controls it" is pretty much the single biggest factor in that decision.

In simple terms, unless there is some kind of streamlined Judicial process for that kind of case (like there is around things like loans) bringing a case to court to force the side in control of something to act in a certain way with it is only worth it for large monetary amounts, and generally the price of a game is below that.

It just so happens that at the moment for console games having the installer for it in a physical medium in your control is having control of that copy but, as my example above illustrates, if the game does phone-home DRM authorization checks like some PC games go, it's actually not really under your control even if you have it in physical media.