How I deal with this is when the Xbox 360 is booting up and the game already in the drive starts to load, I hit the middle silver Button (???) and then hit the Y button and select the 360 desktop. This will stop the loading of the game but allow you to keep it in the drive.
Then if you want to start the game again from the 360 desktop, just open and close the drive by remote with the controller. This works for me.