Yeah, if you don't move the camera or Google, this thing is nearly impossible.
I mean, in a single game I got two sections that were 2 lane paved roads with nothing around them. Could have been ANYWHERE.
There are quite a lot of things you can use to deduce where you are - Of course not precisely, but things like climate, terrain, mountains in the distance, car models, vegetation, especially trees are a good indicator of climate, as in subtropical. That said, my experience tells me that most dirt roads are either western Australia or northern America, so either of those are a good guess ;-)
I agree with Engine that I have tons of fun playing detective on those hard to guess areas!