In my experience you can absolutely hear where the sub is located - if you don't play the mains at the same time.
Have you tried it with your eyes covered? Have your friend cover your eyes and rotate you a few times so you loose room orientation and then try to point out the sub location.
Sub localization blind tests have been performed in a controlled manner and virtually nobody managed to localize a sub playing up to 80Hz and many listeners have problems localizing a sub playing as high as 180Hz. Search AES database and you will easilly find articles on such researches.