Though unless you had either a prior knowledge of an album, or some visual reference to tie the sound to you wouldn't know if the channels had been reversed, so I can see where ummmm is coming from with the question.
If you only ever listened to your music through headphones or stereo speakers reversed to "normal" it would make no difference. it's only because you get used to the stereo channels being a particular way round that headphone makers remind you to wear them the correct way round.
It's to make sure you recieve the right signal to the correct ear just as the recordings intended you to hear it. You'l find that most are moulded to fit in the ear canal's specifically to the left and right ear, so able to gain full appreciation of the tune being played, with the full stereo separation to blow you away.