The earliest recorded use of the word to refer to a building was from J Moser in an edition of American Architecture and Building News published in 1883.
Prior to that, of course, it meant a high sail, a tall horse, a tall man, a tall hat and various other things.
I thought the term was first used to describe a building as I have never heard it used to describe anything else but thinking about it it could apply to anything.