I'm definitely in the figurative 1 percent audiophile group, but I'm not wealthy. I know it might seem old-fashioned, but there was a time not so long ago when all sorts of people listened to music at home over a hi-fi. They weren't necessarily audiophiles, but they had a turntable or CD player, an amplifier or stereo receiver, and a pair of speakers. They also listened in cars, but the home hi-fi was where the bulk of their music collection was. Nowadays audiophiles might be the only people listening -- really listening -- to music at home; the other 99 percent of Americans don't.
I think anyone who sometimes gives music their undivided attention is an audiophile, no matter where they listen. The general population listens in a different way; for them, live concerts or recorded music is background sound to other activities, such as talking, texting, etc. So sound quality is no big deal for them; the music is just there. Once you start to focus on the music, sound-quality differences become important.
I enjoy my $35 Koss PortaPro headphones, but I get more out of music when I listen over better headphones, like my $1,099 JH Audio JH-13s. There's more there there. High-end gear can be expensive, but it can provide decades of enjoyment. A pair of Magnepan MMG flat-panel speakers ($600), an NAD C 326BEE integrated amplifier ($550), and a Pro-Ject Carbon turntable ($400) would make for a very respectable budget high-end system. Some folks spend as much as that system's cost on Starbucks or beer in a year, but a comparable system with used gear would be an even more affordable way to be part of the 1 percent audiophile "elite." My $70 starter system would still qualify as a hi-fi, so cost shouldn't stop anyone from joining the 1 percent club.