The thing is: Most hifi freaks know nothing about music. Most hifi freaks are completely surprised, when they get out of their stylish homes with state-of-the-art equipment and listen to pure, straight live music. The real sound and the real volume of natural instruments is sometimes really shocking to them. When I was still active in the scene, many have gotten so far away from what music really is, that live music is strange or even irritating to them. Mind you: When I say "live music" I mean acoustical instruments and voices without amplification, I don't refer to a rock concert!
You are right and wrong at the same time, I'd say. For thinks like classical music, jazz and some kinds of blues you are right, as the live performance is the primary medium of this kind of music and any recording "only" is a reproduction of a live performance at a certain time, date and place - even if said performance was staged for the sole purpose of recording it and no audience except for the engineers was present.
That's why every performance is comparable - no matter if it's a school orchestra trying their hand on Beethoven or Daniel Barenboim, no matter which jazz combo is improvising on "Changes", they all share the same direct relation to the original sheet music - some performances are better, some are worse, but all are directly linked to the sheet music.
For pop music, as a broadly put genre (you can use "rock" or "rock and roll" as a descriptor here as well), it is different. Here, the studio recording, produced in one or a series of sessions by musicians and engineers/producers is the primary medium. "Folsom Prison Blues" by Johnny Cash refers to the song as he and his band recorded it under the auspices of Sam Philips at Sun Studio, Memphis, TN on July 30, 1955. And not to any take (or "cut" in early recording parlor) they put down that day, but the single one they selected for release. Any other recording of the same material, even if done by the same people in the same space, is not the pop song Folsom Prison Blues, but a cover, a copy, a fake. The reason for this is that pop music has always been thought with mass reproduction in mind, with the fact that it is possible to make identical copies of the same recording taking center stage. With the advent of recording techniques like Phil Spector's "Wall of Sound" or the excesses possible by multi-track consoles, like "Dark Side of the Moon", Queen's "Bohemian Rhapsoody" or even any 80s Iron Maiden Album (which generally feature four or more guitar tracks), it becomes blatantly obvious that a huge chunk of pop music was never even be conceptualized to be reproduced in a concert - of course, Floyd and Maiden regularily play these songs on tour, but these live performances have to feature stripped-down versions of the original songs (it should come as no surprise that both Maiden and Floyd have a reputation for excessive retouching of their "live" releases in studio).
So, if you primarily listen to pop music, emphasizing on getting your playback equipment right makes a whole lot of sense. Sadly, many "audiophiles" are jazz or classical music buffs. For them, it's a different story...