I have a confession

[solderdude]

3) It imparts a psychological load to perform, thus stunting capability people usually would have when not under pressure.

Yes, this is the only problem I see with blind tests which are exhausting. The solution is simple but a bit difficult but not impossible to implement.
It would be impossible to do this on just one day as one needs statistical relevance to make a valid test.
Over one day that would be hard to do.

One would need to test over several days, at home, with familiar music, room, transducers and can only swap DAC's, cables, amplifiers.
There can still be hints though... the volume control setting of PC for instance.

You need someone to switch components (well the cables) at random time intervals over weeks and keep notes. It must not be visible on equipment what is playing (displays, indicators).
The tester must make notes while listening (no swapping when listening) and compare.

Differences would need to be really big to be noticed (instead of really small what audiophools seem to think) for someone to pass such test.

I have accidentally fooled myself thinking I heard a certain signature while in reality I left my AB switch in a position I thought it wasn't in for weeks or months. Only to find out later when testing something. This was in my younger years.
With the info I got from normal blind testing and accidental long-haul testing I knew back then already that my ears/brain really isn't as 'good' as some claim it to be. And yes, the speakers, sources and amps used were capable.

 

[Sgt. Ear Ache]

I'd buy the notion that blind testing is "stressful" and therefore unreliable if there was any hint in the audiophool narrative that they have any trouble at all hearing what they think they are hearing under all the various circumstances in which they claim they can hear it. If someone says to me "I can sometimes, under specific fairly stringent circumstances hear the difference between 320kbps mp3 and lossless" I don't have much of an issue. But that's not what we get. What we get is "the difference is night and day! I can tell instantly when I'm out driving around in my car and some random lossy file comes on my stereo! I was at my friends, and he was demoing his rig for me and I was like Dude what's with the lossy?? I heard it the second it started playing!" Or, "I just hooked up my new dac and after listening to it for a few minutes I can easily hear the massive improvement in clarity over my previous dac which I last listened to 3 days ago. It's night and day!" "I had my new headphones on the burn in station for 100 hours and now they have completely opened up! They sound so much better than they did when I listened to them fresh out of the box a week ago for 5 minutes! The difference is night and day!" We aren't talking about claims of subtle, extremely difficult to identify differences that are hard to note in any circumstance. If it's as the audiophools claim, it should be EASILY identified in a blind test...even with all that stress!

I suspect the stress would mostly be a result of the fact that they know they probably aren't going to be able to identify any difference because in their heart of hearts they know no difference exists.

Let's face it, we are often talking about hundreds, or thousands of dollars here. If I'm going to up the ante on a dac by a grand, I sort of hope the improvement in SQ I get for that money is significant enough that it doesn't cease to exist if I try and pick it out in a little blind test.

 

[Sgt. Ear Ache]

Of course, the other issue with blind testing is that anyone can come on to a forum on the internet and say "I did a blind test and I was able to identify my new dac 9 out of 10 times."

OK. Well that's nice, but really, it's not particularly convincing. I don't know you. Did you really do a blind test? Was it set up properly? Who knows? What I do know without any doubt is that I've done a few little tests of my own and they've led to my current convictions about things. I'm really only interested in tests set up by a third party organization...maybe an audio mag or forum group or something. Does Paul M's usb cable make any difference? Well, apparently on his own "super-revealing" system it does. What is it that's so super-revealing about his system? Is there some specific super-revealing thing that needs to be in the chain to reveal the difference? Does anyone else have a system that is super-revealing in the same way? Has any third party run a test that showed the cable to make any difference? I'm not particularly convinced by a test set up to show how good a product is by the company that sells the product. Call me a cynic if you will! lol...

 

[Costas EAR]

Heck, at 70dB, 12 bits, LP records would sound great!

Ehhhh, not really so "great"...

Bass is never "excellent" and of course the annoying hiss and pops of the surface, makes any LP sound worse than the same mastering in digital files.

Except that, i couldn't agree more with your post.

 

[thefsb]

Great post, Tim! Thanks for confessing.

Thanks ASR for letting me gain all this knowledge, direct my $$$ and attention to the important things, and enjoy high quality audio. These days even watching Disney movies with my little ones is a joyful auditory experience.

Clarifying priorities with respect to my personal context is also what I've gained from ASR.

Discussions here with a certain participants have also helped me concentrate what's personally important.

 

[thefsb]

Ehhhh, not really so "great"...

Bass is never "excellent" and of course the annoying hiss and pops of the surface, makes any LP sound worse than the same mastering in digital files.

That can be true at the same time that Tim's point is true for him.

 

[Costas EAR]

Ok, i confess that i really enjoy a good LP mastering, with auro 3D upmixing to match my immersive setup.

Yes i know, analogue to digital and again, ok, no problem from my point of view, i want dsp for room equalization plus target curve and bass management and upmixing and so on, I'm sure the conversion is more transparent than needed.

No worries.

 

[Unclevanya]

I'd buy the notion that blind testing is "stressful" and therefore unreliable if there was any hint in the audiophool narrative that they have any trouble at all hearing what they think they are hearing under all the various circumstances in which they claim they can hear it. If someone says to me "I can sometimes, under specific fairly stringent circumstances hear the difference between 320kbps mp3 and lossless" I don't have much of an issue. But that's not what we get. What we get is "the difference is night and day! I can tell instantly when I'm out driving around in my car and some random lossy file comes on my stereo! I was at my friends, and he was demoing his rig for me and I was like Dude what's with the lossy?? I heard it the second it started playing!" Or, "I just hooked up my new dac and after listening to it for a few minutes I can easily hear the massive improvement in clarity over my previous dac which I last listened to 3 days ago. It's night and day!" "I had my new headphones on the burn in station for 100 hours and now they have completely opened up! They sound so much better than they did when I listened to them fresh out of the box a week ago for 5 minutes! The difference is night and day!" We aren't talking about claims of subtle, extremely difficult to identify differences that are hard to note in any circumstance. If it's as the audiophools claim, it should be EASILY identified in a blind test...even with all that stress!

I suspect the stress would mostly be a result of the fact that they know they probably aren't going to be able to identify any difference because in their heart of hearts they know no difference exists.

Let's face it, we are often talking about hundreds, or thousands of dollars here. If I'm going to up the ante on a dac by a grand, I sort of hope the improvement in SQ I get for that money is significant enough that it doesn't cease to exist if I try and pick it out in a little blind test.

I've done my own blind testing on high bitrate lossy vs wav and taken a few blind tests like the npr one as well. I tended to properly ID lossless as sounding better 5 out of 6 times at best (lower bitrates and familiar music) and 4 of 6 at worst (higher bitrate unfamiliar music). However these were a/b comparisons of which is preferred, not "pin the tail on the lossy file". Never once did I cringe over lossy, there were only subtle differences in the sound and in many of these I was using recordings I knew well. In those cases my preference for the lossless version might be simple familiarity more than strictly sound quality. In some tests the music was less familiar and that's where I tended to score closer to 4 of 6.

What was I hearing? I am not certain, but I can tell you I had to strain and concentrate and focus on the sound of voices and specific instruments and sometimes go back and forth a few times before making my pick. I honestly hated the experience, it wasn't listening to music, it was exhausting mentally and unsatisfying as a listening session. Personally all I take from the experience is that there are subtle cues and differences in the music generated by lossy vs lossless, and that in side by side listening I had a small preference for the lossless.

Practically what this means is that I prefer to feed my home system with cd quality sources and I don't sweat it if I can't. The car gets mp3/aac etc due to size and bandwidth (the noise floor in the car is so high anyway I doubt I could tell the difference.)

One thing I wish we had more data on is how measurements can predict when the sound of one thing is preferred over another. The high degree of love for tube amps for example suggests that some of the metrics we use aren't that important to people in real world listening sessions. I suspect many of us wouldn't immediately realize tubes vs transistors were being used if we walked in while a song was playing.

 

[Sgt. Ear Ache]

When it comes to preferences, I'm at a point now where what I want is a system that is neutral - one that doesn't color the signal in any way. Given that, I can EQ from there for my preference (and for my room as well) if I need to. But I'm finding that getting things to my ears as neutral as possible and just living with it for a bit has been really eye (erm...ear)-opening.

 

[Sgt. Ear Ache]

What was I hearing? I am not certain, but I can tell you I had to strain and concentrate and focus on the sound of voices and specific instruments and sometimes go back and forth a few times before making my pick. I honestly hated the experience, it wasn't listening to music, it was exhausting mentally and unsatisfying as a listening session.

well...the goal of course isn't to have a satisfying music listening experience. I mean it is very definitely a test. But I do get what you're saying. I've taken several of those online tests myself and usually what happens is I reach the IDGAF point long before I finish the test. For me, the fact that it takes honest effort and dedication to maybe, possibly hear a difference (and I'm talking strictly about high br 256kb or better lossy here) tells me that it's just not even worth worrying about for any real-world listening situation. If I have to struggle to hear a difference on my excellent headphone rig then there's no way I'm hearing it in my living room with it's minimum 35db ambient noise level or in my car for goodness sake! Contrary to what you'll hear on various audio forai, what the tests show me is that high br lossy is definitely NOT "shit."

 

[murraycamp]

Agreed. I don't really go chasing hi-res audio for my music listening. Redbook is plenty fine for me.

I agree that it's quite unlikley I could hear the difference between 12/44 and 24/44 or 96. What I am finding (anecdotally) however is that the hi-res tracks often seem to be taken from better masters, at least for recordings where there were re-mastered issues.

 

[T.M.Noble]

I read that story somewhere and the participants actually preferred the Delta Sigma version compared to the Multibit version of the Gungnir DAC on that blind test

Correct that was one of them.

 

[T.M.Noble]

Sounds interesting. Could you give a bit of detail on some of the interesting results? Thanks.

I will get into more details soon. Jason Stoddard posts them every time we have a blind test on the Schiit thread on head fi.

 

[Unclevanya]

well...the goal of course isn't to have a satisfying music listening experience. I mean it is very definitely a test. But I do get what you're saying. I've taken several of those online tests myself and usually what happens is I reach the IDGAF point long before I finish the test. For me, the fact that it takes honest effort and dedication to maybe, possibly hear a difference (and I'm talking strictly about high br 256kb or better lossy here) tells me that it's just not even worth worrying about for any real-world listening situation. If I have to struggle to hear a difference on my excellent headphone rig then there's no way I'm hearing it in my living room with it's minimum 35db ambient noise level or in my car for goodness sake! Contrary to what you'll hear on various audio forai, what the tests show me is that high br lossy is definitely NOT "shit."

I think we are in complete agreement. Essentially, while I can beat the test, I don't do so easily or without effort. And in the absence of the test it isn't likely that I will complain about quality.

I will say that anecdotally, in past tests of vbr and 320bit mp3 encoding programs, I found some tracks that sounded "off" on specific passages using a few of the encoders. This effort was done years ago and the encoders are likely much better now. So in general lossy isn't going to be easily identified by anyone, and typically won't be heard as sounding sub-par, there may be a few odd tracks with passages that might encode poorly. I haven't heard any "in the wild" so in theory this risk exists but in practice it likely isn't an issue.

In a similar way I liked my NAD D3020 V2 until I compared it subjectively to my Yamaha WXA-50. Either would have been satisfying - only when heard back to back did I note a preference.

 

[MattHooper]

Of course, the other issue with blind testing is that anyone can come on to a forum on the internet and say "I did a blind test and I was able to identify my new dac 9 out of 10 times."

OK. Well that's nice, but really, it's not particularly convincing. I don't know you. Did you really do a blind test? Was it set up properly? Who knows? What I do know without any doubt is that I've done a few little tests of my own and they've led to my current convictions about things. I'm really only interested in tests set up by a third party organization...maybe an audio mag or forum group or something.

I'm a bit confused. If I understood: you have done some personal blind tests that you feel justified some of your current convictions, but then say you are only really interested in tests set up by third party organisations. How do your own blind tests fit in there? Did you draw anything useful from them?

Your post hits at a problem for many of us mere consumers. Blind testing if done right can be very helpful and illuminating, especially once you have personal experience having had a blind test overturn a belief you'd built on sighted testing

But they can be tricky to set up right. And it seems our results often are sort of restricted to personal use for the reasons you give above.

So, an example from my experience. In the late 90's I was on the highend audio newsgroups a lot, on which there was always a firestorm between the "objectivists" and subjectivists, the objectivists usually being the ones who had some science or engineering under their belt, or both.

The objectivists would tell us that DACs were essentially a solved problem; that any properly built DAC or CDP should sound identical to another.
I understood the rational, but I seemed to be having some very strong subjective experience that the 3 digital players I had at the time - a Sony CDP, Meridian CDP and a Mietner DAC - seemed to have subtle but distinct differences in sound. Knowing full well the problems of sighted bias I had a friend help me blind test between them and I posted the results: I could easily identify each unit by it's sound with essentially 100 percent accuracy.

The objectivist crowd were rightly suspicious and when we went through the protocol the told me how I could tighten things up. So I repeated the blind test with help from my father-in-law (engineer), matching output levels at the speaker terminals etc. The result again was my being able to identify the players with just about 100 percent accuracy. When I posted the results, to little surprise, the objectivists didn't want to just accept them. Which I completely understood. Because, as you said: I'm just some other person on the internet to them! I could be lying. Or exaggerating. Even though the procedures I followed matched what they told me, they weren't there to see if I goofed up somewhere along the line.

So I simply would not expect them to accept my results as a "fact" they had to accommodate in their viewpoints. ** So, really, the best I could do is ask "in what ways could I have screwed up the test to produce those results?" In looking at the replies I knew, having done the test, that I had essentially accounted for the possibilities in the test. So, while my results couldn't be definitive for THOSE people, they could at least provide provisional data for me, in my own reckoning of these things.

Someone like Amir can produce results with more confidence than I ever could. But even THERE, if we want to be truly scientific, we can say "Ok, but we weren't there, Amir could be screwing up the results somewhere....so we need the results to be replicated reliably by other parties."

Utlimately, though, when it gets down to trying to understand the world, how to spend our money etc, we end up calling 'em as we see 'em, combining our personal experience with what we take to be realiable-enough data provided by other parties.

**(There is of course nothing impossible about the proposition the digital devices produced audible differences. The engineer/objectivist stance is that any *properly designed* DAC/CDP should sound indistinguishable. Any of the DACs I had on hand may have been designed to fudge the sound a little bit in one way or another).

 

[MattHooper]

Thanks ASR for letting me gain all this knowledge, direct my $$$ and attention to the important things, and enjoy high quality audio. These days even watching Disney movies with my little ones is a joyful auditory experience.

That's awesome. It's nice to see someone happy with their audio and music, and great to see a place like this can help.

Good sound is Good.

Which, for me, brings up all the issues of "what do I want from my sound system?" Ultimately I want "good sound" and I think, at bottom, that's what most of us want. Even though we may diverge here and there on what gets us there.

There is a mindset, represented often here, that the goal is "accuracy." I just want to hear what's in the source with as little distortion - hence greatest accuracy - as possible. So one chases components with as little deviation from linearity etc as possible.

So..."once I know I have an accurate system I can relax, not worry about things, sit back and enjoy the music."

That certainly works for a particular mindset. And some seem to believe this is a sort of recipe for "getting off the gear treadmill of chasing 'sound' (associated with audiophile subjectivism) instead of 'accuracy.'" But, again, though that may work for some people, like different diets, it doesn't work for all.

After all: One can also "get off" the chasing equipment treadmill by simply lowering one's standards of what they demand as well. My wife is very "off the equipment treadmill" as she completely enjoys music from a laptop or our smart speaker, without any itch at all to upgrade to something "more accurate." So even going the "more accuracy" route will often keep someone quite gear-focused. And as accuracy increases, you have similar impetus to keep an eye on advances in gear and upgrade along the way. And there are plenty of subjectivist audiophiles who have been very satisfied owning the same gear for decades as well. One approach to finding satisfaction does not fit all; depends on the individual mindset in any case.

The thing is that "Sounds Good" can be separated from "Accuracy." Accuracy does not automatically entail "good sound." So if we have a choice...what would it be, and why? (That is not to suggest, btw, that "accuracy does not sound good," only that it is a separable subject from what people may find pleasing or "good sound.")

Personally, I'm settling toward a "what do I enjoy?" "What sounds good to me" approach. The reason is that accuracy as a goal, while a laudable north star, ultimately disappears down a rabbit hole in terms of actually achieving this. (You aren't going to truly escape the "circle of confusion," especially when you follow the logic "to what end do I care about accuracy in the first place?"). Though I retain a keen interest in how technology advances to lower distortion, with audible results.

This doesn't lead me only to wanting terribly coloured sound quality. I can enjoy speakers, for instance, that are "more accurate" and some that are "less accurate/more colored" depending if they push some of my joy-buttons. But while, in the case of enjoying a bit of a colored sound, I accept some compromise in accuracy, in my view it is not much of a compromise. The type of colorations I may enjoy are utterly swamped in sonic importance by the sonic signature of the artistic content. I'm hearing all the artistic choices on the track. And enjoying them.

In contrast, the accuracy-is-all-that-counts person accepts sonic compromises when the system "accurately" exposes the poor sound of a badly produced/recorded/mastered track.

Of course this person can always say "but I can just re-EQ those tracks if I want to, to make them sound better." Which is of course fine...but then don't say that you are all about "accuracy" if you are so ready to alter the sound to your taste ;-). You may do it with an EQ, I may do it with some slight coloration in a speaker or tube amp or whatever. (And the "but I can defeat the EQ at any time" rebuttal, making my system more flexible" really doesn't address the meat of the issue. Once you are introducing EQ to taste sometimes, you are altering the sound to taste. Even if you don't do this all the time, or often, on what principled grounds can you defend this against the preference-loving audiophile? "Well, I only alter the sound to taste *sometimes*. Well...who exactly made you the arbiter as to when the sound can be altered or not? It seems completely arbitrary, for instance "I like some bass punch and if a track is really missing that I'll add some." How is that any more principled from someone who says "I like a full-bodied sound, generally speaking, and this speaker with a bit of coloration adds that. ?

So I generally approach a system as to how much I enjoy it, and there tend to be sonic characteristics that I enjoy - for instance when I hear acoustic guitars or other acoustic instruments with a certain "right-to-my-ears" tonal quality, some body to the sound, dynamic life, etc, or just noticing that something is pulling me in to wanting to keep listening in a way some other system isn't.

I have a number of different speakers all of which have, in their own way, a bit of "magic tone" for me, that draw me in to sit down and spend long time listening. I just came back from a friend's house (reviewer) listening to 3 different expensive speakers. They were impressive in many ways, but I wouldn't take a single one if you gave it to me. None made me want to keep listening. They were tonally artificial and uninvolving for me. I came home, fired up my system and it was "aaaahhhh...." I couldn't stop listening.

Anyway, random thoughts from an audio-loving geek, which occasionally go against the grain here .

 

[Sgt. Ear Ache]

My blind tests are for my own convincing. I'm not using them to convince anyone else of anything. Nor would I expect anyone to gain much from them because like I said, nobody on the net knows me or what I've actually tested. Blind tests that I've taken have no real bearing on anyone else...they just prove to me what I myself have come to understand. Which leads back to my really only being interested in testing done by respected third party groups...

 

[Sgt. Ear Ache]

**(There is of course nothing impossible about the proposition the digital devices produced audible differences. The engineer/objectivist stance is that any *properly designed* DAC/CDP should sound indistinguishable. Any of the DACs I had on hand may have been designed to fudge the sound a little bit in one way or another).

Yep, in which case I'd assume if you were to record the output of each of the dacs and compare there would probably be some measureable difference?

 

[Sgt. Ear Ache]

That's awesome. It's nice to see someone happy with their audio and music, and great to see a place like this can help.



Which, for me, brings up all the issues of "what do I want from my sound system?" Ultimately I want "good sound" and I think, at bottom, that's what most of us want. Even though we may diverge here and there on what gets us there.

There is a mindset, represented often here, that the goal is "accuracy." I just want to hear what's in the source with as little distortion - hence greatest accuracy - as possible. So one chases components with as little deviation from linearity etc as possible.

So..."once I know I have an accurate system I can relax, not worry about things, sit back and enjoy the music."

That certainly works for a particular mindset. And some seem to believe this is a sort of recipe for "getting off the gear treadmill of chasing 'sound' (associated with audiophile subjectivism) instead of 'accuracy.'" But, again, though that may work for some people, like different diets, it doesn't work for all.

After all: One can also "get off" the chasing equipment treadmill by simply lowering one's standards of what they demand as well. My wife is very "off the equipment treadmill" as she completely enjoys music from a laptop or our smart speaker, without any itch at all to upgrade to something "more accurate." So even going the "more accuracy" route will often keep someone quite gear-focused. And as accuracy increases, you have similar impetus to keep an eye on advances in gear and upgrade along the way. And there are plenty of subjectivist audiophiles who have been very satisfied owning the same gear for decades as well. One approach to finding satisfaction does not fit all; depends on the individual mindset in any case.

The thing is that "Sounds Good" can be separated from "Accuracy." Accuracy does not automatically entail "good sound." So if we have a choice...what would it be, and why? (That is not to suggest, btw, that "accuracy does not sound good," only that it is a separable subject from what people may find pleasing or "good sound.")

Personally, I'm settling toward a "what do I enjoy?" "What sounds good to me" approach. The reason is that accuracy as a goal, while a laudable north star, ultimately disappears down a rabbit hole in terms of actually achieving this. (You aren't going to truly escape the "circle of confusion," especially when you follow the logic "to what end do I care about accuracy in the first place?"). Though I retain a keen interest in how technology advances to lower distortion, with audible results.

This doesn't lead me only to wanting terribly coloured sound quality. I can enjoy speakers, for instance, that are "more accurate" and some that are "less accurate/more colored" depending if they push some of my joy-buttons. But while, in the case of enjoying a bit of a colored sound, I accept some compromise in accuracy, in my view it is not much of a compromise. The type of colorations I may enjoy are utterly swamped in sonic importance by the sonic signature of the artistic content. I'm hearing all the artistic choices on the track. And enjoying them.

In contrast, the accuracy-is-all-that-counts person accepts sonic compromises when the system "accurately" exposes the poor sound of a badly produced/recorded/mastered track.

Of course this person can always say "but I can just re-EQ those tracks if I want to, to make them sound better." Which is of course fine...but then don't say that you are all about "accuracy" if you are so ready to alter the sound to your taste ;-). You may do it with an EQ, I may do it with some slight coloration in a speaker or tube amp or whatever. (And the "but I can defeat the EQ at any time" rebuttal, making my system more flexible" really doesn't address the meat of the issue. Once you are introducing EQ to taste sometimes, you are altering the sound to taste. Even if you don't do this all the time, or often, on what principled grounds can you defend this against the preference-loving audiophile? "Well, I only alter the sound to taste *sometimes*. Well...who exactly made you the arbiter as to when the sound can be altered or not? It seems completely arbitrary, for instance "I like some bass punch and if a track is really missing that I'll add some." How is that any more principled from someone who says "I like a full-bodied sound, generally speaking, and this speaker with a bit of coloration adds that. ?

So I generally approach a system as to how much I enjoy it, and there tend to be sonic characteristics that I enjoy - for instance when I hear acoustic guitars or other acoustic instruments with a certain "right-to-my-ears" tonal quality, some body to the sound, dynamic life, etc, or just noticing that something is pulling me in to wanting to keep listening in a way some other system isn't.

I have a number of different speakers all of which have, in their own way, a bit of "magic tone" for me, that draw me in to sit down and spend long time listening. I just came back from a friend's house (reviewer) listening to 3 different expensive speakers. They were impressive in many ways, but I wouldn't take a single one if you gave it to me. None made me want to keep listening. They were tonally artificial and uninvolving for me. I came home, fired up my system and it was "aaaahhhh...." I couldn't stop listening.

Anyway, random thoughts from an audio-loving geek, which occasionally go against the grain here .

I haven't found all that terribly many albums that are so badly produced that they sound worse on a neutral system then they do on any "good" sound system. I don't EQ to improve the sound quality of individual recordings. I listen to em as they are.

 

[MattHooper]

Yep, in which case I'd assume if you were to record the output of each of the dacs and compare there would probably be some measureable difference?

Presuming my blind tests were valid, that is just what I'd expect!

I haven't found all that terribly many albums that are so badly produced that they sound worse on a neutral system then they do on any "good" sound system. I don't EQ to improve the sound quality of individual recordings. I listen to em as they are.

Same here. I do use tube amps with *may or may not* be acting as a bit of EQ, depending on the speaker. But in any case I simply like what I hear and never feel the need to EQ. In fact I had a digital EQ in my system for almost 20 years and found so little use for it I sold it not too long ago.