Catalogue of blind tests

[Spkrdctr]

I doubt anything will "resolve this debate finally", because the objectivists will continue to believe in science, and the subjectivists will continue to believe in "feel".

That is the truth!

 

[ahofer]

But I think it would be great to add to the body of tests. Thresholds of audibility, for instance, of electronics with inferior measurements.

 

[Spkrdctr]

But I think it would be great to add to the body of tests. Thresholds of audibility, for instance, of electronics with inferior measurements.

The audibility of anything that we find in testing is where I go to for pretty much all of my posts. Say we find that wire/amp/interconnect has a difference of resistance of .050 of an ohm. It is measurable every single time it is tested and 100% repeatable every time. Then I come along and say, is it audible? It will not be. So many things in the audio chain will swamp that measurement that it in effect does not matter. Room interaction, speaker ability to resolve at that level, speaker placement, listener placement on and on. that little bit of an ohm will NEVER affect the sound to the listener. So, it is an interesting measurement, but it means nothing for the audio chain. Most of our current audio gear is pretty good to very good at sound reproduction. The room, our ears, our brain and many other factors tend to degrade the accuracy of the sound by a large amount.

This is where Amir's measurements come into play. A consumer (not an engineer type) can look at the table and graphs and recommendations and say, well in the chart of all of the equipment tested, lets say DACS, just pick anything you like from the top 25% of the chart. You will be good to go. Features etc. then will come into play. People do not have to go to the absolute best item that Amir has tested, but they can though if they want. Amir DOES give us great info that allows us to dodge a buying mistake by not getting anything in the bottom half ot the chart. Or wherever someone wants to draw their own personal line in performance. So once again between Amir and Audioholics, you can buy very good equipment that will not need to be replaced every 3 years. Plus, it cuts through all the marketing BS. Claiming to be the best and then testing to be the best are two different things.

 

[Gorgonzola]

I doubt anything will "resolve this debate finally", because the objectivists will continue to believe in science, and the subjectivists will continue to believe in "feel".

It is one thing to be skeptical of the extravagant claims of subjectivists who claim to hear "huge" differences between, say, cables, and, OTOH, to mock and ridicule anyone and everyone who says they have heard differences between, say, amplifiers. There is a core of denialists here at ASR who do the latter on a regular basis.

Going right back to the OP, referring to Ian Masters article in the January 1987 edition of Stereo Review, it is clear that some listeners were able to hear differences between specific amps at well above statistic significance. Master's general conclusion was that, "... Certainly there are still differences between amps, but we are unlikely to hear them." "Unlikely" is the same thing as never by anyone.

I suggest our local denialist cadre back off a bit when other posters say they can hear differences -- because maybe they can and do.

 

[MarkS]

This is where Amir's measurements come into play. A consumer (not an engineer type) can look at the table and graphs and recommendations and say, well in the chart of all of the equipment tested, lets day DACS, just pick anything you like from the top 25% of the chart. You will be good to go.

Actually you will be good to go if you pick anything from the top 99% of the chart, because almost no modern electronics measures badly enough to make an audible difference.

This is where I differ from most here at ASR. Yes, it's good to weed out the badly designed and/or built 1%, but other than that, there is really nothing to discuss, and nothing to care about.

In this sense, measurements really don't matter.

 

[MarkS]

I suggest our local denialist cadre back off a bit when other posters say they can hear differences -- because maybe they can and do.

If they say they can hear differences between two amps when they don't know which one is in the system, then I am happy to believe them.

But no one ever does this. No one ever goes to the effort to listen blind before declaring that amp A sounds better than amp B.

And, since decades of research shows that sight heavily influences perceived sound, there is no particular reason to take them seriously. And so I don't.

If that makes be a member of the "local denialist cadre", so be it.

 

[Gorgonzola]

If they say they can hear differences between two amps when they don't know which one is in the system, then I am happy to believe them.

But no one ever does this. No one ever goes to the effort to listen blind before declaring that amp A sounds better than amp B.

And, since decades of research shows that sight heavily influences perceived sound, there is no particular reason to take them seriously. And so I don't.

If that makes be a member of the "local denialist cadre", so be it.

I'm not arguing with the fact that sight can influence perceived sound, however Master's typical method was blind ABX where 'X' is unidentifiable and the participate must choose whether it is 'A' or 'B'.

By the way, there is no problem if participants knows that 'A' is the Futterman, and 'B' is the Hafler. The crux of the matter is if they can with better than random chance identify the hidden 'X' as 'A' when it's Futterman and 'B' when it's Hafler. This proves that differences can be heard: QED. Whether 'A' sounds "better" than 'B' is largely subjective anyway, and is irrelevant in the test context.

Master concluded that differences generally tiny and were usually unidentifiable: this is true. But he didn't deny that some times differences could heard.

 

[SIY]

I suggest our local denialist cadre back off a bit when other posters say they can hear differences -- because maybe they can and do.

When they do, absent a significant measured error, and then cannot provide any evidence, no need to "back off" insisting on verification before anyone can accept it, your word-parsing (and neglect of qualifiers) notwithstanding. Bring evidence and there's no problem. Wave hands and stomp your feet, you're in Make-Believe Land.

 

[MarkS]

Master concluded that differences generally tiny and were usually unidentifiable: this is true. But he didn't deny that some times differences could heard.

Sure, sometimes differences can honestly be heard. And ABX, where A and B are known and X is not, is a perfectly valid methodology (but one that requires special equipment for set-up, and precise level matching).

My point is that the overwhelming majority of sound quality claims are made with full knowledge; e.g. I put in the Futterman instead of the Hafler, and I think it sounds better. I switch back to the Hafler, I think it sounds worse.

I reject all such claims as providing any evidence of sonic difference.

 

[Kalessin]

I doubt anything will "resolve this debate finally", because the objectivists will continue to believe in science, and the subjectivists will continue to believe in "feel".

It is one thing to be skeptical of the extravagant claims of subjectivists who claim to hear "huge" differences between, say, cables, and, OTOH, to mock and ridicule anyone and everyone who says they have heard differences between, say, amplifiers. There is a core of denialists here at ASR who do the latter on a regular basis.

Going right back to the OP, referring to Ian Masters article in the January 1987 edition of Stereo Review, it is clear that some listeners were able to hear differences between specific amps at well above statistic significance. Master's general conclusion was that, "... Certainly there are still differences between amps, but we are unlikely to hear them." "Unlikely" is the same thing as never by anyone.

I suggest our local denialist cadre back off a bit when other posters say they can hear differences -- because maybe they can and do.

I was trying to say that nothing will "resolve the debate finally" because there is no argument that will cause one side to completely accept the other side's position and abandon their own. Similar to "Mac vs. PC" or "religion vs. atheism" debates, there is no win condition, no final resolution, possible in this debate.

 

[Roland]

Actually you will be good to go if you pick anything from the top 99% of the chart, because almost no modern electronics measures badly enough to make an audible difference.

This is where I differ from most here at ASR. Yes, it's good to weed out the badly designed and/or built 1%, but other than that, there is really nothing to discuss, and nothing to care about.

In this sense, measurements really don't matter.

99% of dacs and amps sound the same so there is no reason to buy anything other than a Denon AVR-X3700/4700/6700 depending on your requirements for power, channels and features. I can’t understand why there is so much discussion around other amps and dacs.

 

[Geoffkait]

There are many reasons blind tests can fail. System issues, hearing or listening skill issues, psychological issues perhaps, not following instructions, and others. The main point is that if the results of a single blind test are negative the results have no meaning. That’s because of all the things that can go wrong with any test. The best laid plans of mice and men oft go awry. To obtain more validity tests must be repeatable and transferrable, thus conclusions might be drawn after many tests by different testers on many systems.

 

[Gorgonzola]

There are many reasons blind tests can fail. System issues, hearing or listening skill issues, psychological issues perhaps, not following instructions, and others. The main point is that if the results of a single blind test are negative the results have no meaning. That’s because of all the things that can go wrong with any test. The best laid plans of mice and men oft go awry. To obtain more validity tests must be repeatable and transferrable, thus conclusions might be drawn after many tests by different testers on many systems.

Apart from the fact that Ian Masters, (as an example), was able to establish that differences were sometimes audible using ABX testing, I agree that this test paradigm is flawed as proof for the bigger picture.

Simply put, ABX isn't how audiophile establish their impressions and preferences for equipment they evaluate. Such impressions and preferences are typically establish over many hours, days, weeks, and indeed, years of listening.

I agree that many of the sound difference that audiophiles insist they hear are imaginary -- on the other hand some are not. It's presumptuous of deniers to mock individuals any time and every time they claim to hear a difference.

 

[Gorgonzola]

Sure, sometimes differences can honestly be heard. And ABX, where A and B are known and X is not, is a perfectly valid methodology (but one that requires special equipment for set-up, and precise level matching).

My point is that the overwhelming majority of sound quality claims are made with full knowledge; e.g. I put in the Futterman instead of the Hafler, and I think it sounds better. I switch back to the Hafler, I think it sounds worse.

I reject all such claims as providing any evidence of sonic difference.

Of course you're right: sighted listening impressions are NOT PROOF or even a whole lot of evidence that sound differences exist in any given instance. But the consistent testimony of dozens, hundreds, or indeed thousands of audiophiles of are of value and are certainly powerful evidence (if not proof) that, for example, tube equipment typically sounds different from solid state.

Further the concept of "scientific" proof, let's not forget the proper science is always seeking to disprove its theories -- there is no such thing as "settled science". Proper science looks to find new evidence and improve methods. Standard ABX testing has demonstrated that differences can, at least sometime, be heard then ipso facto it has not demonstrated that they cannot. Those who want to prove that the consistent testimony of thousands is wrong will have to come up with a different approach than ABX testing.

 

[SIY]

There are many reasons blind tests can fail. System issues, hearing or listening skill issues, psychological issues perhaps, not following instructions, and others. The main point is that if the results of a single blind test are negative the results have no meaning. That’s because of all the things that can go wrong with any test. The best laid plans of mice and men oft go awry. To obtain more validity tests must be repeatable and transferrable, thus conclusions might be drawn after many tests by different testers on many systems.

Still running your fraud business and cheating the gullible?

 

[Geoffkait]

I’ve been unmasked by a “technical expert“ and “major contributor.” Gosh, just when I thought it was safe to go back in the water.

 

[SIY]

So, yes, you're still running your scams.

This is probably not the place for you to find your target audience.

 

[Geoffkait]

Ouch. Very ouch!

 

[Spkrdctr]

But the consistent testimony of thousands of audiophiles of are of value and are certainly powerful evidence (if not proof) that, for example, tube equipment typically sounds different from solid state.

Further the concept of "scientific" proof, let's not forget the proper science is always seeking to disprove its theories -- there is no such thing as "settled science". Those who want to prove that the consistent testimony of thousands is wrong will have to come up with a different approach than ABX testing.

Ok, I changed the original post to show what I wanted to comment on. First off, the statement that thousands of audiophiles believing something is "near proof" of its validity. This is wrong. Dead wrong. This is the entire reason that DBT exists. We know for a fact that after thousands of tests over the last 40 years that not one of the audiophiles who was tested could pass ANY test. Many of these tests were set up without the extreme engineering level that many want on this forum. Because the tests were done to see if they could tell a difference between two products and not for product development which is the engineering side of testing. These tests, were done to see what if any, products in the audio chain made a difference that was audible to an audiophile. With no one ever passing a test, it is what I call settled science. If in the future, if someone can pass a test, then it will be thrown wide open on that piece of wire, interconnect etc.

The testing I was involved in was far simpler than what is talked about on this forum. These tests were done with people who knew that the difference in sound was so obvious, anyone could hear it. It was just a settled issue across the country (by reading magazines at the time) that they would fly through the test and easily pass. This would once and for all show the darn testing people, that people could and did hear differences all the time and did it very, very easily. Since we had a 100% failure rate during the testing, we skewed the tests to give the product or idea the maximum help we could. The tests ended up being what our engineers on here would call "whacky", we figured well, certainly they could pass the tests now. STILL no one passed. Ever.

The audiophiles had such massive failure that they refused to take part in testing. This has been the situation for about 30 years now. They know to not be involved in any testing as they know they can't pass the test. So, that has in effect stopped much of the testing nationwide. It is easier to say I don't believe in your tests than to test and fail. Especially if you are a recognized audiophile working at reviewing equipment, tweaks and snake oil.

All this to say, we can measure (as Amir does) a lot of stuff that is different between equipment, wires etc, but none of it is audible to the human hearing system. So, you have to remember, the human hearing system is lousy. No matter how great you think yours is, it is not that good. That is for many reasons including the ear brain interaction. So in the end, in the last 20 years all of the old disproved theories have resurfaced again and are being spouted by YouTube video producers, magazines and snake oil manufacturers. Many of the snake oil producers are well respected manufacturers of audio equipment/tweaks. So what you stated about thousands of audiophiles cant be wrong (my paraphrasing), is really nothing but bringing up old disprove ideas yet once again. Just remember if ANY sight or "tell" of any kind is involved the test is worthless.

 

[Gorgonzola]

Ok, I changed the original post to show what I wanted to comment on. First off, the statement that thousands of audiophiles believing something is "near proof" of its validity. This is wrong. Dead wrong. This is the entire reason that DBT exists. We know for a fact that after thousands of tests over the last 40 years that not one of the audiophiles who was tested could pass ANY test. Many of these tests were set up without the extreme engineering level that many want on this forum. Because the tests were done to see if they could tell a difference between two products and not for product development which is the engineering side of testing. These tests, were done to see what if any, products in the audio chain made a difference that was audible to an audiophile. With no one ever passing a test, it is what I call settled science. If in the future, if someone can pass a test, then it will be thrown wide open on that piece of wire, interconnect etc.

The testing I was involved in was far simpler than what is talked about on this forum. These tests were done with people who knew that the difference in sound was so obvious, anyone could hear it. It was just a settled issue across the country (by reading magazines at the time) that they would fly through the test and easily pass. This would once and for all show the darn testing people, that people could and did hear differences all the time and did it very, very easily. Since we had a 100% failure rate during the testing, we skewed the tests to give the product or idea the maximum help we could. The tests ended up being what our engineers on here would call "whacky", we figured well, certainly they could pass the tests now. STILL no one passed. Ever.

The audiophiles had such massive failure that they refused to take part in testing. This has been the situation for about 30 years now. They know to not be involved in any testing as they know they can't pass the test. So, that has in effect stopped much of the testing nationwide. It is easier to say I don't believe in your tests than to test and fail. Especially if you are a recognized audiophile working at reviewing equipment, tweaks and snake oil.

All this to say, we can measure (as Amir does) a lot of stuff that is different between equipment, wires etc, but none of it is audible to the human hearing system. So, you have to remember, the human hearing system is lousy. No matter how great you think yours is, it is not that good. That is for many reasons including the ear brain interaction. So in the end, in the last 20 years all of the old disproved theories have resurfaced again and are being spouted by YouTube video producers, magazines and snake oil manufacturers. Many of the snake oil producers are well respected manufacturers of audio equipment/tweaks. So what you stated about thousands of audiophiles cant be wrong (my paraphrasing), is really nothing but bringing up old disprove ideas yet once again. Just remember if ANY sight or "tell" of any kind is involved the test is worthless.

I'll keep it short.

Ian Masters in 1987 effectively proved that sometime differences could be heard between specific amplifiers, so no need to belabor the fact that DBT, (not double blind in the sense of drug testing, BTW), have proven that people can never heard differences -- because sometimes they can.

As for claims by some audiophiles that the can hear a difference between, e.g. on between on brand of non-inductive resistor and another brand, do not have to be taken seriously. Evoking such an example is a straw man argument as a basis for saying that difference can never be heard.

So we fundamentally disagree about the relevance of the consistent impressions of experienced audiophiles. You'll grant that I didn't say that even extensive anecdotal evidence is proof . What I said (or clearly implied) is that such evidence indicates legitimate avenues for investigation. Beware that any scientific test, especially the simple ones, thought they meet scientific rigor and be replicable, may be too limited in scope or applicability to enlighten the big picture.