Just how legit is this person's blind test results?

[rdenney]

Yes, years and years of data. In the 70s the add copy in the magazines was filled with zeros... 0.01% distortion fighting with 0.0001% distortion claims... that was just as silly as discussing the pace rhythm and timing of a device.

At the end of the day, we either can or can not HEAR a difference. The correlation between what we measure and what we hear is getting better, but I do not believe we have 100% correlation yet.

This argument is used to defend the notion that we can't measure everything we can hear, but with each passing improvement in measurement and with each new properly controlled subjective test, the notion that emerges is just the opposite. We can measure far more than what we can hear.

I'm sorta hoping that was your point.

Rick "who does not worship measurements because they are now too precise for the needed accuracy, not because they are likely to be missing something important" Denney

 

[kemmler3D]

This argument is used to defend the notion that we can't measure everything we can hear, but with each passing improvement in measurement and with each new properly controlled subjective test, the notion that emerges is just the opposite. We can measure far more than what we can hear.

I'm sorta hoping that was your point.

Rick "who does not worship measurements because they are now too precise for the needed accuracy, not because they are likely to be missing something important" Denney

Could not agree more. In any given listening test, there is no guarantee that everything you're hearing also has a corresponding graph already published, out there on the web, for you to look at. That doesn't mean it can't be measured, just that nobody's bothered to publish the measurement yet.

If it can be recorded in a studio, it can be measured, because measuring and recording are basically the same thing. So anything "unmeasurable" is already below the SINAD of your actual program material, probably far below since most studio gear isn't quite lab-grade. Or somehow it exists outside the plane of either oscilloscopes or FFTs, in which case you have to wonder if audiophiles have some kind of extradimensional, time-traveling hearing ability.

 

[Mr. Widget]

This argument is used to defend the notion that we can't measure everything we can hear, but with each passing improvement in measurement and with each new properly controlled subjective test, the notion that emerges is just the opposite. We can measure far more than what we can hear.

First, I and most audio enthusiasts who were not first introduced to audio through the pages of ASR, was brought up believing what my biased ears were telling me and therefore believed that CD players, DACs, amps, and preamps, all sounded subtly different. And of course this has been reinforced by most of the audio press for decades. To be fair to the audio press, in the golden age of Hi-Fi none of the gear was as linear in FR or low in distortion or noise as modern equipment.

Now, with that out of the way, I am fully open to the strong possibility that our sighted bias and expectations can interfere with our perception of what we are hearing.

You mention, "each new properly controlled subjective test". So far, when I request information on such tests, I am referred to one magazine article of a blind test of amps from decades ago. The objectivists call it definitive and the subjectivists call foul. Are you aware of other blind/double blind comparisons ideally with larger groups of listeners? From your post you must be aware of many more, can you provide links or other documentation?

 

[oleg87]

First, I and most audio enthusiasts who were not first introduced to audio through the pages of ASR

The unreliability of sighted tests with respect to eyebrow-raising claims by audiophiles is hardly a concept that originates with ASR. I certainly remember reading such arguments on internet forums circa early 2000s (and as far as I'm aware, no compelling "sound quality" factor has been proposed that eludes the usual set of measurements an engineer uses to quantify the performance of an amp (or cable or dac) in the intervening decades).

 

[stringham]

If spending $50 or $5,000 on a power cable that has zero effect on the system's performance but does not present a fire risk and it makes the purchaser happy then where is the harm.

It's fraud. Fraud distorts markets and undermines trust, even when the immediate victim has no complaints.

As for fire safety: How can you trust a UL label on a product from one of these companies?

 

[Newman]

This test result will be rushed into print in Stereophile, along with a firm middle finger from Jim Austin to ASR.

 

[Mr. Widget]

The unreliability of sighted tests with respect to eyebrow-raising claims by audiophiles is hardly a concept that originates with ASR. I certainly remember reading such arguments on internet forums circa early 2000s (and as far as I'm aware, no compelling" sound quality" factor has been proposed that eludes the usual set of measurements an engineer uses to quantify the performance of an amp in the intervening decades).

Of course I agree with you when discussing the outrageous descriptions by some writers, but at ASR the most vocal posters are certain that all amplifiers that measure well sound identical. This is not a common belief outside of these threads.

It's fraud. Fraud distorts markets and undermines trust, even when the immediate victim has no complaints.

As for fire safety: How can you trust a UL label on a product from one of these companies?

I agree that it is BS, but if it is fraud, why has no one sued Transparent, Audioquest, Nordost, or the others after spending ungodly sums on these ridiculous, frivolous, and in your words fraudulent products?

 

[Mr. Widget]

FWIW: I think I have shared my opinions on measurements, blind testing, and snake oil fraud and really do not need to keep lobbing the ball back over the net, so I'll step back on this thread and watch where it goes.

 

[BDWoody]

And of course this has been reinforced by most of the audio press for decades.

To the dismay of many. Here is a quote from an interview with J Gordon Holt describing how he felt about it,:

https://www.stereophile.com/asweseeit/1107awsi/index.html#:~:text=Do you see,spreading my gospel.

"Do you see any signs of future vitality in high-end audio?

Vitality? Don't make me laugh. Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me, because I am associated by so many people with the mess my disciples made of spreading my gospel."

To be fair to the audio press, in the golden age of Hi-Fi none of the gear was as linear in FR or low in distortion or noise as modern equipment.

Maybe not, but it was likely honestly advertised based largely on it's measured performance, not on the buzzword bingo game we have today.

It was a pursuit of high fidelity...as in, accuracy to the source.

I would love to find a poster of this ad for my study:

 

[nyxnyxnyx]

I want to add another point that wasn't brought up here:

So in the post, Purr1n kinda said something to the effect of:

1) "audibly transparent threshold (around 110-120dB?)" sourced by Amir/ASR is something he made up out of nowhere, there's a lack of evidence/conclusive researches for it.
2) Other equipment like transducers and headphones will at most only have something like ~75dB of "amirNAD" so the point of "audibly transparent" is obsolete. 60dB+ is where it's already 'ok'.

I myself do understand the logic a bit, I mean, I've listened to several badly measured products and they sound fine, or even on par/not noticeable with way better measuring products (of course that depends on the setting/pairing too). I don't know what you guys think about this.

 

[majingotan]

I myself do understand the logic a bit, I mean, I've listened to several badly measured products and they sound fine, or even on par/not noticeable with way better measuring products (of course that depends on the setting/pairing too). I don't know what you guys think about this.

Badly measuring products =/ it sounds bad. They just don't represent the original source content faithfully which I don't mind personally since I have audibly transparent chain if I crave the most accurate representation on occasion

 

[nyxnyxnyx]

Badly measuring products =/ it sounds bad. They just don't represent the original source content faithfully which I don't mind personally since I have audibly transparent chain if I crave the most accurate representation on occasion

I understnad your point and I think so too. What I'm bringing up here is Purr1n thinks our standard of audibly transparent is something we kinda made up/inaccurate or hyped up rather than a real thing with backed up evidences and such.

And at this point I don't really want to search for every audio research ever to see if it's true or not lol.

 

[Newman]

Correct me if I’m wrong but Purr1n (Marv?) operates SBAF right?

Don’t know, but he’s definitely a staff member there, per his ID block.

He also has form, “blind testing himself” on 2 DACs (ODAC vs tweaked Modi 2) in 2015 and scoring 10/12 correct on day 1 and 8/8 on day 2, link.

Biased testing is a massive problem (never mind angry, biased testers). So many avenues for the desired outcome to occur, not all of them deliberate. It is difficult to give credibility to a solo, self-administered, self-designed test in these conditions.

 

[Newman]

Humour note: I see the following in the header block of the Audio Science forum on SBAF: “Excessive insistence on only very difficult to implement double blind testing results in an automatic one week ban or worse.”

So, real science is banned from their science forum.

 

[oleg87]

I want to add another point that wasn't brought up here:

So in the post, Purr1n kinda said something to the effect of:

1) "audibly transparent threshold (around 110-120dB?)" sourced by Amir/ASR is something he made up out of nowhere, there's a lack of evidence/conclusive researches for it.
2) Other equipment like transducers and headphones will at most only have something like ~75dB of "amirNAD" so the point of "audibly transparent" is obsolete. 60dB+ is where it's already 'ok'.

I myself do understand the logic a bit, I mean, I've listened to several badly measured products and they sound fine, or even on par/not noticeable with way better measuring products (of course that depends on the setting/pairing too). I don't know what you guys think about this.

A problem with boiling audio fidelity down to one number is that different sources of degradation are not equally audible or objectionable across all circumstances. Your ears can take magnitudes more 2nd order harmonic distortion without complaining than, say, 7th, and I suspect distortion that reaches audibility on transient peaks but not 95% of the audio waveform may be far less objectionable than crossover distortion generated every time the signal crosses zero.

I prefer to just err on the side of paranoia and get something that vastly outperforms my ears.

 

[Bugal1998]

Unfortunately we can't teach or legislate common sense.

True story. Just before 2008 an older friend mortgaged her paid for home to buy a vacation condo. I told her to get out of the sale because it was a bad idea, she said she would lose $50K. I told her she would be ahead if she lost $150K. She kept the condo. A few years later she had lost both houses because of that bad decision.

If someone is inclined to make bad decisions, they will make bad decisions.

I'm with you on this, we must have personal accountability. Whether the product performs as promised or not, if someone spends their money irresponsibly, that's absolutely on them.

But that in no way means it's ok for manufacturers, vendors, and sales people to promise impossible outcomes.

 

[HarmonicTHD]

Humour note: I see the following in the header block of the Audio Science forum on SBAF: “Excessive insistence on only very difficult to implement double blind testing results in an automatic one week ban or worse.”

So, real science is banned from their science forum.

You are right. It reeks.

 

[Newman]

Here is purr1n’s claim, paraphrased: “it is ridiculously easy to reliably identify audio electronics, in this case headphone amps, that have SNR better than 103 dB”.

Well, of course it is. They might have very different FR. He is assuming that both the amps have dead flat FR, so he thinks he is also testing the claim (actually a straw man claim) that all amps sound the same. As if anyone claimed that.

Here are a few missing points from his test:-

1. He needs to have the actual device samples in his hands measured, while connected to his headphones, to check that there are no measured differences that exceed commonly-understood thresholds of audibility. Show us the high-res FR plots of all 4 channels, as well as SNR, distortion. After all, if it turns out that there are such differences, then a 10/10 correct score only validates what ASR (ie audio science) would expect, and he is proving us right.
2. Is there any audible hiss or hum through the headphones with either amp?
3. A great point by Blumlein 88: if the volume pots + circuits have different channel matching, then it’s easy to tell them apart. So both channels of both amps need to be matched to the test voltage of 0.455 volts. This will almost certainly need in-circuit attenuators, and I am pretty sure it wasn’t done. Without this step, it’s just a test of potentiometer variability and its audibility. [Edit: or just do the test in single channel and save some work.]
4. It appears he simply retires to the next room between tests. One might be able to hear whether the cables are being switched by some giveaway sound or pattern to footsteps, etc. Or the time taken between kids entering and leaving the room.
5. He might not even be aware of himself subconsciously noticing whether the devices have been moved ever so slightly, indicating a swap of amp.

 

[Galliardist]

Here is purr1n’s claim, paraphrased: “it is ridiculously easy to reliably identify audio electronics, in this case headphone amps, that have SNR better than 103 dB”.

Well, of course it is. They might have very different FR. He is assuming that both the amps have dead flat FR, so he thinks he is also testing the claim (actually a straw man claim) that all amps sound the same. As if anyone claimed that.

Here are a few missing points from his test:-

1. He needs to have measurements of the two amps, meaning the actual devices in his hands, while connected to his headphones, to check that there are no measured differences that exceed commonly-understood thresholds of audibility. Show us the high-res FR plots of all 4 channels, as well as SNR, distortion. After all, if it turns out that there are such differences, then a 10/10 correct score only validates what ASR (ie audio science) would expect, and he is proving us right.
2. Is there any audible hum through either headphone?
3. A great point by Blumlein 88: if the volume pots + circuits have different channel matching, then it’s easy to tell them apart. So both channels of both amps need to be matched to the test voltage of 0.455 volts. This will almost certainly need in-circuit attenuators, and I am pretty sure it wasn’t done. Without this step, it’s just a test of potentiometer variability and its audibility.
4. It appears he simply retires to the next room between tests. One might be able to hear whether the cables are being switched by some giveaway sound or pattern to footsteps, etc. Or the time taken between kids entering and leaving the room.
5. He might not even be aware of himself subconsciously noticing whether the devices have been moved ever so slightly, indicating a swap of amp.

He actually gives us some supporting evidence to work with.
The tells.

• Magni+ was smoother and less grainy. Heretic had grainier jumpier highs. This was the biggest giveaway.
• Heretic was flatter and boring sounding, but not by much. This was the second giveaway.

I'd suggest that the second tell indicates that levels were not properly matched. I'd suggest that the first tell suggests that the Heretic was not properly driving the headphones at the somewhat loud volume level used in the test. When I did my one disciplined single blind test with amps, graininess and a slight hardening of the sound was the way I could tell the two apart at high volume, and I could not tell them apart at normal listening levels, so that chimes with me.

A test like this, even if flawed, gives a genuine researcher a starting point to answer the research question about the amps sounding different. It is at least above the level of my usual faffing about... The biggest problem here is that PurrIn shows no interest in finding out WHY he got this result. That he could tell what was happening as easily as he reports should indicate a large difference to be found somewhere.

"I can hear better than you" is not science, and if that was the only purpose of the test, it's a waste of his and our time. Has he no scientific curiosity? It's not like we can all go and join there and make any of these suggestions because we will be instantly banned from there, even if we do our best to be polite and helpful. He admits himself in the comments that all we are going to do is pick this to pieces (which outside of a very tight test environment will always happen). So what is the point, apart from massaging his own ego that he can hear better than another researcher?

As well as the tests you suggest, a null test from electrical samples at the outputs of the amps would tell us quickly much of what is going on, and it strikes me he should have the wherewithal to do that much.

Over to PurrIn, if he reads this far. If he's genuine, he can do any or all of the things suggested to reinforce this result. We haven't just been negative here, we've also suggested various ways forward.

 

[ROOSKIE]

Of course I agree with you when discussing the outrageous descriptions by some writers, but at ASR the most vocal posters are certain that all amplifiers that measure well sound identical. This is not a common belief outside of these threads.

Howdy, I think that is said by some but that would not be a controlled enough statement.

Amps of course can sound different. There are amps that don't even meet an average speakers +/-3db 20hrz-20khrz frequency spec and amps that ruin straight out of gas in the low frequencies. Even with very nice products there are amps that double their power from 8ohms to 4ohms and even again to 2ohm and amps that produce very little extra power when impedence drop below 6 or 8ohms.
I am sure you and others get that but so what is being claimed is that when amps are objectively neutral to the threshold of audibility and they are operating within their power output ability they should be so identical that humans can not tell them apart.

The question then is what is the threshold of audibility? The 'what' and then we really need the 'where' and 'why'.
In lab or extreme critically focused conditions it is likely folks can pick out traits they will not pick out ever when listening for enjoyment - and I postulate the ultra minute differences increase said enjoyment by zero. (in fact straining to hear them may in fact decrease actual enjoyment)

So what are the essential absolute limits of audibility and beyond that at what point is 'audibility' so outside of the artform and experience of media playback that straining to occasionally hear something is utter stupidity ----------> what is 'the edge of practical limits' when listening at home or creating media.

Common beliefs are not truths, it was once most common to think the world was flat, it was once very common in many cultures to see a wife as one's personal property. It was once common for some of us in the past to believe in Santa Claus and the Tooth fairy. Christopher Columbus discovered the West, thinking heat 'rises', I found out Napoleon was not actually short. There are many common beliefs that are simply not true enough. Common beliefs are very often sh7t stories.

And technology changes. It was very recently once common to have a rotary phone that one rented from the local phone provider. Now many of us have a near supercomputer in our hands that has a direct near speed of thought wireless connection to actual supercomputers dishing out the great library of info and connection to others.

Any company who wants can measure their gear to perfection, make a well controlled blind test, pass the blind test showing differences exist between truly neutral measuring amps and publish that result. Nobody does, why?? Deep down we surely know why, they simply can't pass. Or they have resorted to making amps with house sounds and/or flaws that are distinguishable and not audibly neutral.