Is Audio Science Review going about it all wrong? Or partly wrong? Or all right?

[FrantzM]

My own epiphany came after a very famous audiophile went onto an ABX or close to ABX trying to identify his own expensive cables versus run of the mill Monster cable speaker cables. I was honestly certain he was going to ace the test and make us all subjectivist audiophiles proud. He didn’t. I replicated this test in some ways having a friend switch the cables in an order unknown to me; very approximate level matching with a radio shack SPL meter on some songs. Other audiophiles present and myself were utterly unable to distinguish my rather expensive cables from a Pair of 6 AWG electrical cables.
That was a turning point for me. I continued to test myself and many other things including electronics. For the most part they sounded the same. I find and somewhat maintain that amps, Speakers are a completely different story.
One can go somewhere and listen to various gear in presence of those who sell them. The better salespersons will influence you. Your own biases will influence in a not subtle way. Many of those biases are from difficult to impossible to remove. And don’t believe yourself thinking you can will them out. We can’t. You can’t. The best we have are ABX blind tests, they are not easy to conduct. The second best are level matched blind test. They can educate many.
I have been an audiophile for almost 50 years. The best site that I have come across is ASR. It educates challenges and tests. There are luminaries on this site. There are people whose level in things Audio and Engineering are World Elite. They share their knowledge and steer us toward better audio reproduction in our home. Long live this site.

 

[Blumlein 88]

That wouldn't be a proper visual test, just a different one. Despite all the standard explanations, I could still see the 1-pixel-wide line from a great distance. It shows that the narrative is incomplete.

Yes, it would be a different test. Why would I suggest the same test if I thought it wasn't telling you something useful? And yes your narrative about the single line is incomplete. The line isn't giving you one pixel information on an HD display it is giving 1080 pixels of information. Try what I suggested a one pixel separation between two lines or what Amir suggested which is a 1 pixel dot. You'll find your ability to see that does end well short of 30 feet.

Well, then this gets harder and harder. Is a software adjustment of volume not going to affect the sound? I'd think it would. Otherwise we'd have all software volume control, which is far from standard in the industry. I think a perfect AB is close to impossible.

You'd be incorrect on the software volume control. Pretty much any good software does it with 32 bit float, most now 64 bit. No mechanical volume control is as good. If you wish you could do it mechanically. The point being if one amp has mismatched channels and one doesn't then yes they might sound different. But that is an uninteresting difference that says little about the inherent fidelity or quality of playback.

So is it your opinion that perfect AB tests are close to impossible so just don't try and do them well? Listening to various gear with lots of music over an afternoon can be fun, and will almost always result in you hearing differences. But the conclusions are simply not reliable at all. Matching volume is not some impossible task, I've been doing it for 30 years even before digital playback with digital volume control was available. To my way of thinking taking the bother to match levels is not so much compared to spending hours reaching conclusions that are gut felt and unreliable or misleading. And sorry, but levels not being matched dooms your efforts from the start. If might not feel that way to you, but it is the truth.

 

[Blumlein 88]

To add on to this...
Could I perceive the difference between two lines separated by one pixel versus a single line two or three pixels wide? Perhaps the thickness would appear the same, but perhaps the color would seem different? The fact that I can perceive a single line from 30 feet means that the single line carries information to my eyes. How much then?

Yes a pair of lines 1 pixel wide and 1 pixel apart would appear different vs a 3 pixel wide line. A line has more information than a dot. That is what is throwing you off here.

 

[Blumlein 88]

Being relaxed at such a test makes things easier.

Yes, many complain of the pressure of doing an ABX. And I get it. Some of it is how an audiophile approaches this. It becomes a test of his manhood to show he can hear differences. Or a validation of him personally. And such stress would negatively effect performance. Especially as a common experience is someone trying blind listening with some obvious difference they aren't even worried about hearing. Only to find, "uh, uh, uh...................boy this is hard.............uh.........how could this be?" So it is a short step with blind experience feeling so completely different and 'obvious' differences disappearing altogether to think there is something wrong with this kind of testing. The difference is you didn't realize how much sight or knowing your sources influences what you hear.

If one will continue until they've enough experience to be comfortable and not stressed, they'll likely get the same result it just is more believable. Of course rarely would such people want to continue further which I also understand.

So your method of starting with differences that one can show themselves can be heard blind is a good step in the right direction I think.

 

[tpaxadpom]

My own epiphany came after a very famous audiophile went onto an ABX or close to ABX trying to identify his own expensive cables versus run of the mill Monster cable speaker cables. I was honestly certain he was going to ace the test and make us all subjectivist audiophiles proud. He didn’t. I replicated this test in some ways having a friend switch the cables in an order unknown to me; very approximate level matching with a radio shack SPL meter on some songs. Other audiophiles present and myself were utterly unable to distinguish my rather expensive cables from a Pair of 6 AWG electrical cables.
That was a turning point for me. I continued to test myself and many other things including electronics. For the most part they sounded the same. I did find and somewhat maintains that amps, power amps may sound different but for the most part they sound the same over a wide range of situations speakers and all. I insist on amps. Speakers are a completely different story.
One can go somewhere and listen to various gear in presence of those who sell them. The better salesperson will influence you. Your own biases will influence in a not subtle way. Many of those biases are difficult to impossible to remove. And don’t believe yourself thinking you can will them out. We can’t. You can’t. The best we have are ABX blind tests they are not easy to conduct. The second best are level matched blind test. They can educate many.
I have been an audiophile for almost 50 years. The best site that I have come across is ASR. It educates challenges and tests. There are luminaries on this site. There are people whose level in things Audio and Engineering are World Elite. They share their knowledge and steer us toward better audio reproduction in our home. Long live this site.

What you are saying is that bias has a significant effect on perception and cannot be eliminated in sighted comparison. As the result it is not a discussion between being audible or not but rather perception of whether it is audible. Blind testing is suited well for product development from scientific point of view and audio voodoo on the market is well justified as long as the owner gets the positive bias. Is this the world we live in? When are they going to invent the audiophile nirvana pills?
I don't understand the nature of the bias though. I get it when audiophiles are aware of product cost, but when they are not? What if neither product is attractive? What is the mechanism by which the audiophile gets the placebo effect?

1. Objectivist <=> Class A amplifier with no local feedback (enjoys distortions cause they are musical)
2. Subjectivist <=> Class AB amplifier with lots of feedback (show me the graph!)
3. Subjectivist + <=>Class D amplifier? (it is all about efficiency...)
When subject to properly setup blind test neither category can determine the difference between Class A (no feedback), Class AB nor Class D. Who is right, who is wrong???

 

[SIY]

What you are saying is that bias has a significant effect on perception and cannot be eliminated in sighted comparison.

It's not him saying that, it's over a century of experimental research into perception and sensory analysis. It's a fundamental fact.

 

[tpaxadpom]

It's not him saying that, it's over a century of experimental research into perception and sensory analysis. It's a fundamental fact.

I guess we should talk about audio perception and remove the word audible unless it is mentioned in the context of ABX test.

BTW I participate in blind test on a regular basis (research purposes). What I am trying to get out of this discussion is how blind test can help someone to perceive audio components without bias. If it is not possible then what is the point of the blind test for regular user if the results do not apply to sighted test conditions?

PS: is that an audiophile pill I was referring to in your avatar?

 

[Blumlein 88]

I don't understand the nature of the bias though. I get it when audiophiles are aware of product cost, but when they are not? What if neither product is attractive? What is the mechanism by which the audiophile gets the placebo effect?

Well it has been shown, and known for a long time people are likely to hear differences even when there are none. Now however it developed we have people who have a motto of "everything matters". Listening with a propensity to hear differences means when sighted you are likely to hear them when they aren't even there. You'll likely hear that even if you think two pieces under comparison are ugly or not very good. You'll have a propensity to hear a difference.

Now if you go with that experience, you'll continue to look for and hear more and more nuances. That is how you get things like cable risers, audible power cords, and audible interconnects. Or if you know of the propensity and that such things don't sound different, you'll still experience what sounds like a difference, but not attribute it to being real.

So blind tests don't help one perceive without bias all that much. They do perhaps mitigate actions one might take based upon bias if they know better. They also help rein in one's tendency to pursue and invent additional perceived differences that aren't there.

 

[Habu]

PS: is that an audiophile pill I was referring to in your avatar?

Edison Gold-Moulded Cylinders (1902–1912)
http://cylinders.library.ucsb.edu/history-goldmoulded.php

 

[Shadrach]

I think it's important to distinguish between blind tests and ABX tests.
They are not the same and can give differing results.
It's also important to accept that to the individual listener if they perceive a difference, be it real or imagined, that difference then exists for that listener.
This was in the end the downfall of J Atkinson. No matter how hard he tried when he sat and listened to his stereo system and stared at the quad amp it just didn't press all the necessary buttons that the other amp did (I can't remember what it was) His mistake was not to accept that it wasn't the amps performance that caused the difference in perception, it was his brain.
This is a form of arrogance and at that point if J Atkinson had proven the world flat I would want a second opinion.

 

[beefkabob]

Yes a pair of lines 1 pixel wide and 1 pixel apart would appear different vs a 3 pixel wide line. A line has more information than a dot. That is what is throwing you off here.

You (at least me) can't perceive the three colored lines that make up a white line on the screen from 3 inches .

Hopefully, you can fit that into your theory someplace.

I have a 220ppi OLED display for my laptop. On an all white area, close up I can perceive what looks like individual pixels in a grid. However, if I make a line of white on black, I can see that the edges of the white don't fall right onto the grid, with a bit of a saw pattern around it. I don't know the tech of this screen, but OLEDs aren't typically RGBRGBRGB or something so simple. There are many patterns of different colored subpixels and they're not in a grid like traditionally thought of. They're not even the same size sub pixels, or the same number per color necessarily.

With an all black background, which is pixels off on an OLED, in a daylit room, I can see a single white pixel from ~8 feet. The screen is glossy and dusty, reflecting other stuff, which doesn't help.

Everything you guys have said are different ways to see if something is perceptible. 8 feet back with a 13.3" screen is way smaller than any chart says its worth it to have a 2560x1440 display, but I can still see the pixel. In a dark room, I bet I could see it from much farther away. Will that single pixel matter in the overall brain processing of a moving image? Probably not, but I can still perceive it, despite the prevailing wisdom. And if the prevailing wisdom is wrong in video, is it also wrong in audio? There are two competing prevailing wisdoms in audio, the measurement type and the audiophile type.

I'm not wedded to either of these horses. I'm going to find an SPL I like with my speakers then measure the voltage and see what I can figure out.

 

[solderdude]

There is nothing more sobering than a well performed blind test.

Unfortunately the majority of people don't really get the 'well performed' and 'blind test' part of it all and have their own ideas of what such is.

 

[SIY]

I think it's important to distinguish between blind tests and ABX tests.

Yes indeed. All ABX tests are blind. Not all blind tests are ABX. And not all blind tests are double-blind, which is the most basic control before any experiment can be taken seriously.

 

[Blumlein 88]

I have a 220ppi OLED display for my laptop. On an all white area, close up I can perceive what looks like individual pixels in a grid. However, if I make a line of white on black, I can see that the edges of the white don't fall right onto the grid, with a bit of a saw pattern around it. I don't know the tech of this screen, but OLEDs aren't typically RGBRGBRGB or something so simple. There are many patterns of different colored subpixels and they're not in a grid like traditionally thought of. They're not even the same size sub pixels, or the same number per color necessarily.

With an all black background, which is pixels off on an OLED, in a daylit room, I can see a single white pixel from ~8 feet. The screen is glossy and dusty, reflecting other stuff, which doesn't help.

Everything you guys have said are different ways to see if something is perceptible. 8 feet back with a 13.3" screen is way smaller than any chart says its worth it to have a 2560x1440 display, but I can still see the pixel. In a dark room, I bet I could see it from much farther away. Will that single pixel matter in the overall brain processing of a moving image? Probably not, but I can still perceive it, despite the prevailing wisdom. And if the prevailing wisdom is wrong in video, is it also wrong in audio? There are two competing prevailing wisdoms in audio, the measurement type and the audiophile type.

I'm not wedded to either of these horses. I'm going to find an SPL I like with my speakers then measure the voltage and see what I can figure out.

The same reason you can see stars with an angular size too small to resolve. You are seeing the pin point of light, but you aren't resolving anything at that size.

http://curious.astro.cornell.edu/ab...uestions/679-why-do-we-see-stars-intermediate

You can see it is the light output by trying a 50% gray for the same dot on a black background. If you are resolving the size, the gray dot is the same size. You'll see it at the same distance. If you are seeing light output, you'll see the white dot further away than the gray one. You'll find the gray one goes away much closer than the white one.

 

[Soniclife]

With an all black background, which is pixels off on an OLED, in a daylit room, I can see a single white pixel from ~8 feet.

You can see light, not the size of the light. Do the test Amir described, the standard test.

 

[MRC01]

...
How does any of this translate into SINAD or THD? I don't see much of a link. I just want the music to sound good, maybe even lifelike, though that's a stretch to achieve without really big bucks.
...

In my view, ASR gets it mostly right, but not quite "all right". We are all human so getting it "all right" is a process, not a state. ASR has the best S/N ratio I've seen across various audio related forums, incredibly useful information and great well informed discussion.

...
... There is no, and I mean NO examples of anything demonstrated to be audible which is not also easily measurable.

True in principle, as long as you know what to measure, and the measurements represent actual listening conditions. But that isn't always so, and without these assumptions measurements can be misleading.

One simple example is measuring SINAD. Since Amir started measuring at 50 mV we've seen that performance at full volume (which is the standard method) is not representative of performance at 50 mV. Yet 50 mV is much closer to the actual levels we listen to on headphones. For example, consider the Yulong DA10. It measured SINAD of 112 which is in the top tier. But at 50 mV it measured only 70. Conversely, the Corda Jazz measured only 87.6 which is in the bottom tier. But at 50 mV it measured 90 which is one of the best Amir has measured.

Before we had the 50 mV measurements, if someone made a subjective observation that the Jazz sounded cleaner than the Yulong, someone would say his ears must be wrong because the Yulong measures more than 20 dB cleaner. Now we know that measurement can be misleading, and the Jazz is actually about 20 dB cleaner at normal listening levels.

Again that's just one simple example. Measurements are useful, even essential. In principle you can measure anything you hear, and more. But one must be careful what you're measuring, understand its limitations and applicability to actual listening.

 

[MattHooper]

"..even when a real difference exists, it is very difficult to produce anything but statistically null results..."

Bullshit. There's no other word for it.

But maintaining that facade despite knowing better is what has made him a millionaire.

That was a particularly silly bit of obvious question-begging.

Essentially:

"We are doing a blind test to determine if our subjective impressions are reliable and the differences are real. Well, the blind tests are showing no support for a difference, and since we know the differences actually exist due to the reliability of our subjective impressions, we can reject the blind test."

JA is a very smart man, but even smart people can conjure awfully bad arguments if it supports a personal bias.

 

[MattHooper]

My own epiphany came after a very famous audiophile went onto an ABX or close to ABX trying to identify his own expensive cables versus run of the mill Monster cable speaker cables. I was honestly certain he was going to ace the test and make us all subjectivist audiophiles proud. He didn’t. I replicated this test in some ways having a friend switch the cables in an order unknown to me; very approximate level matching with a radio shack SPL meter on some songs. Other audiophiles present and myself were utterly unable to distinguish my rather expensive cables from a Pair of 6 AWG electrical cables.
That was a turning point for me.

I have long been interested in the subject of skepticism and extraordinary claims, and so I think I've been pretty rational even about audio for a long time. However, I would say the beginning of my personal epiphany about sighted bias came in the 90's when a friend had me do a meagre "blind test" between two speakers he owned: The Quad ESL 63 electrostatics that we'd been listening to at his place for months, and a pair of Spendor speakers (BC2?..can't quite remember) he'd just acquired. He'd set them up near each other, matched levels, gave me a blindfold (night mask) and he'd switch between them asking me to identify which was which. I could in not too much time correctly identify them, but the main take away was how similar they actually sounded! Far more so than I'd ever have expected based on sighted listening. I actually had to concentrate more than I would have guessed to know which was which.

Not too much longer after that I started the occaisional blind testing of items that I acquired to test out (cd players/DACs, power cables...) and it's always been enlightening whether the results have been positive or negative.

 

[MRC01]

There is nothing more sobering than a well performed blind test.
Unfortunately the majority of people don't really get the 'well performed' and 'blind test' part of it all and have their own ideas of what such is.

On the subject of blind testing, keep in mind the difference between precision & recall. 95% confidence is a commonly used threshold. It means means false positives are only 5% likely. But it also means you will have false negatives. This is high precision, low recall. The ideal confidence threshold depends on the purpose of the test.

If the purpose is to be as sure as possible that the differences being tested really were heard, that's high precision. 95% confidence / 5% chance to get the same results by guessing is appropriate. However, this means low recall: there will be cases where subjects really did hear a difference, but the test failed to recognize it. They got it right better than random guessing, but not 95% of the time. Those are the false negatives.

If the purpose is to determine the most conservative thresholds of audibility that we should engineer in order to ensure sonic transparency, that's high recall. 55% confidence / 45% chance to get the same results by guessing is appropriate. However, this means low precision: there will be cases where subjects didn't really hear a difference, but got lucky in the test. Those are the false positives. You may end up over-engineering to lower thresholds, but the benefit is you can be statistically certain that it is transparent.

One of the problems I've seen (not necessarily saying anyone here has made this mistake) is taking the results of a high precision test and making incorrect assumptions about recall. "In this test, none of the subjects passed the 95% threshold. Therefore, the difference is inaudible." That is an incorrect assumption. If you record all the test responses, then you can apply the precision/recall tradeoff in interpreting the results. You don't have to re-test.

 

[Blumlein 88]

I have a 220ppi OLED display for my laptop. On an all white area, close up I can perceive what looks like individual pixels in a grid. However, if I make a line of white on black, I can see that the edges of the white don't fall right onto the grid, with a bit of a saw pattern around it. I don't know the tech of this screen, but OLEDs aren't typically RGBRGBRGB or something so simple. There are many patterns of different colored subpixels and they're not in a grid like traditionally thought of. They're not even the same size sub pixels, or the same number per color necessarily.

With an all black background, which is pixels off on an OLED, in a daylit room, I can see a single white pixel from ~8 feet. The screen is glossy and dusty, reflecting other stuff, which doesn't help.

Everything you guys have said are different ways to see if something is perceptible. 8 feet back with a 13.3" screen is way smaller than any chart says its worth it to have a 2560x1440 display, but I can still see the pixel. In a dark room, I bet I could see it from much farther away. Will that single pixel matter in the overall brain processing of a moving image? Probably not, but I can still perceive it, despite the prevailing wisdom. And if the prevailing wisdom is wrong in video, is it also wrong in audio? There are two competing prevailing wisdoms in audio, the measurement type and the audiophile type.

I'm not wedded to either of these horses. I'm going to find an SPL I like with my speakers then measure the voltage and see what I can figure out.

Okay here is another one you could try on a black background.

Vertical line of 1 pixel width white. You can see it from quite far off.
Two vertical lines of 1 pixel width white separated by 1 pixel of black. You can see it from far off too.
One vertical line of 3 pixels width in white.
One vertical line of 3 pixels width in 67% gray.


Up close you see a skinny white line, a pair of skinny white lines, a thick white line and a thick gray line.

You won't move back all that far until you'll see a dimmer white line,
A gray white line (actually two white lines and a black line)
A thicker white line.
A gray line that looks about the same as the two white lines with a black line between them.

The single pixel white line isn't resolved by size further back,but by light output. It looks like the gray line and the pair of white lines from a distance. You see the line, but see a brightness difference vs the 3 pixel thick white line. All the lines look to be about the same width from further back.