Why is audio objectivism so frequently focused on all the wrong things?

[restorer-john]

I hope @GrimSurfer's ban is temporary and he feels that he can return.

I hope so. @GrimSurfer may have other distractions or issues that may have caused the out of character outbursts.

 

[Mikey]

It is perfectly okay to prefer the triode sound. Claiming it is of superior fidelity because it meets your preference is wrong.

I want to be clear here, my critique isn't about the sound, it's about the methodology. Your test was sound(ish): You took the actual physical devices, tested them separately, and then tested them in conjunction, and were in principle able to determine whether the differences were caused by addition or subtraction. (But it was a sighted test relying on auditory memory, so obviously it's not actually proving anything.)

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

 

[DonH56]

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

Actually, that sort of methodology is very common, and relies on measurable (audible) differences when devices are inserted. That in turn depends upon the signal chain having distortion and noise below the device under test (DUT). That is often the case when SS and tube components are compared since typical SS components have order-of-magnitude lower noise and distortion than their tube counterparts. That is basically what @Blumlein 88 proved for his test setup. If that is not true, comparisons can still be made, using additional measurements (or listening tests) to de-embed the contributions of the chain and focus on the DUT. Whenever you measure anything you have to deal with the noise and distortion of the measuring device and/or everything else in the chain. Ideally low enough to be insignificant, and thus not impact the test results, but they'll never be zero.

 

[Sal1950]

I hesitate to even post this, because I know precisely what the response will be -- it's impossible to do, therefore it didn't actually happen, they must all have screwed up somehow or be lying -- and I don't know, maybe that's actually true.

That would be something you'd have to take into account and much more so on the "screwed up" than "lying". But any time anyone posts the results of tests that you know had agendas to going in, questions must be asked. For tests like this to have relevance to me there should be a couple of people with, first the technical background to check the correctness of the playback chain, and second they have the "scientific" mindset that would ensure they are doing their best to ensure correct results.
Trust but Verify must be the at the top of considerations in areas that are so guided by emotions and passions.

If they do the test blind, and get statistically meaningful results, well, now you've got yourself a puzzle to figure out.

Agreed, once the above level of detail has been applied and they show audible differences, then it's time to figure out what, at the current level of science, is being missed by known measurements. As things stand today, no undeniable results have ever been presented to make me want to believe in magic dust science.

 

[Cbdb2]

I want to be clear here, my critique isn't about the sound, it's about the methodology. Your test was sound(ish): You took the actual physical devices, tested them separately, and then tested them in conjunction, and were in principle able to determine whether the differences were caused by addition or subtraction. (But it was a sighted test relying on auditory memory, so obviously it's not actually proving anything.)

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

The point off most of most his polls is to show theres no audible difference, one thing a blind abx test is rather good at. And many are a response to people claiming supper hearing. He also often shows measurements of the different devices. Not perfect but fairly objective.

 

[BDWoody]

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

To simply wave away what this handful of very patient people are trying to break down for you as absurd is simply revealing your unwillingness to hear anything that doesn't back up your 'point.'

You are casually dismissing a lot of very well intentioned information, seemingly without considering that you may be completely wrong about all of this.

You have some very mistaken ideas. You came to the right place to get answers. Try actually processing them, rather than simply dismissing them because they are inconvenient.

 

[PierreV]

If they do the test blind, and get statistically meaningful results, well, now you've got yourself a puzzle to figure out.

Leaving aside

1) the issues with self-reported data, which will always be biased
2) the correct determination and use of p-values (almost always incorrect on audio forums)
3) the incorrect framing/understanding of a correctly obtained p=0.05 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4448847/)

In practice, assuming the above has been dealt with, the puzzle would be solved by checking the replicability of the experiment. That's what happens in science when a result point to "a puzzle" if

1) the puzzle is interesting enough
2) the field welcomes replicability
3) has funds/resources for replicability

Somehow, I think the audio is automatically disqualified by the above constraints

 

[ernestcarl]

I super-hate those kinds of "tests," because they rest on the assumption that the only difference between devices is an additive distortion.

A bit different but I was reminded yet again of no audiophile's near-field recordings of a song (which I've listened to now more times than I care to repeat):

https://www.audiosciencereview.com/...-on-axis-and-smooth-off-axis.8090/post-252479

Warning! You might super hate these types of rough listening comparisons too.

*I know they are not level-matched. If you download the files, JRiver or Foobar can do that easily after some automated analysis...

 

[Blumlein 88]

I want to be clear here, my critique isn't about the sound, it's about the methodology. Your test was sound(ish): You took the actual physical devices, tested them separately, and then tested them in conjunction, and were in principle able to determine whether the differences were caused by addition or subtraction. (But it was a sighted test relying on auditory memory, so obviously it's not actually proving anything.)

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

Can you explain why it would be an absurd method?

I've posted music made with 8 loops thru an DAC/ADC. So you can hear 8 times the normal degradation of such a process. The difference isn't much. I think it would give credence to the idea a 1st gen recording with an ADC could accurately capture what other gear sounds like. So if you had a copy of the original input file for a system, and an ADC recorded result say at the speaker terminals it would let you hear what the other system does to the sound quality. You'd miss out on what speakers do. I hardly think that is absurd.

In case you'd like to hear these 8th generation copies yourself, I still have the files up to be downloaded.

https://www.audiosciencereview.com/...-choose-the-8th-generation-digital-copy.6827/

 

[Rja4000]

The test where you take a recording of the output of sound run through a device, and then run it through another whole chain and treat that as if you were evaluating the device is methodologically absurd.

I get your point.

But, as said above by others, it all depend of the relative negative impact of the DUT vs the negative impact of the other components.

It's like in measurement, by the way:
your measurement device has to be at least 10dB better than your DUT, for the measurement to be somehow accurate.
So, possibly, blind listening an amp with 60dB SINAD still make sense if you use a 90dB SINAD amp?

Of course, it all comes back to the question:
do we measure the right thing ?
SINAD doesn't tell all.
We have to make sure our 90dB SINAD doesn't screw any of the other reasonable measurements.

 

[Rja4000]

they rest on the assumption that the only difference between devices is an additive distortion.

Well, either the signal is as the original, either it is distorted.
So, yes, the only difference is "additive" distortion, be it added by the DUT or by your measurement chain.

There are different types of distortion, though: frequency response change, dynamic compression or even level change, phase change, time-domain change should also be listed as "distortion".

 

[krabapple]

I think you are being wrongly characterized in this thread as supporting the ideas of "those bad subjectivists".

Indeed. It was eyebrow-raising to read through this and see a few posters bizarrely misconstruing what Mikey wrote...and branding him a 'subjectivist'. w.t.f. ?

 

[krabapple]

Since when is blind testing about determining preferences? It seems to me the primary benefit of blind testing is to determine if there is ANY DIFFERENCE AT ALL between a given set of options. I don't particularly care if a person "prefers" one thing over another. I want to know if they can even hear a difference between the things at all (because in so many situations where they insist they can I don't really believe it). Are $500 speaker cables "better" than lamp cord? Let's first determine if any difference can be heard at all.

Blind testing is about preference, and has been for decades in sensory and product testing, if and when A and B can reasonably be expected to sound/taste/smell differently. In the case of audio, that would be : transducers. Toole and Olive's blind tests of loudspeakers, for example, were tests of *preference*, not difference.

 

[jasonq997]

Blind testing is about preference, and has been for decades in sensory and product testing, if and when A and B can reasonably be expected to sound/taste/smell differently. In the case of audio, that would be : transducers. Toole and Olive's blind tests of loudspeakers, for example, were tests of *preference*, not difference.

I agree. I think it has clearly has utility in both cases. In the context of what we are concerned with on this website, I think most of the discussions center on the question of "difference", which likely eliminates most claimed "preferences" with regard to audio electronics.

 

[krabapple]

I am most definitely not producing a stock argument.
What I am saying is it appears you do not understand the purpose of DBT. It should have nothing to do with preference. If you set the test for preference then you've already decided that a difference is there, otherwise the test is pointless.

And I'm saying you're utterly wrong. DBT is a method that can be (and has been) applied to research into perception of difference OR preference. It is an effective way of applying controls for 'sighted' bias in either case.

 

[krabapple]

I agree. I think it has clearly has utility in both cases. In the context of what we are concerned with on this website, I think most of the discussions center on the question of "difference", which likely eliminates most claimed "preferences" with regard to audio electronics.

Erroneous claims are being made on this thread about what DBT is 'for'... even in the midst of Toole and Olive's *DBT for preference* work being referenced repeatedly.

The only reason 'difference' questions could be considered more relevant than preference on this site is because Amir mostly measures solid state/digital gear, and not electromechanical gear or 'tube' gear. (He did measure a tape deck, a class of devices which might also be usefully tested for 'preference' , but that was a rarity)

 

[Blumlein 88]

Leaving aside

1) the issues with self-reported data, which will always be biased
2) the correct determination and use of p-values (almost always incorrect on audio forums)
3) the incorrect framing/understanding of a correctly obtained p=0.05 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4448847/)

In practice, assuming the above has been dealt with, the puzzle would be solved by checking the replicability of the experiment. That's what happens in science when a result point to "a puzzle" if

1) the puzzle is interesting enough
2) the field welcomes replicability
3) has funds/resources for replicability

Somehow, I think the audio is automatically disqualified by the above constraints

Yes, good link you provided. I've thought for some time there is little point in anything that isn't a 3 sigma result. Manufacturing quality control, and established sciences such as physics go even further. Both made much better and much less ambiguous progress when they used a 3 sigma threshold. That is what allowed them to go even further. Various medical test results are all over the map saying one thing and then another largely for this reason. That and using too few subjects.

There have been recent proposals for medical and psychological results to use p equal to or less than .005 instead of .05, but I think they should go a little further. Of course if you did one at p=.05 and replicate it with another at p=.05 the two results combine for a 3 sigma result, and reduce type II errors.

 

[Rja4000]

The only reason 'difference' questions could be considered more relevant than preference on this site is because Amir mostly measures solid state/digital gear, and not electromechanical gear or 'tube' gear

I guess when differences are so preponderant that they are obvious for everyone, preference is the only way to go.
On top of this, Harman's goal was to identify a set of measurements that help them building loudspeakers that common people prefer at a cheaper price, to get a competitive advantage.

But transducers are the exception.
Why would any other component sound differently nowadays ?
(if it's not meant to add a special coloration, in which case, it should be categorized as an "effect" device)

 

[Tks]

 

[nhunt]

I guess when differences are so preponderant that they are obvious for everyone, preference is the only way to go.
On top of this, Harman's goal was to identify a set of measurements that help them building loudspeakers that common people prefer at a cheaper price, to get a competitive advantage.

I think we established that this research began before they joined Harman. I also think one of the goals was to identify common (not universal!) standards for music production and listening equipment to reduce the circle of confusion.