Master Thread: Are Measurements Everything or Nothing?

[Chagall]

Collecting subjective data and using statistics to transform it into objective evidence (Harmon curve?) doesn't mean it will apply to me or any other singular individual. It's just a group tendency. I would rather look to objective performance data first to identify well engineered devices, then demo them in my space to see if they meet my subjective preferences.

That’s fine, but I was responding to your point that judgments aren’t a metric.
Subjective judgments under controlled conditions - and that part is crucial - become objective evidence.

It's not that statistics or quantity make it evidence, because a million sighted impressions don't mean as much as 20 blind controlled ones.

 

[danadam]

Collecting subjective data and using statistics to transform it into objective evidence (Harmon curve?) doesn't mean it will apply to me

Subjective data can be collected only from you and then the statistics of it will apply to you.

And evidence of what is Harman curve supposed to be? It's just averaged preference afaiu.

 

[goryu]

Subjective data can be collected only from you and then the statistics of it will apply to you.

And evidence of what is Harman curve supposed to be? It's just averaged preference afaiu.

I think we are saying the same thing here...

 

[goryu]

That’s fine, but I was responding to your point that judgments aren’t a metric.
Subjective judgments under controlled conditions - and that part is crucial - become objective evidence.

Only if those judgments are mine and applied only to me.

It's not that statistics or quantity make it evidence, because a million sighted impressions don't mean as much as 20 blind controlled ones.

the quality of the test controls would certainly be of importance but subjective measurements are still subjective and while the sample size should allow convergence about a mean, there is never any guarantee that my particular preferences will not be an outlier.

 

[NTK]

Subjective data can be collected only from you and then the statistics of it will apply to you.

And evidence of what is Harman curve supposed to be? It's just averaged preference afaiu.

No. It is not an average. It is the curve preferred by 64% of the listeners in the tests. If you say it is an average, what is it an average of, and what is the standard deviation?

 

[Pearljam5000]

This is terrible logic. Let me reframe your statements to bare the illogic.

If measurements were everything and everyone was willing to learn it
Then everyone would buy speakers without even hearing them at all and would just rely on the measurements... indeed
Which is not the case because huge numbers of people resist the truth even when explained with all the supporting data, and the education/information channels are rife with misinformation, both wilful and ignorant, resulting in mass confusion and ignorant belief in falsehoods
So measurements... Aren't everything even when they are

cheers

That's not my logic
That's your interpretation of my logic
My logic is yes the measurements are a good base
But it's it not everything
Because if it was there would be no need to demo any audio component ever
And yet that doesn't happen for some reason

 

[Chagall]

Only if those judgments are mine and applied only to me.

the quality of the test controls would certainly be of importance but subjective measurements are still subjective and while the sample size should allow convergence about a mean, there is never any guarantee that my particular preferences will not be an outlier.

But that doesn’t make the evidence any less objective. Controlled tests don’t dictate what you personally will prefer - they establish what is reliably detectable or preferred across listeners. Of course, individual preferences or perceptions can be outliers; that’s always true for both subjective impressions and objective evidence.

For example, say we run a hypothetical double-blind test comparing Amp A and Amp B, and 70% of listeners consistently judge Amp A as sounding better. That is objective evidence that most people perceive Amp A as preferable, and you will probably like it more too.

 

[MAB]

Only if those judgments are mine and applied only to me.



the quality of the test controls would certainly be of importance but subjective measurements are still subjective and while the sample size should allow convergence about a mean, there is never any guarantee that my particular preferences will not be an outlier.

If testing subjects for statistically relevant preference, then yes the result preferred by the population may not be the same as your preference.

If (as is more often the case) testing subjects for detectable differences, then it is likely that the results do apply to you. Unless your personal hearing is much better than the hearing of the study's subjects, or much worse, which is a bit of a corner case study then.

Actually, a third possibility is tests like ABX tests aren't for everybody, need a ton of concentration to perform especially if the differences are small, and some people just can't complete the test with rigor from start to finish. I sometimes get flat out bored, loose interest... In which case a null result means nothing.

 

[Josh83]

Since the “How Can DACs have a sound signature?” thread is locked, I thought I’d mention here that I just acquired a Van Alstine ABX box. I bought the ABX box specifically to see if I can hear the difference between DACs. I have several DACs on hand, including the Matrix XSP, Gustard X30, RME ADI-2, and Holo May, and I plan to write up my results— regardless of what they show — over at Audiophile Style.

 

[solderdude]

Since the “How Can DACs have a sound signature?” thread is locked, I thought I’d mention here that I just acquired a Van Alstine ABX box. I bought the ABX box specifically to see if I can hear the difference between DACs. I have several DACs on hand, including the Matrix XSP, Gustard X30, RME ADI-2, and Holo May, and I plan to write up my results— regardless of what they show — over at Audiophile Style.

Just make sure the DACs are level matched (each channel within 0.1dB (1% using a voltmeter) using a dB meter or an ADC that can resolve those small level differences) and there is no difference in time delay to prevent tells.
Depending on the selected filter in the DAC there can be small timing differences that might become a clue (a tell).
This can be solved by allowing a small delay between switching moments as audible memory is very short.
Ensure there are about minimum 16 'switching moments'.

Tip: spread the test over 4x 4 'switch moments' and take a break in between those series of 4. This prevents listening fatigue. It is really difficult to concentrate when differences are inaudible and you have to make a decision each time.

 

[Josh83]

Just make sure the DACs are level matched (both channels within 0.1dB using a meter or an ADC that can resolve those small level differences) and there is no difference in time delay to prevent tells.
Depending on the selected filter in the DAC there can be small timing differences that might become a clue.
This can be solved by allowing a small delay when switching.
Ensure there are about minimum 16 'switching moments'

Thanks for the tips.

The ABX box has a passive attenuator that includes 0.1 dB increments. I’m going to run the output of the ABX box into my Cosmos ADC and REW, play a 1 kHz test tone, and match the levels. Early measurements has made me confident I can match the two DACs exactly down to 0.1 dB.

The same USB->SPDIF converter will be feeding both DACs. One will get the AES output, and the other will get the SPDIF. (I don’t think that makes any difference sonically, but I suppose some folks might.)

When available on a DAC, I always choose a linear phase sharp rolloff filter. On DACs without that, like the Holo May, I prefer to use NOS and software upsampling. I plan to try 24/96 audio and software upsampling for some tests to avoid the possibility that filter rolloff will be a factor, too.

When you say 16 switching moments, do you mean trials? The Van Alstine box’s ABX program has 8 trials per run. So it seems I need a minimum of 7/8 for statistical validity in a single run, but I can obviously use multiple runs to increase statistical power.

 

[IPunchCholla]

That's not my logic
That's your interpretation of my logic
My logic is yes the measurements are a good base
But it's it not everything
Because if it was there would be no need to demo any audio component ever
And yet that doesn't happen for some reason

It does happen. I’ve bought ALL of my current gear without demoing it. Enough is measured, that my room is the biggest unknown. I’d be comfortable spending 10k without a showroom demo, since I understand the measurements. However, if it was free, I would demo them in my room, since I am not 100% sure of my own preference in my room. My current speakers are MTM and I switch orientations occasionally.

 

[MAB]

It does happen. I’ve bought ALL of my current gear without demoing it.

Me too. Sighted sales often cause or increase confusion.

 

[ahofer]

Me too. Sighted sales often cause or increase confusion.

It’s fine with electronics. And measurements are a great way to triage speakers to audition.

It’s one thing to say your ears are the final arbiter, but another to believe that something with distinctly inferior measurements *might* somehow be superior. The latter seems very unlikely to me.

 

[guy48065]

One has little choice but to trust one's own ears to make one's own final choices. The important point is one shouldn't trust the ears of others to make one's own choices.

Some of us have found designers, manufacturers, or reviewers we trust explicitly.
The guy who designed my (previous) speakers I've loved for 40 years I would (and have) bought anything he produces. I absolutely know his talents, taste, and priorities and have spent many hours in his sound room.

 

[solderdude]

Thanks for the tips.

The ABX box has a passive attenuator that includes 0.1 dB increments. I’m going to run the output of the ABX box into my Cosmos ADC and REW, play a 1 kHz test tone, and match the levels. Early measurements has made me confident I can match the two DACs exactly down to 0.1 dB.

neat.

The same USB->SPDIF converter will be feeding both DACs. One will get the AES output, and the other will get the SPDIF. (I don’t think that makes any difference sonically, but I suppose some folks might.)

When available on a DAC, I always choose a linear phase sharp rolloff filter. On DACs without that, like the Holo May, I prefer to use NOS and software upsampling. I plan to try 24/96 audio and software upsampling for some tests to avoid the possibility that filter rolloff will be a factor, too.

Even when applying the same signal 2 different DACs may not output the signal at exactly the same time. This can turn out to be a tell.
There is always some delay between the applied digital signal and the actual conversion due to filter lengths, buffering etc.

When you say 16 switching moments, do you mean trials? The Van Alstine box’s ABX program has 8 trials per run. So it seems I need a minimum of 7/8 for statistical validity in a single run, but I can obviously use multiple runs to increase statistical power.

Yes, do 2 runs of 8 and add the scores. If needed split each run of 8 in half (take a break after 4 trials) if the scores indicate this.
7 out of 8 is not the same as 14 out of 16 despite it beingf the same percentage (so I am told)

 

[MAB]

It’s fine with electronics. And measurements are a great way to triage speakers to audition.

It’s one thing to say your ears are the final arbiter, but another to believe that something with distinctly inferior measurements *might* somehow be superior. The latter seems very unlikely to me.

I fortunately have measured my room.

I unfortunately don’t have a speaker carousel so no way to audition in a meaningful way.

I listen to music on speakers at shows and stores (and friend’s) for fun.

I certainly use measurements to triage speakers.

Edit: mangled my initial attempt at this post!

 

[Jim Taylor]

Measurements show us what an item objectively does. Listening tells us how much we subjectively like it. Two entirely different things.

If all components were perfect, there would be no need for listening tests .... but they're not. Speakers, especially, deviate from perfection. They also differ one from another in bandwidth, distortion, spectral balance and audible interactions with the room.
So the measurements tell us how close a speaker comes to the ideal (or how miserably it fails). Even if ten different speakers approach the ideal rather closely, they still fall short in relation to technical perfection. That's why some people consider listening before purchase to be important.

If you lined up ten gorgeous gemstones, you might agree that all of them were beautiful. It would be obvious, however, that there would be some differences. The one that you pick would probably not be the one that someone else picked.

So it goes with all of us.

 

[Josh83]

Even when applying the same signal 2 different DACs may not output the signal at exactly the same time. This can turn out to be a tell.
There is always some delay between the applied digital signal and the actual conversion due to filter lengths, buffering etc.

The ABX box itself inserts a short “tick” when switching, so I’m not sure I’d notice any small hypothetical delay imposed by a DAC’s filter. Plus, since I have so many DACs on hand and plan to test numerous pairs, if one had some sort of noticeable filter delay, I could just exclude it from my testing.

 

[goryu]

But that doesn’t make the evidence any less objective. Controlled tests don’t dictate what you personally will prefer - they establish what is reliably detectable or preferred across listeners. Of course, individual preferences or perceptions can be outliers; that’s always true for both subjective impressions and objective evidence.

But it's what I personally prefer that matters to me and as you agree, there can always be outliers- maybe the guy asking me my opinion about a set of speaker is an outlier. Maybe I am... Thus, I can not rely on such evidence. This kind of data may be valid in gross, but I don't find that reliable for anyone in particular. I just don't see much confidence in its predictive power regardless of how it was obtained. I am at a loss to understand the logic behind taking subjective data in a controlled way, summing it, and then saying the sum of a sample of subjective data has now been transformed into an objective result....

For example, say we run a hypothetical double-blind test comparing Amp A and Amp B, and 70% of listeners consistently judge Amp A as sounding better. That is objective evidence that most people perceive Amp A as preferable, and you will probably like it more too.

I would rather call it evidence that 70% of listeners subjectively preferred A to B. Not sure how you can make the leap that I would "probably" like it too- I don't buy equipment on a "probably".