Master Thread: Are measurements Everything or Nothing?

[JiiPee]

I think it would simplify and clarify our discussion, if we could agree what the various testing methods actually mean:

A properly executed blind test with one listener reveals listeners opinions regarding the tested sound reasonably reliably, as other factors that could affect his opinions are carefully controlled.

Similarly executed tests with a larger group of participating listeners reveal reasonably reliably something about peoples preferences generally speaking.

Properly executed measurements with high quality instruments like APX 555 and Klippel reveal reliably how the the tested piece of equipment performed in the used test cases.

If I like A better than B in a properly controlled blind test, and buy A for myself, does that mean that A is categorically better than B ?

If then properly controlled blind tests by large group of people show that a clear majority of people prefer B over A, does that mean that my choice was a wrong one ?

If then properly executed measurements show that B performed better than A in most tested areas, should I lower my head in shame and quietly get rid of A ?

 

[MattHooper]

Ok. Fair enough. There’s enough interesting commentary to chew on without us turning this into a Matt/Newman analysis.

back to the subjects at hand…

 

[Blumlein 88]

I think it would simplify and clarify our discussion, if we could agree what the various testing methods actually mean:

A properly executed blind test with one listener reveals listeners opinions regarding the tested sound reasonably reliably, as other factors that could affect his opinions are carefully controlled.

Similarly executed tests with a larger group of participating listeners reveal reasonably reliably something about peoples preferences generally speaking.

Properly executed measurements with high quality instruments like APX 555 and Klippel reveal reliably how the the tested piece of equipment performed in the used test cases.

If I like A better than B in a properly controlled blind test, and buy A for myself, does that mean that A is categorically better than B ?

If then properly controlled blind tests by large group of people show that a clear majority of people prefer B over A, does that mean that my choice was a wrong one ?

If then properly executed measurements show that B performed better than A in most tested areas, should I lower my head in shame and quietly get rid of A ?

Mostly blind tests are to determine if the difference heard is really heard. Then you can go from there. So many audiophile beliefs heard turn out not to be audible. The other common situation is people believing something is audible which doesn't show up in measurements. So they believe no explanation for why it is unlikely. A blind test is to confirm audibility or not.

 

[Newman]

I posted a rather lengthy reply. It was quite preachy. I deleted it. It's not my business to preach to people.

If you want to continue, then continue. If you want to stop, then stop.

Jim

We have had our answer to that question of “when is he going to stop” for quite some time now. In his defence I conclude that personal blind spots are there for a reason, and that opening to new learning in those blind spots is never going to be easy. But making it painful for the teachers is where I draw the line. This narcissistic carry-on has dragged multiple audio luminaries and countless well-meaning members into an unnecessary vortex, for years now.

“So it goes.” - Kurt Vonnegut​

Cheers

 

[JiiPee]

Mostly blind tests are to determine if the difference heard is really heard. Then you can go from there. So many audiophile beliefs heard turn out not to be audible. The other common situation is people believing something is audible which doesn't show up in measurements. So they believe no explanation for why it is unlikely. A blind test is to confirm audibility or not.

Sure, I agree on that. My post was just a attempt to clarify (or maybe muddle further ) the role of testing & measuring when trying to decide what to buy.

 

[JaMaSt]

No, it doesn't. For the second (or third) time, I offer to our members .....

You might be interested in this.

 

[ThomasD]

Welcome!

You'll find that it's not just that thread, but pretty much the whole place. For most of us, we are hoping the gear doesn't really have a sound, at least I am. I just want to hear the music, not the music with additions or subtractions from gear that gets in the way, rather than just doing that job well.

I moved your post here, where the discussion of this topic is never-ending.

I do not quite agree with the moving of my posts. I only wanted to state that for me the Topping L30II maybe does not sound as superior as Amirs measurements showed and that those measurements maybe not tell the whole truth about this Topping product. I thought thread was about the Topping L30II and not only the measurements?

Note. My first Magni and my loved Devialet Expert (it has served me well since 2013) does not sound as bad as Amirs measurements indicate. At least not for me and my favourite records. But if you a priori say gear has no sound and measurements show all I understand you.

 

[Galliardist]

I do not quite agree with the moving of my posts. I only wanted to state that for me the Topping L30II maybe does not sound as superior as Amirs measurements showed and that those measurements maybe not tell the whole truth about this Topping product. I thought thread was about the Topping L30II and not only the measurements?

Note. My first Magni and my loved Devialet Expert (it has served me well since 2013) does not sound as bad as Amirs measurements indicate. At least not for me and my favourite records. But if you a priori say gear has no sound and measurements show all I understand you.

The Devialet Expert amp section measures poorly, but with easy to drive speakers, the one Amir measured would be fine. The drop in power into harder loads was what got it a headless panther, rather than the DAC section or its ability into easier loads, so with an easy to drive speaker it should be OK. Measurements, and ratings/panthers, should always be taken in context. All comments about gear having no sound, or whatever, will be based on the assumption that the product is being used within its performance envelope - even a seemingly pristine product like the L30 II can be misused, of course.

You made some claims about other DACs sounding different in a review thread for a product,, which is almost certainly why that post was sent here. A review thread is meant to be about that product and as such is sometimes more heavily moderated than some other threads.

Your post also mentioned that you modified amplifiers to reduce feedback and that that improved their sound. That claim will not normally fly with the experts here unless backed up with a well described example of what you did and appropriate testing: that kind of claim too will bring posts here. Be prepared for some hard questioning.

It's easy for a newcomer to stumble into things like this, stick with us and learn the ropes and there is a wealth of knowledge here to be gained (and challenged, of course: but do look around here first and learn to read the room!). Welcome to ASR.

 

[MaxwellsEq]

I do not quite agree with the moving of my posts. I only wanted to state that for me the Topping L30II maybe does not sound as superior as Amirs measurements showed and that those measurements maybe not tell the whole truth about this Topping product. I thought thread was about the Topping L30II and not only the measurements?

Note. My first Magni and my loved Devialet Expert (it has served me well since 2013) does not sound as bad as Amirs measurements indicate. At least not for me and my favourite records. But if you a priori say gear has no sound and measurements show all I understand you.

Your post is entirely why this thread exists. There is no "magic something" in electronics that can not be measured, but is audible.

Don't take the move of your post personally, many other posts with exactly the same starting point as yours get moved here.

 

[Chagall]

I do not quite agree with the moving of my posts. I only wanted to state that for me the Topping L30II maybe does not sound as superior as Amirs measurements showed and that those measurements maybe not tell the whole truth about this Topping product. I thought thread was about the Topping L30II and not only the measurements?

Note. My first Magni and my loved Devialet Expert (it has served me well since 2013) does not sound as bad as Amirs measurements indicate. At least not for me and my favourite records. But if you a priori say gear has no sound and measurements show all I understand you.

Not to talk for the admins, but:

1. 90% of these kinds of posts are the result of sighted bias (expensive gear = better sound) - no bad intentions from the poster just lack of a controlled test on the poster side
2. 9% are trollers and users with bad intentions to stir up some sh***t
3. less than 1% - defective gear and genuine differences

So the device in question isn't the thing that matters.

 

[ahofer]

I own the Magni 2 uber, Magni 3+ and Topping L30II. I think all three are very good headphone amps. But I prefer the Magni due so slightly less crisp sound and a round more airy high frequencies for examples from cymbals. The uber is slightly warmer compared with the 3+ .

I am a bit suprised how much emphasis on this thread is put on measurements but less so on sound, Long time ago I tweaked amplifier stages by changing components and decrease the feedback. In my opinion this improved sound but measurements got worse. What I understand today's op amps have a lot of feedback, or?

Maybe our ears are a better judge than some measurements?

Many of us have had the experience of increasing listening controls and seeing how dispositive measurements are. Certainly the research literature supports that idea as well.

My own view is that they are comprehensive for electronics and a very useful triage for speakers (ie if they measure poorly no need to audition).

This is one of the few places on the internet where I am not in a tiny minority, despite this being the more obvious science-based reasoning path.

Descriptions of subjective listening impressions from uncontrolled tests in different acoustic environments just don’t seem very useful to me.

 

[solderdude]

Measurements do not say much about perceived 'sound quality'.
Measurements can say something about signal fidelity which is an entirely different matter and what ASR is mostly about.
In general, measurements have to be really bad for that to become audible in well performed blind tests (which are very difficult to do properly).
There usually isn't any clear relation with subjectively determined 'sound quality'.

 

[ThomasD]

The Devialet Expert amp section measures poorly, but with easy to drive speakers, the one Amir measured would be fine. The drop in power into harder loads was what got it a headless panther, rather than the DAC section or its ability into easier loads, so with an easy to drive speaker it should be OK. Measurements, and ratings/panthers, should always be taken in context. All comments about gear having no sound, or whatever, will be based on the assumption that the product is being used within its performance envelope - even a seemingly pristine product like the L30 II can be misused, of course.

You made some claims about other DACs sounding different in a review thread for a product,, which is almost certainly why that post was sent here. A review thread is meant to be about that product and as such is sometimes more heavily moderated than some other threads.

Your post also mentioned that you modified amplifiers to reduce feedback and that that improved their sound. That claim will not normally fly with the experts here unless backed up with a well described example of what you did and appropriate testing: that kind of claim too will bring posts here. Be prepared for some hard questioning.

It's easy for a newcomer to stumble into things like this, stick with us and learn the ropes and there is a wealth of knowledge here to be gained (and challenged, of course: but do look around here first and learn to read the room!). Welcome to ASR.

For the moment I drive my Confidence C2 signature with my Devialet. Do not know how difficult they are but they need a lot of power/current to open up. My old Primare I32 did not manage that.
What I know I never stated anything about different DAC sounds?? I have limited experience as I mainly run analog records. I only said headphone amps can, according to me, sound and work differently.

I am no expert!! My records of building and modifying amps and TV sets goes a long time back about 50 years or so to the time of my studies at Chalmers University of technology/ Electronics. The first amp I build was tube amplifier and it became a real disaster. At that time amplifier circuits were quite simple and it was easy to get hold of a circuit diagram and make calculations and change components. Amplifiers I studied at that time were NAD, Tandberg, Braun, Kenwood (?), Philips, Audio Pro . I must admit I have forgotten details about the feedback changes on a Japanese amplifier I made. It was done in 1975. Today Everything is so much more complicated and modifications/reparations are impossible for me. Its the same with electronics in modern cars

 

[ThomasD]

Your post is entirely why this thread exists. There is no "magic something" in electronics that can not be measured, but is audible.

Don't take the move of your post personally, many other posts with exactly the same starting point as yours get moved here.

I always said "to measure is to know" but I still believe correct and complete measurements can be a complicated subject. Magic I do not believe in!

 

[MaxwellsEq]

but I still believe correct and complete measurements can be a complicated subject.

I'm an engineer. I worked in recording studios. Competency in measuring audio and electronics is a fundamental skill for myself, my colleagues and every engineer in the media industry. I'm absolutely confident I could have measured a piece of audio gear, then let a colleague measure the same piece of gear and she would have got exactly the same results. Reliable, professional measurement is the only way that a massive studio and production facility with 1000s of ins and out could ever work reliably.

We've known for a century what measurements are needed to test good audio gear. Even 50 years ago we could measure bandwidth wider than we can hear and distortion lower than we can hear. Modern gear can not just measure noise figures lower than we can hear but also show noise spectra in real time; permit multitone modulation etc. Perfectly competent gear can be had for several hundred dollars.

Magic I do not believe in!

Probably a good thing hereabouts

 

[Axo1989]

The notion of conducting blind testing at home suffers from the same hubris as speaker measurements at home: experimental error becomes a very big issue.

The risk is too high that the experiment itself was not conducted and analysed with sufficient expertise to warrant the confidence level the home tester applies to the results.

Certainly on this forum we can discount any arguments about audio science itself that are leaning on home blind testing for their validity.

You know that our august and generous host (and others like Erin) conduct their tests at home, yes?

Your proposed heuristic "at home" = "hubris" is ragged and self-serving. It isn't a serious proposition though, and we can certainly discount such rhetoric. Testing may be done sufficiently well, or not, regardless of whether the setting is residential. What discussions usually do here when these things come up is to go into the details of the testing and validity of consequent claims. Which may stand up to scrutiny, or not. Which is as it should be. Better that than constantly falling into a credentials fallacy.

 

[MattHooper]

You know that our august and generous host (and others like Erin) conduct their tests at home, yes?

Your proposed heuristic "at home" = "hubris" is ragged and self-serving. It isn't a serious proposition though, and we can certainly discount such rhetoric. Testing may be done sufficiently well, or not, regardless of whether the setting is residential. What discussions usually do here when these things come up is to go into the details of the testing and validity of consequent claims. Which may stand up to scrutiny, or not. Which is as it should be. Better that than constantly fall into a credentials fallacy.

I like the cut of your jib!

A nice summation of blind testing, be it at home or not.

I feel seen.

 

[pablolie]

You know that our august and generous host (and others like Erin) conduct their tests at home, yes?

Your proposed heuristic "at home" = "hubris" is ragged and self-serving. It isn't a serious proposition though, and we can certainly discount such rhetoric. Testing may be done sufficiently well, or not, regardless of whether the setting is residential. What discussions usually do here when these things come up is to go into the details of the testing and validity of consequent claims. Which may stand up to scrutiny, or not. Which is as it should be. Better that than constantly falling into a credentials fallacy.

I find this a bit of a spaghetti-wall proposition. Are you validating home testing or not, and if so, with what purpose?

To me it remains simple. Competent measurements in a controlled environment tell me about fundamental design goals and virtues that were met.

They tend to tell me little about what stuff will do in my home environment. With tower speakers because they'll need correction no matter what. With bookshelves because no test reveals optimally measured sub xover frequencies and such.

 

[MattHooper]

I find this a bit of a spaghetti-wall proposition. Are you validating home testing or not, and if so, with what purpose?

It seems to me axo is making a point similar to the one I was making. It’s certainly true that skilled scientists are more likely to produce experiments, like blind testing gear, without experimenter error vs your average audiophile at home. (but of course the scientists work also has to be checked for errors!! They don’t get a pass just because they are professional scientists).

However, ultimately any test should be scrutinized on its merits, not on who happened to be doing the test and where (otherwise you start moving towards Argument from authority type mistakes).

And, as I’ve said, just as scientists themselves may, after scrutiny , differ over the merits of some peer’s experiments and results, if a blind test is presented here it can be scrutinized for its merits, and then certainly ASR members can decide for themselves how much merit to put in the test and results.

We don’t get absolute certainty in this world, and if somebody wishes to scale down their confidence in an ASR member’s blind test… fine and dandy. But they can’t necessarily speak for everyone else.

 

[pablolie]

It seems to me axo is making a point similar to the one I was making. It’s certainly true that skilled scientists are more likely to produce experiments, like blind testing gear, without experimenter error vs your average audiophile at home. (but of course the scientists work also has to be checked for errors!! They don’t get a pass just because they are professional scientists).

However, ultimately any test should be scrutinized on its merits, not on who happened to be doing the test and where (otherwise you start moving towards Argument from authority type mistakes).

And, as I’ve said, just as scientists themselves may, after scrutiny , differ over the merits of some peer’s experiments and results, if a blind test is presented here it can be scrutinized for its merits, and then certainly ASR members can decide for themselves how much merit to put in the test and results.

We don’t get absolute certainty in this world, and if somebody wishes to scale down their confidence in an ASR member’s blind test… fine and dandy. But they can’t necessarily speak for everyone else.

OK but your argument seems completely circular without actually making a. point one way or the other. Is that by design?