Master Thread: Are measurements Everything or Nothing?

[Mart68]

Fair point, that's newer than the music most ASR members enjoy, but it was the earliest stuff I could find.



Not telephony, much of the research is health and OH&S related (so medical and acoustic journals).



Long duration ABX is still ABX, so unlikely to reduce demand on short-term memory (but it might increase it).

I thought that long term memory was demonstrably unreliable so fast switching ABX actually makes it easier to spot differences?

Aside from that if it really is that difficult to reliably spot a difference that does exist, regardless of the method, then it seems to me that difference is so tiny it is not worth worrying about, at least in terms of the practical endpoint of just enjoying music.

 

[phoenixdogfan]

Interesting that all these cheap Chinese dacs all must sound exactly the same, because the small differences are all far below the level we can hear. Why do you keep testing this? A race to the bottom to find the cheapest do-it-all dac? And then this forum will be closed?

Listen, I love to read every test report and even sometimes the comments. But someone must repeat that the truth out here is only a part of the lovely experience that hifi can be. The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

Has someone heard the difference between a Topping D90se and a RME ADI-2 fs? Think not. And everyone will jump on this statement because it is not triple-blind tested. Ofcourse not, because you can hear it easily yourself.

I love gear with excellent objective AND subjective test results. So I completely enjoy my Neumann KH420’s (relatively cheap in Europe). Problem is that you can hear everything you throw at it. So I prefer a warm sounding dac/pre. I bought the KH420’s after my experience with the K&H 300, good subjective reviews and the good objective review on this forum. But to find that warm sounding dac/pre-amplifier I had to look somewhere else.

If you want to "tune it to the human ear" (whatever that means), then that's what parametric eq is for. EAPO is free and will make it sound like anything you like. Using a DAC/AMP as a tone control is a haphazard and expensive way to go about it. If you find a coloration in the DAC you like, that's pure serendipty, nothing else, and it won't be exactly what you're looking for because only something like PEQ could possibly do that. That's why so many of us insist on completely uncolored components (along with the fact we don't want any tuning except the one put on the recording) in the amplification chain---because it gives us a clean canvas to create on if that's what we choose to do.

And yes, he does keep testing them because there's nothing better than perfect and cheap unless it's even cheaper and even more perfect.

 

[usersky]

The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

We talk about audible perfection here already. If one golden ears salesman want to tune by ear and let's say he really hears a difference (that's why you tune, to make it sound different), then by the simplest logic, sounding different than perfect IS IMPERFECT. Is this what you wish for? If yes, then there is a large market of expensive devices to choose from. We have some measured here.

Has someone heard the difference between a Topping D90se and a RME ADI-2 fs? Think not. And everyone will jump on this statement because it is not triple-blind tested. Ofcourse not, because you can hear it easily yourself.

No audible difference, a testament of audio perfection, something audiophiles should strive for. Would it be better one to be tuned by ear to sound different? Or do you imply you can easily hear differences between Topping D90se and RME ADI-2FS? That even better, maybe one was secretly tuned by ear (the most expensive one I would guess).

 

[Axo1989]

I thought that long term memory was demonstrably unreliable so fast switching ABX actually makes it easier to spot differences?

I didn't suggest long-duration ABX, so take that up with @solderdude

Aside from that if it really is that difficult to reliably spot a difference that does exist, regardless of the method, then it seems to me that difference is so tiny it is not worth worrying about, at least in terms of the practical endpoint of just enjoying music.

If you are interested in demonstrating presence/absence of subtle sonic differences, obviously use a method more likely to reveal them. If you aren't interested (I'm not saying anyone needs to be) then you probably aren't testing for them anyway. But as this thread is for discussion of measurements (or a honey-trap for same, depending how you look at it) so why not?.

Edit: not @Spkrdctr

 

[Sputnik]

Interesting that all these cheap Chinese dacs all must sound exactly the same, because the small differences are all far below the level we can hear. Why do you keep testing this? A race to the bottom to find the cheapest do-it-all dac? And then this forum will be closed?

Listen, I love to read every test report and even sometimes the comments. But someone must repeat that the truth out here is only a part of the lovely experience that hifi can be. The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

Has someone heard the difference between a Topping D90se and a RME ADI-2 fs? Think not. And everyone will jump on this statement because it is not triple-blind tested. Ofcourse not, because you can hear it easily yourself.

I love gear with excellent objective AND subjective test results. So I completely enjoy my Neumann KH420’s (relatively cheap in Europe). Problem is that you can hear everything you throw at it. So I prefer a warm sounding dac/pre. I bought the KH420’s after my experience with the K&H 300, good subjective reviews and the good objective review on this forum. But to find that warm sounding dac/pre-amplifier I had to look somewhere else.

If you go into it thinking 'these two Chinese products measure the same, so they will sound the same', they likely will. If you go into it thinking 'but this will sound much warmer', it probably will.

For you, the only thing that matters is how you feel about it. But if you want to convey to others how a product sounds, the only valid ways are measurements and blind tests. 'Air, detail, warmth, ...' are things that your brain processes, and your ears are only part of it. Preconceived notions, the price, your feelings about a brand, reviews you read,... All play a factor in how it sounds to you. Your ears are just a part if the equation.

But every audiophile will rave about 'hearing what the artist intended' and 'transparancy' in their ridiculously expensive system. Yet they don't want transparent devices, they want something that is 'tuned by ear' by the manufacturer, which by definition makes it not what the artist intended. They keep clinging to overpriced products that are demonstrably badly engineered. They keep raving about products that add distortion, as if the artists couldn't add distortion or 'warmth' themselves, if that was what they intended.

It's all very strange to me..

 

[Madlop26]

The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

I love gear with excellent objective AND subjective test results.

Tuned to human ears? to me is just common sense that the best "tuned" gear is the one which gives absolute transparency to the final sound. Also, no company who claims what you are saying gives any data of the research of what you call so "human tuned" or a detailed explanation of how they were able to achieve that sound they claim.

Gear with Excellent subjective results? all I have heard from subjective audiophiles are anecdotal results which are worthless. Conveniently there is always an absence of any reliable data backed by double-blind testing.

 

[Mart68]

I didn't suggest long-duration ABX, so take that up with @Spkrdctr



If you are interested in demonstrating presence/absence of subtle sonic differences, obviously use a method more likely to reveal them. If you aren't interested (I'm not saying anyone needs to be) then you probably aren't testing for them anyway. But I guess this thread is for discussion of measurements (or a honey-trap for same, depending how you look at it).

Long duration ABX is the obvious answer to those suggesting that fast switching obscures differences, but I don't see a third alternative that dispenses with ABX testing altogether. Except for sighted listening which is the cause of the dissention in the first place.

I think everyone appreciates that listening tests can deliver false negatives. But it seems to me that all that is required is to find just one person who can consistently spot differences. That won't be an impossible search if the differences are real.

As always the burden of proof is on those making the unlikely claims.

 

[xnor]

How do you guys deal with the issue that newbies (new visitors to this site with little or incorrect knowledge about measurements) read way too much into measurement results, coming out of a review thinking that that blip on the chart 120 dB down makes for a flawed product with audible deficiencies while completely overlooking significantly more important features of the product, leading to poor purchasing choices, disappointment, and subsequent rejection of measurements?
To a degree this is provoked by the reviewer's understandable perfectionism that nonetheless highlights measurable but acoustically completely and utterly irrelevant differences and can therefore mislead.

Without training on the consumer's side (which will never happen) or very careful presentation and description (which a lot don't read) of the data, this whole thing obviously backfires.

 

[Peluvius]

How do you guys deal with the issue that newbies (new visitors to this site with little or incorrect knowledge about measurements) read way too much into measurement results, coming out of a review thinking that that blip on the chart 120 dB down makes for a flawed product with audible deficiencies while completely overlooking significantly more important features of the product, leading to poor purchasing choices, disappointment, and subsequent rejection of measurements?
To a degree this is provoked by the reviewer's understandable perfectionism that nonetheless highlights measurable but acoustically completely and utterly irrelevant differences and can therefore mislead.

Without training on the consumer's side (which will never happen) or very careful presentation and description (which a lot don't read) of the data, this whole thing obviously backfires.

I am no scientist however I do work in an environment where measurements are critical as a starting point to achieve the required outcome. For me in audio, measurements come first and last. I need to know the kit has the potential to be great then I need to listen to it. I would hesitate to buy most gear just on measurements.

I can't say I have read one of the reviews here that didn't give some context to a measurement outcome.

 

[solderdude]

How do you guys deal with the issue that newbies (new visitors to this site with little or incorrect knowledge about measurements) read way too much into measurement results

Try to explain it (when they ask or comment) or point them to threads/videos where explanations can be found.

Of course there are those that do not ask anything (I suspect the vast majority of visitors) and just presume things or take it elsewhere and make fun of 'objectivists'. There is nothing that can be done about that. The whole 'SINAD chart' thing being the stick to whack the dog with as an example. And even that has been questioned and explained to death here.

 

[DSJR]

I love gear with excellent objective AND subjective test results. So I completely enjoy my Neumann KH420’s (relatively cheap in Europe). Problem is that you can hear everything you throw at it. So I prefer a warm sounding dac/pre. I bought the KH420’s after my experience with the K&H 300, good subjective reviews and the good objective review on this forum. But to find that warm sounding dac/pre-amplifier I had to look somewhere else.

With deepest respect sir, had I just relied on my ears in recent times (assuming I had the funds to choose), I'd have gone for a thin toned screamer of a speaker and sources inclined to a falsely 'clinical' presentation due to my hearing loss.. At least now with 'aided hearing,' *everything* sounds great, so I'll still look in here while accepting I'm out of it all now and am now easily pleased

 

[Mart68]

The panther system simple enough for a child to understand. No head = don't buy it. Anything else will probably be fine.

 

[FrantzM]

Interesting that all these cheap Chinese dacs all must sound exactly the same, because the small differences are all far below the level we can hear. Why do you keep testing this? A race to the bottom to find the cheapest do-it-all dac? And then this forum will be closed?

Listen, I love to read every test report and even sometimes the comments. But someone must repeat that the truth out here is only a part of the lovely experience that hifi can be. The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

Has someone heard the difference between a Topping D90se and a RME ADI-2 fs? Think not. And everyone will jump on this statement because it is not triple-blind tested. Ofcourse not, because you can hear it easily yourself.

I love gear with excellent objective AND subjective test results. So I completely enjoy my Neumann KH420’s (relatively cheap in Europe). Problem is that you can hear everything you throw at it. So I prefer a warm sounding dac/pre. I bought the KH420’s after my experience with the K&H 300, good subjective reviews and the good objective review on this forum. But to find that warm sounding dac/pre-amplifier I had to look somewhere else.

ROTFLMAO....

Peace.

 

[Trell]

Has someone heard the difference between a Topping D90se and a RME ADI-2 fs? Think not. And everyone will jump on this statement because it is not triple-blind tested. Ofcourse not, because you can hear it easily yourself.

The RME has DSP, like Dynamic volume and PEQ that I use myself. That for sure will make a difference, but with DSP disabled? Not at all.

 

[SIY]

Long duration ABX is the obvious answer to those suggesting that fast switching obscures differences, but I don't see a third alternative that dispenses with ABX testing altogether. Except for sighted listening which is the cause of the dissention in the first place.

There are many alternative test formats. Triangle and sorting, for two examples. I showed several others in my old Linear Audio article.

 

[Axo1989]

Long duration ABX is the obvious answer to those suggesting that fast switching obscures differences, but I don't see a third alternative that dispenses with ABX testing altogether. ...

The papers linked ran comparative methods, pairwise 4AIX outperformed ABX. If you find that interesting, read them. The discussion of different listening modes and other aspects was also useful.

 

[Mart68]

The papers linked ran comparative methods, pairwise 4AIX outperformed ABX. If you find that interesting, read them. The discussion of different listening modes and other aspects was also useful.

Thanks I will have a look at that. Every day's a school day on here.

 

[SIY]

The papers linked ran comparative methods, pairwise 4AIX outperformed ABX.

It is a capital mistake to assume that any specific test format is universally the best one. It very much depends on the hypothesis being tested. This is why there's multiple test formats.

The only absolute, non-negotiable part is that there has to be sufficient controls. In audio, that means ears-only, double blind, no peeking.

 

[Geert]

The more expensive gear sometimes is tuned to the human ear. That can mean the test results are not as good as you would expect. That kind of gear is ridiculed here.

It can be tuned to the ear, to sound different. Doesn't mean it's always better for everyone and in all conditions. Neither does it means it needs to be more expensive. Allowing or adding distortion is easier than making a flawless design. You can even get if for free if you're just messing around.

 

[Curvature]

It is a capital mistake to assume that any specific test format is universally the best one. It very much depends on the hypothesis being tested. This is why there's multiple test formats.

The only absolute, non-negotiable part is that there has to be sufficient controls. In audio, that means ears-only, double blind, no peeking.

Agreed.

Here's a newer linguistics paper using the same 4IAX technique, where "Listeners are asked to discriminate between two pairs of stimuli and choose which pair is different": https://www.researchgate.net/public..._a_4IAX_task_The_effects_of_acoustic_distance

Certainly complicates experimental design. In this particular test a difference was presumed and the problem shifted to ranking differences given listener's histories and cognitive development within a particular linguistic context. There some analogy here to listener training and familiarity with certain errors in audio.