Master Thread: Are Measurements Everything or Nothing?

[pkane]

I fully agree we are not here to discuss philosophy but our experience, training and education of the broader world view does inform how we view specific topics. And yes it’s generally science research which may or may not lead to proofs. I’m not making any definitive claims just merely pointing out what has been shared with me doesn’t constitute what would normally be considered as firm scientific research. We have some helpful limited reseach into certain specific aspects. I’m just saying we don’t have definite absolute proof of most matters especially when it comes to the human aspects. I don’t see how that is particularly controversial at all. Just a statement of the reality and therefore it is best to be open minded.

Sure, and you're stating something very obvious to anyone who's been involved in science. But making any conclusions based on "what has been shared with me" isn't science, it's just an opinion. And you know what they say opinions are like... everyone has them.

Another quote that's been repeated many times before: keep an open mind but not so open that your brains fall out

 

[Mart68]

I fully agree we are not here to discuss philosophy but our experience, training and education of the broader world view does inform how we view specific topics. And yes it’s generally science research which may or may not lead to proofs. I’m not making any definitive claims just merely pointing out what has been shared with me doesn’t constitute what would normally be considered as firm scientific research. We have some helpful limited reseach into certain specific aspects. I’m just saying we don’t have definite absolute proof of most matters especially when it comes to the human aspects. I don’t see how that is particularly controversial at all. Just a statement of the reality and therefore it is best to be open minded.

There may be science that is disputed at the limits of knowledge but that does not include varying an electrical voltage to energise a magnet to make a cone vibrate. No scientist is still looking into that because it's long since a done deal.

It always amuses me that the peddlers of products such as magic power cables, fuses, grounding boxes, magic crystals and so forth do not step forward to claim their Nobel Prize in Physics. These things can only work if the existing paradigm is wrong or missing important elements.

Is that likely? Of course it is not. This is not a case of one set of 'beliefs' versus another. It's decades of testing and evidence vs nonsense.

 

[TTT15]

Yes I’m not disputing the fundamentals of science of course. . I’m still not saying measurements or research are not useful quite the contrary. I’m only debating at the margins and not making any claims but only questioning which is good science. I know from my work that many questions will turn out to be stupid or irrelevant but without questioning you don’t get to the truth or potential issues or potential uncertainties. My job is to question. I don’t think that is a bad thing in a world where often people just seem to accept what ever they want to believe.

I also accept that I’ve a harder definition of what constitutes Science which comes from working with a renowned science journal business. These days scientists pay for the journals to get their research peer reviewed. We understandable don’t have that for HiFi but that is the measure of what constitutes science in my world. The rest for me is studies and research which is good but not the same as published peer reviewed research. The research journals with peer reviewed research are worth billions for a reason. Truth still does matter. We don’t have that for HiFi.

We have good research for speaker measurements but the tests seem to be limited to certain brands so we don’t have universal tests for all speakers and whether they tie into consumer preferences. And we have research into headphones preferred average frequency response but not which are the preferred headphones across all brands and products and how they link to measurements. iPods are the most popular but how well do they measure? Ideally we would have measurements for all brands and products in all segments and they could be linked to blind test preferences . That is simple not realistic so we end up with subjective You Tube reviews being the most that people see. Still doesn’t mean that measurements aren’t useful. I’m arguing that I wish ideally that there was more tests and research.

I don’t know why I enjoy my new headphone set up over my old headphones set up for instance and it isn’t related to the harman curve. I can clearly measure that I spend more time listening to the new set up and it is clear I enjoy them more. I can also clearly measure that I spend more time listening to set ups which are good to use but have no measurements for ease of use. So I would tend to agree with Amir that headphones are only 50pc measurements and that inherently measurements cover some important but not all the many other factors that go into purchase decisions.

 

[Purité Audio]

Personally I wouldn’t invest in a pair of speakers or electronics for that matter which haven’t been independently measured and the results published.
Keith

 

[MAB]

I also accept that I’ve a harder definition of what constitutes Science which comes from working with a renowned science journal business. These days scientists pay for the journals to get their research peer reviewed. We understandable don’t have that for HiFi but that is the measure of what constitutes science in my world. The rest for me is studies and research which is good but not the same as published peer reviewed research. The research journals with peer reviewed research are worth billions for a reason. Truth still does matter. We don’t have that for HiFi.

You seem to be unaware AES is a peer reviewed journal specific to audio.
And that topics on human perception of sound, and DSP, and lots of other audio things are published in peer reviewed journals. You make points that ignore the truth.
Does it not count since it isn't Science or Nature?
Does Phys Rev D then also not count?
How about Journal of American Vacuum Society?

 

[TTT15]

And I think that it is an entirely reasonable choice to only look at equipment with published independent measurements. The equipment I’ve got and are considering are also Independently measured and published. My choice is also to demo.

Yes I would like to see more references for published peer reviewed research thanks as I’ve read up on some but I’m mostly get referred to you tube videos and blogs which as you have gathered I don’t think is necessarily wrong but don’t find to be compelling.

 

[solderdude]

Yes I would like to see more references for published peer reviewed research

Become a member of the AES and you will have plenty of reading material.
Some of it is not very good but quite a lot of it is.
Determining what papers are valuable and which ones are not might be the most difficult part.

 

[Newman]

Become a member of the AES and you will have plenty of reading material.
Some of it is not very good but quite a lot of it is.
Determining what papers are valuable and which ones are not might be the most difficult part.

Toole is the best curator you can get for that task. TTT has already attempted to downrate Toole as just one guy in one company, so he is either trolling or very naive about the person he chose to so quickly and dismissively judge.

 

[TTT15]

Saying that Toole is the best research I have seen and that you intend to demo speakers based on their research doesn’t look like seeking to downgrade a piece of research to me. I’ve see You Tube videos from Toole and Olive and Erin’s Corner and they come across well in my view. When you look at as many businesses as I do you will see that they all claim to have some differentiating point. I’m not contradicting their conclusions. I’m merely pointing out that while their research is published I’ve no doubt that other major manufacturers of speakers and headphones do their own research. Toole and Olive research is published so it isn’t an industry secret. Some people do listen to speakers and headphones before purchasing so it seems reasonable that other major manufacturers might consider the research and do their own research and come to their own conclusions. It seems to me unlikely that speaker and headphone manufacturers are unaware of the research or that they deliberately set out to make speakers and headphones that the public don’t like. I’m just questioning as is good professionally and scientific practice. What is not good practice if you want to be objective is to just dismiss any potential issues or limitations or questions.

 

[pkane]

[/QUOTE]

I’m just questioning as is good professionally and scientific practice. What is not good practice if you want to be objective is to just dismiss any potential issues or limitations or questions.

There’s no professional or scientific practice that encourages questioning real scientific results on the basis of no evidence, experience, or knowledge. Potential issues or questions have to be substantiated with something solid and not just one’s beliefs or opinion.

 

[solderdude]

so it seems reasonable that other major manufacturers might consider the research and do their own research and come to their own conclusions.

They do.
And some of their results, not surprisingly, differs from Harman research.
That doesn't invalidate nor validate the research of others nor Harman's.
When it is about preference all bets are off and there will be a range that would be acceptable for the largest group of people.
But even in that range you can have all kinds of tonality differences which someone may still 'like' but others don't.
At some point in time one could draw an average 'line' and one could call that a target or reference.

I’m just questioning as is good professionally and scientific practice.

Yep, but at one point you would have to draw a conclusion and not keep questioning and questioning... etc.

 

[pkane]

I’m merely pointing out that while their research is published I’ve no doubt that other major manufacturers of speakers and headphones do their own research.

Just as a point of interest, Toole and Olive both conducted acoustic research as scientists at a non-commercial entity -- NRC. Their work at Harman was a continuation of that work with a somewhat bigger budget and more expensive toys (I presume )

 

[Newman]

Toole is the best curator you can get for that task. TTT has already attempted to downrate Toole as just one guy in one company, so he is either trolling or very naive about the person he chose to so quickly and dismissively judge.

Saying that Toole is the best research I have seen and that you intend to demo speakers based on their research doesn’t look like seeking to downgrade a piece of research to me.

Oh yeah? Then what about your statements that, if it is not in Science and Nature, then you don't count it as real science? Does look like seeking do downgrade every single available piece of research to me.

Even though you must be well aware that Science and Nature are not going to be accepting research papers on the topic of what a certain type of consumer hobby product sounds like.. no matter how well the research is conceived, designed and executed. The pinnacle journal for such material is the Journal of the Audio Engineering Society. If you now proceed to try and diss that journal, I will add any such reply to all the evidence that you have already provided that you are here in bad faith.

I’ve see You Tube videos from Toole and Olive and Erin’s Corner and they come across well in my view.

Putting those two in the same breath as Erin's Corner is ridiculous and suggests your naivety. Or your motive. Like I said.

When you look at as many businesses as I do you will see that they all claim to have some differentiating point. I’m not contradicting their conclusions.

Oh, so Toole's research is one business's research? See my comment above.

I’m merely pointing out that while their research is published I’ve no doubt that other major manufacturers of speakers and headphones do their own research. Toole and Olive research is published so it isn’t an industry secret. Some people do listen to speakers and headphones before purchasing so it seems reasonable that other major manufacturers might consider the research and do their own research and come to their own conclusions. It seems to me unlikely that speaker and headphone manufacturers are unaware of the research or that they deliberately set out to make speakers and headphones that the public don’t like. I’m just questioning as is good professionally and scientific practice.

Maybe in your naivety you are doing so innocently. Maybe. Oh, and just in case you think you are giving us lectures on the nature of the scientific method that we need to hear, don't kid yourself. Every wannabe troll comes here and tries that on, using tactics they learned from the climate science denial social media. Weaponising science in order to deny it doesn't wash, including your far-from-novel tactic of claiming to be honouring the scientific method by questioning the research, and implying that your doing so is more 'science-like' than any of us who may be accepting the findings of the best available research. Sorry mate, that only holds true among actual experts on the topic. Also far from novel for you to conflate Toole with a corporate hack so you can claim his findings only apply to the type of speaker that one company sells.

What is not good practice if you want to be objective is to just dismiss any potential issues or limitations or questions.

Just as a point of interest, Toole and Olive both conducted acoustic research as scientists at a non-commercial entity -- NRC. Their work at Harman was a continuation of that work with a somewhat bigger budget and more expensive toys (I presume )

Exactly. Someone is jumping to naive conclusions instead of asking polite questions...even though he repeatedly claims the latter is his specialty.

I still smell the whiff of trolling of the sort that makes me wonder if he needs to defend his sighted listening impressions that aren't supported by the best available research to date.

 

[solderdude]

I can clearly measure that I spend more time listening to the new set up and it is clear I enjoy them more.

There can be reasons for that.
Comfort and fit, new toy syndrome, performance of the headphone is audible 'better' in areas you care about, could even be technical performance is better or even worse but you still prefer it.
And indeed it has nothing to do with Harman research which is about preferred tonality.
It has to do with your personal preference and usage circumstances which can (and does) differ from those of others.
The reason for that is the human aspect and the rather large delta there.

Any research on that will only reveal how big that delta will be and not what someone's personal delta will be from the 'mean'.
This is why you won't find 100% correlation with personal findings.

Plenty of research and attempts have been made regarding perception, usually for single aspects of audibility like tonality, distortion types etc.
In the electrical plane performance is well understood and there don't seem to be surprises coming up there.
Perception and preference combined with transducers is a totally different ballgame.

One also needs to realize that as soon as perception comes into play there will also be recordings (music) into play and there is a necessity for transducers and the accompanying acoustic challenges.
Fortunately, when researching the electric device differences all those acoustic issues (other than perception) remain the same.
Thus when testing electronics by ear you only have 2 variables, the electric plane (measurable and constant) and perception (variable).

Start reading studies on these subjects and stop listening to opinions vented on YouTube and 'reviews'.
If you are so into 'the studies and science' stop looking at consumer reports and 'reviews'. They can be 'colored' in numerous ways for many reasons and do not warrant more research as they are just personal opinions... even when they appear to be science.

 

[TTT15]

Im surprised at some of the reactions. I’ve absolutely nothing against Toole and Olive or their findings per se as I’ve indicated many times.. I think it is good research and they seem like good people. What they found based on the research that they did is what they found. I merely question how universally their findings apply as we are lacking research and measurement from other manufacturers and I question the extent to which their measurements correlate to all the factors that influence individual choice. They are not claiming they do either only average preferences in respect of certain measurements.

The ultimate metric for a consumer product is sales and profits. So to what extent do their measurements correspond to human preferences expressed in sales figures. I’m merely suggesting that there maybe more factors in play in human preferences that some specific measurements of certain specific aspects don’t fully capture.

Why do I question? We use data analytics all the time at work to analyses businesses but they are only useful for deeply analysing certain aspects but they are never suffecient to cover all aspects that matter. My football team has been put together based on detailed data analytics and they are awful. I‘m merely pointing out that measurements can only ever cover certain specific aspects which may well be true in themselves but have to weighted somehow and may not be extrapolated to cover all potential aspects that matter especially when it comes to individual human interaction. I will continue to consider measurements but also other factors that are not measured and what I enjoy. If someone isn’t interested in the features or enjoyment that is fine.

 

[solderdude]

The ultimate metric for a consumer product is sales and profits

That's important for any product and valuable for manufacturers and sellers.
For consumers it is price, availability and preference (in many aspects).

I will continue to consider measurements but also other factors that are not measured and what I enjoy.

Measurements say something about technical performance.
Price, longevity, availability, warranty, ease of operation, feel, taste and preference (enjoyment) is what matters to an individual.

There is no direct link between measurements and enjoyment nor will there ever be. Period.

You may want to see research and explanations but the human brain is involved and thus too many variables.
Enjoyment is personal... of course people are interested in personal enjoyment... but its personal and not science referenced to some reference.

 

[TTT15]

There can be reasons for that.
Comfort and fit, new toy syndrome, performance of the headphone is audible 'better' in areas you care about, could even be technical performance is better or even worse but you still prefer it.
And indeed it has nothing to do with Harman research which is about preferred tonality.
It has to do with your personal preference and usage circumstances which can (and does) differ from those of others.
The reason for that is the human aspect and the rather large delta there.

Any research on that will only reveal how big that delta will be and not what someone's personal delta will be from the 'mean'.
This is why you won't find 100% correlation with personal findings.

Plenty of research and attempts have been made regarding perception, usually for single aspects of audibility like tonality, distortion types etc.
In the electrical plane performance is well understood and there don't seem to be surprises coming up there.
Perception and preference combined with transducers is a totally different ballgame.

One also needs to realize that as soon as perception comes into play there will also be recordings (music) into play and there is a necessity for transducers and the accompanying acoustic challenges.
Fortunately, when researching the electric device differences all those acoustic issues (other than perception) remain the same.
Thus when testing electronics by ear you only have 2 variables, the electric plane (measurable and constant) and perception (variable).

Start reading studies on these subjects and stop listening to opinions vented on YouTube and 'reviews'.
If you are so into 'the studies and science' stop looking at consumer reports and 'reviews'. They can be 'colored' in numerous ways for many reasons and do not warrant more research as they are just personal opinions... even when they appear to be science.

May I just say that I appreciate the way that you always seek to play the ball. There is certainly new toy syndrome at play but I didn’t feel like that about my new toy last time. I perceive levels of detail and enjoyment that I’ve never heard before. So I’m not disputing the finding from Toole and Olive per se at all as some seem to incorrectly read. The point I’m seeking to make is that as you say the harman curve covers a particular aspect on average, tonality. So I completely agree that to make valid comparisons you need to volume match. But should you also match frequency response by EQing to extent possible for transducers, in particular headphones ? And if you did would you find other factors that influence preferences and how do they measure. Olive certainly is only claiming an average frequency response preference not to what extent that correlates to headphone preference overall which is all I’m saying. . I’m not making any claims just questioning how one measured factor correlates to overall preference, the extent to which there are other factors and how do you weigh them.

A more simple example. I had a preference for the sound of a particular earbud but the cable was fragile and would break. I don’t have a preference for no sound. Therefore I’m merely postulating that particular measures or factors are inherently limited as no one specific measurement can capture all factors pertinent to a human consumer purchasing decision. That doesn’t seem like a claim to me but a reasonable question and that is all I’m seeking to understand.

 

[solderdude]

So I completely agree that to make valid comparisons you need to volume match. But should you also match frequency response by EQing to extent possible for transducers, in particular headphones ?

That is simply not possible because of the differences in human anatomy and in test fixtures + targets.
This means that if one EQ's any headphone on a specific fixture with a target suited for that fixture you can get a 'near perfect to that standards' tonality on that fixture.
Put it on any other standard fixture / target and you will get different results.
Put it on someones head and you get different results once again (for many reasons). It is NOT exact science though some pretend it to be.
Headphone (and in a lesser degree speaker) measurements are indicative at best, not a predictor of enjoyment.

Then there are resonances, nulls, cone break-up, driver angles, driver sizes, seal issues, clamping force, physical aspects (pinnae deforming/touching) etc.

Olive certainly is only claiming an average frequency response preference not to what extent that correlates to headphone preference overall which is all I’m saying

Dr. Olive is not 'claiming' anything. He (and his team) cooked up a tonal preference curve based on extensive research which is a 'target' for his (former) employer Harman.
They were kind enough to make most of of it public.
The goal was to design headphones that would be preferred to most people and thus increase sales (when it comes to sound, tonal balance that is).
The research showed there was quite some variance between preferred tonality and they arrived at an average that should 'suit' the majority (thus not all) of customers.

Again.... measurements are for technical performance. Measurement should conform to standard(s).
'Enjoyment' is something personal... for many reasons.
There is a loose link there but no 100% correlation.

 

[Heinrich]

Then there are resonances, nulls, cone break-up, driver angles, driver sizes, seal issues, clamping force, physical aspects (pinnae deforming/touching) etc.

What you don‘t get into the audience‘s head is the quality of those targets as being the least common domintor on average, and the heavy smoothing applied.
Then they complain about dips n‘ peaks like it would fry their very demanding ears. To ask them to just equalize to personal taste is felt to be an insult. They simply can‘t do that. Because they don’t know what their taste is, still complaining about deviances from a non-exsisting ideal.

That‘s the flip-side of measurement driven evaluation. People do not long for the listening experience, but for pride in possession. Individual taste doesn’t sell—it‘s not comparable to others‘, it doesn’t compete.

 

[TTT15]

That is simply not possible because of the differences in human anatomy and in test fixtures + targets.
This means that if one EQ's any headphone on a specific fixture with a target suited for that fixture you can get a 'near perfect to that standards' tonality on that fixture.
Put it on any other standard fixture / target and you will get different results.
Put it on someones head and you get different results once again (for many reasons). It is NOT exact science though some pretend it to be.
Headphone (and in a lesser degree speaker) measurements are indicative at best, not a predictor of enjoyment.

Then there are resonances, nulls, cone break-up, driver angles, driver sizes, seal issues, clamping force, physical aspects (pinnae deforming/touching) etc.



Dr. Olive is not 'claiming' anything. He (and his team) cooked up a tonal preference curve based on extensive research which is a 'target' for his (former) employer Harman.
They were kind enough to make most of of it public.
The goal was to design headphones that would be preferred to most people and thus increase sales (when it comes to sound, tonal balance that is).
The research showed there was quite some variance between preferred tonality and they arrived at an average that should 'suit' the majority (thus not all) of customers.

Again.... measurements are for technical performance. Measurement should conform to standard(s).
'Enjoyment' is something personal... for many reasons.
There is a loose link there but no 100% correlation.

Thank you. That was the answer I was looking for.