Poll: Should We Get Into Testing Headphones or Speakers?

[svart-hvitt]

Does he? I think he just greatly favors buying good speakers and doing room treatment first, and then doing room correction as a last step.

https://www.audiosciencereview.com/...-theory-without-measurement.7127/#post-162524

 

[flipflop]

IMO the fit of clothing isn't quite the right metaphor

It's more of a comparison rather than a metaphor/analogy.
You wear your clothes, you wear your headphones. Wearing clothing that doesn't fit will make you feel uncomfortable. Wearing headphones that don't fit will introduce leakage effects, and probably be uncomfortable, too. Neither should be assumed to be reasonable use cases.

unless designs that tend to couple poorly and have large leakage effects are similar to clothing that barely anyone can fit into.

Both would be examples of bad designs not worthy of paying attention to, let alone study or purchase.

It is, however, well worth keeping in mind when making inferences from the estimated preference ratings that Olive's model produces about real headphones, because it substantially increases the potential error in some cases.

What you're saying here mirrors the part of my last post where I said "leakage effects will make the predictions less accurate". Having thought a little more about it, I will have to partly backtrack/disagree, though.
Leakage effects can measured as a loss of SPL in the bass and lower end of the midrange. This is essentially a change in the frequency response. The model is not any worse at predicting preference ratings for hypothetical headphone A and headphone B (both with good seals) than it is at predicting preference ratings for headphone C with a flawless seal and headphone C with a poor seal.
In other words, as long as the leakage effects are being measured, the "correlation coefficient for the predicted versus observed headphone sound quality preference ratings [of] r = 0.91" remains.

 

[Mad_Economist]

It's more of a comparison rather than a metaphor/analogy.
You wear your clothes, you wear your headphones. Wearing clothing that doesn't fit will make you feel uncomfortable. Wearing headphones that don't fit will introduce leakage effects, and probably be uncomfortable, too. Neither should be assumed to be reasonable use cases.

Both would be examples of bad designs not worthy of paying attention to, let alone study or purchase.

I think that you are underestimating how widespread variations in leak are, and how it can manifest - the Sennheiser HD800 is almost universally agreed to be impeccably ergonomic, but I have verified leak-induced bass variations >6dB at 20hz (its large earpads in many cases have gaps in contact somewhere on the head) on human heads. Almost all closed front volume designs display significant variation in the presence of leaks, and for many - including Harman high-scorers such as the N700NC and MDR-7506 - even something a simple as wearing glasses will have a marked impact on low-frequency response (although the N700NC, and some other active DSP'd designs, does have some capability to compensate for this).

What you're saying here mirrors the part of my last post where I said "leakage effects will make the predictions less accurate". Having thought a little more about it, I will have to partly backtrack/disagree, though.
Leakage effects can measured as a loss of SPL in the bass and lower end of the midrange. This is essentially a change in the frequency response. The model is not any worse at predicting preference ratings for hypothetical headphone A and headphone B (both with good seals) than it is at predicting preference ratings for headphone C with a flawless seal and headphone C with a poor seal.
In other words, as long as the leakage effects are being measured, the "correlation coefficient for the predicted versus observed headphone sound quality preference ratings [of] r = 0.91" remains.

Leakage effects are, of course, an impact on frequency response at the eardrum, and I'd agree that if at-eardrum FR is relatively stable and known, models such as Olive's can offer quite robust prediction. This is not necessarily the case in practice, however, and so I feel it's important to (somewhat) temper our confidence in extrapolations from frequency response measurements alone. I think that they're extremely indicative of perceived sound under the circumstances which were measured, and for many headphones this is accurate to most cases of end user interaction, but there are a number of uncertainties that can crop up between the HATS and the user that merit a bit of caution or some additional measures - which I don't think we disagree with, as we both seem to like RTings' positional variation measurement, for example.

 

[amirm]

So like @Floyd Toole derides room correction as a marketing story, you don’t see HRTF as a problem that should be compensated for in headphones listening?

What? Did you read what I wrote?

 

[svart-hvitt]

This debate on headphones and the Harman method is fascinating.

In another thread on headphones, HRTF was all the rage: https://www.audiosciencereview.com/...uring-hrtf-for-headphone-use.3962/#post-93816

In this thread, I asked:

«SCIENTIFIC REFERENCES?

This is the 14th comment in this thread on measuring headphones and HRTF, but I wonder:

Is it too soon to ask for some peer reviewed references, say from JAES or equivalent, on this very interesting but confusing topic?»

And another question I asked:

«If it were a mature area I would ask for the canonical texts.

Such texts would cast light on background for inquiry. Because this is an application oriented crowd I wonder how our insight into the problem can help us design and find neutral headphones, cfr. @Floyd Toole ’s many practical insights into speakers».

So I pushed the ASR community to come up with explanation why one cannot simply find a «neutral» headphones curve, like the one @Floyd Toole searched for about 40 years ago and which has been refined by Olive et al. recently.

The response from the community was luke warm. One example:

«You are off again. You need to stop asserting that this is an unknown unresearched unscientific area. See olive, as mentioned, or maybe this, the Audeze article etc etc
https://www.bksv.com/en/about/waves/WavesArticles/2014/inside-headphones

Or maybe some of your own searches on the subject. The same message keeps coming out. It's actually a simple problem, but difficult to solve, until we find a cheap and consistent way of measuring individuals ears».
Source: https://www.audiosciencereview.com/...hrtf-for-headphone-use.3962/page-2#post-93920

Has this «difficult to solve» problem, which means «a consistent way of measuring individual ears» been solved now?

I can’t find anything about HRTF in those three Olive papers from 2017 and 2018 that were mentioned previously.

What good is a Harman curve if you don’t take HRTF («measuring indicidual ears») into consideration?

 

[amirm]

What good is a Harman curve if you don’t take HRTF («measuring indicidual ears») into consideration?

What good is it? This good:

With no EQ, that Sennheiser headphone in listening tests across large number of listeners, gets a score of 2.88. If equalized to Harman curve, its score jumps way up to 5.85.

Rankings of headphones doesn't change with experience in music:

Trained or untrained:

Nationality of people:

Or even age (although there is a bit of variation here):

I can go on but the bottom line like speakers is a fortuitous thing: that we are much more alike than we think. Brain adapts to the HRTF of the hearing system and makes it less of a variable.

 

[Mad_Economist]

I can’t find anything about HRTF in those three Olive papers from 2017 and 2018 that were mentioned previously.

What good is a Harman curve if you don’t take HRTF («measuring indicidual ears») into consideration?

Sufficiently good to, conditional on a few things (the use of fairly consistent in FR on head headphones, for example) correlate quite well with preference. There is quite likely some merit in individualized measurements of HRTF, and certainly some merit in in situ measurements of headphone response on users' ears, but the data from the Olive experiments strongly supports that frequency response on a generic/average test fixture can predict preference fairly reliably.

 

[svart-hvitt]

What good is it? This good:

View attachment 28516

With no EQ, that Sennheiser headphone in listening tests across large number of listeners, gets a score of 2.88. If equalized to Harman curve, its score jumps way up to 5.85.

Rankings of headphones doesn't change with experience in music:

View attachment 28517

Trained or untrained:

View attachment 28518

Nationality of people:

View attachment 28519

Or even age (although there is a bit of variation here):

View attachment 28520

I can go on but the bottom line like speakers is a fortuitous thing: that we are much more alike than we think. Brain adapts to the HRTF of the hearing system and makes it less of a variable.

That’s my whole point: Harman has skipped the whole HRTF issue to promote one curve to rule them all. I wonder, is it consensus in audio science that measuring individuals’ ears (HRTF) is unnecessary?

From Samsung’s point of view I see why they may want one single curve instead of the hassle of measuring each individuals’ ears. If you want to sell millions of headphones, you want a smooth operation and measuring individuals’ ears is not smooth. Needless to say: All Samsung headphones will get top marks in your tests because they comply with the Harman curve. It’s like taking an exam where you made all the questions yourself. Adding to the complex is the fact that the Harman curve is not based on a theory of hearing, but a poll of preferences. So the Harman curve will change over time; it already has been altered at least once.

There is a big gap between how a product pusher thinks and how an advisor thinks. So my point is: Is it good advice to take the individual’s unique HRTF out of the equation? Should ASR choose the path chosen by a product pusher or take the role as an independent advisor?

I wrote previously that headphones are a can of worms. I still think that is a correct description.

 

[amirm]

That’s my whole point: Harman has skipped the whole HRTF issue to promote one curve to rule them all.

No. HRTF is included for the standardized human head/ear. They then tweaked it based on listening test results. In other words, this is all driven by listening tests, not some magic curve taken out of thin air.

 

[amirm]

Needless to say: All Samsung headphones will get top marks in your tests because they comply with the Harman curve.

There is no evidence of Samsung adopting this research across the board. But yes, let's invent something not backed by any research/science to make them look worse. That would be the ticket.....

 

[Mad_Economist]

Adding to the complex is the fact that the Harman curve is not based on a theory of hearing, but a poll of preferences.

If you're inclined toward something grounded more explicitly in theory and less in tests of preferences, the work of Günther Theile (which I have linked before, but Theile 1986 and Theile 2016 are two worthy of attention) may be of interest to you. Hammershøi & Møller also proposed some hypothetical targets in the 90s I believe, including a "hybrid" HRTF which weighted some free field HRTFs more heavily in an average - I think it was their paper from 96?

Of course, for targets intended for the playback of music, the subjective perceptions of listeners are a core component, which is why Toole's work in speakers also included it...

 

[amirm]

There is a big gap between how a product pusher thinks and how an advisor thinks. So my point is: Is it good advice to take the individual’s unique HRTF out of the equation? Should ASR choose the path chosen by a product pusher or take the role as an independent advisor.

I plan to perform listening tests if I get into this. The data if we go this way, will indicate how far a headphone deviates from the Harman proposed curve. You all can use such data to ignore any recommendation in text.

Right now, Harman has a very large body of headphone data but it is all private. If we get into this testing, we will be able to have it out in the public to touch and feel.

 

[svart-hvitt]

No. HRTF is included for the standardized human head/ear. They then tweaked it based on listening test results. In other words, this is all driven by listening tests, not some magic curve taken out of thin air.

They skipped the process of measuring individuals’ ears. Instead, they chose an «average» human ear. This is a cost-efficient choice but hardly optimal.

The curve they chose is not magic; I didn’t say that. But the Harman curve is based on polls of preferences only, not a theory of human hearing. So we should expect this curve to change as preferences change of as alternative polls surface.

To me, it seems like the Harman method is not different from @Floyd Toole ‘s work from back in the 1980s or so. Toole had great intuition to foresee what Harman would do decades later. The question is, are Toole’s ideas from the 1980s state of the art 40 years later? Why not measure individuals’ ears if one has the means to do it? And why not develop a theory on human hearing relevant for headphones instead of resorting to polls of preferences?

 

[amirm]

Adding to the complex is the fact that the Harman curve is not based on a theory of hearing, but a poll of preferences.

Of course it is based on theory of hearing. Where do you think it's odd shape comes from? All the resonances and filtering of the human hearing system are included in all the HRTFs used as the basis of the curve. Their modifications comes mostly from testing what people like in listening to speakers and relating that to headphones.

 

[amirm]

They skipped the process of measuring individuals’ ears. Instead, they chose an «average» human ear. This is a cost-efficient choice but hardly optimal.

Your comments don't make any sense. A fixture is used that is standardized by the industry/research to be representative of all of us. By definition, the correction HRTF then is of this average since that is the test fixture that is used to make measurements.

The curve they chose is not magic; I didn’t say that. But the Harman curve is based on polls of preferences only, not a theory of human hearing. So we should expect this curve to change as preferences change of as alternative polls surface.

I am at a loss as what to say about this. You haven't understood the theory of their work, or research in this field at all.

Companies need to make one product to fit the entire population. Harman research shows that instead of random targets, if you follow certain ones and use their measurement gear, you will do a lot better.

Once there, because of lack of standardization in audio production, they suggest that tone controls be made available for the user to tweak the response.

We have a compass now, not a GPS locator. Before, we didn't have a compass and were wondering around randomly.

 

[svart-hvitt]

Your comments don't make any sense. A fixture is used that is standardized by the industry/research to be representative of all of us. By definition, the correction HRTF then is of this average since that is the test fixture that is used to make measurements.


I am at a loss as what to say about this. You haven't understood the theory of their work, or research in this field at all.

Companies need to make one product to fit the entire population. Harman research shows that instead of random targets, if you follow certain ones and use their measurement gear, you will do a lot better.

Once there, because of lack of standardization in audio production, they suggest that tone controls be made available for the user to tweak the response.

We have a compass now, not a GPS locator. Before, we didn't have a compass and were wondering around randomly.

You don’t understand my point, which is BTW the exact same point many others made in a thread on HRTF.

Instead of making one headphone to rule them all, it is more logical that every single headphone - if you want to do it right - should be calibrated to each individuals unique ear and HRTF. To save money, Samsung dislikes words like «individualized», «unique» etc. So it’s not surprising that a research arm of Samsung comes up with a solution that supports Samsung’s logistics and distribution strategy. This is not to say that the Harman curve is bad; but it’s not individualized, tailored, it’s not unique for every single headphones user.

Fascinating how this point - which the community tried to teach me previously - suddenly is irrelevant, replaced by a standard, an average ear and HRTF.

 

[Mad_Economist]

While I'm generally sympathetic to the sentiment that individualized HRTF compensation is ideal, I will point out that in addition to being difficult to test, the lit as stands shows reasonable results from measurements conducted on standardized ears. As a somewhat old example, Spikofski 1988 (which I believe is actually properly dated somewhat earlier in its German publication, but sadly ich spreche kein Deutsch so I'm not certain) included some analysis of probe measurements of headphones and DF-HRTFs of individuals, including disparities between individual and population mean results.

It'd be a good area for acquiring more data, and if anyone's got any spare ears they feel inclined to indefinitely loan I might be able to spare a probe mic, but I feel that its significance may be slightly overstated here.

 

[amirm]

You don’t understand my point, which is BTW the exact same point many others made in a thread on HRTF.

Oh, I know your motivation. You have an irrational fear of anything associated with Harman as a company.

Instead of making one headphone to rule them all, it is more logical that every single headphone - if you want to do it right - should be calibrated to each individuals unique ear and HRTF.

And what should be your starting point? Some random frequency response that doesn't have listener preference and then attempt to twist it into your HRTF?

Why don't you take a clock radio then and DSP it to make it sound the way you want for your stereo?

 

[Mad_Economist]

And what should be your starting point? Some random frequency response that doesn't have listener preference and then attempt to twist it into your HRTF?

In principle, this would be possible if not for the SLD effect - indeed, this is roughly what Griesinger advocates doing (albeit, for some reason, with a frontally located speaker). Some people report subjectively positive results, although my read of the literature is that it doesn't make sense.

 

[DDF]

Sufficiently good to, conditional on a few things (the use of fairly consistent in FR on head headphones, for example) correlate quite well with preference. There is quite likely some merit in individualized measurements of HRTF, and certainly some merit in in situ measurements of headphone response on users' ears, but the data from the Olive experiments strongly supports that frequency response on a generic/average test fixture can predict preference fairly reliably.

When I worked in telecom audio, we had headset targets for "wideband" audio (7 kHz BW) very similar to the Harman targets, but peaking at 7 dB vs 10 dB. These were arrived at from 100s or 1000s of DBTs (mean opinion score, MOS, a form of psychoacoustic preference testing) from numerous independant parties including us (Northern Telecom/BNR) and other members in the relevant ITU forums. I'm pretty confident that the Harman targets are close to a median that gives best chance for success without individualized customization. It's better than the alternatives of doing nothing, or designing to a notably different target. The next step up is to measure individualized HRTF for which there are commercial products providing this capability. Even in this case, starting from the Harman target minimizes the average eq needed, a good thing in terms of preserving system dynamic range.