Poll: Should We Get Into Testing Headphones or Speakers?

[SIY]

We'll have to save that for the first ASR Megafestival, for which I'm sure the Tacoma Dome will be sold out...

Well, when you get tired of gray and rain, pack a few pairs of headphones into a suitcase and come to sunny Arizona. It's supposed to be a balmy 112° here this weekend.

 

[Juhazi]

I voted for headphones testing. Mostly because of logistic and economical issues with loudspeaker testing. Both have also technical challenges and one must learn to analyze test findings, how they correlate with listening impressions.

The best coverage of standardized and public HP tests can be found here https://www.innerfidelity.com/headphone-measurements
but unfortunately Tyll has retired https://www.innerfidelity.com/content/thats-itim-out

 

[Krunok]

I voted for heaphones testing.

Traitor!

 

[svart-hvitt]

Should headphones testing be anchored to a «Harman method»? Is this Audio Science Review or Harman Review? Is Harman the truth in headphones science? Is it a good avenue to put all the eggs in one basket that has «HARMAN» written on it?

 

[Juhazi]

Traitor!

Thanks, I have this wonderful personality with mixture of pessimist, sarcast and realist - ad nauseam!

 

[Krunok]

Thanks, I have this wonderful personality with mixture of pessimist, sarcast and realist - ad nauseam!

I believe you forgot to put "modesty"in that mix!

 

[Floyd Toole]

Should headphones testing be anchored to a «Harman method»? Is this Audio Science Review or Harman Review? Is Harman the truth in headphones science? Is it a good avenue to put all the eggs in one basket that has «HARMAN» written on it?

Sounds like a rather biased comment to me. If you are looking for a "basket" into which to put "eggs", you should select the one with the most credible scientific track record. Right now, that is the evidence published in AES documents by the Harman research group. It is public information, in peer reviewed papers. Call it the "Harman method" if you like, but until someone else provides a credible alternative, it is the current method of choice. Frankly, I think it is generous of Harman to pay for this research and then share the results with the competition. That is how my NRCC research and the subsequent loudspeaker work at Harman proceeded. That is the scientific method - put your work out there for comment and criticism. Unsubstantiated opinions don't count.

 

[Krunok]

Should headphones testing be anchored to a «Harman method»? Is this Audio Science Review or Harman Review? Is Harman the truth in headphones science? Is it a good avenue to put all the eggs in one basket that has «HARMAN» written on it?

ANSI/CEA 2034-A method of measurements of home loudspeakers was also derived from Harman Spinorama method. Are you saying it should not be used because of that? Or maybe it should serve as an example to audio scientists working for other companies to also publish their findings publicly so their work can be used by all of us.

 

[svart-hvitt]

Sounds like a rather biased comment to me. If you are looking for a "basket" into which to put "eggs", you should select the one with the most credible scientific track record. Right now, that is the evidence published in AES documents by the Harman research group. It is public information, in peer reviewed papers. Call it the "Harman method" if you like, but until someone else provides a credible alternative, it is the current method of choice. Frankly, I think it is generous of Harman to pay for this research and then share the results with the competition. That is how my NRCC research and the subsequent loudspeaker work at Harman proceeded. That is the scientific method - put your work out there for comment and criticism. Unsubstantiated opinions don't count.

My point is, despite Harman’s research, there seems to be a lack of consensus on measuring headphones. Publishing through AES doesn’t automatically make a technology truth or even best practice.

Let me use a recent example to illustrate my point.

High resolution audio is an area of much debate. In May 2019 JAES made a special issue on high resolution audio: http://www.aes.org/journal/online/JAES_V67/5/

There are many engineers today that have a lot to say about high resolution audio. What struck my eye was that MQA and J. Robert Stuart with colleagues wrote 3 out of 5 articles on high resolution in this special edition on high-res, which was published no less than 15 years after AES’s first special edition on high-res. MQA and Stuart wrote all of the technical papers, while the two other authors wrote an historical summary and a tutorial. Two points can be made out of this observation:

(1) AES gives the impression to be closely linked to one engineer who has a big commercial stake in a technology he is promoting. Is Stuart the only engineer who has something interesting to say on high-res after 15 years? Such close links are not good in a governance perspective, where AES can be accused of giving MQA and Stuart special treatment that has commercial value.

(2) Even if Stuart and MQA publish in AES doesn’t mean their technology is uncontroversial.

Are there similarities between this AES observation and my point on letting ASR be associated with Harman, «the Harman method»?

Speaker measurements are an area where consensus resides today, not least due to your work, while headphones measurements are an area where consensus is at best emerging. How smart is it of ASR to base the major part of its headphones work on a method that is not consensus based and still linked to the name of one commercial party? I didn’t invent the name «Harman method». Other users used the term here before me.

Publishing research in AES can be a great service to the community. But even if research has been AES reviewed, doesn’t make it truth or uncontested. MQA and Stuart hi-jacked AES’s special edition on high resolution, so should we better use the MQA technology without any afterthoughts? Who profited the most on the AES special edition on high-resolution; the audio society or MQA?

My point has nothing to do with audio science per se. My point has to do with «public image», how to build a reputation built on trust. I think ASR can do better than even AES (in some cases).

 

[617]

I won't get into a discussion on the credibility of Harman's reasearch, but I would hope that we can break it down so that semi technical people like me can participate in testing and understanding results.

Can someone explain the relevance of HRTF to headphones? Is it really more of an inner ear RTF?

 

[Krunok]

Can someone explain the relevance of HRTF to headphones? Is it really more of an inner ear RTF?

I believe topic of such complexity requires a separate thread.

 

[amirm]

Are there similarities between this AES observation and my point on letting ASR be associated with Harman, «the Harman method»?

The first time I got exposed to "harman research" I rolled my eyes. I then set out on a nearly 2 year project to not only read and understand their research, but also all the underlying and supporting research of which there are many. Dr. Toole's book alone has 270 references! I also then participated twice in their double blind test of speakers. By then the evidence was so overwhelming that no person with an ounce of common sense would refuse to accept it.

Everyone seems to have a "story" about what makes good sound in speakers and rooms. But none come remotely to the completeness and comprehensiveness of what is advocated by Dr. Toole and Sean Olive. To give up on what that research tells us, makes no sense at all. Indeed many competitors of Harman use the research in their development of their products.

Let's remember that a lot of what is said to be "harman" research was discovered and developed by Dr. Toole/Olive at canadian NRC so the commercial interest here is but a distant factor.

Again, others are welcome to put forward their stories and what backs them. No commercial entity though that I know of can remotely match 1% of the careful listening tests and studies that has been performed on this front.

And no, I am not and will not marry Harman research. I will be marrying the only solid and comprehensive research we have about good sound. I am not going to throw that away and go by some gray haired speaker designer at some company thinks.

That said, if we get into this measurement field, we will create our own body of data, both subjective and objective. So if you care about an alternative to "harman method," then you should help fund this activity. Nothing in Klippel system is about Harman other than generating the same set of measurements they use which is simply how the speaker radiates energy at different frequencies and angles. I can then setup listening tests and compare our findings to that of Harman.

 

[Shadrach]

I've voted loudspeakers. It's an endeavor fraught with problems none of which I can help with or pay for.
My preference would be to concentrate on the ranges that don't get covered by the pro audio sites which imo do a reasonable job, sometimes with measurements but more often with some degree of rationality.

 

[Frank Dernie]

I am not interested by headphones but I would like to see distortion measurements at various power levels with speakers, in addition to the usual.

 

[q3cpma]

Speakers too. Doing at least what Neumann does would be nice (see the bottom of https://en-de.neumann.com/kh-310-a#technical-data, for example).
About the money problem, I'm sure that someone like amirm has a lot of loudspeakers to review already, which would give some useful information.

I am not interested by headphones but I would like to see distortion measurements at various power levels with speakers, in addition to the usual.

Same, this is an important measurement that unraveled an important problem with "The Ones" by Genelec.

 

[DuxServit]

Looks like we have “Brexit” event for Amir

The Pro-speakers people definitely win, but the logistics of sending physically large speakers may become the equivalent of the Irish “backstop” problem

 

[Mad_Economist]

I don't really see this as much of a problem. No one is suggesting that all headphones will fit across all people. There will sometimes be leakage effects, but those can be measured and corrected, if not too severe.

It's something that has to be considered individually. Rtings frequency response consistency can give you an idea of how good the seal is, but it's no substitute for trying them out yourself to see how they fit.

The bottomline is that the Harman curve was developed for a dummy head, not for actual people, however the curve enables us to make accurate predictions about what people will prefer. Yes, leakage effects will make the predicitons less accurate, but choosing a headphone that suits your head is no different from choosing clothing that suits your body.

Bear in mind, what appears - and feels - to be a reasonable fit may still introduce leakage in the front chamber, and these leaks will be of varying, and in some cases quite substantial, impact depending on the design of the headphones. IMO the fit of clothing isn't quite the right metaphor, unless designs that tend to couple poorly and have large leakage effects are similar to clothing that barely anyone can fit into.

This of course is why RTings tests response consistency, and similar efforts would be a good addition to any attempt to predict how headphones will sound, but as far as I'm aware none has been implemented in a model that has validated predictive power.

None of this is a criticism of the Harman work, mind you - this isn't its intention as such, given that it is primarily a search for the most preferred frequency response all else equal. It is, however, well worth keeping in mind when making inferences from the estimated preference ratings that Olive's model produces about real headphones, because it substantially increases the potential error in some cases.

My takeaway was the fact that distortion was demonstrated to be audible in headphones. That alone makes it worth considering an measuring.

I would agree with that. My contention was with the idea that distortion is necessarily all that remained in factors that influence perception, but it's certainly possible for it to be one of the things influencing results, and given how easily measured it is these days, there's little enough reason not to measure and consider it.

 

[amirm]

I am not interested by headphones but I would like to see distortion measurements at various power levels with speakers, in addition to the usual.

This is something I actually thought about doing for headphones a while back as with powerful amplifiers, I can easily push them into severe distortion.

My quick search for ultra-low distortion microphone has not resulted in much though. If people know of such sources, let me know.

 

[watchnerd]

What, no plans for measuring phono carts, @amirm ?

With a test record, it's probably not harder than speakers.

(But I voted for speakers)

 

[amirm]

What, no plans for measuring phono carts, @amirm ?

With a test record, it's probably not harder than speakers.

I have bought a test record already. Have not yet opened it.