Poll: Should We Get Into Testing Headphones or Speakers?

[scott wurcer]

If plotting the measurement results is the only issue, it is a much much easier problem to solve

True but the measurements needed to be processed by the algorithms and then plotted. I gave up on the thread where the guy doing all the work was simply sending data to the library functions and trying to guess if he was getting garbage or not.

 

[NTK]

It is the idea of measuring a speaker in a room but eliminating the effects of the room on what you measure. Reflections need to be extracted and dialed out of the measurements. And this is not easy.

It is NOT a simulation by the way. We are measuring a real speaker.

Ah, I see. Thanks. Because I believe Dr. Geddes also did simulations, so I was wondering. So we are purely trying to reverse engineer the Klippel system. I'd agree with the people here who thinks this would be too ambitious. Not only we need to know the math, we also need to have the hardware to test/debug/verify the software.

 

[bravomail]

I created a thread on motivation to tackle speaker testing: https://www.audiosciencereview.com/...-i-get-into-speaker-testing-measurement.7904/

I kinda solved the problem of good headphones and good dac/amp for myself. I'm now looking into good speakers/good poweramp combo. For headphones the good site is diyaudio. For headphone DAC and AMP - this site (thx, Amir!). We now see more and more good poweramp reviews too. But not any objective speaker reviews. I got pair of Elac B6 on Zeus advice. In light of this whole measurement revolution, Amir started, I still don't know if these speakers are objectively good.

 

[Julf]

For headphones the good site is diyaudio.

They do DIY headphones?

 

[jhaider]

I took quick look at the manual, there was nothing in the examples that several open source plot libraries could not do these days.

I wish I knew how to do something like that, or had the time to learn.

 

[scott wurcer]

I wish I knew how to do something like that, or had the time to learn.

You could go to the documentation for Python's matplotlib and find the examples section. They all come with the code so you can run it from the command line and figure out what each command is doing. Sort of learn by fiddling.

 

[jhaider]

You could go to the documentation for Python's matplotlib and find the examples section. They all come with the code so you can run it from the command line and figure out what each command is doing. Sort of learn by fiddling.

I'm sure that made sense to most of the participants here, but not to this coding-illiterate person. The phrase "command line" gives me hives. I don't even think my MacBook Pro has one of those!

Ditto complicated Excel formulas. I know CEA2034 power response can be plotted in Excel (Manny LaCarrubba does it) but it involves setting up formulas to do some complicated unit conversions to and then going back again.

 

[MZKM]

I spent the entire day yesterday reviewing what they have done at Rting. They are doing a first class job in the quality of the graphs and data they present. There are some issues with what they are doing though. Specifically, since they have a different measurement rig than Harman's, they have proceeded to make their own changes to target curves and measurements. For Bass they use human subjects and for treble, they use the HRTF graph from their HATS manufacturer.

They have justification for these changes but the problem is, they have performed no controlled testing to see if the changes are validated with respect to listening preference. Their own anecdotal testing is not good substitute for that.

Well, Harman took it upon themselves to take their data and correlate it with their objective criteria. This is what they found:

View attachment 28503

Ideal correlation would be 1.0 (objective data matching subjective). As you see, RTING is way down there around 0.5.

Sadly, there is more bad news in that their reviews seem to be biased toward price (the higher the price, the better the headphone):

View attachment 28504

The question Sean asks at the end is interesting one: is this real or a bias?

I also listened to the top rated headphone by Harman. It was exceptional in fidelity. Yet, it got an average rating from Rting.

This is why I say I don't want to manufacture my own science here. I like to start with Harman work and refine from there, not invent something new and hope it is as good or better than Harman's.

Rting also doesn't do any listening tests. I plan to do that.

But yes, your larger point is valid. Unlike many sites with poor graphs, hard to read and understand data, Rting has done an incredible job of presenting their data and certainly put a lot more work in their reviews than I do or plan to do.

Rtings is doing an AMA on Reddit soon and I plan to ask them about this.
In your opinion, what could they do to improve correlation?
• Change curve for treble to not be as relied on their dummy’s HRTF.
• Get the same rig as Harman’s.
• Get a better headphone amp (they use a Schiit Ragnarok).
• Change scoring algorithm for tests (how much deviation results in what score).
• Change weighting in the sound/critical listening category (what % each test contributes).
• etc.

 

[Dave Zan]

Are you referring to "The Waveguide" thread on diyaudio? If not, can you please provide a link to the one...

I think the thread he meant was probably https://www.diyaudio.com/forums/software-tools/318151-klippel-near-field-scanner-shoestring.html .
Sorry if this is a bit late, I was only just alerted to this discussion.

 

[dc655321]

I think he thread he meant was probably https://www.diyaudio.com/forums/software-tools/318151-klippel-near-field-scanner-shoestring.html .
Sorry if this is a bit late, I was only just alerted to this discussion.

Better late than never. Thanks for this link, Dave.
And welcome to ASR.

When I see someone say, "You pay some $40,000 for the software alone, telling you how confident Klippel is that this is too hard for just about anyone to solve.", I immediately file that as a case of mistaking unfamiliarity with improbability...

 

[Dave Zan]

And welcome to ASR.

Thank you

"You pay some $40,000 for the software alone....

I did mention more or less the same point in the DIYAudio thread, the software is expensive at least partially because it is difficult.
But of course, once someone works it out then it's no problem - especially if it's someone else who works it out.

Best wishes
David

 

[dc655321]

I did mention more or less the same point in the DIYAudio thread, the software is expensive at least partially because it is difficult.
But of course, once someone works it out then it's no problem - especially if it's someone else who works it out.

That's a fair point.
It's easy and incorrect for me to judge the difficulty of the problem based on a cursory study.
I simply object to categorizing this as "...too hard for just about anyone to solve."
I see it as a v. challenging mechanical system measurement problem, not of the same category as LIGO.

 

[Dave Zan]

...the difficulty of the problem...

It's difficult but I am with you, it looks doable.
As I studied more in that thread I learned a reasonable outline of the maths and never saw cause to alter that assessment.
So I am pleased that other people have decided to work on it.

Best wishes
David

 

[amirm]

In your opinion, what could they do to improve correlation?
• Change curve for treble to not be as relied on their dummy’s HRTF.
• Get the same rig as Harman’s.
• Get a better headphone amp (they use a Schiit Ragnarok).
• Change scoring algorithm for tests (how much deviation results in what score).
• Change weighting in the sound/critical listening category (what % each test contributes).
• etc.

#2: get the same rig and protocol as Harman.

Let me state that between speaker and headphone research at Harman, I far more trust the former than the latter. I have participated in those tests (speakers) myself, and countless other researchers find them definitive. Headphone research has been far less definitive and somewhat a moving target. The problem set itself is more challenging than speakers.

 

[dc655321]

It's difficult but I am with you, it looks doable.
As I studied more in that thread I learned a reasonable outline of the maths and never saw cause to alter that assessment.
So I am pleased that other people have decided to work on it.

I'm not quite half way through the DIY audio thread. Interesting reading, as are the NAH papers linked therein.
Thanks again for posting the link and your contributions.

 

[MZKM]

#2: get the same rig and protocol as Harman.

Let me state that between speaker and headphone research at Harman, I far more trust the former than the latter. I have participated in those tests (speakers) myself, and countless other researchers find them definitive. Headphone research has been far less definitive and somewhat a moving target. The problem set itself is more challenging than speakers.

Thanks, I asked them and here is the chain of comments.
Some stand out quotes:

Our current target uses the diffuse field response of our dummy head (Head Acoustics HMS) as the basis for the treble range. This may seem unjustified in the beginning, but if you compare the treble response of the AKG N700NC (which is tuned to the Harman target) to the DF response of our dummy head, you will see they are remarkably similar. The N700NC basically just has more bass. I have even discussed this with Harman.

In the upcoming test bench update, which should be published in less than a week, we are going even further away from Harman's model.

The number of headphones and subjects used in Harman's studies is a lot smaller than the number of our visitors and headphones. So they probably haven't experienced some of the issues we have run into. I believe even if we used Harman's model fully we would still feel the need to account for other factors, like the variation in user preference

I also stated that the Schiit they use to power their headphones has an output impedance of >10ohm and how it’ll effect low-impedance headphones, I recomended the JDS Atom and they said they’ll test it out as well as other options.

 

[Xyrium]

Bring on the speakers, love to see some stand mounters being measured. My curiosity about such measurements is: Are we measuring output of multiple frequencies at the same time, or just doing a sweep across from 20-20? Wouldn't a more realistic measurement for a speaker system's behavior be multiple frequencies (straight up pink noise?) in order to determine how a system can play, say, 50Hz, while that same driver is trying to reproduce varying amplitudes of a multitude of other frequencies, as is the case in real music.

I'm not an engineer (if that wasn't obvious , LOL) but I don't recall anyone doing this, perhaps it's just not realistic? If a system plays 6 octaves of frequencies at the same time, for an average SPL of 75dB, what is the average distortion product? Stuff like that would be interesting to me....

 

[Julf]

Bring on the speakers, love to see some stand mounters being measured. My curiosity about such measurements is: Are we measuring output of multiple frequencies at the same time, or just doing a sweep across from 20-20? Wouldn't a more realistic measurement for a speaker system's behavior be multiple frequencies (straight up pink noise?) in order to determine how a system can play, say, 50Hz, while that same driver is trying to reproduce varying amplitudes of a multitude of other frequencies, as is the case in real music.

I'm not an engineer (if that wasn't obvious , LOL) but I don't recall anyone doing this, perhaps it's just not realistic? If a system plays 6 octaves of frequencies at the same time, for an average SPL of 75dB, what is the average distortion product? Stuff like that would be interesting to me....

It is pretty much standard procedure - both multi-tone IM measurements and wide-band white noise or impulse response.

 

[solderdude]

I'm not an engineer (if that wasn't obvious , LOL) but I don't recall anyone doing this, perhaps it's just not realistic? If a system plays 6 octaves of frequencies at the same time, for an average SPL of 75dB, what is the average distortion product? Stuff like that would be interesting to me....

I have tested headphones using sweep (REW) and by capturing white noise over 1 minute and then FFT.
They matched pretty well so stopped measuring this way.
Only with BT headphones I still check FR this way because of practical issues with my setup.
Not with speakers though.

 

[Xyrium]

Nice, that's what I was hoping. Thanks for the clarification folks!