Should we (I) get into speaker testing & measurement

[617]

The problem is that the response is inextricably tied to the shape of the pinnae (and no two are the same) and the way the headphones fit and compress the ears.

If I even do something as simple as taking the headphones off the jig, then replacing them, I see 5 dB or more frequency response swings. By contrast, when I repeated measurements on some speakers taken in my basement in Illinois before I moved and in my lab in Phoenix after I moved, they were almost identical.

Now, that said, if I see a frequency response feature that persists even when using very different test jigs, I can attribute that to the headphones. But unlike speakers, I can't correlate all of the other parts of the measurement to the sound.

That's very interesting. I wasn't aware they exhibited so much variability. I wonder what would happen if they were taken off and replaced 20 times and measurements averaged?

 

[Krunok]

The problem is that the response is inextricably tied to the shape of the pinnae (and no two are the same) and the way the headphones fit and compress the ears.

Does that mean that same headphones will sound differently to different people, not subjectively but objectively, for the reasons you just mentioned?

 

[SIY]

That's very interesting. I wasn't aware they exhibited so much variability. I wonder what would happen if they were taken off and replaced 20 times and measurements averaged?

That's pretty much how I did my headphone measurements for AX reviews. But there's a saying in my native language, "Es vet hilfin vi a toten bankes," which roughly corresponds to "bandage on a corpse." Change the jig and now you have a totally different frequency response.

For speakers, if changing calibrated microphones changed my measurements that dramatically, I'd give up measuring speakers.

 

[617]

That's pretty much how I did my headphone measurements for AX reviews. But there's a saying in my native language, "Es vet hilfin vi a toten bankes," which roughly corresponds to "bandage on a corpse." Change the jig and now you have a totally different frequency response.

For speakers, if changing calibrated microphones changed my measurements that dramatically, I'd give up measuring speakers.

We really should be able to do better. Perhaps we can take after Harman and objectify a panel of subjective impressions?

 

[SIY]

Does that mean that same headphones will sound differently to different people, not subjectively but objectively, for the reasons you just mentioned?

Yes. And I have experimental evidence for that- when @jan.didden and I reviewed the Smyth Realiser, part of the calibration process is measuring response inside the headphone cup while the user is wearing them. He and I swapped cal files and the magic that the Smyth did totally disappeared. From an audibility standpoint, it was grossly different.

 

[MZKM]

Related question - is there a reliable way to measure headphones? Could someone give me a rundown? The ear/transducer interface seems incredibly variable, and the measurements I've seen of headphones suggest really poor performance.

Are you looking at raw FR measurements? If so, that’s not gonna help, it‘s reading in binary, sure you can do it, but takes to much time, you gotta look at it after being converted to a response one would usually see from say a speaker.

For instance (from Rtings):
Raw measurement of HD800S:

Converted:

 

[svart-hvitt]

That's pretty much how I did my headphone measurements for AX reviews. But there's a saying in my native language, "Es vet hilfin vi a toten bankes," which roughly corresponds to "bandage on a corpse." Change the jig and now you have a totally different frequency response.

For speakers, if changing calibrated microphones changed my measurements that dramatically, I'd give up measuring speakers.

Can of worms, really.

 

[SIY]

Can of worms, really.

Yep, exactly.

 

[Krunok]

Yes. And I have experimental evidence for that- when @jan.didden and I reviewed the Smyth Realiser, part of the calibration process is measuring response inside the headphone cup while the user is wearing them. He and I swapped cal files and the magic that the Smyth did totally disappeared. From an audibility standpoint, it was grossly different.

Huh.. So no measurement and no correction unless done with headphones on your head, right? And even then all this would be valid only for you..

 

[SIY]

Huh.. So no measurement and no correction unless done with hehadphones on your head, right? And even then all this would be valid only for you..

Yep, exactly.

 

[tw99]

I voted no because I don't see that the ROI is worth it. Speakers are a very personal choice, they interact heavily with the room, there are definite aesthetic considerations, so I'd argue that measurements are less critical in a buying decision than with electronics, even for those with an objectivist approach.

 

[daftcombo]

I voted no because I don't see that the ROI is worth it. Speakers are a very personal choice, they interact heavily with the room, there are definite aesthetic considerations, so I'd argue that measurements are less critical in a buying decision than with electronics, even for those with an objectivist approach.

For me, there would be 0% aesthetic in the choice of a speaker if I'm convinced I make the best choice in terms of sound quality.

Measurements should be more critical, since a difference in frequency response will be much more audible than any difference in DACs or amps. Try and play a bit with an EQ if you're not convinced.

 

[617]

I voted no because I don't see that the ROI is worth it. Speakers are a very personal choice, they interact heavily with the room, there are definite aesthetic considerations, so I'd argue that measurements are less critical in a buying decision than with electronics, even for those with an objectivist approach.

The current ethic in speaker design is to design speakers which are relatively indifferent to rooms. Polar speaker measurements are highly predictive of sound quality although certainly not to the same degree as source units and amps.

 

[PierreV]

No, as well for me.

DACs can be purely objective, in a mathematical way.

Amplifiers can also be, but there is also some (a lot?) of fuzziness in the way they interact with their load. It is objective, but not as clear cut as DACs.

Speakers... the field is not as objective. There is a huge difference, imho, between comparing an input function and an output function (DACs) and assessing the conformity of the measurements of one subset of characteristics of a speaker among many with a standard version of that subset of characteristics that has been defined by manufacturer supported studies. Don't get me wrong, I do not mean that I don't respect those studies or that they are wrong (in fact, I tend to find my own taste correlates well with them, I just seem to belong to different subsets on different types of music) but, let's face it, self-reported preferences on a limited set of test devices would be considered very weak evidence in other fields...

DAC = very formal framework. Provability. Amps = somewhat less formal framework, provability assuming the real world (the load) collaborates. Speakers : at best... assuming the measurement show a good fit with the "reference", what you get is a predictor (uncertainty) of what most people (again uncertainty) apparently prefer. That's a good test for businesses - how should I design my speaker in order to have the best chance to sell it to most people - not really a test for individual purchases.

It's a swamp. Teetering perilously on the edge of subjectivity.

And, ofc, all the issues with logistics, test conditions, probable impossibility to test bigger speakers, different speaker markets in different parts of the world. Also, flagships and huge sellers are usually fairly well documented.

Recommended DAC? OK, perfectly rational. Recommended AMP? OK, provided it interacts well with my load. Recommended speaker? Hmmmmmmm...

 

[MSNWatch]

Yes but as pointed out here considerable hurdles both financial and logistic. I for one would be willing to chip in with money to help make this happen. Maybe set up a Gofundme account and see if there is sufficient financial support? If target is not reached then maybe refund the money? Also might be worth exploring purchasing options - perhaps a wholesale price can be negotiated with Klippel given the huge amount of exposure they will be getting from this site? Or some kind of rent to purchase option so payments can be spread out?

 

[amirm]

perhaps a wholesale price can be negotiated with Klippel given the huge amount of exposure they will be getting from this site?

I am working on this angle but even in the best case scenario, we are talking big dollars.

 

[maty]

Too much money and time for one person I think. You should think more about your family and your business.

 

[oivavoi]

I voted yes, but with some ambivalence. I see two questions here:

1) Can it be done? -- in some sense, I'm pretty sure the answer to that is yes (would be awesome if we could crowdfund a Klippel thingy for Amir/ASR!)

2) Can it be done well - and can it generate results that will not be misleading with regards to preference or perceived fidelity?

-- On that question, I'm not so sure. It sounds to me like @KSTR really know what he's talking about. Are there other members here who have been involved in speaker testing and speaker production at a comparable level? If so, I think it would be worthwhile to use their competence and experience when thinking about this. One reason I'm a bit skeptical is that I don't think speaker design is a solved issue - that there is one correct way of doing it that Harman has uncovered. Should dispersion be broad or narrow? Dr. Toole says broad, dr. Geddes says narrow. Both of them know more about psychoacoustic science than the rest of the world combined. The speaker that received the highest rating ever at the NRC was as far as I know a bipole quasi-omni speaker from Mirage. But Harman never made such a speaker. Was that based on comparisons between the Mirage and other speakers, or based on commercial considerations?

And there have been done psychoacoustic work on speakers since Toole/Harman which seem to emphasize other aspects, beyond dispersion - like this article, which was mentioned in another thread by @svart-hvitt recently: http://www.aes.org/tmpFiles/elib/20190626/18729.pdf

Other examples can be given. @KSTR mentioned in another thread that one difference between the speakers he developed at Hedd and the Neumann monitors is that Neumann prioritized completely flat frequency response on-axis, while Hedd gave somewhat higher priority to the on-axis behaviour. Can we be certain that Hedd was right and Neumann wrong on this issue? I'm not sure.

Or, last example: The blind test comparison between the Revel Salon and the JBL M2 which was documented at AVS forum: https://www.avsforum.com/forum/89-s...ccurate-well-reviewed-speakers-ever-made.html
I'm still not sure anybody really knows why the Salon was rated as the better speaker.

But I don't mean to only be negative here... I have a suggestion!
There are two things that are missing online. More complete speaker measurements is one thing. But valid blind listening impressions is another thing. And in particular work which links measurements to listening impressions.

So here is my proposal: Say that @amirm manages to get this Klippel thing going. Awesome! How about for each speaker that is reviewed/measured, one also assembles a small panel of listeners who listen to that speaker behind a blind or something, without knowing which speaker it is, and compare it in mono to one of Amir's Salon speakers? - and provide notes/scores? This would be super cool, I think, as it would provide further hints about the correlation between certain objective facets of speakers, and how we actually perceive them.

The implication of this would of course be that each review would be more time consuming. But I think it would make the reviews even more valuable. The LTS does this for the electronics they review, with their elaborate bypass tests... so timewise it should be possible, given that one can find panels in the Seattle area that are willing to participate.

Anyway, great initiative and great idea, and I'm looking forward to following the continued discussion.

 

[amirm]

So here is my proposal: Say that @amirm manages to get this Klippel thing going. Awesome! How about for each speaker that is reviewed/measured, one also assembles a small panel of listeners who listen to that speaker behind a blind or something, without knowing which speaker it is, and compare it in mono to one of Amir's Salon speakers? - and provide notes/scores? This would be super cool, I think, as it would provide further hints about the correlation between certain objective facets of speakers, and how we actually perceive them.

I am planning to do a limited version of this, recruiting my son who is in his 20s and has superb high frequency hearing. Mine are shot and this is one of the reasons I want to bring instrumentation into this rather than using my ears as proxy for all of you. Of course I still hear well in lower octaves but won't be a complete answer.

Our local audiophile society could be recruited to do larger scale testing but obviously this is not something we can do frequently. We could do shoot outs though like folks organized which got talked about at AVS.

On the rest of your questions, all of them require data and that is what is missing. While I plan to start with body of knowledge from Harman/NCR work by Dr. Toole/Olive, I think we will all learn more together as we get large sample size of speakers all measured the same. This has happened with electronic testing as evidenced by refinement of the tests I run.

 

[amirm]

Speakers... the field is not as objective. There is a huge difference, imho, between comparing an input function and an output function (DACs) and assessing the conformity of the measurements of one subset of characteristics of a speaker among many with a standard version of that subset of characteristics that has been defined by manufacturer supported studies. Don't get me wrong, I do not mean that I don't respect those studies or that they are wrong (in fact, I tend to find my own taste correlates well with them, I just seem to belong to different subsets on different types of music) but, let's face it, self-reported preferences on a limited set of test devices would be considered very weak evidence in other fields...

I hear you on all of this. As with electronics though, I think there are some real dogs out there as far as speakers. The race to the bottom as far as pricing, and focusing on marketing and looks, has most likely created pretty horrible speakers out there. Measurements of them I suspect will be so revealing that there won't be a lot of doubt about their lack of fidelity.

As I noted, I also hope to setup some kind of rudimentary blind testing to go with the objective measurements.