Poll: Should We Get Into Testing Headphones or Speakers?

[Blumlein 88]

Headphones, test headphones. The market is bigger for us and it represents much better value.

Grow the forum with this , maybe try and get other outlets involved to creat a standardized method for testing .

Then in 18 months or so look at where we are and if the ( hopefully larger ) community can support a move to speakers .

Headphones also pair nice with the electronic you have been testing , mostly DACs and headphone amps.

Makes more sense to do this imo.

You make good sense Thomas. Except for two things. I don't think headphone testing that can ferret out good performance vs not good is well developed. And just being selfish, I hate listening on headphones.

I do have a question directed @amirm and @Floyd Toole .

Its been awhile sense I read Toole's book. Maybe it was covered there, and I forget, so partly I'm being lazy. I remember the predictive correlation of spin-o-rama results with listening testing was very close to 100% with small speakers with no bass below 100 hz and more than 80% with full range speakers. Is Harman's method of determining that publicly available or has that been kept to themselves? I've a general idea what is good and bad in seeing such test results from reading about them. If Amir can test speakers, and with 80% correlation tell us which is the better speaker at any given level that is really hugely useful news. If he is only going to tell us more generally this is good and this a problem, it is still useful, but less so.

I find with Stereophile vertical and horizontal results for speakers you can go back and see which speakers are more problematic which includes some big names. As well as those that are more reliable choices to pick from. But I couldn't use what's there or even total spin-o-rama results to do a fine tooth combing of possible speaker candidates.

Also I'm not convinced you get anywhere nearly that good a predictive ability from testing headphones. Without that headphone testing is mostly about finding the truly horrid designs, and not as much help if the phone isn't a total hack job.

 

[Thomas Savage]

You make good sense Thomas. Except for two things. I don't think headphone testing that can ferret out good performance vs not good is well developed. And just being selfish, I hate listening on headphones.

I do have a question directed @amirm and @Floyd Toole .

Its been awhile sense I read Toole's book. Maybe it was covered there, and I forget, so partly I'm being lazy. I remember the predictive correlation of spin-o-rama results with listening testing was very close to 100% with small speakers with no bass below 100 hz and more than 80% with full range speakers. Is Harman's method of determining that publicly available or has that been kept to themselves? I've a general idea what is good and bad in seeing such test results from reading about them. If Amir can test speakers, and with 80% correlation tell us which is the better speaker at any given level that is really hugely useful news. If he is only going to tell us more generally this is good and this a problem, it is still useful, but less so.

I find with Stereophile vertical and horizontal results for speakers you can go back and see which speakers are more problematic which includes some big names. As well as those that are more reliable choices to pick from. But I couldn't use what's there or even total spin-o-rama results to do a fine tooth combing of possible speaker candidates.

Also I'm not convinced you get anywhere nearly that good a predictive ability from testing headphones. Without that headphone testing is mostly about finding the truly horrid designs, and not as much help if the phone isn't a total hack job.

Just rig something up with this guy, it's what he would of wanted..,

 

[Krunok]

Headphones, test headphones. The market is bigger for us and it represents much better value.

Grow the forum with this , maybe try and get other outlets involved to creat a standardized method for testing .

Then in 18 months or so look at where we are and if the ( hopefully larger ) community can support a move to speakers .

Headphones also pair nice with the electronic you have been testing , mostly DACs and headphone amps.

Makes more sense to do this imo.

People of this forum have voted for loudspeakers so you better prepare some more used underware to sell instead of bragging about phones.

 

[flipflop]

Also I'm not convinced you get anywhere nearly that good a predictive ability from testing headphones.

I'm convinced you haven't read the relevant papers:
https://www.docdroid.net/enYZsTS/a-...tings-of-around-ear-and-on-ear-headphones.pdf
https://mega.nz/#!E9gCiKIb!q46zYi_ANJftB60KmaznOLRHo2HiBt1CcKAUIBfXo5M
https://mega.nz/#!Aww2Qa5A!AC8PgQjndsH33mgf4bLL3WWbUzs7Zzu7q6oUJTYxyLA

 

[Blumlein 88]

I'm convinced you haven't read the relevant papers:
https://www.docdroid.net/enYZsTS/a-...tings-of-around-ear-and-on-ear-headphones.pdf
https://mega.nz/#!E9gCiKIb!q46zYi_ANJftB60KmaznOLRHo2HiBt1CcKAUIBfXo5M
https://mega.nz/#!Aww2Qa5A!AC8PgQjndsH33mgf4bLL3WWbUzs7Zzu7q6oUJTYxyLA

I've read them. I'll look over them again.

Two things. One is they already changed their target curve. If their original data was good, why did that happen? Too many complaints the bass portion of the target curve didn't match up with results.

And at least some of that was emulations of various phones on the listener. I see the reason you would start with this, and that the results would be enlightening. But that isn't the same as testing the various phones which fit various ways and which are sized differently onto different ears. I don't recall them showing that was a non-issue, but I'll read back through the papers. Emulations with a single phone would simplify much about what makes headphones perform differently for me vs you.

 

[flipflop]

Two things. One is they already changed their target curve. If their original data was good, why did that happen? Too many complaints the bass portion of the target curve didn't match up with results.

Different test subjects give different results. The first iterations of the Harman target curve used a small sample of trained listeners, the second iteration used a larger number of untrained listeners. Both of those (now outdated) curves were fairly similar, though.

And at least some of that was emulations of various phones on the listener. I see the reason you would start with this, and that the results would be enlightening. But that isn't the same as testing the various phones which fit various ways and which are sized differently onto different ears. I don't recall them showing that was a non-issue, but I'll read back through the papers. Emulations with a single phone would simplify much about what makes headphones perform differently for me vs you.

If you have an issue with the emulations, I suggest you read this: https://seanolive.blogspot.com/2016/04/a-virtual-headphone-listening-test.html
Full paper here: https://mega.nz/#!kphGRC5C!22Et9gpCRXboR5ZNSdQAa0dDXxlDyxUepVDK8Q7W2eE
IEM study: https://mega.nz/#!s4wEXaTD!miMecGEKhsZMjZbCzHcfRnzM-f-oaXsDKS8c3eIVutU

I fail to see what your criticism has to do with the predictive capabilities of the models.

 

[Mad_Economist]

I fail to see what your criticism has to do with the predictive capabilities of the models.

As an example, fit issues have a meaningful impact in correlating frequency response on HATS/other measurement systems to perceived response and preferences on the head. This caused some of the disparity between the simulated and real headphones in Olive's paper on the virtual listening method, even, so it's certainly a factor that's worth considering and isn't included in the Harman model.

Which doesn't mean that it can't be evaluated, mind you - tests of behavior with broken coupling can be conducted, or headphones can be measured with multiple placements and positionings and variation plotted as a range rather than being averaged together as it often is, or headphones can be measured on real ears for low-frequency data (RTings does the latter two, for example, in their testing).

In general, I think that's true of many of the issues that can be raised with trying to infer results purely from a single FR plot - it's surprisingly viable, but it's not the be-all end-all, there are additional bits of information we can include to improve our analysis!

 

[flipflop]

As an example, fit issues have a meaningful impact in correlating frequency response on HATS/other measurement systems to perceived response and preferences on the head. This caused some of the disparity between the simulated and real headphones in Olive's paper on the virtual listening method, even, so it's certainly a factor that's worth considering and isn't included in the Harman model.

A proper seal might have been an issue during the development of the virtual headphone method, but in 'Preference Ratings of Around-Ear and On-Ear Headphones' they went to extreme lengths to make sure it wasn't the case:

The measured magnitude response of each headphone was then simulated over a replicator headphone (AKG K712) chosen for its low distortion, relatively smooth and extended frequency response. The open-back design of the replicator headphone provided a natural leak thus ensuring a more consistent response at low frequencies across listeners. [...] The replicator headphone was modified using a stiff, curved piece of wire to increase clamping force, which preliminary testing showed would decrease variability of leakage.

Which doesn't mean that it can't be evaluated, mind you - tests of behavior with broken coupling can be conducted, or headphones can be measured with multiple placements and positionings and variation plotted as a range rather than being averaged together as it often is, or headphones can be measured on real ears for low-frequency data (RTings does the latter two, for example, in their testing).

I'm glad you brought up Rtings. 1 of their 5 test subjects wears glasses while taking measurements of the frequency response. This results in crazy leakage effects for many of their headphones, especially the closed ones. With K712 there is none in the left channel and a modest -1 dB drop, at the lowest frequencies, in the right channel.
https://www.rtings.com/headphones/1-2/graph#327/2056
https://www.rtings.com/headphones/1-2/graph#327/2057

In general, I think that's true of many of the issues that can be raised with trying to infer results purely from a single FR plot - it's surprisingly viable, but it's not the be-all end-all, there are additional bits of information we can include to improve our analysis!

Those additional bits of information would primarily come from distortions, discussed in The Correlation Between Distortion Audibility and Listener Preference in Headphones, which I assume you have read. Distortion measurements combined with FR measurements would render all other factors close to negligible.

 

[StevenEleven]

My preference is just speakers, or speakers and headphones.

Speakers are the cornerstone to the best sound available to consumers. We don’t want to be pigeon-hold in the crowded headphone measurement and subjective review space.

A high performing low cost speaker is the most accessible route to an excellent sound system.

I agree headphone reviews will bring interest in from other places, and I like headphones, though I have enough headphones for a lifetime ; ). Cutting edge is good unbiased guidance on commercially available speakers though. If cost and practicality are factors, you could start in the budget space instead of the high end space, as others have suggested. My intuition is that some giant electronics companies are putting in really good efforts in that space, and others are mailing it in and manufacturing stuff with way below par sound.

Just my two cents!

 

[SIY]

Just out of curiosity, have any of you read the Kaspar Sprogis patents? He developed a very novel means of headphone measurement. Of course, it can't be used here, but it seems to have a great deal of merit...

 

[Mad_Economist]

A proper seal might have been an issue during the development of the virtual headphone method, but in 'Preference Ratings of Around-Ear and On-Ear Headphones' they went to extreme lengths to make sure it wasn't the case:

I feel that you're missing the key point that what Blumlein is asking about - and what I was commenting on - is making inferences about subjective impressions of actual rather than simulated headphones based on measurements. The K712 - and the HD518, which has also been used as a simulator by Olive - does not vary much in low-frequency response on users' heads, but many headphones do, and this is a meaningful factor to consider in interpreting how their measurements will correlate to our perceptions of them in use...

Those additional bits of information would primarily come from distortions, discussed in The Correlation Between Distortion Audibility and Listener Preference in Headphones, which I assume you have read. Distortion measurements combined with FR measurements would render all other factors close to negligible.

I have indeed read it, but that seems like a somewhat atypical takeaway from it to me - the conclusion itself states that "Nonlinear distortion in headphones of this high caliber [...] seems to not be a significant factor in how good it sounds.", reinforcing the primacy of frequency response, and not necessarily excluding other causes of variation in perceptions.

 

[Mad_Economist]

Just out of curiosity, have any of you read the Kaspar Sprogis patents? He developed a very novel means of headphone measurement. Of course, it can't be used here, but it seems to have a great deal of merit...

Are those the Sonarworks patents? If so, I have - it seems quite novel, but I'm not entirely certain that some parts of it make sense to me - IIRC the main test fixture leaves the pad empty, but measures response at many points, and there's something involving an "ear sock" with a bunch of mic capsules?

 

[SIY]

I judge it by the results- the EQ that resulted absolutely transformed the sound of some inexpensive headphones I had on hand. "Intuitive" it is not, but that doesn't mean that it isn't more correct than the conventional approaches. Nor that it is, if I want to be honest, but damn, it seems to work really, really well.

 

[617]

I see the value in both. As someone who's simulated and measured a few speakers, I don't find their responses particularly surprising or unpredictable, but for affordable speakers and smaller speakers it would be useful for the general public to have better data. The polar responses of different speakers of the same topology are compromised in the same ways, so the axial FR ends up being the most telling differentiator. For conventional dynamic speakers, if I know the cone sizes and materials I can pretty tell you where dips in power response will be.

The speakers which I'd like to see detailed measurements of are panel speakers such as quads and magnepans. People like how these speakers sound but I have no idea what their polar response looks like (probably not great in terms of treble dispersion.)

On the other hand, as we've been discussing, headphone measurement is very difficult, and I would like to see if we can create a class-leading protocol for assessing headphone performance in the public space.

 

[Mad_Economist]

I judge it by the results- the EQ that resulted absolutely transformed the sound of some inexpensive headphones I had on hand. "Intuitive" it is not, but that doesn't mean that it isn't more correct than the conventional approaches. Nor that it is, if I want to be honest, but damn, it seems to work really, really well.

Out of curiosity, have you compared it to equalizing to ex. the Harman target? This database contains a fairly large number of headphones and filter values for Harman equalization of said, if that's helpful. Subjectively, I have some quibbles with Sonarworks' compensation, but this is fairly common (all such tests have shown a healthy range of variation in preference) and not necessarily a specific issue with their methodology (Harman doesn't quite do it for me either, 2013, 2015, or 2017).

 

[SIY]

Out of curiosity, have you compared it to equalizing to ex. the Harman target? This database contains a fairly large number of headphones and filter values for Harman equalization of said, if that's helpful.

Unfortunately, none of the headphones I have are in that database.

 

[Mad_Economist]

Unfortunately, none of the headphones I have are in that database.

That's unfortunate. I do hope you get a chance to compare, sometime - it's always interesting to see the differences in such compensated results (and, IMO pertinent to our prior discussion today, how two headphones compensated to the same target may differ to my ear ).

 

[SIY]

Wish you lived closer, we could do some comparisons. Ah well.

 

[Mad_Economist]

We'll have to save that for the first ASR Megafestival, for which I'm sure the Tacoma Dome will be sold out...

 

[flipflop]

I feel that you're missing the key point that what Blumlein is asking about - and what I was commenting on - is making inferences about subjective impressions of actual rather than simulated headphones based on measurements.

I don't really see this as much of a problem. No one is suggesting that all headphones will fit across all people. There will sometimes be leakage effects, but those can be measured and corrected, if not too severe.

The K712 - and the HD518, which has also been used as a simulator by Olive - does not vary much in low-frequency response on users' heads, but many headphones do, and this is a meaningful factor to consider in interpreting how their measurements will correlate to our perceptions of them in use...

It's something that has to be considered individually. Rtings frequency response consistency can give you an idea of how good the seal is, but it's no substitute for trying them out yourself to see how they fit.

The bottomline is that the Harman curve was developed for a dummy head, not for actual people, however the curve enables us to make accurate predictions about what people will prefer. Yes, leakage effects will make the predicitons less accurate, but choosing a headphone that suits your head is no different from choosing clothing that suits your body.

I have indeed read it, but that seems like a somewhat atypical takeaway from it to me - the conclusion itself states that "Nonlinear distortion in headphones of this high caliber [...] seems to not be a significant factor in how good it sounds.", reinforcing the primacy of frequency response, and not necessarily excluding other causes of variation in perceptions.

My takeaway was the fact that distortion was demonstrated to be audible in headphones. That alone makes it worth considering an measuring.