Poll: Should We Get Into Testing Headphones or Speakers?

[watchnerd]

I have bought a test record already. Have not yet opened it.

Let me know when you open up the flood gates for people to mail in carts.

 

[Fledermaus]

Hi,
Been lurking for a while now, 1st message here. Voted speakers, for I'm marginally interested in headphones, but would like to know whether there are decent yet affordable speakers on the market, the way there are such electronic devices. If so, I think those should be brought into light.
As for crappy expensive ones, I'm not holding my breath

 

[oivavoi]

The first time I got exposed to "harman research" I rolled my eyes. I then set out on a nearly 2 year project to not only read and understand their research, but also all the underlying and supporting research of which there are many. Dr. Toole's book alone has 270 references! I also then participated twice in their double blind test of speakers. By then the evidence was so overwhelming that no person with an ounce of common sense would refuse to accept it.

Everyone seems to have a "story" about what makes good sound in speakers and rooms. But none come remotely to the completeness and comprehensiveness of what is advocated by Dr. Toole and Sean Olive. To give up on what that research tells us, makes no sense at all. Indeed many competitors of Harman use the research in their development of their products.

Let's remember that a lot of what is said to be "harman" research was discovered and developed by Dr. Toole/Olive at canadian NRC so the commercial interest here is but a distant factor.

Again, others are welcome to put forward their stories and what backs them. No commercial entity though that I know of can remotely match 1% of the careful listening tests and studies that has been performed on this front.

And no, I am not and will not marry Harman research. I will be marrying the only solid and comprehensive research we have about good sound. I am not going to throw that away and go by some gray haired speaker designer at some company thinks.

That said, if we get into this measurement field, we will create our own body of data, both subjective and objective. So if you care about an alternative to "harman method," then you should help fund this activity. Nothing in Klippel system is about Harman other than generating the same set of measurements they use which is simply how the speaker radiates energy at different frequencies and angles. I can then setup listening tests and compare our findings to that of Harman.

I don't think anybody advocates throwing out the very valuable research done by the people at Harman!

I think it's rather a question about the level of certainty one can have about some of Harman's own conclusions, and how to present measurements. Everybody agrees that good behaviour on-axis is important, and everybody agrees that good behavior off-axis important. Harman's research (and dr. Toole's previous research at the NRC) has been very important for solidfying these important basic points. But how does one determine trade-offs? Say that speaker A behaves significantly better off-axis than speaker B, but somewhat worse on-axis. This can be a real trade-off when designing speakers. Should these speakers be ranked according to the Harman design philosophy, which probably would put speaker A above speaker B, even for listeners sitting exactly in the sweetspot? Is the evidence presented by Harman sufficient to establish that this indeed the case? I'm not 100 percent sure about that.

In this case, I think it's better to just say that "speaker A behaves better off-axis, and speaker B better on-axis" - and ideally couple it with additional blind listening impressions. My own guess - which could be wrong - would be that speaker B would sound better if listening exactly in the sweetspot relatively close to the speakers, but that speaker A would sound better everywhere else. I think that kind of nuanced information would be more valuable than a simple ranking of A over B (or vice versa).

It's also important remember that no scientific investigations are infallible. Almost all scientific findings can turn out to be imprecise, more or less close to the truth, etc. That's one of the reasons why it's vitally important to have more than one research group working on similar questions, so as to get a vigorous methodological debate. The problem in loudspeaker research is that there are so few research groups working with it, so the scientific debate is still in its infancy (and will probably stay there, alas, because hifi is waning as high-funded hobby). We know some things - thanks in no small part to dr. Toole and Harman - but we don't know everything.

Here are some areas where I think we still don't know enough, and it would be good with more available data:
- loudspeaker directivity: are some directivity patterns better than others, and what are the trade-offs?
- dynamics and maximum spl: Does more available headroom make a difference?
- active vs passive: all else being equal, do active crossovers matter?

And there may be more. The point is just to avoid overconfidence, I guess. But the more I think about it, the more I like the idea of ASR getting into measuring loudspeakers!

 

[Fledermaus]

My own guess - which could be wrong - would be that speaker B would sound better if listening exactly in the sweetspot relatively close to the speakers, but that speaker A would sound better everywhere else. I think that kind of nuanced information would be more valuable than a simple ranking of A over B (or vice versa).

My guess would be that all else being equal, it would depend on the listening room and distance, which are conditioning the relative quanta of direct and reflected sound you're exposed to. Listening at close range in a dead room emphasizes direct sound, hence - in my view - in favor of B, while beyond critical distance in a reflective room, maybe A would fare better. So the ranking and choice depends in part on listening conditions - which still underlines the importance of that choice being a technically informed one.

 

[oivavoi]

My guess would be that all else being equal, it would depend on the listening room and distance, which are conditioning the relative quanta of direct and reflected sound you're exposed to. Listening at close range in a dead room emphasizes direct sound, hence - in my view - in favor of B, while beyond critical distance in a reflective room, maybe A would fare better. So the ranking and choice depends in part on listening conditions - which still underlines the importance of that choice being a technically informed one.

Good points, I agree. My own subjective and sighted experience is that speakers of type B - a typical small 2-way monitor, basically - can sound very good if listened to in the near-field, either in a treated room or in a normal room but well away from walls. But it may not work at all if listened to at a distance.

 

[Fledermaus]

IME, at a distance, it's the same but with more directive speakers (big 2-ways, dual 12" + 1' compression) with somewhat interrupted or absent boundaries on the sides of the room and a rather wide space above, so still a major part of direct sound. In that regard, I'd be more in E. Winer's than in Dr Toole's camp - but that's just me with my subjective tastes based on my modest experience - never heard Salon's in a cosy, semi-reflective lounge !

 

[jazzendapus]

Voted for speakers testing.
While headphone testing is obviously much easier logistically and less cost prohibitive, there's simply a very strong competition in the form of rtings.com. They're strong on science (tight with HARMAN), have a good amount of followers and backers and are seemingly independent. They also deal with issues of headphones testing that were raised here and in other threads seriously. It just looks like a well oiled machinery at this point.
So what do you think you can offer that rtings don't cover already when it comes to headphone testing?

The arguments for speaker testing (there's not enough reliable data on the web and too much payola) and against (very expensive and time consuming, logistically difficult) are clear. But it seems like the ASR project has pretty much completed its mission when it comes to DACs and headphone amps - get a JDS Atom and something like Topping D50 and it's pretty much game over. For good. Reviewing devices with a few more/a few less db's of SINAD for +-50$ isn't too interesting imo. Why bother when you can just slap 300-350$ and be done with this ordeal for life?

There's still some stuff to do when it comes to power amps/integrateds/avrs. For example is there a sub 1000$ all-in-one, multichannel unicorn with guaranteed transparency for CD standards? I'd really love to get a clear answer to this question. But this too is a goal with a clear end in sight. Going for speaker testing, even if limited to small bookshelves at first, is simply the next logical step for the ASR project.

 

[amirm]

While headphone testing is obviously much easier logistically and less cost prohibitive, there's simply a very strong competition in the form of rtings.com. They're strong on science (tight with HARMAN), have a good amount of followers and backers and are seemingly independent. They also deal with issues of headphones testing that were raised here and in other threads seriously. It just looks like a well oiled machinery at this point.
So what do you think you can offer that rtings don't cover already when it comes to headphone testing?

I spent the entire day yesterday reviewing what they have done at Rting. They are doing a first class job in the quality of the graphs and data they present. There are some issues with what they are doing though. Specifically, since they have a different measurement rig than Harman's, they have proceeded to make their own changes to target curves and measurements. For Bass they use human subjects and for treble, they use the HRTF graph from their HATS manufacturer.

They have justification for these changes but the problem is, they have performed no controlled testing to see if the changes are validated with respect to listening preference. Their own anecdotal testing is not good substitute for that.

Well, Harman took it upon themselves to take their data and correlate it with their objective criteria. This is what they found:

Ideal correlation would be 1.0 (objective data matching subjective). As you see, RTING is way down there around 0.5.

Sadly, there is more bad news in that their reviews seem to be biased toward price (the higher the price, the better the headphone):

The question Sean asks at the end is interesting one: is this real or a bias?

I also listened to the top rated headphone by Harman. It was exceptional in fidelity. Yet, it got an average rating from Rting.

This is why I say I don't want to manufacture my own science here. I like to start with Harman work and refine from there, not invent something new and hope it is as good or better than Harman's.

Rting also doesn't do any listening tests. I plan to do that.

But yes, your larger point is valid. Unlike many sites with poor graphs, hard to read and understand data, Rting has done an incredible job of presenting their data and certainly put a lot more work in their reviews than I do or plan to do.

 

[amirm]

I must say, I am really surprised by the results of the poll so far, with nearly 70% asking for speaker testing. I thought for sure our audience would be far more biased toward headphones at this point.

 

[jazzendapus]

That's interesting, do you have any idea what kind of scores from rtings' database were used in order to calcualte the correlation in the first slide? I have a feeling the correlation is low bcause rtings weight in other factors besides frequency response (like soundstage perception, distortion etc) into the calculation of the final score, if that what was used, while Sean Olive, afaik, only uses frequency response in his models to predict preference. If the correlation was simply based on rtings' freq. response measurement, it would be very high as they use something very similar to Harman headphone curve.

 

[svart-hvitt]

I spent the entire day yesterday reviewing what they have done at Rting. They are doing a first class job in the quality of the graphs and data they present. There are some issues with what they are doing though. Specifically, since they have a different measurement rig than Harman's, they have proceeded to make their own changes to target curves and measurements. For Bass they use human subjects and for treble, they use the HRTF graph from their HATS manufacturer.

They have justification for these changes but the problem is, they have performed no controlled testing to see if the changes are validated with respect to listening preference. Their own anecdotal testing is not good substitute for that.

Well, Harman took it upon themselves to take their data and correlate it with their objective criteria. This is what they found:

View attachment 28503

Ideal correlation would be 1.0 (objective data matching subjective). As you see, RTING is way down there around 0.5.

Sadly, there is more bad news in that their reviews seem to be biased toward price (the higher the price, the better the headphone):

View attachment 28504

The question Sean asks at the end is interesting one: is this real or a bias?

I also listened to the top rated headphone by Harman. It was exceptional in fidelity. Yet, it got an average rating from Rting.

This is why I say I don't want to manufacture my own science here. I like to start with Harman work and refine from there, not invent something new and hope it is as good or better than Harman's.

Rting also doesn't do any listening tests. I plan to do that.

But yes, your larger point is valid. Unlike many sites with poor graphs, hard to read and understand data, Rting has done an incredible job of presenting their data and certainly put a lot more work in their reviews than I do or plan to do.

Do you intend to take HRTF out of the equation when giving headphone advice?

 

[amirm]

That's interesting, do you have any idea what kind of scores from rtings' database were used in order to calcualte the correlation in the first slide? I have a feeling the correlation is low bcause rtings weight in other factors besides frequency response (like soundstage perception, distortion etc) into the calculation of the final score, if that what was used, while Sean Olive, afaik, only uses frequency response in his models to predict preference. If the correlation was simply based on rtings' freq. response measurement, it would be very high as they use something very similar to Harman headphone curve.

It is actually on the slide: they tested the frequency response and sound scores. That is why there are two bar graphs there for Rting.

 

[amirm]

Do you intend to take HRTF out of the equation when giving headphone advice?

Just to clarify, "HRTF" here means the manufacturer provided free-field or diffused field correction data for their test rig. Both of these have been shown to have poor correlation with listening preference. So I don't intend to use these HRTF data but use Harman curve for correction. Is this what you are asking?

 

[jazzendapus]

It is actually on the slide: they tested the frequency response and sound scores. That is why there are two bar graphs there for Rting.

I see now.
Their targets aren't as similar as I thought, so that's what throws the correlation off.
But practically they could simply use a Harman target option for calculating the sound score in addition to their own. Given how neatly they collect and present the data, I think it would be close to effortless for them to implement something like that. But yeah, I agree on the point that their hybrid target curve needs some serious backing with blind tests.

The higher than Harman's correlation of price and score is something I'm not too worried about as their scoring system is completely transparent. It really doesn't look like they took the more expensive headphones and derived their targets from their performance, the correlation in the 2nd slide would be much higher if that were the case. There's at least a theoretically sensible basis for ideal targets in each category.

 

[oivavoi]

I also listened to the top rated headphone by Harman. It was exceptional in fidelity. Yet, it got an average rating from Rting.

Got me curious: Which one was this? If it's open to the public to know? If not, I'd love to hear it on pm

 

[amirm]

Got me curious: Which one was this? If it's open to the public to know? If not, I'd love to hear it on pm

Sean wouldn't say in public so I am not going to do that either.

 

[oivavoi]

Sean wouldn't say in public so I am not going to do that either.

Ok!

 

[MZKM]

I also listened to the top rated headphone by Harman. It was exceptional in fidelity. Yet, it got an average rating from Rting.
Rting also doesn't do any listening tests. I plan to do that.

Odd how their FR score was so low, not sure what can be explained as it’s modeled after Harman, maybe it’s the weighting/threshold of how they give a score. I wonder how they rated Rtings on just frequency response and not just Sound, I assume they averaged the scores, which can be a slight issue as I feel bass and midrange are a more important than treble (Rtings does 21% for bass, 20% midrange, 19% treble; I would go even further).

As for their score increasing with higher price, their score for sound is objective, only things like comfort as subjective, so I don’t see how a case for a large bias could be made.

Rtings does listen to the headphones, and sometimes state that in the review when talking about the bass or whathaveyou, but they of course don’t publish a listening test score.

I agree though, that since they use a different rig than Harman, and that their “anechoic chamber” is just a large box, that there are things to improve on.

 

[svart-hvitt]

Just to clarify, "HRTF" here means the manufacturer provided free-field or diffused field correction data for their test rig. Both of these have been shown to have poor correlation with listening preference. So I don't intend to use these HRTF data but use Harman curve for correction. Is this what you are asking?

So like @Floyd Toole derides room correction as a marketing story, you don’t see HRTF as a problem that should be compensated for in headphones listening?

In other words:

(1) Don’t use room correction (ok below Schroeder)
(2) Don’t use HRTF correction

Is this what science says?

 

[MZKM]

So like @Floyd Toole derides room correction as a marketing story

Does he? I think he just greatly favors buying good speakers and doing room treatment first, and then doing room correction as a last step.