I cannot trust the Harman speaker preference score

[Putter]

I may have expressed myself poorly. I was not criticising the statistical model but the methodology used to gather the data.

All this fanboyism in regard to Harman's research is not healthy...

It's better if people buy a better speaker than one recommended by a 'subjective' reviewer who likes everything. Remember that at this time there are very few places where you can listen to different speakers, esp. those in the budget category. I also HATE the 'fanboy' tag. It's an insult and implies a mindlessness rather than an excess of enthusiasm. It's a 'cancel culture' term that says that a person's opinion is not worthwhile.

 

[Kennyknetter]

As somebody commented earlier, I think the understanding of the score gets more intuitive if you call it a tonality score, rather than a preference score.

What tonality does the speaker have in any direction, and how close does the tonality match the general preference. That's the question that the score answers.

Now, how important is a smooth vertical off axis response when compared to a reduced level of distortion?

And, how important is a wide horizontal beamwidth in the uppermost octave, if you have two speakers?

The score is helpful, but not everything.

When comparing mostly the same speakers, I do take the preference score into account. Less so otherwise.

 

[Bjorn]

Finally, I think that the Harman research is right. Tonality does reign supreme. Yes it doesn't measure everything. But if a speaker doesn't get the tonality and directivity right than don't bother with that speaker. It is much more important than distortion. It is also much more important than output and sometimes we have to admit to ourselves that past a certain point, more driver area does nothing if you listen below "hearing damage" levels in medium rooms.

But which it doesn't do very well for the reasons I mentioned in the post below.

I cannot trust the Harman speaker preference score

I always like to tell myself that the Harman Score is the Harman Tonality Score. Tonality is 95% of why we like speakers, especially at lower volumes, so of course it is very very important and the most important thing, in fact. Once you know that it is only a tonality score and not a volume /...

www.audiosciencereview.com

A speaker with a collapasing polar high in frequency, and that doesn't minimize floor bounce and floor/ceiling reflections will never be great in tonality in a traditional room.

Whether the response is more important than distortion and other areas will of course depend on large the deviations are and what type of devaitions we're talking about. There's a good reason why an old Klipsch heritage design is capable of giving you to illusion of hearing real instruments and voices, while the traditional front firing speaker isn't. Obviously we want both a correct tonality and low distortion as well as other things.

 

[FrantzM]

Hi

I see a lot of "Appeal to Authorities" fallacies in this discussion.
The people who came up with this score are luminaries no doubt and are deserving of praises and respect. Their contributions to the Arts and Science of Music reproduction may well be unmatched.
This metric, the Preference Score, however should be seen as a good starting point, not a definitive score. I don't see other ways to put it: it is inconsistent. Let's not bend backward trying to make of it what it isn't. As it is , it doesn't fully grade speakers. Even its basis are not as clear as many others from the same people .e.g. Blind test and peer reviews. Was it Peer-reviewed BTW? Not sure.

Peace

 

[abdo123]

and that doesn't minimize floor bounce and floor/ceiling reflections will never be great in tonality in a traditional room.

Can you give examples? Some supporting measurements perhaps?

 

[amper42]

Take the Revel M105 and compare it side by side with the Revel F328Be. You can place a screen in front of the speakers so you can't see each speaker and the audio is so radically different each person can easily identify each speaker in a blind test.

F328Be - 6.5 score posted on ASR, with sub 7.7
M105 - 5.9 score posted on ASR, with sub 8.2

I purchased the M105 based on this tremendous rating. I already had the F328Be. I thought it would be interesting to compare the two. Here what I discovered:
1. The M105 is a fine speaker but low end is lacking and distorts easily if not crossed over at 80Hz with a sub.
2. The M105 doesn't have the same wonderful downward sloping FR as the F328Be.
3. The M105 tweeter doesn't offer the same "realistic" detail as the Be.
4. The F328Be can easily fill a large room while the M105 is best suited for a smaller 12'x12' space.
5. There is absolutely no way the M105 sounds better than the F328Be yet the score with sub indicates it does.
6. No one who has spent time with both speakers next to each other would argue the M105 is within a 0.6 score difference of the F328Be. It's a bold conclusion between a large tower and a dual driver bookshelf that's totally false in actual listening tests. Yet, it's displayed as truth in the speaker scoring system.

Next, I compared the current shipping BMR Monitor to the Revel M105.
The BMR Monitor isn't listed on the ASR Speaker Measurement Index. The BMR Monitor is listed on the https://pierreaubert.github.io/spinorama/index.html site.

BMR Monitor - 5.1 score posted on Pierre Site, with sub 7.3
M105 - 5.8 score posted on Pierre Site, with sub 8.2

When I compare the BMR Monitor to the M105 - this is what I experience in a 12'x13' room (my office).
1. The BMR Monitor has a full strong low end extension that's audibly missing in the M105.
2. The fullness of the 3 way BMR Monitor is much preferred by my ear compared to the M105.
3. The M105 is 13.5" tall while the BMR Monitor is 20". It's basically 50% taller with a larger rear port.

The RAAL 64-10X tweeter, 2.5” Tectonic Balanced Mode Radiator and SB Acoustics 6" Ceramic woofer combination offers a much fuller, engaging sound than the two-way M105 (1" aluminum tweeter/5.25" cone). Yet, the speaker score rates the the M105 a full 0.7 higher than the BMR Monitor. For me, this was another example of why I should assign the Harmon speaker score little value. As far as I am concerned the Speaker score can easily misrepresent which speaker will sound best for my use. Caveat emptor!

Bottom line, when you can't count on the Harmon score to reflect your taste it's best to assign it a lower weight in speaker purchase decisions. If you think it is a very reliable tool for selecting your next speaker you will be fooled.

 

[IPunchCholla]

KH120s have a significantly higher preference score than JBL 708Ps (6.7 vs 4.9 sans sub). KH120s, while certainly excellent speakers, do not have the same level of capability as the 708P w/r/t dynamic reproduction or low bass response. It just doesn't match with reality.

I’m a newbie here, and I am looking for new speakers. While I appreciate having all the Kipple measurements, it is a cumbersome way to go through speakers. The things that concern me are FR, low distortion, sensitivity, price, looks, bass extension. So not having heard either, I would assume I would like the JBLs better. But sound quality is way down on my wife’s list. She would actually prefer listening on her laptop. Good dynamic range seems to be painful to her. I think I, and pretty much everyone here, might have a completely warped idea about what a statistically significant preference might be. The things that I love and are difficult and therefore expensive just might not matter to most people.
for that reason it would be nice to sort by other elements that would allow grouping by purpose; bass F3 or FS, sensitivity and then use the preference score.

 

[TurtlePaul]

It is also important to remember when this research was written. In the 1990s, every audiophile speaker advertised 50 hz - 20 khz +/- 3 dB, yet they all sounded completely different. At least Harman created the spin methodology which for the first time could explain and describe how speakers actually sound.

 

[Randy Bessinger]

In reading this thread, I am curious about the scoring with eq. Has Toole changed his view of eq above the transition frequency? I see that now it seems scores are adjusted based on eq and it appears some of the eq is adjusting not just the bass but other speaker deviations higher in the range?

FWIW, I am not talking about powered speakers with factory adjusted crossovers.

 

[sarumbear]

I’m a newbie here, and I am looking for new speakers. While I appreciate having all the Kipple measurements, it is a cumbersome way to go through speakers. The things that concern me are FR, low distortion, sensitivity, price, looks, bass extension. So not having heard either, I would assume I would like the JBLs better. But sound quality is way down on my wife’s list. She would actually prefer listening on her laptop. Good dynamic range seems to be painful to her. I think I, and pretty much everyone here, might have a completely warped idea about what a statistically significant preference might be. The things that I love and are difficult and therefore expensive just might not matter to most people.
for that reason it would be nice to sort by other elements that would allow grouping by purpose; bass F3 or FS, sensitivity and then use the preference score.

According to the researcher who came up with the preference score trained or untrained doesn’t matter. This is what he says.

http://seanolive.blogspot.com/2008/12/loudspeaker-preferences-of-trained.html

Part 2 - Differences in Performances of Trained Versus Untrained Listeners

Part 1 of this article summarized the results of a controlled listening experiment conducted by the author where 300+ listeners, both trai...

seanolive.blogspot.com

 

[Frgirard]

Hi

I see a lot of "Appeal to Authorities" fallacies in this discussion.
The people who came up with this score are luminaries no doubt and are deserving of praises and respect. Their contributions to the Arts and Science of Music reproduction may well be unmatched.
This metric, the Preference Score, however should be seen as a good starting point, not a definitive score. I don't see other ways to put it: it is inconsistent. Let's not bend backward trying to make of it what it isn't. As it is , it doesn't fully grade speakers. Even its basis are not as clear as many others from the same people .e.g. Blind test and peer reviews. Was it Peer-reviewed BTW? Not sure.

Peace

Peer review? That is the most funny thing I read in this thread.
You post is an great authority argument.

 

[Frgirard]

In reading this thread, I am curious about the scoring with eq. Has Toole changed his view of eq above the transition frequency? I see that now it seems scores are adjusted based on eq and it appears some of the eq is adjusting not just the bass but other speaker deviations higher in the range?

FWIW, I am not talking about powered speakers with factory adjusted crossovers.

Toole is not God but a psycholog.

 

[Randy Bessinger]

Toole is not God but a psycholog.

I didn’t know I said he was god? I have met him and no miracles were performed. And what is a psycholog?

 

[abdo123]

Take the Revel M105 and compare it side by side with the Revel F328Be. You can place a screen in front of the speakers so you can't see each speaker and the audio is so radically different each person can easily identify each speaker in a blind test.

F328Be - 6.5 score posted on ASR, with sub 7.7
M105 - 5.9 score posted on ASR, with sub 8.2

I purchased the M105 based on this tremendous rating. I already had the F328Be. I thought it would be interesting to compare the two. Here what I discovered:
1. The M105 is a fine speaker but low end is lacking and distorts easily if not crossed over at 80Hz with a sub.
2. The M105 doesn't have the same wonderful downward sloping FR as the F328Be.
3. The M105 tweeter doesn't offer the same "realistic" detail as the Be.
4. The F328Be can easily fill a large room while the M105 is best suited for a smaller 12'x12' space.
5. There is absolutely no way the M105 sounds better than the F328Be yet the score with sub indicates it does.
6. No one who has spent time with both speakers next to each other would argue the M105 is within a 0.6 score difference of the F328Be. It's a bold conclusion between a large tower and a dual driver bookshelf that's totally false in actual listening tests. Yet, it's displayed as truth in the speaker scoring system.

Next, I compared the current shipping BMR Monitor to the Revel M105.
The BMR Monitor isn't listed on the ASR Speaker Measurement Index. The BMR Monitor is listed on the https://pierreaubert.github.io/spinorama/index.html site.

BMR Monitor - 5.1 score posted on Pierre Site, with sub 7.3
M105 - 5.8 score posted on Pierre Site, with sub 8.2

When I compare the BMR Monitor to the M105 - this is what I experience in a 12'x13' room (my office).
1. The BMR Monitor has a full strong low end extension that's audibly missing in the M105.
2. The fullness of the 3 way BMR Monitor is much preferred by my ear compared to the M105.
3. The M105 is 13.5" tall while the BMR Monitor is 20". It's basically 50% taller with a larger rear port.

The RAAL 64-10X tweeter, 2.5” Tectonic Balanced Mode Radiator and SB Acoustics 6" Ceramic woofer combination offers a much fuller, engaging sound than the two-way M105 (1" aluminum tweeter/5.25" cone). Yet, the speaker score rates the the M105 a full 0.7 higher than the BMR Monitor. For me, this was another example of why I should assign the Harmon speaker score little value. As far as I am concerned the Speaker score can easily misrepresent which speaker will sound best for my use. Caveat emptor!

Bottom line, when you can't count on the Harmon score to reflect your taste it's best to assign it a lower weight in speaker purchase decisions. If you think it is a very reliable tool for selecting your next speaker you will be fooled.

Thank you for sharing! It was quite entertaining to read.

Do you have any other experiences with speakers with score difference higher than one whole point? because that is when the model becomes more significant in predicting preference.

 

[Bjorn]

Can you give examples? Some supporting measurements perhaps?

This measurement below is from a very big room with great distances to side walls. So in the smaller room with a speaker that collapses in the directivity fairly high in frequency, the response will normally get much worse than this. Still it gives you an idea.

A typical speaker with same weaknesses in directivity as high score Harman speakers:

And vs a speaker that minimizes floor and ceiling reflections:

Or as I mentioned, you can google the in room response of speakers that have scored well and you'll often see that the response is quite uneven. One example below.

Great speaker designs ends up measuring much more even in the actual room. You also have to consider the time domain. While the frequency response follows the time domain, it's still crucial to look at the time domain alone in order to judge the tonality.

 

[Randy Bessinger]

^^That looks like a pretty reflective floor? It also looks like a Don Keele type line array, I have the Parts Express version and the design is for a reflective floor…no? Perhaps, I am way off base!

 

[abdo123]

This measurement below is from a very big room with great distances to side walls. So in the smaller room with a speaker that collapses in the directivity fairly high in frequency, the response will normally get much worse than this. Still it gives you an idea.

A typical speaker with same weaknesses in directivity as high score Harman speakers:
View attachment 190787

And vs a speaker that minimizes floor and ceiling reflections:
View attachment 190788


Or as I mentioned, you can google the in room response of speakers that have scored well and you'll often see that the response is quite uneven. One example below.

View attachment 190790


Great speaker designs ends up measuring much more even in the actual room. You also have to consider the time domain. While the frequency response follows the time domain, it's still crucial to look at the time domain alone in order to judge the tonality.

I thought you were talking about polar map aberrations caused by the vertical distance between drivers and how these specific abberations would show up in in-room measurements, not general abberations simply because the the floor and ceiling are radiated with sound.

Then there is no point in mentioning that controlled vertical directivity specifically is important, but rather that controlled directivity in general is important.

 

[IPunchCholla]

According to the researcher who came up with the preference score trained or untrained doesn’t matter. This is what he says.

http://seanolive.blogspot.com/2008/12/loudspeaker-preferences-of-trained.html

Part 2 - Differences in Performances of Trained Versus Untrained Listeners

Part 1 of this article summarized the results of a controlled listening experiment conducted by the author where 300+ listeners, both trai...

seanolive.blogspot.com

Holy crap. That graph. Take ten seconds to think about how colors can convey meaning. I had to spend a lot of time just figuring out what the graph represented, after reading the explanation. What do you mean highlighted in red, isn’t that the group results for one speaker? Oh you mean under? Is that last group on the right trained?

Anyway. Good info. This would seem to reinforce the idea that the preference score might be a useful bit of info to rank order speakers within a category of speakers that meet my specific criteria. If I could specify bass extension, sensitivity, distortion and then rank the results by preference score, it would likely give me a reasonable ordered shortlist.

 

[Xulonn]

If anyone here on ASR think the score alone decides quality, then the site is failing spectacularly in educating people.

Education only works for those who are willing to learn - and perhaps set their egos aside when evidence (not "proof", which is a term that does not belong in science-based discussions) demonstrates that they were wrong. As with the denial by many - even Ph.D.'s - of the reality of anthropogenic global warming and its likely detrimental effects on human civilization, education doesn't work on those who refuse to accept solid scientific evidence.

Not just a single person, as you imply, but many people here at ASR fail to learn and will not accept basic truths about audio and human audio perception, even when presented with solid evidence that they cannot dispute. It is quite silly to characterize this as a "spectacular failure".

I would like to believe that most intelligent and rational ASR members and visitors would agree with what @Frank Dernie and others have said, indicating these scores are useful for further refining a short list, e.g. eliminating some - but not all - deficient speakers from consideration.

If you don't have the perfect tool for a task, it is necessary to work with the tools that are actually available.

 

[tuga]

Because the majority of the criticism of this thread is basically:

"There is no way the model is accurate because the lower scoring speaker Y has bigger woofers and is more expensive than the higher scoring, smaller, speaker X. people HAVE to perfer speaker Y more than Speaker X even though there is no research that backs my opinion up"

This is no different in my opinion to "Expensive cables have to make a difference" or " I shove a crystal up my **** and it makes me feel much more relaxed through out the day".

In that case I apologise. I have selective reading abilities and have missed those comments.

Edit - maybe those comments where made in the Sointuva topic?