Master Preference Ratings for Loudspeakers

[MZKM]

Thanks. I also found this:

http://seanolive.blogspot.com/2008/12/are-consumer-reports-loudspeaker.html?m=1

Yep, and Sean talks about how other people‘s scoring is poor in the patent I linked to in my explanation.

 

[bobbooo]

Because how much better is a 7 from a 6?

6/$200 is better than 7/$300? Or $100/4 is the same as $50/2?

That's why I suggest specifically calling it 'Performance per $', which is just a definite ratio of two values we're already assuming are 'correct', rather than something more vague and subjective like 'Value for Money Score'. In that way, a speaker scoring 6 and costing $200 is better performance per dollar than one scoring 7 for $300. Whether it's better 'value for money', with all the contextual subjectivity that phrase has, is another matter, and ultimately up to the reader. (You could even call it Score/Price if you think Performance per $ still sounds ambiguous and open to interpretation.)

As for the scale formatting error, are you on mobile? It looks correct for me when using Chrome for Mac, but Chrome for iOS screws it up.

No I'm on PC, both Chrome and Firefox are messed up:

As are the score charts on mobile Chrome (although not with 'desktop site' view enabled funnily enough). Not sure you need the scale anyway, as you have the exact scores in the bars for each speaker.

 

[MZKM]

No I'm on PC, both Chrome

Huh, on Chrome for Mac it's fine, I'll get rid of it.

I'm not against the $/score or score/$, but I think it's way too deceptive looking for most people, as $100 vs $200 is much smaller than a 2 vs 4 score, at least IMO. The scatter plot allows you to easily see which speaker is the cheapest for a given score.

 

[bobbooo]

Huh, on Chrome for MAc it's fine, I'll get rid of it.

I'm not against the $/score or score/$, but I think it's way too deceptive looking for most people, as $100 vs $200 is much smaller than a 2 vs 4 score, at least IMO. The scatter plot allows you to easily see which speaker is the cheapest for a given score.

Ok I'll ask pozz if he can include it in his table then. One thing it would offer over the scatter plot, once enough speakers are measured, is you could easily see whether a particular brand offer consistently good performance for the price, over their whole product range, which a lot of people would consider a good indicator of an honest company selling well-engineered products (conversely, companies that are ripping off consumers would be easily identified).

 

[MZKM]

Ok I'll ask pozz if he can include it in his table then. One thing it would offer over the scatter plot, once enough speakers are measured, is you could easily see whether a particular brand offer consistently good performance for the price, over their whole product range, which a lot of people would consider a good indicator of an honest company selling well-engineered products (and conversely showing which companies are ripping off consumers).

That's interesting, but for sure would take a lot of time for Amir to amass enough data, as right now we just have multiple models from JBL.

 

[sweetchaos]

Consider adding a separate column: "Review" which links to the original ASR review, as it makes jumping between reviews much easier.
Also, this will help any new reddit users who click on your master spreadsheet and who don't understand where this data actually came from.

Also, the link for Kef LS50 goes to ASR , not the google sheets graphs.

 

[MZKM]

Consider adding a separate column: "Review" which links to the original ASR review, as it makes jumping between reviews much easier.
Also, this will help any new reddit users who click on your master spreadsheet and who don't understand where this data actually came from.

Also, the link for Kef LS50 goes to ASR , not the google sheets graphs.

Done and fixed.

 

[LTig]

Just had a look. Great work! Fantastic what this membership is able to create.

 

[bunkbail]

Google Sheets Based Publication

This is a godsent. Thanks a lot @MZKM and @amirm.

 

[beefkabob]

I think the ignore LFX score is misleading. Who crosses over their sub at 300hz? That would only make any sense if you had stereo subs right under your speakers and had crap speakers. I think most people are crossing over around 60-80hz so that the sub noises are all non-directional. My sub sure isn't centered, but I cross over at 50-60hz.

 

[MZKM]

I think the ignore LFX score is misleading. Who crosses over their sub at 300hz? That would only make any sense if you had stereo subs right under your speakers and had crap speakers. I think most people are crossing over around 60-80hz so that the sub noises are all non-directional. My sub sure isn't centered, but I cross over at 50-60hz.

?

The LFX score is based on the -6dB point in the bass that’s referenced to the average SPL of the speaker (from 300Hz-10kHz).

The score for ignoring it assumes using a subwoofer that can get down to ~15Hz (-6dB). Rythmik’s cheapest sub is -6dB @12Hz, so not uncommon.

 

[bobbooo]

That's interesting, but for sure would take a lot of time for Amir to amass enough data, as right now we just have multiple models from JBL.

Just one more thing - could you add a 'Score Calculation' / 'Spreadsheet' column to the 'List' table, that links to the raw data and calculations, like you posted for the first few speakers. It would be great to have all of this easily viewable to everyone in one place for possible error- and cross-checking.

 

[MZKM]

Just one more thing - could you add a 'Score Calculation' / 'Spreadsheet' column to the 'List' table, that links to the raw data and calculations, like you posted for the first few speakers. It would be great to have all of this easily viewable to everyone in one place for possible error- and cross-checking.

I can't double post a spreadsheet (one for the whole thing and one for specific tabs only).
Here is my master file, I can duplicate a spreadsheet for a speaker's calculation upon request, but it'd be a pain to make duplicates and separate publications for every speaker.

 

[MZKM]

Just one more thing - could you add a 'Score Calculation' / 'Spreadsheet' column to the 'List' table, that links to the raw data and calculations, like you posted for the first few speakers. It would be great to have all of this easily viewable to everyone in one place for possible error- and cross-checking.

Here is the Kali's entire spreadsheet:
https://docs.google.com/spreadsheets/d/1idvvv5eB1t8PTNuxfgWn45wx9KV0y3GhdA-HRxcudZw/edit?usp=sharing

 

[napilopez]

Tagging @MZKM , but this is also for general discussion.

As noted in the other Kali thread, I want to advocate for calculating a score that includes the listening window in addition to the on-axis preference ranking. I know you've tested such for the Kali and some other models. I don't mean to clutter the data too much, but I'm concerned about future readers simply looking at the preference chart and trying to determine which speaker is better from a simple numerical value, when I don't believe the exact olive formula is representative of our goal of ranking the best speakers.

While the preference score is a super useful tool it does clearly disadvantage some speakers, as shown with the Kali In-8, which scores a 6.7 with the On-Axis curve vs a 7.4 using the listening window. Such discrepancies will continue to happen for almost every coaxial, but also for any speaker designed to be smoother off-axis such as some designs by Andrew Jones and many modern speakers meant for domestic listening. In fact, I'd say roughly half of the speakers I've measured are smoother some amount between 10-30 degrees off-axis. Maybe more.

An additional column with preference ratings based on LW can help provide important context for a speaker's rating using the standard formula. Alternatively, if we don't want to mess with the score too much and just want to highlight the improvement from using the listening window, we can just include a "Listening Window Bonus" score or something of the sort to highlight how much of a difference there is between the on-axis reference off-axis. This should make it clear the speaker should be listened to some degree off-axis, or that the sound should balance out.

Some food for thought:

• It's worth considering the context of the Olive paper, which had different intentions than ours. The goal was not to find which speakers were the best, it was to correlate measurements with preference. It's a subtle but important distinction; Olive did not make it a priority to optimize a particular speaker's performance, he makes no consideration of how people would actually listen to and position the speakers, nor what the intended horizontal listening axis is. His concern was only how captured data could be correlated with preferences. This is understandable, given his goal was to provide a better preference metric than the popular Consumer Reports one (at the time). On-axis was presumably chosen as the direct sound axis because it is easier to set up a turntable for a variety of speakers this way than finding the optimal reference axis for each speaker.
• More specifically, the listening room procedure is describes in "Part 1" of the test, in which Olive states:
  
  "All tests were performed with a single listener in the room situated in the same seating location, who controlled the switching of the loudspeakers. The listener sat 3 m. away from the loudspeaker, on-axis to the loudspeakers. All loudspeakers were positioned so that the design axis or tweeter was approximately positioned in height to the listeners’ ears. "
  
  So the vertical axis was optimized, but not the horizontal. Think about it; it's actually pretty obvious that on-axis would correlate with preference better than the listening window, because it is a far more accurate representation of the direct sound the listeners heard. At 3 meters, allowing for about a foot of horizontal movement, you'd never be more than about 3 or 4 degrees off-axis. Off-axis listening wasn't even an option for the tested speakers, even if that sound cleared up just a little bit off-axis.
  
  In other words, all the preference rating really tells us is that the first sound to hit our ears correlates more strongly with the preference than a 30-degree horizontal and 10-degree vertical average. This is not surprising.
• However, we know from the Devantier study (which determined the listening window from studying positioning in 15 homes), that the average setup is 10 degrees off-axis. Moreover, more people listen off-axis than on-axis, and nearly as many people listen at 20 degrees off-axis as do at 0 degrees. So any way you cut it, the on-axis measurement simply isn't representative of most real-world use.
  
  
  
• Of course, the Listening Window isn't perfect. It's a wide window and will be smoother than the on-axis simply by virtue of being an average of 9 measurements. However, again, our goals are different than Olive's. He wanted to develop a preference formula. We're trying to figure out which speakers are the best. The Listening window is arguably a fairer representation for a diverse body of speakers positioning a variety of ways, many of which are designed to be listened to off-axis.
  
  You could argue that what we really should be doing is using the best listening axis, but that's a lot of work still leaves coaxials at a bit of a disadvantage. I think simply seeing the difference between the listening window and on-axis based preference scores should be enough to tell you whether you should be listening off-axis.
• Dr. Toole himself makes a similar argument for using the listening window instead of on-axis when it comes to calculating the Directivity Indexes. In his book, when discussing how engineers shifted from calculating the directivity index from the on-axis to the listening window, he says:
  
  "It was decided to depart from this convention because it is often found that, because of symmetry in the layout of transducers on baffles, the on-axis frequency response contains acoustical interference artifacts, due to diffraction, that do not appear in any other measurement. It seems fundamentally wrong to burden the directivity index with irregularities that can have no consequential effects in real listening circumstances."
  
  In that vein, I'd also argue it seems fundamentally wrong to burden the preference score with irregularities that have no consequential effects in real listening circumstances.

So I propose showing both a score with a listening window and one with the on-axis. Otherwise, speakers designed for good on-axis sound will always rank better better, even if that curve isn't actually representative of the first sound to hit our ears.

If we wanted to get real funky, we could also calculate a score with the smoothest NBD axis and include that in reviews for positioning advice, but that's probably asking too much.

 

[MZKM]

Tagging @MZKM , but this is also for general discussion.

As noted in the other Kali thread, I want to advocate for calculating a score that includes the listening window in addition to the on-axis preference ranking. I know you've tested such for the Kali and some other models. I don't mean to clutter the data too much, but I'm concerned about future readers simply looking at the preference chart and trying to determine which speaker is better from a simple numerical value.

While the preference score is a super useful tool, but it does clearly disadvantage some speakers, as shown with the Kali In-8. This will continue to happen for almost every coaxial, but also for any speaker designed to be smoother off-axis such as some designs by Andrew Jones and many speakers meant for domestic listening. In fact, I'd say roughly half of the speakers I've measured are smoother some amount between 10-30 degrees off-axis.

An additional column with preference ratings based on LW can help provide important context for a speaker's rating using the standard formula. Alternatively, if we don't want to mess with the score too much and just want to highlight the improvement from using the listening window, we can just include a "Listening Window Bonus" score or something to highlight how much of a difference there is between the on-axis reference off-axis.

Some food for thought:

• It's worth considering the context of the Olive paper: the goal was not to find which speakers sound the best, as we are doing here on ASR. Rather, it was to correlate measurements with preference. It's a subtle but important distinction; Olive did not make it a priority to optimize a particular speaker's performance, he makes no consideration of how people would actually listen to and position the speakers, nor what the intended listening axis is. His concern was only how captured data could be correlated with preferences. This is understandable, given his goal was to provide a better preference metric than the popular Consumer Reports one (at the time). On-axis was presumably chosen as the direct sound axis because it is easier to set up a turntable for a variety of speakers this way than finding the optimal reference axis for each speaker.
• More specifically, the preference formula paper used the same listening setup as the "part 1" test, in which Olive states:
  
  "All tests were performed with a single listener in the room situated in the same seating location, who controlled the switching of the loudspeakers. The listener sat 3 m. away from the loudspeaker, on-axis to the loudspeakers. All loudspeakers were positioned so that the design axis or tweeter was approximately positioned in height to the listeners’ ears. "
  
  So think about it; it's actually pretty obvious that on-axis would correlate with preference better than the listening window, because it is a far more accurate representation of the direct sound the listeners heard! At 3 meters, allowing for about a foot of horizontal movement, you'd never be more than about 3 or 4 degrees off-axis. Off-axis listening wasn't even an option for the tested speakers.
  
  In other words, all the preference rating really tells us is that the first sound to hit our ears correlates more strongly with the preference than a 30-degree horizontal and 10-degree vertical average. This is not surprising.
• However, we know from the Devantier study (which determined the listening window from studying positioning in 15 homes), that the average setup is 10 degrees off-axis. Moreover, more people listen off-axis than on-axis, and nearly as many people listen at 20 degrees off-axis as do at 0 degrees. So any way you cut it, the on-axis measurement simply isn't representative of most real-world use.
  
  View attachment 48421
• Of course, the Listening Window isn't perfect. It's a wide window and will be smoother than the on-axis simply by virtue of being an average of 9 measurements. However, again, our goals are different than Olive's. He wanted to develop a preference formula. We're trying to figure out which speakers are the best. The Listening window is arguably a fairer representation for a diverse body of speakers positioning a variety of ways, many of which are designed to be listened to off-axis.
  
  You could argue that what we really should be doing is using the best listening axis, but that's a lot of work still leaves coaxials at a bit of a disadvantage. I think simply seeing the difference between the listening window and on-axis based preference scores should be enough to tell you whether you should be listening off-axis.
• Dr. Toole himself makes a similar argument for using the listening window instead of on-axis when it comes to calculating the Directivity Indexes. In his book, when discussing how engineers shifted from calculating the directivity index from the on-axis to the listening window, he says:
  
  "It was decided to depart from this convention because it is often found that, because of symmetry in the layout of transducers on baffles, the on-axis frequency response contains acoustical interference artifacts, due to diffraction, that do not appear in any other measurement. It seems fundamentally wrong to burden the directivity index with irregularities that can have no consequential effects in real listening circumstances."
  
  In that vein, I'd also argue it seems fundamentally wrong to burden the preference score with irregularities that have no consequential effects in real listening circumstances.

So I propose showing both a score with a listening window and one with the on-axis. Otherwise, speakers designed for good on-axis sound will always sound better, even if that curve isn't actually representative of the first sound to hit our ears.

If we wanted to get real funky, we could also calculate a score with the smoothest NBD axis and include that in reviews for positioning advice, but that's probably asking too much.

I already calculate this for all the speakers, so it can be easily added, I'll probably have to have some disclaimer that it is not a proven formula.

10 degree variations are too large in my opinion to calculate which axis is more flat, as I did that as a test and while for most of the band the 10 degree curve was flatter, the upper treble lost too much energy, lowering it's score. This can be eye-balled from the directivity graphs.

 

[napilopez]

I already calculate this for all the speakers, so it can be easily added, I'll probably have to have some disclaimer that it is not a proven formula.

10 degree variations are too large in my opinion to calculate which axis is more flat, as I did that as a test and while for most of the band the 10 degree curve was flatter, the upper treble lost too much energy, lowering it's score. This can be eye-balled from the directivity graphs.

Awesome, I think it's a great idea. A disclaimer would be good. It might be good to also note 86 percent accuracy somewhere in the chart too. Maybe a separate tab with some disclaimers on interpretation?

 

[napilopez]

Oh also, though the listening window is usually better than the on-axis, it is occasionally worse. Such is the case for the Neumann KH80 in my measurements. So it could help to identify whether you should be listening on or off axis at a glance

 

[MZKM]

Oh also, though the listening window is usually better than the on-axis, it is occasionally worse. Such is the case for the Neumann KH80 in my measurements. So it could help to identify whether you should be listening on or off axis at a glance

Everything should be updated now.

 

[sweetchaos]

I would clarify it further:

1.

'On-Axis w/o subwoofer'
or
'On-Axis without subwoofer'

2.

'On-Axis w/ subwoofer'

3.

Good.