Preference rating: What score speaker do you use? What can you tolerate? Where do you start to deviate? Where can you hear until?

[Sancus]

But if you were starting from scratch in identifying a pool of speakers to evaluate, the Preference Rating could narrow the choices

This is the actual real benefit of the score, without it the review index is long and unsortable.

People trying to compare 4.5s to 5.2s purely via score have missed the point.

It's always surprising to me how much people fixate. Like, are people worried about the validity of whathifi's 5 star ratings or rtings arbitrary scoring systems?? The score is flawed but it's also the only scoring system I've ever seen on a consumer review site that has ANY research paper backing it.

 

[MZKM]

Is @MZKM calculating the preference ratings for each reviewed speaker from Amir's published measurement data?

Others do too, but yes I do.

 

[kimmosto]

7.5 without sub, 8.9 with sub. Speaker is quite small conventional 2-way with passive radiators. Calculated with VituixCAD during XO simulation so result is not entirely compatible with NFS measurement.

Rating would be ca. 7.7 w/o sub if crossover is optimized by preference rating (equation 9), but that sounds worse because equation 9 is too simplified containing assumptions about listening distance and room acoustics. Few statistical problems with PR equation 9:
1) Does not include SM_ON or SM_LW. On-axis and listening window are included in SM_PIR with quite small weight. This allows significant error in on-axis and listening window responses. Equation 9 contains kinda assumption that listening distance is not short and room acoustics is not damped. "Error" in PR result is increased if DI response is not straight.
2) Does not include SL_ON or SL_LW. On-axis tilt -> sound balance could be total crap while PR looks perfect. This is severe problem in equation 9 preventing optimising crossover or EQ with PR / patent application.
3) Does not detect unbalanced HF/LF directivity: SL_PIR, SL_ER or SL_SP is not included.

Other fundamental problems in PR calculation / patent application:
4) Response slope is not eliminated from SM calculation. Slope other than 0 dB/oct. causes worse SM result so e.g. SM_PIR and SM_SP have variable bias -> SM_PIR and SM_SP are not valid for optimizing speaker.
5) LFX is calculated from sound power (SP) so it's not ready for directive bass applications.
6) LFQ is calculated from sound power (SP) so it's not ready for directive bass applications.

Other problems with preference rating in general:
7) Does not detect dynamic and timbre problems due to bad timing.
8) Does not detect dynamic and sound balance problems due to compression.
9) Does not detect sound stage problems due to diffraction and other multi-source effects.
10) Does not detect non-linearities and intermodulations.
11) Does not detect resonances outside NBD range.
12) Does not fully understand other than small...medium conventional full space applications.

Ranking speakers with PR eq 9 is not very wise.

 

[sdiver68]

7.5 without sub, 8.9 with sub. Speaker is quite small conventional 2-way with passive radiators. Calculated with VituixCAD during XO simulation so result is not entirely compatible with NFS measurement.

Rating would be ca. 7.7 w/o sub if crossover is optimized by preference rating (equation 9), but that sounds worse because equation 9 is too simplified containing assumptions about listening distance and room acoustics. Few statistical problems with PR equation 9:
1) Does not include SM_ON or SM_LW. On-axis and listening window are included in SM_PIR with quite small weight. This allows significant error in on-axis and listening window responses. Equation 9 contains kinda assumption that listening distance is not short and room acoustics is not damped. "Error" in PR result is increased if DI response is not straight.
2) Does not include SL_ON or SL_LW. On-axis tilt -> sound balance could be total crap while PR looks perfect. This is severe problem in equation 9 preventing optimising crossover or EQ with PR / patent application.
3) Does not detect unbalanced HF/LF directivity: SL_PIR, SL_ER or SL_SP is not included.

Other fundamental problems in PR calculation / patent application:
4) Response slope is not eliminated from SM calculation. Slope other than 0 dB/oct. causes worse SM result so e.g. SM_PIR and SM_SP have variable bias -> SM_PIR and SM_SP are not valid for optimizing speaker.
5) LFX is calculated from sound power (SP) so it's not ready for directive bass applications.
6) LFQ is calculated from sound power (SP) so it's not ready for directive bass applications.

Other problems with preference rating in general:
7) Does not detect dynamic and timbre problems due to bad timing.
8) Does not detect dynamic and sound balance problems due to compression.
9) Does not detect sound stage problems due to diffraction and other multi-source effects.
10) Does not detect non-linearities and intermodulations.
11) Does not detect resonances outside NBD range.
12) Does not fully understand other than small...medium conventional full space applications.

Ranking speakers with PR eq 9 is not very wise.

CAD calculated.

Please send it in for testing, would love to see how this correlates with real testing.

 

[kimmosto]

CAD calculated.

Please send it in for testing, would love to see how this correlates with real testing.

Several years old freeware for DIY with thousands of active users so no need to send. Just download from https://kimmosaunisto.net/. XO simulation uses measurement data so it will be very close to exact match with "measured reality".

Note: No user support for first time testers and users who don't use measurement data for XO simulation so read and follow measurement instructions and user manual carefully, and play with sample projects before asking any questions from anybody/anywhere. Limited/occasional support for freeware users.

 

[BoredErica]

Right --- but the naive reader doesn't know that.

If all the score means is that two speakers with similar scores have a similar level of aggregate "quality", but sound very different, is that useful?

That means I need to look at other data and/or listen to get the full picture -- and why I pay little attention to it. It doesn't tell me anything about how the speaker sounds.



At which point, other than weeding out designs that are bad, what's the use of the score for a consumer?

Like minute differences in SINAD, I think preference score gets more attention than it deserves on ASR.

In practice when people don't use preference score they start giving their opinions on how they sound in their own room or showroom sighted without EQ or sub. It's so far off from how it should be done to the point where those impressions are almost totally useless. I don't see people compare speaker A vs B and start busting out spinorama graphs often. I'd rather select from a variety of speakers above a preference rating score based on looks than listen to what people say about how it sounds otherwise I'd be buying cable risers before I know it.

If we're actually comparing looking at different measurements vs the preference score, then yeah maybe. Even then if it's so simple to tell what is better, surely we could improve the preference rating ourselves? But people don't do it because they're not really sure if it results in a more accurate formula because it's not tested.

Preference score is very limited, but can I really be that sure my interpretation of spinorama results are more accurate? If I have the advantage over the preference rating due to knowing my preferences well, how do I know those are my preferences without controlled testing? And how do I know how much to compromise from other aspects of performance for whatever metric I think I care about more? It seems to me there is just too much room to make mistakes in interpreting spins or falling for my own biases to just toss out the preference score.

Basically I fear by relying on my own judgement of spins too much I am opening a pandora's box by introducing subjective evaluations from a very error-prone and biased human being (myself) who is also knowledge limited. And yeah, we can say I should just study more. But audio is just a lot more complicated than people inside the hobby give it for. I've been in ASR for weeks and weeks and weeks trying to learn everything and watching Amir's and Erin's tutorials on spins and I'm going to die if I cram more information into my brain in a short period of time.

 

[PH73]

Rating would be ca. 7.7 w/o sub if crossover is optimized by preference rating (equation 9), but that sounds worse . . .

Sounds like a case of Goodhart's law.

 

[sdiver68]

Several years old freeware for DIY with thousands of active users so no need to send. Just download from https://kimmosaunisto.net/. XO simulation uses measurement data so it will be very close to exact match with "measured reality".

Note: No user support for first time testers and users who don't use measurement data for XO simulation so read and follow measurement instructions and user manual carefully, and play with sample projects before asking any questions from anybody/anywhere. Limited/occasional support for freeware users.

Not the software, the speaker. Id be interested in how closely the sim matches the results from a variety of perspectives.

 

[kimmosto]

Not the software, the speaker. Id be interested in how closely the sim matches the results from a variety of perspectives.

Accuracy of electrical simulation is about 0.001 dB so accuracy of response data defines accuracy of simulated acoustical output.
What is "variety of perspectives"?

 

[kimmosto]

Sounds like a case of Goodhart's law.

Maybe... or PR (eq 9) is just too much simplified based on too static acoustical conditions and setup while listening tests for PR development. For example sound balance in PR (eq 9) can be simplified to:
1) Predicted in-room response (PIR) alone defines sound balance (weighting factors of PIR: LW=12%, ER=44% and SP=44%).
2) SM_PIR=1.0 i.e. straight PIR is the best/preferred balance.
Quite easy to say that it's not entire truth according "everything" we know about sound reproduction with speakers.

Simplification may be acceptable for some casual purposes and evaluation of bad...crap speakers with PR<<7, but speaker won't necessarily be very versatile without "certain" directivity index response. On-axis and listening window should be included with bigger weight or root responses (LW, ER, SP) should have tolerance window while optimizing. Of course SL_LW or SL_ON is mandatory, but that is somehow forgotten from PR (eq 9).

 

[sdiver68]

Accuracy of electrical simulation is about 0.001 dB so accuracy of response data defines accuracy of simulated acoustical output.
What is "variety of perspectives"?

Accuracy of software vs accuracy of spinorama vs quality of build.

If they are all accurate to spec should see little deviation.

 

[SpanishPassion]

If all the score means is that two speakers with similar scores have a similar level of aggregate "quality", but sound very different, is that useful?

Sorry for the late reply... I think that by itself, the preference rating is not very useful, but this is not the only information that we get here on ASR or other competent resources.

Combined with spinorama itself and also with the subjective impressions, you get a much better picture. Personally I like to see the preference score and I'm always looking forward to it. It adds some additional information about the quality of the speakers from the engineering point of view. A summary of its characteristics, to speak so.

At which point, other than weeding out designs that are bad, what's the use of the score for a consumer?

I think another use is just the reality check. Sometimes you can see people complaining about some perceived flaws in the spinorama or how some speaker is horrible, but then the preference score is actually the same or similar to preference scores of speakers that we objectively know that are state of art and well engineered. When this happens you know that what they say is not really based on any objective data, but just on personal bias.

 

[OldUI11]

I'm curious about what the preference score represents. Is it the value of a function that takes numerical representations of various properties of a speaker as its arguments? That number, or rather the scale it is on, represents a preference ordering, at least for most listeners. Is that right? What are the properties of the scale? Is it an ordinal scale, or is it a cardinal scale of some type? I assume it's the former, but it occurred to me that I hadn't seen a description of the properties of the scale.

 

[MattHooper]

I don't think I've ever looked at the preference scores.

I do enjoy peeking at speaker measurements, both here and at Stereophile. It scratches a sort of intellectual interest in that regard.

But, with some exceptions, I generally don't purchase speakers based on preference scores/measurements. Though haven't been in the market for a number of years now for speakers.

Generally speaking, in terms of my buying decisions it's always through auditioning speakers directly, and it can include a broad range of speakers that simply intrigue me. I'm not using measurements a lot to narrow things down (though do to some degree) because I really just enjoy hearing different speaker designs. It can be enlightening or entertaining to me to listen to a speaker that even has wonky measurements.
And I can't precisely predict my own reaction based on measurements (my own failing to a degree, no doubt).

So, I still go out and have fun listening to various speakers and choose the one I liked best. (Or, I also will purchase speakers I'm interested in second hand to try out and sell if they don't fully satisfy. I've never regretted trying out speakers that way too).

 

[ayane]

Answering the OP, 5.5 6.5 is my threshold for tolerating without subs, and basically everything with a lower score just falls apart.

I don't heed the score too much, as it dumbs down the comprehensive measurement data too much, but it is fairly accurate in terms of boiling down the most important factors. Directivity is the most important factor for me, and I weight that much higher than a flat response. Bass extension is also important, but more specifically, how far do the speakers extend at a typical listening level before choking up with distortion? >160 Hz is unacceptable, and par is 80 Hz (two octaves above threshold of 20 Hz. The fewer octaves the sub has to cover, the better).

A speaker with ruler straight directivity - especially a listening window that is almost identical to the on-axis - that extends to at least 80 Hz with distortion below 1% at 90 dB SPL from 2 meters away is a good speaker. High standard, but that's what I'm familiar with - and Genelecs deliver all of that.

I'm really familiar with Genelecs from production - 8341A to be specific - and after learning to monitor (and listen for flaws in reproduction side too) as well as critical listening tests with pink noise, it's hard to stand worse speakers. It's a large part of the reason why I stuck with headphones, because it's too expensive to justify crappy speakers at home when a $1k headphone rig can run circles around stuff 5x as expensive.

Currently using a pair of modded Swans HiVi DIY3.1-A and they're decent. However, after having them for a few weeks, their flaws are becoming harder and harder to tolerate, and I'm probably going to build a pair of ASR Directiva R1s in the very near future. I guess you truly do get what you pay for, haha.

Maybe my threshold is higher than 5.5 (without sub), but I think that's the lowest I can go before my standards prevent enjoyment. It's hard to stand most speakers when you get familiar with Genelecs.

Edit: The Directiva R1 is now my benchmark for all other speakers - if it's worse than the Directiva in terms of directivity, flatness, or distortion, I don't like it. Same goes for looks too!

 

[Descartes]

To weed out brands and speakers, look at Amid & Erin’s reviews
Then my ears and they need to look the part!
Finally the value proposition has to match the price.