Random musings on The Preference Ratings of speaker reviews

[tuga]

What's the concern about the use of untrained listeners? If the intent is to develop a model that is appealing to consumers, why would it be preferable to use trained listeners?

Use of untrained listeners is fine for tasting polls, but not for determining accuracy.

 

[Inner Space]

Frequency response and directivity seem to go a long way to characterize speaker performance.

It seems on axis frequency response will characterize a speaker's sound quite well unless some off axis problems corrupt it.

Of course beyond that you need to know about loudness capabilities for the space you intend to use it in. And distortion should be low enough ...

I would add an assessment of cabinet talk, via accelerometer or whatever. Theoretically one could imagine a speaker with native dips to some extent filled in by cabinet resonances, such that it measured OK, but sounded annoying.

 

[weasels]

Use of untrained listeners is fine for stasting polls, but not for determining accuracy.

Thanks - that makes sense.

 

[bobbooo]

Use of untrained listeners is fine for tasting polls, but not for determining accuracy.

18 of the 19 studies the preference formula is based on exclusively used trained listeners only, and the one remaining study featuring untrained listeners still had a minority of trained listeners.

 

[KaiserSoze]

I haven't paid much attention to the preference ratings because, being the natural-born skeptic that I am, my immediate reaction was to scoff at the idea. Various people here have identified a list of reasons why it is folly to try to reduce the quality of loudspeakers to a single metric. Re-defining the singular metric or inventing a new one isn't going to help. The idea just isn't a sound idea.

The "preference score with subwoofer" seems especially iffy. For me personally, it is a struggle to get my head around the idea that a "preference score with subwoofer" would be about anything other than the ability of a speaker to integrate properly with a subwoofer. And a speaker that does especially well in this respect is simply a speaker with especially good deep bass extension. The "preference score with subwoofer" seems to be a sort of "what the speaker's rating would be if you place no importance whatsoever on the need for the speaker to integrate properly with a subwoofer". The incongruity is manifest, and it seems to be trying to tell us that when you try to do something that doesn't inherently make good sense, that you'll most likely end up with some sort of logical absurdity.

 

[HooStat]

The idea just isn't a sound idea.

I think your skepticism is reasonable. However, just to be clear, the idea is solid. The underlying concept of building a model of the process and using it to improve products is used in many industries. The problem is in how it is used and interpreted. But it is no more or less useful than a reviewer's opinion about a speaker, or a rating from a panel of experts.

EDIT: It seems that the post I was responding to was deleted. Hence this seems out of context.

 

[KaiserSoze]

I think your skepticism is reasonable. However, just to be clear, the idea is solid. The underlying concept of building a model of the process and using it to improve products is used in many industries. The problem is in how it is used and interpreted. But it is no more or less useful than a reviewer's opinion about a speaker, or a rating from a panel of experts.

Just to be clear, the concept of building a model and using it to improve products is not in question. However this does not defend the practice of reducing an extremely complex n-dimensional variable - where the number n is not even readily identifiable - to a simple scalar number. Numerous reasons why this doesn't work have been identified here by various people. One such reason is the reason I gave, i.e., the inherent logical absurdity of the "preference score with subwoofer", which is logically absurd for the simple reason that it disregards the need for the speaker to properly integrate with a subwoofer.

 

[HooStat]

Just to be clear, the concept of building a model and using it to improve products is not in question. However this does not defend the practice of reducing an extremely complex n-dimensional variable - where the number n is not even readily identifiable - to a simple scalar number. Numerous reasons why this doesn't work have been identified here by various people. One such reason is the reason I gave, i.e., the inherent logical absurdity of the "preference score with subwoofer", which is logically absurd for the simple reason that it disregards the need for the speaker to properly integrate with a subwoofer.

This is done all the time in statistics. And it works quite well.

 

[amirm]

Before we focus on the formula, a spin describes the dispersion characteristics of a given speaker, its tonal balance on- and off-axis.
Is directivy enough to characterise speaker performance?
Is frequency response enough to characterise speaker performance?

Its effect is huge! I know because when I equalize to what the target should be, the improvements are dramatic and positive almost all of the time.

I usually start with the spin data but other times I use PIR. Haven't decided which one should be the target but having both is invaluable in evaluating speaker performance and importantly, correcting its imperfections.

 

[Blumlein 88]

The only time performance <95Hz is considered is for the LFX, which gets fixed in place for the w/ sub score.

NBD uses 1/2 octave bands starting at 100Hz, and I am using 14, one could delete the bottom 3 bands (~ 95Hz-270). Smoothness starts at 100Hz, but one could also delete the lower end of responses.

However, usually there aren’t many issues <300Hz, most of the issues are >500Hz dealing with baffle step, crossover implementation, directivity matching, etc.

Sorry for my poor reading comprehension. When I looked back at the formula it says right there use between 100hz and 12 khz for one part and 100 hz and 16 khz for another part.

Also noticed looking in Toole's book about Olive's work on this, something I had forgotten. That subjectively judged response below 250 hz typically is several decibels higher than the anechoic curves indicate. Usually 10 db higher at 50 hz and 6 db higher at 35 hz. Something which would seem to explain exactly the now much discussed lack of low end as measured on the Magnepan LRS vs what people hear. People hear it as lightweight and in need of a sub, but not nearly so lightweight as the Klippel measurements show. And this is perfectly expected from the prior work. It just looks so glaring because the LRS has that peak around 400hz and rolls from that point on down.

 

[bobbooo]

Just to be clear, the concept of building a model and using it to improve products is not in question. However this does not defend the practice of reducing an extremely complex n-dimensional variable - where the number n is not even readily identifiable - to a simple scalar number.

What do you think you have to do internally when you make a purchase choice, say whether to buy speaker A with attributes x,y,z (and others unknown) or speaker B with attributes i,j,k etc? You subconsciously reduce the known 'n-dimensional variables' x,y,z and i,j,k to a one-dimemsional variable for each speaker that can be rated on the same internal scale. The one with the resultant higher overall internal rating becomes your purchase choice and so 'wins'. Oh wait, that's exactly what the preference formula does, except its domain is restricted purely to sound quality attributes, and it eliminates cognitive biases and other nuisance variables by being based on scientifically controlled, double-blind listening tests.

 

[bobbooo]

Also noticed looking in Toole's book about Olive's work on this, something I had forgotten. That subjectively judged response below 250 hz typically is several decibels higher than the anechoic curves indicate. Usually 10 db higher at 50 hz and 6 db higher at 35 hz.

Chapter, section, page? Are you just talking about the difference between anechoic and in-room measurements (i.e. mostly room gain in the bass)?

 

[Blumlein 88]

Chapter, section, page? Are you just talking about the difference between anechoic and in-room measurements (i.e. mostly room gain in the bass)?

It is in chapter 20.2 under Olive's experiments. I'm using the earlier 2008 edition. It is discussed in regard to some graphs in figure 20.4 in the earlier edition. It shows the FR as judged/perceived by trained listeners vs the anechoicly measured on axis and early reflections window. Until 250 hz there is a very close match with measurements and it goes off as you go lower in frequency. The response as perceived by listeners shows higher bass response vs measurements. It would have been nice to have the in room measurements and see if that was just room gain (which is the most obvious conclusion) or something else. But those aren't in the text.

 

[MZKM]

Sorry for my poor reading comprehension. When I looked back at the formula it says right there use between 100hz and 12 khz for one part and 100 hz and 16 khz for another part.

Also noticed looking in Toole's book about Olive's work on this, something I had forgotten. That subjectively judged response below 250 hz typically is several decibels higher than the anechoic curves indicate. Usually 10 db higher at 50 hz and 6 db higher at 35 hz. Something which would seem to explain exactly the now much discussed lack of low end as measured on the Magnepan LRS vs what people hear. People hear it as lightweight and in need of a sub, but not nearly so lightweight as the Klippel measurements show. And this is perfectly expected from the prior work. It just looks so glaring because the LRS has that peak around 400hz and rolls from that point on down.

The PIR is supposed to help this issue. 2 issues though:

1) It is tilted; thus I normalize it to its own slope (I used to do it to the target slope, not anymore). I then do a weighted average with the on-axis (weights per the formula). This is my Tonal Balance tab in my files.
2) It is going to still under-estimate the bass, as it doesn’t account for boundary gain. This effects all speakers though, and of course the gain is different per room and placement. But yes, when it is very lean in the bass, the data makes it look even moreso.

EDIT: The LRS is very directive in the bass, and it has the cancellations at +/-90°, thus the bass it puts out into the room has to be less than other speakers, even if the on-axis was flat, but since that is also down, no favors are being given. But, since the panel sits close to the floor, there will also be some reinforcement there (not much more compared to a 3-way pair of towers, but more than a 2-way bookshelf on stands).

 

[stevenswall]

This seems impressive to me -- for the Genelec. It's a tiny thing, after all, and not even close to the top of the line in the range.

One thing to note here is that even though the 8341 isn't the top of the range with Genelec, it more or less sounds identical to the rest of the 8000 series. The only things you really get with the 8351 or 8260 are a larger waveguide for more controlled directivity at slightly lower frequencies, and a bit more bass extension and spl. Other than that they are all incredibly flat and seem to have the same tonality. Heck, even the S360 sounds very similar to me, and a 1031a was also quite uncolored (though there was some port chuffing and little bass.)