Sony SS-CS5 3-way Speaker Review

[QMuse]

The way I see it is that we may be sensitive to small changes in tonal balance which wouldn't be reflected at all in scoring components. Those 2 radar charts are practically identical, but LW and PIR curves aren't.

 

[QMuse]

Final question for @MZKM, @QMuse et al.

It seems also the case that correlation coefficient would be biased in favour of a steeply upward-sloping SM_PIR slope - is this correct?

I don't think so. -30deg slope and +30deg slope should result in the same corr. coefficient, just the sign would be opposite.

 

[andreasmaaan]

I don't think so. -30deg slope and +30deg slope should result in the same corr. coefficient, just the sign would be opposite.

Yes sorry, I meant in comparison with a flat or shallow slope.

So yeh, I think in other words you are confirming what I understood to be the case, i.e. that a speaker with a steeply upward-sloping PIR would score higher (all else equal) than a speaker with a flat or shallowly-downward sloping PIR.

EDIT: actually no, I believe that's false. The opposite sign would negatively affect the score, IIUC.

 

[QMuse]

So yeh, I think in other words you are confirming what I understood to be the case, i.e. that a speaker with a steeply upward-sloping PIR would score higher (all else equal) than a speaker with a flat or shallowly-downward sloping PIR.

Not necessairly, it all depends how correlation coef is used. Upward sloping PIR may as well score worse than the one with flat PIR.

 

[andreasmaaan]

Not necessairly, it all depends how correlation coef is used. Upward sloping PIR may as well score worse than the one with flat PIR.

Yes precisely, see my edit. An upwards slope would result in a negative SM_PIR, which would negatively affect the preference rating.

 

[andreasmaaan]

Ok, so do we think that Olive deliberately built in SM_PIR such that it would advantage speakers with the steepest possible downward-sloping PIR?

If so, it seems hard not to conclude that, on the basis of the data set, listeners tended (all else equal) to prefer speakers with steeply narrowing directivity over speakers with wider/more constant directivity.

 

[QMuse]

Ok, so do we think that Olive deliberately built in SM_PIR such that it would advantage speakers with the steepest possible downward-sloping PIR?

If so, it seems hard not to conclude that, on the basis of the data set, listeners tended to prefer speakers with steeply narrowing directivity over speakers with wider/more constant directivity.

This simply seems wrong to me, as after you reach certain negative slope with PIR that is perceived optimal making the slope further negative should sound less optimal IMHO.

 

[andreasmaaan]

This simply seems wrong to me, as after you reach certain negative slope with PIR that is perceived optimal making the slope further negative should sound less optimal IMHO.

It seems wrong to me too, yet a difficult conclusion to escape.

 

[QMuse]

The other thing I'm struggling to understand is why he used on-axis and not LW.

 

[BenB]

Would you be able to do it with 70 speakers and only 4 of these variables though?

Nope.

Here's an excerpt from section 2.1 of the paper:

And, later in section 4 (this is about Test One though):

So it does look like Olive was well-aware of the over-fitting problem. I do agree that the model being evaluated against the same dataset it was trained with is worrying, though.

We know the model is far from perfect. But it's the best we've got right now, and it's still miles ahead of anything that has ever been attempted with regard to producing reliable objective ratings of loudspeakers.

There are discussions in the paper about overfitting, and statistical techniques are applied that are supposed to prevent or account for it. I'm not a statistician, though I do deal with some statistical methods. Here's a brief summary of how these have been utilized, as far as I can tell (I got the paper last night, so I could be mistaken. I hope others will help characterize this summary):

A model was developed on 13 loudspeakers that when applied to those 13 loudspeakers could match the preferences of the listeners with a correlation of 0.995. An adjusted correlation estimate was made in an attempt to account for any overfitting to the limited sample size, which resulted in a drop to 0.96. That estimate assumed that the 13 speaker were representative of any loudspeaker to which the algorithm would be applied. When the algorithm developed on 13 speakers was applied to 70 speakers, the actual result was only 0.7. That is much lower than the adjusted correlation estimate.

The authors offer these seemingly contradictory insights:

The Mallow’s CP value is 4 indicating that the model is not too over-fitted for the number of variables used

The lower correlation was likely related to the model being too tightly fitted to the small sample (13 loudspeakers) and/or the loss of precision from combining subjective data from 18 unrelated tests.

An additional explanation of my own:
The 13 loudspeakers in sample 1 were not representative of the 70 loudspeakers in sample 2.

A second algorithm was developed on the 70 speaker set. This algorithm has an uncorrected correlation with listener preferences of 0.86. I don't see where a corrected number has been supplied. It's possible the correction is simply very small, which could be appropriate if 4 variables are used with 70 samples.

There is some discussion about the level of control realized in the 70 speaker sample. The authors imply that they can't achieve better than 0.86 correlation because there is effectively too much noise or differing biases from subset to subset in the accumulated preference information.

The authors identify 4 limitations of the model

1) Up to this point, the model has been tested in one listening room.
2) The model doesn’t include variables that account for nonlinear distortion (and to a lesser extent, perceived spatial attributes).
3) The model is limited to the specific types of loudspeakers in our sample of 70.
4) The model’s accuracy is limited by the accuracy of the subjective measurements.

As an amateur speaker designer, I think limitation 3 is the most important. The algorithm and any insights implied cannot necessarily be expected to apply to any variation in design that wasn't well represented (with multiple models) in the sample. Since I almost exclusively undertake projects that are different from what's typically done in the industry, it makes it more difficult for me to predict what will optimize the listener preferences.

Additionally, I would make the wording of limitation 2 more broad. The model simply doesn't incorporate variables for many things that probably impact preference. It's dangerous to increase the number of variables when starved for samples, so it's a necessary limitation based on sample size.

 

[QMuse]

It seems wrong to me too, yet a difficult conclusion to escape.

Maybe @amirm can help us test it.

I created another filter which increases the slope of the PIR and I have attached a filter so if you have time for another listening test we would greatly appreciate.

PIR - original in red, initial correction in blue, increased slope correction in green. Filter is attached (.txt -> .wav). Plz also apply your room EQ 100Hz filter as you did with 1st test.

 

[napilopez]

How many buyers of this inexpensive speaker have the inclination, ability and means to EQ this speaker?

How many audiophiles would bother?

Just askin'. View attachment 65320

Probably not many, but seems like a perfectly reasonable thing to do for a secondary speaker system on the cheap. Since the issues can also be dealt with broad Q filters, a decent AV room correction system might be able to give you a good result. Just a thought.

 

[Gatordaddy]

How many buyers of this inexpensive speaker have the inclination, ability and means to EQ this speaker?

How many audiophiles would bother?

Just askin'. View attachment 65320

Considering that young people use computers for sources, have no budget but like to tinker, and just how enormous the headphone tweaker crowd is I think there'd be a good amount of interest.

 

[napilopez]

The other thing I'm struggling to understand is why he used on-axis and not LW.

Because on-axis had higher correlation with the sampled preference data. As I've mentioned in other threads, this is the expected result, as the on-axis is likely more representative of the direct sound, because every speaker was listened to directly on-axis from a single position. This is not representative of real world use, however, where we know most people listen off-axis, where people tend to move more, and where some speakers are meant to be heard off axis.

But the paper was not interested in optimizing the tested speaker's performance, finding out which specific speakers are the best, or what real world use is like. It was interested in correlating listening impressions to measured frequency responses.

At best, it maybe just tells us that for the speakers tested, a 60 degree wide and 20 degree tall listening window is not as representative to preference to a speaker listened on axis from a single chair as the actual on-axis curve is. That's not a surprise. I'd be willing to bet good money a smaller listening window, maybe +/- 10 degrees, would've fared better, but we don't have the raw data to know.

For our purposes of trying to figure out which speakers are better than others, imo the listening window is still more useful. Harman engineers themselves seem to prefer optimizing for the listening window, as can be seen in so many spinoramas. Kevin Voecks of revel says in this thread: "As our research has long indicated, the listening window is a far better indicator of direct sound quality than is any on-axis curve. "

 

[QMuse]

Because on-axis had higher correlation with the sampled preference data. As I've mentioned in other threads, this is the expected result, as the on-axis is likely more representative of the direct sound, because every speaker was listened to directly on-axis from a single position. This is not representative of real world use, however, where we know most people listen off-axis, where people tend to move more, and where some speakers are meant to be heard off axis.

But the paper was not interested in optimizing the tested speaker's performance, finding out which specific speakers are the best, or what real world use is like. It was interested in correlating listening impressions to measured frequency responses.

At best, it maybe just tells us that for the speakers tested, a 60 degree wide and 20 degree tall listening window is not as representative to preference to a speaker listened on axis from a single chair as the actual on-axis curve is. That's not a surprise. I'd be willing to bet good money a smaller listening window, maybe +/- 10 degrees, would've fared better, but we don't have the raw data to know.

For our purposes of trying to figure out which speakers are better than others, imo the listening window is still more useful. Harman engineers themselves seem to prefer optimizing for the listening window, as can be seen in so many spinoramas. Kevin Voecks of revel says in this thread: "As our research has long indicated, the listening window is a far better indicator of direct sound quality than is any on-axis curve. "

Exactly, on-axis has nothing to do with real world. One can argue that on-axis, as a single point measurement is hardly a better representative of direct sound than LW. I remain sceptical, especially considering such high weighting.

 

[MZKM]

Because on-axis had higher correlation with the sampled preference data. As I've mentioned in other threads, this is the expected result, as the on-axis is likely more representative of the direct sound, because every speaker was listened to directly on-axis from a single position. This is not representative of real world use, however, where we know most people listen off-axis, where people tend to move more, and where some speakers are meant to be heard off axis.

But the paper was not interested in optimizing the tested speaker's performance, finding out which specific speakers are the best, or what real world use is like. It was interested in correlating listening impressions to measured frequency responses.

At best, it maybe just tells us that for the speakers tested, a 60 degree wide and 20 degree tall listening window is not as representative to preference to a speaker listened on axis from a single chair as the actual on-axis curve is. That's not a surprise. I'd be willing to bet good money a smaller listening window, maybe +/- 10 degrees, would've fared better, but we don't have the raw data to know.

For our purposes of trying to figure out which speakers are better than others, imo the listening window is still more useful. Harman engineers themselves seem to prefer optimizing for the listening window, as can be seen in so many spinoramas. Kevin Voecks of revel says in this thread: "As our research has long indicated, the listening window is a far better indicator of direct sound quality than is any on-axis curve. "

Well, it‘s also my understanding the waveguides inherently cause issues on-axis that are not seen even a few degrees off-axis. Since Harman uses waveguides, it would benefit them to focus on the listening window.

 

[QMuse]

@MZKM Can you plz calculate rating for this filter for PIR with increased slope?

 

[MZKM]

Maybe @amirm can help us test it.

I created another filter which increases the slope of the PIR and I have attached a filter so if you have time for another listening test we would greatly appreciate.

PIR - original in red, initial correction in blue, increased slope correction in green. Filter is attached (.txt -> .wav). Plz also apply your room EQ 100Hz filter as you did with 1st test.

View attachment 65384

This won’t prove much. You need to compare speakers that are near identical in every aspect except the directivity. Increasing the slope via EQ will bring down the on-axis and listening window so they are no longer neutral.

 

[QMuse]

This won’t prove much. You need to compare speakers that are near identical in every aspect except the directivity. Increasing the slope via EQ will bring down the on-axis and listening window so they are no longer neutral.

My idea was to compare this non-optimal EQ with the scoring for it.

 

[MZKM]

@MZKM Can you plz calculate rating for this filter for PIR with increased slope?

I’m about to be out of the house for a while, so it’ll have to wait (I foolishly deleted the rows to match the octave smoothing rather than hide them, so I manually will have to do the adjustment again).