Evidence-based Speaker Designs

[noobie1]

So if they have theories, they need to prove them with evidence, while if you have theories, they need to disprove them with evidence? Hmm..

Now I agree with you that Olive and Toole’s research is not gospel (obviously) and is not going to be the final word, but I don’t see your criticisms as consistent here.

No no. I am not saying they need to disprove my specific theory. I am only saying if I have a credible counter theory with my own set of evidence (experimental or even thought experiment) then generally those espousing the theory may want to consider designing experiments to disprove my theory. This comes up in academic settings not in internet forums.

 

[andreasmaaan]

No no. I am not saying they need to disprove my specific theory. I am only saying if I have a credible counter theory with my own set of evidence (experimental or even thought experiment) then generally those espousing the theory may want to consider designing experiments to disprove my theory. This comes up in academic settings not in internet forums.

Ok, well what’s your credible counter theory? What evidence or theory is it based on?

Let me add, I’m genuinely interested in having a constructive discussion on this topic so I’m not waiting behind my keyboard here to rip apart whatever it is you put forward

 

[noobie1]

Ok, well what’s your credible counter theory? What evidence or theory is it based on?

I don't disagree with O'Toole's design principles. I think they are fine. But people are taking it here as the gospel without independent verification by another lab. They undoubtedly have some flaws in their approach as do all research groups.

I was a PhD student many years ago when one of my classmates published an article in a scientific journal announcing the discovery of a new chemical element. Only a few years later did people discover there was academic fraud and dishonesty. The paper was retracted. I'm not suggesting O'Toole of such things. Generally, scientists don't accept scientific conclusions as definitive and established if only one lab has performed the tests. It must be repeated independently in order to become established.

 

[March Audio]

Ok, well what’s your credible counter theory? What evidence or theory is it based on?

Let me add, I’m genuinely interested in having a constructive discussion on this topic so I’m not waiting behind my keyboard here to rip apart whatever it is you put forward

Me too. I want to hear a different view. However there doesn't appear to be one.

 

[March Audio]

I don't disagree with O'Toole's design principles. I think they are fine. But people are taking it here as the gospel without independent verification by another lab. They undoubtedly have some flaws in their approach as do all research groups.

I was a PhD student many years ago when one of my classmates published an article in a scientific journal announcing the discovery of a new chemical element. Only a few years later did people discover there was academic fraud and dishonesty. The paper was retracted. I'm not suggesting O'Toole of such things. Generally, scientists don't accept scientific conclusions as definitive and established if only one lab has performed the tests. It must be repeated independently in order to become established.

No one is taking it as gospel, however there are no obvious reasons that I see to disagree with it or contradict it.

Not only me, but seemingly yourself, and the general professional audio community.

 

[noobie1]

No one is taking it as gospel, however there are no obvious reasons that I see to disagree with it or contradict it.

Not only me, but seemingly yourself, and the general professional audio community.

This whole thread is based on the notion that if you don't follow O'Toole principle, you somehow don't use evidence to design speakers. That implies that O'Toole is the only Truth that matters.

 

[RayDunzl]

O'Toole?

 

[March Audio]

This whole thread is based on the notion that if you don't follow O'Toole principle, you somehow don't use evidence to design speakers. That implies that O'Toole is the only Truth that matters.

Is it?

So what's your alternative? Design based on personal unproven pet theories? Audiophile observation?

 

[Juhazi]

Audio market knows, that peope have different desires and even cultural differencies in what kind of (measured) reproduced sound they like. And this changes by time. One important thing is that USA is still one of the leading markets with lots of large and small manufacturers. Japan and GB used to be big and "independent" too, with different aims and styles, but the whole field is nowdays much more even and diffuse compared to 1960-80s. Still, in each and every country and culture, each customer has her own desires and likes. Science and ideals are elementary for development, but we need variation to satisfy all markets and customers! Same with food, clothes, cars, computers etc.

Harman is a global company owned by Samsung now. They have installed several listening rooms around the world and are seriously trying to make good sound quality and get customer satisfaction. They have also hired many of the best designers and openly published lots of reseach. No one else comes even near. But this doesn't mean that everyone likes all their products.

 

[andreasmaaan]

I don't disagree with O'Toole's design principles. I think they are fine. But people are taking it here as the gospel without independent verification by another lab. They undoubtedly have some flaws in their approach as do all research groups.

I was a PhD student many years ago when one of my classmates published an article in a scientific journal announcing the discovery of a new chemical element. Only a few years later did people discover there was academic fraud and dishonesty. The paper was retracted. I'm not suggesting O'Toole of such things. Generally, scientists don't accept scientific conclusions as definitive and established if only one lab has performed the tests. It must be repeated independently in order to become established.

Yes, we often have this problem in this field because (1) the resources are fewer than in other scientific fields and (2) a lot of research is done informally and is unpublished.

A couple of points, however:

• The Harman research has been repeated (IIRC) in a number of different listening rooms with sample groups from a number of different continents. There was not a great difference in outcomes from sample group to sample group.
• There is no competing research that tends to discredit the Harman research (that I know of).
• The findings accord with a much larger and more established body of research into the human auditory system (for example, research into the precedence effect, thresholds of detection for linear distortion, etc.).

I would also like to see these studies repeated under more varied conditions.

In the meantime, I see this research as (by far) the most comprehensive and convincing in the field (I know you agree with this). I am interested in looking at it critically, however. But I'd like to hear specific criticisms of the research methodology, and/or counter-hypotheses, rather than generalised criticisms such as those you've made so far.

However, if I interpret your basic point here to be "these findings are not gospel and further research is warranted", then I agree with you.

 

[noobie1]

Audio market knows, that peope have different desires and even cultural differencies in what kind of (measured) reproduced sound they like. And this changes by time. One important thing is that USA is still one of the leading markets with lots of large and small manufacturers. Japan and GB used to be big and "independent" too, with different aims and styles, but the whole field is nowdays much more even and diffuse compared to 1960-80s. Still, in each and every country and culture, each customer has her own desires and likes. Science and ideals are elementary for development, but we need variation to satisfy all markets and customers! Same with food, clothes, cars, computers etc.

Harman is a global company owned by Samsung now. They have installed several listening rooms around the world and are seriously trying to make good sound quality and get customer satisfaction. They have also hired many of the best designers and openly published lots of reseach. No one else comes even near. But this doesn't mean that everyone likes all their products.

I’ve worked at several large companies similar in size to Samsung and was intimately aware of proprietary research data, I know Toole did his research before Harman was bought out by Samsung. Based on my experiences, companies never give out important proprietary data for free. I suspect we don’t have the full story when it comes to speaker measurements because it doesn’t serve Harman any good to share them.

 

[noobie1]

Yes, we often have this problem in this field because (1) the resources are fewer than in other scientific fields and (2) a lot of research is done informally and is unpublished.

A couple of points, however:

• The Harman research has been repeated (IIRC) in a number of different listening rooms with sample groups from a number of different continents. There was not a great difference in outcomes from sample group to sample group.
• There is no competing research that tends to discredit the Harman research (that I know of).
• The findings accord with a much larger and more established body of research into the human auditory system (for example, research into the precedence effect, thresholds of detection for linear distortion, etc.).

I would also like to see these studies repeated under more varied conditions.

In the meantime, I see this research as (by far) the most comprehensive and convincing in the field (I know you agree with this). I am interested in looking at it critically, however. But I'd like to hear specific criticisms of the research methodology, and/or counter-hypotheses, rather than generalised criticisms such as those you've made so far.

However, if I interpret your basic point here to be "these findings are not gospel and further research is warranted", then I agree with you.

Thanks for sharing. I’m not all that literate in this field so I appreciate any chance to learn.

My main concern with the methodology is the use of single speakers when most consumer listening is done in stereo. In my opinion, you’re always going to have a disconnect if your testing conditions are reduced from real world scenarios. I’m worried that there is an inherent bias in a single speaker setup toward their own design principles. Moreover, if the trend is that adding a second speakers reduces the listeners ability distinguish these characteristics, what happens when you add a third or fourth speaker? If trend continues, the listener will no longer have a preference for Toole speakers or worse, they will prefer non-Toole speakers. What happens to the design principles then? Also, I perceive that people refer to Toole principle as the gold standard but I don’t believe it’s been exemplified. In other words, perhaps you can slightly compromise one of their core principles in favor of another design parameter that may be even more preferred in stereo setups.

 

[andreasmaaan]

My main concern with the methodology is the use of single speakers when most consumer listening is done in stereo. In my opinion, you’re always going to have a disconnect if your testing conditions are reduced from real world scenarios. I’m worried that there is an inherent bias in a single speaker setup toward their own design principles. Moreover, if the trend is that adding a second speakers reduces the listeners ability distinguish these characteristics, what happens when you add a third or fourth speaker? If trend continues, the listener will no longer have a preference for Toole speakers or worse, they will prefer non-Toole speakers. What happens to the design principles then?

I think others have explained why this specific criticism is (IMHO of course) not a strong one. I won't repeat what others have said here already, but Dr Toole is a member here and perhaps he can weigh in himself at some point and add something that others have not adequately managed to.

To restate the point anyway, Toole and Olive found that (a) listeners were able to distinguish more subtle differences between speakers in mono and (2) there was no significant difference in listener preference whether the test setup was in mono or stereo.

Have you read Toole on this particular subject? You might find that doing so allays this specific concern of yours.

 

[noobie1]

I have Toole’s latest book and I’ve quoted him in this thread. Based on my reading, the preference is always for Toole speakers. This preference is very strong in mono mode but is less so in stereo. In my humble opinion, if you design a human preference test in which test subjects always choose one preference over another, the methodology itself is highly specious.

If there were such an overwhelming preference for Toole speakers, it should dominate the market. People here point to sighted bias which I believe is real but I don’t think it credibly accounts for everything. There are of course other factors but a 100% preference should be more compelling than any of these secondary considerations.

Again my background is in physical science and most ppl in hard sciences view psycho acoustics or psycho anything with a lot of skepticism.

 

[svart-hvitt]

Interesting discussion.

One faulty logic I observe on ASR is to conflate Toole with Harman. Toole is a person with his own set of values. The corporation Harman’s prime value has always been to make profit. Profit can make scientific insights into an afterthought.

Toole is a merited engineer. I believe noone has more AES recognition than Toole has. But there is some idolators on ASR, interpreting everything Toole has written as Truth and thinking Toole’s affiliation with Harman make Harman into a company making products of unquestionable scientific quality.

Please note, this is no criticism of Toole per se. It is a criticism of idolatry and some people’s way of reading and interpreting research.

 

[March Audio]

I have Toole’s latest book and I’ve quoted him in this thread. Based on my reading, the preference is always for Toole speakers. This preference is very strong in mono mode but is less so in stereo. In my humble opinion, if you design a human preference test in which test subjects always choose one preference over another, the methodology itself is highly specious.

If there were such an overwhelming preference for Toole speakers, it should dominate the market. People here point to sighted bias which I believe is real but I don’t think it credibly accounts for everything. There are of course other factors but a 100% preference should be more compelling than any of these secondary considerations.

Again my background is in physical science and most ppl in hard sciences view psycho acoustics or psycho anything with a lot of skepticism.

No, to quote the previously linked article the preference is for:

a speaker exhibits flat and linear on-axis frequency response with consistent off-axis performance to preserve critical early reflections, then the speaker will score very highly in blind listening tests and also provide more consistent performance from room to room. The speaker must also be free of pesky resonances and the transition between drivers must be as seamless as possible.

Specious? Well you need to actually say why. What are your specific criticisms of the testing methodology? If you cant articulate them here then you have no basis for the comment.

The market is extremely disconnected from actual product performance. You are right, Its not just sighted bias, although this on its own destroys any chance of performance being evaluated accurately. Its also the dealers bias and obviously the availability of speakers for individuals to actually audition, etc etc. I dont know what you mean by 100% preference being more compelling.

Psycho acoustics is as "hard" a science as any other. Why do you think MP3 and other lossy codecs work?

Can you please listen to Tooles comments about mono stereo preference. Its at 8:20 to 9:50. There is NO difference in the preference outcome. Try looking at it in a different way, as you seem to be repeating this one. The preference is not different in stereo, its just that the more thats going on, the less easy it is for listeners to home in on the problems with the speakers reproduction. Thats not really surprising is it? You could say that mono increases the sensitivity of the test.

 

[March Audio]

Interesting discussion.

One faulty logic I observe on ASR is to conflate Toole with Harman. Toole is a person with his own set of values. The corporation Harman’s prime value has always been to make profit. Profit can make scientific insights into an afterthought.

Toole is a merited engineer. I believe noone has more AES recognition than Toole has. But there is some idolators on ASR, interpreting everything Toole has written as Truth and thinking Toole’s affiliation with Harman make Harman into a company making products of unquestionable scientific quality.

Please note, this is no criticism of Toole per se. It is a criticism of idolatry and some people’s way of reading and interpreting research.

I think Toole himslef has commented on the challenges of dealing with Harman marketing dept . Final products wont be based purely on the best technical design. They are also designed to sell and must appeal to the publics desires and conceptions (and misconceptions) of whats good. A prime example is the earlier discussion of wide baffles. Even if they are the right thing to do (no comment from me about that) I think most of us would agree the thin box will probably win on aesthetic grounds.

Im not sure I agree with the comments about idolators on ASR. Its just that there appears to be little, maybe no contradictory scientific views on what Toole has presented. What is presented is entirely un-contentious IMO. Are we surprised that speakers designed to those principles are found to be preferred? If so I dont know why as those principles are entirely logical and concurr with wider research.

 

[Cosmik]

It's still the case that no one can prove that a person who knows they are taking part in a listening test doesn't have changed perception or reduced critical faculties. Without a Truman Show experiment this will always remain the case.

And testing existing speakers means that many variables are in play, not just the on-axis vs. off-axis characteristics (EQ, distortion, phase, delay, power compression, resonances, etc.). This would explain (among many other reasons that may never be teased apart) why listening test-based research isn't completely predictive.

Just looking at horns versus direct drivers is a case in point. The 'aggregate' performance of a horn may be exemplary, but listeners anecdotally report that they sound 'coloured' - and I think they are, too. It may not even be measurable, in that while the characteristic can be captured by a laptop and mic, the coloration is related to microscopic head movements and the subtle dynamic interactions in the 'fur' of the phase and frequency response graphs while the listener is moving, and is not a visible characteristic on the laptop screen. Phase and frequency response may simply not be a useful way of interpreting certain transducer characteristics.

But get some listeners and ask them to sway about in a prescribed oscillatory motion while listening to Bach and then try to get them to quantify their experiences..? It doesn't sound like fun.

 

[andreasmaaan]

In my humble opinion, if you design a human preference test in which test subjects always choose one preference over another, the methodology itself is highly specious.

IMHO you have it backwards here. The outcome can't render the methodology specious.

I take specious to mean something along the lines of "superficially plausible, but in fact wrong". But what you seem to be saying is that, because the outcome is superficially implausible (i.e. too perfect), the methodology must be wrong.

The methodology might be wrong, but looking simply at the outcome won't help us determine whether or not this is so. To make this determination, we need to look at the methodology per se. You've raised the stereo/mono point as a potential methodological issue. That's a way of looking at the problem that makes sense, in my view (even though I don't share this particular concern).

If there were such an overwhelming preference for Toole speakers, it should dominate the market. People here point to sighted bias which I believe is real but I don’t think it credibly accounts for everything. There are of course other factors but a 100% preference should be more compelling than any of these secondary considerations.

Essentially, you're asserting here that how a speaker sounds is the most important factor in its market success. What evidence do you have for this?

Again my background is in physical science and most ppl in hard sciences view psycho acoustics or psycho anything with a lot of skepticism.

Again, I feel that this is a criticism that is so general in scope as to be nebulous. Asking what specifically is amiss in the research is a more fruitful approach than making a vague attempt to undermine the credibility of the entire field, surely?

 

[andreasmaaan]

...the coloration is related to microscopic head movements and the subtle dynamic interactions in the 'fur' of the phase and frequency response graphs while the listener is moving, and is not a visible characteristic on the laptop screen.

What do you mean by "fur"? And do you have any evidence or any theoretical reason why this would be the case?