• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Master Thread: Are measurements Everything or Nothing?

So, imo, there are no parts of the reproduction chain where sighted listening is any good at all, except speakers.
And is that even any good? Harman's early results indicate maybe not or at the most really limited. REALLY LIMITED.

But for all of history until recently, there simply was not really any other way than sighted listening for most consumers. Effectively still isn't if you wonder about speakers for which spinorama data is not available. We know without listening (as much as I love them) panels are problematic. Anything with a 1st order crossovers have off axis lobing issues. Some designs really should now be discounted simply for those reasons.

The speaker thing is a quandary. Spinorama data isn't quite to the point of effectiveness that electronics are measured. And you'll always be effected by looks, and craftmanship. But shouldn't anything that cost very much at all also include good design? Yes. It will never be that way for all speakers just like there are expensive vacuum tube amps of poor performance that find an audience. But Toole has literally wrote the book for anyone serious about speaker manufacturing.

Some dealers, those few that remain, actually wouldn't find it outrageous to set up a good room with the ability to switch speakers. Have in that room an acoustically transparent curtain to allow customers to listen without knowing which speaker is which. Do a process of elimination to find the one you like.
 
And of course, the speakers would have to be level-matched. :p:p

Jim
Yes. Toole did that with pink noise rolled off below 500 hz at 12 db per octave and above 2 khz at 12 db per octave. It works pretty well.
 
The power of suggestion had nothing to do with these examples?

Jim

That’s always a possibility. But, It can be a bit too facile to always resort to that, for the many reasons I’ve been giving.

Remember, I have not just spoken about hearing someone else’s description first, and afterward forming an impression. I’ve mentioned numerous instances here (and another threads) where the impression is formed first through one’s own experience with a loudspeaker, and then finding another person’s description of that loudspeaker to be very accurate to your own impressions.

I mentioned for instance, how I formed my sighted impressions of the PMC loudspeakers before reading that Kal described precisely the issues I heard (also backed up my measurements). I mentioned how when I visit my reviewer pal, He has formed his own impressions with loudspeaker before I give mine, and they typically match very well. (and those loudspeakers often end up being measured.)

And I mentioned that other audiophiles have found my descriptions of loudspeakers that they already knew well, were accurate to what they heard as well.

In the specific example I gave, this other person already owned the Harbeth loudspeaker, knew it well, and found my description was very accurate. That helped him gain some trust that I might be describing the Joseph speakers accurately as well. And it turned out he was quite sure I did when he heard them for himself. Did it just happen that my description of Harbeth speakers was accurate, but my description of the Joseph speakers was inaccurate, but then he just imagined they sounded as I described anyways? Possible. But it’s also quite plausible I simply described both accurately.

(how far can the “ power of suggestion” explanation for loudspeaker descriptions be pushed? If I came on here and said, I heard some Neumann KH420 speakers and I gave a description like “lean, weak bass, honky mids and muffled highs” are the owners of that loudspeaker here going to think “ you know what he’s right!” Of course not. They will immediately protest that my description is nothing like what they hear. Not just it doesn’t measure that way, but it also clearly would not match the sound they actually experience. I could also have easily described the Joseph audio speakers or Harbeth speakers so inaccurately that the individual surely would have protested I was off my rocker)
 
Last edited:
Hasn't Dr Toole already answered Hooper's question?

Thanks! I knew that was buried here somewhere!

FT:

“It is not that sighted tests are useless, it is simply that blind tests yield substantially more repeatable (i.e. statistically useful), judgements from a wide population of listeners (most people with “normal” hearing). ”

Works for me! Pretty much what I’ve been saying.
 
Yet later in the same post, he says ...

"Personally, from the perspective of sound quality, I would trust my interpretation of a “spinorama” presentation of anechoic measurements more than my subjective judgment in a typical consumer listening circumstance."

Yup. Exactly. This is why I’ve said the approach of many on ASR, mirroring information from Toole et al, makes excellent sense.

"It remains a problem for consumers that reliable unbiased listening tests are generally not available, so opinions about sound quality are to some extent contaminated by non-auditory factors. Some people may be more resistant than others to such influences, but questions remain."

Yep, that fits in with my perspective.


Your interpretation might be correct, but as for me, I think the good doctor is simply being overly polite. :)

Yes, and no, I think.

No, in the sense that Dr. Toole seems to recognize that sighted listening cannot be totally useless in the big picture. It’s certainly “ useless” if you are looking for scientific level verification, which is of course what Dr Toole seeks. But I think it’s obvious he recognizes there would be problems ruling sighted listening “ totally useless” as a practical every day matter.

But yes, he was also being polite I agree.

I mean, not for a second am I arguing that Toole endorses the methods I’ve been describing for my own decisions, and some other audiophiles I’ve mentioned. He’s obviously not endorsing appeoto sighted listening, subjective reviewers, the reports of other audiophiles using uncontrolled listening.
Of course not. His whole point is that there is a better way: You can take lots of the guesswork out and get much more reliable results by looking to scientifically derived data and principles.

So if Toole were to advise me how to buy loudspeakers I think we can be sure that he would not advise me to have gone on the path I have over the years. It’s not that I don’t care about measurements, I do. But I have clearly put more emphasis on subjective impressions than he ever would advise.

My main point being: yeah there’s a lot of bramble and thickets to cut through if you were going to take some subjective reports as informative, but it’s not totally impossible, the process not totally useless, and it’s actually allowed me to have plenty of extremely satisfying purchases and audio experiences. Going “full ASR/Toole” certainly has a hell of a lot going for it. But it’s not the only way to do the hobby, not the only way to satisfaction, and for people like me who truly love sharing subjective experiences, it can feel like a somewhat richer and more subjectively nourishing process.
 
Last edited:
Hasn't Dr Toole already answered Hooper's question?
Yes. Can you recall if it made a big impression on Hooper? If it did, would he be repeating the exact same 'conundrum' today, 2 years later?

Dr Toole's conclusion, after much generous latitude, was to use the Spinorama, and to EQ your bass in-room to get it right. Why? Because sighted listening cannot be trusted.

I hope he didn't waste his time.

So, imo, there are no parts of the reproduction chain where sighted listening is any good at all, except speakers.
And is that even any good? Harman's early results indicate maybe not or at the most really limited. REALLY LIMITED.

But for all of history until recently, there simply was not really any other way than sighted listening for most consumers. Effectively still isn't if you wonder about speakers for which spinorama data is not available. We know without listening (as much as I love them) panels are problematic. Anything with a 1st order crossovers have off axis lobing issues. Some designs really should now be discounted simply for those reasons.

The speaker thing is a quandary.
If I told you that the sighted listening effect works on amps and DACs and cables too, would you see that as a quandary in the choice of those components too?

Spinorama data isn't quite to the point of effectiveness that electronics are measured. And you'll always be effected by looks, and craftmanship. But shouldn't anything that cost very much at all also include good design? Yes. It will never be that way for all speakers just like there are expensive vacuum tube amps of poor performance that find an audience. But Toole has literally wrote the book for anyone serious about speaker manufacturing.
I have several times spoken on ASR of the opportunity to use the data to make a shortlist of speakers that all meet whatever high standards of reproduction that one demands, then use sighted listening to choose between them with one's cognitive biases in full swing. Sighted Listening With Insurance. ;)

However much I say the above, though, it won't satisfy the professional debaters whose real issue is that they can't believe how very much their sighted listening can't be trusted to adjudicate on the sound waves themselves. The elephant in the room.

Some dealers, those few that remain, actually wouldn't find it outrageous to set up a good room with the ability to switch speakers. Have in that room an acoustically transparent curtain to allow customers to listen without knowing which speaker is which. Do a process of elimination to find the one you like.
Good luck. I would be shocked and delighted if you even find a handful of such dealers across a big country. They have to protect the Price Performance Ladder on which their business depends. Furthermore, the test needs a speaker flipper like Harman Labs used.

But even then, actually knowing the exact two products that are being compared is problematic. There is a tendency to straight away hazard a guess as to which is which, and then sighted bias is in play. It's best if you don't actually know the products being compared, and best if at least 3 are compared.

So: if you can inform the dealer of your price range, tell them to pick 3 speakers and don't tell you what they are, and run a blind test, then you might be onto an opportunity. Good luck getting that to happen.

cheers
 
So the anecdotal path to believing Toole's work at Harman.

My home speakers were Soundlab ESLs. I was spending time away from my home for a year. I was recording music for bands. I fortunately just prior to this had my awful M-Audio monitors blow up. I needed a small monitor to work on these recordings. Mostly two mic minimalist at first. Gear space had posters touting the JBL LSR-305s as head and shoulders above anything near its price. I was somewhat aware of the work that Harman had done. I had heard some Salons that were impressive. I gave the 305s a shot. I was there at the recordings and the 305s were impressively good. Impressively accurate in evaluating the recordings. Gave me a little more belief in the work of Toole. I read his book.

A couple years later moved to a new house. Ran across some Revel F12s cheap. They were an early Toole influenced floorstander design. They had gotten unusually good reviews from several magazines. Put them into my new living room and WOW, they were damn good. Easier for the time being vs moving my Soundlabs. I didn't replace them as quickly as I expected they were so good to live with. When I did replace them it was with some Revel F208s. I also used the 305s as they were small and easy to take places being self powered and only needing a preamp or DAC to show some audiophile friends. All were impressed. Most didn't think much of the work of Toole prior to hearing some speakers from his work. One friend outfitted his small theater room with 5 of them and a couple subs. Another used them for an office system. They were cheaper than what they replaced and much more enjoyable. So some people just from hearing them decided they were a bargain of unusually good performance just from sighted listening. Were they influenced by me? I'm sure they were. I had some credibility with them. I doubt that would have lasted long term if the speakers had been poor.

Now I am not bothered by buying speakers with good spins. It was living with them that convinced me however. I would point out even in the subjective press Revels have gotten pretty much universally good reviews. Often with the reviewer's estimation that not only were they good they were exceptionally good for the price. The science works. It even convinces those who just listen naively and sighted.
 
When one advocates sighted listening as useful, it begs the question, "How useful?" By the same token, when someone says that sighted listening is useless, it begs the question, "How useless?"

There are no grades in "useless", either it's useless or it's not useless, and nothing in between.

"How useful" something can be is another thing, as a specific thing can be weighted against other aspects of things, that may or may not be of higher importance than the particular thing we were questioning the usefulness of.
 
How many well-known reviewers deal with bass EQ in any fashion before doing their sighted listening? I can’t think of any, although it’s been a while since I read any of that stuff.

You can see Fremer’s listening space in a widely-circulated video. Ugh.
 
Yes. Can you recall if it made a big impression on Hooper? If it did, would he be repeating the exact same 'conundrum' today, 2 years later?

You’ve been missing the point.

The problem isn’t with Floyd Toole, the problem is with folks like yourself and sometimes others who push the scepticism about sighted listening too far, and use it as a blunt cudgel to dismiss any appeal to sighted listening. So the problem isn’t with Floyd Toole, it’s more about the type of attitude that often flares up in this forum about sighted listening.

Toole said that sighted listening isn’t useless; but listening with scientific controls yields substantially more reliable information.

That of course, has been my position all along. And it avoids the conundrum. The greater reliability using the scientific controls is a GIVEN. But it’s also reasonable to explore why “ sighted listening isn’t useless.” Which brings the question “ OK when is it reasonable to allow for conclusions based on sighted listening?

This is an incredibly pertinent question because most of us aren’t able to do science on our perception all day long, and in the end, we are using sighted listening in our homes.

This is the area where dogmatism against sighted listening, of the type you often produce, leads to problems of the sort I keep pointing out.

Dr Toole's conclusion, after much generous latitude, was to use the Spinorama, and to EQ your bass in-room to get it right.

Completely consistent with my position.

Why? Because sighted listening cannot be trusted.

Well, there you go again. It all depends exactly what you mean when you say sighted listening “ cannot be trusted.”

If you mean “ cannot be trusted to give scientific levels of confidence.”
Then, of course, it would be no disagreement between us whatsoever. We know sighted listening has liabilities. But if you want to suggest as you typically do that sighted listening is essentially useless for vetting audio gear, then you bring in the problems I keep raising.

Again:

1. We all normally recognize that we can’t have scientific levels of confidence for every decision or conclusion we make: normal life would be rendered impossible on such demands. Therefore, we have to recognize we are justified in coming to conclusions which admit of uncertainty, but which are nonetheless reasonable.

2. Once you’ve acknowledged that obvious fact, it is reasonable to explore the type of conclusions we can draw anywhere, including choices of audio gear, that may admit of uncertainty because we aren’t using scientific controls, but which nonetheless are REASONABLE.

That doesn’t mean “ I’ll just assume it’s a bias effect and not reality.” It’s about being justified in provisionally concluding, without scientific certainty, that you are perceiving the real Sonic characteristics of a piece of gear.

And that is the space I have been talking about.

I keep giving all sorts of examples, once again, a whole bunch in this very thread, where it seems reasonable (in the sense outlined above ) to conclude people, including myself, our hearing real Sonic characteristics and not just imagining it

It is notable that you have not shown any of the examples to have been unreasonable.

Instead, you tend to stick to blanket protestations like “ sighted listening can not be trusted.”

So you are never really addressing the issue.
The reason is pretty obvious: you are worried that giving any credence whatsoever to sighted listening opens the gates to the Subjectivist Barbarians, and possibly even more horrific, the fear of giving any legitimacy to any subjective reviews.

This keeps you on your dogmatic tracks, which leads to the conundrums I raise.

Speaking of which:

I have several times spoken on ASR of the opportunity to use the data to make a shortlist of speakers that all meet whatever high standards of reproduction that one demands, then use sighted listening to choose between them with one's cognitive biases in full swing. Sighted Listening With Insurance. ;)

This clearly does not resolve the conundrum.
What point is there to trying to “ meet high standards of reproduction” if you are not, in sighted conditions, going to benefit from it by perceiving it?

In other words, what use is accurate equipment if sighted listening, how you will perceive the results, is inaccurate?

There is no “ insurance” in any such view.
When you’ve been asked to further justify this “ solution” it’s clear

You don’t seem to have

Understood the problem


The more you crank up the skepticism over sighted listening, the less relevant you make “good speaker measurements” to choosing speakers for the sighted conditions in which we actually listen to our loudspeakers.

The only solution is to be reasonable, and dial back your level of scepticism about sighted listening: when you get your great measuring loudspeakers, you will not be left in a helpless morass of sighted delusion: you’ll be able to perceive “ the sound waves” accurately enough to enjoy the real Sonic characteristics you paid for.

And you know very well that, whatever lip service you and others pay here to the untrustworthiness of sighted listening, nobody here is actually assuming they are sighted listening is THAT untrustworthy.
Is quite obvious that the reason people are looking to measure rents and recommending certain speakers is that the listener is expected to be able to actually perceive and appreciate the ACTUAL Sonic qualities of those loudspeakers under sighted conditions
in their home. That goes for you too.

So it is quite inconsistent for you to turn around and wave off the reasonableness of any of the sighted listening conclusions I’ve pointed to as totally untrustworthy and useless, and not noticing how this undermines your ability to be rational about your own decisions.

The elephant in the room.

…. is currently stepping on your toes. ;-)

As I pointed out before, the irony here is that my position actually justifies the relevance of the scientific data from blind testing to choosing loudspeakers. The ASR approach is completely coherent on my reasoning.

Your inability to allow any legitimacy to sighted listening leaves you unable to produce a coherent justification.
 
Last edited:
If I told you that the sighted listening effect works on amps and DACs and cables too, would you see that as a quandary in the choice of those components too?
FWIW, blind testing confirmed my sighted impressions of some CDPs and DACs sounding different were accurate.

But more to the point, in terms of being pragmatic : the smaller (or more likely nonexistent) Sonic characteristics we are talking about, the more it makes sense to demand scientific controls before reaching any conclusion.

We don’t need scientific controls to assume that Taylor Swift fans can accurately perceive the difference between her singing and Donald Trump speaking.

If we get down to things like amplifiers and DACs where Sonic differences can straddle between subtle and nonexistent, then the demand for scientific controls becomes more pressing.

Loudspeakers occupied something of a gray zone. As with anything at all empirical, using science is what you want for the most reliable results, but in this gray zone various sighted conclusions can be justifiable, even if held with a higher level of uncertainty.
 
Last edited:
FWIW, blind testing confirmed my sighted impressions of some CDPs and DACs sounding different were accurate.
It's interesting that your blind testing confirmed your impressions. Since CDPs and DACs are typically designed to be audibly transparent, could the differences you noticed be identified through measurements or configurations rather than listening alone? Maybe you could have skipped the listening session and realized there could be an audible difference?
 
It's interesting that your blind testing confirmed your impressions. Since CDPs and DACs are typically designed to be audibly transparent, could the differences you noticed be identified through measurements or configurations rather than listening alone? Maybe you could have skipped the listening session and realized there could be an audible difference?

Certainly if they were differences to be heard, they would’ve been measurable.

I didn’t have the equipment or the aptitude to do such measurements. I simply owned those three different pieces seem to notice some subtle but distinct Sonic differences.
I would’ve been interested to see measurements if somebody actually took that on, but ultimately, I’d be more interested in the subjective character of the sound than the measurements.

The Meridian CD player seemed to produce Sonic images with slightly more focus, density, and texture making for a slightly more palpable sound. The DAC struck me as sounding slightly more diffuse, lush, and “ darker” in the highs. That’s exactly what I heard in the blind conditions as well.

Ultimately, I preferred the meridian and sold the DAC. (the Sony player was an older one I’ve had from previous years).

These days, I don’t think about differences between DACS
 
Certainly if they were differences to be heard, they would’ve been measurable.

I didn’t have the equipment or the aptitude to do such measurements. I simply owned those three different pieces seem to notice some subtle but distinct Sonic differences.
I would’ve been interested to see measurements if somebody actually took that on, but ultimately, I’d be more interested in the subjective character of the sound than the measurements.

The Meridian CD player seemed to produce Sonic images with slightly more focus, density, and texture making for a slightly more palpable sound. The DAC struck me as sounding slightly more diffuse, lush, and “ darker” in the highs. That’s exactly what I heard in the blind conditions as well.

Ultimately, I preferred the meridian and sold the DAC. (the Sony player was an older one I’ve had from previous years).

These days, I don’t think about differences between DACS
I hate saying this, but double blind tests should exclude anyone over 25. Especially males.

Back in the days of CRTs, my wife hated the mall and couldn’t stand being in a store or department that sold TVs.

I could hear the horizontal oscillator, but it was faint. For her, it was oppressive.
 
I hate saying this, but double blind tests should exclude anyone over 25. Especially males.

Back in the days of CRTs, my wife hated the mall and couldn’t stand being in a store or department that sold TVs.

I could hear the horizontal oscillator, but it was faint. For her, it was oppressive.

Oppressive for me as well. If I walked into a room, whether it was business or whatever, I would know in a minute whether there was a CRT somewhere in the room from the high-pitched tone.
 
Oppressive for me as well. If I walked into a room, whether it was business or whatever, I would know in a minute whether there was a CRT somewhere in the room from the high-pitched tone.
I worked in electronic repair shops for 15 years and never repaired a TV, microwave oven or a VCR. I specialized in the small stuff high accuracy mechatronics. I can smell a TV and hear the noise from them when I am in the vicinity and that does not mean close. CRT TVs are annoying I think too.
 
Yup. Exactly. This is why I’ve said the approach of many on ASR, mirroring information from Toole et al, makes excellent sense.



Yep, that fits in with my perspective.




Yes, and no, I think.

No, in the sense that Dr. Toole seems to recognize that sighted listening cannot be totally useless in the big picture. It’s certainly “ useless” if you are looking for scientific level verification, which is of course what Dr Toole seeks. But I think it’s obvious he recognizes there would be problems ruling sighted listening “ totally useless” as a practical every day matter.

But yes, he was also being polite I agree.

I mean, not for a second am I arguing that Toole endorses the methods I’ve been describing for my own decisions, and some other audiophiles I’ve mentioned. He’s obviously not endorsing appeoto sighted listening, subjective reviewers, the reports of other audiophiles using uncontrolled listening.
Of course not. His whole point is that there is a better way: You can take lots of the guesswork out and get much more reliable results by looking to scientifically derived data and principles.

So if Toole were to advise me how to buy loudspeakers I think we can be sure that he would not advise me to have gone on the path I have over the years. It’s not that I don’t care about measurements, I do. But I have clearly put more emphasis on subjective impressions than he ever would advise.

My main point being: yeah there’s a lot of bramble and thickets to cut through if you were going to take some subjective reports as informative, but it’s not totally impossible, the process not totally useless, and it’s actually allowed me to have plenty of extremely satisfying purchases and audio experiences. Going “full ASR/Toole” certainly has a hell of a lot going for it. But it’s not the only way to do the hobby, not the only way to satisfaction, and for people like me who truly love sharing subjective experiences, it can feel like a somewhat richer and more subjectively nourishing process.
I would say sighted listening tests, reported by unknown people, on online forums, with no other data available, are almost completely useless, unless the test is only meant to convey very gross qualities of the equipment in question.

e.g. If someone subjectively describes the differences between an LS60 and a mid-tier KRK monitor I think that will point us in the right direction.

On the other hand, if we are getting descriptions of sighted listening tests between two 200w Class D amps with good SINAD, what am I supposed to do with that?
 
Oppressive for me as well. If I walked into a room, whether it was business or whatever, I would know in a minute whether there was a CRT somewhere in the room from the high-pitched tone.
Mall stores installed high frequency noise makers to drive teenagers away.
 
I would say sighted listening tests, reported by unknown people, on online forums, with no other data available, are almost completely useless, unless the test is only meant to convey very gross qualities of the equipment in question.

I generally agree. This is why I say that of course it doesn’t make sense to just take any and all sighted reports as informative.

That’s why I like to see if there’s anything that adds credibility to a sighted report. How plausible are the differences this person is describing? Do they have any track record of being on the mark before? If this same person has described other loudspeakers I’m familiar with accurately, and especially if this person seems to be listening for the type of things I care about, then that gains my interest. Likewise for individual subjective reviewers.

I find perhaps a majority of subjective reviews to be poorly written and not very informative, even for my purposes.

And even when it comes to the reviewer I like, it’s not like I’m just swallowing everything they say as accurate or reasonable. There is discretion involved. But over the years has been some reviewers whose opinion I’m quite curious to read.

On the other hand, if we are getting descriptions of sighted listening tests between two 200w Class D amps with good SINAD, what am I supposed to do with that?

Personally, I would ignore it. Far too implausible.
 
If I told you that the sighted listening effect works on amps and DACs and cables too, would you see that as a quandary in the choice of those components too?
I'd say that is where the problem is for anyone who is unaware or a subjectivist true believer. My suspicion is that genuine large differences have a chance of cutting through the sighted listening effect. If there is no difference but one is suspected, that is where we are more likely to instead rely on our eyes, our memories and anything else going to help decide.
 
Back
Top Bottom