• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Consumer evaluation strategies personal evaluation or relying on external evidence

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
140
Location
Germany
#1
In the "uptone thread" a post from Thomas Savage:
<snip>

He's basically saying he has no clue whether it works or not but it's not expensive so you can try it..

WE know the fallibility of subjective listening conclusions of this kind so I'd recommend steering clear based on the evidence being presented here.

Give the $300 to a veterans charity instead.
emphasized the fallibility of subjective listening conclusions and concluded that without some evidence for a benefit it would be better to avoid any evaluation.

Which raises the interesting question what a customer should do.
Let´s assume some measured evidence is available, but as often it´s below the known thresholds of hearing, and even some positive results from a controlled listening test is given; what should a consumer do?
Buy the device "blind"?
Or should he/she nevertheless do a listening test, because no one knows, if the conditions of the mentioned test results are relevant for their situation?

As ThomasSavage mentioned, there always is a risk of subjectivity (bias effects are everywhere, even in more socalled objective attempts after all), but would that risk be smaller or even larger given the evidence?

If personal perceptual evaluation is still a good advice, why should that depend on the additional evidence, given that there is a money back garantuee?

I would not restrict it to devices like the mentioned audio tweaks/gadgets, as even for differences above the known thresholds of hearing, the fallability is still there and furthermore, any intersubject personal differences and preferences will still exist.

So, what to do?
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,013
Likes
1,856
Location
UK
#2
It's not just a question of money. We all have limited time, so to spend it listening for phantoms seems a bit like a waste. And every time you do such a 'test' you lose a bit of your ability to simply enjoy listening to music!

...there is a money back garantuee
I don't know why people don't evaluate the likelihood of a particular gizmo doing anything based on their own experiences of 'people' - and indeed their own inner selves. Didn't the "money back guarantee" thing make you laugh back when you were about 17? It did me. Even in my youthful naivety I understood that if you are selling people phantoms, a money back guarantee solves virtually all problems: your conscience is clear; no one can sue you for ripping them off; the authorities won't arrest you for a scam; the existence of the guarantee adds to the power of the phantom, making it less likely that anyone will want their money back anyway; the fact that most people don't want their money back even allows you to convince yourself of the efficacy of your own phantom-based product if you need that particular crutch.
 
Last edited:

fas42

Major Contributor
Joined
Mar 21, 2016
Messages
2,818
Likes
139
Location
Australia
#3
Ummm ... listen to it. If it improves your system it was most likely worth the money - it's your money, and your pleasure in the listening. Strangely enough, people buy TV sets after "sighted" evaluation - somehow, in this area, using your senses is OK - but, not with one's ears ... funny ol' world ...
 

Don Hills

Addicted to Fun and Learning
Joined
Mar 1, 2016
Messages
614
Likes
274
Location
Wellington, New Zealand
#4
Ummm ... listen to it. If you believe it improves your system it was most likely worth the money - it's your money, and your pleasure in the listening.
Fixed it for you...

Strangely enough, people buy TV sets after "sighted" evaluation - somehow, in this area, using your senses is OK - but, not with one's ears ... funny ol' world ...
Strangely enough, most people never bother to change the settings from the "shop demo" (dynamic, etc) settings to more natural settings when they get the set home.
 

Sal1950

Major Contributor
The Chicago Crusher
Patreon Donor
Joined
Mar 1, 2016
Messages
5,923
Likes
3,594
Location
Central Fl
#5
Fixed it for you...
Strangely enough, most people never bother to change the settings from the "shop demo" (dynamic, etc) settings to more natural settings when they get the set home.
You must be talking about TV's with the delusional.
TV's are a perfect example of folks not being able to tell quality reproduction from garbage. They go to the showrooms and pick the one set to the brightest, bluest "flamethrower" picture mode. Then they get one home a set it to the "Vivid or Sports" mode for watching movies. I hear this all the time at AVS, stuff like "the cinema mode is so boring" or "I had the set calibrated but I didn't like it, too dark and soft, went back to Sports mode. Don't understand this big deal over calibration" LOL
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
6,702
Likes
6,735
#6
You must be talking about TV's with the delusional.
TV's are a perfect example of folks not being able to tell quality reproduction from garbage. They go to the showrooms and pick the one set to the brightest, bluest "flamethrower" picture mode. Then they get one home a set it to the "Vivid or Sports" mode for watching movies. I hear this all the time at AVS, stuff like "the cinema mode is so boring" or "I had the set calibrated but I didn't like it, too dark and soft, went back to Sports mode. Don't understand this big deal over calibration" LOL
I purchased a BenQ projector which claimed to go a little bit extra on the settings and conform to Cinema rec 709 protocols. I do basic calibration for my friends on their projectors and they are very happy. This being made to standards is why I purchased this projector. I turned it on and went WOW! So did my friends. It couldn't be improved upon in basic calibration and came close to perfect results in testing it. Just plug it up and go. Couple of my friends wondered why they weren't all setup this way.

Later I purchased a new desktop monitor and paid extra for one of the Dell's with factory calibration to standards. I plugged it in, turned it on and went WOW!! I have never touched adjustments on it. I am so grateful that better video hasn't been invaded by the audiophool mentality. I will admit to subscribing to the Perfect Vision for a time. Anyone remember it?

Right now in audio, if we agreed upon some basic standards there are no components that would need cost more than $1000 which wouldn't meet or exceed those standards excepting speakers and amplifiers (and I aint so sure about amplifiers). That is what is so disturbing about MQA. We have the ability to hear as close as is possible the sound of the master if the record companies would simply make the unadulterated master available. We need nothing else.

Even when people wish to use special processing for special effects and different than life-like results (sound or video) that works just fine as long as the replay chain meets some standards. My projector can show photos and naturally filmed movies in excellent fidelity to reality. It also shows special effects exhibitions equally well.

So personal evaluation or external evidence? My personal evaluation indicates external evidence beats the HELL out of just personal evaluation. So I can short cut the messiness of personal evaluation and get good results easily. Plus I no longer need to personally, continually, re-evaluate or revalidate those results. I can just enjoy movies or music without worry.
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
6,702
Likes
6,735
#7
It's not just a question of money. We all have limited time, so to spend it listening for phantoms seems a bit like a waste. And every time you do such a 'test' you lose a bit of your ability to simply enjoy listening to music!



I don't know why people don't evaluate the likelihood of a particular gizmo doing anything based on their own experiences of 'people' - and indeed their own inner selves. Didn't the "money back guarantee" thing make you laugh back when you were about 17? It did me. Even in my youthful naivety I understood that if you are selling people phantoms, a money back guarantee solves virtually all problems: your conscience is clear; no one can sue you for ripping them off; the authorities won't arrest you for a scam; the existence of the guarantee adds to the power of the phantom, making it less likely that anyone will want their money back anyway; the fact that most people don't want their money back even allows you to convince yourself of the efficacy of your own phantom-based product if you need that particular crutch.
Yes, the worst money back guarantees or LIFETIME guarantees are on inexpensive products. You know, the guarantee says if ever dissatisfied send the product to some address for a full refund. Only shipping is $15 and the product cost $10. I believe most boner pills have such money back guarantees. The market for those is in the billions. Others are like Leupold riflescopes. Pretty good scopes, and always touting the warranty. In time they had to reign it in some as the warranty claims got a bit much.

I also remember back in the early 1980's Honda cars had the most minimal warranty. Others touted the long warranties they had. Honda could do that because you pretty much just didn't have problems. If the car is of good enough quality you don't need much warranty. Back then most Hondas were sold before unloaded from the truck. My first Honda was a 1980 Accord. It did have a couple problems crop up around 185,000 miles. I will say going from nothing except muscle cars to the Honda Accord was an adjustment. The 30 mpg vs 12 and simply no problems were not among them. Never thought I could be happy with a car that had a 93 mph top speed. But I was. Of course that was when Honda's motto was we make it simple. They did, but no longer do. Back then it was AC or no, auto or no, two models to pick. Then maybe 4 colors to choose from. Mine was a beautiful metallic chocolate brown which I must tell you would not be on my top 50 choices for a color. And my Honda motorcycle had about the same power as my Honda car. Still it was better than just okay.
 

RayDunzl

Major Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
7,170
Likes
3,594
Location
Riverview, FL
#8

RayDunzl

Major Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
7,170
Likes
3,594
Location
Riverview, FL
#9
Yes, the worst money back guarantees or LIFETIME guarantees are on inexpensive products.
I bought some 32 gallon trash cans (with a 10 year warranty) to carry yard waste to the curb:

"Any use of the Product for purposes other than the storage of goods will void this limited product warranty."

upload_2017-8-7_4-31-10.png

It's a "refuse" container, but can only be used for storing "goods".
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
140
Location
Germany
#10
It's not just a question of money. We all have limited time, so to spend it listening for phantoms seems a bit like a waste. And every time you do such a 'test' you lose a bit of your ability to simply enjoy listening to music!
Thomas Savage´s argument wasn´t so much about the money but more about the fallability; but anyway, yours seems to be another argument, more related to the limited time available to mortals, but focussing on "phantoms", which neglects the cases in which external reference is available.

The "lose some of your ability" line seems imo to be a non sequitur, but maybe i don´t understand it.

I don't know why people don't evaluate the likelihood of a particular gizmo doing anything based on their own experiences of 'people' - and indeed their own inner selves. Didn't the "money back guarantee" thing make you laugh back when you were about 17? It did me. Even in my youthful naivety I understood that if you are selling people phantoms, a money back guarantee solves virtually all problems: your conscience is clear; no one can sue you for ripping them off; the authorities won't arrest you for a scam; the existence of the guarantee adds to the power of the phantom, making it less likely that anyone will want their money back anyway; the fact that most people don't want their money back even allows you to convince yourself of the efficacy of your own phantom-based product if you need that particular crutch.
All that sounds :) plausible, but is it true? Is there external evidence available?

Reading it i got the impression, that it´s just the exchange of one belief system for another belief system.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
140
Location
Germany
#11
<snip>

Right now in audio, if we agreed upon some basic standards there are no components that would need cost more than $1000 which wouldn't meet or exceed those standards excepting speakers and amplifiers (and I aint so sure about amplifiers). That is what is so disturbing about MQA. We have the ability to hear as close as is possible the sound of the master if the record companies would simply make the unadulterated master available. We need nothing else.
Although you didn´t directly answer the question, what consumers should do, i think you´re advocating the "buy blind" approach for some devices like DACs (after agreement to some basic standards), but what about the other things like loudspeakers, room acoustic treatment (even enhancing software called "room correction")?

Given the experiences over the last decades with digital gear, agreement of those basic standards seems to be non existent. Looking at the standard measurements published for a lot of DACs, none of the problems we have seen in different measurement setup was predicted by those numbers available.

So personal evaluation or external evidence? My personal evaluation indicates external evidence beats the HELL out of just personal evaluation. So I can short cut the messiness of personal evaluation and get good results easily. Plus I no longer need to personally, continually, re-evaluate or revalidate those results. I can just enjoy movies or music without worry.
Would you expand that to the other items, i´ve mentioned above?

Everything in my setup was purchased that way.
Relying on external evidence or recommendations by others?
 

RayDunzl

Major Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
7,170
Likes
3,594
Location
Riverview, FL
#12
Relying on external evidence or recommendations by others?
Manufacturer's statements and specifications, expected device functionality, my own knowledge of "how things work", other user (online) experiences, price, availability, whatever I can find to assist my decision.

It's not a formal procurement process.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,013
Likes
1,856
Location
UK
#13
All that sounds :) plausible, but is it true? Is there external evidence available?

Reading it i got the impression, that it´s just the exchange of one belief system for another belief system.
If you google "astrology money back guarantee" you will get plenty of hits. It might be possible to argue that this, in itself, is external evidence of what I was suggesting. But... I rather think that you are of the persuasion that any claim made by anyone is plausible until checked out with a DBT. A $10,000 wire will make your digits sound better? Only a DBT can prove otherwise. Alignment of the stars will make your digits sound better? Only a DBT..., etc.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
140
Location
Germany
#15
@Cosmik,

is imo a logical fallacy; if charlatans using "money back" does that mean that everyone using "money back" is a charlatan?
Wrt to controlled listening tests; as we know (at least should know by now) they can´t "proof" something, all you get are probabilities.

Furthermore, the conditions for my questions were mentioned also that it isn´t just a matter of "audio mystery" but of decision strategy for just every product. If the argument is "perceptual fallability" it , as we know, matters also when, for example, looking for a new loudspeaker or new/additional room treatment, "room enhancing software" ....
 

Thomas savage

Power hungry desperado
Moderator
The Watchman
Patreon Donor
Joined
Feb 24, 2016
Messages
7,269
Likes
5,257
Location
uk, taunton
#16
@Cosmik,

is imo a logical fallacy; if charlatans using "money back" does that mean that everyone using "money back" is a charlatan?
Wrt to controlled listening tests; as we know (at least should know by now) they can´t "proof" something, all you get are probabilities.

Furthermore, the conditions for my questions were mentioned also that it isn´t just a matter of "audio mystery" but of decision strategy for just every product. If the argument is "perceptual fallability" it , as we know, matters also when, for example, looking for a new loudspeaker or new/additional room treatment, "room enhancing software" ....
There's a certainty of self deception, fallibility and faulty conclusion Jacob with all human subjective appraisals of audio devices.

When there is a choice imo it's best to stick to measurements in determining fidelity BUT preference in terms of things like speakers and room correction etc is reliant on subjective appraisal. This is far from ideal but that's life.

The conclusions audiophiles ( and it's only audiophiles as 'normal ' humans just accept the sound coming from a device more or less) make reminds me of those who wish to assign human feelings and meaning onto animal behaviours.

Johnny Morris and the like.. it's very appealing but it's total bollocks.

There's 2 certainties in life, death and doubt, the rest is just the best we can do.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,013
Likes
1,856
Location
UK
#17
@Cosmik,
is imo a logical fallacy; if charlatans using "money back" does that mean that everyone using "money back" is a charlatan?
No, it is just a question of probabilities...

Wrt to controlled listening tests; as we know (at least should know by now) they can´t "proof" something, all you get are probabilities.
Yes, probabilities and standalone preferences that do not in any way indicate a 'better' system or any applicability outside the experiment. I am left wondering what the point of them is!
 

RayDunzl

Major Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
7,170
Likes
3,594
Location
Riverview, FL
#18
Although you didn´t directly answer the question, what consumers should do, i think you´re advocating the "buy blind" approach for some devices like DACs (after agreement to some basic standards), but what about the other things like loudspeakers, room acoustic treatment (even enhancing software called "room correction")?
Having recently purchased loudspeakers (JBL LSR 308), room treatment (bulk chunks of Roxul 7 x 24 x 48 batts) , a disc spinner (Tascam CD200), a convolver (miniDSP OpenDRC-DI) and software (AcourateDRC), I am comfortable with the idea that I can "buy blind" and not be disappointed.

Maybe I'm too easily amused.

I might have been able to hear the speakers beforehand, if one of the local (30+ miles) pro-audio shops happened to carry them and have a set to listen to (unlikely), the Roxul was a special order from Home Depot, the miniDSP came from Hong Kong, and the software from Germany.

I don't think it is even possible for us Average Joes living out in the boonies to audition things before purchase anymore in our Nu-Perfect Global Economy, if you don't live in New York or Tokyo or Berlin.

Choices become a thought experiment - then you make a commitment, and either live with the results or discover the problems with the vendor's return policy.

Next I'm going to build a new PC. What else can I do but go by specs and hope for the best as I select and order a motherboard, memory, CPU, and a SATA expander? Audio parts and pieces inhabit the same realm now, in my opinion.

At least at the level at which I participate.
 
Last edited:

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
6,702
Likes
6,735
#19
Although you didn´t directly answer the question, what consumers should do, i think you´re advocating the "buy blind" approach for some devices like DACs (after agreement to some basic standards), but what about the other things like loudspeakers, room acoustic treatment (even enhancing software called "room correction")?
Loudspeakers would best be purchased after listening to them. As Ray has said, that isn't often possible in most places now. So reading the specs of speakers, and experiences of those who have them along with a chance to return them if they turn out not to suit one is what I would do. I purchased some LSR305s this way. It is notable they are the only company I am aware of or that makes it known they have a standard to which they are designing. Having heard a few of their designs now it is my opinion this works. Which is not to say all sounded the same. They did have family resemblances that were well balanced. That is the benefit of having a standard to shoot for. They have developed a well considered loudspeaker standard and others lack that.
Given the experiences over the last decades with digital gear, agreement of those basic standards seems to be non existent. Looking at the standard measurements published for a lot of DACs, none of the problems we have seen in different measurement setup was predicted by those numbers available.
I have no idea what you are talking about here. If it is some sighted listening impressions just say so. I'll ignore the issue. Audio electronics should accurately with low distortion, low noise and flat response reproduce the signals they are fed. DACs are widely available at relatively low cost that accomplish this to nigh on theoretical perfection.
Would you expand that to the other items, i´ve mentioned above?



Relying on external evidence or recommendations by others?
I rely on the best evidence available. For items that have some goal or standard those getting closest to that are better. Whenever possible I do not wish to rely on recommendations by others. I prefer external evidence. Some areas and some products are simply not that cleanly delineated. So it will be a mix. If it becomes a preference, then you'll need to figure out what your preferences lean toward and use recommendations of those with similar preferences. Now there are other aspects of products like reliability. If I look on Amazon and actual owners liked how a product works, but are saying it went kaput in a few weeks or it broke or it wore out too soon then I'll of course pay attention to that. I will repeat, I'll use best evidence available.
 

fas42

Major Contributor
Joined
Mar 21, 2016
Messages
2,818
Likes
139
Location
Australia
#20
Fixed it for you...
I'm touched ...

Strangely enough, most people never bother to change the settings from the "shop demo" (dynamic, etc) settings to more natural settings when they get the set home.
Strangely enough, most audio systems I come across are in "shop demo" mode - lots of impressive bass banging, accurate treble reproduction is almost non-existent - you have to sit in exactly the right place, otherwise it sounds pretty awful ... the last thing you can say about it is that it's "natural"; that it always ticks the boxes no matter where you listen to it, and the degree of attention one pays to it.

So, audio is usually desperately in need of "calibration" - and very rarely gets it, IME ..
 
Top Bottom